[00:28:47] I just killed an image while trying to rename it [00:29:19] the system told me "Fatal error (numbers that I didn't copy in time)", and then I tried to rename it again, and now it's gone gone gone [00:30:37] Reedy: ^ [00:30:45] Dragonfly6-7: what image? [00:32:02] moent [00:32:27] originally it was [[File:Kufuor.JPG]] (on en.wiki) [00:34:35] and I tried moving it to [[File:John Agyekum Kufuor (June 28, 2001).jpg]] [00:35:06] i can't help with this, we need an ops member [00:35:14] apergos: maybe you if Reedy's not around? [00:35:39] I doubt she would be, it's even later there [00:35:48] AaronSchulz: ? [00:35:53] * RoanKattouw doesn't see him at his desk [00:36:54] i don't really know who to poke during non-European times [00:37:18] bd808: Maybe you have some minor insight? [00:37:53] * bd808 reads scrollback [00:38:02] fortunately the image can be easily replaced, since it was derived from an image on commons, but... still, this is an issue that should be remedied. [00:39:07] unfortunately that's above my pay grade. sounds like a bug for sure [00:39:46] Ah well [00:39:54] I don't have access to server logs to debug further [00:40:06] Nor sadly do I [00:40:11] bd808, I can look it up for you [00:41:21] 2013-09-27 00:27:28 mw1049 enwiki: [770efea1] /w/index.php?title=Special:MovePage&action=submit Exception from line 1053 of /usr/local/apache/common-l [00:41:21] ocal/php-1.22wmf18/includes/filebackend/SwiftFileBackend.php: Got InvalidResponseException exception. [00:41:43] so Swift burped [00:41:49] yup [00:41:56] * MaxSem bites AaronSchulz [00:44:39] The error handler in SwiftFileBackend looks like it sort of expects InvalidResponseException [00:45:13] There is code to drop the connection to that swift node when it happens [00:45:41] But losing the file is bad. Dragonfly6-7 can you file a bug? [00:46:16] There's probably one for this already but better too many than too few. [00:47:15] No, I can't. I don't know how it's done, and I don't have the time to acquire the skillset. [00:47:24] I actually lost another image like this just the otehr day. [00:49:23] https://en.wikipedia.org/wiki/File:Kufuor.JPG exists for me. [00:49:41] Redir to File:John Agyekum Kufuor (June 28, 2001).jpg [00:49:57] oh. "No file by this name exists." [00:56:27] Dragonfly6-7: I filed https://bugzilla.wikimedia.org/show_bug.cgi?id=54674 on your behalf. [00:57:39] It would be great if you added yourself as a CC recipient when you get a chance. It would also be good place to add more information if you see this happen again. [01:02:25] I wonder if the file would magically appear if the page was moved back? [01:04:41] bd808 - sorry, assumes a skillset I don't have [01:05:09] ah, and here's the one that got eaten by a hiccup on commons [01:05:11] https://commons.wikimedia.org/w/index.php?title=File:Bf_logo.png&action=edit&redlink=1 [01:07:53] That one seems a little different. The wiki history seems to show that the move worked and then the original page was deleted by another user (or bot). [01:08:20] actually, no [01:08:36] it didn't work, and after a few hours, someone else deleted the dead redirect [01:08:45] Ah. ok [01:09:24] although in that case, I didnt' see any "fatal error" message. [01:10:32] hm [01:10:48] looking at the bugzilla interface... would it be at all useful to give you my browser version? [01:11:46] I suppose it wouldn't hurt, but this seems like server side issues [01:12:25] browser version mostly helps with problems where javascript and DOM rendering are involved [01:12:26] play [01:12:33] *ok*ay [01:14:04] Reporting and updating bugzilla bugs is pretty easy. Only "hard" part is setting up yet another account in the bugzilla app. [01:15:07] * bd808 wishes that everything worked with openid [01:16:39] My better half is calling me to come to dinner. [01:18:20] just tried moving back - no bueno [02:04:19] Hello. I was wondering where I could get information about the location of wmf servers for various wikis (including any backup servers) [02:04:53] http://noc.wikimedia.org/dbtree/ [02:06:10] closedmouth: thanks, but I meant the physical location of the servers [02:06:20] as in, where in the US? [02:09:46] http://meta.wikimedia.org/wiki/Wikimedia_servers [02:11:39] Sid-G: pmtpa is near Tampa, Florida. eqiad is near Dulles, Virginia. ulsfo is near San Francisco, California. [02:11:52] ha, ulsfo is not on that page [02:11:53] The last three letters indicate the nearest airport. [02:11:56] closedmouth: thanks for the link. that links to https://wikitech.wikimedia.org/view/Server_roles which doesn't exist :( [02:12:01] To be fair it's only recently had network connectivity AFAIK [02:12:01] That page is horribly out-of-date. [02:12:20] Elsie: Hey at least it says eqiad is primary starting Jan 2013, and that pmtpa is being decommissioned [02:12:26] So it's not nearly as out of date as other stuff on meta :) [02:12:50] :-) [02:13:06] "It could be worse." [02:13:26] right, to put this in perspective, I just read the wmf response at https://meta.wikimedia.org/wiki/Talk:Privacy_policy#How_Do_We_Protect_Your_Data.3F and hence was wondering which US state's laws would apply [02:14:43] I don't know if that depends on the location of the servers or where the managing entity is [02:14:56] hmm, good point [02:15:04] To make things more confusing, WMF is incorporated in Florida but headquartered in California [02:15:17] ... [02:15:38] Historically that's not unreasonable, it used to be headquartered in Florida and then it moved [02:16:59] RoanKattouw: re: https://bugzilla.wikimedia.org/show_bug.cgi?id=54636, is VE opt-in by default now? [02:17:29] But now that the place where the main servers are has also moved to yet another state (Virginia), I have no idea which state's laws apply [02:17:33] * Sid-G jumps at the chance of pointing YuviPanda to gerrit.wikimedia.org/r/#/c/74649/ [02:17:34] But I'm just a coder, not a lawyer :) [02:17:44] !g 74649 [02:17:45] https://gerrit.wikimedia.org/r/#q,74649,n,z [02:18:09] RoanKattouw: thanks for the info. I'll be asking Geoff that in the discussion :) [02:18:16] YuviPanda: wikitech is probably either misconfigured, or the default configs aren't sane [02:18:23] right. [02:18:25] On any given WMF-hosted wiki, VE is either opt-in or opt-out [02:18:41] enwiki was recently moved from opt-out to opt-in, which may have cleared your preference, thus defaulting to opted out [02:18:43] hmm, I just setup a VE enabled wiki on labs to test... [02:19:12] Hmm, yeah I see both prefs on my localhost as well [02:19:14] That sounds like a bug [02:19:17] The default config should be saner [02:19:27] Sanity? In my VisualEditor? [02:19:29] Arguably the default config should be opt-out [02:19:50] If you went and installed VE on a non-WMF wiki, the behavior you most expect is probably to refresh the page and get VE [02:20:04] indeed. [02:20:56] Have people done that? [02:21:05] Installed VE on random test wikis? Yes [02:21:22] That's why we made the VE<->Parsoid config default to a localhost setup [02:21:37] Out of the box it should Just Work [02:21:39] Has anyone set up VE on shared hosting? [02:21:53] Not without shell access, probably [02:21:58] Because you need nodejs [02:22:10] Shell access doesn't mean you can install things. ;-) [02:22:11] But with shell access it's pretty easy [02:22:21] Assuming you can get node to run [02:22:23] RoanKattouw: have you seen https://wikitech.wikimedia.org/wiki/Labsvagrant? [02:22:32] Can node.js be built locally (in your home)? [02:22:34] (which you can install in your user space if you're desperate) [02:22:36] RoanKattouw: can install a VE + parsoid working setup in about a minute or two [02:22:49] most of which is just puppet running [02:22:50] I have used the Vagrant thing locally, which I was gonna tell Elsie about next [02:23:09] :D [02:23:11] I don't think Vagrant is meant for production wikis. [02:23:24] ori-l has built this thing where you can install VirtualBox and Vagrant, then clone a git repo from gerrit, then run 'vagrant up; vagrant enable-role visualeditor; vagrant provision;' and you'll have VE running on localhost:8080 [02:23:34] It's not really, no [02:23:42] MediaWiki-Vagrant is essentially just a bunch of Puppet configs along with Vagrant itself [02:23:49] you can use the Puppet configs without having to use Vagrant [02:23:50] YuviPanda: Nice [02:23:56] which is what the (misnamed?) labsvagrant does [02:24:10] it just re-uses the puppet files [02:24:12] Though arguably that should not be needed, you should be able to enable/disable puppet roles and run puppet, right? [02:24:33] I guess being able to do that from the command line within the instance is nice [02:24:38] RoanKattouw: so, because operations/puppet and mediawiki/vagrant are written very differently, we can't really do that with the regular labs puppetmaster. [02:24:55] I see [02:24:58] RoanKattouw: so literally, there'll be two uppet chains - one from ops/puppet, and one from mediawiki/vagrant, in labsvagrant machines [02:25:05] Right, OK [02:25:08] Well that's not /too/ bd [02:25:10] *bad [02:25:17] the only place they fight each other is MOTD :P [02:25:29] * bd808 dreams of a day when ops-puppet and vagrant-puppet are compatible [02:25:33] it is trivial to factor that out, but I kinda find it funny... [02:25:35] so left it in :D [02:25:46] plus vagrant has the unintentional unicorn from bd808 [02:26:20] bd808: seen https://wikitech.wikimedia.org/wiki/Labsvagrant yet? [02:26:39] Yeah! I read it this morning [02:26:59] now i need you to teach me how to play in labs so I can mess with it [02:27:06] :D [02:27:12] sooooo many things to learn [02:27:17] you're alread part of the multimedia project! [02:27:34] step 1 is to ignore anyone who says 'just use sudo git clone, it is trivial!' :P [02:27:53] step 1: {{done}} [02:28:16] step 2: login to wikitech! [02:28:20] If I can't script it it's not worth doing [02:28:26] bd808: +1 [02:29:10] step 2: {{done}} [02:29:19] bd808: step 3: create a new instance! [02:29:26] bd808: 'manage instances' on the left, in the sidebar [02:29:45] bd808: labs is organized into 'projects', which are units of access control, each of which can have many instances, which are just VMs [02:30:05] there's puppet-dawg, which I should delete now :P [02:30:22] -dawg, -dragons, -alpha [02:30:36] no more -dawg [02:31:19] step 3: {{done}} [02:31:41] bd808: follow https://wikitech.wikimedia.org/wiki/Labsvagrant. [02:31:49] bd808: do you already have the appropriate ssh config setup? [02:31:55] to be able to ssh into labs? [02:32:23] I thought I did, but it's kicking me out [02:32:36] with what error? [02:32:59] "Connection closed by UNKNOWN" [02:33:11] * bd808 checks .ssh/config [02:33:12] bd808: to which host? [02:33:25] ssh bd808@bd808-test.pmtpa.wmflabs [02:33:41] * YuviPanda tries [02:33:53] indeed [02:33:54] i get it too [02:36:35] I can't hop in via ssh bastion -> ssh bd808-test either [02:37:07] bd808: see -labs [02:48:21] http://noc.wikimedia.org/dbtree/ reads "Click shard name for list of wikis." I can't find the wiki names anywhere clicking on the shard names. help?!? [02:49:02] nvm [02:50:28] so s1-s4 are servers. what are dbxxxx ? multiple databases on a single server? [03:07:33] No, s1-s4 (really s1-s7 by now) are sets of databases [03:07:55] s1 is only enwiki, s2 is a group of large wikis, s4 is commons, s5-7 are other groups of largeish wikis, and s3 is everything else [03:08:06] For every sN, there is one master DB server and multiple slave DB servers [03:08:14] sN would be the "shard name" [03:08:35] If you click on the box with s1 you do in fact see "wikis: enwiki" [03:08:39] ( Sid-G ---^^ ) [03:09:15] RoanKattouw: thanks, found the wiki names [03:09:29] The tree structure you see is how the replication flows. So db1056 is the s1 master, which means it's where all the writes (changes) for s1 go. Then db1043, 1049, etc. replicate from it [03:10:01] Then db63 also replicates from db1056, and it in turn replicates to db32, db59, etc. This is because db63 is in pmtpa while db10NN are in eqiad [03:10:01] RoanKattouw: so the slave servers are backups? [03:10:24] As a general rule, servers in the 1000-1999 are in eqiad, <1000 is pmtpa, 3NNN is esams, and 4NNN is ulsfo IIRC [03:10:30] In a way [03:10:37] The slave servers mostly exist for load balancing [03:10:47] We send as many read operations to the slaves as we can [03:10:52] ah [03:10:53] to alleviate load on the master server [03:11:03] so the backups are separate? [03:11:24] Also, if a master server has a problem, we can get one of its slaves to take over the master role [03:11:41] Yeah these aren't off-site backups [03:11:56] I'm assuming you're referring to "backups" as "what happens if all of Northern Virginia is wiped out by a tornado" [03:12:08] yeah [03:12:10] I don't know what we do for off-site backups, but then I'm also not on the ops team :) [03:12:20] those along with what happens if the server crashes backups [03:12:45] Well if a slave server crashes, we stop reading from it and the rest of the slaves will be that much busier [03:13:07] and if the master crashes you just make a slave the master? [03:13:15] If a master crashes, that's a call-people-out-of-bed event (but it has happened, I remember a master server hitting a full disk a couple years back), then you rotate a slave into the master role [03:14:38] right [03:15:06] since we're on the topic, has someone saved wikipedia for wwIII in a nuclear bunker somewhere? [03:16:11] IIRC there's an institute for saving stuff in a nuclear bunker somewhere in Europe (Switzerland?) [03:24:49] Sid-G no [03:24:53] that would be ukraine [03:25:12] US uses nuclear bunkers as housing [03:25:22] Ukraine for servers [03:25:39] ToAruShiroiNeko: right. So Ukraine has a backup of wikipedia in a nuclear bunker? [03:25:50] not to my knowledge [03:26:04] copyright may be an issue for non-free content [03:26:55] hmmm [03:28:17] WMF could mobilize the wikimedia army to take over US nuke bases though [03:28:53] though it is possible they'll be no match to Squirrels [03:28:55] http://blogs.smithsonianmag.com/science/2013/08/how-one-nuclear-missile-base-is-battling-ground-squirrels/ [03:47:17] there's a seed vault under construction in Norway, IIRC [03:47:36] that is designed to withstand all sorts of calamities [03:47:51] including, depressingly, humanity-ending events [03:48:14] ah yes, here we go: https://en.wikipedia.org/wiki/Svalbard_Global_Seed_Vault [03:49:16] though i remembered incorrectly, it's not a facility designed to outlive humanity [03:49:50] If humanity is gone, who would the vault be for? [03:49:56] Svalbard?! [03:50:05] That's not just "somewhere in Norway", that's friggin' Svalbard [03:50:11] Calamity #0 is the climate up there [03:50:29] hmm, I oddly find humanity-ending events not depressing. [03:50:30] oh well [03:50:50] you just want all those seeds for yourself [03:50:54] It's a good choice though, it's so remote that any catastrophe would probably not reah it [03:52:22] is svalbard more famous than i realize? [03:52:41] is it quasi-mythical, like lapland? [03:53:39] Yes [03:53:50] Well, -ish [03:53:56] Most people, even in Europe, have no clue where it is [03:54:05] So to the general public it's definitely less famous than Lapland [03:54:40] But people who are interested in Scandinavia like myself would know about it [03:55:43] It also features prominently in His Dark Materials (the book series that The Golden Compass was based on, and which I intend to read in my copious free time at some point, although at this point I wonder if it'll happen before my retirement) [03:57:08] ah, i was going to say -- if you're into fiction and scandinavia, my wife has been working through (most) of the books listed in http://www.theguardian.com/books/2009/jan/23/scandinavian-crime-fiction and is enjoying them tremendously [03:57:47] I have of course read the Stieg Larsson novels [03:57:58] Perhaps one day my Swedish will be good enough that I can read the originals [03:58:00] puppet's very stupid [03:58:55] RoanKattouw: stieg larsson's just the context for going into scandinavian krimis in general [03:59:03] there's a good list later on in the article [03:59:06] Those books are actually not good candidates for reading in English because that loses the bits where the characters use English words rather than their native language. I read the Dutch translations [04:00:45] Hah, Jo Nesbø, that guy was advertised quite widely in NL around the time I left there [04:00:49] yeah, swedish sounds very cool [04:02:04] do you like 'the knife'? [04:02:15] as in, the band [04:02:20] I already speak Dutch, Frisian, German and English, so there's a lot of Swedish vocabulary that I can't come up with but can understand when I read it [04:02:28] but I need to stop being lazy and actually learn the language [04:02:34] No, never heard of them [04:03:15] there's a cool electronic music scene, most sing in english tho [04:04:28] though when it comes to swedish high art, it's hard to top http://www.youtube.com/watch?v=iPrnduGtgmc [04:04:58] did you speak frisian growing up? [04:05:01] RoanKattouw: ori-l vagrant+labs for setting up VE + UW, about 4 commands in total! http://pearl-hammer.instance-proxy.wmflabs.org/wiki/Main_Page [04:05:15] (not counting the hour of debugging, but that is an unrelated labs issue) [04:05:32] 'pearl hammer'? [04:05:54] ori-l: it'll be destroyed very soon [04:06:18] is that a porn star alias or the code name for a secret military operation? [04:06:29] never mind, i don't want to know [04:06:36] ori-l: I didn't, I learned it by hanging out with people that spoke it in high school [04:06:43] I grew up in an area of Friesland where Frisian isn't spoken [04:07:08] ori-l: I had just started listening to Abbey Road, and the first instance was silver-hammer. And then we had to kill it because labs, so the new one was pearl-hammer [04:07:16] Then when I was 13 I moved to a city (well, "city", 15k-ish people) where Frisian wasn't spoken in the city but was spoken in the surrounding villages [04:07:19] ori-l: in hindsight, I can see how that sounds like a porn star alias [04:07:22] bang bang maxwell silver hammer came down upon her head [04:07:36] playing right now, yes :) [04:07:45] * ori-l looks up recordings of frisian [04:07:50] I usually need to listen to songs 3-5 times to fully comprehend the lyrics [04:08:15] * RoanKattouw tries to find ori-l a recording of "bûter, brea en griene tsiis" [04:09:04] https://en.wikipedia.org/wiki/West_Frisian_language#Folklore_about_relation_to_English_and_Dutch [04:10:04] * RoanKattouw listens to the recording and is slightly irked that the pronunciation is from the middle / eastern part of the West Frisian language area, as opposed to the northwest where he's from [04:11:07] Yes, you heard that right. Our province the size of Rhode Island with pop 1M has its own language spoken by ~50% of the population, which has 3 dialects (3 major ones at least) [04:15:22] that's cool. i've never lived in an area with many dialects [04:15:57] The UK also has incredible dialect density [04:16:14] Lots of very different accents in a small space [04:16:44] I guess in the UK I'm mostly amazed by how far apart the accents are, than by the geographical aspect [04:19:05] yeah, why is that? maybe because the upper classes are careful not to speak too much with the working classes? [04:19:28] I don't know, it's weird [04:20:58] Maybe it's also that people move around less [04:21:39] It's totally normal for someone to move from NY to SF for instance, which is 3k mi, much less normal than even for people from Belgium to move to NL [04:21:39] YuviPanda: what are the four commands? [04:21:55] That may be because of the cultural and linguistic differences, but I wonder which causes which [04:22:12] ori-l: 'puppetd -tv' 'labsvagrant list-roles', 'labsvagrant enable-role visualeditor uploadwizard', 'labsvagrant provision' [04:22:31] ori-l: arguably the first isn't needed if you are willing to wait for 30m for puppet in labs to apply the role::labsvagrant [04:22:36] and second isn't needed either :P [04:23:21] ori-l: docs at https://wikitech.wikimedia.org/wiki/Labsvagrant [04:24:43] RoanKattouw: yes, I had a good Flemish friend in college and I visited him once and we had a long bike tour of Flanders and I remember laughing whenever he pointed at and squinted angrily at Walloonian(sp?) villages [04:24:55] hah [04:25:02] Yeah the Belgians are particularly weird that way [04:25:09] (Walloon) [04:25:24] They insist on being in the same country with people that speak a different language [04:26:51] YuviPanda: nitpicking: 'Labsvagrant' is hard on the eyes [04:27:03] suggestions? [04:27:07] maybe Labs-Vagrant? or LabsVagrant? or something else entirely? [04:27:15] like? [04:27:24] technically this has nothing to do with vagrant [04:28:59] dunno [04:29:08] dunno is a very ungoogleable name :P [04:29:21] anyway, so far it is just me and bd808 testing it out, so open to changing it [04:30:30] `not-really-vagrant` [04:30:43] well, I don't mind it, I just think you'd have an easier time getting people to check it out if it didn't look like the name of a subway station in Oslo [04:30:46] or `bum` [04:30:48] puppet-files-from-vagrant [04:31:16] i guess it's still better than 'mediawiki-singlenode' [04:31:18] `tramp` [04:31:26] `drifter` [04:31:36] 'jetlagged' [04:31:43] `itinerant` [04:31:50] `yuvi` [04:32:16] `sockpuppet` [04:32:21] labs-vagrant? [04:32:33] yeah, I think that's nicer [04:33:55] alright [04:34:11] i'll rename the tool to that. [04:34:17] don't think puppet modules can have - in them tho [04:34:25] so I guess i'll leave the module named as is [04:35:06] they can have _. we make that translation elsewhere [04:35:25] ah, hm [04:35:40] eg the package::foo_bar classes [04:35:45] right [04:36:05] tr/-/_/g [04:36:19] ^ the name of my next ska band [04:36:28] rolls off the tongue quite nicely, I must say [04:36:54] 'tee aaar! slaaaashhh! daaasshhh! slaaasshhh! underscoooooreeee! slaaash! geeeee!' [04:36:57] * bd808 will NEVER start a ska band [04:39:51] 'labs-vagrant' matches 'mediawiki-vagrant' [04:40:21] you guys can help reviewing other vagrant patches, you know :P [04:40:31] maybe -1 / +1 for a bit until we've got a consistent standard [04:40:50] ori-l: oh we are :P [04:41:01] not just each other's, i mean [04:41:13] there are other vagrant patches? [04:41:40] the ones ori-l self merges as soon as they are posted? [04:41:50] [04:44:32] I just set a watch in gerrit for mediawiki/vagrant. I'll try to help [04:48:24] thanks [12:53:37] Am I the only one having SUL issues? I logged in on Commons and it didn't propagate to any other wiki. [12:54:56] Never mind, after logging in on en.wp, it seems I'm now logged in on other wikis as well. [12:54:59] guillom: works for me [12:55:15] logged in on commons, then on random wikisource i was in.. [13:24:37] guillom: note that commons is on varnish (in the US only though) [13:25:42] !gitweb operations/dns [13:25:42] https://git.wikimedia.org/summary/?r=operations/dns [13:25:54] I seem to remember that after logging in (on Commons), I ended up on wikimediafoundation.org, with a "no such special page" error. [13:38:45] guillom: known bug [13:38:58] Oh. Ok, then. [13:39:28] guillom: https://bugzilla.wikimedia.org/show_bug.cgi?id=52206 [13:39:37] Thanks Nemo_bis [13:39:38] Unless it was another page foryou [13:40:40] Nope, it was that one. [14:14:17] Anybody has an explanation why often outdated cached versions of pages are shown to users not logged in? [14:14:20] (at least to de.wp users, see https://bugzilla.wikimedia.org/show_bug.cgi?id=54647 ) [14:17:45] could it have to do with https://bugzilla.wikimedia.org/show_bug.cgi?id=54629 ? [14:20:19] hmm, media storage normally refers to images/video/audio, but this is page content :-/ [14:20:24] but thanks for the idea [14:27:14] andre__, yes, it was about image cache purges not propagating to European servers. i was wondering if page purges wouldn't also be affected by some similar infrastructure [14:27:37] *similar = common [15:09:02] malafaya: usually the two are unrelated [15:09:55] Nemo_bis: thanks [16:32:59] greg-g: Re bug 54647, it seems we've been having a lot of caching issues lately. [16:37:33] * greg-g nods [16:38:52] greg-g: Have you poked the ops people who deal with the caches? [16:40:00] anomie: bd808 just poked [16:40:36] we had a multicast/htcp outage from the 22nd and until yesterday [16:40:46] it was a network issue [16:40:57] that would do it for sure [16:41:52] paravoid: It looks like my attempts to fix the vhtcpd_* ganglia graphs still haven't worked either. Do you know ganglia magic? [16:42:02] not much [16:42:07] what's the issue? [16:42:49] They are showing all-time cumulative values instead of desired "gauge" behavior of delta from prior sample [16:44:00] I tried to fix that in https://gerrit.wikimedia.org/r/#/c/83587/ but it doesn't seem to have made a difference [16:46:10] slope positive is the one you're looking for indeed [16:46:34] but iirc, this is an rrd property, so I'm guessing the rrds might need to be deleted/recreated [16:46:51] or dump/modify/restore maybe [16:47:17] I thought that ganglia magically managed that, but if not I agree [16:47:35] the initial RRDs were most certainly made with the wrong setting [16:48:43] I don't know ganglia well [16:48:46] i'd just delete them and allow them to be recreated [16:48:53] i can do that if you like [16:49:40] ori-l: You might check with bblack first but I don't think that would be a horrible plan [16:50:20] bblack: ^ [16:53:11] I'm going ahead with it -- if we had a five-day outage we didn't know about it's probably better not to regard the monitoring setup as precious [16:54:03] that's not why we had the outage [16:54:29] we had the outage because we don't have something to alert us to them and noone's looking at the graphs all the time [16:54:43] the outage was quite obvious from the graphs and it's how I know it was from the 22nd [16:55:08] okay, fair point [16:57:54] I'm guessing that's an issue "on the roadmap"? [16:58:04] trend monitoring that is [16:58:41] I did some of that with graphite + nagios at $DAYJOB-1. [16:59:05] paravoid: i have the RRDs backed up; is it cool to to delete them? [16:59:14] Well more threshold alerts than trends [17:01:09] Do nagios plugins work out of the box with icinga? https://pypi.python.org/pypi/check_ganglia_metric/ looks interesting [17:02:02] we have a lot of infrastructure, a lot of subsystems, a lot of software, and few opsen [17:02:20] ori-l: I understand completely [17:03:16] adding more alerting mechanisms isn't the best way to approach it; what's needed is really to do a lot of curatorial work to identify a compact set of vital indicators [17:03:31] what ori said [17:03:33] that aren't noisy [17:04:57] yes please [17:05:03] ori-l: yes, it's fine [17:05:06] to delete them [17:06:46] ok, I backed them up here http://noc.wikimedia.org/~olivneh/vhtcpd/ [17:06:55] ^ bd808 if you want to poke at them [17:08:41] thanks ori-l. I'll add it to "the list" :) [17:09:56] bd808: deleted [17:17:15] what is needed is somebody actually looking at new bugs :P [17:38:29] marktraceur: how can I know if an etherpad was lost? specifically http://etherpad.wikimedia.org/wmconf2013-evaluating_programs [17:38:58] this confirms it existed: http://web.archive.org/web/20130717203153/http://etherpad.wikimedia.org/wmconf2013-evaluating-programs [17:43:56] I wanna cry, no pads in here seem to have survived https://meta.wikimedia.org/wiki/Wikimedia_Conference_2013/Schedule/Saturday [18:09:19] csteipp, *: http://dpaste.com/1398244/plain/ [18:09:31] login issues? that was sent 4 hrs ago [18:10:41] jeremyb: is that really their username? [18:10:49] no, of course not :) [18:11:01] i'm posting publicly so i redacted [18:11:05] Good, was trying to figure out how that would get past our filter :) [18:11:15] hah [18:11:22] "well, there's your problem!" [18:11:27] We currently have an issue with logged in users visiting http [18:11:56] csteipp: sent you the real name [18:12:05] Just saw that, thanks [18:13:03]