[00:07:00] PROBLEM - Puppet freshness on db11 is CRITICAL: Puppet has not run in the last 10 hours [00:09:00] RECOVERY - Puppet freshness on db11 is OK: puppet ran at Thu Mar 28 00:08:57 UTC 2013 [00:09:10] PROBLEM - RAID on db11 is CRITICAL: CRITICAL: Defunct disk drive count: 1 [00:10:02] PROBLEM - Puppet freshness on db11 is CRITICAL: Puppet has not run in the last 10 hours [00:12:41] RECOVERY - Puppet freshness on db11 is OK: puppet ran at Thu Mar 28 00:12:35 UTC 2013 [00:13:01] PROBLEM - Puppet freshness on db11 is CRITICAL: Puppet has not run in the last 10 hours [00:14:02] New patchset: Reedy; "(bug 46004) Set $wgCategoryCollation to 'uca-be' on be.wikipedia and be.wikisource" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/54365 [00:14:54] RobH: whatever happened with professor? errors never popped up again? [00:16:00] RECOVERY - Puppet freshness on db11 is OK: puppet ran at Thu Mar 28 00:15:49 UTC 2013 [00:16:00] PROBLEM - Puppet freshness on db11 is CRITICAL: Puppet has not run in the last 10 hours [00:16:20] jeremyb_: which one [00:16:42] mutante: 4619 [00:17:24] Change merged: Reedy; [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/54365 [00:18:14] jeremyb_: i think it says it all, the fan was replaced but not yet the memory [00:18:18] !log reedy synchronized wmf-config/InitialiseSettings.php [00:18:25] Logged the message, Master [00:18:43] mutante: the reason to replace the fan and not the memory was in order to see the memory errors without the flood of fan errors [00:18:55] RECOVERY - Puppet freshness on db11 is OK: puppet ran at Thu Mar 28 00:18:41 UTC 2013 [00:18:55] PROBLEM - Puppet freshness on db11 is CRITICAL: Puppet has not run in the last 10 hours [00:19:12] jeremyb_: yes, i saw that, so? [00:19:47] so, i was just wondering if the new errors that we were waiting for had appeared or not [00:19:49] i wouldnt expect them to replace it without commenting [00:20:16] (it's nearly 2 weeks and seems like it would be more than enough time to get more errors. but maybe i'm wrong) [00:20:54] i don't know, so it was just a "bump?" [00:21:10] RECOVERY - Puppet freshness on db11 is OK: puppet ran at Thu Mar 28 00:21:07 UTC 2013 [00:21:27] lol, that is nice, since db11 was supposed to be decom'ed [00:21:50] PROBLEM - Puppet freshness on db11 is CRITICAL: Puppet has not run in the last 10 hours [00:23:50] RECOVERY - Puppet freshness on db11 is OK: puppet ran at Thu Mar 28 00:23:49 UTC 2013 [00:24:50] PROBLEM - Puppet freshness on db11 is CRITICAL: Puppet has not run in the last 10 hours [00:25:50] RECOVERY - Puppet freshness on db11 is OK: puppet ran at Thu Mar 28 00:25:45 UTC 2013 [00:27:09] !log once again disabling notifications for db11 [00:27:15] Logged the message, Master [00:43:24] New patchset: Aklapper; "bugzilla_report.php: Add query and formatting for list of urgent issues" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/56348 [00:52:11] PROBLEM - Puppet freshness on virt2 is CRITICAL: Puppet has not run in the last 10 hours [00:57:11] PROBLEM - Puppet freshness on virt3 is CRITICAL: Puppet has not run in the last 10 hours [01:05:00] PROBLEM - Puppet freshness on db11 is CRITICAL: Puppet has not run in the last 10 hours [01:07:10] PROBLEM - RAID on db11 is CRITICAL: CRITICAL: Defunct disk drive count: 1 [01:25:52] New patchset: Ram; "Bug: 43544: Improve error handling to not hide internal errors." [operations/debs/lucene-search-2] (master) - https://gerrit.wikimedia.org/r/56354 [01:55:29] New patchset: Reedy; "Remove some comments that just get in the way" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/56356 [01:55:47] Change merged: Reedy; [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/56356 [02:04:51] PROBLEM - Puppet freshness on db11 is CRITICAL: Puppet has not run in the last 10 hours [02:07:01] PROBLEM - RAID on db11 is CRITICAL: CRITICAL: Defunct disk drive count: 1 [02:19:24] !log LocalisationUpdate completed (1.21wmf12) at Thu Mar 28 02:19:24 UTC 2013 [02:19:31] Logged the message, Master [02:25:33] New review: Faidon; "Looks OK to me. Ping me or someone else in ops to merge it when you're around, just in case." [operations/puppet] (production) C: 2; - https://gerrit.wikimedia.org/r/54324 [02:30:52] New review: Faidon; "(3 comments)" [operations/debs/python-voluptuous] (master) C: -1; - https://gerrit.wikimedia.org/r/56168 [02:35:58] New review: Faidon; "The package itself is maintained in git in" [operations/debs/python-jsonschema] (debian/experimental) - https://gerrit.wikimedia.org/r/56064 [02:39:40] New review: Faidon; "(4 comments)" [operations/debs/python-statsd] (master) C: -1; - https://gerrit.wikimedia.org/r/55069 [02:43:11] New review: Faidon; "I think it's okay. But please figure out the answer to your TODO/FIXME and fix it, instead of adding..." [operations/puppet] (production) - https://gerrit.wikimedia.org/r/55302 [02:58:13] New patchset: Reedy; "(bug 46081) Set $wgCategoryCollation to 'uca-default' on Polish Wiktionary" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/54367 [02:58:23] Change merged: Reedy; [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/54367 [02:59:20] !log reedy synchronized wmf-config/InitialiseSettings.php [02:59:25] New patchset: Reedy; "cswikinews: Set autopatrolled group" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/55441 [02:59:27] Logged the message, Master [02:59:32] Change merged: Reedy; [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/55441 [03:00:48] New patchset: Reedy; "(bug 46589) Add localised/v2 logos for Wikipedias without one (second installment)" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/56097 [03:01:04] Change merged: Reedy; [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/56097 [03:01:22] New patchset: Reedy; "(bug 43863) Enabled wgImportSources on the Spanish Wikivoyage. Added eswiki, meta, commons, en.voy, de.voy, fr.voy, it.voy, nl.voy, pt.voy, ru.voy, and sv.voy" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/56113 [03:01:30] Change merged: Reedy; [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/56113 [03:01:46] New patchset: Reedy; "(bug 46461) Set $wgAutoConfirmCount to 50 for Wikidata" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/56150 [03:01:52] Change merged: Reedy; [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/56150 [03:02:06] New patchset: Reedy; "(bug 45638) Modify user group rights on it.wikivoyage Modified wgAddGroups and wgRemoveGroups; changed user rights for autoconfirmed, added patroller group." [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/56118 [03:02:12] Change merged: Reedy; [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/56118 [03:02:31] New patchset: Reedy; "Add tz database time zone settings for wikis in Maldivian language" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/56098 [03:02:37] Change merged: Reedy; [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/56098 [03:02:55] New patchset: Reedy; "(bug 46182) Set LQT as opt-out on se.wikimedia (chapter wiki)" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/56130 [03:03:01] Change merged: Reedy; [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/56130 [03:03:24] New patchset: Reedy; "(bug 44285) config changes for eswikivoyage" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/56055 [03:03:31] Change merged: Reedy; [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/56055 [03:04:16] !log reedy synchronized wmf-config/InitialiseSettings.php [03:04:22] Logged the message, Master [03:06:08] PROBLEM - Puppet freshness on db11 is CRITICAL: Puppet has not run in the last 10 hours [03:08:21] PROBLEM - RAID on db11 is CRITICAL: CRITICAL: Defunct disk drive count: 1 [03:12:18] PROBLEM - Puppet freshness on ms1004 is CRITICAL: Puppet has not run in the last 10 hours [03:26:38] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:28:07] New review: Faidon; "I like this very much." [operations/puppet] (production) C: 2; - https://gerrit.wikimedia.org/r/49710 [03:28:28] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.128 second response time [03:30:02] New review: Faidon; "Oh and the answer to your 0444 question is that when you vi and says it's read-only is a helpful hin..." [operations/puppet] (production) - https://gerrit.wikimedia.org/r/49710 [03:50:34] New review: Faidon; "So, this is a very nice effort. It feels a bit too complicated and I was confused a lot while review..." [operations/puppet] (production) C: -1; - https://gerrit.wikimedia.org/r/53714 [03:51:45] paravoid: s/ger-orig-source/get-orig-source/ [04:06:32] PROBLEM - Puppet freshness on db11 is CRITICAL: Puppet has not run in the last 10 hours [04:08:02] RECOVERY - Puppet freshness on db11 is OK: puppet ran at Thu Mar 28 04:07:58 UTC 2013 [04:08:37] PROBLEM - Puppet freshness on db11 is CRITICAL: Puppet has not run in the last 10 hours [04:08:42] PROBLEM - RAID on db11 is CRITICAL: CRITICAL: Defunct disk drive count: 1 [04:09:32] RECOVERY - Puppet freshness on db11 is OK: puppet ran at Thu Mar 28 04:09:29 UTC 2013 [04:10:32] PROBLEM - Puppet freshness on db11 is CRITICAL: Puppet has not run in the last 10 hours [04:10:42] RECOVERY - Puppet freshness on db11 is OK: puppet ran at Thu Mar 28 04:10:41 UTC 2013 [04:11:32] PROBLEM - Puppet freshness on db11 is CRITICAL: Puppet has not run in the last 10 hours [04:11:52] RECOVERY - Puppet freshness on db11 is OK: puppet ran at Thu Mar 28 04:11:49 UTC 2013 [04:12:32] PROBLEM - Puppet freshness on db11 is CRITICAL: Puppet has not run in the last 10 hours [04:12:53] RECOVERY - Puppet freshness on db11 is OK: puppet ran at Thu Mar 28 04:12:48 UTC 2013 [04:13:33] PROBLEM - Puppet freshness on db11 is CRITICAL: Puppet has not run in the last 10 hours [04:13:42] RECOVERY - Puppet freshness on db11 is OK: puppet ran at Thu Mar 28 04:13:40 UTC 2013 [04:14:32] PROBLEM - Puppet freshness on db11 is CRITICAL: Puppet has not run in the last 10 hours [04:15:12] RECOVERY - Puppet freshness on db11 is OK: puppet ran at Thu Mar 28 04:15:02 UTC 2013 [04:15:32] PROBLEM - Puppet freshness on db11 is CRITICAL: Puppet has not run in the last 10 hours [04:15:34] New patchset: Faidon; "New upstream release" [operations/debs/ruby-jsduck] (master) - https://gerrit.wikimedia.org/r/56362 [04:16:24] Change merged: Faidon; [operations/debs/ruby-jsduck] (master) - https://gerrit.wikimedia.org/r/56362 [04:19:37] !log updating jsduck in apt and upgrading it on gallium [04:19:39] Krinkle: ^^^ [04:19:44] Logged the message, Master [04:21:42] paravoid: Thanks [04:21:53] paravoid: btw, what does this find/print command do? https://gerrit.wikimedia.org/r/#/c/56362/1/debian/rules [04:23:16] Wondering what could cause chmod to be wrong by default. [04:25:24] why not use -exec ? is it not guaranteed to be in every /usr/bin/find ? [04:26:13] (or for that matter chmod -R) [04:26:24] oh, -type f [04:26:35] still, idk. X is useful :) [04:37:29] Krinkle: the gifs were 755 [04:37:45] paravoid: orly, that's messed up. [04:37:53] jeremyb_: chmod -R is for dirs; -exec vs. xargs is the difference between running (fork/exec) N chmods vs. one [04:37:53] I'll file a bug. [04:38:29] paravoid: well i'd do -exec ... + not -exec ... \; [04:39:26] Krinkle: hmm [04:39:29] they're not in the tarball [04:39:54] oh nevermind [04:39:55] they are [04:40:03] 2821891 4 -rwxr-xr-x 1 www-data www-data 856 Mar 28 05:59 ./extjs/resources/themes/images/default/util/splitter/mini-bottom.gif [04:40:06] 2821892 4 -rwxr-xr-x 1 www-data www-data 856 Mar 28 05:59 ./extjs/resources/themes/images/default/util/splitter/mini-top.gif [04:40:10] etc. [04:40:20] > find debian/ruby-jsduck/usr/share/ruby-jsduck/ -type f -exec chmod 644 {} + [04:40:56] oh hah [04:41:02] I didn't know + [04:41:03] that's nifty [04:41:08] must be relatively new [04:42:10] core bins like that usually take very long to get updates spread [04:42:28] but I guess 1997 is new in that case :P [04:42:52] 2005-01-15 James Youngman [04:42:55] First working version of -exec ...+ [04:43:23] so, yeah, not exactly new [04:43:26] but much newer than 1997 [04:43:54] funny [04:43:57] changelog goes back to... [04:44:00] 87/02/21 22:19:25 22:19:25 cire (Eric B. Decker) [04:44:04] 1987! [04:45:49] oh god [04:45:51] ariel woke up [04:45:54] and I still haven't gone to bed [04:48:59] tha's a bad sign [04:49:06] shoo! [04:49:36] I wouldnt' say I "woke up" exactly, 'groggily sitting at keyboard" more like [04:49:51] cursig gnome-shell 3 yet again [05:06:53] PROBLEM - Puppet freshness on db11 is CRITICAL: Puppet has not run in the last 10 hours [05:08:26] Could someone merged https://gerrit.wikimedia.org/r/#/c/38252/ please? [05:08:27] ;p [05:09:03] PROBLEM - RAID on db11 is CRITICAL: CRITICAL: Defunct disk drive count: 1 [05:09:12] I'm one step ahead of you paravoid, I'm in bed, but not asleep... Checked IRC and logged 2 new bugs... [05:09:59] but we also have 2h time difference, so I win anyway :) [05:10:28] uhm, you've commented that it's buggy [05:12:16] "buggy" [05:12:26] fatalmonitor has similar little flaws [05:12:59] For all intents and purposes it works [05:14:55] e.g. [05:14:56] 84 Exception from line 637 of /usr/local/apache/common-local/php-1.21wmf12/includes/cache/MessageCache.php: Message key 'Filepage.css' does not appear to be a full key. [05:14:56] 2 Exception from line 637 of /usr/local/apache/common-local/php-1.21wmf12/includes/cache/MessageCache.php: Message key 'Handheld.css' does not appear to be a full key. [05:21:31] ping [05:26:57] TimStarling: ping [05:27:10] hello preilly [05:27:20] TimStarling: May I PM [05:27:24] yes [05:37:14] PROBLEM - Puppet freshness on lvs1004 is CRITICAL: Puppet has not run in the last 10 hours [05:37:14] PROBLEM - Puppet freshness on lvs1005 is CRITICAL: Puppet has not run in the last 10 hours [05:37:14] PROBLEM - Puppet freshness on lvs1006 is CRITICAL: Puppet has not run in the last 10 hours [05:45:20] New patchset: Aude; "Update fywiki sort order, add note about default Wikibase settings" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/56367 [06:06:44] PROBLEM - Puppet freshness on db11 is CRITICAL: Puppet has not run in the last 10 hours [06:08:54] PROBLEM - RAID on db11 is CRITICAL: CRITICAL: Defunct disk drive count: 1 [06:09:14] PROBLEM - Puppet freshness on cp3010 is CRITICAL: Puppet has not run in the last 10 hours [06:21:14] PROBLEM - Puppet freshness on virt1005 is CRITICAL: Puppet has not run in the last 10 hours [06:25:57] ori-l: ping [06:26:08] hey, preilly [06:27:36] ori-l: I just saw your Vagrant change [06:27:46] ori-l: have you thought about using sshfs? [06:28:31] No, Vagrant doesn't support it by default [06:28:41] instead of? [06:28:56] How's that possible? [06:29:00] VirtualBox Shared Folders and NFS are the options it supports out of the box [06:29:13] ori-l: have you tried the VMWare driver yet? [06:30:07] No -- it isn't free or open-source, so I don't expect it to be very popular with our community [06:30:33] ori-l: hmm [06:30:33] Jasper_Deng_busy: look up? [06:30:43] ori-l: I just meant have you tried it? [06:30:43] * Aaron|home admires http://www.time.com/time/photogallery/0,29307,2036928_2218542,00.html [06:30:50] no, I haven't [06:31:03] Have you? How does it compare to VirtualBox? [06:31:23] Aaron|home: is that server porn? [06:31:35] ori-l: it seems much much faster to me [06:31:46] ori-l: I just started playing with it today [06:33:14] RECOVERY - Puppet freshness on db11 is OK: puppet ran at Thu Mar 28 06:33:08 UTC 2013 [06:33:45] PROBLEM - Puppet freshness on db11 is CRITICAL: Puppet has not run in the last 10 hours [06:34:24] RECOVERY - Puppet freshness on db11 is OK: puppet ran at Thu Mar 28 06:34:14 UTC 2013 [06:34:26] Honestly, I'm mostly hoping that VMWare support motivates Oracle to make the integration with VirtualBox better [06:34:44] PROBLEM - Puppet freshness on db11 is CRITICAL: Puppet has not run in the last 10 hours [06:34:57] their stewardship of open-source projects hasn't been inspiring [06:35:05] ori-l: makes total sense to me [06:35:14] RECOVERY - Puppet freshness on db11 is OK: puppet ran at Thu Mar 28 06:35:09 UTC 2013 [06:35:14] ori-l: yeah totally [06:35:44] PROBLEM - Puppet freshness on db11 is CRITICAL: Puppet has not run in the last 10 hours [06:35:54] RECOVERY - Puppet freshness on db11 is OK: puppet ran at Thu Mar 28 06:35:53 UTC 2013 [06:36:39] based on oracle's current philosophy I'm surprised virtualbox still exists [06:36:44] PROBLEM - Puppet freshness on db11 is CRITICAL: Puppet has not run in the last 10 hours [06:36:50] Ryan_Lane: yeah totally [06:37:10] they have hardly touched it since they acquired sun [06:37:18] Ryan_Lane: yeah [06:37:39] Ryan_Lane: and the delta between it and VMWare is growing all the time [06:37:42] indeed [06:37:47] on Linux it's not a big deal [06:37:55] on OS X and Windows it's a pain in the ass [06:38:16] http://download.virtualbox.org/favicon.ico [06:38:17] yeah [06:38:28] ori-l: :D [06:38:29] ha ha ha [06:38:43] ori-l: that's hilarious [07:06:42] PROBLEM - Puppet freshness on db11 is CRITICAL: Puppet has not run in the last 10 hours [07:08:52] PROBLEM - RAID on db11 is CRITICAL: CRITICAL: Defunct disk drive count: 1 [08:06:24] PROBLEM - Puppet freshness on db11 is CRITICAL: Puppet has not run in the last 10 hours [08:07:35] PROBLEM - RAID on db11 is CRITICAL: CRITICAL: Defunct disk drive count: 1 [08:07:54] RECOVERY - Puppet freshness on db11 is OK: puppet ran at Thu Mar 28 08:07:53 UTC 2013 [08:08:25] PROBLEM - Puppet freshness on db11 is CRITICAL: Puppet has not run in the last 10 hours [08:08:54] RECOVERY - Puppet freshness on db11 is OK: puppet ran at Thu Mar 28 08:08:52 UTC 2013 [08:09:24] PROBLEM - Puppet freshness on db11 is CRITICAL: Puppet has not run in the last 10 hours [08:09:54] RECOVERY - Puppet freshness on db11 is OK: puppet ran at Thu Mar 28 08:09:47 UTC 2013 [08:10:24] PROBLEM - Puppet freshness on db11 is CRITICAL: Puppet has not run in the last 10 hours [08:14:38] RECOVERY - Puppet freshness on db11 is OK: puppet ran at Thu Mar 28 08:14:31 UTC 2013 [08:15:24] PROBLEM - Puppet freshness on db11 is CRITICAL: Puppet has not run in the last 10 hours [08:17:24] PROBLEM - Puppet freshness on mw1160 is CRITICAL: Puppet has not run in the last 10 hours [09:05:16] PROBLEM - Puppet freshness on db11 is CRITICAL: Puppet has not run in the last 10 hours [09:07:26] PROBLEM - RAID on db11 is CRITICAL: CRITICAL: Defunct disk drive count: 1 [09:10:26] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:11:16] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.129 second response time [09:14:41] RECOVERY - Puppet freshness on db11 is OK: puppet ran at Thu Mar 28 09:14:30 UTC 2013 [09:15:16] PROBLEM - Puppet freshness on db11 is CRITICAL: Puppet has not run in the last 10 hours [09:22:26] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:23:16] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.125 second response time [10:04:54] PROBLEM - Puppet freshness on db11 is CRITICAL: Puppet has not run in the last 10 hours [10:06:34] PROBLEM - Disk space on db11 is CRITICAL: DISK CRITICAL - free space: /a 32479 MB (3% inode=99%): [10:07:04] PROBLEM - RAID on db11 is CRITICAL: CRITICAL: Defunct disk drive count: 1 [10:14:03] hi apergos [10:14:09] yo [10:14:35] RECOVERY - Puppet freshness on db11 is OK: puppet ran at Thu Mar 28 10:14:31 UTC 2013 [10:14:42] apergos: did greg tell you that you are going to watch my deployment? [10:14:54] PROBLEM - Puppet freshness on db11 is CRITICAL: Puppet has not run in the last 10 hours [10:15:07] no, he asked if I was available and I said I'd be happy to [10:15:13] :-P [10:15:22] how long from now is that window? [10:15:26] apergos: it's now [10:15:31] all righty [10:16:07] apergos: to put it short... I was not able to reach working solution for https://gerrit.wikimedia.org/r/#/c/56345/ so I'm not deploying it, only Translate which is https://gerrit.wikimedia.org/r/56379 [10:17:42] ok [10:33:45] Nikerabbit: but no revert of the other either? (would be useful to tell somewhere) [10:39:12] !log nikerabbit synchronized php-1.21wmf12/extensions/Translate/ 'Translate to master' [10:39:20] Logged the message, Master [10:39:34] Nemo_bis: Siebrand is updating the bugs [10:40:11] New patchset: Hashar; "package-builder learned 'cowbuilder'" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/56382 [10:40:30] All information should be in bugs https://bugzilla.wikimedia.org/show_bug.cgi?id=1495 and https://bugzilla.wikimedia.org/show_bug.cgi?id=46579#c19 now. [10:40:54] I'm writing my last email to some Wikimedia folks now, and will continue with other things. [10:41:34] last email on the subject. [10:41:46] * Nemo_bis just received bugmail [10:53:11] PROBLEM - Puppet freshness on virt2 is CRITICAL: Puppet has not run in the last 10 hours [10:58:07] PROBLEM - Puppet freshness on virt3 is CRITICAL: Puppet has not run in the last 10 hours [10:58:14] I hate you puppet [10:58:14] really [10:58:22] most of the time [11:02:25] New review: Hashar; "I am not paying attention:" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/56382 [11:06:46] PROBLEM - Puppet freshness on db11 is CRITICAL: Puppet has not run in the last 10 hours [11:08:26] PROBLEM - Disk space on db11 is CRITICAL: DISK CRITICAL - free space: /a 31767 MB (3% inode=99%): [11:08:56] PROBLEM - RAID on db11 is CRITICAL: CRITICAL: Defunct disk drive count: 1 [11:24:34] ssh: connect to host gerrit.wikimedia.org port 29418: Connection timed out [11:31:04] thanks freenode [11:33:26] PROBLEM - search indices - check lucene status page on search20 is CRITICAL: HTTP CRITICAL: HTTP/1.1 200 OK - pattern found - 62805 bytes in 0.118 second response time [11:33:46] PROBLEM - search indices - check lucene status page on search19 is CRITICAL: HTTP CRITICAL: HTTP/1.1 200 OK - pattern found - 62805 bytes in 0.110 second response time [11:35:44] pmtpa right? guess I'm going to not care about those [11:38:30] apergos: the search indices comes from a patch I wrote and got deployed yesterday [11:38:56] ok [11:39:15] apergos: we have an Icinga check for each of the search servers that verify whether each indices are fine. Some search boxes have issues though :( [11:39:34] we should probably have fixed the issues before enabling the module. I will have a look at search19 and search20 [11:39:59] all right. we're basically serving search out of eqiad though, right? [11:40:08] no idea [11:40:22] ram said he will have a look at them anyway [11:40:27] great [11:41:22] ahh [11:41:27] that is the enwiki.prefix db that failed [11:41:29] when I look at network traffic for the search clusters there's steady to eqiad and not really to pmtpa [11:42:08] !log search19 and search20 have enwiki.prefix marked as FAILED. (see: curl --silent http://search19.pmtpa.wmnet:8123/status |grep FAILED and curl --silent http://search20.pmtpa.wmnet:8123/status |grep FAILED). [11:42:15] Logged the message, Master [11:42:27] curl --silent http://search20.pmtpa.wmnet:8123/status |grep FAILED [11:42:28] [FAILED] enwiki.prefix [11:42:28] ;) [11:42:37] I guess this might be a not peter thing [11:42:48] yup and ram or ^demon [11:42:53] ok [11:43:33] I will fill a RT ticket [11:43:53] thanks [11:48:06] apergos: can you possibly acknowledge both errors in Icinga and refer to RT #4845 ? [11:48:38] ah lemme see, last time I tried to ack things there I didn't have permission [11:48:53] !log search19 and search20 issue with enwiki.prefix is in {{rt|4845}} [11:49:00] Logged the message, Master [11:50:43] I will try [11:51:14] not authorized :( [11:51:20] link is https://icinga-admin.wikimedia.org/cgi-bin/icinga/cmd.cgi?cmd_typ=34&host=search19&service=search+indices+-+check+lucene+status+page [11:51:32] and https://icinga-admin.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=search20&service=search+indices+-+check+lucene+status+page [11:53:35] yeah I'm not authorizd still [11:53:36] sorry [11:53:54] at least you tried, thank you for that :-] [11:53:57] sure [11:54:03] I am sure someone will ping notpeter [11:54:40] hopefully not for several hours! [11:55:50] maybe ^demon can fix it [11:56:04] I guess now I shuld be watching the servers for blips (that the lightning deployment is done) [11:56:09] so far it's been nice and boring [11:57:20] are you referring to the icinga notifications? [11:57:26] no [11:57:36] nike rabbit's deployment [11:59:17] ohhh [12:05:29] PROBLEM - Puppet freshness on db11 is CRITICAL: Puppet has not run in the last 10 hours [12:07:09] PROBLEM - Disk space on db11 is CRITICAL: DISK CRITICAL - free space: /a 31258 MB (3% inode=99%): [12:07:41] PROBLEM - RAID on db11 is CRITICAL: CRITICAL: Defunct disk drive count: 1 [12:07:49] RECOVERY - Puppet freshness on db11 is OK: puppet ran at Thu Mar 28 12:07:48 UTC 2013 [12:08:29] PROBLEM - Puppet freshness on db11 is CRITICAL: Puppet has not run in the last 10 hours [12:08:52] RECOVERY - Puppet freshness on db11 is OK: puppet ran at Thu Mar 28 12:08:40 UTC 2013 [12:09:30] PROBLEM - Puppet freshness on db11 is CRITICAL: Puppet has not run in the last 10 hours [12:09:31] RECOVERY - Puppet freshness on db11 is OK: puppet ran at Thu Mar 28 12:09:22 UTC 2013 [12:10:29] PROBLEM - Puppet freshness on db11 is CRITICAL: Puppet has not run in the last 10 hours [12:14:39] RECOVERY - Puppet freshness on db11 is OK: puppet ran at Thu Mar 28 12:14:28 UTC 2013 [12:15:29] PROBLEM - Puppet freshness on db11 is CRITICAL: Puppet has not run in the last 10 hours [12:33:02] New patchset: Hashar; "package-builder learned 'cowbuilder'" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/56382 [12:33:54] notice: /Stage[main]/Misc::Package-builder/Misc::Package-builder::Builder[pbuilder]/Misc::Package-builder::Image[pbuilder-lucid]/Exec[imaging lucid for pbuilder]/returns: E: Could not perform immediate configuration on 'util-linux'.Please see man 5 apt.conf under APT::Immediate-Configure for details. (2) [12:33:57] that is more and more cryptic [12:34:01] i guess I should remove lucid [13:05:46] PROBLEM - Puppet freshness on db11 is CRITICAL: Puppet has not run in the last 10 hours [13:07:56] PROBLEM - RAID on db11 is CRITICAL: CRITICAL: Defunct disk drive count: 1 [13:08:26] PROBLEM - Disk space on db11 is CRITICAL: DISK CRITICAL - free space: /a 30645 MB (3% inode=99%): [13:12:36] PROBLEM - Puppet freshness on ms1004 is CRITICAL: Puppet has not run in the last 10 hours [13:56:15] Change merged: Ottomata; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/49710 [13:59:33] New patchset: Ottomata; "Fixing README and comments" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/56395 [13:59:53] Change merged: Ottomata; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/56395 [14:05:40] PROBLEM - Puppet freshness on db11 is CRITICAL: Puppet has not run in the last 10 hours [14:07:50] PROBLEM - RAID on db11 is CRITICAL: CRITICAL: Defunct disk drive count: 1 [14:08:21] PROBLEM - Disk space on db11 is CRITICAL: DISK CRITICAL - free space: /a 30099 MB (3% inode=99%): [14:17:30] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:18:20] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.142 second response time [14:40:38] New patchset: Matmarex; "(bug 45776) Set $wgCategoryCollation to 'uca-uk' on all Ukrainian-language wikis" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/56400 [14:40:53] New patchset: Ottomata; "Fixing some more comments" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/56401 [14:41:09] Change merged: Ottomata; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/56401 [14:50:48] New patchset: Demon; "Switch nostalgiawiki to use Nostalgia from extension" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/56402 [14:51:28] New patchset: Hashar; "package-builder learned 'cowbuilder'" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/56382 [14:51:55] New patchset: Ottomata; "Adding misc/limn.pp to manage setup of WMF hosted limn sites. Installing reportcard.wikimedia.org on stat1001." [operations/puppet] (production) - https://gerrit.wikimedia.org/r/56403 [14:56:30] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:56:32] New patchset: Ottomata; "Adding misc/limn.pp to manage setup of WMF hosted limn sites. Installing reportcard.wikimedia.org on stat1001." [operations/puppet] (production) - https://gerrit.wikimedia.org/r/56403 [14:57:20] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.162 second response time [14:58:43] New patchset: Ottomata; "Adding misc/limn.pp to manage setup of WMF hosted limn sites. Installing reportcard.wikimedia.org on stat1001." [operations/puppet] (production) - https://gerrit.wikimedia.org/r/56403 [15:06:06] New patchset: Ottomata; "Adding misc/limn.pp to manage setup of WMF hosted limn sites. Installing reportcard.wikimedia.org on stat1001." [operations/puppet] (production) - https://gerrit.wikimedia.org/r/56403 [15:09:32] PROBLEM - Puppet freshness on db11 is CRITICAL: Puppet has not run in the last 10 hours [15:11:41] PROBLEM - RAID on db11 is CRITICAL: CRITICAL: Defunct disk drive count: 1 [15:12:11] PROBLEM - Disk space on db11 is CRITICAL: DISK CRITICAL - free space: /a 29559 MB (3% inode=99%): [15:12:34] New patchset: Hashar; "package-builder learned 'cowbuilder'" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/56382 [15:22:40] New patchset: Hashar; "package-builder learned 'cowbuilder'" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/56382 [15:31:34] New review: Hashar; "Deployed on integration-jobbuilder instance using puppetmaster::self. That is generating the images ..." [operations/puppet] (production) - https://gerrit.wikimedia.org/r/56382 [15:32:31] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:33:21] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.131 second response time [15:35:12] New patchset: Demon; "In sync-dir, actually perform the syntax check" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/56105 [15:35:12] New patchset: Demon; "Move scap source location from fenari to tin" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/56104 [15:35:16] New patchset: Demon; "Basic puppetization of dsh" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/56107 [15:35:17] New patchset: Demon; "Remove some node lists" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/56108 [15:38:11] PROBLEM - Puppet freshness on lvs1004 is CRITICAL: Puppet has not run in the last 10 hours [15:38:11] PROBLEM - Puppet freshness on lvs1005 is CRITICAL: Puppet has not run in the last 10 hours [15:38:11] PROBLEM - Puppet freshness on lvs1006 is CRITICAL: Puppet has not run in the last 10 hours [15:46:07] New patchset: Ottomata; "Adding misc/limn.pp to manage setup of WMF hosted limn sites. Installing reportcard.wikimedia.org on stat1001." [operations/puppet] (production) - https://gerrit.wikimedia.org/r/56403 [15:46:17] hi hashar! [15:46:20] would you look that last one over for me? [15:46:28] https://gerrit.wikimedia.org/r/56403 [15:46:49] my main question is whether or not misc/limn.pp makes sense [15:47:32] i mean, the puppet stuff there makes sense, but i'm not sure if I should create a file called misc/limn.pp [15:47:35] or maybe just limn.pp [15:47:42] or can I put a define in role class? [15:47:44] role/limn.pp? [15:50:44] New review: Hashar; "(2 comments)" [operations/puppet] (production) C: -1; - https://gerrit.wikimedia.org/r/56348 [15:52:06] ottomata: hhhhhiiii [15:52:35] ottomata: don't you have a limn module nowadays ? [15:53:00] ah you have [15:53:04] ottomata: that should be a role class [15:53:11] ottomata: manifests/role/limn.pp [15:53:23] ottomata: consider having a role for production (default) and another for labs [15:53:37] something like: role::limn and role::limn::labs [15:56:00] New review: Ottomata; "Well, this could be an artifact of the way I created this branch." [operations/debs/python-jsonschema] (debian/experimental) - https://gerrit.wikimedia.org/r/56064 [15:56:15] hashar [15:56:20] andrew [15:56:20] well,i don't want a role::limn class directly [15:56:23] :) [15:56:29] but [15:56:32] why not ? [15:56:44] limn is just a piece of software, not a functional thing [15:56:49] that be like having a role::apache class [15:57:00] make sense [15:57:03] a role::reportcard class will make sense [15:57:13] (reportcard is a limn site) [15:57:24] essentially, that's what misc::statistics::sites::reportcard is [15:57:30] I kept that there for consistency [15:57:30] also [15:57:41] i wasn't sure if defines belonged in a role class [15:57:47] define role::limn::instance? [15:57:58] basically, I want a limn instance define that abstracts out WMF specific settings for limn instances [15:58:35]