[00:11:51] !log reedy synchronized wmf-config/InitialiseSettings.php 'wmgArticleFeedbackv5AutoArchiveEnabled to false for enwiki' [00:11:54] Logged the message, Master [00:12:56] (03PS1) 10Reedy: Set wmgArticleFeedbackv5AutoArchiveEnabled to false for enwiki [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/84469 [00:13:15] (03CR) 10Reedy: [C: 032] Set wmgArticleFeedbackv5AutoArchiveEnabled to false for enwiki [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/84469 (owner: 10Reedy) [00:19:45] (03PS1) 10Springle: fix db1037 regex [operations/puppet] - 10https://gerrit.wikimedia.org/r/84470 [00:20:36] (03CR) 10Springle: [C: 032 V: 032] fix db1037 regex [operations/puppet] - 10https://gerrit.wikimedia.org/r/84470 (owner: 10Springle) [00:22:05] (03PS1) 10Yurik: Added Tajikistan's Tcell carrier for zero [operations/puppet] - 10https://gerrit.wikimedia.org/r/84471 [00:26:02] (03PS2) 10Bsitu: Enable Echo and Thanks on various wikis [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/84427 [00:30:47] anyone wants to +2 Tajikistan? ^ [00:58:35] PROBLEM - Disk space on ms-be1011 is CRITICAL: DISK CRITICAL - /var/lib/ceph/osd/ceph-121 is not accessible: Input/output error [00:58:35] PROBLEM - search indices - check lucene status page on search1017 is CRITICAL: HTTP CRITICAL: HTTP/1.1 200 OK - pattern found - 60624 bytes in 0.017 second response time [00:58:35] PROBLEM - profiler-to-carbon on db1014 is CRITICAL: PROCS CRITICAL: 0 processes with regex args ^/usr/bin/python /usr/udpprofile/sbin/profiler-to-carbon [00:58:44] PROBLEM - Disk space on cp1059 is CRITICAL: DISK CRITICAL - free space: /srv/sda3 10034 MB (3% inode=99%): /srv/sdb3 10829 MB (3% inode=99%): [00:58:44] PROBLEM - RAID on db1031 is CRITICAL: CRITICAL: Degraded [00:58:44] PROBLEM - DPKG on analytics1003 is CRITICAL: DPKG CRITICAL dpkg reports broken packages [00:58:45] PROBLEM - mysqld processes on db44 is CRITICAL: PROCS CRITICAL: 0 processes with command name mysqld [00:58:54] PROBLEM - check_job_queue on hume is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [00:58:54] PROBLEM - RAID on db1053 is CRITICAL: CRITICAL: Degraded [01:21:07] PROBLEM - MySQL Processlist on db1051 is CRITICAL: CRIT 0 unauthenticated, 0 locked, 0 copy to table, 37 statistics [01:23:07] RECOVERY - MySQL Processlist on db1051 is OK: OK 0 unauthenticated, 0 locked, 0 copy to table, 3 statistics [01:26:07] PROBLEM - mysqld processes on db1037 is CRITICAL: PROCS CRITICAL: 0 processes with command name mysqld [01:28:06] RECOVERY - mysqld processes on db1037 is OK: PROCS OK: 1 process with command name mysqld [01:50:52] PROBLEM - MySQL Replication Heartbeat on db1047 is CRITICAL: CRIT replication delay 327 seconds [01:51:02] PROBLEM - MySQL Replication Heartbeat on db1017 is CRITICAL: CRIT replication delay 337 seconds [02:14:56] !log LocalisationUpdate completed (1.22wmf17) at Tue Sep 17 02:14:55 UTC 2013 [02:15:00] Logged the message, Master [02:24:07] PROBLEM - mysqld processes on db1037 is CRITICAL: PROCS CRITICAL: 0 processes with command name mysqld [02:25:20] !log starting xtrabackup clone for s1 from db1017 to db1037 [02:25:23] Logged the message, Master [02:28:14] !log LocalisationUpdate completed (1.22wmf16) at Tue Sep 17 02:28:14 UTC 2013 [02:28:18] Logged the message, Master [02:50:54] !log LocalisationUpdate ResourceLoader cache refresh completed at Tue Sep 17 02:50:53 UTC 2013 [02:50:57] Logged the message, Master [03:27:58] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 30 seconds [03:28:28] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.143 second response time [03:34:45] PROBLEM - MySQL Replication Heartbeat on db1017 is CRITICAL: CRIT replication delay 6560 seconds [03:34:45] PROBLEM - check_job_queue on terbium is CRITICAL: JOBQUEUE CRITICAL - the following wikis have more than 9,999 jobs: , enwiktionary (15121), dewiki (84815), enwiki (2115469), nlwiki (96573), frwiki (147858), Total (2472727) [04:26:46] PROBLEM - search indices - check lucene status page on search16 is CRITICAL: HTTP CRITICAL: HTTP/1.1 200 OK - pattern found - 58665 bytes in 0.111 second response time [04:26:47] PROBLEM - mysqld processes on db1037 is CRITICAL: PROCS CRITICAL: 0 processes with command name mysqld [04:26:47] PROBLEM - Memcached on holmium is CRITICAL: Connection refused [04:26:47] PROBLEM - MySQL Replication Heartbeat on db1047 is CRITICAL: CRIT replication delay 9689 seconds [04:28:26] PROBLEM - MySQL Processlist on db1052 is CRITICAL: CRIT 0 unauthenticated, 0 locked, 0 copy to table, 82 statistics [04:29:27] RECOVERY - MySQL Processlist on db1052 is OK: OK 0 unauthenticated, 0 locked, 0 copy to table, 1 statistics [04:32:46] RECOVERY - mysqld processes on db1037 is OK: PROCS OK: 1 process with command name mysqld [04:34:46] PROBLEM - MySQL Replication Heartbeat on db1037 is CRITICAL: CRIT replication delay 10166 seconds [04:38:27] PROBLEM - MySQL Processlist on db1052 is CRITICAL: CRIT 0 unauthenticated, 0 locked, 0 copy to table, 53 statistics [04:39:44] PROBLEM - MySQL Slave Delay on db1037 is CRITICAL: CRIT replication delay 9995 seconds [04:42:34] RECOVERY - MySQL Processlist on db1052 is OK: OK 0 unauthenticated, 0 locked, 0 copy to table, 2 statistics [05:02:16] PROBLEM - RAID on searchidx1001 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [05:03:16] RECOVERY - RAID on searchidx1001 is OK: OK: State is Optimal, checked 4 logical device(s) [05:05:56] RECOVERY - MySQL Slave Delay on db1037 is OK: OK replication delay 114 seconds [05:05:57] RECOVERY - MySQL Replication Heartbeat on db1037 is OK: OK replication delay 109 seconds [05:11:22] (03PS1) 10Springle: deploy db1037 s1 slave [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/84483 [05:12:30] (03CR) 10Springle: [C: 032 V: 032] deploy db1037 s1 slave [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/84483 (owner: 10Springle) [05:13:16] !log springle synchronized wmf-config/db-eqiad.php [05:13:20] Logged the message, Master [05:13:40] !log springle synchronized wmf-config/db-pmtpa.php [05:13:43] Logged the message, Master [05:13:51] !log deployed db1037 as s1 slave [05:13:54] Logged the message, Master [05:17:44] springle: you seem to be fitting well into the wikimedia-i-work-silly-hours group ;) [05:18:09] it's 3 PM, no? [05:19:14] Reedy: ;) [05:19:33] ori-l: No, it's 0620 [05:34:05] (03PS1) 10Springle: adjust s1 slave load after db1037 warm-up [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/84485 [05:35:39] (03CR) 10Springle: [C: 032 V: 032] adjust s1 slave load after db1037 warm-up [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/84485 (owner: 10Springle) [05:37:19] !log springle synchronized wmf-config/db-eqiad.php [06:05:18] PROBLEM - RAID on db1004 is CRITICAL: CRITICAL: Degraded [08:51:23] apergos: i should, but having too much fun trolling someone on #mediawiki [08:52:12] :-D [08:52:15] I'm watching over there actually, I would like to see huggle with python extensibility [08:52:18] I'm less excited about being involved in a c++ rewrite (ugh) [08:52:39] YuviPanda, http://xkcd.com/386/ [08:53:10] yuri_k: i should probably get deploy rights while i'm here [08:53:35] yuri_k: hehe :D [08:53:40] YuviPanda, nah, that would imply that you might be stuck all night deploying crap ;) [08:53:40] i'd actually like that ;) more excuses when I inevitably wake up late [08:53:40] while apergos complains about c++... [08:53:40] apergos: i'd actually like a JS based thing, really. [08:53:52] oh it has its place, for example I support the notion of the parsoid rewrite in c++ [08:54:00] I just don't live the idea for huggle [08:54:02] *love [08:54:15] wasn't huggle originally piles of horrid js? [08:54:17] apergos: works on browser, ships locally with a chromeframe when needed [08:54:31] apergos, i won't support any rewrite unless its in c# [08:54:44] bbaaahhhh [08:55:02] btw, i will stick around because my laundry is still spinning [08:55:08] what's peak time for mobile usage anyways? [08:55:16] yuri_k: goddamn, I should do laundry at some point [08:55:18] will run out tomorrow [08:55:56] YuviPanda, are you in club q? [08:56:00] yuri_k: yeah [08:56:04] yuri_k: 5th floor too :P [08:56:10] i'm downstairs in the laundry room [08:56:20] wifi works everywhere! [08:56:23] hehe [08:56:23] yeah [08:56:25] amaszing [08:56:26] isn't it? [08:56:39] i know... like... its the future ! [08:56:42] :D [08:56:51] the future will be simless mobiles, man [08:56:57] if only laptop batterry would last for a year instead of a few hrs [08:57:07] haswell! [08:57:08] no no no, i like sims [08:57:13] brrrdurrr [08:57:16] nah, 20% extra [08:57:22] not much [08:57:23] apergos: sorry to pollute your channel with random chatter :) [08:57:24] hmm in 3-4 hours according to the graphs we will have peak (how is that possible, all those us smartphone users will be sleeping) [08:57:35] why, we're not doing other work in here at the moment so it's fine [08:57:37] * YuviPanda doesn't trust most graphs anyway [08:57:48] apergos: i like apologizing to people. [08:57:50] i think? [08:57:53] :-D [08:57:58] europe will activelly read en [08:58:10] * yuri_k hates ppl [08:58:15] yes but I would expect that to be trumped by us usage later in the day [08:58:23] I hate all the non-apple manufacturers of laptop. [08:58:29] all I want is a haswell ultrabook [08:58:29] not true though according to the graphs that yuvi mistrusts [08:58:31] is that so hard to get? [08:59:28] is that the one you were linking to earlier with the high res screen? [08:59:30] apergos: the graphs might be correct, sure! I just still don't trust them fully. [08:59:36] problem with the ultrabooks is 1) I want my cable. I don't want to give up a usb port for ethernet. 2) I want my memory and cpu. I crunch things and run vms on my laptop. It's gotta hold up. [09:00:52] oh, that I believe more... [09:00:54] apergos: hah, right. I'm optimizing for battery and screen resolution [09:01:05] apergos, if you find a good laptop with 8+ ram, ssd, 6+ hrs bat life, and must not be apple, i will take it too :) [09:01:19] apergos: Samsing ATIV 9 plus, and Acer S7. [09:01:22] err [09:01:23] yuri_k: ^ [09:01:27] fits all your criteria [09:01:34] YuviPanda, yeah, i was looking at samsungs [09:01:37] thinking... [09:01:39] tbd [09:01:41] ATIV not yet widely available [09:01:50] but has a fucking highres screen [09:01:51] YuviPanda, they haven't released haswell yet [09:01:53] I don't look at apple, they don't have the specs I need [09:01:56] (promised though) [09:02:11] apergos: yuri_k the S7 also has 2 SSDs on RAID 0 [09:02:22] yuri_k: oh, they announced it ~2 weeks ago ;) [09:02:22] Samsung, that is [09:02:26] shipping in a week [09:02:27] yep [09:02:35] will believe it when i see it [09:02:36] S7 is already available tho [09:02:54] apergos: but yeah, not for heavy VM work. [09:03:05] apergos: but I'm planning on putting linux on it and running puppet to manage it from day0 [09:03:06] so... [09:03:33] I run parallel dumps and crunch dumps on my box, among other things [09:03:45] yeah [09:04:00] apergos: so when I used to stay in one spot, I had this super powerful desktop in a corner... [09:04:05] and then rats ate it. [09:04:06] yeah [09:04:11] :( [09:04:39] well I didn't have rats eat mine, but I had to give it away when I moved (it was several years old by then and would have needed serious upgrades) [09:05:27] 1.9ghz base. meh [09:05:50] so I'm only off by 12 hours, peak is in about 15 hours :-D [09:05:57] bleh 465 Warning: Recursion detected in RequestContext::getLanguage in /usr/local/apache/common-local/php-1.22wmf16/includes/context/RequestContext.php on line 281 [09:05:58] you will all be awake for it :-P [09:06:08] i know this is not new [09:06:10] yuri_k: is considered 'normal', according to Reedy earlier today. [09:06:15] apergos: heh :) [09:06:15] yep [09:06:28] apergos: i am waiting to settle in one place so I can build another powerful beast [09:06:57] hmm there was a little spikey for the swift boxen [09:07:06] almost done drying... [09:07:21] seems to be mostly settled [09:08:47] * yuri_k wonders what is swift... the messaging thingy? [09:09:06] no, that's our distributed filesystem that holds all media [09:10:05] we used t have various combinations of 'all images on one box served by nfs, all thumbs on another with nfs, a cache only mirror over here, etc' [09:10:19] at some point you realize that's not gonna scale [09:10:37] apergos, nice, sounds similar to google fs but for files instead of blocks [09:10:51] just read openstack info on it [09:11:00] (03PS1) 10TTO: Add Cookbook and Wikijunior namespaces for hywikibooks [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/84501 [09:11:07] pfft images, who needs those [09:11:16] people who are visual learners [09:11:26] * apergos rains on p858snake|l's parade [09:11:47] * YuviPanda takes away p858snake|l's browser [09:12:04] "a picture is worth a thousand words' [09:12:09] /nick jidanni [09:12:10] >.> [09:12:14] ew all know what the driving force behind the internet was... [09:12:57] * YuviPanda styles p858snake|l to be a particular shade of suave [09:13:51] Pfft, We all know I'm Pantone 2955C. [09:14:29] borrring [09:15:31] * p858snake|l traps apergos in a room of badly designed power point presentations [09:15:52] * apergos screams [09:18:18] sonoffa [09:18:20] http://news.softpedia.com/news/Security-Firm-VUPEN-Confirms-the-NSA-Bought-Exploits-Service-Subscription-383597.shtml [09:18:37] https://www.muckrock.com/foi/united-states-of-america-10/vupen-contracts-with-nsa-6593/ [09:18:47] fscking nsa [09:23:06] i think we should also subscribe to it and get all our users to run SETI... [09:23:26] or at least protein folding [09:24:00] while playing the popcorn song... [09:25:37] ok, off to bed, apergos let me know if something fails... tomorrow [09:28:51] sleep well [10:15:35] (03PS2) 10Petr Onderka: Simple progress reporting for idumps create [operations/dumps/incremental] (gsoc) - 10https://gerrit.wikimedia.org/r/84363 [11:53:55] (03CR) 10Petr Onderka: [C: 032 V: 032] Simple progress reporting for idumps create [operations/dumps/incremental] (gsoc) - 10https://gerrit.wikimedia.org/r/84363 (owner: 10Petr Onderka) [12:22:28] (03CR) 10QChris: "(2 comments)" [operations/puppet] - 10https://gerrit.wikimedia.org/r/84338 (owner: 10Diederik) [12:30:24] (03PS3) 10Dereckson: Add patrolmark right to autoconfirmed user group on ckb.wikipedia [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/84206 (owner: 10Ladsgroup) [12:32:04] (03PS4) 10Dereckson: Add patrolmark right to autoconfirmed user group on ckb.wikipedia [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/84206 (owner: 10Ladsgroup) [12:34:08] (03CR) 10Dereckson: "Ladsgroup, here you can solve this kind of issues." [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/84206 (owner: 10Ladsgroup) [12:49:25] (03PS1) 10Dereckson: Break lines in review Gerrit comments [operations/puppet] - 10https://gerrit.wikimedia.org/r/84522 [12:49:56] (03CR) 10Dereckson: "Follow-up: change Ie4a26b723aec88f17300855b904a6ddd1324328d" [operations/puppet] - 10https://gerrit.wikimedia.org/r/21895 (owner: 10Dereckson) [13:24:07] (03Abandoned) 10Hashar: initial debian directory [operations/debs/python-gear] - 10https://gerrit.wikimedia.org/r/84022 (owner: 10Hashar) [14:03:51] Reedy: if you are around, could you create a gerrit repository for me at https://gerrit.wikimedia.org/r/#/admin/projects/mediawiki/extensions/Elasticsearch [14:04:13] you offered last night and I told you chad was going to. but he didn't. [14:08:50] Hello manybubbles [14:10:06] hi Nemo_bis ! [14:10:08] how are you doing? [14:10:57] I spent the morning doing some bug triaging and WLM paperwork and now I'm testing CirrusSearch on it.wikt :) [14:12:18] Nemo_bis: thanks! I had a look at it and some parts looked better (templates resolved) but I can't tell how the actual search part is working because I don't understand the articles. [14:13:20] manybubbles: yeah, there's also less template expansion magic than in bigger wiktionaries [14:13:53] Nemo_bis: yeah. the engliash wiktionary's search can get pretty unreadable. [14:14:18] this is rather good for instance https://it.wiktionary.org/w/index.php?title=Speciale%3ARicerca&profile=default&search=bellissimi&fulltext=Search&srbackend=CirrusSearch [14:14:45] "very beautiful", male plural, finds the dictionary form without comparative and puts it first [14:17:36] cool! [14:20:18] The ranking seems to be strictly by number of incoming links. The exact title match comes after another term which matches in full text only but has 8 links instead of 7. [14:20:34] Will need to play mor with it. :) [14:21:48] hashar: sounds good to me [14:33:05] AzaToth: :-) [14:33:41] back to moar accounting [14:57:25] manybubbles: dropping this here for you. We had some poolqueue full issues before, but they weren't conclusive (and I don't know exactly how they relate to everything), but, I told the VPT to let me know if they run into it again, well: https://en.wikipedia.org/wiki/User_talk:Greg_%28WMF%29#Pool_queue_is_full_.26_timeouts [14:58:33] greg-g: I'll have a look and see if there is something up in prod. I frequently see lock up like things in beta but beta is beta [14:59:19] greg-g: and since we've got the pool counter set up around search this isn't super surprising. also, we just picked numbers for those out of thin air. [14:59:20] manybubbles: right right [14:59:30] oh, 'good' ;) [15:10:59] (03PS1) 10Petr Onderka: Handle IPv6 addresses that use :: [operations/dumps/incremental] (gsoc) - 10https://gerrit.wikimedia.org/r/84530 [15:20:16] (03PS2) 10Petr Onderka: Handle IPv6 addresses that use :: [operations/dumps/incremental] (gsoc) - 10https://gerrit.wikimedia.org/r/84530 [15:20:56] (03CR) 10Petr Onderka: [C: 032 V: 032] Handle IPv6 addresses that use :: [operations/dumps/incremental] (gsoc) - 10https://gerrit.wikimedia.org/r/84530 (owner: 10Petr Onderka) [15:37:19] (03PS1) 10Petr Onderka: Fixes issue with vector.erase() in GCC [operations/dumps/incremental] (gsoc) - 10https://gerrit.wikimedia.org/r/84533 [15:39:55] (03PS2) 10Petr Onderka: Fixes issue with vector.erase() in GCC [operations/dumps/incremental] (gsoc) - 10https://gerrit.wikimedia.org/r/84533 [15:40:24] (03CR) 10Petr Onderka: [C: 032 V: 032] Fixed issue with vector.erase() in GCC [operations/dumps/incremental] (gsoc) - 10https://gerrit.wikimedia.org/r/84533 (owner: 10Petr Onderka) [15:57:04] (03CR) 10Anomie: "(1 comment)" [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/83225 (owner: 10Umherirrender) [16:03:23] !log deactivated a few peers on ams-ix in order to reduce traffic on congested port [16:03:27] Logged the message, Mistress of the network gear. [16:25:45] (03PS1) 10Reedy: Add Math to extension-list [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/84536 [16:28:45] (03CR) 10Reedy: [C: 032] Add Math to extension-list [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/84536 (owner: 10Reedy) [16:28:56] (03Merged) 10jenkins-bot: Add Math to extension-list [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/84536 (owner: 10Reedy) [16:31:28] !log reedy Started syncing Wikimedia installation... : Rebuild l10ncache to include Math stuffs [16:31:32] Logged the message, Master [16:36:49] !log rebooting neon, ext3 errors (corruption?) remounted ro [16:36:53] Logged the message, Master [16:37:09] I'm going to /forcefsck it and reboot it again when it gets back [16:47:16] mountall: fsck / [391] terminated with status 4 [16:47:16] mountall: Filesystem has errors: / [16:47:27] Inodes that were part of a corrupted orphan linked list found. Fix? yes [16:47:30] Inode 392281 was part of the orphaned inode list. FIXED. [16:47:36] fun [16:59:53] !log reedy Finished syncing Wikimedia installation... : Rebuild l10ncache to include Math stuffs [16:59:57] Logged the message, Master [17:03:56] RECOVERY - Puppet freshness on mw1037 is OK: puppet ran at Tue Sep 17 17:03:55 UTC 2013 [17:03:56] RECOVERY - Puppet freshness on cp1045 is OK: puppet ran at Tue Sep 17 17:03:55 UTC 2013 [17:04:06] RECOVERY - Puppet freshness on lvs3 is OK: puppet ran at Tue Sep 17 17:04:00 UTC 2013 [17:04:06] RECOVERY - Puppet freshness on lvs4 is OK: puppet ran at Tue Sep 17 17:04:00 UTC 2013 [17:04:06] RECOVERY - Puppet freshness on pc2 is OK: puppet ran at Tue Sep 17 17:04:00 UTC 2013 [17:04:06] RECOVERY - Puppet freshness on mw1160 is OK: puppet ran at Tue Sep 17 17:04:00 UTC 2013 [17:04:06] RECOVERY - Puppet freshness on sq67 is OK: puppet ran at Tue Sep 17 17:04:00 UTC 2013 [17:04:16] RECOVERY - Puppet freshness on mc1015 is OK: puppet ran at Tue Sep 17 17:04:05 UTC 2013 [17:04:36] RECOVERY - Puppet freshness on cp3005 is OK: puppet ran at Tue Sep 17 17:04:20 UTC 2013 [17:04:36] RECOVERY - Puppet freshness on db52 is OK: puppet ran at Tue Sep 17 17:04:25 UTC 2013 [17:04:36] RECOVERY - Puppet freshness on mw1113 is OK: puppet ran at Tue Sep 17 17:04:25 UTC 2013 [17:04:36] RECOVERY - Puppet freshness on stafford is OK: puppet ran at Tue Sep 17 17:04:30 UTC 2013 [17:04:36] RECOVERY - Puppet freshness on snapshot1003 is OK: puppet ran at Tue Sep 17 17:04:30 UTC 2013 [17:04:37] RECOVERY - Puppet freshness on tmh2 is OK: puppet ran at Tue Sep 17 17:04:30 UTC 2013 [17:04:37] RECOVERY - Puppet freshness on mw51 is OK: puppet ran at Tue Sep 17 17:04:30 UTC 2013 [17:04:38] RECOVERY - Puppet freshness on search28 is OK: puppet ran at Tue Sep 17 17:04:30 UTC 2013 [17:04:38] RECOVERY - Puppet freshness on mw16 is OK: puppet ran at Tue Sep 17 17:04:30 UTC 2013 [17:04:39] RECOVERY - Puppet freshness on mw1187 is OK: puppet ran at Tue Sep 17 17:04:30 UTC 2013 [17:04:39] RECOVERY - Puppet freshness on srv300 is OK: puppet ran at Tue Sep 17 17:04:30 UTC 2013 [17:04:40] RECOVERY - Puppet freshness on search13 is OK: puppet ran at Tue Sep 17 17:04:30 UTC 2013 [17:04:46] RECOVERY - Puppet freshness on cp3003 is OK: puppet ran at Tue Sep 17 17:04:35 UTC 2013 [17:04:46] RECOVERY - Puppet freshness on mw1164 is OK: puppet ran at Tue Sep 17 17:04:35 UTC 2013 [17:04:46] RECOVERY - Puppet freshness on mw1217 is OK: puppet ran at Tue Sep 17 17:04:35 UTC 2013 [17:04:46] RECOVERY - Puppet freshness on mw1176 is OK: puppet ran at Tue Sep 17 17:04:35 UTC 2013 [17:04:46] RECOVERY - Puppet freshness on mw1100 is OK: puppet ran at Tue Sep 17 17:04:35 UTC 2013 [17:04:47] RECOVERY - Puppet freshness on analytics1012 is OK: puppet ran at Tue Sep 17 17:04:35 UTC 2013 [17:04:47] RECOVERY - Puppet freshness on mw1088 is OK: puppet ran at Tue Sep 17 17:04:40 UTC 2013 [17:04:48] RECOVERY - Puppet freshness on mw1117 is OK: puppet ran at Tue Sep 17 17:04:40 UTC 2013 [17:04:48] RECOVERY - Puppet freshness on mw1099 is OK: puppet ran at Tue Sep 17 17:04:40 UTC 2013 [17:04:49] RECOVERY - Puppet freshness on mc1003 is OK: puppet ran at Tue Sep 17 17:04:41 UTC 2013 [17:04:49] RECOVERY - Puppet freshness on srv287 is OK: puppet ran at Tue Sep 17 17:04:41 UTC 2013 [17:04:50] RECOVERY - Puppet freshness on search1010 is OK: puppet ran at Tue Sep 17 17:04:41 UTC 2013 [17:04:56] RECOVERY - Puppet freshness on mw1103 is OK: puppet ran at Tue Sep 17 17:04:46 UTC 2013 [17:04:56] RECOVERY - Puppet freshness on sq86 is OK: puppet ran at Tue Sep 17 17:04:51 UTC 2013 [17:04:56] RECOVERY - Puppet freshness on mw1018 is OK: puppet ran at Tue Sep 17 17:04:51 UTC 2013 [17:04:56] RECOVERY - Puppet freshness on amssq52 is OK: puppet ran at Tue Sep 17 17:04:51 UTC 2013 [17:04:56] RECOVERY - Puppet freshness on sq83 is OK: puppet ran at Tue Sep 17 17:04:51 UTC 2013 [17:05:06] RECOVERY - Puppet freshness on sq71 is OK: puppet ran at Tue Sep 17 17:04:56 UTC 2013 [17:05:06] RECOVERY - Puppet freshness on db40 is OK: puppet ran at Tue Sep 17 17:04:56 UTC 2013 [17:05:06] RECOVERY - Puppet freshness on es1 is OK: puppet ran at Tue Sep 17 17:05:01 UTC 2013 [17:05:12] (03PS1) 10J: Set $wgFFmpegThreads to 2 for faster transcodes [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/84539 [17:06:06] RECOVERY - Puppet freshness on labsdb1003 is OK: puppet ran at Tue Sep 17 17:05:56 UTC 2013 [17:06:06] RECOVERY - Puppet freshness on williams is OK: puppet ran at Tue Sep 17 17:05:56 UTC 2013 [17:06:06] RECOVERY - Puppet freshness on cp1020 is OK: puppet ran at Tue Sep 17 17:05:56 UTC 2013 [17:06:06] RECOVERY - Puppet freshness on db1019 is OK: puppet ran at Tue Sep 17 17:05:56 UTC 2013 [17:06:06] RECOVERY - Puppet freshness on es1003 is OK: puppet ran at Tue Sep 17 17:05:56 UTC 2013 [17:06:06] RECOVERY - Puppet freshness on search17 is OK: puppet ran at Tue Sep 17 17:05:56 UTC 2013 [17:06:06] RECOVERY - Puppet freshness on mw34 is OK: puppet ran at Tue Sep 17 17:05:56 UTC 2013 [17:07:16] RECOVERY - Puppet freshness on es10 is OK: puppet ran at Tue Sep 17 17:07:06 UTC 2013 [17:07:16] RECOVERY - Puppet freshness on pc1002 is OK: puppet ran at Tue Sep 17 17:07:06 UTC 2013 [17:07:16] RECOVERY - Puppet freshness on mw49 is OK: puppet ran at Tue Sep 17 17:07:06 UTC 2013 [17:07:57] RECOVERY - Puppet freshness on db1052 is OK: puppet ran at Tue Sep 17 17:07:46 UTC 2013 [17:07:57] RECOVERY - Puppet freshness on srv237 is OK: puppet ran at Tue Sep 17 17:07:46 UTC 2013 [17:07:57] RECOVERY - Puppet freshness on mw9 is OK: puppet ran at Tue Sep 17 17:07:51 UTC 2013 [17:07:57] RECOVERY - Puppet freshness on cp1006 is OK: puppet ran at Tue Sep 17 17:07:51 UTC 2013 [17:07:57] RECOVERY - Puppet freshness on search14 is OK: puppet ran at Tue Sep 17 17:07:51 UTC 2013 [17:07:58] RECOVERY - Puppet freshness on search21 is OK: puppet ran at Tue Sep 17 17:07:51 UTC 2013 [17:07:58] RECOVERY - Puppet freshness on analytics1010 is OK: puppet ran at Tue Sep 17 17:07:51 UTC 2013 [17:07:59] RECOVERY - Puppet freshness on mw14 is OK: puppet ran at Tue Sep 17 17:07:51 UTC 2013 [17:07:59] RECOVERY - Puppet freshness on mw1175 is OK: puppet ran at Tue Sep 17 17:07:51 UTC 2013 [17:08:00] RECOVERY - Puppet freshness on mw1182 is OK: puppet ran at Tue Sep 17 17:07:51 UTC 2013 [17:08:00] RECOVERY - Puppet freshness on ms5 is OK: puppet ran at Tue Sep 17 17:07:51 UTC 2013 [17:08:01] RECOVERY - Puppet freshness on sq79 is OK: puppet ran at Tue Sep 17 17:07:51 UTC 2013 [17:08:01] RECOVERY - Puppet freshness on mw5 is OK: puppet ran at Tue Sep 17 17:07:51 UTC 2013 [17:08:02] RECOVERY - Puppet freshness on mw100 is OK: puppet ran at Tue Sep 17 17:07:51 UTC 2013 [17:08:02] RECOVERY - Puppet freshness on mw125 is OK: puppet ran at Tue Sep 17 17:07:51 UTC 2013 [17:08:03] RECOVERY - Puppet freshness on mw81 is OK: puppet ran at Tue Sep 17 17:07:52 UTC 2013 [17:08:03] RECOVERY - Puppet freshness on kaulen is OK: puppet ran at Tue Sep 17 17:07:52 UTC 2013 [17:08:06] RECOVERY - Puppet freshness on mw41 is OK: puppet ran at Tue Sep 17 17:07:54 UTC 2013 [17:08:08] RECOVERY - Puppet freshness on mw47 is OK: puppet ran at Tue Sep 17 17:07:56 UTC 2013 [17:08:10] RECOVERY - Puppet freshness on db1048 is OK: puppet ran at Tue Sep 17 17:08:01 UTC 2013 [17:08:12] RECOVERY - Puppet freshness on search1014 is OK: puppet ran at Tue Sep 17 17:08:01 UTC 2013 [17:08:12] RECOVERY - Puppet freshness on mw1114 is OK: puppet ran at Tue Sep 17 17:08:01 UTC 2013 [17:08:20] RECOVERY - Puppet freshness on pc1003 is OK: puppet ran at Tue Sep 17 17:08:06 UTC 2013 [17:08:21] RECOVERY - Puppet freshness on wtp1024 is OK: puppet ran at Tue Sep 17 17:08:13 UTC 2013 [17:08:21] RECOVERY - Puppet freshness on mw1044 is OK: puppet ran at Tue Sep 17 17:08:13 UTC 2013 [17:08:21] RECOVERY - Puppet freshness on mw1127 is OK: puppet ran at Tue Sep 17 17:08:13 UTC 2013 [17:08:21] RECOVERY - Puppet freshness on mw89 is OK: puppet ran at Tue Sep 17 17:08:13 UTC 2013 [17:08:21] RECOVERY - Puppet freshness on cp1050 is OK: puppet ran at Tue Sep 17 17:08:13 UTC 2013 [17:08:21] RECOVERY - Puppet freshness on cp1069 is OK: puppet ran at Tue Sep 17 17:08:13 UTC 2013 [17:08:26] RECOVERY - Puppet freshness on sq60 is OK: puppet ran at Tue Sep 17 17:08:20 UTC 2013 [17:08:27] RECOVERY - Puppet freshness on cp1051 is OK: puppet ran at Tue Sep 17 17:08:20 UTC 2013 [17:08:27] RECOVERY - Puppet freshness on mw119 is OK: puppet ran at Tue Sep 17 17:08:20 UTC 2013 [17:08:46] RECOVERY - Puppet freshness on mw1126 is OK: puppet ran at Tue Sep 17 17:08:40 UTC 2013 [17:08:59] RECOVERY - Puppet freshness on mw1130 is OK: puppet ran at Tue Sep 17 17:08:48 UTC 2013 [17:09:00] RECOVERY - Puppet freshness on mw1162 is OK: puppet ran at Tue Sep 17 17:08:48 UTC 2013 [17:09:00] RECOVERY - Puppet freshness on mw1196 is OK: puppet ran at Tue Sep 17 17:08:48 UTC 2013 [17:09:00] RECOVERY - Puppet freshness on mw38 is OK: puppet ran at Tue Sep 17 17:08:48 UTC 2013 [17:09:00] RECOVERY - Puppet freshness on mw1125 is OK: puppet ran at Tue Sep 17 17:08:48 UTC 2013 [17:09:00] RECOVERY - Puppet freshness on mw1124 is OK: puppet ran at Tue Sep 17 17:08:48 UTC 2013 [17:09:00] RECOVERY - Puppet freshness on mw83 is OK: puppet ran at Tue Sep 17 17:08:48 UTC 2013 [17:09:01] RECOVERY - Puppet freshness on mw1111 is OK: puppet ran at Tue Sep 17 17:08:48 UTC 2013 [17:09:01] RECOVERY - Puppet freshness on srv290 is OK: puppet ran at Tue Sep 17 17:08:48 UTC 2013 [17:09:02] RECOVERY - Puppet freshness on mw118 is OK: puppet ran at Tue Sep 17 17:08:48 UTC 2013 [17:09:02] RECOVERY - Puppet freshness on snapshot2 is OK: puppet ran at Tue Sep 17 17:08:48 UTC 2013 [17:09:03] RECOVERY - Puppet freshness on cp1070 is OK: puppet ran at Tue Sep 17 17:08:48 UTC 2013 [17:09:03] RECOVERY - Puppet freshness on ms-fe3001 is OK: puppet ran at Tue Sep 17 17:08:48 UTC 2013 [17:09:04] RECOVERY - Puppet freshness on sq51 is OK: puppet ran at Tue Sep 17 17:08:48 UTC 2013 [17:09:04] RECOVERY - Puppet freshness on search36 is OK: puppet ran at Tue Sep 17 17:08:48 UTC 2013 [17:09:05] RECOVERY - Puppet freshness on mw1161 is OK: puppet ran at Tue Sep 17 17:08:48 UTC 2013 [17:09:06] RECOVERY - Puppet freshness on mw1190 is OK: puppet ran at Tue Sep 17 17:08:58 UTC 2013 [17:09:06] RECOVERY - Puppet freshness on rubidium is OK: puppet ran at Tue Sep 17 17:08:58 UTC 2013 [17:09:08] RECOVERY - Puppet freshness on solr2 is OK: puppet ran at Tue Sep 17 17:09:04 UTC 2013 [17:09:09] RECOVERY - Puppet freshness on db43 is OK: puppet ran at Tue Sep 17 17:09:05 UTC 2013 [17:09:09] RECOVERY - Puppet freshness on db45 is OK: puppet ran at Tue Sep 17 17:09:05 UTC 2013 [17:09:09] RECOVERY - Puppet freshness on cp1046 is OK: puppet ran at Tue Sep 17 17:09:05 UTC 2013 [17:09:29] RECOVERY - Puppet freshness on wtp1003 is OK: puppet ran at Tue Sep 17 17:09:20 UTC 2013 [17:09:32] RECOVERY - Puppet freshness on wtp1018 is OK: puppet ran at Tue Sep 17 17:09:23 UTC 2013 [17:09:37] RECOVERY - Puppet freshness on ms-be1012 is OK: puppet ran at Tue Sep 17 17:09:29 UTC 2013 [17:09:37] RECOVERY - Puppet freshness on vanadium is OK: puppet ran at Tue Sep 17 17:09:32 UTC 2013 [17:09:37] RECOVERY - Puppet freshness on ms-be4 is OK: puppet ran at Tue Sep 17 17:09:32 UTC 2013 [17:09:37] RECOVERY - Puppet freshness on ms-be2 is OK: puppet ran at Tue Sep 17 17:09:32 UTC 2013 [17:09:37] RECOVERY - Puppet freshness on wtp1011 is OK: puppet ran at Tue Sep 17 17:09:32 UTC 2013 [17:09:37] RECOVERY - Puppet freshness on analytics1023 is OK: puppet ran at Tue Sep 17 17:09:32 UTC 2013 [17:09:37] RECOVERY - Puppet freshness on cp1063 is OK: puppet ran at Tue Sep 17 17:09:32 UTC 2013 [17:09:38] RECOVERY - Puppet freshness on db1020 is OK: puppet ran at Tue Sep 17 17:09:32 UTC 2013 [17:09:38] RECOVERY - Puppet freshness on mc1005 is OK: puppet ran at Tue Sep 17 17:09:32 UTC 2013 [17:09:39] RECOVERY - Puppet freshness on gadolinium is OK: puppet ran at Tue Sep 17 17:09:32 UTC 2013 [17:09:39] RECOVERY - Puppet freshness on srv244 is OK: puppet ran at Tue Sep 17 17:09:32 UTC 2013 [17:09:40] RECOVERY - Puppet freshness on db1036 is OK: puppet ran at Tue Sep 17 17:09:32 UTC 2013 [17:09:40] RECOVERY - Puppet freshness on fenari is OK: puppet ran at Tue Sep 17 17:09:32 UTC 2013 [17:09:48] RECOVERY - Puppet freshness on srv249 is OK: puppet ran at Tue Sep 17 17:09:37 UTC 2013 [17:09:48] RECOVERY - Puppet freshness on mw30 is OK: puppet ran at Tue Sep 17 17:09:43 UTC 2013 [17:09:48] RECOVERY - Puppet freshness on srv246 is OK: puppet ran at Tue Sep 17 17:09:43 UTC 2013 [17:09:48] RECOVERY - Puppet freshness on srv253 is OK: puppet ran at Tue Sep 17 17:09:43 UTC 2013 [17:09:48] RECOVERY - Puppet freshness on srv245 is OK: puppet ran at Tue Sep 17 17:09:43 UTC 2013 [17:09:48] RECOVERY - Puppet freshness on amssq44 is OK: puppet ran at Tue Sep 17 17:09:43 UTC 2013 [17:09:48] RECOVERY - Puppet freshness on mw1004 is OK: puppet ran at Tue Sep 17 17:09:43 UTC 2013 [17:09:49] RECOVERY - Puppet freshness on search19 is OK: puppet ran at Tue Sep 17 17:09:43 UTC 2013 [17:10:06] RECOVERY - Puppet freshness on mw76 is OK: puppet ran at Tue Sep 17 17:09:58 UTC 2013 [17:10:06] RECOVERY - Puppet freshness on sq48 is OK: puppet ran at Tue Sep 17 17:09:59 UTC 2013 [17:10:06] RECOVERY - Puppet freshness on analytics1015 is OK: puppet ran at Tue Sep 17 17:09:59 UTC 2013 [17:10:06] RECOVERY - Puppet freshness on sq85 is OK: puppet ran at Tue Sep 17 17:09:59 UTC 2013 [17:10:07] RECOVERY - Puppet freshness on sq74 is OK: puppet ran at Tue Sep 17 17:09:59 UTC 2013 [17:10:12] RECOVERY - Puppet freshness on mw1081 is OK: puppet ran at Tue Sep 17 17:10:04 UTC 2013 [17:10:12] RECOVERY - Puppet freshness on sq43 is OK: puppet ran at Tue Sep 17 17:10:04 UTC 2013 [17:10:12] RECOVERY - Puppet freshness on mw1056 is OK: puppet ran at Tue Sep 17 17:10:04 UTC 2013 [17:10:12] RECOVERY - Puppet freshness on solr3 is OK: puppet ran at Tue Sep 17 17:10:04 UTC 2013 [17:10:12] RECOVERY - Puppet freshness on mw1057 is OK: puppet ran at Tue Sep 17 17:10:04 UTC 2013 [17:10:13] RECOVERY - Puppet freshness on mw1089 is OK: puppet ran at Tue Sep 17 17:10:04 UTC 2013 [17:10:13] RECOVERY - Puppet freshness on mw1115 is OK: puppet ran at Tue Sep 17 17:10:04 UTC 2013 [17:10:14] RECOVERY - Puppet freshness on mw1146 is OK: puppet ran at Tue Sep 17 17:10:04 UTC 2013 [17:10:14] RECOVERY - Puppet freshness on mw1147 is OK: puppet ran at Tue Sep 17 17:10:04 UTC 2013 [17:10:15] RECOVERY - Puppet freshness on mw1159 is OK: puppet ran at Tue Sep 17 17:10:04 UTC 2013 [17:10:29] RECOVERY - Puppet freshness on mc1001 is OK: puppet ran at Tue Sep 17 17:10:24 UTC 2013 [17:10:29] RECOVERY - Puppet freshness on db1029 is OK: puppet ran at Tue Sep 17 17:10:24 UTC 2013 [17:10:29] RECOVERY - Puppet freshness on db58 is OK: puppet ran at Tue Sep 17 17:10:24 UTC 2013 [17:10:29] RECOVERY - Puppet freshness on testsearch1003 is OK: puppet ran at Tue Sep 17 17:10:24 UTC 2013 [17:10:29] RECOVERY - Puppet freshness on cp1049 is OK: puppet ran at Tue Sep 17 17:10:24 UTC 2013 [17:10:29] RECOVERY - Puppet freshness on snapshot1 is OK: puppet ran at Tue Sep 17 17:10:24 UTC 2013 [17:10:29] RECOVERY - Puppet freshness on solr1003 is OK: puppet ran at Tue Sep 17 17:10:24 UTC 2013 [17:10:37] RECOVERY - Puppet freshness on mw1207 is OK: puppet ran at Tue Sep 17 17:10:31 UTC 2013 [17:10:47] RECOVERY - Puppet freshness on srv276 is OK: puppet ran at Tue Sep 17 17:10:36 UTC 2013 [17:10:48] RECOVERY - Puppet freshness on search18 is OK: puppet ran at Tue Sep 17 17:10:37 UTC 2013 [17:10:48] RECOVERY - Puppet freshness on mw8 is OK: puppet ran at Tue Sep 17 17:10:42 UTC 2013 [17:10:56]