[00:00:06] do we now have any way to see full stack exceptions in prod? [00:01:37] (03PS1) 10Ori.livneh: If /a/common is a symlink, don't replace it with an empty directory [operations/puppet] - 10https://gerrit.wikimedia.org/r/96669 [00:02:08] yurik_: "full" is vague. fluorine:/a/mw-log/exception.log & ./fatal.log are the best you'll get [00:02:27] thanks ori-l ! [00:02:30] apergos: I told you that it's a bad idea to merge something that is titled "beta" that touches production, didn't I? :-) [00:02:36] salt says that terbium was the only one with a symlink [00:02:44] oh yes [00:02:49] I made that symlink myself manually [00:02:55] because all cronjobs were broken for months [00:03:01] foreachwiki didn't work [00:03:08] we oughta have that in puppet I guess [00:03:19] (03PS1) 10Jgreen: add roots to rhodium [operations/puppet] - 10https://gerrit.wikimedia.org/r/96670 [00:03:26] so it all comes home to roost [00:03:35] I gave puppet a look, saw the /a / /apache madness, then run away crying [00:03:40] :-D [00:03:58] ran even [00:04:47] ok, I'm officially off (I think I said I was checking out hurs ago and instead I got sucked into puppet cert hell, which I willl dig myself out of tomorrow morning) [00:05:00] hope things stay quiet, get some sleep soon [00:05:12] (03CR) 10Jgreen: [C: 032 V: 031] add roots to rhodium [operations/puppet] - 10https://gerrit.wikimedia.org/r/96670 (owner: 10Jgreen) [00:05:34] (03CR) 10Ori.livneh: [C: 032] If /a/common is a symlink, don't replace it with an empty directory [operations/puppet] - 10https://gerrit.wikimedia.org/r/96669 (owner: 10Ori.livneh) [00:06:20] Jeff, I'm merging your change [00:06:38] Jeff_Green1: I merged your change [00:06:43] (roots on rhodium) [00:06:54] on palladium, I mean [00:07:25] i was just gonna ask, I saw the one for tin [00:07:30] thanks! [00:07:32] gzinflate() [function.gzinflate]: data error in /usr/local/apache/common-local/php-1.23wmf4/includes/Revision.php on line 1299 looks funky [00:09:58] !log terbium: symlinked /a/common to /usr/local/apache/common. We should really get rid of the distinction. [00:10:12] Logged the message, Master [00:10:38] grr. Guest... [00:11:57] i forced a puppet run and confirmed that it manages perms but does not otherwise replace the symlink with a directory [00:12:05] (03PS1) 10Se4598: add dewiki help page for Echo [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/96673 [00:14:00] greg-g: do we have someone to merge+deploy this only minor linkchange? https://gerrit.wikimedia.org/r/96673 [00:14:07] bsitu: ^ [00:15:12] ori-l: do you know if we record warning logs anywhere? [00:15:20] (and if those have stack traces) [00:15:34] i'm not sure [00:15:50] se4598: our deployment window is over, since this is not urgent, we will try some other time [00:16:07] isn't Lightning deploy time? [00:16:14] it's tiny, i'll just merge it [00:16:26] (03CR) 10Ori.livneh: [C: 032] add dewiki help page for Echo [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/96673 (owner: 10Se4598) [00:16:29] ori-l: thx [00:16:35] (03Merged) 10jenkins-bot: add dewiki help page for Echo [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/96673 (owner: 10Se4598) [00:17:28] !log ori updated /a/common to {{Gerrit|If961f9365}}: add dewiki help page for Echo [00:17:44] Logged the message, Master [00:18:10] se4598: forgot it's lightning deploy time now, :) [00:18:21] !log ori synchronized wmf-config/InitialiseSettings.php 'If961f9365: add dewiki help page for Echo' [00:18:37] Logged the message, Master [00:20:30] ori-l: thank you so much [00:21:22] se4598: np at all [00:24:06] sigh [00:26:21] tail -f /a/mw-log/apache2 | uniq | bugzilla [00:28:04] oh look, already reported in september: https://bugzilla.wikimedia.org/show_bug.cgi?id=54193 [00:29:57] !log Resynced terbium /usr/local/apache/common-local which was totally empty [00:30:12] Logged the message, Master [00:30:56] AaronSchulz: is there a bug for: [00:30:59] Nov 21 00:28:16 mw1086: PHP Warning: apc_store() [function.apc-store]: GC cache entry '/usr/local/apache/common-local/php-1.23wmf4/extensions/PagedTiffHandler/PagedTiffHandler.image.php' (dev=2049 ino=9064492) was on gc-list for 601 seconds in /usr/local/apache/common-local/php-1.23wmf3/includes/objectcache/APCBagOStuff.php on line 62 [00:31:37] probably the same [00:31:52] yeah, i see different files being cited [00:31:59] so we regularly have +10 min reqs? [00:32:17] both description page view (on metadata upgrade) and render can hit the exif bug [00:32:33] probably the former is what happens on the random non-scaler apaches [00:32:41] of course segfaults mess up the apc ref counting [00:33:17] ah [00:34:03] odd just calling wfFindFile() on that file in eval.php just hangs [00:34:14] I mean I didn't even call getMetadata() yet ;) [00:35:17] that explains the 504, but still, I didn't call any methods yet [00:35:26] it can feel it coming [00:53:21] apergos: on second thought, if Varnish starts crashing tomorrow morning, call me [00:55:58] why so? (happy to do it, just wondering) [00:57:18] I want to grab a backtrace [00:57:24] and you were supposed to be sleeping [00:57:26] ah ha [00:57:32] not yet, I had to wait an hour after food [00:57:34] this was meant to reach your backlog [00:57:37] heh [00:58:27] in about 10 minutes the hour will have passed [01:09:06] ori-l: yeah, so wfLocalFile() is fast, but not wfFindFile()...since the later calls load() due to exists()/isCacheable() calls which trigger metadata fetches which hang the process [01:10:39] * AaronSchulz uses IRC as a clipboard between two computers [01:11:16] heh [01:14:33] yurik: columbia! :P [01:15:11] is this the cause of the left over tifs? [01:16:51] ori-l: the master of wiki-prism :-P [01:17:23] (03PS1) 10Springle: depool pmtpa db hosts, some for decom, some for shipping [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/96683 [01:18:03] (03CR) 10Springle: [C: 032] depool pmtpa db hosts, some for decom, some for shipping [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/96683 (owner: 10Springle) [01:18:53] !log springle synchronized wmf-config/db-pmtpa.php [01:19:09] Logged the message, Master [01:21:26] ooohhh more boxes going away [01:21:35] Importing largetif.tif... [01:21:42] I bet that's locking up my box :) [01:21:45] hahaha [01:21:57] need moar ram [01:22:17] read(8, "identify: Memory allocation fail"..., 8192) = 100 [01:22:19] wait4(30506, 0x7fff5e7000d4, WNOHANG|WSTOPPED, NULL) = 0 [01:22:20] select(12, [8 11], [], [], NULL [01:22:30] ...and stuck [01:24:29] !log depool db60, db54, db34, db31, db65, db47, db37 for decom and/or shipping. promote db68, db74, db72, db71, db69 to pmtpa shard masters [01:24:43] Logged the message, Master [01:26:14] that made dbtree look a lot smaller [01:35:39] wow [01:35:47] winter pruning [01:39:05] !log enwiki revision table schema change test run db60 [01:39:21] Logged the message, Master [02:17:20] !log LocalisationUpdate completed (1.23wmf4) at Thu Nov 21 02:17:19 UTC 2013 [02:17:35] Logged the message, Master [02:32:33] !log LocalisationUpdate completed (1.23wmf3) at Thu Nov 21 02:32:33 UTC 2013 [02:32:49] Logged the message, Master [03:20:28] !log LocalisationUpdate ResourceLoader cache refresh completed at Thu Nov 21 03:20:28 UTC 2013 [03:20:44] Logged the message, Master [04:50:54] quick heads up i'm running a routine w0 script to check that banners are served properly for various carriers [04:51:18] it runs daily in the middle of the day normally, but running it again in a few minutes. [04:54:20] dr0ptp4kt: it's customary to use !log for such notices [04:54:43] that way it also ends up on the SAL (=server admin log, https://wikitech.wikimedia.org/wiki/Server_admin_log) [04:57:26] ori-l, should i just manually post the message right now? [04:57:46] dr0ptp4kt: sure [04:58:07] !log Wikipedia Zero BDD script run [04:58:23] Logged the message, Master [04:58:49] ori-l, is there a page documenting the recommended way to do this automatically? i don't know if we want to fill up the log with the routine daily check messages, but maybe? [04:59:33] if they're daily and routine and low-risk, you probably don't need a log message for each one [05:00:02] the rule of thumb is that if it's unusual or significant enough to merit giving notice on the channel, it might as well go in the sal [05:00:21] you can script a bot to announce it in #wikimedia-mobile if you like [05:01:24] ori-l, cool man. thx [05:01:40] heh [05:01:43] i think i scared him away [05:01:50] * ori-l pets morebots. [05:20:49] morebots: BTW, morebots doesn't identify or its ident changed or something. [05:20:50] I am a logbot running on tools-exec-05. [05:20:50] Messages are logged to wikitech.wikimedia.org/wiki/Server_Admin_Log. [05:20:50] To log a message, type !log . [05:20:54] Helpful. [05:20:58] ori-l: ^^^ [05:21:12] So it doesn't get autovoiced. [05:21:22] Unlike icinga-wm and logmsgbot. [05:21:40] Probably some rule for neon.wm.o. [05:22:11] meh [05:22:16] bugzilla-worthy, to be sure [05:22:25] but i don't feel like dealing with that [05:23:49] Probably just needs a channel op command. [05:24:09] Perhaps Thehelpfulone will read scrollback and deal with it. [07:52:19] !log updated all remaining references to sockpuppet and stafford as puppetmaster on wikitech [07:52:35] Logged the message, Master [07:54:25] icinga? [07:54:45] Max concurrent service checks (3200) has been reached [07:55:12] neon root is r/o [07:55:45] :( [07:59:55] ext3 aborted journal again. bonce it? [07:59:58] bounce* [08:00:09] yeah [08:03:51] springle: and !log it [08:04:19] oh morebots still lives [08:04:37] !log rebooted neon, ext3 root fs read only [08:04:43] yeah, different system [08:04:51] Logged the message, Master [08:05:22] paravoid: i have a package, how can I push it? [08:05:30] and it was hard :) [08:09:37] thought I got em all (sockpuppet on wikitech), huh [08:10:43] how long should neon take to show something on mgmt console after powercycle? [08:11:05] it sure should by now.... not even a post? [08:11:40] no [08:11:48] hm [08:12:02] neon.mgmt.eqiad.wmnet right? [08:12:08] yes [08:12:39] shall I have a look? [08:12:45] yep, please [08:13:36] did you reboot or powrcycle? [08:13:51] reboot. waited. then powercycle [08:14:06] ok well I'll try again [08:16:00] well that's not good (nothing visible) but I'm going to let it sit for awhile I guess [08:21:30] (03PS1) 10Ori.livneh: Enable recursion-guard log bucket to investigate bug 54193 [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/96709 [08:22:35] paravoid: please guide me how to push the jsduck package when you have time [08:24:34] matanya: sorry, not now :( [08:25:00] np, it is waiting in my testing machine [08:26:51] how's neon? [08:27:08] nothing on the console whatsoever [08:27:16] it's definitely powered on and pings [08:27:44] so pretty hard to tell anything about what it's doing or not doing. and when I say 'nothing', it is as sprin gle says, [08:28:07] cursor sits in upper left, as though post and etc never went to serial console [08:28:20] try typing capital S [08:28:20] paravoid: http://ganglia.wikimedia.org/latest/graph.php?r=month&z=xlarge&hreg[]=client-side&mreg[]=%5Ebrowser.redirecting.%28desktop%7Cmobile%29_median%24>ype=line&title=Redirecting%3A+redirectStart+to+redirectEnd&aggregate=1 , MaxSem credits you for the fix [08:28:35] what did I do? [08:28:51] are you still awake just to get that varnish stack trace? :-P [08:28:55] yes [08:29:04] :-D [08:29:05] and to switch eqiad-esams back [08:29:11] but I won't do that without icinga [08:29:24] you usually do so many things that cant tell for which to thank:) [08:29:37] if in doubt, credit paravoid? [08:29:39] ori-l / maxsem: is it the vodafone/semc/whatever match? [08:29:44] didn't you say this was a noop? [08:29:50] or was it the LG/NEC one? [08:30:10] apergos: try hitting capital S [08:30:19] oh, i forgot about those [08:30:20] you could have lt me call you... you would have gotten a few hours anyways [08:30:40] nada [08:30:59] I'll give it another couple mins though [08:31:09] the vodafone change coudn't have fixed anything [08:31:25] MaxSem: there were two changes in that changeset [08:31:32] one was the one you said it was noop [08:31:47] the other one was a wrongly case-sensitive match becoming case insensitive [08:32:06] can't say that I ever thought either of these would make a difference, though :) [08:32:18] apergos: can I try? [08:32:24] sure, lemme get off the console [08:32:38] all yours [08:33:03] thx [08:34:21] (03CR) 10Spage: [C: 031] "We really need this for VE browser tests on beta labs. It should work and will have no effect on production wikis that don't yet enable Fl" [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/96161 (owner: 10EBernhardson) [08:35:23] hrm, I don't even get the BIOS on the console [08:35:38] that is exactly what springle as telling us [08:35:47] no post, no nothing [08:36:05] getting on the idrac [08:37:24] (03CR) 10Ori.livneh: [C: 032] Enable recursion-guard log bucket to investigate bug 54193 [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/96709 (owner: 10Ori.livneh) [08:44:04] ah the page with the sockpuppet ref was obsolete, I see [08:44:17] !log with neon down, scap / syncs not logged to irc [08:44:27] !log Synced wmf-config/InitialiseSettings.php "I5e8317db6: Enable recursion-guard log bucket to investigate bug 54193" [08:44:31] Logged the message, Master [08:44:45] Logged the message, Master [08:46:28] mmm, 54 lines in 1 stacktrace [08:46:56] how is it looking? [08:47:47] MaxSem: I wonder if that's a record [08:48:04] for MW development, maybe [08:48:44] for world, I've seen nice 3 floor struts traces screenshots [08:49:16] yup, 54 is very easy on Android too [08:49:58] nope, exception.log-20130612.gz has one with 73 [08:50:10] * ori-l is being productive [08:51:33] heh [08:55:00] hmm, so that bug appears to be caused by CA [08:57:43] you should update the bug if you discover anything [08:58:13] i got an insta-headache looking at that trace and immediately alt-tabbed to youtube [09:01:57] another part is AF: it manages to divide by 0 on page views [09:02:30] ...while executing a CA hook, of course [09:04:58] icinga looks down [09:05:27] yes, host is down [09:05:47] (03CR) 10Akosiaris: [C: 04-1] "Nitpicks. Other than that LGTM" (033 comments) [operations/puppet] - 10https://gerrit.wikimedia.org/r/96538 (owner: 10QChris) [09:06:00] ah right, neon; does this machine do anything else or can I redirect [[neon]] to [[incinga]] [09:06:05] -typos [09:06:38] (03CR) 10Akosiaris: [C: 032] "LGTM. As soon as the dependency is ammended I will merge both" [operations/puppet] - 10https://gerrit.wikimedia.org/r/95363 (owner: 10QChris) [09:07:49] Nemo_bis: it runs tcpircbot, which relays syncs / scaps to IRC [09:08:04] and it's back? [09:09:21] * apergos waits to hear the details [09:09:55] serial console is broken [09:09:59] I tried a few settings on the bios [09:10:09] I gave up after the 6th reboot or so [09:10:19] went to grub, removed console=ttyS1,... [09:10:29] so I got a proper console with output from fsck [09:10:34] it found errors, needed F to fix [09:10:40] after that it was pretty straightforward [09:10:49] ok [09:15:35] (03PS1) 10Odder: (bug 57345) Raise account creation limit for cawiki GLAM event [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/96718 [09:15:46] RECOVERY - Puppet freshness on cp1018 is OK: puppet ran at Thu Nov 21 09:15:36 UTC 2013 [09:15:46] RECOVERY - Puppet freshness on cp4019 is OK: puppet ran at Thu Nov 21 09:15:41 UTC 2013 [09:16:01] RECOVERY - Puppet freshness on ms6 is OK: puppet ran at Thu Nov 21 09:15:46 UTC 2013 [09:16:01] RECOVERY - Puppet freshness on ssl4 is OK: puppet ran at Thu Nov 21 09:15:46 UTC 2013 [09:16:01] RECOVERY - Puppet freshness on ssl1005 is OK: puppet ran at Thu Nov 21 09:15:51 UTC 2013 [09:16:01] RECOVERY - Puppet freshness on analytics1002 is OK: puppet ran at Thu Nov 21 09:15:51 UTC 2013 [09:16:01] RECOVERY - Puppet freshness on db1027 is OK: puppet ran at Thu Nov 21 09:15:56 UTC 2013 [09:16:02] RECOVERY - Puppet freshness on amssq51 is OK: puppet ran at Thu Nov 21 09:15:56 UTC 2013 [09:16:02] RECOVERY - Puppet freshness on db1026 is OK: puppet ran at Thu Nov 21 09:15:56 UTC 2013 [09:16:11] RECOVERY - Puppet freshness on srv283 is OK: puppet ran at Thu Nov 21 09:16:01 UTC 2013 [09:16:11] RECOVERY - Puppet freshness on mw61 is OK: puppet ran at Thu Nov 21 09:16:01 UTC 2013 [09:16:11] RECOVERY - Puppet freshness on cp1066 is OK: puppet ran at Thu Nov 21 09:16:01 UTC 2013 [09:16:11] RECOVERY - Puppet freshness on ms-be5 is OK: puppet ran at Thu Nov 21 09:16:01 UTC 2013 [09:16:11] RECOVERY - Puppet freshness on analytics1022 is OK: puppet ran at Thu Nov 21 09:16:01 UTC 2013 [09:16:11] RECOVERY - Puppet freshness on mw74 is OK: puppet ran at Thu Nov 21 09:16:01 UTC 2013 [09:16:12] RECOVERY - Puppet freshness on palladium is OK: puppet ran at Thu Nov 21 09:16:07 UTC 2013 [09:16:12] RECOVERY - Puppet freshness on elastic1006 is OK: puppet ran at Thu Nov 21 09:16:07 UTC 2013 [09:16:13] RECOVERY - Puppet freshness on amssq56 is OK: puppet ran at Thu Nov 21 09:16:07 UTC 2013 [09:16:13] RECOVERY - Puppet freshness on ssl1003 is OK: puppet ran at Thu Nov 21 09:16:07 UTC 2013 [09:16:21] RECOVERY - Puppet freshness on srv261 is OK: puppet ran at Thu Nov 21 09:16:12 UTC 2013 [09:16:21] RECOVERY - Puppet freshness on mw93 is OK: puppet ran at Thu Nov 21 09:16:12 UTC 2013 [09:16:21] RECOVERY - Puppet freshness on mw13 is OK: puppet ran at Thu Nov 21 09:16:12 UTC 2013 [09:16:21] RECOVERY - Puppet freshness on srv299 is OK: puppet ran at Thu Nov 21 09:16:12 UTC 2013 [09:16:21] RECOVERY - Puppet freshness on mw1128 is OK: puppet ran at Thu Nov 21 09:16:12 UTC 2013 [09:16:22] RECOVERY - Puppet freshness on mw1194 is OK: puppet ran at Thu Nov 21 09:16:12 UTC 2013 [09:16:22] RECOVERY - Puppet freshness on search1003 is OK: puppet ran at Thu Nov 21 09:16:12 UTC 2013 [09:16:23] RECOVERY - Puppet freshness on amslvs1 is OK: puppet ran at Thu Nov 21 09:16:12 UTC 2013 [09:16:23] RECOVERY - Puppet freshness on mw1105 is OK: puppet ran at Thu Nov 21 09:16:12 UTC 2013 [09:16:24] RECOVERY - Puppet freshness on search1005 is OK: puppet ran at Thu Nov 21 09:16:12 UTC 2013 [09:16:24] RECOVERY - Puppet freshness on hume is OK: puppet ran at Thu Nov 21 09:16:17 UTC 2013 [09:16:25] RECOVERY - Puppet freshness on mw1047 is OK: puppet ran at Thu Nov 21 09:16:17 UTC 2013 [09:16:25] RECOVERY - Puppet freshness on snapshot1002 is OK: puppet ran at Thu Nov 21 09:16:17 UTC 2013 [09:16:26] RECOVERY - Puppet freshness on analytics1009 is OK: puppet ran at Thu Nov 21 09:16:17 UTC 2013 [09:16:26] RECOVERY - Puppet freshness on cp1060 is OK: puppet ran at Thu Nov 21 09:16:17 UTC 2013 [09:16:31] RECOVERY - Puppet freshness on mw1084 is OK: puppet ran at Thu Nov 21 09:16:22 UTC 2013 [09:16:31] RECOVERY - Puppet freshness on sq62 is OK: puppet ran at Thu Nov 21 09:16:22 UTC 2013 [09:16:31] RECOVERY - Puppet freshness on mw1180 is OK: puppet ran at Thu Nov 21 09:16:27 UTC 2013 [09:16:31] RECOVERY - Puppet freshness on cp1016 is OK: puppet ran at Thu Nov 21 09:16:27 UTC 2013 [09:16:41] RECOVERY - Puppet freshness on mw1049 is OK: puppet ran at Thu Nov 21 09:16:32 UTC 2013 [09:16:41] RECOVERY - Puppet freshness on labstore2 is OK: puppet ran at Thu Nov 21 09:16:32 UTC 2013 [09:16:41] RECOVERY - Puppet freshness on mw1137 is OK: puppet ran at Thu Nov 21 09:16:32 UTC 2013 [09:16:41] RECOVERY - Puppet freshness on tmh1001 is OK: puppet ran at Thu Nov 21 09:16:32 UTC 2013 [09:16:41] RECOVERY - Puppet freshness on sanger is OK: puppet ran at Thu Nov 21 09:16:32 UTC 2013 [09:16:41] RECOVERY - Puppet freshness on cp4005 is OK: puppet ran at Thu Nov 21 09:16:32 UTC 2013 [09:16:42] RECOVERY - Puppet freshness on db69 is OK: puppet ran at Thu Nov 21 09:16:37 UTC 2013 [09:16:42] RECOVERY - Puppet freshness on es9 is OK: puppet ran at Thu Nov 21 09:16:37 UTC 2013 [09:16:43] RECOVERY - Puppet freshness on db67 is OK: puppet ran at Thu Nov 21 09:16:37 UTC 2013 [09:16:43] RECOVERY - Puppet freshness on amslvs4 is OK: puppet ran at Thu Nov 21 09:16:37 UTC 2013 [09:16:44] RECOVERY - Puppet freshness on hafnium is OK: puppet ran at Thu Nov 21 09:16:37 UTC 2013 [09:16:49] !log reenable ospfv3 between esams-eqiad [09:16:51] RECOVERY - Puppet freshness on tungsten is OK: puppet ran at Thu Nov 21 09:16:47 UTC 2013 [09:16:51] RECOVERY - Puppet freshness on db1005 is OK: puppet ran at Thu Nov 21 09:16:47 UTC 2013 [09:16:52] RECOVERY - Puppet freshness on solr1001 is OK: puppet ran at Thu Nov 21 09:16:47 UTC 2013 [09:16:52] RECOVERY - Puppet freshness on srv291 is OK: puppet ran at Thu Nov 21 09:16:47 UTC 2013 [09:17:01] RECOVERY - Puppet freshness on mw45 is OK: puppet ran at Thu Nov 21 09:16:52 UTC 2013 [09:17:01] RECOVERY - Puppet freshness on db1009 is OK: puppet ran at Thu Nov 21 09:16:52 UTC 2013 [09:17:01] RECOVERY - Puppet freshness on cp1062 is OK: puppet ran at Thu Nov 21 09:16:52 UTC 2013 [09:17:01] RECOVERY - Puppet freshness on search1015 is OK: puppet ran at Thu Nov 21 09:16:57 UTC 2013 [09:17:07] Logged the message, Master [09:17:11] RECOVERY - Puppet freshness on stat1002 is OK: puppet ran at Thu Nov 21 09:17:02 UTC 2013 [09:17:11] RECOVERY - Puppet freshness on mw1165 is OK: puppet ran at Thu Nov 21 09:17:02 UTC 2013 [09:17:11] RECOVERY - Puppet freshness on mw1179 is OK: puppet ran at Thu Nov 21 09:17:02 UTC 2013 [09:17:11] RECOVERY - Puppet freshness on mw1030 is OK: puppet ran at Thu Nov 21 09:17:02 UTC 2013 [09:17:11] RECOVERY - Puppet freshness on mw1181 is OK: puppet ran at Thu Nov 21 09:17:02 UTC 2013 [09:17:12] RECOVERY - Puppet freshness on mw1020 is OK: puppet ran at Thu Nov 21 09:17:02 UTC 2013 [09:17:12] RECOVERY - Puppet freshness on mw1075 is OK: puppet ran at Thu Nov 21 09:17:07 UTC 2013 [09:17:13] RECOVERY - Puppet freshness on mw1078 is OK: puppet ran at Thu Nov 21 09:17:07 UTC 2013 [09:17:13] RECOVERY - Puppet freshness on lvs1004 is OK: puppet ran at Thu Nov 21 09:17:07 UTC 2013 [09:17:27] dammit [09:17:30] not even close to a gig [09:17:31] RECOVERY - Puppet freshness on nickel is OK: puppet ran at Thu Nov 21 09:17:27 UTC 2013 [09:17:33] I'm too early [09:17:41] RECOVERY - Puppet freshness on db77 is OK: puppet ran at Thu Nov 21 09:17:32 UTC 2013 [09:17:41] RECOVERY - Puppet freshness on ms-fe2 is OK: puppet ran at Thu Nov 21 09:17:32 UTC 2013 [09:17:41] RECOVERY - Puppet freshness on cp4015 is OK: puppet ran at Thu Nov 21 09:17:37 UTC 2013 [09:17:41] RECOVERY - Puppet freshness on amssq35 is OK: puppet ran at Thu Nov 21 09:17:37 UTC 2013 [09:17:51] RECOVERY - Puppet freshness on amssq37 is OK: puppet ran at Thu Nov 21 09:17:42 UTC 2013 [09:17:51] RECOVERY - Puppet freshness on amssq31 is OK: puppet ran at Thu Nov 21 09:17:42 UTC 2013 [09:17:51] RECOVERY - Puppet freshness on db1058 is OK: puppet ran at Thu Nov 21 09:17:47 UTC 2013 [09:17:51] RECOVERY - Puppet freshness on mw95 is OK: puppet ran at Thu Nov 21 09:17:47 UTC 2013 [09:17:51] RECOVERY - Puppet freshness on analytics1008 is OK: puppet ran at Thu Nov 21 09:17:47 UTC 2013 [09:18:01] RECOVERY - Puppet freshness on srv272 is OK: puppet ran at Thu Nov 21 09:17:52 UTC 2013 [09:18:01] RECOVERY - Puppet freshness on sq68 is OK: puppet ran at Thu Nov 21 09:17:52 UTC 2013 [09:18:01] RECOVERY - Puppet freshness on mw99 is OK: puppet ran at Thu Nov 21 09:17:52 UTC 2013 [09:18:01] RECOVERY - Puppet freshness on cp1038 is OK: puppet ran at Thu Nov 21 09:17:52 UTC 2013 [09:18:01] RECOVERY - Puppet freshness on db65 is OK: puppet ran at Thu Nov 21 09:17:52 UTC 2013 [09:18:01] RECOVERY - Puppet freshness on lvs6 is OK: puppet ran at Thu Nov 21 09:17:52 UTC 2013 [09:18:02] RECOVERY - Puppet freshness on wtp1002 is OK: puppet ran at Thu Nov 21 09:17:52 UTC 2013 [09:18:02] RECOVERY - Puppet freshness on erbium is OK: puppet ran at Thu Nov 21 09:17:57 UTC 2013 [09:18:03] RECOVERY - Puppet freshness on mw39 is OK: puppet ran at Thu Nov 21 09:17:57 UTC 2013 [09:18:03] RECOVERY - Puppet freshness on mw1198 is OK: puppet ran at Thu Nov 21 09:17:57 UTC 2013 [09:18:11] RECOVERY - Puppet freshness on srv264 is OK: puppet ran at Thu Nov 21 09:18:02 UTC 2013 [09:18:11] RECOVERY - Puppet freshness on lvs4001 is OK: puppet ran at Thu Nov 21 09:18:07 UTC 2013 [09:18:11] RECOVERY - Puppet freshness on search1021 is OK: puppet ran at Thu Nov 21 09:18:07 UTC 2013 [09:18:12] RECOVERY - Puppet freshness on ms-be1009 is OK: puppet ran at Thu Nov 21 09:18:07 UTC 2013 [09:18:21] RECOVERY - Puppet freshness on search1023 is OK: puppet ran at Thu Nov 21 09:18:12 UTC 2013 [09:18:21] RECOVERY - Puppet freshness on mw1169 is OK: puppet ran at Thu Nov 21 09:18:12 UTC 2013 [09:18:21] RECOVERY - Puppet freshness on ms-be1007 is OK: puppet ran at Thu Nov 21 09:18:17 UTC 2013 [09:18:21] RECOVERY - Puppet freshness on mw1136 is OK: puppet ran at Thu Nov 21 09:18:17 UTC 2013 [09:18:21] RECOVERY - Puppet freshness on mw1116 is OK: puppet ran at Thu Nov 21 09:18:17 UTC 2013 [09:18:22] RECOVERY - Puppet freshness on mw1074 is OK: puppet ran at Thu Nov 21 09:18:17 UTC 2013 [09:18:22] RECOVERY - Puppet freshness on mw1094 is OK: puppet ran at Thu Nov 21 09:18:17 UTC 2013 [09:18:23] RECOVERY - Puppet freshness on mw1083 is OK: puppet ran at Thu Nov 21 09:18:17 UTC 2013 [09:18:31] RECOVERY - Puppet freshness on mw1053 is OK: puppet ran at Thu Nov 21 09:18:22 UTC 2013 [09:18:31] RECOVERY - Puppet freshness on brewster is OK: puppet ran at Thu Nov 21 09:18:27 UTC 2013 [09:18:31] RECOVERY - Puppet freshness on sq50 is OK: puppet ran at Thu Nov 21 09:18:27 UTC 2013 [09:18:41] RECOVERY - Puppet freshness on es5 is OK: puppet ran at Thu Nov 21 09:18:32 UTC 2013 [09:18:41] RECOVERY - Puppet freshness on dataset2 is OK: puppet ran at Thu Nov 21 09:18:32 UTC 2013 [09:18:41] RECOVERY - Puppet freshness on cp4017 is OK: puppet ran at Thu Nov 21 09:18:32 UTC 2013 [09:18:51] RECOVERY - Puppet freshness on mc1008 is OK: puppet ran at Thu Nov 21 09:18:42 UTC 2013 [09:18:51] RECOVERY - Puppet freshness on mw117 is OK: puppet ran at Thu Nov 21 09:18:42 UTC 2013 [09:18:51] RECOVERY - Puppet freshness on db1024 is OK: puppet ran at Thu Nov 21 09:18:42 UTC 2013 [09:18:51] RECOVERY - Puppet freshness on mw36 is OK: puppet ran at Thu Nov 21 09:18:42 UTC 2013 [09:18:51] RECOVERY - Puppet freshness on mw26 is OK: puppet ran at Thu Nov 21 09:18:42 UTC 2013 [09:18:52] RECOVERY - Puppet freshness on mw33 is OK: puppet ran at Thu Nov 21 09:18:47 UTC 2013 [09:18:52] RECOVERY - Puppet freshness on wtp1013 is OK: puppet ran at Thu Nov 21 09:18:47 UTC 2013 [09:18:53] RECOVERY - Puppet freshness on mw64 is OK: puppet ran at Thu Nov 21 09:18:47 UTC 2013 [09:18:53] RECOVERY - Puppet freshness on cp1065 is OK: puppet ran at Thu Nov 21 09:18:47 UTC 2013 [09:19:01] RECOVERY - Puppet freshness on db57 is OK: puppet ran at Thu Nov 21 09:18:52 UTC 2013 [09:19:01] RECOVERY - Puppet freshness on mw1138 is OK: puppet ran at Thu Nov 21 09:18:57 UTC 2013 [09:19:01] RECOVERY - Puppet freshness on analytics1011 is OK: puppet ran at Thu Nov 21 09:18:57 UTC 2013 [09:19:01] RECOVERY - Puppet freshness on analytics1019 is OK: puppet ran at Thu Nov 21 09:18:57 UTC 2013 [09:19:32] RECOVERY - Puppet freshness on tarin is OK: puppet ran at Thu Nov 21 09:19:27 UTC 2013 [09:19:41] RECOVERY - Puppet freshness on ssl1007 is OK: puppet ran at Thu Nov 21 09:19:32 UTC 2013 [09:19:41] RECOVERY - Puppet freshness on testsearch1002 is OK: puppet ran at Thu Nov 21 09:19:37 UTC 2013 [09:19:42] RECOVERY - Puppet freshness on snapshot4 is OK: puppet ran at Thu Nov 21 09:19:37 UTC 2013 [09:19:51] RECOVERY - Puppet freshness on labsdb1001 is OK: puppet ran at Thu Nov 21 09:19:42 UTC 2013 [09:19:51] RECOVERY - Puppet freshness on mw107 is OK: puppet ran at Thu Nov 21 09:19:43 UTC 2013 [09:19:51] RECOVERY - Puppet freshness on db1010 is OK: puppet ran at Thu Nov 21 09:19:43 UTC 2013 [09:19:51] RECOVERY - Puppet freshness on mw40 is OK: puppet ran at Thu Nov 21 09:19:43 UTC 2013 [09:19:51] RECOVERY - Puppet freshness on wtp1021 is OK: puppet ran at Thu Nov 21 09:19:48 UTC 2013 [09:19:52] RECOVERY - Puppet freshness on cp1043 is OK: puppet ran at Thu Nov 21 09:19:48 UTC 2013 [09:20:01] RECOVERY - Puppet freshness on mw109 is OK: puppet ran at Thu Nov 21 09:19:53 UTC 2013 [09:20:01] RECOVERY - Puppet freshness on mw70 is OK: puppet ran at Thu Nov 21 09:19:53 UTC 2013 [09:20:01] RECOVERY - Puppet freshness on srv296 is OK: puppet ran at Thu Nov 21 09:19:53 UTC 2013 [09:20:01] RECOVERY - Puppet freshness on mw1185 is OK: puppet ran at Thu Nov 21 09:19:53 UTC 2013 [09:20:01] RECOVERY - Puppet freshness on es1009 is OK: puppet ran at Thu Nov 21 09:19:58 UTC 2013 [09:20:01] RECOVERY - Puppet freshness on mw1218 is OK: puppet ran at Thu Nov 21 09:19:58 UTC 2013 [09:20:02] RECOVERY - Puppet freshness on mw1132 is OK: puppet ran at Thu Nov 21 09:19:58 UTC 2013 [09:20:02] RECOVERY - Puppet freshness on mw1040 is OK: puppet ran at Thu Nov 21 09:19:58 UTC 2013 [09:20:03] RECOVERY - Puppet freshness on mw53 is OK: puppet ran at Thu Nov 21 09:19:58 UTC 2013 [09:20:11] RECOVERY - Puppet freshness on mw1001 is OK: puppet ran at Thu Nov 21 09:20:03 UTC 2013 [09:20:11] RECOVERY - Puppet freshness on mw1062 is OK: puppet ran at Thu Nov 21 09:20:08 UTC 2013 [09:20:11] RECOVERY - Puppet freshness on mw1022 is OK: puppet ran at Thu Nov 21 09:20:08 UTC 2013 [09:20:21] RECOVERY - Puppet freshness on sq64 is OK: puppet ran at Thu Nov 21 09:20:18 UTC 2013 [09:20:31] RECOVERY - Puppet freshness on sq55 is OK: puppet ran at Thu Nov 21 09:20:23 UTC 2013 [09:20:31] RECOVERY - Puppet freshness on amslvs2 is OK: puppet ran at Thu Nov 21 09:20:23 UTC 2013 [09:20:31] RECOVERY - Puppet freshness on sq81 is OK: puppet ran at Thu Nov 21 09:20:28 UTC 2013 [09:20:31] RECOVERY - Puppet freshness on amssq40 is OK: puppet ran at Thu Nov 21 09:20:28 UTC 2013 [09:20:31] RECOVERY - Puppet freshness on lvs1001 is OK: puppet ran at Thu Nov 21 09:20:28 UTC 2013 [09:20:32] RECOVERY - Puppet freshness on aluminium is OK: puppet ran at Thu Nov 21 09:20:28 UTC 2013 [09:20:32] RECOVERY - Puppet freshness on virt2 is OK: puppet ran at Thu Nov 21 09:20:28 UTC 2013 [09:20:41] RECOVERY - Puppet freshness on analytics1001 is OK: puppet ran at Thu Nov 21 09:20:38 UTC 2013 [09:20:41] RECOVERY - Puppet freshness on ssl1006 is OK: puppet ran at Thu Nov 21 09:20:38 UTC 2013 [09:20:51] RECOVERY - Puppet freshness on mc1011 is OK: puppet ran at Thu Nov 21 09:20:43 UTC 2013 [09:20:51] RECOVERY - Puppet freshness on ms-fe1003 is OK: puppet ran at Thu Nov 21 09:20:43 UTC 2013 [09:20:51] RECOVERY - Puppet freshness on testsearch1001 is OK: puppet ran at Thu Nov 21 09:20:43 UTC 2013 [09:20:51] RECOVERY - Puppet freshness on professor is OK: puppet ran at Thu Nov 21 09:20:48 UTC 2013 [09:21:01] RECOVERY - Puppet freshness on db31 is OK: puppet ran at Thu Nov 21 09:20:53 UTC 2013 [09:21:01] RECOVERY - Puppet freshness on mw111 is OK: puppet ran at Thu Nov 21 09:20:53 UTC 2013 [09:21:01] RECOVERY - Puppet freshness on mw1139 is OK: puppet ran at Thu Nov 21 09:20:53 UTC 2013 [09:21:01] RECOVERY - Puppet freshness on cp3006 is OK: puppet ran at Thu Nov 21 09:20:53 UTC 2013 [09:21:01] RECOVERY - Puppet freshness on mw87 is OK: puppet ran at Thu Nov 21 09:20:53 UTC 2013 [09:21:02] RECOVERY - Puppet freshness on mw1178 is OK: puppet ran at Thu Nov 21 09:20:58 UTC 2013 [09:21:02] RECOVERY - Puppet freshness on db1011 is OK: puppet ran at Thu Nov 21 09:20:58 UTC 2013 [09:21:11] RECOVERY - Puppet freshness on mw1093 is OK: puppet ran at Thu Nov 21 09:21:03 UTC 2013 [09:21:11] RECOVERY - Puppet freshness on db29 is OK: puppet ran at Thu Nov 21 09:21:03 UTC 2013 [09:21:11] RECOVERY - Puppet freshness on amssq59 is OK: puppet ran at Thu Nov 21 09:21:03 UTC 2013 [09:21:11] RECOVERY - Puppet freshness on mw1219 is OK: puppet ran at Thu Nov 21 09:21:09 UTC 2013 [09:21:11] RECOVERY - Puppet freshness on srv285 is OK: puppet ran at Thu Nov 21 09:21:09 UTC 2013 [09:21:12] RECOVERY - Puppet freshness on ms-be10 is OK: puppet ran at Thu Nov 21 09:21:09 UTC 2013 [09:21:12] RECOVERY - Puppet freshness on manutius is OK: puppet ran at Thu Nov 21 09:21:09 UTC 2013 [09:21:21] RECOVERY - Puppet freshness on search1011 is OK: puppet ran at Thu Nov 21 09:21:14 UTC 2013 [09:21:21] RECOVERY - Puppet freshness on mw1134 is OK: puppet ran at Thu Nov 21 09:21:14 UTC 2013 [09:21:21] RECOVERY - Puppet freshness on mw1072 is OK: puppet ran at Thu Nov 21 09:21:14 UTC 2013 [09:21:21] RECOVERY - Puppet freshness on search1006 is OK: puppet ran at Thu Nov 21 09:21:14 UTC 2013 [09:21:21] RECOVERY - Puppet freshness on mw1031 is OK: puppet ran at Thu Nov 21 09:21:14 UTC 2013 [09:21:21] RECOVERY - Puppet freshness on mw1080 is OK: puppet ran at Thu Nov 21 09:21:19 UTC 2013 [09:21:31] RECOVERY - Puppet freshness on sq78 is OK: puppet ran at Thu Nov 21 09:21:29 UTC 2013 [09:21:41] RECOVERY - Puppet freshness on magnesium is OK: puppet ran at Thu Nov 21 09:21:34 UTC 2013 [09:21:51] RECOVERY - Puppet freshness on sq82 is OK: puppet ran at Thu Nov 21 09:21:44 UTC 2013 [09:21:51] RECOVERY - Puppet freshness on mc1016 is OK: puppet ran at Thu Nov 21 09:21:44 UTC 2013 [09:21:51] RECOVERY - Puppet freshness on mc1013 is OK: puppet ran at Thu Nov 21 09:21:44 UTC 2013 [09:21:51] RECOVERY - Puppet freshness on wtp1010 is OK: puppet ran at Thu Nov 21 09:21:49 UTC 2013 [09:21:51] RECOVERY - Puppet freshness on db73 is OK: puppet ran at Thu Nov 21 09:21:49 UTC 2013 [09:22:01] RECOVERY - Puppet freshness on amssq43 is OK: puppet ran at Thu Nov 21 09:21:54 UTC 2013 [09:22:01] RECOVERY - Puppet freshness on mw115 is OK: puppet ran at Thu Nov 21 09:21:54 UTC 2013 [09:22:01] RECOVERY - Puppet freshness on mchenry is OK: puppet ran at Thu Nov 21 09:21:59 UTC 2013 [09:22:01] RECOVERY - Puppet freshness on amssq32 is OK: puppet ran at Thu Nov 21 09:21:59 UTC 2013 [09:22:01] RECOVERY - Puppet freshness on srv259 is OK: puppet ran at Thu Nov 21 09:21:59 UTC 2013 [09:22:02] RECOVERY - Puppet freshness on mw52 is OK: puppet ran at Thu Nov 21 09:21:59 UTC 2013 [09:22:02] RECOVERY - Puppet freshness on stat1001 is OK: puppet ran at Thu Nov 21 09:21:59 UTC 2013 [09:22:12] RECOVERY - Puppet freshness on cp1015 is OK: puppet ran at Thu Nov 21 09:22:04 UTC 2013 [09:22:12] RECOVERY - Puppet freshness on ms-be1004 is OK: puppet ran at Thu Nov 21 09:22:05 UTC 2013 [09:22:12] RECOVERY - Puppet freshness on srv282 is OK: puppet ran at Thu Nov 21 09:22:05 UTC 2013 [09:22:21] RECOVERY - Puppet freshness on db36 is OK: puppet ran at Thu Nov 21 09:22:10 UTC 2013 [09:22:21] RECOVERY - Puppet freshness on srv280 is OK: puppet ran at Thu Nov 21 09:22:11 UTC 2013 [09:22:21] RECOVERY - Puppet freshness on srv240 is OK: puppet ran at Thu Nov 21 09:22:11 UTC 2013 [09:22:21] RECOVERY - Puppet freshness on mw15 is OK: puppet ran at Thu Nov 21 09:22:11 UTC 2013 [09:22:21] RECOVERY - Puppet freshness on mw1090 is OK: puppet ran at Thu Nov 21 09:22:16 UTC 2013 [09:22:21] RECOVERY - Puppet freshness on ssl3001 is OK: puppet ran at Thu Nov 21 09:22:16 UTC 2013 [09:22:22] RECOVERY - Puppet freshness on mw1141 is OK: puppet ran at Thu Nov 21 09:22:16 UTC 2013 [09:22:22] RECOVERY - Puppet freshness on mw1145 is OK: puppet ran at Thu Nov 21 09:22:16 UTC 2013 [09:22:31] RECOVERY - Puppet freshness on mw114 is OK: puppet ran at Thu Nov 21 09:22:21 UTC 2013 [09:22:31] RECOVERY - Puppet freshness on srv270 is OK: puppet ran at Thu Nov 21 09:22:21 UTC 2013 [09:22:31] RECOVERY - Puppet freshness on mw1220 is OK: puppet ran at Thu Nov 21 09:22:21 UTC 2013 [09:22:31] RECOVERY - Puppet freshness on linne is OK: puppet ran at Thu Nov 21 09:22:26 UTC 2013 [09:22:31] RECOVERY - Puppet freshness on mw1086 is OK: puppet ran at Thu Nov 21 09:22:26 UTC 2013 [09:22:32] RECOVERY - Puppet freshness on mw65 is OK: puppet ran at Thu Nov 21 09:22:26 UTC 2013 [09:22:32] RECOVERY - Puppet freshness on mw1045 is OK: puppet ran at Thu Nov 21 09:22:26 UTC 2013 [09:22:33] RECOVERY - Puppet freshness on sq59 is OK: puppet ran at Thu Nov 21 09:22:26 UTC 2013 [09:22:33] RECOVERY - Puppet freshness on search1004 is OK: puppet ran at Thu Nov 21 09:22:26 UTC 2013 [09:22:41] RECOVERY - Puppet freshness on analytics1017 is OK: puppet ran at Thu Nov 21 09:22:31 UTC 2013 [09:22:41] RECOVERY - Puppet freshness on mw1059 is OK: puppet ran at Thu Nov 21 09:22:31 UTC 2013 [09:22:41] RECOVERY - Puppet freshness on ms-be1002 is OK: puppet ran at Thu Nov 21 09:22:31 UTC 2013 [09:22:41] RECOVERY - Puppet freshness on cp4009 is OK: puppet ran at Thu Nov 21 09:22:31 UTC 2013 [09:22:41] RECOVERY - Puppet freshness on chromium is OK: puppet ran at Thu Nov 21 09:22:31 UTC 2013 [09:22:42] RECOVERY - Puppet freshness on mw1203 is OK: puppet ran at Thu Nov 21 09:22:31 UTC 2013 [09:22:42] RECOVERY - Puppet freshness on terbium is OK: puppet ran at Thu Nov 21 09:22:31 UTC 2013 [09:22:43] RECOVERY - Puppet freshness on mw1215 is OK: puppet ran at Thu Nov 21 09:22:36 UTC 2013 [09:22:43] RECOVERY - Puppet freshness on mw1200 is OK: puppet ran at Thu Nov 21 09:22:36 UTC 2013 [09:22:44] RECOVERY - Puppet freshness on mw1112 is OK: puppet ran at Thu Nov 21 09:22:36 UTC 2013 [09:22:51] RECOVERY - Puppet freshness on lvs4002 is OK: puppet ran at Thu Nov 21 09:22:41 UTC 2013 [09:22:51] RECOVERY - Puppet freshness on analytics1018 is OK: puppet ran at Thu Nov 21 09:22:41 UTC 2013 [09:22:51] RECOVERY - Puppet freshness on sq37 is OK: puppet ran at Thu Nov 21 09:22:46 UTC 2013 [09:22:51] RECOVERY - Puppet freshness on mc1006 is OK: puppet ran at Thu Nov 21 09:22:46 UTC 2013 [09:22:51] RECOVERY - Puppet freshness on elastic1001 is OK: puppet ran at Thu Nov 21 09:22:46 UTC 2013 [09:23:01] RECOVERY - Puppet freshness on ssl1002 is OK: puppet ran at Thu Nov 21 09:22:51 UTC 2013 [09:23:01] RECOVERY - Puppet freshness on wtp1008 is OK: puppet ran at Thu Nov 21 09:22:51 UTC 2013 [09:23:01] RECOVERY - Puppet freshness on mw91 is OK: puppet ran at Thu Nov 21 09:22:51 UTC 2013 [09:23:01] RECOVERY - Puppet freshness on srv268 is OK: puppet ran at Thu Nov 21 09:22:51 UTC 2013 [09:23:01] RECOVERY - Puppet freshness on db1045 is OK: puppet ran at Thu Nov 21 09:22:56 UTC 2013 [09:23:01] RECOVERY - Puppet freshness on wtp1020 is OK: puppet ran at Thu Nov 21 09:22:56 UTC 2013 [09:23:02] RECOVERY - Puppet freshness on es1008 is OK: puppet ran at Thu Nov 21 09:22:56 UTC 2013 [09:23:02] RECOVERY - Puppet freshness on cp1039 is OK: puppet ran at Thu Nov 21 09:22:56 UTC 2013 [09:23:03] RECOVERY - Puppet freshness on db1057 is OK: puppet ran at Thu Nov 21 09:22:56 UTC 2013 [09:23:03] RECOVERY - Puppet freshness on emery is OK: puppet ran at Thu Nov 21 09:22:56 UTC 2013 [09:23:04] RECOVERY - Puppet freshness on mw92 is OK: puppet ran at Thu Nov 21 09:22:56 UTC 2013 [09:23:04] RECOVERY - Puppet freshness on db1056 is OK: puppet ran at Thu Nov 21 09:22:56 UTC 2013 [09:23:05] RECOVERY - Puppet freshness on cp1068 is OK: puppet ran at Thu Nov 21 09:22:56 UTC 2013 [09:23:11] RECOVERY - Puppet freshness on mw113 is OK: puppet ran at Thu Nov 21 09:23:01 UTC 2013 [09:23:11] RECOVERY - Puppet freshness on wtp1006 is OK: puppet ran at Thu Nov 21 09:23:01 UTC 2013 [09:23:11] RECOVERY - Puppet freshness on mw59 is OK: puppet ran at Thu Nov 21 09:23:01 UTC 2013 [09:23:11] RECOVERY - Puppet freshness on cp1054 is OK: puppet ran at Thu Nov 21 09:23:01 UTC 2013 [09:23:11] RECOVERY - Puppet freshness on srv271 is OK: puppet ran at Thu Nov 21 09:23:01 UTC 2013 [09:23:11] RECOVERY - Puppet freshness on mw1 is OK: puppet ran at Thu Nov 21 09:23:01 UTC 2013 [09:23:12] RECOVERY - Puppet freshness on ms-be1003 is OK: puppet ran at Thu Nov 21 09:23:01 UTC 2013 [09:23:12] RECOVERY - Puppet freshness on srv298 is OK: puppet ran at Thu Nov 21 09:23:01 UTC 2013 [09:23:13] RECOVERY - Puppet freshness on cp1053 is OK: puppet ran at Thu Nov 21 09:23:06 UTC 2013 [09:23:13] RECOVERY - Puppet freshness on mw1110 is OK: puppet ran at Thu Nov 21 09:23:06 UTC 2013 [09:23:21] RECOVERY - Puppet freshness on mw1158 is OK: puppet ran at Thu Nov 21 09:23:11 UTC 2013 [09:23:21] RECOVERY - Puppet freshness on mw1174 is OK: puppet ran at Thu Nov 21 09:23:11 UTC 2013 [09:23:21] RECOVERY - Puppet freshness on mw1135 is OK: puppet ran at Thu Nov 21 09:23:16 UTC 2013 [09:23:21] RECOVERY - Puppet freshness on analytics1020 is OK: puppet ran at Thu Nov 21 09:23:16 UTC 2013 [09:23:21] RECOVERY - Puppet freshness on mw1121 is OK: puppet ran at Thu Nov 21 09:23:16 UTC 2013 [09:23:21] RECOVERY - Puppet freshness on mw1026 is OK: puppet ran at Thu Nov 21 09:23:16 UTC 2013 [09:23:22] RECOVERY - Puppet freshness on mw1082 is OK: puppet ran at Thu Nov 21 09:23:16 UTC 2013 [09:23:31] RECOVERY - Puppet freshness on pdf3 is OK: puppet ran at Thu Nov 21 09:23:21 UTC 2013 [09:23:31] RECOVERY - Puppet freshness on sq77 is OK: puppet ran at Thu Nov 21 09:23:26 UTC 2013 [09:23:41] RECOVERY - Puppet freshness on ms1004 is OK: puppet ran at Thu Nov 21 09:23:31 UTC 2013 [09:23:41] RECOVERY - Puppet freshness on sq49 is OK: puppet ran at Thu Nov 21 09:23:31 UTC 2013 [09:23:41] RECOVERY - Puppet freshness on cp1001 is OK: puppet ran at Thu Nov 21 09:23:31 UTC 2013 [09:23:41] RECOVERY - Puppet freshness on es2 is OK: puppet ran at Thu Nov 21 09:23:31 UTC 2013 [09:23:41] RECOVERY - Puppet freshness on cp3021 is OK: puppet ran at Thu Nov 21 09:23:31 UTC 2013 [09:23:42] RECOVERY - Puppet freshness on cp3019 is OK: puppet ran at Thu Nov 21 09:23:36 UTC 2013 [09:23:51] RECOVERY - Puppet freshness on virt1005 is OK: puppet ran at Thu Nov 21 09:23:41 UTC 2013 [09:23:51] RECOVERY - Puppet freshness on elastic1012 is OK: puppet ran at Thu Nov 21 09:23:46 UTC 2013 [09:23:51] RECOVERY - Puppet freshness on mc1002 is OK: puppet ran at Thu Nov 21 09:23:46 UTC 2013 [09:23:51] RECOVERY - Puppet freshness on srv251 is OK: puppet ran at Thu Nov 21 09:23:46 UTC 2013 [09:24:01] RECOVERY - Puppet freshness on amssq50 is OK: puppet ran at Thu Nov 21 09:23:51 UTC 2013 [09:24:01] RECOVERY - Puppet freshness on harmon is OK: puppet ran at Thu Nov 21 09:23:51 UTC 2013 [09:24:01] RECOVERY - Puppet freshness on srv293 is OK: puppet ran at Thu Nov 21 09:23:51 UTC 2013 [09:24:01] RECOVERY - Puppet freshness on wtp1016 is OK: puppet ran at Thu Nov 21 09:23:51 UTC 2013 [09:24:01] RECOVERY - Puppet freshness on srv243 is OK: puppet ran at Thu Nov 21 09:23:51 UTC 2013 [09:24:02] RECOVERY - Puppet freshness on virt8 is OK: puppet ran at Thu Nov 21 09:23:56 UTC 2013 [09:24:02] RECOVERY - Puppet freshness on mw25 is OK: puppet ran at Thu Nov 21 09:23:56 UTC 2013 [09:24:11] RECOVERY - Puppet freshness on ms-be1010 is OK: puppet ran at Thu Nov 21 09:24:01 UTC 2013 [09:24:11] RECOVERY - Puppet freshness on mw3 is OK: puppet ran at Thu Nov 21 09:24:01 UTC 2013 [09:24:11] RECOVERY - Puppet freshness on mw7 is OK: puppet ran at Thu Nov 21 09:24:01 UTC 2013 [09:24:11] RECOVERY - Puppet freshness on mw1073 is OK: puppet ran at Thu Nov 21 09:24:06 UTC 2013 [09:24:11] RECOVERY - Puppet freshness on mw1120 is OK: puppet ran at Thu Nov 21 09:24:06 UTC 2013 [09:24:12] RECOVERY - Puppet freshness on mw1148 is OK: puppet ran at Thu Nov 21 09:24:06 UTC 2013 [09:24:12] RECOVERY - Puppet freshness on mw1009 is OK: puppet ran at Thu Nov 21 09:24:06 UTC 2013 [09:24:13] RECOVERY - Puppet freshness on mw1060 is OK: puppet ran at Thu Nov 21 09:24:06 UTC 2013 [09:24:13] RECOVERY - Puppet freshness on mw1199 is OK: puppet ran at Thu Nov 21 09:24:06 UTC 2013 [09:24:21] RECOVERY - Puppet freshness on mw1008 is OK: puppet ran at Thu Nov 21 09:24:11 UTC 2013 [09:24:21] RECOVERY - Puppet freshness on analytics1021 is OK: puppet ran at Thu Nov 21 09:24:11 UTC 2013 [09:24:21] RECOVERY - Puppet freshness on mw1108 is OK: puppet ran at Thu Nov 21 09:24:11 UTC 2013 [09:24:21] RECOVERY - Puppet freshness on lvs1 is OK: puppet ran at Thu Nov 21 09:24:11 UTC 2013 [09:24:31] RECOVERY - Puppet freshness on cp4010 is OK: puppet ran at Thu Nov 21 09:24:26 UTC 2013 [09:24:41] RECOVERY - Puppet freshness on sq66 is OK: puppet ran at Thu Nov 21 09:24:32 UTC 2013 [09:24:41] RECOVERY - Puppet freshness on sq75 is OK: puppet ran at Thu Nov 21 09:24:32 UTC 2013 [09:24:41] RECOVERY - Puppet freshness on nitrogen is OK: puppet ran at Thu Nov 21 09:24:32 UTC 2013 [09:24:41] RECOVERY - Puppet freshness on cp1009 is OK: puppet ran at Thu Nov 21 09:24:32 UTC 2013 [09:24:41] RECOVERY - Puppet freshness on cp1008 is OK: puppet ran at Thu Nov 21 09:24:32 UTC 2013 [09:24:42] RECOVERY - Puppet freshness on db1046 is OK: puppet ran at Thu Nov 21 09:24:37 UTC 2013 [09:24:42] RECOVERY - Puppet freshness on hydrogen is OK: puppet ran at Thu Nov 21 09:24:37 UTC 2013 [09:24:43] RECOVERY - Puppet freshness on cp4016 is OK: puppet ran at Thu Nov 21 09:24:37 UTC 2013 [09:24:51] RECOVERY - Puppet freshness on elastic1009 is OK: puppet ran at Thu Nov 21 09:24:47 UTC 2013 [09:24:51] RECOVERY - Puppet freshness on db64 is OK: puppet ran at Thu Nov 21 09:24:47 UTC 2013 [09:24:51] RECOVERY - Puppet freshness on zinc is OK: puppet ran at Thu Nov 21 09:24:47 UTC 2013 [09:25:01] RECOVERY - Puppet freshness on srv257 is OK: puppet ran at Thu Nov 21 09:24:52 UTC 2013 [09:25:01] RECOVERY - Puppet freshness on db1018 is OK: puppet ran at Thu Nov 21 09:24:52 UTC 2013 [09:25:01] RECOVERY - Puppet freshness on elastic1005 is OK: puppet ran at Thu Nov 21 09:24:52 UTC 2013 [09:25:01] RECOVERY - Puppet freshness on elastic1010 is OK: puppet ran at Thu Nov 21 09:24:52 UTC 2013 [09:25:01] RECOVERY - Puppet freshness on mw12 is OK: puppet ran at Thu Nov 21 09:24:52 UTC 2013 [09:25:02] RECOVERY - Puppet freshness on srv274 is OK: puppet ran at Thu Nov 21 09:24:52 UTC 2013 [09:25:02] RECOVERY - Puppet freshness on srv235 is OK: puppet ran at Thu Nov 21 09:24:52 UTC 2013 [09:25:03] RECOVERY - Puppet freshness on ms-fe1004 is OK: puppet ran at Thu Nov 21 09:24:52 UTC 2013 [09:25:03] RECOVERY - Puppet freshness on mw48 is OK: puppet ran at Thu Nov 21 09:24:52 UTC 2013 [09:25:04] RECOVERY - Puppet freshness on cp1064 is OK: puppet ran at Thu Nov 21 09:24:57 UTC 2013 [09:25:04] RECOVERY - Puppet freshness on db1015 is OK: puppet ran at Thu Nov 21 09:24:57 UTC 2013 [09:25:11] RECOVERY - Puppet freshness on mw56 is OK: puppet ran at Thu Nov 21 09:25:02 UTC 2013 [09:25:11] RECOVERY - Puppet freshness on mw1170 is OK: puppet ran at Thu Nov 21 09:25:02 UTC 2013 [09:25:11] RECOVERY - Puppet freshness on holmium is OK: puppet ran at Thu Nov 21 09:25:02 UTC 2013 [09:25:11] RECOVERY - Puppet freshness on db1049 is OK: puppet ran at Thu Nov 21 09:25:02 UTC 2013 [09:25:11] RECOVERY - Puppet freshness on mw1085 is OK: puppet ran at Thu Nov 21 09:25:07 UTC 2013 [09:25:11] RECOVERY - Puppet freshness on mw1058 is OK: puppet ran at Thu Nov 21 09:25:07 UTC 2013 [09:25:12] RECOVERY - Puppet freshness on virt0 is OK: puppet ran at Thu Nov 21 09:25:07 UTC 2013 [09:25:12] RECOVERY - Puppet freshness on mw1157 is OK: puppet ran at Thu Nov 21 09:25:07 UTC 2013 [09:25:21] RECOVERY - Puppet freshness on mw1070 is OK: puppet ran at Thu Nov 21 09:25:12 UTC 2013 [09:25:21] RECOVERY - Puppet freshness on mw1061 is OK: puppet ran at Thu Nov 21 09:25:12 UTC 2013 [09:25:21] RECOVERY - Puppet freshness on mw1019 is OK: puppet ran at Thu Nov 21 09:25:12 UTC 2013 [09:25:21] RECOVERY - Puppet freshness on mw1119 is OK: puppet ran at Thu Nov 21 09:25:17 UTC 2013 [09:25:31] RECOVERY - Puppet freshness on mw1102 is OK: puppet ran at Thu Nov 21 09:25:22 UTC 2013 [09:25:31] RECOVERY - Puppet freshness on sq56 is OK: puppet ran at Thu Nov 21 09:25:22 UTC 2013 [09:25:31] RECOVERY - Puppet freshness on amssq36 is OK: puppet ran at Thu Nov 21 09:25:27 UTC 2013 [09:25:41] RECOVERY - Puppet freshness on ersch is OK: puppet ran at Thu Nov 21 09:25:32 UTC 2013 [09:25:51] RECOVERY - Puppet freshness on mw96 is OK: puppet ran at Thu Nov 21 09:25:47 UTC 2013 [09:25:51] RECOVERY - Puppet freshness on amssq41 is OK: puppet ran at Thu Nov 21 09:25:47 UTC 2013 [09:25:51] RECOVERY - Puppet freshness on ssl1004 is OK: puppet ran at Thu Nov 21 09:25:47 UTC 2013 [09:26:01] RECOVERY - Puppet freshness on wtp1014 is OK: puppet ran at Thu Nov 21 09:25:52 UTC 2013 [09:26:01] RECOVERY - Puppet freshness on db1028 is OK: puppet ran at Thu Nov 21 09:25:52 UTC 2013 [09:26:01] RECOVERY - Puppet freshness on db38 is OK: puppet ran at Thu Nov 21 09:25:52 UTC 2013 [09:26:01] RECOVERY - Puppet freshness on analytics1006 is OK: puppet ran at Thu Nov 21 09:25:52 UTC 2013 [09:26:01] RECOVERY - Puppet freshness on analytics1024 is OK: puppet ran at Thu Nov 21 09:25:52 UTC 2013 [09:26:01] RECOVERY - Puppet freshness on srv239 is OK: puppet ran at Thu Nov 21 09:25:58 UTC 2013 [09:26:02] RECOVERY - Puppet freshness on praseodymium is OK: puppet ran at Thu Nov 21 09:25:58 UTC 2013 [09:26:02] RECOVERY - Puppet freshness on db68 is OK: puppet ran at Thu Nov 21 09:25:58 UTC 2013 [09:26:03] RECOVERY - Puppet freshness on cp1057 is OK: puppet ran at Thu Nov 21 09:25:58 UTC 2013 [09:26:03] RECOVERY - Puppet freshness on mw78 is OK: puppet ran at Thu Nov 21 09:25:58 UTC 2013 [09:26:04] RECOVERY - Puppet freshness on mw120 is OK: puppet ran at Thu Nov 21 09:25:58 UTC 2013 [09:26:04] RECOVERY - Puppet freshness on gallium is OK: puppet ran at Thu Nov 21 09:25:58 UTC 2013 [09:26:11] RECOVERY - Puppet freshness on mw1184 is OK: puppet ran at Thu Nov 21 09:26:03 UTC 2013 [09:26:11] RECOVERY - Puppet freshness on ms-be7 is OK: puppet ran at Thu Nov 21 09:26:03 UTC 2013 [09:26:11] RECOVERY - Puppet freshness on mw1177 is OK: puppet ran at Thu Nov 21 09:26:03 UTC 2013 [09:26:11] RECOVERY - Puppet freshness on virt12 is OK: puppet ran at Thu Nov 21 09:26:03 UTC 2013 [09:26:11] RECOVERY - Puppet freshness on mw1172 is OK: puppet ran at Thu Nov 21 09:26:03 UTC 2013 [09:26:12] RECOVERY - Puppet freshness on ms-be1005 is OK: puppet ran at Thu Nov 21 09:26:03 UTC 2013 [09:26:12] RECOVERY - Puppet freshness on mw1191 is OK: puppet ran at Thu Nov 21 09:26:08 UTC 2013 [09:26:13] RECOVERY - Puppet freshness on srv247 is OK: puppet ran at Thu Nov 21 09:26:08 UTC 2013 [09:26:13] RECOVERY - Puppet freshness on srv252 is OK: puppet ran at Thu Nov 21 09:26:08 UTC 2013 [09:26:21] RECOVERY - Puppet freshness on mw67 is OK: puppet ran at Thu Nov 21 09:26:13 UTC 2013 [09:26:21] RECOVERY - Puppet freshness on amssq60 is OK: puppet ran at Thu Nov 21 09:26:13 UTC 2013 [09:26:21] RECOVERY - Puppet freshness on mw1039 is OK: puppet ran at Thu Nov 21 09:26:13 UTC 2013 [09:26:21] RECOVERY - Puppet freshness on tmh1002 is OK: puppet ran at Thu Nov 21 09:26:13 UTC 2013 [09:26:21] RECOVERY - Puppet freshness on lvs1003 is OK: puppet ran at Thu Nov 21 09:26:14 UTC 2013 [09:26:31] RECOVERY - Puppet freshness on search1013 is OK: puppet ran at Thu Nov 21 09:26:24 UTC 2013 [09:26:32] RECOVERY - Puppet freshness on search1009 is OK: puppet ran at Thu Nov 21 09:26:29 UTC 2013 [09:26:32] RECOVERY - Puppet freshness on sq61 is OK: puppet ran at Thu Nov 21 09:26:29 UTC 2013 [09:26:32] RECOVERY - Puppet freshness on nfs2 is OK: puppet ran at Thu Nov 21 09:26:29 UTC 2013 [09:26:41] RECOVERY - Puppet freshness on sq57 is OK: puppet ran at Thu Nov 21 09:26:34 UTC 2013 [09:26:41] RECOVERY - Puppet freshness on cp4013 is OK: puppet ran at Thu Nov 21 09:26:34 UTC 2013 [09:26:41] RECOVERY - Puppet freshness on ms-fe4 is OK: puppet ran at Thu Nov 21 09:26:39 UTC 2013 [09:26:41] RECOVERY - Puppet freshness on cp1019 is OK: puppet ran at Thu Nov 21 09:26:39 UTC 2013 [09:26:51] RECOVERY - Puppet freshness on es7 is OK: puppet ran at Thu Nov 21 09:26:44 UTC 2013 [09:26:51] RECOVERY - Puppet freshness on pdf2 is OK: puppet ran at Thu Nov 21 09:26:49 UTC 2013 [09:26:51] RECOVERY - Puppet freshness on db37 is OK: puppet ran at Thu Nov 21 09:26:49 UTC 2013 [09:27:01] RECOVERY - Puppet freshness on wtp1005 is OK: puppet ran at Thu Nov 21 09:26:54 UTC 2013 [09:27:01] RECOVERY - Puppet freshness on hooper is OK: puppet ran at Thu Nov 21 09:26:54 UTC 2013 [09:27:01] RECOVERY - Puppet freshness on wtp1012 is OK: puppet ran at Thu Nov 21 09:26:54 UTC 2013 [09:27:01] RECOVERY - Puppet freshness on mw84 is OK: puppet ran at Thu Nov 21 09:26:54 UTC 2013 [09:27:01] RECOVERY - Puppet freshness on db1043 is OK: puppet ran at Thu Nov 21 09:26:54 UTC 2013 [09:27:01] RECOVERY - Puppet freshness on amssq58 is OK: puppet ran at Thu Nov 21 09:26:54 UTC 2013 [09:27:02] RECOVERY - Puppet freshness on db1003 is OK: puppet ran at Thu Nov 21 09:26:59 UTC 2013 [09:27:02] RECOVERY - Puppet freshness on mw71 is OK: puppet ran at Thu Nov 21 09:26:59 UTC 2013 [09:27:03] RECOVERY - Puppet freshness on silver is OK: puppet ran at Thu Nov 21 09:26:59 UTC 2013 [09:27:03] RECOVERY - Puppet freshness on db33 is OK: puppet ran at Thu Nov 21 09:26:59 UTC 2013 [09:27:04] RECOVERY - Puppet freshness on mw69 is OK: puppet ran at Thu Nov 21 09:26:59 UTC 2013 [09:27:04] RECOVERY - Puppet freshness on es1007 is OK: puppet ran at Thu Nov 21 09:26:59 UTC 2013 [09:27:05] RECOVERY - Puppet freshness on virt10 is OK: puppet ran at Thu Nov 21 09:26:59 UTC 2013 [09:27:11] RECOVERY - Puppet freshness on srv254 is OK: puppet ran at Thu Nov 21 09:27:04 UTC 2013 [09:27:11] RECOVERY - Puppet freshness on mw63 is OK: puppet ran at Thu Nov 21 09:27:04 UTC 2013 [09:27:11] RECOVERY - Puppet freshness on mw1167 is OK: puppet ran at Thu Nov 21 09:27:04 UTC 2013 [09:27:11] RECOVERY - Puppet freshness on db1041 is OK: puppet ran at Thu Nov 21 09:27:04 UTC 2013 [09:27:11] RECOVERY - Puppet freshness on mw17 is OK: puppet ran at Thu Nov 21 09:27:04 UTC 2013 [09:27:12] RECOVERY - Puppet freshness on mw1013 is OK: puppet ran at Thu Nov 21 09:27:09 UTC 2013 [09:27:12] RECOVERY - Puppet freshness on mw80 is OK: puppet ran at Thu Nov 21 09:27:09 UTC 2013 [09:27:21] RECOVERY - Puppet freshness on arsenic is OK: puppet ran at Thu Nov 21 09:27:14 UTC 2013 [09:27:21] RECOVERY - Puppet freshness on snapshot1001 is OK: puppet ran at Thu Nov 21 09:27:14 UTC 2013 [09:27:21] RECOVERY - Puppet freshness on mw1036 is OK: puppet ran at Thu Nov 21 09:27:14 UTC 2013 [09:27:21] RECOVERY - Puppet freshness on mw1076 is OK: puppet ran at Thu Nov 21 09:27:14 UTC 2013 [09:27:21] RECOVERY - Puppet freshness on mw1035 is OK: puppet ran at Thu Nov 21 09:27:19 UTC 2013 [09:27:22] RECOVERY - Puppet freshness on mw1055 is OK: puppet ran at Thu Nov 21 09:27:19 UTC 2013 [09:27:22] RECOVERY - Puppet freshness on analytics1016 is OK: puppet ran at Thu Nov 21 09:27:19 UTC 2013 [09:27:23] RECOVERY - Puppet freshness on mw1195 is OK: puppet ran at Thu Nov 21 09:27:19 UTC 2013 [09:27:31] RECOVERY - Puppet freshness on mw1149 is OK: puppet ran at Thu Nov 21 09:27:24 UTC 2013 [09:27:31] RECOVERY - Puppet freshness on sq53 is OK: puppet ran at Thu Nov 21 09:27:29 UTC 2013 [09:27:41] RECOVERY - Puppet freshness on nfs1 is OK: puppet ran at Thu Nov 21 09:27:34 UTC 2013 [09:27:41] RECOVERY - Puppet freshness on es3 is OK: puppet ran at Thu Nov 21 09:27:34 UTC 2013 [09:27:51] RECOVERY - Puppet freshness on ssl1008 is OK: puppet ran at Thu Nov 21 09:27:44 UTC 2013 [09:27:51] RECOVERY - Puppet freshness on ytterbium is OK: puppet ran at Thu Nov 21 09:27:44 UTC 2013 [09:27:51] RECOVERY - Puppet freshness on mc1014 is OK: puppet ran at Thu Nov 21 09:27:44 UTC 2013 [09:27:51] RECOVERY - Puppet freshness on sq69 is OK: puppet ran at Thu Nov 21 09:27:44 UTC 2013 [09:27:51] RECOVERY - Puppet freshness on osm-db1001 is OK: puppet ran at Thu Nov 21 09:27:49 UTC 2013 [09:27:52] RECOVERY - Puppet freshness on mc1010 is OK: puppet ran at Thu Nov 21 09:27:49 UTC 2013 [09:27:52] RECOVERY - Puppet freshness on cp3022 is OK: puppet ran at Thu Nov 21 09:27:49 UTC 2013 [09:27:53] RECOVERY - Puppet freshness on db1039 is OK: puppet ran at Thu Nov 21 09:27:49 UTC 2013 [09:28:01] RECOVERY - Puppet freshness on mw88 is OK: puppet ran at Thu Nov 21 09:27:54 UTC 2013 [09:28:01] RECOVERY - Puppet freshness on db1007 is OK: puppet ran at Thu Nov 21 09:27:54 UTC 2013 [09:28:01] RECOVERY - Puppet freshness on es1006 is OK: puppet ran at Thu Nov 21 09:27:54 UTC 2013 [09:28:01] RECOVERY - Puppet freshness on ssl3003 is OK: puppet ran at Thu Nov 21 09:27:54 UTC 2013 [09:28:01] RECOVERY - Puppet freshness on analytics1026 is OK: puppet ran at Thu Nov 21 09:27:59 UTC 2013 [09:28:02] RECOVERY - Puppet freshness on mw18 is OK: puppet ran at Thu Nov 21 09:27:59 UTC 2013 [09:28:02] RECOVERY - Puppet freshness on mw86 is OK: puppet ran at Thu Nov 21 09:27:59 UTC 2013 [09:28:03] RECOVERY - Puppet freshness on mw32 is OK: puppet ran at Thu Nov 21 09:27:59 UTC 2013 [09:28:03] RECOVERY - Puppet freshness on mw1014 is OK: puppet ran at Thu Nov 21 09:27:59 UTC 2013 [09:28:11] RECOVERY - Puppet freshness on mw1216 is OK: puppet ran at Thu Nov 21 09:28:04 UTC 2013 [09:28:11] RECOVERY - Puppet freshness on mw1098 is OK: puppet ran at Thu Nov 21 09:28:04 UTC 2013 [09:28:11] RECOVERY - Puppet freshness on mw1202 is OK: puppet ran at Thu Nov 21 09:28:04 UTC 2013 [09:28:11] RECOVERY - Puppet freshness on mw1151 is OK: puppet ran at Thu Nov 21 09:28:09 UTC 2013 [09:28:11] RECOVERY - Puppet freshness on mw1192 is OK: puppet ran at Thu Nov 21 09:28:09 UTC 2013 [09:28:12] RECOVERY - Puppet freshness on mw1079 is OK: puppet ran at Thu Nov 21 09:28:09 UTC 2013 [09:28:21] RECOVERY - Puppet freshness on mw1109 is OK: puppet ran at Thu Nov 21 09:28:14 UTC 2013 [09:28:21] RECOVERY - Puppet freshness on search1002 is OK: puppet ran at Thu Nov 21 09:28:16 UTC 2013 [09:28:21] RECOVERY - Puppet freshness on mw1168 is OK: puppet ran at Thu Nov 21 09:28:16 UTC 2013 [09:28:31] RECOVERY - Puppet freshness on analytics1013 is OK: puppet ran at Thu Nov 21 09:28:21 UTC 2013 [09:28:31] RECOVERY - Puppet freshness on mw1051 is OK: puppet ran at Thu Nov 21 09:28:21 UTC 2013 [09:28:31] RECOVERY - Puppet freshness on mw1133 is OK: puppet ran at Thu Nov 21 09:28:21 UTC 2013 [09:28:31] RECOVERY - Puppet freshness on db49 is OK: puppet ran at Thu Nov 21 09:28:26 UTC 2013 [09:28:41] RECOVERY - Puppet freshness on cp4018 is OK: puppet ran at Thu Nov 21 09:28:31 UTC 2013 [09:28:41] RECOVERY - Puppet freshness on virt1007 is OK: puppet ran at Thu Nov 21 09:28:32 UTC 2013 [09:28:41] RECOVERY - Puppet freshness on ssl3002 is OK: puppet ran at Thu Nov 21 09:28:37 UTC 2013 [09:28:42] RECOVERY - Puppet freshness on formey is OK: puppet ran at Thu Nov 21 09:28:37 UTC 2013 [09:28:42] RECOVERY - Puppet freshness on cp1011 is OK: puppet ran at Thu Nov 21 09:28:37 UTC 2013 [09:28:42] RECOVERY - Puppet freshness on cp1004 is OK: puppet ran at Thu Nov 21 09:28:37 UTC 2013 [09:28:51] RECOVERY - Puppet freshness on iron is OK: puppet ran at Thu Nov 21 09:28:42 UTC 2013 [09:28:51] RECOVERY - Puppet freshness on mw97 is OK: puppet ran at Thu Nov 21 09:28:42 UTC 2013 [09:28:51] RECOVERY - Puppet freshness on mw4 is OK: puppet ran at Thu Nov 21 09:28:42 UTC 2013 [09:28:51] RECOVERY - Puppet freshness on rdb1001 is OK: puppet ran at Thu Nov 21 09:28:47 UTC 2013 [09:28:51] RECOVERY - Puppet freshness on virt7 is OK: puppet ran at Thu Nov 21 09:28:47 UTC 2013 [09:28:51] RECOVERY - Puppet freshness on oxygen is OK: puppet ran at Thu Nov 21 09:28:47 UTC 2013 [09:28:52] RECOVERY - Puppet freshness on srv263 is OK: puppet ran at Thu Nov 21 09:28:47 UTC 2013 [09:29:01] RECOVERY - Puppet freshness on mw123 is OK: puppet ran at Thu Nov 21 09:28:57 UTC 2013 [09:29:01] RECOVERY - Puppet freshness on ssl1009 is OK: puppet ran at Thu Nov 21 09:28:57 UTC 2013 [09:29:01] RECOVERY - Puppet freshness on db1030 is OK: puppet ran at Thu Nov 21 09:28:57 UTC 2013 [09:29:01] RECOVERY - Puppet freshness on mw85 is OK: puppet ran at Thu Nov 21 09:28:57 UTC 2013 [09:29:11] RECOVERY - Puppet freshness on mw1034 is OK: puppet ran at Thu Nov 21 09:29:02 UTC 2013 [09:29:11] RECOVERY - Puppet freshness on db72 is OK: puppet ran at Thu Nov 21 09:29:02 UTC 2013 [09:29:11] RECOVERY - Puppet freshness on mw72 is OK: puppet ran at Thu Nov 21 09:29:02 UTC 2013 [09:29:11] RECOVERY - Puppet freshness on cp1048 is OK: puppet ran at Thu Nov 21 09:29:02 UTC 2013 [09:29:11] RECOVERY - Puppet freshness on sq70 is OK: puppet ran at Thu Nov 21 09:29:02 UTC 2013 [09:29:12] RECOVERY - Puppet freshness on mc1004 is OK: puppet ran at Thu Nov 21 09:29:02 UTC 2013 [09:29:12] RECOVERY - Puppet freshness on ms-be1008 is OK: puppet ran at Thu Nov 21 09:29:07 UTC 2013 [09:29:21] RECOVERY - Puppet freshness on db1016 is OK: puppet ran at Thu Nov 21 09:29:12 UTC 2013 [09:29:21] RECOVERY - Puppet freshness on mw1050 is OK: puppet ran at Thu Nov 21 09:29:17 UTC 2013 [09:29:21] RECOVERY - Puppet freshness on mw1183 is OK: puppet ran at Thu Nov 21 09:29:17 UTC 2013 [09:29:21] RECOVERY - Puppet freshness on mw1156 is OK: puppet ran at Thu Nov 21 09:29:17 UTC 2013 [09:29:21] RECOVERY - Puppet freshness on hooft is OK: puppet ran at Thu Nov 21 09:29:17 UTC 2013 [09:29:31] RECOVERY - Puppet freshness on lvs2 is OK: puppet ran at Thu Nov 21 09:29:22 UTC 2013 [09:29:31] RECOVERY - Puppet freshness on ms1002 is OK: puppet ran at Thu Nov 21 09:29:27 UTC 2013 [09:29:37] I guess icinga-wm is back [09:29:41] RECOVERY - Puppet freshness on amssq46 is OK: puppet ran at Thu Nov 21 09:29:37 UTC 2013 [09:29:51] RECOVERY - Puppet freshness on ssl3 is OK: puppet ran at Thu Nov 21 09:29:47 UTC 2013 [09:29:51] RECOVERY - Puppet freshness on virt5 is OK: puppet ran at Thu Nov 21 09:29:47 UTC 2013 [09:29:51] RECOVERY - Puppet freshness on elastic1011 is OK: puppet ran at Thu Nov 21 09:29:47 UTC 2013 [09:29:51] RECOVERY - Puppet freshness on wtp1023 is OK: puppet ran at Thu Nov 21 09:29:47 UTC 2013 [09:30:01] RECOVERY - Puppet freshness on srv241 is OK: puppet ran at Thu Nov 21 09:29:52 UTC 2013 [09:30:01] RECOVERY - Puppet freshness on wtp1022 is OK: puppet ran at Thu Nov 21 09:29:52 UTC 2013 [09:30:01] RECOVERY - Puppet freshness on amssq62 is OK: puppet ran at Thu Nov 21 09:29:52 UTC 2013 [09:30:01] RECOVERY - Puppet freshness on cp1055 is OK: puppet ran at Thu Nov 21 09:29:57 UTC 2013 [09:30:01] RECOVERY - Puppet freshness on virt6 is OK: puppet ran at Thu Nov 21 09:29:57 UTC 2013 [09:30:01] RECOVERY - Puppet freshness on db1059 is OK: puppet ran at Thu Nov 21 09:29:57 UTC 2013 [09:30:02] RECOVERY - Puppet freshness on es8 is OK: puppet ran at Thu Nov 21 09:29:57 UTC 2013 [09:30:02] RECOVERY - Puppet freshness on mw102 is OK: puppet ran at Thu Nov 21 09:29:57 UTC 2013 [09:30:11] RECOVERY - Puppet freshness on srv288 is OK: puppet ran at Thu Nov 21 09:30:02 UTC 2013 [09:30:11] RECOVERY - Puppet freshness on mw44 is OK: puppet ran at Thu Nov 21 09:30:02 UTC 2013 [09:30:11] RECOVERY - Puppet freshness on db1053 is OK: puppet ran at Thu Nov 21 09:30:02 UTC 2013 [09:30:11] RECOVERY - Puppet freshness on mw6 is OK: puppet ran at Thu Nov 21 09:30:02 UTC 2013 [09:30:11] RECOVERY - Puppet freshness on srv279 is OK: puppet ran at Thu Nov 21 09:30:02 UTC 2013 [09:30:12] RECOVERY - Puppet freshness on mw1097 is OK: puppet ran at Thu Nov 21 09:30:02 UTC 2013 [09:30:12] RECOVERY - Puppet freshness on mw1002 is OK: puppet ran at Thu Nov 21 09:30:02 UTC 2013 [09:30:13] RECOVERY - Puppet freshness on mw27 is OK: puppet ran at Thu Nov 21 09:30:02 UTC 2013 [09:30:21] RECOVERY - Puppet freshness on search1016 is OK: puppet ran at Thu Nov 21 09:30:12 UTC 2013 [09:30:21] RECOVERY - Puppet freshness on mw1163 is OK: puppet ran at Thu Nov 21 09:30:12 UTC 2013 [09:30:21] RECOVERY - Puppet freshness on mw1023 is OK: puppet ran at Thu Nov 21 09:30:12 UTC 2013 [09:30:21] RECOVERY - Puppet freshness on mw1029 is OK: puppet ran at Thu Nov 21 09:30:12 UTC 2013 [09:30:21] RECOVERY - Puppet freshness on mw1077 is OK: puppet ran at Thu Nov 21 09:30:12 UTC 2013 [09:30:22] RECOVERY - Puppet freshness on mw1212 is OK: puppet ran at Thu Nov 21 09:30:17 UTC 2013 [09:30:22] RECOVERY - Puppet freshness on mw1096 is OK: puppet ran at Thu Nov 21 09:30:17 UTC 2013 [09:30:31] RECOVERY - Puppet freshness on search1017 is OK: puppet ran at Thu Nov 21 09:30:22 UTC 2013 [09:30:31] RECOVERY - Puppet freshness on snapshot1004 is OK: puppet ran at Thu Nov 21 09:30:22 UTC 2013 [09:30:31] RECOVERY - Puppet freshness on amssq38 is OK: puppet ran at Thu Nov 21 09:30:28 UTC 2013 [09:30:31] RECOVERY - Puppet freshness on sq80 is OK: puppet ran at Thu Nov 21 09:30:28 UTC 2013 [09:30:41] RECOVERY - Puppet freshness on pc3 is OK: puppet ran at Thu Nov 21 09:30:33 UTC 2013 [09:30:41] RECOVERY - Puppet freshness on db47 is OK: puppet ran at Thu Nov 21 09:30:38 UTC 2013 [09:30:51] RECOVERY - Puppet freshness on rdb1004 is OK: puppet ran at Thu Nov 21 09:30:43 UTC 2013 [09:30:51] RECOVERY - Puppet freshness on db62 is OK: puppet ran at Thu Nov 21 09:30:48 UTC 2013 [09:30:51] RECOVERY - Puppet freshness on ms1001 is OK: puppet ran at Thu Nov 21 09:30:48 UTC 2013 [09:30:51] RECOVERY - Puppet freshness on pc1001 is OK: puppet ran at Thu Nov 21 09:30:48 UTC 2013 [09:30:51] RECOVERY - Puppet freshness on wtp1017 is OK: puppet ran at Thu Nov 21 09:30:48 UTC 2013 [09:31:01] RECOVERY - Puppet freshness on cp1059 is OK: puppet ran at Thu Nov 21 09:30:53 UTC 2013 [09:31:01] RECOVERY - Puppet freshness on db1047 is OK: puppet ran at Thu Nov 21 09:30:53 UTC 2013 [09:31:01] RECOVERY - Puppet freshness on wtp1019 is OK: puppet ran at Thu Nov 21 09:30:53 UTC 2013 [09:31:01] RECOVERY - Puppet freshness on db1006 is OK: puppet ran at Thu Nov 21 09:30:58 UTC 2013 [09:31:01] RECOVERY - Puppet freshness on db1038 is OK: puppet ran at Thu Nov 21 09:30:58 UTC 2013 [09:31:02] RECOVERY - Puppet freshness on db35 is OK: puppet ran at Thu Nov 21 09:30:58 UTC 2013 [09:31:02] RECOVERY - Puppet freshness on ms-be1011 is OK: puppet ran at Thu Nov 21 09:30:58 UTC 2013 [09:31:03] RECOVERY - Puppet freshness on locke is OK: puppet ran at Thu Nov 21 09:30:58 UTC 2013 [09:31:11] RECOVERY - Puppet freshness on mw89 is OK: puppet ran at Thu Nov 21 09:31:03 UTC 2013 [09:31:11] RECOVERY - Puppet freshness on mw21 is OK: puppet ran at Thu Nov 21 09:31:03 UTC 2013 [09:31:11] RECOVERY - Puppet freshness on cp1052 is OK: puppet ran at Thu Nov 21 09:31:03 UTC 2013 [09:31:11] RECOVERY - Puppet freshness on search1020 is OK: puppet ran at Thu Nov 21 09:31:03 UTC 2013 [09:31:11] RECOVERY - Puppet freshness on mw1067 is OK: puppet ran at Thu Nov 21 09:31:08 UTC 2013 [09:31:12] RECOVERY - Puppet freshness on es1010 is OK: puppet ran at Thu Nov 21 09:31:08 UTC 2013 [09:31:12] RECOVERY - Puppet freshness on ms-be1001 is OK: puppet ran at Thu Nov 21 09:31:08 UTC 2013 [09:31:21] RECOVERY - Puppet freshness on ms-be6 is OK: puppet ran at Thu Nov 21 09:31:13 UTC 2013 [09:31:21] RECOVERY - Puppet freshness on mw1005 is OK: puppet ran at Thu Nov 21 09:31:13 UTC 2013 [09:31:21] RECOVERY - Puppet freshness on analytics1027 is OK: puppet ran at Thu Nov 21 09:31:13 UTC 2013 [09:31:21] RECOVERY - Puppet freshness on mw1209 is OK: puppet ran at Thu Nov 21 09:31:13 UTC 2013 [09:31:21] RECOVERY - Puppet freshness on mw1016 is OK: puppet ran at Thu Nov 21 09:31:13 UTC 2013 [09:31:22] RECOVERY - Puppet freshness on mw1028 is OK: puppet ran at Thu Nov 21 09:31:18 UTC 2013 [09:31:22] RECOVERY - Puppet freshness on mw1152 is OK: puppet ran at Thu Nov 21 09:31:18 UTC 2013 [09:31:31] RECOVERY - Puppet freshness on sq60 is OK: puppet ran at Thu Nov 21 09:31:28 UTC 2013 [09:31:41] RECOVERY - Puppet freshness on amssq45 is OK: puppet ran at Thu Nov 21 09:31:33 UTC 2013 [09:31:41] RECOVERY - Puppet freshness on es1001 is OK: puppet ran at Thu Nov 21 09:31:33 UTC 2013 [09:31:41] RECOVERY - Puppet freshness on zhen is OK: puppet ran at Thu Nov 21 09:31:33 UTC 2013 [09:31:41] RECOVERY - Puppet freshness on carbon is OK: puppet ran at Thu Nov 21 09:31:33 UTC 2013 [09:31:41] RECOVERY - Puppet freshness on cp1014 is OK: puppet ran at Thu Nov 21 09:31:33 UTC 2013 [09:31:42] RECOVERY - Puppet freshness on ms-fe1 is OK: puppet ran at Thu Nov 21 09:31:38 UTC 2013 [09:31:51] RECOVERY - Puppet freshness on amssq33 is OK: puppet ran at Thu Nov 21 09:31:43 UTC 2013 [09:31:51] RECOVERY - Puppet freshness on db71 is OK: puppet ran at Thu Nov 21 09:31:43 UTC 2013 [09:31:51] RECOVERY - Puppet freshness on cp4020 is OK: puppet ran at Thu Nov 21 09:31:48 UTC 2013 [09:31:51] RECOVERY - Puppet freshness on bast4001 is OK: puppet ran at Thu Nov 21 09:31:48 UTC 2013 [09:31:51] RECOVERY - Puppet freshness on db1017 is OK: puppet ran at Thu Nov 21 09:31:48 UTC 2013 [09:31:52] RECOVERY - Puppet freshness on amslvs3 is OK: puppet ran at Thu Nov 21 09:31:48 UTC 2013 [09:32:01] RECOVERY - Puppet freshness on ssl2 is OK: puppet ran at Thu Nov 21 09:31:53 UTC 2013 [09:32:01] RECOVERY - Puppet freshness on amssq42 is OK: puppet ran at Thu Nov 21 09:31:53 UTC 2013 [09:32:01] RECOVERY - Puppet freshness on cp1070 is OK: puppet ran at Thu Nov 21 09:31:53 UTC 2013 [09:32:01] RECOVERY - Puppet freshness on elastic1003 is OK: puppet ran at Thu Nov 21 09:31:53 UTC 2013 [09:32:01] RECOVERY - Puppet freshness on wtp1009 is OK: puppet ran at Thu Nov 21 09:31:53 UTC 2013 [09:32:01] RECOVERY - Puppet freshness on cp3020 is OK: puppet ran at Thu Nov 21 09:31:53 UTC 2013 [09:32:02] RECOVERY - Puppet freshness on mw11 is OK: puppet ran at Thu Nov 21 09:32:00 UTC 2013 [09:32:02] RECOVERY - Puppet freshness on cp1040 is OK: puppet ran at Thu Nov 21 09:32:00 UTC 2013 [09:32:03] RECOVERY - Puppet freshness on strontium is OK: puppet ran at Thu Nov 21 09:32:00 UTC 2013 [09:32:03] RECOVERY - Puppet freshness on cerium is OK: puppet ran at Thu Nov 21 09:32:00 UTC 2013 [09:32:04] RECOVERY - Puppet freshness on snapshot2 is OK: puppet ran at Thu Nov 21 09:32:00 UTC 2013 [09:32:04] RECOVERY - Puppet freshness on cp1044 is OK: puppet ran at Thu Nov 21 09:32:00 UTC 2013 [09:32:05] RECOVERY - Puppet freshness on mw73 is OK: puppet ran at Thu Nov 21 09:32:00 UTC 2013 [09:32:11] RECOVERY - Puppet freshness on mw31 is OK: puppet ran at Thu Nov 21 09:32:05 UTC 2013 [09:32:11] RECOVERY - Puppet freshness on mw24 is OK: puppet ran at Thu Nov 21 09:32:05 UTC 2013 [09:32:11] RECOVERY - Puppet freshness on lvs5 is OK: puppet ran at Thu Nov 21 09:32:05 UTC 2013 [09:32:11] RECOVERY - Puppet freshness on cp3008 is OK: puppet ran at Thu Nov 21 09:32:05 UTC 2013 [09:32:11] RECOVERY - Puppet freshness on srv277 is OK: puppet ran at Thu Nov 21 09:32:05 UTC 2013 [09:32:12] RECOVERY - Puppet freshness on srv265 is OK: puppet ran at Thu Nov 21 09:32:05 UTC 2013 [09:32:12] RECOVERY - Puppet freshness on mw22 is OK: puppet ran at Thu Nov 21 09:32:10 UTC 2013 [09:32:13] RECOVERY - Puppet freshness on srv236 is OK: puppet ran at Thu Nov 21 09:32:10 UTC 2013 [09:32:13] RECOVERY - Puppet freshness on amssq49 is OK: puppet ran at Thu Nov 21 09:32:10 UTC 2013 [09:32:14] RECOVERY - Puppet freshness on neon is OK: puppet ran at Thu Nov 21 09:32:10 UTC 2013 [09:32:14] RECOVERY - Puppet freshness on mw1193 is OK: puppet ran at Thu Nov 21 09:32:10 UTC 2013 [09:32:21] RECOVERY - Puppet freshness on srv248 is OK: puppet ran at Thu Nov 21 09:32:15 UTC 2013 [09:32:21] RECOVERY - Puppet freshness on mw1006 is OK: puppet ran at Thu Nov 21 09:32:15 UTC 2013 [09:32:21] RECOVERY - Puppet freshness on srv256 is OK: puppet ran at Thu Nov 21 09:32:15 UTC 2013 [09:32:21] RECOVERY - Puppet freshness on amssq54 is OK: puppet ran at Thu Nov 21 09:32:15 UTC 2013 [09:32:21] RECOVERY - Puppet freshness on mw19 is OK: puppet ran at Thu Nov 21 09:32:15 UTC 2013 [09:32:22] RECOVERY - Puppet freshness on mw77 is OK: puppet ran at Thu Nov 21 09:32:15 UTC 2013 [09:32:22] RECOVERY - Puppet freshness on yvon is OK: puppet ran at Thu Nov 21 09:32:20 UTC 2013 [09:32:23] RECOVERY - Puppet freshness on mw1064 is OK: puppet ran at Thu Nov 21 09:32:20 UTC 2013 [09:32:23] RECOVERY - Puppet freshness on cp1067 is OK: puppet ran at Thu Nov 21 09:32:20 UTC 2013 [09:32:31] RECOVERY - Puppet freshness on mw1012 is OK: puppet ran at Thu Nov 21 09:32:25 UTC 2013 [09:32:31] RECOVERY - Puppet freshness on tridge is OK: puppet ran at Thu Nov 21 09:32:25 UTC 2013 [09:32:31] RECOVERY - Puppet freshness on amssq39 is OK: puppet ran at Thu Nov 21 09:32:30 UTC 2013 [09:32:32] RECOVERY - Puppet freshness on mw1071 is OK: puppet ran at Thu Nov 21 09:32:30 UTC 2013 [09:32:32] RECOVERY - Puppet freshness on eeden is OK: puppet ran at Thu Nov 21 09:32:30 UTC 2013 [09:32:41] RECOVERY - Puppet freshness on analytics1004 is OK: puppet ran at Thu Nov 21 09:32:40 UTC 2013 [09:32:42] RECOVERY - Puppet freshness on cp3005 is OK: puppet ran at Thu Nov 21 09:32:40 UTC 2013 [09:32:42] RECOVERY - Puppet freshness on virt9 is OK: puppet ran at Thu Nov 21 09:32:40 UTC 2013 [09:32:42] RECOVERY - Puppet freshness on cp3004 is OK: puppet ran at Thu Nov 21 09:32:40 UTC 2013 [09:32:51] RECOVERY - Puppet freshness on cp4012 is OK: puppet ran at Thu Nov 21 09:32:50 UTC 2013 [09:32:51] RECOVERY - Puppet freshness on elastic1007 is OK: puppet ran at Thu Nov 21 09:32:50 UTC 2013 [09:32:51] RECOVERY - Puppet freshness on amssq61 is OK: puppet ran at Thu Nov 21 09:32:50 UTC 2013 [09:32:51] RECOVERY - Puppet freshness on cp4006 is OK: puppet ran at Thu Nov 21 09:32:50 UTC 2013 [09:32:52] RECOVERY - Puppet freshness on mw105 is OK: puppet ran at Thu Nov 21 09:32:50 UTC 2013 [09:32:52] RECOVERY - Puppet freshness on mw108 is OK: puppet ran at Thu Nov 21 09:32:50 UTC 2013 [09:32:52] RECOVERY - Puppet freshness on mw103 is OK: puppet ran at Thu Nov 21 09:32:50 UTC 2013 [09:33:01] RECOVERY - Puppet freshness on mw54 is OK: puppet ran at Thu Nov 21 09:32:55 UTC 2013 [09:33:01] RECOVERY - Puppet freshness on cp1045 is OK: puppet ran at Thu Nov 21 09:32:55 UTC 2013 [09:33:01] RECOVERY - Puppet freshness on analytics1025 is OK: puppet ran at Thu Nov 21 09:32:55 UTC 2013 [09:33:01] RECOVERY - Puppet freshness on db1037 is OK: puppet ran at Thu Nov 21 09:32:55 UTC 2013 [09:33:01] RECOVERY - Puppet freshness on srv295 is OK: puppet ran at Thu Nov 21 09:32:55 UTC 2013 [09:33:02] RECOVERY - Puppet freshness on mw29 is OK: puppet ran at Thu Nov 21 09:32:55 UTC 2013 [09:33:02] RECOVERY - Puppet freshness on analytics1005 is OK: puppet ran at Thu Nov 21 09:32:55 UTC 2013 [09:33:03] RECOVERY - Puppet freshness on cp1047 is OK: puppet ran at Thu Nov 21 09:32:55 UTC 2013 [09:33:03] RECOVERY - Puppet freshness on xenon is OK: puppet ran at Thu Nov 21 09:32:55 UTC 2013 [09:33:04] RECOVERY - Puppet freshness on mw94 is OK: puppet ran at Thu Nov 21 09:32:55 UTC 2013 [09:33:04] RECOVERY - Puppet freshness on mw101 is OK: puppet ran at Thu Nov 21 09:32:55 UTC 2013 [09:33:05] RECOVERY - Puppet freshness on mexia is OK: puppet ran at Thu Nov 21 09:32:55 UTC 2013 [09:33:05] RECOVERY - Puppet freshness on db1035 is OK: puppet ran at Thu Nov 21 09:32:55 UTC 2013 [09:33:11] RECOVERY - Puppet freshness on lvs4004 is OK: puppet ran at Thu Nov 21 09:33:05 UTC 2013 [09:33:11] RECOVERY - Puppet freshness on mw1187 is OK: puppet ran at Thu Nov 21 09:33:05 UTC 2013 [09:33:11] RECOVERY - Puppet freshness on mw1207 is OK: puppet ran at Thu Nov 21 09:33:10 UTC 2013 [09:33:11] RECOVERY - Puppet freshness on mw1160 is OK: puppet ran at Thu Nov 21 09:33:10 UTC 2013 [09:33:12] RECOVERY - Puppet freshness on mw1113 is OK: puppet ran at Thu Nov 21 09:33:10 UTC 2013 [09:33:12] RECOVERY - Puppet freshness on snapshot1003 is OK: puppet ran at Thu Nov 21 09:33:10 UTC 2013 [09:33:12] RECOVERY - Puppet freshness on mw1037 is OK: puppet ran at Thu Nov 21 09:33:10 UTC 2013 [09:33:31] RECOVERY - Puppet freshness on maerlant is OK: puppet ran at Thu Nov 21 09:33:20 UTC 2013 [09:33:41] RECOVERY - Puppet freshness on pc2 is OK: puppet ran at Thu Nov 21 09:33:31 UTC 2013 [09:33:41] RECOVERY - Puppet freshness on cp3003 is OK: puppet ran at Thu Nov 21 09:33:31 UTC 2013 [09:33:41] RECOVERY - Puppet freshness on sq67 is OK: puppet ran at Thu Nov 21 09:33:36 UTC 2013 [09:33:41] RECOVERY - Puppet freshness on stafford is OK: puppet ran at Thu Nov 21 09:33:36 UTC 2013 [09:33:51] RECOVERY - Puppet freshness on mc1015 is OK: puppet ran at Thu Nov 21 09:33:41 UTC 2013 [09:33:51] RECOVERY - Puppet freshness on mw16 is OK: puppet ran at Thu Nov 21 09:33:46 UTC 2013 [09:33:51] RECOVERY - Puppet freshness on mc1003 is OK: puppet ran at Thu Nov 21 09:33:46 UTC 2013 [09:33:51] RECOVERY - Puppet freshness on srv287 is OK: puppet ran at Thu Nov 21 09:33:46 UTC 2013 [09:34:01] RECOVERY - Puppet freshness on srv300 is OK: puppet ran at Thu Nov 21 09:33:51 UTC 2013 [09:34:01] RECOVERY - Puppet freshness on mw51 is OK: puppet ran at Thu Nov 21 09:33:56 UTC 2013 [09:34:01] RECOVERY - Puppet freshness on mw1176 is OK: puppet ran at Thu Nov 21 09:33:56 UTC 2013 [09:34:11] RECOVERY - Puppet freshness on mw1117 is OK: puppet ran at Thu Nov 21 09:34:01 UTC 2013 [09:34:11] RECOVERY - Puppet freshness on mw1164 is OK: puppet ran at Thu Nov 21 09:34:01 UTC 2013 [09:34:11] RECOVERY - Puppet freshness on labstore4 is OK: puppet ran at Thu Nov 21 09:34:01 UTC 2013 [09:34:11] RECOVERY - Puppet freshness on search1010 is OK: puppet ran at Thu Nov 21 09:34:01 UTC 2013 [09:34:11] RECOVERY - Puppet freshness on mw1018 is OK: puppet ran at Thu Nov 21 09:34:01 UTC 2013 [09:34:11] RECOVERY - Puppet freshness on mw1217 is OK: puppet ran at Thu Nov 21 09:34:01 UTC 2013 [09:34:12] RECOVERY - Puppet freshness on mw1099 is OK: puppet ran at Thu Nov 21 09:34:01 UTC 2013 [09:34:12] RECOVERY - Puppet freshness on mw1100 is OK: puppet ran at Thu Nov 21 09:34:06 UTC 2013 [09:34:13] RECOVERY - Puppet freshness on mw1103 is OK: puppet ran at Thu Nov 21 09:34:06 UTC 2013 [09:34:21] RECOVERY - Puppet freshness on fenari is OK: puppet ran at Thu Nov 21 09:34:11 UTC 2013 [09:34:21] RECOVERY - Puppet freshness on mw1088 is OK: puppet ran at Thu Nov 21 09:34:11 UTC 2013 [09:34:22] seems like icinga is back [09:34:31] RECOVERY - Puppet freshness on sq86 is OK: puppet ran at Thu Nov 21 09:34:21 UTC 2013 [09:34:31] RECOVERY - Puppet freshness on cp4011 is OK: puppet ran at Thu Nov 21 09:34:26 UTC 2013 [09:34:41] RECOVERY - Puppet freshness on labsdb1003 is OK: puppet ran at Thu Nov 21 09:34:36 UTC 2013 [09:34:51] RECOVERY - Puppet freshness on cp1007 is OK: puppet ran at Thu Nov 21 09:34:41 UTC 2013 [09:34:51] RECOVERY - Puppet freshness on db1054 is OK: puppet ran at Thu Nov 21 09:34:41 UTC 2013 [09:34:51] RECOVERY - Puppet freshness on sq83 is OK: puppet ran at Thu Nov 21 09:34:46 UTC 2013 [09:34:52] RECOVERY - Puppet freshness on es1 is OK: puppet ran at Thu Nov 21 09:34:46 UTC 2013 [09:34:52] RECOVERY - Puppet freshness on labsdb1002 is OK: puppet ran at Thu Nov 21 09:34:46 UTC 2013 [09:34:52] RECOVERY - Puppet freshness on sq71 is OK: puppet ran at Thu Nov 21 09:34:46 UTC 2013 [09:35:01] RECOVERY - Puppet freshness on db40 is OK: puppet ran at Thu Nov 21 09:34:51 UTC 2013 [09:35:01] RECOVERY - Puppet freshness on srv297 is OK: puppet ran at Thu Nov 21 09:34:51 UTC 2013 [09:35:01] RECOVERY - Puppet freshness on cp3007 is OK: puppet ran at Thu Nov 21 09:34:56 UTC 2013 [09:35:01] RECOVERY - Puppet freshness on mw82 is OK: puppet ran at Thu Nov 21 09:34:56 UTC 2013 [09:35:01] RECOVERY - Puppet freshness on mw66 is OK: puppet ran at Thu Nov 21 09:34:56 UTC 2013 [09:35:11] RECOVERY - Puppet freshness on srv238 is OK: puppet ran at Thu Nov 21 09:35:01 UTC 2013 [09:35:11] RECOVERY - Puppet freshness on mw20 is OK: puppet ran at Thu Nov 21 09:35:06 UTC 2013 [09:35:11] RECOVERY - Puppet freshness on mw1065 is OK: puppet ran at Thu Nov 21 09:35:06 UTC 2013 [09:35:11] RECOVERY - Puppet freshness on mw1123 is OK: puppet ran at Thu Nov 21 09:35:06 UTC 2013 [09:35:11] RECOVERY - Puppet freshness on mw1017 is OK: puppet ran at Thu Nov 21 09:35:06 UTC 2013 [09:35:21] RECOVERY - Puppet freshness on cp1056 is OK: puppet ran at Thu Nov 21 09:35:11 UTC 2013 [09:35:21] RECOVERY - Puppet freshness on mw122 is OK: puppet ran at Thu Nov 21 09:35:11 UTC 2013 [09:35:21] RECOVERY - Puppet freshness on mw1042 is OK: puppet ran at Thu Nov 21 09:35:11 UTC 2013 [09:35:21] RECOVERY - Puppet freshness on mw1095 is OK: puppet ran at Thu Nov 21 09:35:11 UTC 2013 [09:35:21] RECOVERY - Puppet freshness on lanthanum is OK: puppet ran at Thu Nov 21 09:35:11 UTC 2013 [09:35:22] RECOVERY - Puppet freshness on amssq52 is OK: puppet ran at Thu Nov 21 09:35:11 UTC 2013 [09:35:22] RECOVERY - Puppet freshness on mw34 is OK: puppet ran at Thu Nov 21 09:35:16 UTC 2013 [09:35:23] RECOVERY - Puppet freshness on ms-be8 is OK: puppet ran at Thu Nov 21 09:35:16 UTC 2013 [09:35:23] RECOVERY - Puppet freshness on search1008 is OK: puppet ran at Thu Nov 21 09:35:16 UTC 2013 [09:35:31] RECOVERY - Puppet freshness on mw46 is OK: puppet ran at Thu Nov 21 09:35:21 UTC 2013 [09:35:31] RECOVERY - Puppet freshness on mw1101 is OK: puppet ran at Thu Nov 21 09:35:21 UTC 2013 [09:35:31] RECOVERY - Puppet freshness on search1018 is OK: puppet ran at Thu Nov 21 09:35:21 UTC 2013 [09:35:31] RECOVERY - Puppet freshness on cp1061 is OK: puppet ran at Thu Nov 21 09:35:21 UTC 2013 [09:35:31] RECOVERY - Puppet freshness on search1019 is OK: puppet ran at Thu Nov 21 09:35:21 UTC 2013 [09:35:32] RECOVERY - Puppet freshness on mw1052 is OK: puppet ran at Thu Nov 21 09:35:21 UTC 2013 [09:35:32] RECOVERY - Puppet freshness on mw1144 is OK: puppet ran at Thu Nov 21 09:35:21 UTC 2013 [09:35:33] RECOVERY - Puppet freshness on sq65 is OK: puppet ran at Thu Nov 21 09:35:26 UTC 2013 [09:35:33] RECOVERY - Puppet freshness on mw1015 is OK: puppet ran at Thu Nov 21 09:35:26 UTC 2013 [09:35:41] RECOVERY - Puppet freshness on es1003 is OK: puppet ran at Thu Nov 21 09:35:31 UTC 2013 [09:35:41] RECOVERY - Puppet freshness on cp1020 is OK: puppet ran at Thu Nov 21 09:35:31 UTC 2013 [09:35:41] RECOVERY - Puppet freshness on cp1012 is OK: puppet ran at Thu Nov 21 09:35:36 UTC 2013 [09:35:51] RECOVERY - Puppet freshness on cp4003 is OK: puppet ran at Thu Nov 21 09:35:41 UTC 2013 [09:35:51] RECOVERY - Puppet freshness on cp4008 is OK: puppet ran at Thu Nov 21 09:35:41 UTC 2013 [09:35:51] RECOVERY - Puppet freshness on ms-be1 is OK: puppet ran at Thu Nov 21 09:35:46 UTC 2013 [09:35:51] RECOVERY - Puppet freshness on db1023 is OK: puppet ran at Thu Nov 21 09:35:46 UTC 2013 [09:36:01] RECOVERY - Puppet freshness on es1005 is OK: puppet ran at Thu Nov 21 09:35:51 UTC 2013 [09:36:01] RECOVERY - Puppet freshness on srv286 is OK: puppet ran at Thu Nov 21 09:35:51 UTC 2013 [09:36:01] RECOVERY - Puppet freshness on amssq55 is OK: puppet ran at Thu Nov 21 09:35:51 UTC 2013 [09:36:01] RECOVERY - Puppet freshness on srv237 is OK: puppet ran at Thu Nov 21 09:35:51 UTC 2013 [09:36:01] RECOVERY - Puppet freshness on mw47 is OK: puppet ran at Thu Nov 21 09:35:51 UTC 2013 [09:36:02] RECOVERY - Puppet freshness on mw14 is OK: puppet ran at Thu Nov 21 09:35:51 UTC 2013 [09:36:02] RECOVERY - Puppet freshness on mw125 is OK: puppet ran at Thu Nov 21 09:35:51 UTC 2013 [09:36:03] RECOVERY - Puppet freshness on mw9 is OK: puppet ran at Thu Nov 21 09:35:51 UTC 2013 [09:36:03] RECOVERY - Puppet freshness on db1019 is OK: puppet ran at Thu Nov 21 09:35:51 UTC 2013 [09:36:04] RECOVERY - Puppet freshness on srv250 is OK: puppet ran at Thu Nov 21 09:35:51 UTC 2013 [09:36:04] RECOVERY - Puppet freshness on srv258 is OK: puppet ran at Thu Nov 21 09:35:51 UTC 2013 [09:36:05] RECOVERY - Puppet freshness on mw62 is OK: puppet ran at Thu Nov 21 09:35:51 UTC 2013 [09:36:05] RECOVERY - Puppet freshness on mw49 is OK: puppet ran at Thu Nov 21 09:35:56 UTC 2013 [09:36:06] RECOVERY - Puppet freshness on srv284 is OK: puppet ran at Thu Nov 21 09:35:56 UTC 2013 [09:36:11] RECOVERY - Puppet freshness on analytics1010 is OK: puppet ran at Thu Nov 21 09:36:01 UTC 2013 [09:36:11] RECOVERY - Puppet freshness on mw1175 is OK: puppet ran at Thu Nov 21 09:36:01 UTC 2013 [09:36:11] RECOVERY - Puppet freshness on mw1214 is OK: puppet ran at Thu Nov 21 09:36:01 UTC 2013 [09:36:11] RECOVERY - Puppet freshness on mw1182 is OK: puppet ran at Thu Nov 21 09:36:01 UTC 2013 [09:36:11] RECOVERY - Puppet freshness on mw1127 is OK: puppet ran at Thu Nov 21 09:36:06 UTC 2013 [09:36:21] RECOVERY - Puppet freshness on mw1126 is OK: puppet ran at Thu Nov 21 09:36:11 UTC 2013 [09:36:21] RECOVERY - Puppet freshness on mw1114 is OK: puppet ran at Thu Nov 21 09:36:11 UTC 2013 [09:36:21] RECOVERY - Puppet freshness on pdf1 is OK: puppet ran at Thu Nov 21 09:36:11 UTC 2013 [09:36:21] RECOVERY - Puppet freshness on dobson is OK: puppet ran at Thu Nov 21 09:36:16 UTC 2013 [09:36:31] RECOVERY - Puppet freshness on sq73 is OK: puppet ran at Thu Nov 21 09:36:26 UTC 2013 [09:36:31] RECOVERY - Puppet freshness on sq72 is OK: puppet ran at Thu Nov 21 09:36:26 UTC 2013 [09:36:31] RECOVERY - Puppet freshness on es4 is OK: puppet ran at Thu Nov 21 09:36:26 UTC 2013 [09:36:41] RECOVERY - Puppet freshness on cp4004 is OK: puppet ran at Thu Nov 21 09:36:31 UTC 2013 [09:36:41] RECOVERY - Puppet freshness on cp1003 is OK: puppet ran at Thu Nov 21 09:36:32 UTC 2013 [09:36:41] RECOVERY - Puppet freshness on nescio is OK: puppet ran at Thu Nov 21 09:36:37 UTC 2013 [09:36:41] RECOVERY - Puppet freshness on cp3010 is OK: puppet ran at Thu Nov 21 09:36:37 UTC 2013 [09:36:51] RECOVERY - Puppet freshness on amssq34 is OK: puppet ran at Thu Nov 21 09:36:42 UTC 2013 [09:36:51] RECOVERY - Puppet freshness on es10 is OK: puppet ran at Thu Nov 21 09:36:42 UTC 2013 [09:36:51] RECOVERY - Puppet freshness on cp1013 is OK: puppet ran at Thu Nov 21 09:36:47 UTC 2013 [09:36:51] RECOVERY - Puppet freshness on es1002 is OK: puppet ran at Thu Nov 21 09:36:47 UTC 2013 [09:36:51] RECOVERY - Puppet freshness on ms5 is OK: puppet ran at Thu Nov 21 09:36:47 UTC 2013 [09:37:01] RECOVERY - Puppet freshness on mw100 is OK: puppet ran at Thu Nov 21 09:36:52 UTC 2013 [09:37:01] RECOVERY - Puppet freshness on pc1003 is OK: puppet ran at Thu Nov 21 09:36:52 UTC 2013 [09:37:01] RECOVERY - Puppet freshness on srv275 is OK: puppet ran at Thu Nov 21 09:36:52 UTC 2013 [09:37:01] RECOVERY - Puppet freshness on pc1002 is OK: puppet ran at Thu Nov 21 09:36:52 UTC 2013 [09:37:01] RECOVERY - Puppet freshness on cp1069 is OK: puppet ran at Thu Nov 21 09:36:52 UTC 2013 [09:37:01] RECOVERY - Puppet freshness on amssq57 is OK: puppet ran at Thu Nov 21 09:36:57 UTC 2013 [09:37:02] RECOVERY - Puppet freshness on db1052 is OK: puppet ran at Thu Nov 21 09:36:57 UTC 2013 [09:37:11] RECOVERY - Puppet freshness on mw1044 is OK: puppet ran at Thu Nov 21 09:37:02 UTC 2013 [09:37:11] RECOVERY - Puppet freshness on zirconium is OK: puppet ran at Thu Nov 21 09:37:02 UTC 2013 [09:37:11] RECOVERY - Puppet freshness on mw41 is OK: puppet ran at Thu Nov 21 09:37:02 UTC 2013 [09:37:11] RECOVERY - Puppet freshness on mw5 is OK: puppet ran at Thu Nov 21 09:37:02 UTC 2013 [09:37:11] RECOVERY - Puppet freshness on mw90 is OK: puppet ran at Thu Nov 21 09:37:02 UTC 2013 [09:37:12] RECOVERY - Puppet freshness on mw81 is OK: puppet ran at Thu Nov 21 09:37:02 UTC 2013 [09:37:12] RECOVERY - Puppet freshness on search1014 is OK: puppet ran at Thu Nov 21 09:37:07 UTC 2013 [09:37:13] RECOVERY - Puppet freshness on cp1051 is OK: puppet ran at Thu Nov 21 09:37:07 UTC 2013 [09:37:13] RECOVERY - Puppet freshness on mw1196 is OK: puppet ran at Thu Nov 21 09:37:07 UTC 2013 [09:37:21] RECOVERY - Puppet freshness on mw1162 is OK: puppet ran at Thu Nov 21 09:37:12 UTC 2013 [09:37:31] RECOVERY - Puppet freshness on cp4001 is OK: puppet ran at Thu Nov 21 09:37:27 UTC 2013 [09:37:31] RECOVERY - Puppet freshness on sq79 is OK: puppet ran at Thu Nov 21 09:37:27 UTC 2013 [09:37:41] RECOVERY - Puppet freshness on cp1006 is OK: puppet ran at Thu Nov 21 09:37:32 UTC 2013 [09:37:51] RECOVERY - Puppet freshness on db1048 is OK: puppet ran at Thu Nov 21 09:37:42 UTC 2013 [09:37:51] RECOVERY - Puppet freshness on lvs4 is OK: puppet ran at Thu Nov 21 09:37:42 UTC 2013 [09:37:51] RECOVERY - Puppet freshness on wtp1024 is OK: puppet ran at Thu Nov 21 09:37:47 UTC 2013 [09:37:51] RECOVERY - Puppet freshness on mw118 is OK: puppet ran at Thu Nov 21 09:37:47 UTC 2013 [09:37:52] RECOVERY - Puppet freshness on lvs3 is OK: puppet ran at Thu Nov 21 09:37:47 UTC 2013 [09:37:52] RECOVERY - Puppet freshness on db1036 is OK: puppet ran at Thu Nov 21 09:37:47 UTC 2013 [09:37:52] RECOVERY - Puppet freshness on mw119 is OK: puppet ran at Thu Nov 21 09:37:47 UTC 2013 [09:37:53] RECOVERY - Puppet freshness on mc1005 is OK: puppet ran at Thu Nov 21 09:37:47 UTC 2013 [09:37:53] RECOVERY - Puppet freshness on mw83 is OK: puppet ran at Thu Nov 21 09:37:47 UTC 2013 [09:38:01] RECOVERY - Puppet freshness on tin is OK: puppet ran at Thu Nov 21 09:37:52 UTC 2013 [09:38:01] RECOVERY - Puppet freshness on mw38 is OK: puppet ran at Thu Nov 21 09:37:52 UTC 2013 [09:38:01] RECOVERY - Puppet freshness on ms-be4 is OK: puppet ran at Thu Nov 21 09:37:52 UTC 2013 [09:38:01] RECOVERY - Puppet freshness on gadolinium is OK: puppet ran at Thu Nov 21 09:37:52 UTC 2013 [09:38:01] RECOVERY - Puppet freshness on cp1050 is OK: puppet ran at Thu Nov 21 09:37:57 UTC 2013 [09:38:01] RECOVERY - Puppet freshness on cp1046 is OK: puppet ran at Thu Nov 21 09:37:57 UTC 2013 [09:38:11] RECOVERY - Puppet freshness on srv301 is OK: puppet ran at Thu Nov 21 09:38:02 UTC 2013 [09:38:11] RECOVERY - Puppet freshness on srv290 is OK: puppet ran at Thu Nov 21 09:38:02 UTC 2013 [09:38:11] RECOVERY - Puppet freshness on mw1161 is OK: puppet ran at Thu Nov 21 09:38:02 UTC 2013 [09:38:11] RECOVERY - Puppet freshness on mw1125 is OK: puppet ran at Thu Nov 21 09:38:02 UTC 2013 [09:38:11] RECOVERY - Puppet freshness on mw1190 is OK: puppet ran at Thu Nov 21 09:38:02 UTC 2013 [09:38:12] RECOVERY - Puppet freshness on analytics1015 is OK: puppet ran at Thu Nov 21 09:38:02 UTC 2013 [09:38:12] RECOVERY - Puppet freshness on mw1147 is OK: puppet ran at Thu Nov 21 09:38:02 UTC 2013 [09:38:13] RECOVERY - Puppet freshness on lvs4003 is OK: puppet ran at Thu Nov 21 09:38:07 UTC 2013 [09:38:13] RECOVERY - Puppet freshness on mw1130 is OK: puppet ran at Thu Nov 21 09:38:07 UTC 2013 [09:38:14] RECOVERY - Puppet freshness on mw1124 is OK: puppet ran at Thu Nov 21 09:38:07 UTC 2013 [09:38:21] RECOVERY - Puppet freshness on mw1111 is OK: puppet ran at Thu Nov 21 09:38:12 UTC 2013 [09:38:21] RECOVERY - Puppet freshness on lvs1006 is OK: puppet ran at Thu Nov 21 09:38:12 UTC 2013 [09:38:31] RECOVERY - Puppet freshness on sq51 is OK: puppet ran at Thu Nov 21 09:38:27 UTC 2013 [09:38:31] RECOVERY - Puppet freshness on amssq44 is OK: puppet ran at Thu Nov 21 09:38:27 UTC 2013 [09:38:41] RECOVERY - Puppet freshness on rubidium is OK: puppet ran at Thu Nov 21 09:38:32 UTC 2013 [09:38:41] RECOVERY - Puppet freshness on cp4007 is OK: puppet ran at Thu Nov 21 09:38:37 UTC 2013 [09:38:51] RECOVERY - Puppet freshness on ms-fe3 is OK: puppet ran at Thu Nov 21 09:38:42 UTC 2013 [09:38:51] RECOVERY - Puppet freshness on ms-fe3001 is OK: puppet ran at Thu Nov 21 09:38:42 UTC 2013 [09:38:51] RECOVERY - Puppet freshness on mc1009 is OK: puppet ran at Thu Nov 21 09:38:42 UTC 2013 [09:38:51] RECOVERY - Puppet freshness on wtp1011 is OK: puppet ran at Thu Nov 21 09:38:47 UTC 2013 [09:39:01] RECOVERY - Puppet freshness on vanadium is OK: puppet ran at Thu Nov 21 09:38:52 UTC 2013 [09:39:01] RECOVERY - Puppet freshness on db1020 is OK: puppet ran at Thu Nov 21 09:38:52 UTC 2013 [09:39:01] RECOVERY - Puppet freshness on srv246 is OK: puppet ran at Thu Nov 21 09:38:52 UTC 2013 [09:39:01] RECOVERY - Puppet freshness on ms-be2 is OK: puppet ran at Thu Nov 21 09:38:52 UTC 2013 [09:39:01] RECOVERY - Puppet freshness on srv249 is OK: puppet ran at Thu Nov 21 09:38:52 UTC 2013 [09:39:01] RECOVERY - Puppet freshness on srv245 is OK: puppet ran at Thu Nov 21 09:38:52 UTC 2013 [09:39:02] RECOVERY - Puppet freshness on wtp1003 is OK: puppet ran at Thu Nov 21 09:38:58 UTC 2013 [09:39:02] RECOVERY - Puppet freshness on srv289 is OK: puppet ran at Thu Nov 21 09:38:58 UTC 2013 [09:39:03] RECOVERY - Puppet freshness on srv253 is OK: puppet ran at Thu Nov 21 09:38:58 UTC 2013 [09:39:03] RECOVERY - Puppet freshness on wtp1018 is OK: puppet ran at Thu Nov 21 09:38:58 UTC 2013 [09:39:04] RECOVERY - Puppet freshness on srv244 is OK: puppet ran at Thu Nov 21 09:38:58 UTC 2013 [09:39:04] RECOVERY - Puppet freshness on solr2 is OK: puppet ran at Thu Nov 21 09:38:58 UTC 2013 [09:39:11] RECOVERY - Puppet freshness on cp1063 is OK: puppet ran at Thu Nov 21 09:39:03 UTC 2013 [09:39:11] RECOVERY - Puppet freshness on mw30 is OK: puppet ran at Thu Nov 21 09:39:03 UTC 2013 [09:39:11] RECOVERY - Puppet freshness on mw76 is OK: puppet ran at Thu Nov 21 09:39:03 UTC 2013 [09:39:11] RECOVERY - Puppet freshness on mw1038 is OK: puppet ran at Thu Nov 21 09:39:03 UTC 2013 [09:39:11] RECOVERY - Puppet freshness on mw1057 is OK: puppet ran at Thu Nov 21 09:39:03 UTC 2013 [09:39:12] RECOVERY - Puppet freshness on mw1115 is OK: puppet ran at Thu Nov 21 09:39:03 UTC 2013 [09:39:12] RECOVERY - Puppet freshness on db1004 is OK: puppet ran at Thu Nov 21 09:39:08 UTC 2013 [09:39:13] RECOVERY - Puppet freshness on mw1056 is OK: puppet ran at Thu Nov 21 09:39:08 UTC 2013 [09:39:13] RECOVERY - Puppet freshness on ms-be1012 is OK: puppet ran at Thu Nov 21 09:39:08 UTC 2013 [09:39:14] RECOVERY - Puppet freshness on mw1004 is OK: puppet ran at Thu Nov 21 09:39:08 UTC 2013 [09:39:21] RECOVERY - Puppet freshness on mw1089 is OK: puppet ran at Thu Nov 21 09:39:13 UTC 2013 [09:39:21] RECOVERY - Puppet freshness on mw1146 is OK: puppet ran at Thu Nov 21 09:39:13 UTC 2013 [09:39:21] RECOVERY - Puppet freshness on mw1159 is OK: puppet ran at Thu Nov 21 09:39:18 UTC 2013 [09:39:21] RECOVERY - Puppet freshness on mw1048 is OK: puppet ran at Thu Nov 21 09:39:18 UTC 2013 [09:39:31] RECOVERY - Puppet freshness on mw1081 is OK: puppet ran at Thu Nov 21 09:39:23 UTC 2013 [09:39:31] RECOVERY - Puppet freshness on sq85 is OK: puppet ran at Thu Nov 21 09:39:28 UTC 2013 [09:39:32] RECOVERY - Puppet freshness on sq74 is OK: puppet ran at Thu Nov 21 09:39:28 UTC 2013 [09:39:41] RECOVERY - Puppet freshness on solr3 is OK: puppet ran at Thu Nov 21 09:39:33 UTC 2013 [09:39:51] RECOVERY - Puppet freshness on es6 is OK: puppet ran at Thu Nov 21 09:39:43 UTC 2013 [09:39:51] RECOVERY - Puppet freshness on sq43 is OK: puppet ran at Thu Nov 21 09:39:48 UTC 2013 [09:39:52] RECOVERY - Puppet freshness on solr1003 is OK: puppet ran at Thu Nov 21 09:39:48 UTC 2013 [09:39:52] RECOVERY - Puppet freshness on wtp1004 is OK: puppet ran at Thu Nov 21 09:39:48 UTC 2013 [09:39:52] RECOVERY - Puppet freshness on snapshot1 is OK: puppet ran at Thu Nov 21 09:39:48 UTC 2013 [09:39:52] RECOVERY - Puppet freshness on testsearch1003 is OK: puppet ran at Thu Nov 21 09:39:48 UTC 2013 [09:39:52] RECOVERY - Puppet freshness on ms-be9 is OK: puppet ran at Thu Nov 21 09:39:48 UTC 2013 [09:40:01] RECOVERY - Puppet freshness on srv276 is OK: puppet ran at Thu Nov 21 09:39:53 UTC 2013 [09:40:01] RECOVERY - Puppet freshness on db1029 is OK: puppet ran at Thu Nov 21 09:39:53 UTC 2013 [09:40:01] RECOVERY - Puppet freshness on mw8 is OK: puppet ran at Thu Nov 21 09:39:58 UTC 2013 [09:40:01] RECOVERY - Puppet freshness on ms-be3 is OK: puppet ran at Thu Nov 21 09:39:58 UTC 2013 [09:40:01] RECOVERY - Puppet freshness on srv294 is OK: puppet ran at Thu Nov 21 09:39:58 UTC 2013 [09:40:01] RECOVERY - Puppet freshness on srv292 is OK: puppet ran at Thu Nov 21 09:39:58 UTC 2013 [09:40:02] RECOVERY - Puppet freshness on mw58 is OK: puppet ran at Thu Nov 21 09:39:58 UTC 2013 [09:40:02] RECOVERY - Puppet freshness on cp1049 is OK: puppet ran at Thu Nov 21 09:39:58 UTC 2013 [09:40:11] RECOVERY - Puppet freshness on mw1171 is OK: puppet ran at Thu Nov 21 09:40:03 UTC 2013 [09:40:11] RECOVERY - Puppet freshness on mc1001 is OK: puppet ran at Thu Nov 21 09:40:03 UTC 2013 [09:40:11] RECOVERY - Puppet freshness on mw121 is OK: puppet ran at Thu Nov 21 09:40:03 UTC 2013 [09:40:11] RECOVERY - Puppet freshness on stat1 is OK: puppet ran at Thu Nov 21 09:40:08 UTC 2013 [09:40:11] RECOVERY - Puppet freshness on mw1007 is OK: puppet ran at Thu Nov 21 09:40:08 UTC 2013 [09:40:12] RECOVERY - Puppet freshness on mw1210 is OK: puppet ran at Thu Nov 21 09:40:08 UTC 2013 [09:40:12] RECOVERY - Puppet freshness on mw1186 is OK: puppet ran at Thu Nov 21 09:40:08 UTC 2013 [09:40:13] RECOVERY - Puppet freshness on mw1188 is OK: puppet ran at Thu Nov 21 09:40:08 UTC 2013 [09:40:13] RECOVERY - Puppet freshness on mw1087 is OK: puppet ran at Thu Nov 21 09:40:08 UTC 2013 [09:40:21] RECOVERY - Puppet freshness on mw1106 is OK: puppet ran at Thu Nov 21 09:40:13 UTC 2013 [09:40:21] RECOVERY - Puppet freshness on mw1140 is OK: puppet ran at Thu Nov 21 09:40:13 UTC 2013 [09:40:21] RECOVERY - Puppet freshness on mw1063 is OK: puppet ran at Thu Nov 21 09:40:13 UTC 2013 [09:40:21] RECOVERY - Puppet freshness on mw1041 is OK: puppet ran at Thu Nov 21 09:40:18 UTC 2013 [09:40:21] RECOVERY - Puppet freshness on search1024 is OK: puppet ran at Thu Nov 21 09:40:18 UTC 2013 [09:40:22] RECOVERY - Puppet freshness on mw1043 is OK: puppet ran at Thu Nov 21 09:40:18 UTC 2013 [09:40:22] RECOVERY - Puppet freshness on mw1197 is OK: puppet ran at Thu Nov 21 09:40:18 UTC 2013 [09:40:31] RECOVERY - Puppet freshness on ekrem is OK: puppet ran at Thu Nov 21 09:40:23 UTC 2013 [09:40:31] RECOVERY - Puppet freshness on ms10 is OK: puppet ran at Thu Nov 21 09:40:28 UTC 2013 [09:40:32] RECOVERY - Puppet freshness on cp1005 is OK: puppet ran at Thu Nov 21 09:40:28 UTC 2013 [09:40:32] RECOVERY - Puppet freshness on cp1002 is OK: puppet ran at Thu Nov 21 09:40:28 UTC 2013 [09:40:32] RECOVERY - Puppet freshness on db34 is OK: puppet ran at Thu Nov 21 09:40:28 UTC 2013 [09:40:41] RECOVERY - Puppet freshness on pc1 is OK: puppet ran at Thu Nov 21 09:40:33 UTC 2013 [09:40:41] RECOVERY - Puppet freshness on sq76 is OK: puppet ran at Thu Nov 21 09:40:38 UTC 2013 [09:40:51] RECOVERY - Puppet freshness on mc1007 is OK: puppet ran at Thu Nov 21 09:40:43 UTC 2013 [09:40:51] RECOVERY - Puppet freshness on cp3012 is OK: puppet ran at Thu Nov 21 09:40:43 UTC 2013 [09:40:51] RECOVERY - Puppet freshness on rdb1002 is OK: puppet ran at Thu Nov 21 09:40:43 UTC 2013 [09:40:51] RECOVERY - Puppet freshness on srv273 is OK: puppet ran at Thu Nov 21 09:40:43 UTC 2013 [09:40:51] RECOVERY - Puppet freshness on cp3009 is OK: puppet ran at Thu Nov 21 09:40:43 UTC 2013 [09:40:52] RECOVERY - Puppet freshness on mw57 is OK: puppet ran at Thu Nov 21 09:40:43 UTC 2013 [09:40:52] RECOVERY - Puppet freshness on wtp1015 is OK: puppet ran at Thu Nov 21 09:40:48 UTC 2013 [09:40:53] RECOVERY - Puppet freshness on potassium is OK: puppet ran at Thu Nov 21 09:40:48 UTC 2013 [09:40:53] RECOVERY - Puppet freshness on mw43 is OK: puppet ran at Thu Nov 21 09:40:48 UTC 2013 [09:40:54] RECOVERY - Puppet freshness on amssq53 is OK: puppet ran at Thu Nov 21 09:40:48 UTC 2013 [09:40:54] RECOVERY - Puppet freshness on srv255 is OK: puppet ran at Thu Nov 21 09:40:48 UTC 2013 [09:40:55] RECOVERY - Puppet freshness on helium is OK: puppet ran at Thu Nov 21 09:40:48 UTC 2013 [09:40:55] RECOVERY - Puppet freshness on db1031 is OK: puppet ran at Thu Nov 21 09:40:48 UTC 2013 [09:41:01] RECOVERY - Puppet freshness on db1001 is OK: puppet ran at Thu Nov 21 09:40:53 UTC 2013 [09:41:01] RECOVERY - Puppet freshness on mw124 is OK: puppet ran at Thu Nov 21 09:40:53 UTC 2013 [09:41:01] RECOVERY - Puppet freshness on elastic1004 is OK: puppet ran at Thu Nov 21 09:40:58 UTC 2013 [09:41:11] RECOVERY - Puppet freshness on ms-be1006 is OK: puppet ran at Thu Nov 21 09:41:08 UTC 2013 [09:41:11] RECOVERY - Puppet freshness on mw1032 is OK: puppet ran at Thu Nov 21 09:41:08 UTC 2013 [09:41:11] RECOVERY - Puppet freshness on analytics1014 is OK: puppet ran at Thu Nov 21 09:41:08 UTC 2013 [09:41:21] RECOVERY - Puppet freshness on searchidx1001 is OK: puppet ran at Thu Nov 21 09:41:13 UTC 2013 [09:41:31] RECOVERY - Puppet freshness on sq54 is OK: puppet ran at Thu Nov 21 09:41:23 UTC 2013 [09:41:31] RECOVERY - Puppet freshness on sq58 is OK: puppet ran at Thu Nov 21 09:41:23 UTC 2013 [09:41:51] RECOVERY - Puppet freshness on rdb1003 is OK: puppet ran at Thu Nov 21 09:41:45 UTC 2013 [09:41:51] RECOVERY - Puppet freshness on elastic1008 is OK: puppet ran at Thu Nov 21 09:41:45 UTC 2013 [09:41:51] RECOVERY - Puppet freshness on wtp1007 is OK: puppet ran at Thu Nov 21 09:41:45 UTC 2013 [09:41:51] RECOVERY - Puppet freshness on db1044 is OK: puppet ran at Thu Nov 21 09:41:43 UTC 2013 [09:41:51] RECOVERY - Puppet freshness on ms-fe1001 is OK: puppet ran at Thu Nov 21 09:41:50 UTC 2013 [09:41:52] RECOVERY - Puppet freshness on cp1010 is OK: puppet ran at Thu Nov 21 09:41:50 UTC 2013 [09:41:52] RECOVERY - Puppet freshness on mw98 is OK: puppet ran at Thu Nov 21 09:41:50 UTC 2013 [09:41:53] RECOVERY - Puppet freshness on mw79 is OK: puppet ran at Thu Nov 21 09:41:50 UTC 2013 [09:41:53] RECOVERY - Puppet freshness on mw106 is OK: puppet ran at Thu Nov 21 09:41:50 UTC 2013 [09:42:01] RECOVERY - Puppet freshness on db1022 is OK: puppet ran at Thu Nov 21 09:41:55 UTC 2013 [09:42:01] RECOVERY - Puppet freshness on db63 is OK: puppet ran at Thu Nov 21 09:41:55 UTC 2013 [09:42:01] RECOVERY - Puppet freshness on mw2 is OK: puppet ran at Thu Nov 21 09:41:55 UTC 2013 [09:42:01] RECOVERY - Puppet freshness on mw35 is OK: puppet ran at Thu Nov 21 09:42:00 UTC 2013 [09:42:01] RECOVERY - Puppet freshness on mw1024 is OK: puppet ran at Thu Nov 21 09:42:00 UTC 2013 [09:42:11] RECOVERY - Puppet freshness on mw1189 is OK: puppet ran at Thu Nov 21 09:42:05 UTC 2013 [09:42:11] RECOVERY - Puppet freshness on srv193 is OK: puppet ran at Thu Nov 21 09:42:05 UTC 2013 [09:42:12] RECOVERY - Puppet freshness on mw1150 is OK: puppet ran at Thu Nov 21 09:42:05 UTC 2013 [09:42:12] RECOVERY - Puppet freshness on mw1153 is OK: puppet ran at Thu Nov 21 09:42:06 UTC 2013 [09:42:21] RECOVERY - Puppet freshness on mw1205 is OK: puppet ran at Thu Nov 21 09:42:11 UTC 2013 [09:42:21] RECOVERY - Puppet freshness on mw1046 is OK: puppet ran at Thu Nov 21 09:42:11 UTC 2013 [09:42:21] RECOVERY - Puppet freshness on mw1068 is OK: puppet ran at Thu Nov 21 09:42:16 UTC 2013 [09:42:21] RECOVERY - Puppet freshness on virt1000 is OK: puppet ran at Thu Nov 21 09:42:16 UTC 2013 [09:42:21] RECOVERY - Puppet freshness on mw1069 is OK: puppet ran at Thu Nov 21 09:42:16 UTC 2013 [09:42:22] RECOVERY - Puppet freshness on mw1201 is OK: puppet ran at Thu Nov 21 09:42:16 UTC 2013 [09:42:22] RECOVERY - Puppet freshness on mw1173 is OK: puppet ran at Thu Nov 21 09:42:16 UTC 2013 [09:42:23] RECOVERY - Puppet freshness on mw1122 is OK: puppet ran at Thu Nov 21 09:42:16 UTC 2013 [09:42:31] RECOVERY - Puppet freshness on mw1003 is OK: puppet ran at Thu Nov 21 09:42:21 UTC 2013 [09:42:31] RECOVERY - Puppet freshness on mw1033 is OK: puppet ran at Thu Nov 21 09:42:21 UTC 2013 [09:42:31] RECOVERY - Puppet freshness on sq63 is OK: puppet ran at Thu Nov 21 09:42:26 UTC 2013 [09:42:41] RECOVERY - Puppet freshness on calcium is OK: puppet ran at Thu Nov 21 09:42:31 UTC 2013 [09:42:45] (03PS5) 10Nemo bis: Make the monthly querypages updates not hit each cluster on the same day [operations/puppet] - 10https://gerrit.wikimedia.org/r/95889 [09:42:51] RECOVERY - Puppet freshness on virt11 is OK: puppet ran at Thu Nov 21 09:42:41 UTC 2013 [09:42:51] RECOVERY - Puppet freshness on ssl1001 is OK: puppet ran at Thu Nov 21 09:42:41 UTC 2013 [09:42:51] RECOVERY - Puppet freshness on solr1002 is OK: puppet ran at Thu Nov 21 09:42:46 UTC 2013 [09:42:51] RECOVERY - Puppet freshness on srv260 is OK: puppet ran at Thu Nov 21 09:42:46 UTC 2013 [09:42:51] RECOVERY - Puppet freshness on mw104 is OK: puppet ran at Thu Nov 21 09:42:46 UTC 2013 [09:42:51] RECOVERY - Puppet freshness on srv267 is OK: puppet ran at Thu Nov 21 09:42:46 UTC 2013 [09:42:52] RECOVERY - Puppet freshness on iodine is OK: puppet ran at Thu Nov 21 09:42:46 UTC 2013 [09:42:52] RECOVERY - Puppet freshness on db48 is OK: puppet ran at Thu Nov 21 09:42:46 UTC 2013 [09:43:01] RECOVERY - Puppet freshness on mw68 is OK: puppet ran at Thu Nov 21 09:42:51 UTC 2013 [09:43:01] RECOVERY - Puppet freshness on ms-be11 is OK: puppet ran at Thu Nov 21 09:42:51 UTC 2013 [09:43:01] RECOVERY - Puppet freshness on ruthenium is OK: puppet ran at Thu Nov 21 09:42:51 UTC 2013 [09:43:01] RECOVERY - Puppet freshness on srv269 is OK: puppet ran at Thu Nov 21 09:42:51 UTC 2013 [09:43:01] RECOVERY - Puppet freshness on srv262 is OK: puppet ran at Thu Nov 21 09:42:51 UTC 2013 [09:43:02] RECOVERY - Puppet freshness on db1040 is OK: puppet ran at Thu Nov 21 09:42:56 UTC 2013 [09:43:02] RECOVERY - Puppet freshness on cp1037 is OK: puppet ran at Thu Nov 21 09:42:56 UTC 2013 [09:43:03] RECOVERY - Puppet freshness on mw112 is OK: puppet ran at Thu Nov 21 09:42:56 UTC 2013 [09:43:11] RECOVERY - Puppet freshness on mw37 is OK: puppet ran at Thu Nov 21 09:43:01 UTC 2013 [09:43:11] RECOVERY - Puppet freshness on mw1091 is OK: puppet ran at Thu Nov 21 09:43:01 UTC 2013 [09:43:11] RECOVERY - Puppet freshness on mw1025 is OK: puppet ran at Thu Nov 21 09:43:01 UTC 2013 [09:43:11] RECOVERY - Puppet freshness on mw1142 is OK: puppet ran at Thu Nov 21 09:43:06 UTC 2013 [09:43:11] RECOVERY - Puppet freshness on db1002 is OK: puppet ran at Thu Nov 21 09:43:06 UTC 2013 [09:43:21] RECOVERY - Puppet freshness on mw1010 is OK: puppet ran at Thu Nov 21 09:43:11 UTC 2013 [09:43:21] RECOVERY - Puppet freshness on mw1166 is OK: puppet ran at Thu Nov 21 09:43:11 UTC 2013 [09:43:21] RECOVERY - Puppet freshness on fluorine is OK: puppet ran at Thu Nov 21 09:43:11 UTC 2013 [09:43:21] RECOVERY - Puppet freshness on search1001 is OK: puppet ran at Thu Nov 21 09:43:12 UTC 2013 [09:43:21] RECOVERY - Puppet freshness on mw1118 is OK: puppet ran at Thu Nov 21 09:43:12 UTC 2013 [09:43:31] RECOVERY - Puppet freshness on mw1092 is OK: puppet ran at Thu Nov 21 09:43:22 UTC 2013 [09:43:31] RECOVERY - Puppet freshness on search1022 is OK: puppet ran at Thu Nov 21 09:43:22 UTC 2013 [09:43:41] RECOVERY - Puppet freshness on db54 is OK: puppet ran at Thu Nov 21 09:43:32 UTC 2013 [09:43:41] RECOVERY - Puppet freshness on sockpuppet is OK: puppet ran at Thu Nov 21 09:43:37 UTC 2013 [09:43:41] RECOVERY - Puppet freshness on copper is OK: puppet ran at Thu Nov 21 09:43:37 UTC 2013 [09:43:41] RECOVERY - Puppet freshness on db1051 is OK: puppet ran at Thu Nov 21 09:43:37 UTC 2013 [09:43:51] RECOVERY - Puppet freshness on analytics1003 is OK: puppet ran at Thu Nov 21 09:43:42 UTC 2013 [09:43:51] RECOVERY - Puppet freshness on osm-db1002 is OK: puppet ran at Thu Nov 21 09:43:42 UTC 2013 [09:43:51] RECOVERY - Puppet freshness on lvs1002 is OK: puppet ran at Thu Nov 21 09:43:42 UTC 2013 [09:43:51] RECOVERY - Puppet freshness on elastic1002 is OK: puppet ran at Thu Nov 21 09:43:47 UTC 2013 [09:44:01] RECOVERY - Puppet freshness on db1042 is OK: puppet ran at Thu Nov 21 09:43:52 UTC 2013 [09:44:01] RECOVERY - Puppet freshness on amssq48 is OK: puppet ran at Thu Nov 21 09:43:52 UTC 2013 [09:44:01] RECOVERY - Puppet freshness on db1021 is OK: puppet ran at Thu Nov 21 09:43:52 UTC 2013 [09:44:01] RECOVERY - Puppet freshness on db66 is OK: puppet ran at Thu Nov 21 09:43:57 UTC 2013 [09:44:11] RECOVERY - Puppet freshness on amssq47 is OK: puppet ran at Thu Nov 21 09:44:02 UTC 2013 [09:44:11] RECOVERY - Puppet freshness on mw10 is OK: puppet ran at Thu Nov 21 09:44:02 UTC 2013 [09:44:11] RECOVERY - Puppet freshness on mw28 is OK: puppet ran at Thu Nov 21 09:44:02 UTC 2013 [09:44:11] RECOVERY - Puppet freshness on mw60 is OK: puppet ran at Thu Nov 21 09:44:02 UTC 2013 [09:44:11] RECOVERY - Puppet freshness on mw116 is OK: puppet ran at Thu Nov 21 09:44:02 UTC 2013 [09:44:12] RECOVERY - Puppet freshness on db1034 is OK: puppet ran at Thu Nov 21 09:44:02 UTC 2013 [09:44:12] RECOVERY - Puppet freshness on mw1129 is OK: puppet ran at Thu Nov 21 09:44:07 UTC 2013 [09:44:13] RECOVERY - Puppet freshness on mw1213 is OK: puppet ran at Thu Nov 21 09:44:07 UTC 2013 [09:44:21] RECOVERY - Puppet freshness on mw1066 is OK: puppet ran at Thu Nov 21 09:44:12 UTC 2013 [09:44:21] RECOVERY - Puppet freshness on mw23 is OK: puppet ran at Thu Nov 21 09:44:12 UTC 2013 [09:44:21] RECOVERY - Puppet freshness on mw1107 is OK: puppet ran at Thu Nov 21 09:44:13 UTC 2013 [09:44:21] RECOVERY - Puppet freshness on search1007 is OK: puppet ran at Thu Nov 21 09:44:18 UTC 2013 [09:44:21] RECOVERY - Puppet freshness on mw1211 is OK: puppet ran at Thu Nov 21 09:44:18 UTC 2013 [09:44:21] RECOVERY - Puppet freshness on lvs1005 is OK: puppet ran at Thu Nov 21 09:44:18 UTC 2013 [09:44:31] RECOVERY - Puppet freshness on mw1027 is OK: puppet ran at Thu Nov 21 09:44:23 UTC 2013 [09:44:31] RECOVERY - Puppet freshness on sq52 is OK: puppet ran at Thu Nov 21 09:44:23 UTC 2013 [09:44:31] RECOVERY - Puppet freshness on mw1011 is OK: puppet ran at Thu Nov 21 09:44:23 UTC 2013 [09:44:31] RECOVERY - Puppet freshness on search1012 is OK: puppet ran at Thu Nov 21 09:44:23 UTC 2013 [09:44:31] RECOVERY - Puppet freshness on mw1143 is OK: puppet ran at Thu Nov 21 09:44:23 UTC 2013 [09:44:31] RECOVERY - Puppet freshness on mw1204 is OK: puppet ran at Thu Nov 21 09:44:23 UTC 2013 [09:44:32] RECOVERY - Puppet freshness on mw1054 is OK: puppet ran at Thu Nov 21 09:44:28 UTC 2013 [09:44:41] RECOVERY - Puppet freshness on labstore1 is OK: puppet ran at Thu Nov 21 09:44:33 UTC 2013 [09:44:41] RECOVERY - Puppet freshness on labstore1001 is OK: puppet ran at Thu Nov 21 09:44:33 UTC 2013 [09:44:42] RECOVERY - Puppet freshness on cp3011 is OK: puppet ran at Thu Nov 21 09:44:38 UTC 2013 [09:44:42] RECOVERY - Puppet freshness on dataset1001 is OK: puppet ran at Thu Nov 21 09:44:38 UTC 2013 [09:44:42] RECOVERY - Puppet freshness on solr1 is OK: puppet ran at Thu Nov 21 09:44:38 UTC 2013 [09:44:42] RECOVERY - Puppet freshness on mc1012 is OK: puppet ran at Thu Nov 21 09:44:38 UTC 2013 [09:44:51] RECOVERY - Puppet freshness on cp4002 is OK: puppet ran at Thu Nov 21 09:44:43 UTC 2013 [09:44:51] RECOVERY - Puppet freshness on ms-be12 is OK: puppet ran at Thu Nov 21 09:44:43 UTC 2013 [09:44:51] RECOVERY - Puppet freshness on cp4014 is OK: puppet ran at Thu Nov 21 09:44:43 UTC 2013 [09:44:51] RECOVERY - Puppet freshness on osm-cp1001 is OK: puppet ran at Thu Nov 21 09:44:43 UTC 2013 [09:44:51] RECOVERY - Puppet freshness on ssl1 is OK: puppet ran at Thu Nov 21 09:44:43 UTC 2013 [09:44:52] RECOVERY - Puppet freshness on mw55 is OK: puppet ran at Thu Nov 21 09:44:48 UTC 2013 [09:44:52] RECOVERY - Puppet freshness on labstore3 is OK: puppet ran at Thu Nov 21 09:44:48 UTC 2013 [09:44:53] RECOVERY - Puppet freshness on antimony is OK: puppet ran at Thu Nov 21 09:44:48 UTC 2013 [09:44:53] RECOVERY - Puppet freshness on wtp1001 is OK: puppet ran at Thu Nov 21 09:44:48 UTC 2013 [09:44:54] RECOVERY - Puppet freshness on mw42 is OK: puppet ran at Thu Nov 21 09:44:48 UTC 2013 [09:44:54] RECOVERY - Puppet freshness on ms-fe1002 is OK: puppet ran at Thu Nov 21 09:44:48 UTC 2013 [09:45:01] RECOVERY - Puppet freshness on srv242 is OK: puppet ran at Thu Nov 21 09:44:53 UTC 2013 [09:45:01] RECOVERY - Puppet freshness on mw75 is OK: puppet ran at Thu Nov 21 09:44:53 UTC 2013 [09:45:01] RECOVERY - Puppet freshness on cp1058 is OK: puppet ran at Thu Nov 21 09:44:58 UTC 2013 [09:45:11] RECOVERY - Puppet freshness on mw1155 is OK: puppet ran at Thu Nov 21 09:45:03 UTC 2013 [09:45:11] RECOVERY - Puppet freshness on mw1104 is OK: puppet ran at Thu Nov 21 09:45:03 UTC 2013 [09:45:11] RECOVERY - Puppet freshness on mw1208 is OK: puppet ran at Thu Nov 21 09:45:03 UTC 2013 [09:45:11] RECOVERY - Puppet freshness on mw1021 is OK: puppet ran at Thu Nov 21 09:45:08 UTC 2013 [09:45:11] RECOVERY - Puppet freshness on mw1206 is OK: puppet ran at Thu Nov 21 09:45:08 UTC 2013 [09:45:12] RECOVERY - Puppet freshness on mw1154 is OK: puppet ran at Thu Nov 21 09:45:08 UTC 2013 [09:45:12] RECOVERY - Puppet freshness on mw1131 is OK: puppet ran at Thu Nov 21 09:45:08 UTC 2013 [09:45:31] RECOVERY - Puppet freshness on bast1001 is OK: puppet ran at Thu Nov 21 09:45:23 UTC 2013 [09:45:31] RECOVERY - Puppet freshness on sodium is OK: puppet ran at Thu Nov 21 09:45:29 UTC 2013 [09:45:41] RECOVERY - Puppet freshness on es1004 is OK: puppet ran at Thu Nov 21 09:45:34 UTC 2013 [09:45:41] RECOVERY - Puppet freshness on cp1017 is OK: puppet ran at Thu Nov 21 09:45:34 UTC 2013 [09:45:51] RECOVERY - Puppet freshness on db9 is OK: puppet ran at Thu Nov 21 09:45:49 UTC 2013 [09:45:51] RECOVERY - Puppet freshness on sq84 is OK: puppet ran at Thu Nov 21 09:45:49 UTC 2013 [09:46:01] RECOVERY - Puppet freshness on capella is OK: puppet ran at Thu Nov 21 09:45:54 UTC 2013 [10:04:30] so the interesting thing about both cp3013 and cp3014 is that they have puppet certs, from june of this year [10:10:09] I love how the new puppet docs say palladium [10:10:12] and right after it [10:10:17] talk about solaris & blastwave :) [10:11:24] ugh what? [10:12:00] (03PS1) 10ArielGlenn: pull cp3013, 1014 from mobile cache list [operations/puppet] - 10https://gerrit.wikimedia.org/r/96729 [10:12:34] (03PS2) 10ArielGlenn: pull cp3013, 3014 from mobile cache list [operations/puppet] - 10https://gerrit.wikimedia.org/r/96729 [10:13:25] holdover from something that didn't work very well nor for that long [10:13:39] it's a noop though now :-P [10:14:26] (03CR) 10ArielGlenn: [C: 032] pull cp3013, 3014 from mobile cache list [operations/puppet] - 10https://gerrit.wikimedia.org/r/96729 (owner: 10ArielGlenn) [10:17:49] do we keep jar files in apt.wikimedia.org ? or pull them from git, or how do we handle them? [10:20:47] (03PS1) 10Ori.livneh: Add Gdash role class; apply to tungsten [operations/puppet] - 10https://gerrit.wikimedia.org/r/96732 [10:21:48] /a/graphite? [10:21:55] have we merged this already? [10:23:04] no, good catch [10:23:50] matanya: dunno [10:24:27] ori-l: i'm writing the logstash module, but it is just a giant jar, which i'm not sure how to handle [10:24:54] i can download from github, but that seems inscure and not nice [10:25:30] yeah, don't do that [10:25:39] have you gone through http://pkg-java.alioth.debian.org/docs/tutorial.html? [10:26:31] yes, along time ago. it is sort of a pain [10:26:36] (03PS6) 10Springle: Make the monthly querypages updates not hit each cluster on the same day [operations/puppet] - 10https://gerrit.wikimedia.org/r/95889 (owner: 10Nemo bis) [10:26:45] (03CR) 10Springle: [C: 032] Make the monthly querypages updates not hit each cluster on the same day [operations/puppet] - 10https://gerrit.wikimedia.org/r/95889 (owner: 10Nemo bis) [10:27:01] hrm [10:27:11] why does gdash need to know the whisper storage dir, anyway? [10:27:22] it should only be constructing URLs [10:27:37] I can do that if it is needed, but far from ideal [10:27:55] gdash queryies the carbon doesn't it? [10:28:32] haha [10:28:32] https://github.com/ripienaar/gdash/issues/91 [10:28:43] "Is this necessary and used somewhere in the code?" [10:29:55] seems like you need to update gdash [10:30:24] we're running a fork, sadly [10:30:51] backports are so much fun :/ [10:30:52] I bet apergos could fix this in a few seconds or such and set a personal record https://bugzilla.wikimedia.org/show_bug.cgi?id=52170#c2 >.> [10:31:07] busy right now [10:31:23] (can look later) [10:31:34] anyways, it looks like if you don't provide a value it defaults to "/var/lib/carbon/whisper", and then is never used [10:31:38] so i'll just leave it unset [10:31:56] jesus [10:32:05] or $deity, even [10:34:28] ori-l: i found out how to package it. is there any docs on how to submit .debs? [10:35:51] it's done via git [10:36:05] well, we version debian/ [10:36:24] the actual making of the package and uploading to reprepo is done manually by an ops person [10:37:01] ottomata and akosiaris recently-ish packaged a few java things, a starting point might be to look at their gerrit histories and see how they did it [10:37:27] good to know ori-l thanks. [10:38:20] ah, here's a good example: https://gerrit.wikimedia.org/r/#/q/status:merged+project:operations/debs/kafka,n,z [10:38:40] kafka is distributed as a jar file too, iirc [10:39:19] https://gerrit.wikimedia.org/r/#/c/68026/ is instructive (both commit & comments) [10:39:26] was just looking at that [10:40:44] oh, this seems like a lot of work :) [10:41:57] i think a lot of the files under debian/ are generated for you by whatever packaging tool they're using [10:42:27] Ah! just found out the logstash has an official debs [10:42:52] yay [10:43:51] ori-l: download.elasticsearch.org/logstash/logstash/packages/debian/logstash_1.2.2-debian1_all.deb [10:44:20] quoting from #logstash: " we will also be publishing our public repo's soon " [10:48:01] yes, there's a bug open on which I've commented [10:48:04] about an apt repo [10:48:15] Nik has commented too [10:48:25] before me even I think [10:48:55] https://github.com/elasticsearch/elasticsearch/issues/3286#issuecomment-28900359 [10:49:34] 72M deb? [10:49:35] ffs [10:49:51] anyway [10:49:55] off now [10:50:00] see you later [10:54:36] good night paravoid [10:56:14] (03PS2) 10Ori.livneh: Add Gdash role class; apply to tungsten [operations/puppet] - 10https://gerrit.wikimedia.org/r/96732 [10:57:18] (03CR) 10Ori.livneh: [C: 032] Add Gdash role class; apply to tungsten [operations/puppet] - 10https://gerrit.wikimedia.org/r/96732 (owner: 10Ori.livneh) [11:00:20] it should be good noon paravoid :) [11:18:31] even better: http://build.logstash.net/job/logstash.jar-flat.daily/469/artifact/pkg/logstash_1.2.2-ubuntu1_all.deb [11:46:35] (03PS1) 10Ori.livneh: applicationserver: retab [operations/puppet] - 10https://gerrit.wikimedia.org/r/96746 [12:06:54] ori-l: is brewster still proxing puppet? [12:07:59] is gerrit->gitblit replication broken? [12:08:18] e.g. https://gerrit.wikimedia.org/r/#/c/96652/ has no copy on gitblit [12:12:23] matanya: yes and no [12:12:45] akosiaris: can please add more on this? [12:12:46] yes the functionality is still enabled, no we now have a private link that allows direct connections [12:12:56] from esams to eqiad so no need [12:13:16] so it is no longer used [12:13:26] i know haproxy proxies brewster to stafford, so this is redundant? [12:14:05] I don't know if it proxies directly to stafford or puppet.* [12:14:26] but it is unused now. We might need to use it again though [12:15:03] akosiaris: see files/puppet/haproxy.cfg direct to stafford [12:15:39] in that case it is wrong [12:15:59] i would like to merge that into the module, and fix the config [12:16:11] which module? [12:16:14] what is the correct setup now? [12:16:16] puppet or puppetmaster ? [12:16:19] haproxy [12:18:04] damn there is both a haproxy module and a misc::haproxy class [12:18:26] and of course nobody uses the module [12:18:38] feel free to merge and amend and please merge the two classes [12:18:49] or even better [12:18:54] create a role class and a module [12:19:03] but stop the duplication please :-) [12:20:34] that was what i was inteding to do. just not sure what is the correct setup for proxing, it is clearly incorrect to proxy directly to stafford. i'm fixing right now, but not sure what should be done with this proxing issue [12:20:50] given there is no use case [12:21:04] I 'd say you should parameterize it [12:21:14] as an erb? [12:21:18] yes [12:21:36] and a parametered class for at least the end host (maybe other things as well) [12:21:54] ok, thanks. i'll push something soon, and ask you to review [12:21:58] ok [12:22:04] thanks as well :-) [13:06:57] PROBLEM - Varnish HTTP mobile-backend on cp3014 is CRITICAL: Connection refused [13:10:48] (03CR) 10Edenhill: Setting up varnishkafka on mobile varnish caches (031 comment) [operations/puppet] - 10https://gerrit.wikimedia.org/r/94169 (owner: 10Ottomata) [13:12:07] why is gerrit->gitblit replication broken? [13:12:09] e.g. https://gerrit.wikimedia.org/r/#/c/96652/ has no copy on gitblit [13:12:29] qchris_away: ^ [13:13:04] * YuviPanda mumbles UseGitHub [13:15:28] * apergos glares at YuviPanda [13:15:53] apergos: it's significantly faster for me to clone from github than gerrit [13:16:30] then note the difference in times please and bugzilla it, so that folks can look into it [13:16:48] I think I did [13:16:51] * YuviPanda hunt [13:16:51] s [13:21:26] apergos: apparently I had filed it specifically for vagrant https://bugzilla.wikimedia.org/show_bug.cgi?id=56046 [13:21:31] let me file a general one with time numbers [13:21:38] that sounds good [13:22:05] (03PS1) 10Matanya: haproxy: get rid of misc, add a role and template [operations/puppet] - 10https://gerrit.wikimedia.org/r/96759 [13:24:58] akosiaris: ^ [13:36:57] RECOVERY - Varnish HTTP mobile-backend on cp3014 is OK: HTTP OK: HTTP/1.1 200 OK - 189 bytes in 0.191 second response time [13:39:59] (03CR) 10Akosiaris: [C: 04-1] "See comments inline. Plus brewster's definition in site.pp should be amended to no longer use misc::haproxy but role::haproxy." (034 comments) [operations/puppet] - 10https://gerrit.wikimedia.org/r/96759 (owner: 10Matanya) [13:50:49] any idea why we call puppet with --no-splay ? [13:52:10] akosiaris: just to make sure i understand, you expect the variables to be called from the puppet class as endpoint_hostname=$endpoint_hostname ? [13:53:18] class haproxy ($endpoint_hostname, $endpoint_ip) { etcetcetc for the definition [13:53:19] and [13:53:26] class { 'haproxy': [13:53:48] endpoint_hostname => 'thisisahostname', [13:53:48] endpoint_ip => 'thisisanip', [13:53:48] } [13:53:57] for the role [13:54:12] the role class should call the module class this way [13:55:04] have a look at puppet parameterized class in http://docs.puppetlabs.com/guides/parameterized_classes.html [13:55:54] !log restarted Zuul, had some jobs stuck in its queue ({{gerrit|96762}} [13:56:09] Logged the message, Master [13:58:10] (03PS2) 10Matanya: haproxy: get rid of misc, add a role and template [operations/puppet] - 10https://gerrit.wikimedia.org/r/96759 [13:59:11] (03CR) 10jenkins-bot: [V: 04-1] haproxy: get rid of misc, add a role and template [operations/puppet] - 10https://gerrit.wikimedia.org/r/96759 (owner: 10Matanya) [14:00:55] (03PS3) 10Matanya: haproxy: get rid of misc, add a role and template [operations/puppet] - 10https://gerrit.wikimedia.org/r/96759 [14:01:38] (03CR) 10jenkins-bot: [V: 04-1] haproxy: get rid of misc, add a role and template [operations/puppet] - 10https://gerrit.wikimedia.org/r/96759 (owner: 10Matanya) [14:16:36] akosiaris: what am i missing here? ^ [14:17:11] * matanya didn't get the parameterized class thing until the end [14:17:41] gimme a sec, figuring something out [14:21:59] (03PS2) 10Akosiaris: Change the way cron run times are calculated [operations/puppet] - 10https://gerrit.wikimedia.org/r/96247 [14:26:03] matanya: a { [14:26:49] apergos: https://bugzilla.wikimedia.org/show_bug.cgi?id=57354 [14:26:53] akosiaris: where? i ran puppet parser validate, and it doesn't show me where [14:26:57] had to wait a while just to get the data :D [14:27:28] ok, thanks [14:28:20] err: Could not parse for environment production: Syntax error at ','; expected '}' at /srv/ssd/jenkins-slave/workspace/operations-puppet-validate/modules/haproxy/manifests/init.pp:3 [14:28:29] line 3 of that file [14:28:50] huh [14:28:54] wait ... it is worse [14:29:13] it needs to be class haproxy($endpoint_hostname, $endpoint_ip) { [14:29:19] and not what you have there [14:31:42] (03PS4) 10Matanya: haproxy: get rid of misc, add a role and template [operations/puppet] - 10https://gerrit.wikimedia.org/r/96759 [14:34:55] (03PS3) 10Akosiaris: Change the way cron run times are calculated [operations/puppet] - 10https://gerrit.wikimedia.org/r/96247 [14:35:37] MatmaRex: Thanks. No idea why it's broken, and I lack access to the relevant machines. The commit is on github. So it really seems gerrit -> gitblit only. [14:35:46] MatmaRex: I'm filing a bug. [14:36:07] qchris: [14:36:10] dont [14:36:20] akosiaris: Ok :-) [14:36:30] it was created yesterday during a firewall changing [14:36:43] and somethings have not been pushed to gitblit [14:37:03] i assume if a new change is pushed in gerrit, it will sync [14:37:11] (03PS1) 10coren: Tool Labs: install libhtml-template-perl [operations/puppet] - 10https://gerrit.wikimedia.org/r/96766 [14:37:17] cause other repos are up to date now [14:37:29] If it's just a fixed firewalling problem, then you're cretainly right. [14:37:59] akosiaris: Thanks. [14:38:09] your are welcome [14:38:21] akosiaris: i removed brewster totally [14:38:48] (03CR) 10coren: [C: 032] Tool Labs: install libhtml-template-perl [operations/puppet] - 10https://gerrit.wikimedia.org/r/96766 (owner: 10coren) [14:40:00] matanya: that won't unconfigure brewster as a haproxy though. it will still act as one and with a wrong config at that. [14:40:40] i see. so i'll two patches to get this right [14:40:46] i suggest you just use the new role::class to correctly configure brewster instead [14:41:22] all in one patch. No need for more than one. It's a rather simple change. [14:42:16] oh and instead of stafford in the role, put palladium and the ip 10.64.16.160 [14:46:15] (03PS5) 10Matanya: haproxy: get rid of misc, add a role and template [operations/puppet] - 10https://gerrit.wikimedia.org/r/96759 [14:47:41] akosiaris: i think it is ready now [14:48:45] (03PS6) 10Matanya: haproxy: get rid of misc, add a role and template [operations/puppet] - 10https://gerrit.wikimedia.org/r/96759 [14:48:54] (03CR) 10Akosiaris: [C: 04-1] "Not quite yet." (033 comments) [operations/puppet] - 10https://gerrit.wikimedia.org/r/96759 (owner: 10Matanya) [14:49:39] (03PS1) 10coren: Tool Labs: Multiple dependencies installed [operations/puppet] - 10https://gerrit.wikimedia.org/r/96767 [14:51:09] (03PS7) 10Matanya: haproxy: get rid of misc, add a role and template [operations/puppet] - 10https://gerrit.wikimedia.org/r/96759 [14:51:46] grrr [14:52:08] can't revoke by serial number until 2.7.20 in spite of what docs say [14:52:10] i'm annoying, i know :/ [14:52:33] * apergos makes a note to save that cleanup until someday when we upgrade >_< [14:55:10] (03CR) 10Akosiaris: [C: 032] haproxy: get rid of misc, add a role and template [operations/puppet] - 10https://gerrit.wikimedia.org/r/96759 (owner: 10Matanya) [14:55:38] thanks akosiaris sorry, it got so complicated [14:56:43] i still don't get how it will be fixed on brewster [14:57:07] matanya: meaning ? puppet will run and populated the configuration [14:57:17] and palladium and the new ip are the correct setup [14:58:04] but since we change it from stafford to palladium, the wrong config will stay on stafford, won't it? [14:58:07] (03CR) 10coren: [C: 032] Tool Labs: Multiple dependencies installed [operations/puppet] - 10https://gerrit.wikimedia.org/r/96767 (owner: 10coren) [14:58:19] what wrong config ? [14:58:55] what config to be more exact ? [14:59:03] the haproxy [14:59:06] (03PS3) 10QChris: Backup geowiki's data-private bare repository [operations/puppet] - 10https://gerrit.wikimedia.org/r/95363 [14:59:07] (03PS2) 10QChris: Extract geowiki paramaters into separate class [operations/puppet] - 10https://gerrit.wikimedia.org/r/96538 [14:59:10] we didn't remove it [14:59:24] yes, we fixed it [14:59:29] not remove it [14:59:47] it now points to a server that is actually working (palladium) and not to a server that was not (stafford) [14:59:48] so we need one more patch to remove it from stafford, correct? [15:00:15] what makes you think that something needs to be removed from stafford ? [15:00:46] the proxy setup only has config on brewster? nothing on the endpoint? [15:00:56] yup [15:01:03] (03CR) 10QChris: Extract geowiki paramaters into separate class (033 comments) [operations/puppet] - 10https://gerrit.wikimedia.org/r/96538 (owner: 10QChris) [15:01:14] now i get it, i'm very slow today :) [15:01:24] :-) [15:02:44] i'm out, see you later [15:03:44] PROBLEM - Apache HTTP on mw1188 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:03:54] PROBLEM - Apache HTTP on mw1216 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:04:04] PROBLEM - Apache HTTP on mw1044 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:04:04] PROBLEM - Apache HTTP on mw1077 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:04:04] PROBLEM - Apache HTTP on mw1075 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:04:14] PROBLEM - Apache HTTP on mw1091 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:04:14] PROBLEM - Apache HTTP on mw1097 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:04:15] PROBLEM - Apache HTTP on mw1082 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:04:31] huh [15:04:44] PROBLEM - Apache HTTP on mw1093 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:04:44] PROBLEM - Apache HTTP on mw1065 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:04:55] PROBLEM - Apache HTTP on mw1164 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:05:05] PROBLEM - Apache HTTP on mw1081 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:05:24] PROBLEM - Apache HTTP on mw1168 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:06:04] PROBLEM - Apache HTTP on mw1041 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:06:04] PROBLEM - Apache HTTP on mw1062 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:06:24] PROBLEM - Apache HTTP on mw1102 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:06:54] RECOVERY - Apache HTTP on mw1216 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 808 bytes in 4.924 second response time [15:06:54] RECOVERY - Apache HTTP on mw1044 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 808 bytes in 3.410 second response time [15:07:14] RECOVERY - Apache HTTP on mw1168 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 808 bytes in 4.997 second response time [15:07:27] (03PS1) 10coren: Tool Labs: fix pysvn installation [operations/puppet] - 10https://gerrit.wikimedia.org/r/96770 [15:07:54] RECOVERY - Apache HTTP on mw1041 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 808 bytes in 1.377 second response time [15:07:55] PROBLEM - Apache HTTP on mw1100 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:08:04] RECOVERY - Apache HTTP on mw1091 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 808 bytes in 0.051 second response time [15:08:04] PROBLEM - Apache HTTP on mw1056 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:08:14] RECOVERY - Apache HTTP on mw1102 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 808 bytes in 1.576 second response time [15:08:45] PROBLEM - Apache HTTP on mw1163 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:09:04] PROBLEM - Apache HTTP on mw1060 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:09:44] PROBLEM - Apache HTTP on mw1048 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:09:54] PROBLEM - Apache HTTP on mw1216 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:10:04] PROBLEM - Apache HTTP on mw1044 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:10:28] damn big outage [15:10:44] RECOVERY - Apache HTTP on mw1093 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 808 bytes in 6.505 second response time [15:10:54] RECOVERY - Apache HTTP on mw1077 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 808 bytes in 5.516 second response time [15:10:54] RECOVERY - Apache HTTP on mw1044 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 808 bytes in 5.811 second response time [15:11:14] PROBLEM - Apache HTTP on mw1035 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:11:23] Indeed. [15:11:54] RECOVERY - Apache HTTP on mw1216 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 808 bytes in 5.625 second response time [15:12:23] springle-afk: I see you on the DBs now. Are you looking at this or really afk? [15:13:02] two switches on eqiad are reporting weird errors [15:13:04] PROBLEM - Apache HTTP on mw1026 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:13:13] <^d> ^ Assuming you're all talking about the 503s I woke up to this morning [15:13:17] Coren: am here. phone beeped [15:13:19] * ^d goes back to breakfast [15:13:44] uh [15:13:44] PROBLEM - Apache HTTP on mw1093 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:13:53] (03PS1) 10Dereckson: New extra language for wikidata: Ottoman Turkish (ota) [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/96771 [15:13:54] RECOVERY - Apache HTTP on mw1164 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 808 bytes in 9.387 second response time [15:13:55] those just dtarted [15:14:05] withint the last ten minutes [15:14:11] (03CR) 10jenkins-bot: [V: 04-1] New extra language for wikidata: Ottoman Turkish (ota) [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/96771 (owner: 10Dereckson) [15:14:12] https://gdash.wikimedia.org/dashboards/reqerror/ [15:14:29] it more general than that i am afraid [15:14:41] well than yesterdays 503s i mean [15:14:44] RECOVERY - Apache HTTP on mw1048 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 808 bytes in 5.613 second response time [15:14:49] sorry, I was doing the scrollback and only saw demon's remark [15:14:57] not paged >_< [15:15:10] <^d> I just woke up to 503s this morning, wasn't saying they're related to yesterdays. [15:15:25] I'm seeing requests stuck for ~10m in the databases. [15:15:44] RECOVERY - Apache HTTP on mw1100 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 808 bytes in 0.045 second response time [15:15:54] PROBLEM - Apache HTTP on mw1046 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:16:12] <^d> Anyway, I'm not nearly awake enough to be of much use. I'll go back to my breakfast and lurk if anyone needs me. [15:16:44] PROBLEM - Apache HTTP on mw1099 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:16:44] PROBLEM - Apache HTTP on mw1215 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:16:50] (03PS2) 10Dereckson: New extra language for wikidata: Ottoman Turkish (ota) [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/96771 [15:17:04] RECOVERY - Apache HTTP on mw1056 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 808 bytes in 7.370 second response time [15:17:10] icinga resets my connections [15:17:15] apparently it's been very slow to edit wikidata (e.g. ordinary wiki text pages) [15:17:17] network issue ? [15:17:23] * aude tries [15:17:34] RECOVERY - Apache HTTP on mw1163 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 808 bytes in 2.475 second response time [15:17:44] pc1001 has a load of open connections [15:17:44] RECOVERY - Apache HTTP on mw1046 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 808 bytes in 1.615 second response time [15:17:53] seems ok now [15:18:04] RECOVERY - Apache HTTP on mw1075 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 808 bytes in 6.687 second response time [15:18:14] springle: Another rash of slow queries? [15:18:28] pfw1-eqiad RT_FLOW : RT_FLOW: RT_FLOW_SESSION_DENY: session denied 10.64.40.100/3306->208.80.154.6/40144 None 6(0) deny_other fundraising untrust UNKNOWN UNKNOWN N/A(N/A) vlan.1135 [15:18:34] Coren: no it's wierd. this is parsercache stuff trying to commit [15:18:43] do those mean anything to anyone ? [15:18:44] RECOVERY - Apache HTTP on mw1215 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 808 bytes in 6.819 second response time [15:18:44] "morning" [15:18:50] no clue, sorry [15:18:55] PROBLEM - Apache HTTP on mw1054 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:19:01] db1010 is groaning under the strain. [15:19:14] PROBLEM - Apache HTTP on mw1079 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:19:42] Coren: db1010? it has a lot of sleepers, not much action... [15:20:04] PROBLEM - Apache HTTP on mw1056 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:20:11] springle: Oh, nevermind, I was looking at the wrong window. [15:20:19] * Coren needs moar caffeine. [15:20:44] PROBLEM - Apache HTTP on mw1049 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:21:04] PROBLEM - Apache HTTP on mw1075 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:22:14] RECOVERY - Apache HTTP on mw1079 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 808 bytes in 8.656 second response time [15:22:44] RECOVERY - Apache HTTP on mw1093 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 808 bytes in 6.091 second response time [15:23:04] RECOVERY - Apache HTTP on mw1075 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 808 bytes in 9.518 second response time [15:24:04] RECOVERY - Apache HTTP on mw1081 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 808 bytes in 2.850 second response time [15:24:04] RECOVERY - Apache HTTP on mw1035 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 808 bytes in 0.056 second response time [15:24:05] RECOVERY - Apache HTTP on mw1082 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 808 bytes in 4.140 second response time [15:24:34] RECOVERY - Apache HTTP on mw1049 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 808 bytes in 0.103 second response time [15:24:34] RECOVERY - Apache HTTP on mw1065 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 808 bytes in 0.056 second response time [15:24:34] RECOVERY - Apache HTTP on mw1188 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 808 bytes in 0.061 second response time [15:24:35] RECOVERY - Apache HTTP on mw1099 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 808 bytes in 0.056 second response time [15:24:45] RECOVERY - Apache HTTP on mw1054 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 808 bytes in 0.073 second response time [15:24:54] RECOVERY - Apache HTTP on mw1026 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 808 bytes in 0.071 second response time [15:24:55] RECOVERY - Apache HTTP on mw1056 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 808 bytes in 0.068 second response time [15:24:55] RECOVERY - Apache HTTP on mw1060 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 808 bytes in 0.091 second response time [15:24:55] RECOVERY - Apache HTTP on mw1062 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 808 bytes in 0.095 second response time [15:25:04] RECOVERY - Apache HTTP on mw1097 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 808 bytes in 0.080 second response time [15:34:27] !log bounced pc1001 mysqld. massive spike of writes exhausted innodb txn log slots and wouldn't be killed [15:34:37] apergos: ^ [15:34:40] I see it [15:34:41] Logged the message, Master [15:34:45] what was running over there? [15:35:15] thousands of transactions waiting to COMMIT /* SqlBagOStuff::set */ [15:35:24] etf [15:35:28] *wtf [15:35:59] write spike started about 30m ago [15:36:43] 30 mins.. meh [15:39:34] a little after fr [15:39:35] well [15:40:08] we could ask them to turn it back on and watch for a bit... either confirm or rule it out [15:40:28] might be better than not knowing [15:43:40] (03CR) 10coren: [C: 032] Tool Labs: fix pysvn installation [operations/puppet] - 10https://gerrit.wikimedia.org/r/96770 (owner: 10coren) [15:44:50] ok, so fr is going to renable 100% again and we're going to see, bearing in mind it might take a while for it to pileup [15:47:04] Hm. Was the cross-project banner suppression thing disabled? [15:48:15] apergos: some 10-15 mins [15:48:32] I have the feeling that is might indeed be it [15:49:21] we'll see [15:54:48] Coren, it was [15:54:54] caching [15:55:02] ...of which it had none:P [16:17:35] andrewbogott: regarding : https://gerrit.wikimedia.org/r/#/c/91573/3/manifests/mail.pp i didn't get your comment about the dependcy [16:19:39] (03CR) 10PiRSquared17: "Thanks for this patch." [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/95846 (owner: 10Odder) [16:21:22] (03CR) 10PiRSquared17: "BTW this is bug 56634, not bug 56334. Can you edit the message after it's submitted? (new to gerrit)" [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/95846 (owner: 10Odder) [16:28:00] matanya: This line "Exec["mkdir /var/spool/exim4/scan"] -> Mount["/var/spool/exim4/scan"] -> File["/var/spool/exim4/scan"] [16:28:13] That ensures that the mount points exist before the 'mount' happens. [16:28:33] Without that line, there's some chance that puppet will try to mount before the dir is created. A 'requires' line can fix that. [16:28:35] exec mkdir ? [16:28:44] please make it a file resource :-) [16:29:31] andrewbogott: you mean puppet might the mouting before creating the directory that is declared right after? [16:29:40] *do the [16:29:53] (03PS1) 10Akosiaris: Renaming role::haproxy to role::puppetproxy [operations/puppet] - 10https://gerrit.wikimedia.org/r/96782 [16:29:58] akosiaris: yep, the patch we're discussing is exactly about fixing that :) [16:30:09] :-D :-D :-D [16:30:15] matanya: right, exactly. [16:30:32] ok andrewbogott i'll add a require ilne [16:30:43] thx [16:31:13] (03CR) 10Akosiaris: [C: 032] Renaming role::haproxy to role::puppetproxy [operations/puppet] - 10https://gerrit.wikimedia.org/r/96782 (owner: 10Akosiaris) [16:36:47] (03PS4) 10Matanya: mail.pp: change exec to file [operations/puppet] - 10https://gerrit.wikimedia.org/r/91573 [16:37:23] (03CR) 10jenkins-bot: [V: 04-1] mail.pp: change exec to file [operations/puppet] - 10https://gerrit.wikimedia.org/r/91573 (owner: 10Matanya) [16:39:21] (03PS5) 10Matanya: mail.pp: change exec to file [operations/puppet] - 10https://gerrit.wikimedia.org/r/91573 [16:44:47] (03CR) 10Odder: "Yes, but not after the patch had been merged. I only realized this mistake yesterday, but it doesn't matter that much now. Apologies for i" [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/95846 (owner: 10Odder) [16:47:51] andrewbogott: pushed [16:48:04] ok! [16:49:01] matanya: I saw in scrollback that you are working on a logstash puppet module [16:49:11] I just wanted to say \o/ for that [16:49:34] Have you seen the bits I've been working on in labs related to logstash? [17:10:36] (03CR) 10Andrew Bogott: mail.pp: change exec to file (031 comment) [operations/puppet] - 10https://gerrit.wikimedia.org/r/91573 (owner: 10Matanya) [17:18:48] csteipp: around? [17:18:58] yurik: yep [17:19:51] <^d> !log reloaded zuul to pick up Ibc354db7 [17:20:07] Logged the message, Master [17:20:14] csteipp: https://gerrit.wikimedia.org/r/#/c/96654/ [17:20:17] what are you thoughts about it? [17:21:06] My initial thought was that belonged in apache, but I haven't taken a deep look at it [17:21:21] csteipp: what do you mean by apache? [17:21:25] it is php :) [17:22:34] Apache config. But I guess since you're trying to work with the dynamic configs, you have to do it in php [17:22:53] Needs sanitization [17:23:28] csteipp: what kind of sanitazation is missing? [17:23:53] $redirect [17:25:02] csteipp: but that redirect is generated specifically for this? [17:27:00] (03CR) 10CSteipp: [C: 04-1] Mobile m. and zero. landing page redirect handling by ZERO (033 comments) [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/96654 (owner: 10Yurik) [17:28:03] yurik: It probably is, but please do sanity check right there, in case someone modifies that class later on. [17:28:05] csteipp: thanks! [17:28:22] what kind of sanitation were you thinking about? [17:28:51] i will fix it soon [17:34:09] (03PS6) 10Matanya: mail.pp: change exec to file [operations/puppet] - 10https://gerrit.wikimedia.org/r/91573 [17:35:12] csteipp: i am still not sure how sanitize redirect url better - we can't do wfUrlencode or anything like that, and that redirect is the same code as we use for all other redirects like when user sees a warning message [17:35:48] RobH: Can you help me figure out what's going on with virt100x? I can ping virt1000 and virt1004-1007… 1002 and 1003 don't seem to be in dns and 1001 is in dns but unresponsive. [17:35:58] RobH: My guess is there's a big chart someplace that explains what's happening? [17:36:06] so if it is broken inside the class, it would affect other redirects as well [17:36:15] (03PS7) 10Matanya: mail.pp: change exec to file [operations/puppet] - 10https://gerrit.wikimedia.org/r/91573 [17:36:16] there aint a chart [17:36:34] andrewbogott: so all the mgmt should be working, i have no clue to OS states [17:36:52] as i wasnt directly involved in setting up labs, you may need to ask coren or ryan [17:37:06] yurik_: Maybe he's worried about header splitting? PHP theoretically takes care of that internally since v5.1.2 though. [17:37:10] (i can stop doing rfp stuff and try to figure it out, but it would be going in from scratch) [17:37:49] andrewbogott: I haven't really touched the virt1000s yet. [17:37:53] RobH: I can check via mgmt, I just thought maybe this stuff was tracked. If it isn't that's fine, I'll just wing it :) [17:38:03] So I don't know what their statii are. [17:38:04] Some of 'em are broken because I broke 'em :) [17:38:06] well, you can check if it has signed puppet certs, etc... [17:38:20] It's the ones that aren't in DNS that puzzle me. [17:38:31] prolly never setup then, or someone got a bit too delte happy [17:39:04] Ryan_Lane: do you know if virt1001 and virt1002 are among those boxes 'loaned' to analytics? I note that they aren't in DNS [17:39:23] andrewbogott: some were also stolen for parser cache [17:39:30] * apergos runs the dreaded report [17:39:54] 0, 5 and 7 have certs [17:40:04] and respond to salt and have stored configs [17:40:13] the rest do not [17:40:22] this is for virt100x [17:40:32] how can I find out if the systems formerly known as 1001 and 1002 are there but off (or hung) vs. actively repurposed? [17:40:36] yurik: At least make sure there are no newlines in it. Checking filter_var( $redirect, FILTER_VALIDATE_URL ) would be even better. [17:41:15] https://rt.wikimedia.org/Ticket/Display.html?id=3687 [17:41:21] by looking in rt if you have access [17:41:40] virt1001-3 -> pc1001-3 [17:41:45] so there is that piece of the answer [17:42:10] (03CR) 10Nemo bis: "For reference, this was for https://bugzilla.wikimedia.org/show_bug.cgi?id=41362" [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/29913 (owner: 10Tim Starling) [17:42:15] yikes, ok. [17:42:18] apergos: thanks! [17:42:20] sure [17:42:41] for any of this stuff I always hunt through all the back tickets putting in the first part of the name like virt1 or something [17:42:56] not super swesome but we don't have better [17:43:16] apergos: OK, I guess I know how to do that, just assumer there was… a better way :) [17:43:21] RobH: Post-migration will it be reasonably straightforward for us to ship some of the virtx servers in tampa to eqiad and add them to the new labs cluster? [17:43:36] post migration + wipe [17:43:44] sure. [17:43:48] not that I have found. one goes through: git logs for puppet, git logs for dns, rt, and even digs into racktables and hopes hthat [17:43:48] but you also need to keep in mind that what we take for eqiad [17:43:51] you wont have for new datacenter [17:43:56] or you wont have right away and identical [17:43:58] between those and old pages on wikitech typically found via google.. [17:44:02] the information is available [17:44:09] So we can't use them as space /during/ the migration but can rely on them for growth space in the long run. [17:44:14] which reminds me anyone here knw about the server 'pascal' ? :-D [17:44:24] andrewbogott: do you want growth space to be identical between sites? [17:44:40] right now the idea is labs has same # of servers in each location (or it was in the beginning) [17:45:04] RobH: Unknown. We were set up to use both datacenters for labs at the same time but never actually used more than one. [17:45:21] So… going forward we'll have to sort out whether we're trying to distribute things or just keep it all in one basket. [17:45:24] Does this andrew otto fellow have his classes today? [17:45:49] its be nice if that was figured out before we take the time to setup the existing ciscos from tampa in eqiad ;] [17:46:04] but yes, we can ship tampa virt servers to ashburn (they must be wiped before shipment!) [17:46:20] I just would assume it would be ideal to keep the two sites identical [17:46:25] ashburn + future tbd site [17:46:33] and add new hw to both virt clusters at same time [17:46:43] so virt1001 and virt2001 always match spec [17:46:46] so on... [17:46:50] (but doesnt matter really) [17:47:07] as long as folks realize they aren't mirrors, then its cool. [17:47:21] And we don't really have a way to get more ciscos because they were a one-time donation, right? [17:47:40] what happened to analytics giving us some back? [17:48:11] they're using, what? 4 systems? [17:48:11] Ryan_Lane: I just sent an email about that. [17:48:26] But we are /very/ short right now, only have five in eqiad atm [17:48:58] yeah, and analytics at some point had like 10 or so [17:49:10] two more (virt13 and 14) will get sent to you [17:49:25] * Ryan_Lane nods [17:49:34] (there is ticket) [17:50:00] the fast way is get some back in eqiad [17:50:11] andrewbogott: just note i am not telling you no, im asking about our plans =] [17:50:31] so reclaim from analytics is faster than having tampa ciscos shipped [17:50:48] RobH: Yep, I follow. The idea of migrating servers from pmtpa is a very-long-term thought, nothing related to the immediate migration. [17:50:50] and yes, we should not expect to get more ciscos (unless we purchase them) is my understanding [17:50:55] ahh, cool [17:52:17] I"m gonna... be afk for awhile (horrors!)... see folks later [17:52:35] bye apergos [17:56:45] (03PS3) 10Yurik: Mobile m. and zero. landing page redirect handling by ZERO [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/96654 [17:57:49] (03CR) 10Yurik: Mobile m. and zero. landing page redirect handling by ZERO (033 comments) [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/96654 (owner: 10Yurik) [18:10:18] Reedy: what's the difference between extensions listed in normalExtensions vs branchedExtensions in make-wmf-branch? [18:10:35] Reedy: ie, aren't the one in normal branched? [18:10:40] (when core is branched) [18:12:37] (03CR) 10Manybubbles: [C: 04-1] "Going to revise update timeout. It should be faster so we don't clog the queue if updates start to bunch up." [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/96064 (owner: 10Manybubbles) [18:13:20] greg-g: i think a branch of visual editor gets made same time as core [18:13:31] so they can cherry pick bug fixes in? [18:13:49] aude: right, so, I guess, isn't that what happens to the normalExtensions list? [18:13:56] or am I misunderstanding [18:14:02] https://git.wikimedia.org/blob/mediawiki%2Ftools%2Frelease.git/4f0733efadb05bd4ab4ae69e3f21aaea0eec40a8/make-wmf-branch%2Fdefault.conf [18:14:23] for normal extensions, core branch points to submodule point in master of those extensions [18:14:45] * aude not expert here, though :) [18:15:29] so, not a real branch, ust a reference to their master? [18:15:29] but i see visual editor has branches for each branch of core, and normal extensions do not [18:15:32] yep [18:15:35] * greg-g nods [18:15:37] cool [18:15:39] thanks aude [18:15:40] :) [18:15:53] 15/15 [18:15:55] probably similar situation to wikibase, that stuff is changing so much that a fixed branch is helpful [18:16:01] awesome [18:17:14] paravoid: I joked that we need a sign in the office "X days since last outage" or "X days with an outage since last stable day" :) [18:17:33] (that last one needs a wordsmith) [18:18:51] :( [18:19:50] http://devopsreactions.tumblr.com/post/64194256689/11-hour-on-call-marathon [18:21:13] haha, yeah [18:21:20] * greg-g hugs paravoid  [18:21:49] Ryan_Lane: is wikitech-static hosted on AWS or by rackspace, or… someone else? [18:22:05] I'm just answering a random question from my brother about what hosting service he should use. [18:22:37] it was rackspace, at least [18:23:00] rackspace [18:23:10] used to be linode [18:25:03] Any idea what it costs and if we're happy with their service? [18:25:13] * andrewbogott is, apparently, too lazy to google [18:25:48] I think rackspace is decent, I dont' know about price/performance/features/etc for a personal site [18:26:16] (03PS1) 10Manybubbles: Graph components of Elasticsearch health [operations/puppet] - 10https://gerrit.wikimedia.org/r/96796 [18:31:35] (03PS2) 10Manybubbles: Graph components of Elasticsearch health [operations/puppet] - 10https://gerrit.wikimedia.org/r/96796 [18:31:47] andrewbogott: i'm with them if you want another to compare https://www.vr.org/ [18:32:48] mutante: thanks [18:33:49] Reedy: heya, could you please not branch/deploy VE today? no one from VE team is around to deal with issues. [18:35:18] In the same way the older version of VE might not work with current core master [18:35:49] (03PS2) 10Manybubbles: Increase Cirrus pool counter for new servers [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/96064 [18:35:59] riiight [18:36:02] * greg-g hrmmms [18:36:27] greg-g, no upfront communication from the VE team with you about what's going out and how to handle it? [18:36:35] Eloquence: no. [18:36:40] grumble grumble [18:37:02] It's not like it matters so much [18:37:07] it's just VE ;) [18:37:10] It's only test wikis plus mediawikiwiki [18:37:18] it's default-enabled on craploads of wikis [18:37:27] But we're not deploying the new code anywhere [18:37:34] Not till tuesday [18:37:50] true [18:37:56] The code to go to the wikipedias has been on test wikis for a week, and the rest of the non wikipedias since monday [18:38:07] unless you're worried about the phase1 -> phase2 wiki change, Eloquence [18:38:17] yes, I'm worried about it [18:38:18] ie: non-wikipedias to wikipedias [18:38:40] just because it's been on test2 on mediawiki.org doesn't mean we've caught significant mission-critical VE issues during that time [18:38:50] ori-l...I received the new disk for vanadium...I will need to shut it down to replace. let me know when it will be ok [18:38:56] if it's going to all WPs today with wmf4 and breaks crap, who'll fix it? [18:39:14] Me likely [18:39:16] yeah [18:39:18] By revertig [18:39:22] to an older version [18:39:25] which may not work in core [18:39:30] yeah right. [18:39:31] see above, ad-infinitum [18:39:34] not good enough. [18:39:38] It's very much their fault if VE is broken on master when they know it's due to be branched and deployed [18:39:45] Even if travelling [18:39:47] we need a VE person to be on point during VE deploys. [18:39:48] Don't submit! [18:40:20] cmjohnson: how soon could it be done? [18:40:47] Reedy, here's what we can do to not mess up the deploy cycle [18:40:56] once I shut the server down ...5-10 mins total [18:41:07] so: option 1: hold VE back while updating core on phase2 wikis ... option 2) update both to whats on non-wikipedias ... option 3) don't deploy anything to wikipedias [18:41:24] greg-g, Reedy, I'd like to see what test2 looks like with core on wmf4 and VE on wmf3. [18:41:31] hi greg-g and Reedy [18:41:59] Reedy: can you do that first? [18:42:00] I think the risk of breakage with the new version core is less likely than major regressions in VE with nobody around to fix them [18:42:09] so, we made a new branch for wikibase, but i would like another day to test it before updating our stuff on test2 and test.wikidata [18:42:23] if I'm wrong, I'd prefer to hold off on wmf4 entirely. I'm not comfortable signing off on a VE deploy with wmf4 to prod wikis without any VE developers around. [18:42:30] Look at all the special requests. [18:42:32] aude: no deploys on friday :) [18:42:38] greg-g: hmmmm [18:42:42] * aude panics [18:42:46] This is the jumping-off point! :P [18:42:48] what's going on? [18:42:58] paravoid: s'ok [18:43:02] paravoid: nothing [18:43:11] Reedy: My only request is that you not break the cluster and remember to create the BetaFeatures table on every wiki before enabling it. :) [18:43:12] doesn't sound like nothing :) [18:43:24] paravoid: just figuring out whether to update VE etc today, nothing's broken yet :) [18:43:30] Miiinor issue. [18:43:31] ok, I'll stay out of your way :) [18:43:33] I wish the VE team would communicate their release plans to the release manager when traveling :( [18:43:36] greg-g: how about just update test.wikidata? [18:43:50] the issue i am looking at and have fixed (in gerrit) affects only test2 [18:43:52] RECOVERY - RAID on vanadium is OK: OK: Active: 5, Working: 5, Failed: 0, Spare: 0 [18:44:04] cmjohnson: OK, i alerted folks [18:44:09] oh, already done? wow. [18:44:17] ori-l....no [18:44:23] greg-g, edsanders is around. poking him now. [18:44:34] i just removing it from software raid....have to do that before the physical disk change [18:44:38] aude: interesting [18:44:44] Eloquence: thanks [18:44:47] cmjohnson: ok, no rush at all [18:45:14] aude: so, the breakage on test/test2.wikipedia, how bad? [18:45:24] only test2 [18:45:30] -test [18:45:31] ;) [18:45:31] what is broken? [18:45:34] ori-l: nothing [18:45:36] (yet) [18:45:48] I should have said "theoretical/potential breakage" [18:45:53] affects our parser function, so i wouldn't like to deploy without my fix [18:45:56] we're all a bit tense this week, eh? [18:46:00] but i also don't want to self merge it :) [18:46:32] which change? [18:46:34] aude: who do you need to review in the next 15 minutes? [18:46:40] ori-l's offering :) [18:46:44] if we must wait until monday, i suppose it's okay but we want to get quantities onto test.wikidata so people can try them :) [18:47:01] greg-g: https://gerrit.wikimedia.org/r/#/c/96792/ [18:47:07] * ori-l looks [18:47:11] although probably best for someone from the wikidata team to look [18:47:17] yeah :/ [18:47:27] you can see jenkins passes and i added more tests for the issue [18:47:43] so, last minute bug in software that'll break stuff usually delays deploy, I undersatnd the panic since next week is a no deploy week [18:47:45] yeah, it's not a 15 minute review [18:47:51] ori-l..i am ready if you wanna take it down now [18:47:56] greg-g: oh, that's bad [18:48:06] we need people to test quantities on test.wikidata [18:48:08] cmjohnson: yeah, go for it [18:48:19] test.wikidata is not hooked up to any clients yet [18:48:20] okay cool [18:48:33] !log shutting down vanadium to replace failed hard drive [18:48:50] Logged the message, Master [18:48:57] aude: so, let me rephrase/clarify: the issue *only* manifests on test2 with the new wikibase, not testwikidata (nor test.wikipedia), right? [18:49:07] greg-g: correct [18:49:17] why is that? [18:49:21] what's special about test2? [18:49:23] the code is in the client [18:49:27] ahhhhhhhh [18:49:28] client is only on test2 [18:49:34] our parser function [18:49:40] makes sense now, sorry [18:49:48] and an issue with a parser function can affect entire rendering of apage [18:50:02] is the client code being updated otherwise today? [18:50:16] is there no wikidata instance in beta labs? [18:50:19] hmmmm [18:50:26] ori-l: good question [18:50:27] ori-l: there is [18:50:30] :) [18:50:42] PROBLEM - Host vanadium is DOWN: PING CRITICAL - Packet loss = 100% [18:51:20] * aude can try to find a reviewer [18:52:14] heh, noc.wikimedia.org times out because it loads styles from svn.wikimedia.org, which is dead :P [18:52:19] * aude realizes it is tricky to update test wikdiata and not test2 [18:52:33] i should just get my patch reviewed [18:52:49] (03PS1) 10Andrew Bogott: Simplify our attempt to fix repo dir permissions. [operations/puppet] - 10https://gerrit.wikimedia.org/r/96802 [18:53:00] aude: right, and to rephrase again, the breakage in your parser will cause all of test2 to have issues, not just pages with testwikidata data? [18:53:25] only pages with the parser function and only if they reference a non-existing property [18:53:33] hi edsanders [18:53:40] edsanders, greg-g, Reedy - can we push the wmf4 deploy by a couple hours? [18:53:41] hi [18:53:57] doesn't affect lua [18:54:26] greg-g, it looks like VE-ers will be more reachable then in case things go wrong. [18:54:31] Eloquence: yeah, the only other thing is OAuth which should go out after this update, planned for 1, but can be pushed a little (chris s is on point) [18:54:40] RoanKattouw_away, Krinkle|detached and James_F|Away are catching up on jet lag and will awake in an hour or so [18:55:00] is that ok with you Reedy ? I know it's late for you [18:55:42] RECOVERY - Host vanadium is UP: PING OK - Packet loss = 0%, RTA = 0.49 ms [18:55:59] Yeah, fine [18:56:04] i'll fix noc.wm.o [18:56:06] Reedy: thanks man [18:56:09] thanks :) [18:56:12] ori-l: I'm fixing svn [18:56:19] greg-g: it looks like wikibase is on the old branch in wmf4 [18:56:23] paravoid will fix noc.wm.o [18:56:35] :P [18:56:43] <^d> What happened to svn? [18:56:49] the firewall [18:56:58] maybe in an hour, i can try to poke reviewers again or see if it gets merged [18:57:10] aude: ok, let me know [18:57:21] ok [18:57:23] is there a reason for noc.wm.o to point to svn.wm.o for css? [18:57:30] they should just live on the same host [18:57:39] ori-l: historical reasons presumably [18:57:40] it was probably an expedient hack [18:57:41] cod reuse! [18:57:41] yeah [18:57:42] clarify, is there a "good" reason? [18:57:53] <^d> code reuse is a great reason imho [18:57:57] :) [18:58:20] (03PS3) 10Dzahn: etherpad - tabbing, quoting & aligning [operations/puppet] - 10https://gerrit.wikimedia.org/r/96354 [18:58:42] (03PS1) 10Faidon Liambotis: Add firewall rules for SVN [operations/puppet] - 10https://gerrit.wikimedia.org/r/96803 [18:58:43] ^d: I prefer reusing cod [18:59:05] marktraceur: see above, we're delaying a bit until VE people get back online [18:59:08] (03CR) 10Faidon Liambotis: [C: 032] Add firewall rules for SVN [operations/puppet] - 10https://gerrit.wikimedia.org/r/96803 (owner: 10Faidon Liambotis) [18:59:11] <^d> ori-l: Anyway, no it shouldn't anymore. Either point it at git/gerrit if it must, or put the file on noc itself. [18:59:44] (03CR) 10Dzahn: etherpad - tabbing, quoting & aligning (032 comments) [operations/puppet] - 10https://gerrit.wikimedia.org/r/96354 (owner: 10Dzahn) [18:59:50] I saw [18:59:59] c'mon jenkins [19:00:03] marktraceur: just making sure/doing due diligence [19:00:16] <^d> paravoid: Did you do the secret dance and pray 9 times? [19:00:32] greg-g: Yup yup [19:00:41] greg-g: fabriceflorin incoming, at some point [19:00:47] So he can be in the loop [19:01:12] cmjohnson: a6, b5, c2 have PDU warnings [19:01:25] yes i know [19:01:28] k [19:01:51] can't do anything with a6 until we can move some of the mw servers to a new row [19:01:57] same with b5 [19:02:00] :( [19:02:11] c2 is odd [19:02:22] !log restored service for svn.wm.org [19:02:38] Logged the message, Master [19:02:57] Eloquence: ^^, should work now; ori-l is still working on the proper fix afaik [19:03:17] \o/ [19:03:20] thanks paravoid :) [19:04:04] so, opinions [19:04:14] check_job_queue is way too slow to be run from tampa [19:04:37] and it times out all the time [19:05:10] shall we fix the check to potentially be faster, or stop running the check from misc::maintenance hosts? [19:05:44] Aaron|home: ^^ [19:05:51] <^d> paravoid: There was a bug 57334 for svn, I've closed it for you. [19:05:55] yes, I see [19:06:03] ^d: oh I hadn't see that, thanks :) [19:06:10] Aaron|home: sorry [19:06:27] <^d> I think it should stop running on all misc::maintenance boxes. [19:06:35] <^d> It's currently running on arsenic, which is *wrong* [19:07:03] eww [19:07:17] check_job_queue should not run on fenari and hume anymore [19:07:22] right [19:07:23] make that role::applicationserver::maintenance [19:07:49] * Krinkle catches up. Ping me right away if you need anything. [19:09:07] somehow reduce this to 1 https://icinga.wikimedia.org/cgi-bin/icinga/status.cgi?search_string=check_job_queue [19:09:44] where the ones on arsenic and terbium are duplicate but work and the old ones havent been working for long [19:11:06] annnnd he detaches again :) [19:11:49] brb [19:15:04] Ryan_Lane: in trebuchet / puppet -- when I install my upstart script; how can I ensure that `initctl reload-configuration` has been run? [19:15:05] ... or is it better to just not use upstart [19:16:58] (03PS1) 10Faidon Liambotis: Run the check_job_queue check only on terbium [operations/puppet] - 10https://gerrit.wikimedia.org/r/96804 [19:18:10] (03CR) 10Faidon Liambotis: [C: 032] Run the check_job_queue check only on terbium [operations/puppet] - 10https://gerrit.wikimedia.org/r/96804 (owner: 10Faidon Liambotis) [19:20:36] guillom: en-planet.log:ERROR:planet.runner:Error 403 while updating feed http://www.gpaumier.org/blog/topic/wikimedia/feed/ [19:21:29] !log reedy synchronized php-1.23wmf5 'Staging code' [19:21:30] guillom: changes it to guillaumepaumier.com [19:21:44] Logged the message, Master [19:23:12] (03CR) 10Dzahn: "thanks, the duplicate checks have been bugging me for a while but i wasn't sure which approach to take about the roles and putting it in s" [operations/puppet] - 10https://gerrit.wikimedia.org/r/96804 (owner: 10Faidon Liambotis) [19:23:17] (03PS1) 10Reedy: Add new symlinks [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/96805 [19:23:38] (03CR) 10Reedy: [C: 032] Add new symlinks [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/96805 (owner: 10Reedy) [19:23:47] (03Merged) 10jenkins-bot: Add new symlinks [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/96805 (owner: 10Reedy) [19:24:29] !log reedy synchronized docroot and w [19:24:45] Logged the message, Master [19:24:47] Krinkle|detached: says you're detached, you available for the deploy? [19:25:02] mutante: I thought I had redirected the feeds [19:25:56] guillom: planet-venus says it gets 403 , maybe user agent? but i wouldnt worry too much, i'm fixing the URL in config [19:26:14] mutante: awesome; thank you :) [19:26:18] in browser i get redirected so i saw the new one [19:26:26] ok [19:26:44] The ExtensionMessages bug didn't appear this week [19:26:47] * Reedy grumbles [19:27:30] !log reedy Started syncing Wikimedia installation... : testwiki to 1.23wmf5 and build l10n cache [19:27:46] Logged the message, Master [19:28:08] Reedy: of course [19:29:27] apergos: Can you give me a 1-minute explanation about how the puppet freshness checks work and are reported? I'd like to do something similar in labs -- not IRC nagging but maybe a report about which instances are stale. [19:30:53] (03PS1) 10Ori.livneh: noc.wm.o: use local CSS files rather than point to svn.wm.o [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/96806 [19:31:03] andrewbogott: a puppet run triggers sending out UDP packet and as long as Icinga receives them it's OK, if they stop coming in it becomes CRIT.. (passive check) [19:31:19] monitor_service { .. passive = true ... [19:31:30] smpttrap [19:31:46] apergos: really? [19:31:50] yes really [19:32:08] (03CR) 10Ori.livneh: [C: 032] noc.wm.o: use local CSS files rather than point to svn.wm.o [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/96806 (owner: 10Ori.livneh) [19:32:13] you know this, we've discussed the oid and all [19:32:32] When you say 'a puppet run triggers' is that a custom feature we added or something that puppet always does and we're just currently ignoring on labs? [19:32:38] we added [19:32:39] apergos: for PDU monitoring [19:32:52] ok. I will dig through puppet code and see if I can understand… thanks. [19:32:53] !log ori updated /a/common to {{Gerrit|Ia451d47bd}}: noc.wm.o: use local CSS files rather than point to svn.wm.o [19:33:01] bbiaw [19:33:08] Logged the message, Master [19:33:15] I take it the puppetmaster itself has no idea if a given run succeeded or failed? [19:33:37] andrewbogott: it does [19:33:46] !log ori synchronized docroot/noc 'Ia451d47bd: noc.wm.o: use local CSS files rather than point to svn.wm.o' [19:33:49] the report framework has access to that [19:33:54] i've been meaning to do something with that [19:34:02] Logged the message, Master [19:34:04] ./files/snmp/snmptt.conf.icinga:EXEC /usr/lib/nagios/plugins/eventhandlers/submit_check_result $r "Puppet freshness" 0 "puppet ran at `date`" [19:34:13] if you have time, it'd be worth investigating that [19:34:20] Hm, ok… that might work well for me then, I'll be making my reports on the same system as the labs puppetmaster nayway. [19:34:27] neither does puppet know if the run was without errors [19:34:42] the notice is emitted upon run, not on 'successful run' [19:34:48] *the puppet client [19:35:22] Ah, so the freshness alerts are about puppet failing to run, not puppet failing to run cleanly? [19:35:23] the puppetmaster could parse the report if it received it, uck [19:35:26] right [19:35:36] failing to cache the catalog [19:35:40] or failing to run at all [19:35:52] as soon as we get passed the catalog being cached and into actually applying it, [19:35:57] Wasn't there a thing last month, though, where I broke a puppet dependency and it resulted in icinga flooding IRC? [19:35:59] we'll get notification [19:36:02] I thought that was because... [19:36:20] if you break it so that the catalog can't be cached [19:36:23] then that will do it [19:36:25] Oh, is that because I broke compilation rather than run? [19:36:29] Ah! Got it. [19:36:30] OK. Hm... [19:36:48] now you could organize your method in labs to run instages such that [19:36:58] the notification goes out as the last stage [19:37:13] Yeah, that's not crazy. [19:37:14] hmm I wonder how you can detect error notifications from earlier stages, I don't know [19:37:22] But maybe more than I want to bite off just now. [19:37:24] you would need that to make any difference [19:37:26] sure [19:37:34] you could at least detect the severe cases [19:37:45] So far my system involves asking people to run puppet and watch to see if it throws errors :) [19:38:01] Which presumably about 5% of volunteers are actually doing [19:38:25] :-D [19:38:28] yeah probably 1% [19:38:36] and about 0% are checking their cron jobs [19:39:40] I have a stupid little script I use for my error reporting... one thing it does as a stupid little script is to check the puppet logs on all the hosts (where salt is responsive :-P so there's a dependency) and tell me which ones have the last run as disabled or errs [19:40:00] but that would be a manual check across the instances [19:40:16] and assumes a salt master that can communicate with the instances [19:41:00] Hm… now that I think of it, maybe 'puppet running at all' is the thing I care about. Because if I add a new class that will still get implemented even if other things are failing. [19:41:05] Presuming compilation. [19:41:24] sure [19:41:36] well you can sure start with that and see if it meets your needs [19:41:45] * andrewbogott nods [19:42:24] so on the icinga end there's a file with the trap data written into a spool directory [19:42:59] and then these are processed by icinga which updates its 'time of last successful check' [19:43:03] https://icinga.wikimedia.org/cgi-bin/icinga/status.cgi?host=all&servicestatustypes=28&hoststatustypes=3&serviceprops=2097162&nostatusheader [19:43:05] all that is in puppet [19:43:06] down to 12 now [19:43:21] !log reedy Finished syncing Wikimedia installation... : testwiki to 1.23wmf5 and build l10n cache [19:43:21] good [19:43:33] 5 ps1 [19:43:36] Logged the message, Master [19:43:36] (03PS1) 10Ori.livneh: noc.wm.o: add images referenced by stylesheets. [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/96808 [19:43:46] Reedy: how's the speed relative to pre-'--compress'? [19:44:01] OK. So I want to gather the data but not have icinga actually send alerts. Will that still involve icinga, or is the data-gathering stage done by someone else? [19:44:19] you can get the traps into the directory without icinga [19:44:28] iirc [19:44:32] ok. [19:44:35] * andrewbogott reads puppet code [19:44:39] ori-l: 16 minutes or so [19:44:51] compared to? [19:44:52] yeah best to do that, my brain always gets a little fuzzy at this time of day [19:44:53] or is that the difference? [19:44:55] (clocked out) [19:45:03] (03CR) 10Ori.livneh: [C: 032] noc.wm.o: add images referenced by stylesheets. [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/96808 (owner: 10Ori.livneh) [19:45:25] !log ori updated /a/common to {{Gerrit|I7ea8302f2}}: noc.wm.o: add images referenced by stylesheets. [19:45:39] Logged the message, Master [19:45:40] Quick look back suggests 20-30 minutes for the last couple of weeks [19:46:24] ori-l: So presumably 10-15 minutes saved [19:46:43] !log ori synchronized docroot/noc 'I7ea8302f2: noc.wm.o: add images referenced by stylesheets.' [19:46:43] that's not bad [19:46:58] Indeed [19:46:59] Logged the message, Master [19:47:44] ori-l: Might be worth adding a start/stop time and duration to the end of the scap output to console [19:48:24] Reedy: better yet, to the !log msg [19:48:35] Yeah, true [19:48:42] Well, don't need the times so much there [19:48:46] As the SAL will have them [19:48:51] But the duration at the end would be good [19:48:56] (03PS1) 10Mwalker: OCG Collection [operations/puppet] - 10https://gerrit.wikimedia.org/r/96811 [19:50:13] test !== test2 [19:50:59] greg-g: So what exactly am I holding off doing? [19:51:24] Reedy: updating the wikipedias [19:51:37] Reedy: also, aude would like some time before updating testwikis [19:51:52] so, like everything [19:52:08] greg-g: if test2/test wikidata are kept on the current branch, then go ahead [19:52:14] i can update submodules if/when ready [19:52:42] !log Created BetaFeatures extension tables on all wikis [19:52:44] aude: k, just hopefully later today, as no deploys after today until Dec 2nd [19:52:46] Yayyy [19:52:52] greg-g: yeah [19:52:55] * greg-g nods [19:52:58] Logged the message, Master [19:55:22] 10.64.32.53 is having mass segfaults [19:55:27] 10.64.32.53 is having masses of segfaults [19:55:45] mw1183 [19:55:59] Reedy: depooled [19:56:00] ugh [19:56:01] !log mw1183/10.64.32.53 is having masses of segfaults [19:56:11] Thanks [19:56:17] Logged the message, Master [19:56:18] And they've stopped [19:56:22] how convienent! ;) [19:56:27] !log depooling mw1183 [19:56:32] bad memory probably [19:56:42] Logged the message, Master [19:57:27] First one looks to be at 15:23:09 [19:58:00] 55826 of them [19:58:18] marktraceur: Can we do your deploy while we're waiting for everyone else? :P [19:58:30] I have no problem with that :D [19:58:36] fabriceflorin, ^^ [19:58:40] (03PS1) 10Reedy: testwiki to 1.23wmf5 [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/96815 [19:58:45] Reedy: opened as RT #6365 [19:58:58] The only thing being will the wikipedias still being on 1.23wmf3 be a problem? [19:59:05] Uhhh [19:59:07] Yes! [19:59:08] I think so. [19:59:14] if so we can just 'wikipedia' => false for the moment [19:59:19] and deploy elsewhere [19:59:22] OK, no problemo [19:59:27] ugh [19:59:29] 'wiki' [19:59:48] paravoid: thanks [20:00:21] 33 Fatal error: ParserCache::getKey() [parsercache.getkey]: The script tried to execute a method or access a property of an incomplete object. Please ensure that the class definition "params" of the object you are trying to operate on was loaded _before_ unserialize() gets called or provide a __autoload() function to load the class definition in /usr/local/apache/common-local/php-1. [20:00:21] 23wmf3/includes/parser/ParserCache.php on line 139 [20:00:24] More trouble in paradise [20:00:58] Reedy: Related to what? [20:01:16] wmf3? [20:01:17] Just checking stack traces [20:02:45] https://bugzilla.wikimedia.org/show_bug.cgi?id=57370 [20:03:08] All seem to be from frwiki [20:03:21] And all from //fr.wikipedia.org/wiki/Théorème_des_accroissements_finis [20:03:29] mhoover: Your wifi, she is suckiiing! [20:03:29] Hrm [20:03:32] mhoover: all I was saying was… you can use 'iron' as the production bastion. [20:03:41] From there you should be able to ssh root@virt1000 [20:03:46] (03PS1) 10Aude: Enable quantity data type for test.wikidata [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/96817 [20:04:07] (03CR) 10Aude: [C: 04-1] "needs our branch to be deployed first" [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/96817 (owner: 10Aude) [20:04:10] ahh ok [20:04:12] We can also give you access to the mgmt console on those boxes if necessary… I don't know what's installed on them currently. [20:04:13] got it thank you [20:04:17] * aude would love a sticky -2 [20:04:30] heh [20:04:50] * aude not trusted to give myself -2 :D [20:05:09] trying that now [20:05:09] is that iron.wmflabs? [20:05:12] mhoover: right now there are only five designated boxes for labs in eqiad. I'm in the process of scaring up (hopefully) five more. Awkwardly the boxes we have now are misnumbered... [20:05:26] :) [20:05:38] Oh, no -- iron is in production. So, iron.wikimedia.org [20:06:25] mhoover: Has someone already walked you through the process of getting your key on production? [20:06:32] If not I can do that now... [20:06:33] hmm no [20:06:45] my key is in my prefs [20:06:55] but perhaps that is for the labs only [20:07:06] Yep, that's just labs. [20:07:39] ok, if you point me at some docs i can do it myself [20:07:42] mwalker: you should still have an upstart [20:07:43] Best if you create a new keypair just for production, and use a separate agent when you're using the real boxes. That way your production key won't bleed into labs which is modestly insecure. [20:07:55] (03PS1) 10Ori.livneh: scap: measure & output scap run time [operations/puppet] - 10https://gerrit.wikimedia.org/r/96822 [20:07:55] right [20:08:08] mwalker: you want to use checkout_module_calls [20:08:18] andrewbogott: mhoover: perhaps it'd be best if we moved to some random channel to avoid spamming -operations? [20:08:35] checkout_module_calls => { 'service.restart' => ['pdf-server'] } [20:08:46] Coren Ryan_Lane mhoover, join me in #wikimedia-labs-migration? [20:08:54] Reedy: ^ [20:08:56] mwalker: switch pdf-server with whatever your service name actually is [20:09:00] (03PS2) 10Mwalker: OCG Collection [operations/puppet] - 10https://gerrit.wikimedia.org/r/96811 [20:09:08] Reedy: if i merge that, could you rerun scap to verify? [20:09:29] it should be very fast with no actual changes to push [20:09:39] mwalker: checkout_module_calls lets you run any number of salt modules with any number of arguments [20:09:51] or you can write a custom module, then call that custom module from that [20:10:04] greg-g: i will be back in 30 min or so [20:10:24] meanwhile, looks like my patch will go in [20:10:49] (03PS3) 10Mwalker: OCG Collection [operations/puppet] - 10https://gerrit.wikimedia.org/r/96811 [20:10:56] Ryan_Lane: ^ so that's what I have right now [20:11:24] if that's all correct; I'm only missing one bit; and that's how to deploy the configuration file [20:11:44] you should likely have a configuration repo [20:11:54] this is what parsoid does [20:12:10] heh; configuration repo for one file [20:12:10] and they symlink the config file into the directory on the target side [20:12:21] yeah, it kind of sucks [20:12:55] though it also means you can deploy config separate from the code [20:13:17] *nods* what should that repo be called? and I can create a repo request for it [20:13:19] greg-g: would it be ok to run a no-op scap? [20:13:38] to verify https://gerrit.wikimedia.org/r/96822 [20:13:45] mwalker: it can be a local repo if you want [20:13:49] or you can have it in gerrit [20:14:01] on the deployment system it can be ocg/config [20:14:16] if you want it to be a local repo, then just don't specify an upstream [20:14:26] I'm guessing it would be better to have it in gerrit? [20:14:28] (03PS1) 10Dr0ptp4kt: Change order of extension vars. [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/96823 [20:14:32] (03CR) 10jenkins-bot: [V: 04-1] Change order of extension vars. [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/96823 (owner: 10Dr0ptp4kt) [20:14:41] yeah, unless you think there's going to be private stuff in it [20:15:06] * ori-l JFDI. [20:15:17] (03CR) 10Ori.livneh: [C: 032] scap: measure & output scap run time [operations/puppet] - 10https://gerrit.wikimedia.org/r/96822 (owner: 10Ori.livneh) [20:15:17] no private stuff unless we password protect our redis servers [20:15:29] * Ryan_Lane nods [20:15:39] worst case we make a private config file and git ignore it [20:15:47] then force add it locally when it needs to be updated [20:17:06] ori-l: not right now, things are primed [20:17:19] greg-g: k [20:17:43] ok; repo request in for operations/ocg-config [20:19:23] ori-l: cool though, that'll happen when reedy scaps soon, right? [20:20:15] greg-g: yes, and it'll work [20:20:20] cool [20:20:55] I already scapped.. [20:21:21] (03Abandoned) 10Dr0ptp4kt: Change order of extension vars. [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/96823 (owner: 10Dr0ptp4kt) [20:22:41] greg-g: ? [20:23:10] oh [20:23:16] just not the config change [20:23:23] 10.64.32.47 mw1177 is also segfaulting a lot [20:23:24] edsanders: Krinkle|detached how are you all? [20:23:30] Not as much as it's prior counterpart [20:24:10] Only 1658 today [20:24:19] !mw117 is also segfaulting quite a bit [20:24:19] Key was added [20:24:25] !mw117 del [20:24:26] Successfully removed mw117 [20:24:30] !log mw117 is also segfaulting quite a bit [20:24:38] Reedy: any objection to moving the scap start !log message to the actual start of scap, rather than after mw-update-l10n? [20:24:39] (03PS1) 10Dr0ptp4kt: W0 globals order. [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/96824 [20:24:46] Logged the message, Master [20:25:01] WFM [20:25:14] (03PS4) 10Mwalker: Initial Puppet Try for OCG::Collection Role [operations/puppet] - 10https://gerrit.wikimedia.org/r/96811 [20:25:16] Reedy: Did you figure out the frwiki problem? [20:25:21] Nope [20:25:25] actually, that will make it harder to assess the effect of --compress [20:25:34] so let's wait on that for a bit [20:25:45] Or just add an earlier message [20:26:21] marktraceur: I'm guessing that the objects of that page are corrupt/similar in the parser cache [20:26:31] (03PS5) 10Mwalker: Initial Puppet Try for OCG::Collection Role [operations/puppet] - 10https://gerrit.wikimedia.org/r/96811 [20:26:43] 'kay [20:26:57] As long as it's nothing too complex [20:27:10] Or, you know, related to what we're trying to push out [20:28:12] (03CR) 10jenkins-bot: [V: 04-1] Initial Puppet Try for OCG::Collection Role [operations/puppet] - 10https://gerrit.wikimedia.org/r/96811 (owner: 10Mwalker) [20:28:59] (03PS3) 10Dr0ptp4kt: WIP: DO NOT MERGE YET. Apply FlaggedRevs to metawiki for W0. [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/95662 [20:29:00] (03CR) 10jenkins-bot: [V: 04-1] Initial Puppet Try for OCG::Collection Role [operations/puppet] - 10https://gerrit.wikimedia.org/r/96811 (owner: 10Mwalker) [20:29:07] Reedy: oh, I nearly forgot: would you be able to merge https://gerrit.wikimedia.org/r/#/c/96718/ or can you do that tomorrow (bloody Fridays)? [20:31:04] (03PS1) 10Ori.livneh: slight tweak to format of scap duration text message [operations/puppet] - 10https://gerrit.wikimedia.org/r/96831 [20:31:48] (03CR) 10Reedy: [C: 032] testwiki to 1.23wmf5 [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/96815 (owner: 10Reedy) [20:31:52] marktraceur: There's no reason to believe it's anything to do with that [20:32:51] (03CR) 10Ori.livneh: [C: 032] slight tweak to format of scap duration text message [operations/puppet] - 10https://gerrit.wikimedia.org/r/96831 (owner: 10Ori.livneh) [20:32:58] (03Merged) 10jenkins-bot: testwiki to 1.23wmf5 [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/96815 (owner: 10Reedy) [20:33:22] (03PS6) 10Mwalker: Initial Puppet Try for OCG::Collection Role [operations/puppet] - 10https://gerrit.wikimedia.org/r/96811 [20:34:22] marktraceur: What's next then? [20:35:40] Reedy: What do you mean? You enabled the extensions in the config? [20:38:07] csteipp: just fyi, you might be delayed today for oauth, we're waiting on someone from the VE team to get online before pushing that out (ie: updating wikipedias to wmf4) [20:40:10] might? [20:40:10] haha [20:43:49] right [20:43:54] csteipp: just fyi, you will be delayed today for oauth, we're waiting on someone from the VE team to get online before pushing that out (ie: updating wikipedias to wmf4) [20:44:01] :) [20:45:57] Reedy: Did you need direction from me still? [20:46:02] If so, where are you now? [20:46:45] I've no idea what I'm doing [20:46:55] Deploy Beta Features on all wikis worldwide: [20:46:55] * Beta Features [20:46:55] * Media Viewer [20:46:55] * CommonsMetaData [20:46:55] * VisualEditor Formulae [20:46:56] * Typography (VectorBeta) [20:46:59] * VisualEditor Language Editing [20:47:01] * Nearby [20:47:03] This is a platform train deployment, which Greg and Reedy will oversee. Multimedia, Design and other teams will need to test it as soon as it's live, then announce and socialize the release -- and hear what our users think. :) [20:47:07] The first 3 are extensions [20:47:10] What the other 4? [20:47:29] Hah, cool. [20:47:31] VE is just VE [20:47:40] VectorBeta is the only other extension you need to enable [20:47:43] !log sq44 powering down for decom [20:47:44] Reedy: ^^ [20:47:48] greg-g: So delayed hours? Or until monday? [20:47:54] csteipp: hours, hopefully [20:47:59] Logged the message, Master [20:48:09] How does that work? [20:48:24] BetaFeatures, MediaViewer and CommonsMetaData aren't enablede on all wikis [20:48:37] Reedy: That's what we were planning on doing today. [20:48:37] Reedy: that's the point, they want them to be now [20:49:09] sorry, I tried to get a more concise clear explanation in that thread :/ [20:49:32] (03PS1) 10Yurik: Removed ZERO namespace protection in META [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/96873 [20:57:40] !log sq48 powering down for decom [20:57:48] greg-g: Reedy we are ready to update test wikidata and test2, soon as i make submodules patch [20:57:56] Logged the message, Master [20:58:06] aude: whew, nick o time [20:58:18] :D [20:59:56] !log reedy updated /a/common to {{Gerrit|I6433f2b0d}}: testwiki to 1.23wmf5 [20:59:59] (03PS1) 10Reedy: Enable BetaFeatures everywhere except Wikipedias [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/96876 [21:00:10] Logged the message, Master [21:00:46] (03CR) 10jenkins-bot: [V: 04-1] Removed ZERO namespace protection in META [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/96873 (owner: 10Yurik) [21:01:57] https://gerrit.wikimedia.org/r/#/c/96877/ [21:01:58] yay there goes sq48 [21:01:59] Reedy: ^ [21:02:06] soon to be off my trouble list! [21:03:03] (03CR) 10Reedy: [C: 032] Enable BetaFeatures everywhere except Wikipedias [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/96876 (owner: 10Reedy) [21:03:15] yay, beta features :) [21:07:03] what the hell is jenkins upto now? [21:07:16] hey, can anybody unban all yurik's (yurik / yurik_) ? his irc client went bananas. he's on ip 50.74.69.177 [21:08:08] dr0ptp4kt: done [21:08:34] thanks paravoid [21:09:36] (03PS1) 10Cmjohnson: decom'ing sq44 and 48 -removed from dsh, dhcpd, role.pp and added to decom.pp [operations/puppet] - 10https://gerrit.wikimedia.org/r/96878 [21:09:49] dr0ptp4kt: got a minute to talk about X-F-B? [21:10:03] paravoid, yes. [21:10:18] (03Merged) 10jenkins-bot: Enable BetaFeatures everywhere except Wikipedias [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/96876 (owner: 10Reedy) [21:10:26] Wooo [21:10:52] could you clarify the numbers a bit? [21:12:01] paravoid, sure. can you hop on gchat for a moment so we can discuss verbally? [21:12:06] (google hangout) [21:12:22] sorry, can we do this over irc? [21:12:39] sure. sorry in advance if any confusion, in that case :) [21:12:48] what's up? [21:13:35] Holding other factors constant, my log analysis suggests something like an additional 55,000 distinct origin hits required for each additional proxy in the X-Forwarded-By field in order to fully populate the caches to account for the new varying field. There are somewhere between 50,000-60,000 cache hit/200 distinct URLs from review of a full day of a recent W0 log [21:14:03] have you kept perhaps the greps you did? [21:14:17] Variations on this theme: https://gist.github.com/anonymous/7589732 [21:14:18] (03CR) 10Cmjohnson: [C: 032 V: 032] decom'ing sq44 and 48 -removed from dsh, dhcpd, role.pp and added to decom.pp [operations/puppet] - 10https://gerrit.wikimedia.org/r/96878 (owner: 10Cmjohnson) [21:14:32] oh, nice [21:16:48] !log reedy synchronized php-1.23wmf5/extensions/Wikibase [21:17:03] Logged the message, Master [21:17:10] and what is the 50-60k figure? [21:17:15] paravoid, you can toggle the ! for the preg_match on zero= to make it check against zero-tagged traffic, "obviously" [21:18:02] (03PS1) 10Cmjohnson: Removing all dns entries for sq44 and sq48 [operations/dns] - 10https://gerrit.wikimedia.org/r/96879 [21:18:04] (air quotes) [21:18:12] heh [21:18:14] !log reedy synchronized wmf-config/InitialiseSettings.php 'Enable BetaFeatures on all non wikipedias' [21:18:17] (03CR) 10jenkins-bot: [V: 04-1] Removing all dns entries for sq44 and sq48 [operations/dns] - 10https://gerrit.wikimedia.org/r/96879 (owner: 10Cmjohnson) [21:18:29] Logged the message, Master [21:18:38] Doing anything else now is going to upset APC [21:18:45] It doesn't like 3 versions of MW concurrently... [21:18:48] Oof, looks like the CSS didn't go [21:19:15] Didn't go how so? [21:19:20] it represents the places where there would be cache degradation. if a url didn't have a cache hit, chances are pretty good that something about it makes caching hard or unlikely. basic assumption, and i can think of counterarguments against that basic premise, like in a multiday set, for example... [21:19:41] The betafeatures preference tab on wikibooks looks uuuugly [21:19:51] greg-g, can i do a setting push at some point? [21:20:00] minor oneliner adjustment [21:20:10] https://en.wikibooks.org/w/index.php?title=Special:Preferences#mw-prefsection-betafeatures [21:20:11] yurik-road: not right now, and what does -road mean? [21:20:21] yurik-road: what is it? [21:20:25] yurik-road: can you do LD? [21:20:37] no beta features on wikidata? [21:20:43] event.returnValue is deprecated. Please use the standard event.preventDefault() instead. load.php?debug=false&lang=en&modules=jquery%2Cmediawiki%2CSpinner%7Cjquery.triggerQueueCallback%2Cl…:48 [21:20:44] (03Abandoned) 10Cmjohnson: Removing all dns entries for sq44 and sq48 [operations/dns] - 10https://gerrit.wikimedia.org/r/96879 (owner: 10Cmjohnson) [21:21:01] https://bits.wikimedia.org/en.wikivoyage.org/load.php?debug=false&lang=en&modules=jquery%2Cmediawiki%2CSpinner%7Cjquery.triggerQueueCallback%2CloadingSpinner%2CmwEmbedUtil%7Cmw.MwEmbedSupport&only=scripts&skin=vector&version=20131121T024736Z [21:21:05] aude: I don't see why it wouldn't be, but I don't know how wikidata is configured [21:21:05] paravoid, here's another gist with the same general principle at work. you can just run these things, commenting and uncommenting, against zero.tsv.log and sampled-1000 files to get a feel for unique URLs, unique URL-carrier pairs, etc. https://gist.github.com/anonymous/7589841 [21:21:12] purge all the things! [21:21:17] :) [21:21:29] I should write a script to do the JS touching [21:21:48] greg-g - road is because i am on the laptop, to differentiate from the home one (instead of yurik_ or some other ones, and it has a cloak - safer from identity perspective). The patch is https://gerrit.wikimedia.org/r/#/c/96873/1/wmf-config/InitialiseSettings.php [21:21:53] Reedy: Better, thanks [21:22:02] I've not do anything yet [21:22:05] could be 'wiki' => false, [21:22:20] paravoid, also scroll up for message "it represents the places..." in case you didn't see that response [21:22:21] i suppose we can wait until dec 2 [21:22:28] dr0ptp4kt: looking [21:22:30] aude: It should be later tonight [21:22:34] Reedy: ok [21:22:44] dr0ptp4kt: where are you running these? [21:22:46] The issue was wikipedias on 1.23wmf3 which the extensions are a bit out of date [21:22:52] k [21:22:55] which one of all of the log boxes that we have? [21:24:45] yurik-road: excuse my ignorance, what does that do? [21:24:53] the patch [21:25:39] (03PS1) 10Jgreen: add deploy redis module to ocg test server [operations/puppet] - 10https://gerrit.wikimedia.org/r/96881 [21:26:23] !log reedy updated /a/common to {{Gerrit|I4360e8a34}}: Enable BetaFeatures everywhere except Wikipedias [21:26:37] Logged the message, Master [21:26:38] greg-g, we currently protect meta's ZERO: namespace from editing by anyone but us. About a month ago I checked in some code do custom user right checking inside the extension, and couldnt figure out why it wasn't working - all because we had this blanket ban on editing. Once removed, my code will do the actual checking. It was reviewed by Doug_Weller [21:26:49] could I get a few eyeballs on the redis commit I just did? I'd like to make sure I'm doing sane things there [21:26:55] (03PS1) 10Reedy: Wikipedias to 1.23wmf4 [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/96883 [21:26:56] (03PS1) 10Reedy: phase1 wikis to 1.23wmf5 [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/96884 [21:27:01] (03PS1) 10Mhoover: Adding Mike Hoover [operations/puppet] - 10https://gerrit.wikimedia.org/r/96885 [21:27:19] greg-g, weird autocorrect - it was reviewed by csteip [21:28:10] (03PS2) 10Reedy: Removed ZERO namespace protection in META [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/96873 (owner: 10Yurik) [21:28:45] paravoid, some logs to look at are stat1002:/a/squid/archive/sampled/sampled-1000.tsv.log-20131118.gz and stat1001:/a/squid/archive/zero/zero.tsv.log-20131114.gz (gunzip them) [21:28:57] oops, i meant stat1002 there on the second example [21:29:45] yurik-road: gotcha [21:29:47] Reedy, you want to deploy it? [21:29:49] yeah, during LD [21:31:32] well, if Reedy wants to, thats fine, he'll just do a quais-continuous deploy thing today :/ [21:32:02] greg-g, can't today during LD - 7pm here and dr0ptp4kt will go home (he's in NYC too) [21:32:15] either earlier or another day [21:33:06] (03PS1) 10Andrew Bogott: Give mhoover root on a bunch of labs hosts [operations/puppet] - 10https://gerrit.wikimedia.org/r/96887 [21:33:20] (03CR) 10Jgreen: [C: 032 V: 031] add deploy redis module to ocg test server [operations/puppet] - 10https://gerrit.wikimedia.org/r/96881 (owner: 10Jgreen) [21:34:04] yurik-road: another day is >= Dec 2nd, just fyi, nothing goes out next wseek [21:34:08] -s [21:36:07] (03PS1) 10Jgreen: add standard to ocg base server [operations/puppet] - 10https://gerrit.wikimedia.org/r/96888 [21:38:10] (03PS2) 10coren: Adding Mike Hoover [operations/puppet] - 10https://gerrit.wikimedia.org/r/96885 (owner: 10Mhoover) [21:38:18] (03CR) 10coren: [C: 032] "It is he!" [operations/puppet] - 10https://gerrit.wikimedia.org/r/96885 (owner: 10Mhoover) [21:38:23] :) [21:38:47] (03CR) 10Jgreen: [C: 032 V: 031] add standard to ocg base server [operations/puppet] - 10https://gerrit.wikimedia.org/r/96888 (owner: 10Jgreen) [21:39:40] (03PS1) 10CSteipp: Enable OAuth on all wikis [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/96890 [21:40:57] Reedy: For some reason BF, MMV, CMD are no longer on the sites it was on (Meta, Commons, mw.org) [21:41:08] they were on* [21:41:18] I thought the config change looked OK but I guess not [21:41:56] :( [21:42:15] lol [21:42:28] They must be in the special "wiki" group [21:43:57] The easiest fix of course, is being to upgrade the Wikipedias... [21:44:04] right... [21:44:15] wiki... [21:44:16] Krinkle|detached: James_F|Away RoanKattouw_away anyone awake yet? [21:44:30] i think wiki=false includes commonswiki metawiki etc [21:44:38] mediawikiwiki wikidatawiki [21:44:40] Hmm [21:44:47] * aude thinks so [21:44:53] Does 'wikipedia' work now? [21:45:02] * Reedy tires it [21:45:19] I seem to recall liagent making a change... [21:46:04] Here goes nothing [21:46:21] (03PS1) 10Ori.livneh: log scap timing to graphite [operations/puppet] - 10https://gerrit.wikimedia.org/r/96891 [21:46:22] * marktraceur sighs [21:46:24] !log reedy synchronized wmf-config/InitialiseSettings.php 'wiki -' [21:46:40] Logged the message, Master [21:46:48] it works! [21:46:50] (03PS2) 10coren: Give mhoover root on a bunch of labs hosts [operations/puppet] - 10https://gerrit.wikimedia.org/r/96887 (owner: 10Andrew Bogott) [21:46:51] Nice [21:46:52] * aude sees beta on wikidata [21:47:00] Ahhhh [21:47:01] https://en.wikipedia.org/wiki/Main_Page [21:47:02] Nooooo [21:47:10] Reedy: Ctrl Z ctrl z [21:47:22] too late [21:47:24] Reverting though [21:47:33] huh [21:47:33] (03CR) 10coren: [C: 032] "WMF." [operations/puppet] - 10https://gerrit.wikimedia.org/r/96887 (owner: 10Andrew Bogott) [21:47:43] !log reedy synchronized wmf-config/InitialiseSettings.php 'rv' [21:47:58] Logged the message, Master [21:48:38] (03CR) 10Chad: [C: 031] Graph components of Elasticsearch health [operations/puppet] - 10https://gerrit.wikimedia.org/r/96796 (owner: 10Manybubbles) [21:49:19] ugh [21:49:29] so, what the hell [21:49:36] is't almost 2, nothing from VE? [21:49:50] greg-g: Want I should call James_F|Away or RoanKattouw_away ? [21:50:19] marktraceur: yes [21:50:24] * Elitre wants her Beta link on it.wp or she'll cry. [21:50:46] <^d> greg-g: Is it cool if I sync two changes to PoolCounter conf? It's all for Cirrus, so it's no-op at the moment, but I'd like it in place well before we start ramping up again. [21:50:46] yeah [21:50:47] Elitre: Honestly as long as it doesn't cause issues on wmf3 I'm cool with pushing it pre-wmf4 [21:50:57] ^d: can you do it in LD? [21:51:01] <^d> I can, sure :) [21:51:02] I just want to keep things clear right now [21:51:03] Apparently the code is there, I just forgot it was [21:51:04] sorry [21:51:15] ^d: things are liable to get real confusing real quick [21:51:19] (03CR) 10Chad: [C: 031] "Will do this during today's LD." [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/96064 (owner: 10Manybubbles) [21:51:49] greg-g: Does ori-l's yes extend to you? [21:51:56] * marktraceur got confused...real quick [21:52:02] marktraceur: in that moment, ori was my surrogate [21:52:16] ori-l: You had so much power and you squandered it on one word [21:52:38] marktraceur: yeah, please call [21:52:52] <^d> He's on the phone now, I see. [21:53:34] cool [21:53:34] oh, I'll get it when it's time. Just hope to see it live before I go to sleep so I can alert you if something has gone horribly wrong :p [21:53:35] <^d> marktraceur: YOU WERE SUPPOSED TO CALL VE TEAM, NOT ORDER PIZZA! [21:53:50] greg-g: James_F|Away is in London apparently but has sent an email sorting it out? [21:53:56] oh? [21:54:08] I see no email from James_F|Away [21:54:16] * greg-g offlineimaps agin [21:54:36] there it is [21:54:53] Huzzah [21:54:53] well, if by asking a question he means 'sorting it out'... [21:55:07] ...I could also call RoanKattouw_away but he may be similarly in a different country [21:57:02] (03CR) 10BryanDavis: [C: 031] log scap timing to graphite (031 comment) [operations/puppet] - 10https://gerrit.wikimedia.org/r/96891 (owner: 10Ori.livneh) [21:58:29] greg-g: It's a last-ditch attempt, but mooeypoo is online [21:58:39] can they diagnose/fix issues with VE? [21:58:45] fixing being the operative word [21:58:50] And InezK [21:58:52] Well [21:59:05] They have different areas of expertise, but they have experience with the codebase [21:59:17] InezK practically wrote an entire subsystem on his own [21:59:46] I'm not comfortable calling on him to be around during our deploys [21:59:52] 'kay [22:00:06] mooeypoo is mostly involved in language support for VE [22:00:10] So it may not be enough [22:00:12] right [22:00:36] I'll assume unless I get one of roan, timo, or james online in here I'll prevent VE from going [22:01:07] Does that mean stopping wmf4 until a week from Tuesday? [22:01:43] Krinkle|detached: Are you not still around? [22:01:58] mark, they're not here [22:02:07] we have to move on [22:02:13] (03CR) 10Bsitu: [C: 031] enable Echo on all beta.wmflabs.org-wikis [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/95450 (owner: 10Umherirrender) [22:02:14] ori-l: Figured it was worth a shot :) [22:02:14] contradictory reports of sleeping or on plane [22:06:32] alright, at this point, since this is holding up the rest of the day, and the rest of my work day (still haven't done my work for today because of this).... [22:07:05] I was going to ask Reedy to be around to revert if needed, but it's really late his time [22:07:16] it's 10pm [22:07:23] I was still awake at 6am this morning [22:07:49] hah, well [22:08:20] alright, stop the blockage, push it through, revert as quickly as possible without asking questions if something arises. marktraceur can you ping mooey to watch the relevant places for issues? [22:08:39] Can do [22:08:50] Reedy: ^ [22:08:55] (03PS2) 10CSteipp: Enable OAuth on all wikis [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/96890 [22:09:29] {{done}} in -visualeditor [22:09:38] uhm, my OTR doesn't seem to work. [22:09:50] (03PS1) 10Jgreen: add role::ocg::test to rhodium [operations/puppet] - 10https://gerrit.wikimedia.org/r/96895 [22:10:02] Elitre: yeah, no idea, was seeing weird messages [22:10:23] (03CR) 10jenkins-bot: [V: 04-1] Enable OAuth on all wikis [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/96890 (owner: 10CSteipp) [22:10:31] Thanks mooeypoo [22:10:58] aye, np. I'm hoping I can help. [22:11:31] is there a problem with VE? [22:11:42] Not yet [22:11:52] We just want to be equipped to deal with anything that crops up [22:12:57] (03PS2) 10Ori.livneh: log scap timing to graphite; parametrize statsd host/port [operations/puppet] - 10https://gerrit.wikimedia.org/r/96891 [22:13:07] (03CR) 10Jgreen: [C: 032 V: 031] add role::ocg::test to rhodium [operations/puppet] - 10https://gerrit.wikimedia.org/r/96895 (owner: 10Jgreen) [22:13:36] paravoid: you mentioned you have a node 0.10 package somewhere? [22:13:46] it's behind his desk [22:13:50] (03PS3) 10CSteipp: Enable OAuth on all wikis [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/96890 [22:13:51] j/k [22:13:53] heh [22:14:12] ori-l: npm wouldn't take it so we sent it through UPS instead [22:15:24] mwalker: I'd guess on zirconium (from https://wikitech.wikimedia.org/wiki/Etherpad) [22:16:10] Reedy: can you do that? [22:18:29] (03CR) 10Reedy: [C: 032] Wikipedias to 1.23wmf4 [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/96883 (owner: 10Reedy) [22:18:39] thanks [22:18:41] (03CR) 10jenkins-bot: [V: 04-1] Wikipedias to 1.23wmf4 [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/96883 (owner: 10Reedy) [22:18:43] heh [22:19:03] Jenkins is like "Nope, you're totally not ready for this" [22:19:29] (03CR) 10BryanDavis: [C: 04-1] "Sneaky whitespace is sneaky." (032 comments) [operations/puppet] - 10https://gerrit.wikimedia.org/r/96891 (owner: 10Ori.livneh) [22:20:07] (03CR) 10Reedy: [V: 032] "STFU jenkins" [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/96883 (owner: 10Reedy) [22:20:23] Hey who fixed grrrit-wm so it doesn't just say "(2 comments)"? Give that person a cupcake! [22:20:54] * marktraceur raises hand [22:21:07] !log reedy rebuilt wikiversions.cdb and synchronized wikiversions files: Wikipedias to 1.23wmf4 [22:21:21] Logged the message, Master [22:21:25] (03PS3) 10Chad: Add icinga monitoring for Gerrit and Gitblit [operations/puppet] - 10https://gerrit.wikimedia.org/r/75777 [22:21:31] (03CR) 10Chad: Add icinga monitoring for Gerrit and Gitblit (036 comments) [operations/puppet] - 10https://gerrit.wikimedia.org/r/75777 (owner: 10Chad) [22:21:44] I'll be back in a bit, I need to go take something for this headache and do something else for work other than this. ping if needed [22:21:57] bd808: I'll take a box of oreos instead [22:22:13] oh man oreos and milk would be great for this headache [22:22:15] Golden ones if you think of it [22:22:28] golden oreos? [22:22:31] Yeah [22:22:32] what is this blasphemy [22:22:34] marktraceur: I was all ready to send you some until you said that [22:22:35] They delicious [22:22:43] they not oreos, they're cookies. [22:22:48] And also… NOT oreos [22:22:52] and they're not golden :P [22:22:53] they're not delicious at all [22:23:03] they're just normal sugar cookies with frosting [22:23:15] -2 points for Griffendor [22:23:19] haha [22:23:27] I didn't realize you guys took your cookies so seriously [22:23:29] Christ [22:23:56] LeslieCarr: Coincidentally frosted sugar cookies from e.g. Safeway are like my favourite. [22:24:06] well, not all cookies [22:24:16] * bd808 sends marktraceur some cookies he likes that I would never eat [22:24:31] though I learned something new yesteday. I got myself gingersnap cookies... they're *spicy!* [22:24:37] who the heck thought about making a spicy cookie [22:25:20] Homemade gingersnaps rank high in my cookie pantheon [22:25:39] they are pretty good, but the fact they are spicy confuses me to no end. [22:26:06] mooeypoo: https://en.wikipedia.org/wiki/Ginger_snaps says the UK and maybe Scandinavia [22:26:12] A little below mexican wedding cakes and oatmeal chocolate chip [22:26:16] Which is hilarious because no other food from there has spice [22:26:24] James_F: Welcome back, now we're definitely set [22:26:30] Oh and Krinkle too [22:26:31] Wow [22:26:33] heya James_F [22:26:34] Great job guys [22:26:38] wait, that's it? [22:26:39] it's over? [22:27:29] Hello. [22:27:46] But I was all ready and stuff. [22:28:02] * James_F was too. Four hours ago, at the normal time. [22:28:08] mooeypoo: We'll still call you if we need it :) [22:28:19] (03CR) 10Aaron Schulz: [C: 031] "As long as this gets moved to meta soon." [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/96890 (owner: 10CSteipp) [22:28:43] I will be happy to help, but I feel a lot better to have James_F and Krinkle around [22:28:53] mooeypoo: Of course. :-) [22:31:12] !log reedy synchronized php-1.23wmf5/extensions/MobileFrontend/ [22:31:27] Logged the message, Master [22:31:42] (03CR) 10Reedy: [C: 032] phase1 wikis to 1.23wmf5 [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/96884 (owner: 10Reedy) [22:31:52] (03Merged) 10jenkins-bot: phase1 wikis to 1.23wmf5 [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/96884 (owner: 10Reedy) [22:32:34] Ah [22:32:36] !log reedy rebuilt wikiversions.cdb and synchronized wikiversions files: phase1 wikis to 1.23wmf5 [22:32:39] I hope that's 1-indexed. [22:32:49] andrewbogott: when you have time, please merge my patch [22:32:52] Logged the message, Master [22:33:51] marktraceur: Yeah, there's debate about whether they're meant to be 1-indexed or 0-indexed. [22:34:02] marktraceur: Which is totally not confusing. :-) [22:34:05] Nope [22:34:09] * marktraceur starts 2-indexing it [22:34:14] Or no [22:34:19] I'll start doing them in reverse [22:34:54] (03CR) 10Aude: [C: 031] "this is ready" [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/96817 (owner: 10Aude) [22:35:01] Reedy: ^ [22:36:31] This is Reedy* FTFY [22:38:47] boy SqlBagOStuff really took a shit on dberror.log [22:39:13] maybe errors should be truncated when sent from MW [22:39:28] ori-l: maybe :) [22:42:39] bd808: have you attempted to find a way to use something other than the environment for configuration? [22:43:06] jeremyb: I actually use env on purpose. Is that a bad thing? [22:43:29] It could easily read a config file as an alternate method [22:43:47] I'm kind of a fan of 12factors style applications [22:43:59] matanya: Bah, sorry, I had a second comment but I must not have saved it. We're defining those directories twice now, aren't we? I think the original mkdir was essentially superfluous. [22:44:23] bd808: i don't care so much about env or not env. i do care about it being in apache conf [22:44:46] andrewbogott: honstly, i'm confused now, mind looking at recent change and comment? [22:44:56] ok [22:45:06] bd808: AFAIK, nothing else (at least on the main cluster) uses apache conf for app conf [22:45:18] zhwiki SqlBagOStuff::set: Transaction already in progress (from SqlBagOStuff::getMulti), performing implicit commit! [22:45:28] how does that end up in a trx at all? [22:45:29] jeremyb: Ok. It's easily fixable at this point [22:45:35] LeslieCarr: ipv6 don't work for dickson. not a dns issue unless you have the wrong AAAA in wikimedia.org zone [22:45:36] gah, so much broken stuff [22:45:38] PING dickson.wikimedia.org(2620:0:861:52:208:80:155:68) 56 data bytes [22:45:41] From xe-5-3-3-500.cr1-eqiad.wikimedia.org icmp_seq=1 Destination unreachable: Address unreachable [22:45:52] (just reverified that) [22:46:18] (03CR) 10Andrew Bogott: mail.pp: change exec to file (032 comments) [operations/puppet] - 10https://gerrit.wikimedia.org/r/91573 (owner: 10Matanya) [22:46:24] but it's not arp'ing [22:46:30] does dickson have that ipv6 addy ? [22:46:40] well neighbor discovery'ing technically [22:46:46] stupid ipv6 terms [22:46:59] LeslieCarr: are you talking about https://rt.wikimedia.org/Ticket/Display.html?id=6326 ? [22:47:06] yes [22:47:16] LeslieCarr: oh, so maybe it's a conf issue on that box [22:47:17] i double checked i actually had set up ipv6 a while ago and forgot [22:47:38] think so, i see a neighbor with the same beginning and then what looks like a mac address-based address [22:47:38] LeslieCarr: it's a linked ticket! just look at the links section! [22:47:46] :) [22:47:49] so i am guessing it's config on the box [22:47:50] hehehe [22:47:57] dude with all the rfp's, my brain is shot [22:48:00] and so are my wrists! [22:48:16] mutante: so do you have shell yet on dickson? want to check that out? [22:50:06] LeslieCarr: you shot up your wrists? :p [22:50:32] yep :p [22:50:35] whatever helps you relax [22:50:42] it was the only way to deal with reading 50+ proposals ;) [22:51:12] just don't OD or won't have any ops team to pester in person anymore [22:51:19] (03CR) 10CSteipp: Mobile m. and zero. landing page redirect handling by ZERO (031 comment) [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/96654 (owner: 10Yurik) [22:53:05] aude: https://gerrit.wikimedia.org/r/#/c/96817/1/wmf-config/Wikibase.php,unified [22:53:08] That can't be right? [22:54:12] looking [22:54:17] PROBLEM - Disk space on xenon is CRITICAL: DISK CRITICAL - free space: / 0 MB (0% inode=96%): [22:54:23] (03PS8) 10Matanya: mail.pp: change exec to file [operations/puppet] - 10https://gerrit.wikimedia.org/r/91573 [22:55:03] Reedy: testwikidata is getting the default wikibase settings [22:55:14] we are overriding everything else for now [22:55:14] (03PS1) 10Bsitu: Schema:Echo has outdated revision id [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/96901 [22:56:55] (03PS2) 10Reedy: Enable quantity data type for test.wikidata [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/96817 (owner: 10Aude) [22:57:04] (03CR) 10Reedy: [C: 032] Enable quantity data type for test.wikidata [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/96817 (owner: 10Aude) [22:57:08] Woo, yay. [22:57:10] :) [22:57:22] (03Merged) 10jenkins-bot: Enable quantity data type for test.wikidata [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/96817 (owner: 10Aude) [22:57:26] quantities already are enabled [22:57:40] it just means when we update wikidata etc. they won't get them prematurely [22:57:45] * James_F nods. [22:57:48] andrewbogott: lets hope this^ is last try :) [22:58:33] (03PS4) 10Reedy: Mobile m. and zero. landing page redirect handling by ZERO [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/96654 (owner: 10Yurik) [22:58:42] (03CR) 10Reedy: [C: 032] Mobile m. and zero. landing page redirect handling by ZERO [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/96654 (owner: 10Yurik) [22:59:00] (03Merged) 10jenkins-bot: Mobile m. and zero. landing page redirect handling by ZERO [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/96654 (owner: 10Yurik) [22:59:41] Crap [23:00:28] Reedy: What's up? [23:00:37] (03CR) 10Reedy: "Probably shouldn't have merged this, but as it's currently unused (ie not symlinked into any docroots) I'm going to leave it in" [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/96654 (owner: 10Yurik) [23:00:50] Whew. [23:00:55] (03PS3) 10Reedy: Removed ZERO namespace protection in META [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/96873 (owner: 10Yurik) [23:00:59] (03CR) 10Reedy: [C: 032] Removed ZERO namespace protection in META [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/96873 (owner: 10Yurik) [23:01:13] (03Merged) 10jenkins-bot: Removed ZERO namespace protection in META [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/96873 (owner: 10Yurik) [23:02:13] (03CR) 10DarTar: [C: 031] Schema:Echo has outdated revision id [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/96901 (owner: 10Bsitu) [23:02:40] !log reedy synchronized wmf-config 'last few config updates' [23:02:56] Logged the message, Master [23:03:03] jeremyb: i did a little while ago and it worked [23:03:20] !log reedy updated /a/common to {{Gerrit|I3ac8621d9}}: Removed ZERO namespace protection in META [23:03:25] (03PS1) 10Reedy: Enable BetaFeatures everywhere [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/96903 [23:03:35] Logged the message, Master [23:04:50] (03CR) 10Reedy: [C: 032] Enable BetaFeatures everywhere [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/96903 (owner: 10Reedy) [23:05:01] (03Merged) 10jenkins-bot: Enable BetaFeatures everywhere [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/96903 (owner: 10Reedy) [23:05:18] Just a heads up to Opsen: the improved version of the CIDR patch for $wgSquidServersNoPurge/wfIsConfiguredProxy() is in 1.23wmf4 which is now on enwiki [23:05:46] So if load starts spiking on the app servers that is one possible/probably cause [23:06:06] !log reedy synchronized wmf-config/InitialiseSettings.php 'Enable BetaFeatures everywher' [23:06:11] (03CR) 10Andrew Bogott: mail.pp: change exec to file (033 comments) [operations/puppet] - 10https://gerrit.wikimedia.org/r/91573 (owner: 10Matanya) [23:06:19] mutante: so can you `ip addr` there at dickson? `ping6 ipv6test.google.com` from there too? [23:06:22] Logged the message, Master [23:06:31] matanya: sorry for serial reviews rather than getting them all in one go :( [23:06:38] Hopefully the changes Reedy and I made to the original patch will keep the load increase down to a reasonable level [23:06:54] https://ganglia.wikimedia.org/latest/graph.php?r=day&z=xlarge&c=Application+servers+eqiad&m=cpu_report&s=by+name&mc=2&g=load_report [23:06:59] Nothing scary [23:07:10] bd808: can discuss later if you want. see https://git.wikimedia.org/summary/?r=operations/apache-config.git for the current config [23:07:18] (or other people can weigh in) [23:07:18] Reedy: May need to touch resources again but it's looking good so far [23:07:21] * jeremyb runs away [23:08:32] jeremyb: No worries. I've looked at that config. If it was puppet-ized my stuff would be easier but I can do something else easily enough [23:09:07] * bd808 <3 per-vhost config snippets collections for things like this [23:10:20] jeremyb: Changing to fit prod needs is on the list: https://www.mediawiki.org/w/index.php?title=Wikimania_Scholarship_app/Backlog&diff=825207&oldid=825056 [23:10:52] jeremyb: http://paste.debian.net/67093/ [23:12:21] mutante: Try `ping6 ipv6.google.com` [23:12:47] PING ipv6.google.com(iad23s07-in-x12.1e100.net) 56 data bytes [23:12:48] 64 bytes from iad23s07-in-x12.1e100.net: icmp_seq=1 ttl=61 time=148 ms [23:12:53] bd808: indeed:) [23:13:11] jeremyb: ^ works, just not the test name [23:13:13] normal google.com is now v6'ed as well [23:13:38] True enough. That's how I connect to gmail etc [23:13:38] yea, he said ipv6test.google.com though [23:13:41] http://ipv6test.google.com/ [23:14:03] and pinging that was unknown host [23:14:35] that's because ipv6test doesn't have a quad-A record [23:14:40] so ping6 wouldn't work on it [23:14:48] ping6 google.com is 64 bytes from iad23s07-in-x00.1e100.net [23:15:06] LeslieCarr: yep, but isnt that odd to not have it when it's the page about it [23:16:00] it is pretty funny [23:16:11] well actually it makes sense [23:16:21] is that the point of the test then ? heh [23:16:21] it's to test that you wont have problems when ipv6 day comes [23:16:36] and if you would have problems, then having a AAAA record would prevent you from even getting to the page [23:16:55] so, it wouldn't work... so yeah, actually does make sense [23:16:59] damn you google for being logical! [23:17:08] hehe, yea, that makes sense [23:20:37] mutante: The IPv6 addr shown in that paste [2620:0:861:52:7a2b:cbff:fe09:c5a] doesn't match dig [2620::861:52:208:80:155:68] [23:22:47] bd808: hrmm.. my guess is the freenode folks configured it differently but didn't reach out for a DNS change, i have no idea besides jeremyb asking so far [23:22:56] PROBLEM - Disk space on mw1194 is CRITICAL: DISK CRITICAL - free space: /tmp 563 MB (3% inode=92%): [23:23:16] The address from the trace is an autogenerated one based on the MAC of the NIC [23:23:16] it's not really operated by us, just some of us have access for emergency [23:23:34] gah, /tmp again? [23:23:41] * greg-g ignores channel [23:24:37] Freenode needs to give it a static IPv6 looks like to me [23:25:47] s/give it/configure eth0 with/ [23:26:51] i'll clean up /tmp if someone digs up the bug and escalates it [23:27:52] bd808: yea. do they actually list it in freenode server lists already? as dickson.wm ? [23:28:22] well, our DNS has AAAA 2620:0:861:52:208:80:155:68 as you said [23:28:51] Coren: fyi ^ [23:29:09] [2620:0:861:52:7a2b:cbff:fe09:c5a] leaks that it's a Dell box FWIW [23:30:04] ori-l: https://bugzilla.wikimedia.org/show_bug.cgi?id=57282 already at high [23:30:17] bd808: I know they've been made aware that they have IPv6, but I don't know whether they've configured it "for real" yet. Do they give a AAAA for it? [23:30:27] ah, thanks for looking it up [23:30:34] Coren: I don't see it listed at all yet [23:30:39] ori-l: is it the same thing as what the previous occurence was diagnosed to? https://bugzilla.wikimedia.org/show_bug.cgi?id=55541 [23:30:46] Segfault when getting EXIF data of some tif images [23:30:56] It's not on http://freenode.net/irc_servers.shtml anyway [23:31:33] # du -ch /tmp/*.tif | tail -1 [23:31:33] 16G total [23:32:06] so: yes [23:32:10] :) [23:32:23] alright, elevating that bug then [23:33:00] done [23:33:20] ipv6calc output for [2620:0:861:52:7a2b:cbff:fe09:c5a] http://paste.debian.net/hidden/585c75ad/ [23:33:55] bd808: Coren : listens on 6667 on tcp , but no tcp6 [23:33:56] RECOVERY - Disk space on mw1194 is OK: DISK OK [23:33:56] !log Purging tifs older than one hour on app servers [23:34:11] mutante: So that's probably just autoconfig. [23:34:11] Logged the message, Master [23:34:22] Coren: yep, as bd808 it's based on MAC [23:34:27] said [23:34:44] so they just didnt do it yet [23:35:18] they can though, they have other stuff listening on tcp6 [23:36:16] chat.freenode.net has 6 AAAA records at the moment [23:37:40] Reedy, thanks for deploying the setting, works beautifully :) [23:43:53] whois 198.73.209.0 [23:44:03] got IPv4? [23:46:25] ori-l: you can use -delete instead of -exec rm {} \; [23:46:31] cajoel: oh? [23:46:42] ori-l: also, when you are using -exec, sometimes + is better than \; [23:46:59] ori-l: \; is one at a time, + batches [23:48:30] cajoel: https://rt.wikimedia.org/Ticket/Display.html?id=4143 is still open :) [23:50:44] jeremyb: cool, thanks for the tips [23:51:31] ori-l: also, maybe \*.tif (with the dot) [23:51:52] * ori-l nods [23:52:37] jeremyb: nice -- I'll run with that. [23:53:19] we'll need some pc router HW [23:53:29] the cisco firewalls we have won't cut it. [23:58:14] (03PS1) 10Dzahn: fix and remove various planet feed URLs [operations/puppet] - 10https://gerrit.wikimedia.org/r/96914 [23:58:40] (03PS1) 10Ori.livneh: Hack: cron job to clean up tifs from /tmp on app servers [operations/puppet] - 10https://gerrit.wikimedia.org/r/96915 [23:59:16] RECOVERY - Disk space on xenon is OK: DISK OK