[00:00:25] no mention of vanadium, either [00:00:37] one sec [00:00:47] I'm fixing another issue right now. [00:03:25] ah, np. i'm probably doing something stupid. [00:03:33] just freed up 160G on virt6 \o/ :) [00:03:42] ok. now I can take a look [00:04:37] hm. [00:06:24] ah. sorry. I forgot something [00:07:11] ori-l: it'll work now [00:07:22] I didn't update the pillars on vanadium [00:07:51] the system is intended to do this automatically when salt runs on puppet, but it wasn't possible before when using regex's. I'll fix this soon [00:09:56] same error [00:10:09] * Ryan_Lane grumbles [00:10:26] https://dpaste.de/hXfua/raw/ [00:10:35] one more time ;) [00:10:42] pillars on tin, too [00:10:48] do i need to make another commit? [00:10:52] nope [00:11:51] I'm going to take some time tomorrow to switch everything to grains to eliminate these manual steps [00:12:00] same error :) [00:12:36] * Ryan_Lane looking [00:12:55] this isn't running the hookl [00:13:00] hah. god damn it [00:13:23] i have to run but if you figure it out feel free to sync or make dummy changes, it's not used by anything yet [00:13:58] I know exactly what it is [00:14:08] missing hookdir [00:14:21] needs a puppet fix [00:14:28] adding it manually for now [00:15:36] ori-l: done, and deployed [00:18:01] meh. I'll fix this when I switch out to sartoris for the frontend [00:33:58] New patchset: Ryan Lane; "Target pillars and states by grain" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/73910 [00:35:08] Change merged: Ryan Lane; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/73910 [00:52:31] !log catrope Started syncing Wikimedia installation... : Update VisualEditor to master again [00:52:41] Logged the message, Master [00:59:57] !log catrope Finished syncing Wikimedia installation... : Update VisualEditor to master again [01:00:08] Logged the message, Master [01:58:40] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:59:30] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 3 [02:15:23] !log LocalisationUpdate completed (1.22wmf10) at Tue Jul 16 02:15:23 UTC 2013 [02:15:35] Logged the message, Master [02:28:34] !log LocalisationUpdate completed (1.22wmf9) at Tue Jul 16 02:28:30 UTC 2013 [02:28:44] Logged the message, Master [02:43:55] !log LocalisationUpdate ResourceLoader cache refresh completed at Tue Jul 16 02:43:55 UTC 2013 [02:44:06] Logged the message, Master [02:57:57] New patchset: TTO; "(bug 51416) New logo for thwiki per community request" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/73919 [06:28:52] bits is being super slow for me here in France [06:29:39] haven't tried looking at anything yet today [06:30:08] seems ok for me [06:30:27] i.e. snappy responses as expected [06:30:27] The anti-French cabal strikes again! [06:30:31] it does [06:30:50] which site were you looking at? [06:30:50] hmm, works better now [06:31:01] A variety of them; they load fine now [06:31:04] ok [06:31:08] weird :) [06:31:14] just needed me to talk nice to 'em [06:31:23] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:31:50] His Master's Voice. [06:31:56] ha! [06:32:23] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 4.777 second response time [06:32:27] this graph is a bit odd: https://ganglia.wikimedia.org/latest/graph.php?r=month&z=xlarge&c=Bits+caches+esams&m=network_report&s=by+name&mc=2&g=network_report [06:33:11] yeah, I'm having a couple issues with loading too [06:33:17] interesting [06:33:19] every time I do a full load there are a couple things that hang… mostly from bits [06:33:21] not everything [06:33:24] just a couple things [06:33:37] couple fonts on one load, common.js on another [06:34:25] I don't have that [06:34:48] I just tried anther project and it loads as expected [06:43:35] for lab watchers, looks like labstore3 is having waitio again...http://ganglia.wikimedia.org/latest/graph_all_periods.php?h=labstore3.pmtpa.wmnet&m=cpu_report&r=4hr&s=by%20name&hc=4&mc=2&st=1373956953&g=cpu_report&z=large&c=Labs%20NFS%20cluster%20pmtpa [06:47:17] guillom: looks like not just you http://status.wikimedia.org/8777/163394/Wiki-platform-[[w:fr:Main-Page]]-(s6) [06:48:12] Jamesofur: ah, thanks. When I looked at status.wm.o, everything was green [06:48:29] yeah, first time it was for me too [06:48:34] seems to have flipped recently [06:52:02] PROBLEM - Puppet freshness on ms-be5 is CRITICAL: No successful Puppet run in the last 10 hours [06:56:23] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:57:13] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.199 second response time [06:57:26] New patchset: ArielGlenn; "app servers should have php-mail and php-mail-mime for Echo (rt 5338)" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/73927 [06:59:47] Change merged: ArielGlenn; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/73927 [07:02:26] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:03:13] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.131 second response time [07:21:27] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:22:17] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.130 second response time [07:27:26] mark: csw1-sdtpa.wikimedia.org is out to lunch, "Global I2C Error Indicator is set (severity: Major)" (see rancid email) [07:27:31] hello [07:27:34] hello [07:56:07] PROBLEM - Puppet freshness on mw1001 is CRITICAL: No successful Puppet run in the last 10 hours [08:22:24] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:23:15] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.137 second response time [08:33:31] New patchset: Siebrand; "Revert "Disable ULS on ml.*"" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/73934 [08:36:44] PROBLEM - Puppet freshness on searchidx1001 is CRITICAL: No successful Puppet run in the last 10 hours [08:38:29] hashar: Can you restart Gerrit? It's unworkably slow... [08:39:17] siebrand: hi :-) works for me [08:39:34] hashar: you mean Gerrit is quick for you? [08:39:43] yup [08:39:46] hashar: Please get a second opinion with Niklas. He's dpeloying... [08:39:51] let me look at ganglia [08:40:07] my (1.22wmf10)$ git submodule update --init has been running for ~10 minutes and is still in extensions starting with letter E [08:40:57] Nikerabbit: if you want to update a single extension, just pass its name to the git submodule update [08:41:03] but yeah maybe gerrit needs a restart [08:41:09] http://ganglia.wikimedia.org/latest/?r=4hr&cs=&ce=&c=Miscellaneous+eqiad&h=manganese.wikimedia.org&tab=m&vn=&mc=2&z=medium&metric_group=ALLGROUPS [08:41:12] there is a bunch of load [08:41:15] let me look at the queue [08:41:49] + ssh -p 29418 hashar@gerrit.wikimedia.org 'gerrit help' [08:41:49] gerrit: help: not found [08:41:50] doh [08:43:25] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:43:44] PROBLEM - Puppet freshness on rubidium is CRITICAL: No successful Puppet run in the last 10 hours [08:44:15] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.134 second response time [08:44:44] PROBLEM - Puppet freshness on ekrem is CRITICAL: No successful Puppet run in the last 10 hours [08:44:44] PROBLEM - Puppet freshness on manganese is CRITICAL: No successful Puppet run in the last 10 hours [08:44:44] PROBLEM - Puppet freshness on mw1007 is CRITICAL: No successful Puppet run in the last 10 hours [08:44:44] PROBLEM - Puppet freshness on mw1041 is CRITICAL: No successful Puppet run in the last 10 hours [08:44:44] PROBLEM - Puppet freshness on mw1043 is CRITICAL: No successful Puppet run in the last 10 hours [08:44:45] PROBLEM - Puppet freshness on mw1063 is CRITICAL: No successful Puppet run in the last 10 hours [08:44:45] PROBLEM - Puppet freshness on mw1087 is CRITICAL: No successful Puppet run in the last 10 hours [08:44:46] PROBLEM - Puppet freshness on mw1171 is CRITICAL: No successful Puppet run in the last 10 hours [08:44:46] PROBLEM - Puppet freshness on mw1197 is CRITICAL: No successful Puppet run in the last 10 hours [08:44:47] PROBLEM - Puppet freshness on mw121 is CRITICAL: No successful Puppet run in the last 10 hours [08:44:47] PROBLEM - Puppet freshness on mw1210 is CRITICAL: No successful Puppet run in the last 10 hours [08:44:48] PROBLEM - Puppet freshness on mw58 is CRITICAL: No successful Puppet run in the last 10 hours [08:44:48] PROBLEM - Puppet freshness on search1024 is CRITICAL: No successful Puppet run in the last 10 hours [08:44:49] PROBLEM - Puppet freshness on search18 is CRITICAL: No successful Puppet run in the last 10 hours [08:44:49] PROBLEM - Puppet freshness on solr1003 is CRITICAL: No successful Puppet run in the last 10 hours [08:44:50] PROBLEM - Puppet freshness on solr3 is CRITICAL: No successful Puppet run in the last 10 hours [08:44:50] PROBLEM - Puppet freshness on sq76 is CRITICAL: No successful Puppet run in the last 10 hours [08:44:51] PROBLEM - Puppet freshness on srv292 is CRITICAL: No successful Puppet run in the last 10 hours [08:44:51] PROBLEM - Puppet freshness on stat1 is CRITICAL: No successful Puppet run in the last 10 hours [08:44:52] PROBLEM - Puppet freshness on titanium is CRITICAL: No successful Puppet run in the last 10 hours [08:45:44] PROBLEM - Puppet freshness on amssq53 is CRITICAL: No successful Puppet run in the last 10 hours [08:45:44] PROBLEM - Puppet freshness on analytics1014 is CRITICAL: No successful Puppet run in the last 10 hours [08:45:44] PROBLEM - Puppet freshness on cp1005 is CRITICAL: No successful Puppet run in the last 10 hours [08:45:44] PROBLEM - Puppet freshness on cp3009 is CRITICAL: No successful Puppet run in the last 10 hours [08:45:44] PROBLEM - Puppet freshness on cp3012 is CRITICAL: No successful Puppet run in the last 10 hours [08:45:45] PROBLEM - Puppet freshness on db1001 is CRITICAL: No successful Puppet run in the last 10 hours [08:45:45] PROBLEM - Puppet freshness on db1031 is CRITICAL: No successful Puppet run in the last 10 hours [08:45:46] PROBLEM - Puppet freshness on db1044 is CRITICAL: No successful Puppet run in the last 10 hours [08:45:46] PROBLEM - Puppet freshness on db39 is CRITICAL: No successful Puppet run in the last 10 hours [08:45:47] PROBLEM - Puppet freshness on helium is CRITICAL: No successful Puppet run in the last 10 hours [08:45:47] PROBLEM - Puppet freshness on mc1007 is CRITICAL: No successful Puppet run in the last 10 hours [08:45:48] PROBLEM - Puppet freshness on ms-be1006 is CRITICAL: No successful Puppet run in the last 10 hours [08:45:48] PROBLEM - Puppet freshness on ms10 is CRITICAL: No successful Puppet run in the last 10 hours [08:45:49] PROBLEM - Puppet freshness on mw1032 is CRITICAL: No successful Puppet run in the last 10 hours [08:45:49] PROBLEM - Puppet freshness on mw124 is CRITICAL: No successful Puppet run in the last 10 hours [08:45:50] PROBLEM - Puppet freshness on pc1 is CRITICAL: No successful Puppet run in the last 10 hours [08:45:50] PROBLEM - Puppet freshness on potassium is CRITICAL: No successful Puppet run in the last 10 hours [08:45:51] PROBLEM - Puppet freshness on praseodymium is CRITICAL: No successful Puppet run in the last 10 hours [08:45:51] PROBLEM - Puppet freshness on mw43 is CRITICAL: No successful Puppet run in the last 10 hours [08:45:52] PROBLEM - Puppet freshness on rdb1002 is CRITICAL: No successful Puppet run in the last 10 hours [08:45:52] PROBLEM - Puppet freshness on sq58 is CRITICAL: No successful Puppet run in the last 10 hours [08:45:53] PROBLEM - Puppet freshness on srv255 is CRITICAL: No successful Puppet run in the last 10 hours [08:45:53] PROBLEM - Puppet freshness on sq54 is CRITICAL: No successful Puppet run in the last 10 hours [08:45:54] PROBLEM - Puppet freshness on srv273 is CRITICAL: No successful Puppet run in the last 10 hours [08:45:54] PROBLEM - Puppet freshness on wtp1015 is CRITICAL: No successful Puppet run in the last 10 hours [08:46:44] PROBLEM - Puppet freshness on cp1010 is CRITICAL: No successful Puppet run in the last 10 hours [08:46:44] PROBLEM - Puppet freshness on db1022 is CRITICAL: No successful Puppet run in the last 10 hours [08:46:44] PROBLEM - Puppet freshness on mw1003 is CRITICAL: No successful Puppet run in the last 10 hours [08:46:44] PROBLEM - Puppet freshness on mw1024 is CRITICAL: No successful Puppet run in the last 10 hours [08:46:44] PROBLEM - Puppet freshness on ms-fe1001 is CRITICAL: No successful Puppet run in the last 10 hours [08:46:45] PROBLEM - Puppet freshness on mw1033 is CRITICAL: No successful Puppet run in the last 10 hours [08:46:45] PROBLEM - Puppet freshness on mw1046 is CRITICAL: No successful Puppet run in the last 10 hours [08:46:46] PROBLEM - Puppet freshness on mw106 is CRITICAL: No successful Puppet run in the last 10 hours [08:46:46] PROBLEM - Puppet freshness on mw1069 is CRITICAL: No successful Puppet run in the last 10 hours [08:46:47] PROBLEM - Puppet freshness on mw1150 is CRITICAL: No successful Puppet run in the last 10 hours [08:46:47] PROBLEM - Puppet freshness on mw1189 is CRITICAL: No successful Puppet run in the last 10 hours [08:46:48] PROBLEM - Puppet freshness on mw1201 is CRITICAL: No successful Puppet run in the last 10 hours [08:46:48] PROBLEM - Puppet freshness on mw1205 is CRITICAL: No successful Puppet run in the last 10 hours [08:46:49] PROBLEM - Puppet freshness on mw2 is CRITICAL: No successful Puppet run in the last 10 hours [08:46:49] PROBLEM - Puppet freshness on mw35 is CRITICAL: No successful Puppet run in the last 10 hours [08:46:50] PROBLEM - Puppet freshness on mw79 is CRITICAL: No successful Puppet run in the last 10 hours [08:46:50] PROBLEM - Puppet freshness on mw98 is CRITICAL: No successful Puppet run in the last 10 hours [08:46:51] PROBLEM - Puppet freshness on rdb1003 is CRITICAL: No successful Puppet run in the last 10 hours [08:46:51] PROBLEM - Puppet freshness on search33 is CRITICAL: No successful Puppet run in the last 10 hours [08:46:52] PROBLEM - Puppet freshness on srv193 is CRITICAL: No successful Puppet run in the last 10 hours [08:46:52] PROBLEM - Puppet freshness on wtp1007 is CRITICAL: No successful Puppet run in the last 10 hours [08:47:44] PROBLEM - Puppet freshness on amslvs1 is CRITICAL: No successful Puppet run in the last 10 hours [08:47:45] PROBLEM - Puppet freshness on amssq48 is CRITICAL: No successful Puppet run in the last 10 hours [08:47:45] PROBLEM - Puppet freshness on analytics1004 is CRITICAL: No successful Puppet run in the last 10 hours [08:47:45] PROBLEM - Puppet freshness on calcium is CRITICAL: No successful Puppet run in the last 10 hours [08:47:45] PROBLEM - Puppet freshness on db1033 is CRITICAL: No successful Puppet run in the last 10 hours [08:49:44] PROBLEM - Puppet freshness on antimony is CRITICAL: No successful Puppet run in the last 10 hours [08:49:44] PROBLEM - Puppet freshness on cp1058 is CRITICAL: No successful Puppet run in the last 10 hours [08:49:44] PROBLEM - Puppet freshness on cp3011 is CRITICAL: No successful Puppet run in the last 10 hours [08:49:44] PROBLEM - Puppet freshness on dataset1001 is CRITICAL: No successful Puppet run in the last 10 hours [08:49:44] PROBLEM - Puppet freshness on db1002 is CRITICAL: No successful Puppet run in the last 10 hours [08:49:45] PROBLEM - Puppet freshness on db1014 is CRITICAL: No successful Puppet run in the last 10 hours [08:49:45] PROBLEM - Puppet freshness on labstore1 is CRITICAL: No successful Puppet run in the last 10 hours [08:49:46] PROBLEM - Puppet freshness on labstore1001 is CRITICAL: No successful Puppet run in the last 10 hours [08:49:46] PROBLEM - Puppet freshness on labstore3 is CRITICAL: No successful Puppet run in the last 10 hours [08:49:47] PROBLEM - Puppet freshness on mc1012 is CRITICAL: No successful Puppet run in the last 10 hours [08:49:47] PROBLEM - Puppet freshness on ms-be12 is CRITICAL: No successful Puppet run in the last 10 hours [08:49:48] PROBLEM - Puppet freshness on ms-fe1002 is CRITICAL: No successful Puppet run in the last 10 hours [08:49:48] PROBLEM - Puppet freshness on mw1027 is CRITICAL: No successful Puppet run in the last 10 hours [08:49:49] PROBLEM - Puppet freshness on mw1104 is CRITICAL: No successful Puppet run in the last 10 hours [08:49:49] PROBLEM - Puppet freshness on mw1206 is CRITICAL: No successful Puppet run in the last 10 hours [08:49:50] PROBLEM - Puppet freshness on mw1208 is CRITICAL: No successful Puppet run in the last 10 hours [08:49:50] PROBLEM - Puppet freshness on mw1211 is CRITICAL: No successful Puppet run in the last 10 hours [08:49:51] PROBLEM - Puppet freshness on mw42 is CRITICAL: No successful Puppet run in the last 10 hours [08:49:51] PROBLEM - Puppet freshness on mw75 is CRITICAL: No successful Puppet run in the last 10 hours [08:49:52] PROBLEM - Puppet freshness on solr1 is CRITICAL: No successful Puppet run in the last 10 hours [08:49:52] PROBLEM - Puppet freshness on search25 is CRITICAL: No successful Puppet run in the last 10 hours [08:49:53] PROBLEM - Puppet freshness on srv242 is CRITICAL: No successful Puppet run in the last 10 hours [08:49:53] PROBLEM - Puppet freshness on ssl1 is CRITICAL: No successful Puppet run in the last 10 hours [08:49:54] PROBLEM - Puppet freshness on virt11 is CRITICAL: No successful Puppet run in the last 10 hours [08:49:54] PROBLEM - Puppet freshness on wtp1001 is CRITICAL: No successful Puppet run in the last 10 hours [08:50:44] PROBLEM - Puppet freshness on amssq51 is CRITICAL: No successful Puppet run in the last 10 hours [08:50:44] PROBLEM - Puppet freshness on amssq56 is CRITICAL: No successful Puppet run in the last 10 hours [08:50:44] PROBLEM - Puppet freshness on analytics1002 is CRITICAL: No successful Puppet run in the last 10 hours [08:50:44] PROBLEM - Puppet freshness on analytics1022 is CRITICAL: No successful Puppet run in the last 10 hours [08:50:44] PROBLEM - Puppet freshness on bast1001 is CRITICAL: No successful Puppet run in the last 10 hours [08:52:24] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:52:44] PROBLEM - Puppet freshness on analytics1008 is CRITICAL: No successful Puppet run in the last 10 hours [08:52:44] PROBLEM - Puppet freshness on cp1038 is CRITICAL: No successful Puppet run in the last 10 hours [08:52:44] PROBLEM - Puppet freshness on db1058 is CRITICAL: No successful Puppet run in the last 10 hours [08:52:44] PROBLEM - Puppet freshness on db65 is CRITICAL: No successful Puppet run in the last 10 hours [08:52:44] PROBLEM - Puppet freshness on db77 is CRITICAL: No successful Puppet run in the last 10 hours [08:53:15] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.127 second response time [08:53:44] PROBLEM - Puppet freshness on analytics1011 is CRITICAL: No successful Puppet run in the last 10 hours [08:53:44] PROBLEM - Puppet freshness on analytics1019 is CRITICAL: No successful Puppet run in the last 10 hours [08:53:44] PROBLEM - Puppet freshness on brewster is CRITICAL: No successful Puppet run in the last 10 hours [08:53:44] PROBLEM - Puppet freshness on cp1065 is CRITICAL: No successful Puppet run in the last 10 hours [08:53:44] PROBLEM - Puppet freshness on dataset2 is CRITICAL: No successful Puppet run in the last 10 hours [08:53:45] PROBLEM - Puppet freshness on db1024 is CRITICAL: No successful Puppet run in the last 10 hours [08:53:45] PROBLEM - Puppet freshness on db32 is CRITICAL: No successful Puppet run in the last 10 hours [08:53:46] PROBLEM - Puppet freshness on db51 is CRITICAL: No successful Puppet run in the last 10 hours [08:53:46] PROBLEM - Puppet freshness on db57 is CRITICAL: No successful Puppet run in the last 10 hours [08:53:47] PROBLEM - Puppet freshness on es5 is CRITICAL: No successful Puppet run in the last 10 hours [08:53:47] PROBLEM - Puppet freshness on mc1008 is CRITICAL: No successful Puppet run in the last 10 hours [08:53:48] PROBLEM - Puppet freshness on mw1138 is CRITICAL: No successful Puppet run in the last 10 hours [08:53:48] PROBLEM - Puppet freshness on mw26 is CRITICAL: No successful Puppet run in the last 10 hours [08:53:49] PROBLEM - Puppet freshness on mw33 is CRITICAL: No successful Puppet run in the last 10 hours [08:53:49] PROBLEM - Puppet freshness on mw36 is CRITICAL: No successful Puppet run in the last 10 hours [08:53:50] PROBLEM - Puppet freshness on mw64 is CRITICAL: No successful Puppet run in the last 10 hours [08:53:50] PROBLEM - Puppet freshness on sq50 is CRITICAL: No successful Puppet run in the last 10 hours [08:53:51] PROBLEM - Puppet freshness on stat1002 is CRITICAL: No successful Puppet run in the last 10 hours [08:53:51] PROBLEM - Puppet freshness on tarin is CRITICAL: No successful Puppet run in the last 10 hours [08:53:52] PROBLEM - Puppet freshness on wtp1013 is CRITICAL: No successful Puppet run in the last 10 hours [08:54:44] PROBLEM - Puppet freshness on db1010 is CRITICAL: No successful Puppet run in the last 10 hours [08:54:44] PROBLEM - Puppet freshness on aluminium is CRITICAL: No successful Puppet run in the last 10 hours [08:54:44] PROBLEM - Puppet freshness on db46 is CRITICAL: No successful Puppet run in the last 10 hours [08:54:44] PROBLEM - Puppet freshness on db55 is CRITICAL: No successful Puppet run in the last 10 hours [08:54:44] PROBLEM - Puppet freshness on es1009 is CRITICAL: No successful Puppet run in the last 10 hours [08:54:45] PROBLEM - Puppet freshness on labsdb1001 is CRITICAL: No successful Puppet run in the last 10 hours [08:54:45] PROBLEM - Puppet freshness on mw1022 is CRITICAL: No successful Puppet run in the last 10 hours [08:54:46] PROBLEM - Puppet freshness on mw1040 is CRITICAL: No successful Puppet run in the last 10 hours [08:54:46] PROBLEM - Puppet freshness on mw1062 is CRITICAL: No successful Puppet run in the last 10 hours [08:54:47] PROBLEM - Puppet freshness on mw107 is CRITICAL: No successful Puppet run in the last 10 hours [08:54:47] PROBLEM - Puppet freshness on mw109 is CRITICAL: No successful Puppet run in the last 10 hours [08:54:48] PROBLEM - Puppet freshness on mw1132 is CRITICAL: No successful Puppet run in the last 10 hours [08:54:48] PROBLEM - Puppet freshness on mw1185 is CRITICAL: No successful Puppet run in the last 10 hours [08:54:49] PROBLEM - Puppet freshness on mw1218 is CRITICAL: No successful Puppet run in the last 10 hours [08:54:49] PROBLEM - Puppet freshness on mw40 is CRITICAL: No successful Puppet run in the last 10 hours [08:54:50] PROBLEM - Puppet freshness on mw53 is CRITICAL: No successful Puppet run in the last 10 hours [08:54:50] PROBLEM - Puppet freshness on mw70 is CRITICAL: No successful Puppet run in the last 10 hours [08:54:51] PROBLEM - Puppet freshness on snapshot4 is CRITICAL: No successful Puppet run in the last 10 hours [08:54:51] PROBLEM - Puppet freshness on sq55 is CRITICAL: No successful Puppet run in the last 10 hours [08:54:52] PROBLEM - Puppet freshness on sq64 is CRITICAL: No successful Puppet run in the last 10 hours [08:54:52] PROBLEM - Puppet freshness on sq81 is CRITICAL: No successful Puppet run in the last 10 hours [08:54:53] PROBLEM - Puppet freshness on srv296 is CRITICAL: No successful Puppet run in the last 10 hours [08:54:53] PROBLEM - Puppet freshness on wtp1021 is CRITICAL: No successful Puppet run in the last 10 hours [08:55:44] PROBLEM - Puppet freshness on amssq59 is CRITICAL: No successful Puppet run in the last 10 hours [08:55:44] PROBLEM - Puppet freshness on colby is CRITICAL: No successful Puppet run in the last 10 hours [08:55:44] PROBLEM - Puppet freshness on cp3006 is CRITICAL: No successful Puppet run in the last 10 hours [08:55:44] PROBLEM - Puppet freshness on db31 is CRITICAL: No successful Puppet run in the last 10 hours [08:55:44] PROBLEM - Puppet freshness on lvs6 is CRITICAL: No successful Puppet run in the last 10 hours [08:55:45] PROBLEM - Puppet freshness on mc1011 is CRITICAL: No successful Puppet run in the last 10 hours [08:55:45] PROBLEM - Puppet freshness on ms-fe1003 is CRITICAL: No successful Puppet run in the last 10 hours [08:55:46] PROBLEM - Puppet freshness on mw1072 is CRITICAL: No successful Puppet run in the last 10 hours [08:55:46] PROBLEM - Puppet freshness on mw1134 is CRITICAL: No successful Puppet run in the last 10 hours [08:55:47] PROBLEM - Puppet freshness on mw117 is CRITICAL: No successful Puppet run in the last 10 hours [08:55:47] PROBLEM - Puppet freshness on mw1178 is CRITICAL: No successful Puppet run in the last 10 hours [08:55:48] PROBLEM - Puppet freshness on mw1219 is CRITICAL: No successful Puppet run in the last 10 hours [08:55:48] PROBLEM - Puppet freshness on mw87 is CRITICAL: No successful Puppet run in the last 10 hours [08:55:49] PROBLEM - Puppet freshness on professor is CRITICAL: No successful Puppet run in the last 10 hours [08:55:49] PROBLEM - Puppet freshness on search1006 is CRITICAL: No successful Puppet run in the last 10 hours [08:55:50] PROBLEM - Puppet freshness on search1011 is CRITICAL: No successful Puppet run in the last 10 hours [08:55:50] PROBLEM - Puppet freshness on srv285 is CRITICAL: No successful Puppet run in the last 10 hours [08:55:51] PROBLEM - Puppet freshness on virt2 is CRITICAL: No successful Puppet run in the last 10 hours [08:56:44] PROBLEM - Puppet freshness on amssq32 is CRITICAL: No successful Puppet run in the last 10 hours [08:56:44] PROBLEM - Puppet freshness on amssq43 is CRITICAL: No successful Puppet run in the last 10 hours [08:56:44] PROBLEM - Puppet freshness on analytics1001 is CRITICAL: No successful Puppet run in the last 10 hours [08:56:44] PROBLEM - Puppet freshness on cp1015 is CRITICAL: No successful Puppet run in the last 10 hours [08:56:44] PROBLEM - Puppet freshness on db1011 is CRITICAL: No successful Puppet run in the last 10 hours [08:56:44] PROBLEM - Puppet freshness on db29 is CRITICAL: No successful Puppet run in the last 10 hours [08:58:44] PROBLEM - Puppet freshness on analytics1020 is CRITICAL: No successful Puppet run in the last 10 hours [08:58:44] PROBLEM - Puppet freshness on cp1001 is CRITICAL: No successful Puppet run in the last 10 hours [08:58:44] PROBLEM - Puppet freshness on cp1039 is CRITICAL: No successful Puppet run in the last 10 hours [08:58:44] PROBLEM - Puppet freshness on cp1054 is CRITICAL: No successful Puppet run in the last 10 hours [08:58:44] PROBLEM - Puppet freshness on cp3019 is CRITICAL: No successful Puppet run in the last 10 hours [08:59:44] PROBLEM - Puppet freshness on amssq36 is CRITICAL: No successful Puppet run in the last 10 hours [08:59:44] PROBLEM - Puppet freshness on cp1008 is CRITICAL: No successful Puppet run in the last 10 hours [08:59:44] PROBLEM - Puppet freshness on cp1009 is CRITICAL: No successful Puppet run in the last 10 hours [08:59:44] PROBLEM - Puppet freshness on cp1064 is CRITICAL: No successful Puppet run in the last 10 hours [08:59:44] PROBLEM - Puppet freshness on cp1068 is CRITICAL: No successful Puppet run in the last 10 hours [09:00:38] Daniel Kinzler reports on #wikimedia-dev that Gerrit is slow. [09:00:43] (report #3) [09:00:44] PROBLEM - Puppet freshness on amssq41 is CRITICAL: No successful Puppet run in the last 10 hours [09:00:45] PROBLEM - Puppet freshness on amssq60 is CRITICAL: No successful Puppet run in the last 10 hours [09:00:45] PROBLEM - Puppet freshness on analytics1006 is CRITICAL: No successful Puppet run in the last 10 hours [09:00:45] PROBLEM - Puppet freshness on cp1057 is CRITICAL: No successful Puppet run in the last 10 hours [09:00:45] PROBLEM - Puppet freshness on db1028 is CRITICAL: No successful Puppet run in the last 10 hours [09:00:45] PROBLEM - Puppet freshness on analytics1024 is CRITICAL: No successful Puppet run in the last 10 hours [09:00:45] PROBLEM - Puppet freshness on db68 is CRITICAL: No successful Puppet run in the last 10 hours [09:00:46] PROBLEM - Puppet freshness on ersch is CRITICAL: No successful Puppet run in the last 10 hours [09:00:46] PROBLEM - Puppet freshness on lvs3 is CRITICAL: No successful Puppet run in the last 10 hours [09:00:47] PROBLEM - Puppet freshness on ms-be1005 is CRITICAL: No successful Puppet run in the last 10 hours [09:00:47] PROBLEM - Puppet freshness on ms-be7 is CRITICAL: No successful Puppet run in the last 10 hours [09:00:48] PROBLEM - Puppet freshness on mw1039 is CRITICAL: No successful Puppet run in the last 10 hours [09:00:48] PROBLEM - Puppet freshness on mw1177 is CRITICAL: No successful Puppet run in the last 10 hours [09:00:49] PROBLEM - Puppet freshness on mw1184 is CRITICAL: No successful Puppet run in the last 10 hours [09:00:49] PROBLEM - Puppet freshness on mw120 is CRITICAL: No successful Puppet run in the last 10 hours [09:00:50] PROBLEM - Puppet freshness on db38 is CRITICAL: No successful Puppet run in the last 10 hours [09:00:50] PROBLEM - Puppet freshness on mw96 is CRITICAL: No successful Puppet run in the last 10 hours [09:00:51] PROBLEM - Puppet freshness on ssl1004 is CRITICAL: No successful Puppet run in the last 10 hours [09:00:51] PROBLEM - Puppet freshness on searchidx2 is CRITICAL: No successful Puppet run in the last 10 hours [09:01:45] PROBLEM - Puppet freshness on amssq58 is CRITICAL: No successful Puppet run in the last 10 hours [09:01:45] PROBLEM - Puppet freshness on db1003 is CRITICAL: No successful Puppet run in the last 10 hours [09:01:45] PROBLEM - Puppet freshness on cp1019 is CRITICAL: No successful Puppet run in the last 10 hours [09:01:45] PROBLEM - Puppet freshness on db1041 is CRITICAL: No successful Puppet run in the last 10 hours [09:01:45] PROBLEM - Puppet freshness on db1043 is CRITICAL: No successful Puppet run in the last 10 hours [09:03:44] PROBLEM - Puppet freshness on amslvs3 is CRITICAL: No successful Puppet run in the last 10 hours [09:03:45] PROBLEM - Puppet freshness on analytics1013 is CRITICAL: No successful Puppet run in the last 10 hours [09:03:45] PROBLEM - Puppet freshness on analytics1026 is CRITICAL: No successful Puppet run in the last 10 hours [09:03:45] PROBLEM - Puppet freshness on cp1004 is CRITICAL: No successful Puppet run in the last 10 hours [09:03:45] PROBLEM - Puppet freshness on cp1011 is CRITICAL: No successful Puppet run in the last 10 hours [09:04:44] PROBLEM - Puppet freshness on amssq46 is CRITICAL: No successful Puppet run in the last 10 hours [09:04:44] PROBLEM - Puppet freshness on amssq62 is CRITICAL: No successful Puppet run in the last 10 hours [09:04:44] PROBLEM - Puppet freshness on db1053 is CRITICAL: No successful Puppet run in the last 10 hours [09:04:44] PROBLEM - Puppet freshness on cp1055 is CRITICAL: No successful Puppet run in the last 10 hours [09:04:44] PROBLEM - Puppet freshness on es8 is CRITICAL: No successful Puppet run in the last 10 hours [09:04:45] PROBLEM - Puppet freshness on ms1002 is CRITICAL: No successful Puppet run in the last 10 hours [09:04:45] PROBLEM - Puppet freshness on mw102 is CRITICAL: No successful Puppet run in the last 10 hours [09:04:46] PROBLEM - Puppet freshness on mw6 is CRITICAL: No successful Puppet run in the last 10 hours [09:04:46] PROBLEM - Puppet freshness on srv241 is CRITICAL: No successful Puppet run in the last 10 hours [09:04:47] PROBLEM - Puppet freshness on ssl3002 is CRITICAL: No successful Puppet run in the last 10 hours [09:04:47] PROBLEM - Puppet freshness on virt5 is CRITICAL: No successful Puppet run in the last 10 hours [09:04:48] PROBLEM - Puppet freshness on virt7 is CRITICAL: No successful Puppet run in the last 10 hours [09:04:48] PROBLEM - Puppet freshness on wtp1023 is CRITICAL: No successful Puppet run in the last 10 hours [09:05:44] PROBLEM - Puppet freshness on analytics1027 is CRITICAL: No successful Puppet run in the last 10 hours [09:05:44] PROBLEM - Puppet freshness on amssq38 is CRITICAL: No successful Puppet run in the last 10 hours [09:05:44] PROBLEM - Puppet freshness on cp1052 is CRITICAL: No successful Puppet run in the last 10 hours [09:05:44] PROBLEM - Puppet freshness on db1006 is CRITICAL: No successful Puppet run in the last 10 hours [09:05:44] PROBLEM - Puppet freshness on db1008 is CRITICAL: No successful Puppet run in the last 10 hours [09:08:05] PROBLEM - Puppet freshness on amssq33 is CRITICAL: No successful Puppet run in the last 10 hours [09:08:05] PROBLEM - Puppet freshness on amssq39 is CRITICAL: No successful Puppet run in the last 10 hours [09:08:05] PROBLEM - Puppet freshness on amssq45 is CRITICAL: No successful Puppet run in the last 10 hours [09:08:05] PROBLEM - Puppet freshness on amssq42 is CRITICAL: No successful Puppet run in the last 10 hours [09:08:05] PROBLEM - Puppet freshness on amssq49 is CRITICAL: No successful Puppet run in the last 10 hours [09:10:48] apergos: sorry for the interrupt but could you restart the gerrit service on manganese please ? The ssh command line does not work as expected [09:10:58] sec [09:10:59] and it several people reported slowness :/ [09:11:05] PROBLEM - Puppet freshness on amssq52 is CRITICAL: No successful Puppet run in the last 10 hours [09:11:05] PROBLEM - Puppet freshness on analytics1015 is CRITICAL: No successful Puppet run in the last 10 hours [09:11:05] PROBLEM - Puppet freshness on cp1007 is CRITICAL: No successful Puppet run in the last 10 hours [09:11:05] PROBLEM - Puppet freshness on cp1056 is CRITICAL: No successful Puppet run in the last 10 hours [09:11:05] PROBLEM - Puppet freshness on analytics1012 is CRITICAL: No successful Puppet run in the last 10 hours [09:11:31] jva at 99% cpu again [09:11:46] :/ [09:11:57] hmm [09:12:00] maybe we can get a stack trace first [09:12:03] !log restarting gerrit [09:12:13] Logged the message, Master [09:12:13] uh. sorry, I was too quick on the trigger [09:12:18] nm :-] [09:12:27] there might be some logs :) [09:12:33] apergos, hashar: I see lots of novaadmin connections on gerrit [09:12:45] not any more you won't :-D [09:12:51] qchris_away: good morning :-) [09:13:04] Too late :-) [09:13:05] PROBLEM - Puppet freshness on cp1013 is CRITICAL: No successful Puppet run in the last 10 hours [09:13:05] PROBLEM - Puppet freshness on cp1050 is CRITICAL: No successful Puppet run in the last 10 hours [09:13:05] PROBLEM - Puppet freshness on cp3010 is CRITICAL: No successful Puppet run in the last 10 hours [09:13:05] PROBLEM - Puppet freshness on db1036 is CRITICAL: No successful Puppet run in the last 10 hours [09:13:05] PROBLEM - Puppet freshness on db1048 is CRITICAL: No successful Puppet run in the last 10 hours [09:13:05] PROBLEM - Puppet freshness on gadolinium is CRITICAL: No successful Puppet run in the last 10 hours [09:13:05] PROBLEM - Puppet freshness on lvs5 is CRITICAL: No successful Puppet run in the last 10 hours [09:13:06] Hehe. [09:13:06] PROBLEM - Puppet freshness on ms-be4 is CRITICAL: No successful Puppet run in the last 10 hours [09:13:06] PROBLEM - Puppet freshness on ms5 is CRITICAL: No successful Puppet run in the last 10 hours [09:13:07] PROBLEM - Puppet freshness on mw1044 is CRITICAL: No successful Puppet run in the last 10 hours [09:13:07] PROBLEM - Puppet freshness on mw1101 is CRITICAL: No successful Puppet run in the last 10 hours [09:13:08] PROBLEM - Puppet freshness on mw1162 is CRITICAL: No successful Puppet run in the last 10 hours [09:13:08] PROBLEM - Puppet freshness on mw119 is CRITICAL: No successful Puppet run in the last 10 hours [09:13:08] yeah sorry [09:13:09] PROBLEM - Puppet freshness on mw1196 is CRITICAL: No successful Puppet run in the last 10 hours [09:13:09] PROBLEM - Puppet freshness on mw5 is CRITICAL: No successful Puppet run in the last 10 hours [09:13:10] PROBLEM - Puppet freshness on mw83 is CRITICAL: No successful Puppet run in the last 10 hours [09:13:10] PROBLEM - Puppet freshness on mw90 is CRITICAL: No successful Puppet run in the last 10 hours [09:13:11] Good morning hashar [09:13:11] PROBLEM - Puppet freshness on search1014 is CRITICAL: No successful Puppet run in the last 10 hours [09:13:11] PROBLEM - Puppet freshness on search21 is CRITICAL: No successful Puppet run in the last 10 hours [09:13:12] PROBLEM - Puppet freshness on sq79 is CRITICAL: No successful Puppet run in the last 10 hours [09:13:12] PROBLEM - Puppet freshness on srv290 is CRITICAL: No successful Puppet run in the last 10 hours [09:13:13] PROBLEM - Puppet freshness on srv275 is CRITICAL: No successful Puppet run in the last 10 hours [09:13:13] PROBLEM - Puppet freshness on wtp1024 is CRITICAL: No successful Puppet run in the last 10 hours [09:13:14] PROBLEM - Puppet freshness on zirconium is CRITICAL: No successful Puppet run in the last 10 hours [09:13:16] for god sake puppet [09:13:17] hurry up and flood out would ya [09:13:30] these are of course all lies [09:14:00] Not sure what novaadmin processes we killed, but seems gerrit ssh is back to normal. [09:14:12] PROBLEM - Puppet freshness on db1020 is CRITICAL: No successful Puppet run in the last 10 hours [09:14:12] PROBLEM - Puppet freshness on lvs1 is CRITICAL: No successful Puppet run in the last 10 hours [09:14:12] PROBLEM - Puppet freshness on mc1009 is CRITICAL: No successful Puppet run in the last 10 hours [09:14:12] PROBLEM - Puppet freshness on ms-be1012 is CRITICAL: No successful Puppet run in the last 10 hours [09:14:12] PROBLEM - Puppet freshness on mw1111 is CRITICAL: No successful Puppet run in the last 10 hours [09:14:12] PROBLEM - Puppet freshness on mw1124 is CRITICAL: No successful Puppet run in the last 10 hours [09:14:12] PROBLEM - Puppet freshness on mw1125 is CRITICAL: No successful Puppet run in the last 10 hours [09:14:12] PROBLEM - Puppet freshness on mw1130 is CRITICAL: No successful Puppet run in the last 10 hours [09:14:12] PROBLEM - Puppet freshness on mw1147 is CRITICAL: No successful Puppet run in the last 10 hours [09:14:12] PROBLEM - Puppet freshness on mw1161 is CRITICAL: No successful Puppet run in the last 10 hours [09:14:12] PROBLEM - Puppet freshness on mw1190 is CRITICAL: No successful Puppet run in the last 10 hours [09:14:12] PROBLEM - Puppet freshness on mw1214 is CRITICAL: No successful Puppet run in the last 10 hours [09:14:12] PROBLEM - Puppet freshness on mw38 is CRITICAL: No successful Puppet run in the last 10 hours [09:14:12] PROBLEM - Puppet freshness on mw8 is CRITICAL: No successful Puppet run in the last 10 hours [09:14:12] PROBLEM - Puppet freshness on pdf2 is CRITICAL: No successful Puppet run in the last 10 hours [09:14:12] PROBLEM - Puppet freshness on search36 is CRITICAL: No successful Puppet run in the last 10 hours [09:14:12] PROBLEM - Puppet freshness on solr2 is CRITICAL: No successful Puppet run in the last 10 hours [09:14:12] PROBLEM - Puppet freshness on sq51 is CRITICAL: No successful Puppet run in the last 10 hours [09:14:12] PROBLEM - Puppet freshness on srv301 is CRITICAL: No successful Puppet run in the last 10 hours [09:14:12] PROBLEM - Puppet freshness on wtp1011 is CRITICAL: No successful Puppet run in the last 10 hours [09:14:12] PROBLEM - Puppet freshness on wtp1018 is CRITICAL: No successful Puppet run in the last 10 hours [09:14:32] qchris: ahh novadmin might be the LDAP username used by Gerrit [09:14:41] which would mean a slowness on LDAP side (on virt0 ? ) [09:14:58] I have noticed that a few weeks ago, doing a query with something like: owner:ldap/ops would be hugely slow [09:15:05] PROBLEM - Puppet freshness on analytics1023 is CRITICAL: No successful Puppet run in the last 10 hours [09:15:05] PROBLEM - Puppet freshness on amssq44 is CRITICAL: No successful Puppet run in the last 10 hours [09:15:05] PROBLEM - Puppet freshness on cp1063 is CRITICAL: No successful Puppet run in the last 10 hours [09:15:05] PROBLEM - Puppet freshness on db1004 is CRITICAL: No successful Puppet run in the last 10 hours [09:15:05] PROBLEM - Puppet freshness on db43 is CRITICAL: No successful Puppet run in the last 10 hours [09:15:07] Remote was virt0.wikimedia.org, yes. [09:16:40] Gerrit slowness appears to have been resolved. [09:17:11] and virt0 is not in ganglia :( [09:17:32] New review: Nikerabbit; "Trying again" [operations/mediawiki-config] (master) C: 2; - https://gerrit.wikimedia.org/r/73934 [09:17:51] Change merged: jenkins-bot; [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/73934 [09:18:59] at least zuul merges :-] [09:19:05] :-D [09:19:08] siebrand: so yeah sorry, seems something went wrong inside Gerrit [09:19:50] siebrand: the course of actions I take is usually: lookup the status of manganese in Ganglia, try to do some gerrit commands over ssh and then ask ops to restart the service. [09:20:14] k [09:20:24] I thought you were the one that would restart Gerrit. [09:20:55] Should I open bugzilla tickets and RT tickets in the future? [09:21:15] I guess RT [09:21:29] ops uses bugzilla only for labs infrastructure [09:21:34] all the prod stuff is handled in RT [09:22:18] not having open tickets on issues with Gerrit means that no Gerrit users can be aware of what's going on. [09:22:21] and I don't think we have a policy of always opening a ticket on incident, so lot of those issues are handled on IRC + !log [09:22:32] (as most people have no access (or even awareness) of RT. [09:23:07] The problem is that it's not "set it and forget about it" for me. [09:23:31] I have a problem. I report it's coming from two ways. You say WFM. [09:23:39] Then there's nothing with your workflow. [09:23:50] Then Daniel Kinzler complains in another channel. [09:24:07] I report he's also observing something different from expected. [09:24:27] cause I was coding some python script, context switched, and my basic assumption was it seemed to works for me [09:24:33] aka I did have access to the gerrit web ui [09:24:41] and managed to do some git-review [09:24:56] finished my script then tried out fetching all extensions and noticed the slowness :-] [09:25:14] PROBLEM - Puppet freshness on amssq34 is CRITICAL: No successful Puppet run in the last 10 hours [09:25:14] PROBLEM - Puppet freshness on cp1002 is CRITICAL: No successful Puppet run in the last 10 hours [09:25:14] PROBLEM - Puppet freshness on cp1012 is CRITICAL: No successful Puppet run in the last 10 hours [09:25:14] PROBLEM - Puppet freshness on db1023 is CRITICAL: No successful Puppet run in the last 10 hours [09:25:14] PROBLEM - Puppet freshness on db34 is CRITICAL: No successful Puppet run in the last 10 hours [09:25:14] PROBLEM - Puppet freshness on es1005 is CRITICAL: No successful Puppet run in the last 10 hours [09:25:14] PROBLEM - Puppet freshness on mc1005 is CRITICAL: No successful Puppet run in the last 10 hours [09:25:14] PROBLEM - Puppet freshness on mw1139 is CRITICAL: No successful Puppet run in the last 10 hours [09:25:14] PROBLEM - Puppet freshness on pc1003 is CRITICAL: No successful Puppet run in the last 10 hours [09:25:14] PROBLEM - Puppet freshness on search1018 is CRITICAL: No successful Puppet run in the last 10 hours [09:25:14] PROBLEM - Puppet freshness on srv258 is CRITICAL: No successful Puppet run in the last 10 hours [09:26:05] PROBLEM - Puppet freshness on cp1046 is CRITICAL: No successful Puppet run in the last 10 hours [09:26:05] PROBLEM - Puppet freshness on db40 is CRITICAL: No successful Puppet run in the last 10 hours [09:26:05] PROBLEM - Puppet freshness on fenari is CRITICAL: No successful Puppet run in the last 10 hours [09:26:05] PROBLEM - Puppet freshness on ms-be1 is CRITICAL: No successful Puppet run in the last 10 hours [09:26:05] PROBLEM - Puppet freshness on mw20 is CRITICAL: No successful Puppet run in the last 10 hours [09:26:05] PROBLEM - Puppet freshness on search14 is CRITICAL: No successful Puppet run in the last 10 hours [09:27:35] !log upgrading packages on gallium [09:27:46] Logged the message, Master [09:29:51] so I am filling a RT to get ganglia on some misc virt devices [09:34:05] PROBLEM - Puppet freshness on erzurumi is CRITICAL: No successful Puppet run in the last 10 hours [09:34:05] PROBLEM - Puppet freshness on lvs1004 is CRITICAL: No successful Puppet run in the last 10 hours [09:34:05] PROBLEM - Puppet freshness on lvs1005 is CRITICAL: No successful Puppet run in the last 10 hours [09:34:05] PROBLEM - Puppet freshness on lvs1006 is CRITICAL: No successful Puppet run in the last 10 hours [09:34:05] PROBLEM - Puppet freshness on mw1173 is CRITICAL: No successful Puppet run in the last 10 hours [09:34:05] PROBLEM - Puppet freshness on virt1 is CRITICAL: No successful Puppet run in the last 10 hours [09:34:05] PROBLEM - Puppet freshness on virt4 is CRITICAL: No successful Puppet run in the last 10 hours [09:34:06] PROBLEM - Puppet freshness on virt3 is CRITICAL: No successful Puppet run in the last 10 hours [09:36:38] New patchset: Aude; "Add WikibaseDataModel dependency for wikidata singlenode" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/73940 [09:37:44] mouth of watercourse [09:37:51] gah [09:40:45] !log Resumed reporting of MediaWiki fatals & exceptions to Ganglia [09:40:57] Logged the message, Master [09:41:10] https://rt.wikimedia.org/Ticket/Display.html?id=5472 :) [09:41:14] * ori-l goes to sleep [09:41:18] ori-l: sleep well [09:41:26] nacht! [09:41:38] thanks, tschuss, cya [09:48:12] !log nikerabbit synchronized php-1.22wmf10/extensions/UniversalLanguageSelector/ 'ULS to master' [09:48:22] Logged the message, Master [09:52:16] !log nikerabbit synchronized php-1.22wmf9/extensions/UniversalLanguageSelector/ 'ULS to master' [09:52:25] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:52:27] Logged the message, Master [09:53:15] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.151 second response time [09:54:10] !log nikerabbit synchronized wmf-config/InitialiseSettings.php 'ULS: Bug 51019' [09:54:20] Logged the message, Master [10:10:28] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:10:55] New patchset: Hashar; "mobile $wgLoadScript now uses $wmfHostnames['bits']" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/71779 [10:11:08] PROBLEM - SSH on cp1044 is CRITICAL: Server answer: [10:11:18] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.139 second response time [10:15:39] New patchset: Hashar; "$wgCopyrightIcon now uses $wmfHostnames['bits']" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/71776 [10:16:08] RECOVERY - SSH on cp1044 is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1.1 (protocol 2.0) [10:18:38] Change merged: jenkins-bot; [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/71776 [10:19:18] PROBLEM - Host cp3011 is DOWN: PING CRITICAL - Packet loss = 100% [10:20:18] RECOVERY - Host cp3011 is UP: PING OK - Packet loss = 0%, RTA = 88.16 ms [10:20:32] !log hashar synchronized wmf-config/CommonSettings.php '$wgCopyrightIcon now vary using $wmfHostnames[bits]' [10:20:42] Logged the message, Master [10:21:17] !log hashar synchronized wmf-config/InitialiseSettings.php '$wgCopyrightIcon now vary using $wmfHostnames[bits]' [10:21:28] Logged the message, Master [10:21:59] New review: Hashar; "deployed" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/71776 [10:22:28] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:23:03] New patchset: Hashar; "$wgEventLoggingBaseUri now uses $wmfHostnames['bits']" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/71777 [10:23:18] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.136 second response time [10:25:34] New review: Hashar; "I was just waiting for some deployment slot :-)" [operations/mediawiki-config] (master) C: 2; - https://gerrit.wikimedia.org/r/71777 [10:25:34] PROBLEM - Host cp3012 is DOWN: PING CRITICAL - Packet loss = 100% [10:25:36] Change merged: jenkins-bot; [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/71777 [10:26:18] RECOVERY - Host cp3012 is UP: PING OK - Packet loss = 0%, RTA = 89.22 ms [10:28:34] !log hashar synchronized wmf-config/CommonSettings.php ' now uses ['bits'] {{gerrit|71777}}' [10:28:45] Logged the message, Master [10:29:09] !log hashar synchronized wmf-config/InitialiseSettings.php ' now uses ['bits'] {{gerrit|71777}}' [10:29:19] Logged the message, Master [10:29:45] New review: Hashar; "deployed:" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/71777 [10:33:28] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:36:45] New review: Odder; "No need to do this with a Gerrit patch." [operations/mediawiki-config] (master) C: -1; - https://gerrit.wikimedia.org/r/73919 [10:38:18] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.131 second response time [10:43:28] RECOVERY - Puppet freshness on mw1025 is OK: puppet ran at Tue Jul 16 10:43:22 UTC 2013 [10:44:06] New patchset: ArielGlenn; "fix syntax error in icinga snmptt trap reporting: submit_check_result" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/73946 [10:44:48] come on jenkins [10:45:01] Change merged: ArielGlenn; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/73946 [10:46:04] and now it's come on neon puppet run [10:52:57] too many facts [10:57:28] RECOVERY - Puppet freshness on mw17 is OK: puppet ran at Tue Jul 16 10:57:25 UTC 2013 [10:57:28] RECOVERY - Puppet freshness on mw63 is OK: puppet ran at Tue Jul 16 10:57:25 UTC 2013 [10:57:28] RECOVERY - Puppet freshness on mw1013 is OK: puppet ran at Tue Jul 16 10:57:25 UTC 2013 [10:57:28] RECOVERY - Puppet freshness on mw1167 is OK: puppet ran at Tue Jul 16 10:57:25 UTC 2013 [10:57:38] RECOVERY - Puppet freshness on snapshot1001 is OK: puppet ran at Tue Jul 16 10:57:30 UTC 2013 [10:57:38] RECOVERY - Puppet freshness on sq53 is OK: puppet ran at Tue Jul 16 10:57:30 UTC 2013 [10:57:38] RECOVERY - Puppet freshness on sq44 is OK: puppet ran at Tue Jul 16 10:57:36 UTC 2013 [10:57:38] RECOVERY - Puppet freshness on es3 is OK: puppet ran at Tue Jul 16 10:57:36 UTC 2013 [10:57:38] RECOVERY - Puppet freshness on nfs1 is OK: puppet ran at Tue Jul 16 10:57:36 UTC 2013 [10:57:38] RECOVERY - Puppet freshness on db1007 is OK: puppet ran at Tue Jul 16 10:57:36 UTC 2013 [10:57:48] RECOVERY - Puppet freshness on mc1014 is OK: puppet ran at Tue Jul 16 10:57:41 UTC 2013 [10:57:48] RECOVERY - Puppet freshness on mc1010 is OK: puppet ran at Tue Jul 16 10:57:46 UTC 2013 [10:57:48] RECOVERY - Puppet freshness on sq69 is OK: puppet ran at Tue Jul 16 10:57:46 UTC 2013 [10:57:58] RECOVERY - Puppet freshness on sq36 is OK: puppet ran at Tue Jul 16 10:57:51 UTC 2013 [10:57:58] RECOVERY - Puppet freshness on ssl3003 is OK: puppet ran at Tue Jul 16 10:57:51 UTC 2013 [10:57:58] RECOVERY - Puppet freshness on es1006 is OK: puppet ran at Tue Jul 16 10:57:51 UTC 2013 [10:57:58] RECOVERY - Puppet freshness on analytics1026 is OK: puppet ran at Tue Jul 16 10:57:56 UTC 2013 [10:58:14] yeah we're gong to see this for the next half hour or so I guess [10:58:14] RECOVERY - Puppet freshness on mw32 is OK: puppet ran at Tue Jul 16 10:58:06 UTC 2013 [10:58:14] RECOVERY - Puppet freshness on mw1168 is OK: puppet ran at Tue Jul 16 10:58:06 UTC 2013 [10:58:14] RECOVERY - Puppet freshness on mw88 is OK: puppet ran at Tue Jul 16 10:58:08 UTC 2013 [10:58:14] RECOVERY - Puppet freshness on db1039 is OK: puppet ran at Tue Jul 16 10:58:08 UTC 2013 [10:58:14] RECOVERY - Puppet freshness on cp3022 is OK: puppet ran at Tue Jul 16 10:58:08 UTC 2013 [10:58:14] RECOVERY - Puppet freshness on mw1109 is OK: puppet ran at Tue Jul 16 10:58:08 UTC 2013 [10:58:14] RECOVERY - Puppet freshness on mw1202 is OK: puppet ran at Tue Jul 16 10:58:08 UTC 2013 [10:58:15] RECOVERY - Puppet freshness on mw1192 is OK: puppet ran at Tue Jul 16 10:58:08 UTC 2013 [10:58:15] RECOVERY - Puppet freshness on mw18 is OK: puppet ran at Tue Jul 16 10:58:08 UTC 2013 [10:58:16] RECOVERY - Puppet freshness on mw1098 is OK: puppet ran at Tue Jul 16 10:58:13 UTC 2013 [10:58:28] then it wil settle down, sorry [10:58:34] RECOVERY - Puppet freshness on pdf1 is OK: puppet ran at Tue Jul 16 10:58:33 UTC 2013 [10:58:44] RECOVERY - Puppet freshness on mw1151 is OK: puppet ran at Tue Jul 16 10:58:43 UTC 2013 [10:58:44] RECOVERY - Puppet freshness on search24 is OK: puppet ran at Tue Jul 16 10:58:43 UTC 2013 [10:58:44] RECOVERY - Puppet freshness on cp1011 is OK: puppet ran at Tue Jul 16 10:58:43 UTC 2013 [10:58:44] RECOVERY - Puppet freshness on analytics1013 is OK: puppet ran at Tue Jul 16 10:58:43 UTC 2013 [10:58:54] RECOVERY - Puppet freshness on db1016 is OK: puppet ran at Tue Jul 16 10:58:48 UTC 2013 [10:58:54] RECOVERY - Puppet freshness on search1002 is OK: puppet ran at Tue Jul 16 10:58:48 UTC 2013 [10:58:54] RECOVERY - Puppet freshness on iron is OK: puppet ran at Tue Jul 16 10:58:48 UTC 2013 [10:58:54] RECOVERY - Puppet freshness on mw1014 is OK: puppet ran at Tue Jul 16 10:58:48 UTC 2013 [10:58:54] RECOVERY - Puppet freshness on oxygen is OK: puppet ran at Tue Jul 16 10:58:48 UTC 2013 [10:58:54] RECOVERY - Puppet freshness on mw1079 is OK: puppet ran at Tue Jul 16 10:58:48 UTC 2013 [10:58:54] RECOVERY - Puppet freshness on rdb1001 is OK: puppet ran at Tue Jul 16 10:58:48 UTC 2013 [10:58:55] RECOVERY - Puppet freshness on sq70 is OK: puppet ran at Tue Jul 16 10:58:48 UTC 2013 [10:58:55] RECOVERY - Puppet freshness on cp1004 is OK: puppet ran at Tue Jul 16 10:58:53 UTC 2013 [10:58:56] RECOVERY - Puppet freshness on mw1051 is OK: puppet ran at Tue Jul 16 10:58:53 UTC 2013 [10:58:56] RECOVERY - Puppet freshness on mw1133 is OK: puppet ran at Tue Jul 16 10:58:53 UTC 2013 [10:58:57] RECOVERY - Puppet freshness on mw86 is OK: puppet ran at Tue Jul 16 10:58:53 UTC 2013 [10:58:57] RECOVERY - Puppet freshness on db1030 is OK: puppet ran at Tue Jul 16 10:58:53 UTC 2013 [10:58:58] RECOVERY - Puppet freshness on formey is OK: puppet ran at Tue Jul 16 10:58:53 UTC 2013 [10:58:58] RECOVERY - Puppet freshness on db49 is OK: puppet ran at Tue Jul 16 10:58:53 UTC 2013 [10:58:59] RECOVERY - Puppet freshness on ms-be1008 is OK: puppet ran at Tue Jul 16 10:58:53 UTC 2013 [10:58:59] RECOVERY - Puppet freshness on mw1216 is OK: puppet ran at Tue Jul 16 10:58:53 UTC 2013 [10:59:00] RECOVERY - Puppet freshness on db74 is OK: puppet ran at Tue Jul 16 10:58:53 UTC 2013 [10:59:00] RECOVERY - Puppet freshness on mc1004 is OK: puppet ran at Tue Jul 16 10:58:53 UTC 2013 [10:59:01] RECOVERY - Puppet freshness on db72 is OK: puppet ran at Tue Jul 16 10:58:53 UTC 2013 [10:59:01] RECOVERY - Puppet freshness on cp1048 is OK: puppet ran at Tue Jul 16 10:58:53 UTC 2013 [10:59:04] RECOVERY - Puppet freshness on srv263 is OK: puppet ran at Tue Jul 16 10:58:53 UTC 2013 [10:59:04] RECOVERY - Puppet freshness on mw85 is OK: puppet ran at Tue Jul 16 10:58:53 UTC 2013 [10:59:04] RECOVERY - Puppet freshness on mw97 is OK: puppet ran at Tue Jul 16 10:58:53 UTC 2013 [10:59:04] RECOVERY - Puppet freshness on mw1183 is OK: puppet ran at Tue Jul 16 10:58:53 UTC 2013 [10:59:04] RECOVERY - Puppet freshness on mw72 is OK: puppet ran at Tue Jul 16 10:58:59 UTC 2013 [10:59:04] RECOVERY - Puppet freshness on mw123 is OK: puppet ran at Tue Jul 16 10:58:59 UTC 2013 [10:59:05] RECOVERY - Puppet freshness on hooft is OK: puppet ran at Tue Jul 16 10:58:59 UTC 2013 [10:59:14] RECOVERY - Puppet freshness on mw1034 is OK: puppet ran at Tue Jul 16 10:59:04 UTC 2013 [10:59:14] RECOVERY - Puppet freshness on mw1050 is OK: puppet ran at Tue Jul 16 10:59:04 UTC 2013 [10:59:14] RECOVERY - Puppet freshness on virt1007 is OK: puppet ran at Tue Jul 16 10:59:04 UTC 2013 [10:59:14] RECOVERY - Puppet freshness on amslvs3 is OK: puppet ran at Tue Jul 16 10:59:04 UTC 2013 [10:59:14] RECOVERY - Puppet freshness on cp1041 is OK: puppet ran at Tue Jul 16 10:59:09 UTC 2013 [10:59:14] RECOVERY - Puppet freshness on search32 is OK: puppet ran at Tue Jul 16 10:59:09 UTC 2013 [10:59:20] only 800 hosts... :-P [10:59:24] RECOVERY - Puppet freshness on ssl3002 is OK: puppet ran at Tue Jul 16 10:59:14 UTC 2013 [10:59:24] RECOVERY - Puppet freshness on mw1156 is OK: puppet ran at Tue Jul 16 10:59:19 UTC 2013 [10:59:24] RECOVERY - Puppet freshness on search20 is OK: puppet ran at Tue Jul 16 10:59:19 UTC 2013 [10:59:24] RECOVERY - Puppet freshness on mw78 is OK: puppet ran at Tue Jul 16 10:59:19 UTC 2013 [10:59:34] RECOVERY - Puppet freshness on virt7 is OK: puppet ran at Tue Jul 16 10:59:24 UTC 2013 [10:59:34] RECOVERY - Puppet freshness on amssq46 is OK: puppet ran at Tue Jul 16 10:59:24 UTC 2013 [10:59:34] RECOVERY - Puppet freshness on searchidx2 is OK: puppet ran at Tue Jul 16 10:59:29 UTC 2013 [10:59:34] RECOVERY - Puppet freshness on ms1002 is OK: puppet ran at Tue Jul 16 10:59:29 UTC 2013 [10:59:39] ignoring icinga-wm was a good decision [10:59:44] PROBLEM - Puppet freshness on cp1041 is CRITICAL: No successful Puppet run in the last 10 hours [10:59:44] RECOVERY - Puppet freshness on virt10 is OK: puppet ran at Tue Jul 16 10:59:39 UTC 2013 [10:59:44] RECOVERY - Puppet freshness on mw4 is OK: puppet ran at Tue Jul 16 10:59:39 UTC 2013 [10:59:44] RECOVERY - Puppet freshness on db1059 is OK: puppet ran at Tue Jul 16 10:59:39 UTC 2013 [10:59:54] RECOVERY - Puppet freshness on es8 is OK: puppet ran at Tue Jul 16 10:59:44 UTC 2013 [10:59:54] RECOVERY - Puppet freshness on wtp1022 is OK: puppet ran at Tue Jul 16 10:59:44 UTC 2013 [10:59:54] RECOVERY - Puppet freshness on virt5 is OK: puppet ran at Tue Jul 16 10:59:49 UTC 2013 [10:59:54] RECOVERY - Puppet freshness on amssq62 is OK: puppet ran at Tue Jul 16 10:59:49 UTC 2013 [11:00:04] RECOVERY - Puppet freshness on mw1212 is OK: puppet ran at Tue Jul 16 10:59:54 UTC 2013 [11:00:04] RECOVERY - Puppet freshness on srv279 is OK: puppet ran at Tue Jul 16 10:59:54 UTC 2013 [11:00:04] RECOVERY - Puppet freshness on mw102 is OK: puppet ran at Tue Jul 16 10:59:59 UTC 2013 [11:00:04] RECOVERY - Puppet freshness on mw6 is OK: puppet ran at Tue Jul 16 10:59:59 UTC 2013 [11:00:04] RECOVERY - Puppet freshness on mw27 is OK: puppet ran at Tue Jul 16 10:59:59 UTC 2013 [11:00:04] RECOVERY - Puppet freshness on mw1097 is OK: puppet ran at Tue Jul 16 11:00:00 UTC 2013 [11:00:04] RECOVERY - Puppet freshness on mw1002 is OK: puppet ran at Tue Jul 16 11:00:00 UTC 2013 [11:00:05] RECOVERY - Puppet freshness on mw44 is OK: puppet ran at Tue Jul 16 11:00:00 UTC 2013 [11:00:05] RECOVERY - Puppet freshness on mw1077 is OK: puppet ran at Tue Jul 16 11:00:00 UTC 2013 [11:00:14] RECOVERY - Puppet freshness on cp1055 is OK: puppet ran at Tue Jul 16 11:00:05 UTC 2013 [11:00:14] RECOVERY - Puppet freshness on db1053 is OK: puppet ran at Tue Jul 16 11:00:05 UTC 2013 [11:00:14] RECOVERY - Puppet freshness on cp1044 is OK: puppet ran at Tue Jul 16 11:00:05 UTC 2013 [11:00:14] RECOVERY - Puppet freshness on wtp1023 is OK: puppet ran at Tue Jul 16 11:00:10 UTC 2013 [11:00:14] RECOVERY - Puppet freshness on srv288 is OK: puppet ran at Tue Jul 16 11:00:10 UTC 2013 [11:00:14] RECOVERY - Puppet freshness on ssl3 is OK: puppet ran at Tue Jul 16 11:00:10 UTC 2013 [11:00:14] RECOVERY - Puppet freshness on srv241 is OK: puppet ran at Tue Jul 16 11:00:10 UTC 2013 [11:00:24] RECOVERY - Puppet freshness on snapshot1004 is OK: puppet ran at Tue Jul 16 11:00:15 UTC 2013 [11:00:24] RECOVERY - Puppet freshness on mw1029 is OK: puppet ran at Tue Jul 16 11:00:15 UTC 2013 [11:00:24] RECOVERY - Puppet freshness on db50 is OK: puppet ran at Tue Jul 16 11:00:15 UTC 2013 [11:00:24] RECOVERY - Puppet freshness on search1016 is OK: puppet ran at Tue Jul 16 11:00:20 UTC 2013 [11:00:24] RECOVERY - Puppet freshness on mw1023 is OK: puppet ran at Tue Jul 16 11:00:20 UTC 2013 [11:00:24] RECOVERY - Puppet freshness on search1017 is OK: puppet ran at Tue Jul 16 11:00:20 UTC 2013 [11:00:25] RECOVERY - Puppet freshness on mw1055 is OK: puppet ran at Tue Jul 16 11:00:20 UTC 2013 [11:00:25] RECOVERY - Puppet freshness on mw1096 is OK: puppet ran at Tue Jul 16 11:00:20 UTC 2013 [11:00:25] PROBLEM - Puppet freshness on cp1044 is CRITICAL: No successful Puppet run in the last 10 hours [11:00:34] RECOVERY - Puppet freshness on sq80 is OK: puppet ran at Tue Jul 16 11:00:25 UTC 2013 [11:00:34] RECOVERY - Puppet freshness on ms1001 is OK: puppet ran at Tue Jul 16 11:00:30 UTC 2013 [11:00:34] RECOVERY - Puppet freshness on amssq38 is OK: puppet ran at Tue Jul 16 11:00:30 UTC 2013 [11:00:44] RECOVERY - Puppet freshness on db47 is OK: puppet ran at Tue Jul 16 11:00:35 UTC 2013 [11:00:44] RECOVERY - Puppet freshness on db1008 is OK: puppet ran at Tue Jul 16 11:00:35 UTC 2013 [11:00:44] RECOVERY - Puppet freshness on db1006 is OK: puppet ran at Tue Jul 16 11:00:35 UTC 2013 [11:00:44] RECOVERY - Puppet freshness on rdb1004 is OK: puppet ran at Tue Jul 16 11:00:35 UTC 2013 [11:00:44] RECOVERY - Puppet freshness on pc3 is OK: puppet ran at Tue Jul 16 11:00:40 UTC 2013 [11:00:44] RECOVERY - Puppet freshness on cp1052 is OK: puppet ran at Tue Jul 16 11:00:40 UTC 2013 [11:00:44] RECOVERY - Puppet freshness on locke is OK: puppet ran at Tue Jul 16 11:00:40 UTC 2013 [11:00:45] RECOVERY - Puppet freshness on ms-be1001 is OK: puppet ran at Tue Jul 16 11:00:40 UTC 2013 [11:00:54] RECOVERY - Puppet freshness on db62 is OK: puppet ran at Tue Jul 16 11:00:45 UTC 2013 [11:00:54] RECOVERY - Puppet freshness on db35 is OK: puppet ran at Tue Jul 16 11:00:45 UTC 2013 [11:00:54] RECOVERY - Puppet freshness on wtp1019 is OK: puppet ran at Tue Jul 16 11:00:45 UTC 2013 [11:00:54] RECOVERY - Puppet freshness on ms-be1011 is OK: puppet ran at Tue Jul 16 11:00:45 UTC 2013 [11:00:54] RECOVERY - Puppet freshness on pc1001 is OK: puppet ran at Tue Jul 16 11:00:45 UTC 2013 [11:00:54] RECOVERY - Puppet freshness on wtp1017 is OK: puppet ran at Tue Jul 16 11:00:45 UTC 2013 [11:00:54] RECOVERY - Puppet freshness on db1047 is OK: puppet ran at Tue Jul 16 11:00:45 UTC 2013 [11:00:55] RECOVERY - Puppet freshness on mw89 is OK: puppet ran at Tue Jul 16 11:00:50 UTC 2013 [11:00:55] RECOVERY - Puppet freshness on cp1059 is OK: puppet ran at Tue Jul 16 11:00:50 UTC 2013 [11:01:04] RECOVERY - Puppet freshness on mw21 is OK: puppet ran at Tue Jul 16 11:00:56 UTC 2013 [11:01:04] RECOVERY - Puppet freshness on analytics1027 is OK: puppet ran at Tue Jul 16 11:00:56 UTC 2013 [11:01:04] RECOVERY - Puppet freshness on ms-be6 is OK: puppet ran at Tue Jul 16 11:00:56 UTC 2013 [11:01:04] RECOVERY - Puppet freshness on mw1209 is OK: puppet ran at Tue Jul 16 11:00:56 UTC 2013 [11:01:04] RECOVERY - Puppet freshness on mw1152 is OK: puppet ran at Tue Jul 16 11:00:56 UTC 2013 [11:01:04] RECOVERY - Puppet freshness on mw1005 is OK: puppet ran at Tue Jul 16 11:01:01 UTC 2013 [11:01:04] RECOVERY - Puppet freshness on search1020 is OK: puppet ran at Tue Jul 16 11:01:01 UTC 2013 [11:01:05] RECOVERY - Puppet freshness on mw1028 is OK: puppet ran at Tue Jul 16 11:01:01 UTC 2013 [11:01:05] RECOVERY - Puppet freshness on mw1067 is OK: puppet ran at Tue Jul 16 11:01:01 UTC 2013 [11:01:06] RECOVERY - Puppet freshness on es1010 is OK: puppet ran at Tue Jul 16 11:01:01 UTC 2013 [11:01:14] RECOVERY - Puppet freshness on mw1016 is OK: puppet ran at Tue Jul 16 11:01:06 UTC 2013 [11:01:44] RECOVERY - Puppet freshness on cp1067 is OK: puppet ran at Tue Jul 16 11:01:37 UTC 2013 [11:01:44] RECOVERY - Puppet freshness on db71 is OK: puppet ran at Tue Jul 16 11:01:37 UTC 2013 [11:01:44] RECOVERY - Puppet freshness on cp1014 is OK: puppet ran at Tue Jul 16 11:01:37 UTC 2013 [11:01:44] RECOVERY - Puppet freshness on es1001 is OK: puppet ran at Tue Jul 16 11:01:38 UTC 2013 [11:01:44] RECOVERY - Puppet freshness on db1017 is OK: puppet ran at Tue Jul 16 11:01:38 UTC 2013 [11:01:44] RECOVERY - Puppet freshness on sq60 is OK: puppet ran at Tue Jul 16 11:01:38 UTC 2013 [11:01:45] RECOVERY - Puppet freshness on amssq54 is OK: puppet ran at Tue Jul 16 11:01:38 UTC 2013 [11:01:45] RECOVERY - Puppet freshness on carbon is OK: puppet ran at Tue Jul 16 11:01:38 UTC 2013 [11:01:46] RECOVERY - Puppet freshness on kaulen is OK: puppet ran at Tue Jul 16 11:01:38 UTC 2013 [11:01:46] RECOVERY - Puppet freshness on zhen is OK: puppet ran at Tue Jul 16 11:01:38 UTC 2013 [11:01:47] RECOVERY - Puppet freshness on amssq49 is OK: puppet ran at Tue Jul 16 11:01:38 UTC 2013 [11:01:47] RECOVERY - Puppet freshness on wtp1009 is OK: puppet ran at Tue Jul 16 11:01:43 UTC 2013 [11:01:48] RECOVERY - Puppet freshness on cp1040 is OK: puppet ran at Tue Jul 16 11:01:43 UTC 2013 [11:01:48] RECOVERY - Puppet freshness on srv265 is OK: puppet ran at Tue Jul 16 11:01:43 UTC 2013 [11:01:49] RECOVERY - Puppet freshness on amssq45 is OK: puppet ran at Tue Jul 16 11:01:43 UTC 2013 [11:01:49] RECOVERY - Puppet freshness on srv248 is OK: puppet ran at Tue Jul 16 11:01:43 UTC 2013 [11:01:50] RECOVERY - Puppet freshness on snapshot2 is OK: puppet ran at Tue Jul 16 11:01:43 UTC 2013 [11:01:50] RECOVERY - Puppet freshness on mw24 is OK: puppet ran at Tue Jul 16 11:01:43 UTC 2013 [11:01:54] RECOVERY - Puppet freshness on virt6 is OK: puppet ran at Tue Jul 16 11:01:48 UTC 2013 [11:01:54] RECOVERY - Puppet freshness on lvs1003 is OK: puppet ran at Tue Jul 16 11:01:48 UTC 2013 [11:01:54] RECOVERY - Puppet freshness on amssq42 is OK: puppet ran at Tue Jul 16 11:01:48 UTC 2013 [11:02:04] RECOVERY - Puppet freshness on cp3020 is OK: puppet ran at Tue Jul 16 11:01:58 UTC 2013 [11:02:04] RECOVERY - Puppet freshness on mw73 is OK: puppet ran at Tue Jul 16 11:01:58 UTC 2013 [11:02:04] RECOVERY - Puppet freshness on mw1193 is OK: puppet ran at Tue Jul 16 11:01:58 UTC 2013 [11:02:24] RECOVERY - Puppet freshness on amssq33 is OK: puppet ran at Tue Jul 16 11:02:18 UTC 2013 [11:02:34] RECOVERY - Puppet freshness on yvon is OK: puppet ran at Tue Jul 16 11:02:28 UTC 2013 [11:02:34] RECOVERY - Puppet freshness on mw77 is OK: puppet ran at Tue Jul 16 11:02:28 UTC 2013 [11:02:34] RECOVERY - Puppet freshness on ms-fe1 is OK: puppet ran at Tue Jul 16 11:02:28 UTC 2013 [11:02:34] RECOVERY - Puppet freshness on db1038 is OK: puppet ran at Tue Jul 16 11:02:28 UTC 2013 [11:02:34] RECOVERY - Puppet freshness on ssl2 is OK: puppet ran at Tue Jul 16 11:02:28 UTC 2013 [11:02:34] RECOVERY - Puppet freshness on cp3008 is OK: puppet ran at Tue Jul 16 11:02:28 UTC 2013 [11:02:34] RECOVERY - Puppet freshness on mw1012 is OK: puppet ran at Tue Jul 16 11:02:28 UTC 2013 [11:02:35] RECOVERY - Puppet freshness on eeden is OK: puppet ran at Tue Jul 16 11:02:33 UTC 2013 [11:02:35] RECOVERY - Puppet freshness on tridge is OK: puppet ran at Tue Jul 16 11:02:33 UTC 2013 [11:02:36] RECOVERY - Puppet freshness on amssq39 is OK: puppet ran at Tue Jul 16 11:02:33 UTC 2013 [11:02:36] RECOVERY - Puppet freshness on cp1070 is OK: puppet ran at Tue Jul 16 11:02:33 UTC 2013 [11:02:44] RECOVERY - Puppet freshness on cp1043 is OK: puppet ran at Tue Jul 16 11:02:38 UTC 2013 [11:02:44] RECOVERY - Puppet freshness on db1035 is OK: puppet ran at Tue Jul 16 11:02:43 UTC 2013 [11:02:44] RECOVERY - Puppet freshness on xenon is OK: puppet ran at Tue Jul 16 11:02:43 UTC 2013 [11:02:54] RECOVERY - Puppet freshness on cp3004 is OK: puppet ran at Tue Jul 16 11:02:48 UTC 2013 [11:02:54] RECOVERY - Puppet freshness on cp3005 is OK: puppet ran at Tue Jul 16 11:02:48 UTC 2013 [11:03:04] RECOVERY - Puppet freshness on cp1047 is OK: puppet ran at Tue Jul 16 11:02:53 UTC 2013 [11:03:04] RECOVERY - Puppet freshness on mw31 is OK: puppet ran at Tue Jul 16 11:02:53 UTC 2013 [11:03:04] RECOVERY - Puppet freshness on mw1071 is OK: puppet ran at Tue Jul 16 11:02:58 UTC 2013 [11:03:04] RECOVERY - Puppet freshness on mw1006 is OK: puppet ran at Tue Jul 16 11:02:58 UTC 2013 [11:03:04] RECOVERY - Puppet freshness on amssq61 is OK: puppet ran at Tue Jul 16 11:02:58 UTC 2013 [11:03:04] RECOVERY - Puppet freshness on mw1064 is OK: puppet ran at Tue Jul 16 11:02:58 UTC 2013 [11:03:04] RECOVERY - Puppet freshness on mw19 is OK: puppet ran at Tue Jul 16 11:02:58 UTC 2013 [11:03:05] RECOVERY - Puppet freshness on virt9 is OK: puppet ran at Tue Jul 16 11:02:58 UTC 2013 [11:03:05] RECOVERY - Puppet freshness on search34 is OK: puppet ran at Tue Jul 16 11:02:58 UTC 2013 [11:03:06] RECOVERY - Puppet freshness on mw22 is OK: puppet ran at Tue Jul 16 11:02:58 UTC 2013 [11:03:06] RECOVERY - Puppet freshness on search15 is OK: puppet ran at Tue Jul 16 11:02:58 UTC 2013 [11:03:07] RECOVERY - Puppet freshness on srv256 is OK: puppet ran at Tue Jul 16 11:02:58 UTC 2013 [11:03:14] RECOVERY - Puppet freshness on mw108 is OK: puppet ran at Tue Jul 16 11:03:04 UTC 2013 [11:03:14] RECOVERY - Puppet freshness on analytics1025 is OK: puppet ran at Tue Jul 16 11:03:04 UTC 2013 [11:03:14] RECOVERY - Puppet freshness on analytics1005 is OK: puppet ran at Tue Jul 16 11:03:04 UTC 2013 [11:03:14] RECOVERY - Puppet freshness on srv295 is OK: puppet ran at Tue Jul 16 11:03:04 UTC 2013 [11:03:14] RECOVERY - Puppet freshness on snapshot1003 is OK: puppet ran at Tue Jul 16 11:03:04 UTC 2013 [11:03:14] RECOVERY - Puppet freshness on search28 is OK: puppet ran at Tue Jul 16 11:03:04 UTC 2013 [11:03:14] RECOVERY - Puppet freshness on mw1187 is OK: puppet ran at Tue Jul 16 11:03:09 UTC 2013 [11:03:15] RECOVERY - Puppet freshness on db1037 is OK: puppet ran at Tue Jul 16 11:03:09 UTC 2013 [11:03:15] RECOVERY - Puppet freshness on mw54 is OK: puppet ran at Tue Jul 16 11:03:09 UTC 2013 [11:03:16] RECOVERY - Puppet freshness on mw1207 is OK: puppet ran at Tue Jul 16 11:03:09 UTC 2013 [11:03:24] RECOVERY - Puppet freshness on mw1037 is OK: puppet ran at Tue Jul 16 11:03:14 UTC 2013 [11:03:24] RECOVERY - Puppet freshness on mw1113 is OK: puppet ran at Tue Jul 16 11:03:14 UTC 2013 [11:03:24] RECOVERY - Puppet freshness on cp1045 is OK: puppet ran at Tue Jul 16 11:03:14 UTC 2013 [11:03:24] RECOVERY - Puppet freshness on mw101 is OK: puppet ran at Tue Jul 16 11:03:14 UTC 2013 [11:03:24] RECOVERY - Puppet freshness on mw94 is OK: puppet ran at Tue Jul 16 11:03:14 UTC 2013 [11:03:24] PROBLEM - Puppet freshness on cp1043 is CRITICAL: No successful Puppet run in the last 10 hours [11:03:25] RECOVERY - Puppet freshness on mw29 is OK: puppet ran at Tue Jul 16 11:03:19 UTC 2013 [11:03:26] RECOVERY - Puppet freshness on mw105 is OK: puppet ran at Tue Jul 16 11:03:19 UTC 2013 [11:03:26] RECOVERY - Puppet freshness on mw1160 is OK: puppet ran at Tue Jul 16 11:03:19 UTC 2013 [11:03:34] RECOVERY - Puppet freshness on maerlant is OK: puppet ran at Tue Jul 16 11:03:24 UTC 2013 [11:03:44] RECOVERY - Puppet freshness on sq67 is OK: puppet ran at Tue Jul 16 11:03:39 UTC 2013 [11:03:44] RECOVERY - Puppet freshness on db52 is OK: puppet ran at Tue Jul 16 11:03:39 UTC 2013 [11:03:44] RECOVERY - Puppet freshness on pc2 is OK: puppet ran at Tue Jul 16 11:03:39 UTC 2013 [11:04:02] RECOVERY - Puppet freshness on mc1015 is OK: puppet ran at Tue Jul 16 11:03:44 UTC 2013 [11:04:02] RECOVERY - Puppet freshness on mc1003 is OK: puppet ran at Tue Jul 16 11:03:44 UTC 2013 [11:04:02] RECOVERY - Puppet freshness on cp3003 is OK: puppet ran at Tue Jul 16 11:03:44 UTC 2013 [11:04:04] RECOVERY - Puppet freshness on srv287 is OK: puppet ran at Tue Jul 16 11:03:55 UTC 2013 [11:04:04] RECOVERY - Puppet freshness on mw51 is OK: puppet ran at Tue Jul 16 11:03:55 UTC 2013 [11:04:04] RECOVERY - Puppet freshness on mw1217 is OK: puppet ran at Tue Jul 16 11:03:55 UTC 2013 [11:04:04] RECOVERY - Puppet freshness on mw16 is OK: puppet ran at Tue Jul 16 11:03:55 UTC 2013 [11:04:04] RECOVERY - Puppet freshness on analytics1012 is OK: puppet ran at Tue Jul 16 11:04:00 UTC 2013 [11:04:04] RECOVERY - Puppet freshness on stafford is OK: puppet ran at Tue Jul 16 11:04:00 UTC 2013 [11:04:04] RECOVERY - Puppet freshness on mw1164 is OK: puppet ran at Tue Jul 16 11:04:00 UTC 2013 [11:04:05] RECOVERY - Puppet freshness on srv300 is OK: puppet ran at Tue Jul 16 11:04:00 UTC 2013 [11:04:05] RECOVERY - Puppet freshness on tmh2 is OK: puppet ran at Tue Jul 16 11:04:00 UTC 2013 [11:04:06] RECOVERY - Puppet freshness on mw1103 is OK: puppet ran at Tue Jul 16 11:04:00 UTC 2013 [11:04:06] RECOVERY - Puppet freshness on search13 is OK: puppet ran at Tue Jul 16 11:04:00 UTC 2013 [11:04:14] RECOVERY - Puppet freshness on mw1176 is OK: puppet ran at Tue Jul 16 11:04:05 UTC 2013 [11:04:14] RECOVERY - Puppet freshness on mw1088 is OK: puppet ran at Tue Jul 16 11:04:05 UTC 2013 [11:04:14] RECOVERY - Puppet freshness on search1010 is OK: puppet ran at Tue Jul 16 11:04:05 UTC 2013 [11:04:14] RECOVERY - Puppet freshness on mw1117 is OK: puppet ran at Tue Jul 16 11:04:05 UTC 2013 [11:04:14] RECOVERY - Puppet freshness on mw1100 is OK: puppet ran at Tue Jul 16 11:04:05 UTC 2013 [11:04:14] RECOVERY - Puppet freshness on mw1018 is OK: puppet ran at Tue Jul 16 11:04:05 UTC 2013 [11:04:44] RECOVERY - Puppet freshness on srv236 is OK: puppet ran at Tue Jul 16 11:04:40 UTC 2013 [11:04:44] RECOVERY - Puppet freshness on amssq52 is OK: puppet ran at Tue Jul 16 11:04:40 UTC 2013 [11:04:44] RECOVERY - Puppet freshness on sq86 is OK: puppet ran at Tue Jul 16 11:04:40 UTC 2013 [11:04:45] RECOVERY - Puppet freshness on sq83 is OK: puppet ran at Tue Jul 16 11:04:40 UTC 2013 [11:04:45] RECOVERY - Puppet freshness on sq71 is OK: puppet ran at Tue Jul 16 11:04:40 UTC 2013 [11:04:45] RECOVERY - Puppet freshness on cp1007 is OK: puppet ran at Tue Jul 16 11:04:40 UTC 2013 [11:04:45] RECOVERY - Puppet freshness on es1 is OK: puppet ran at Tue Jul 16 11:04:40 UTC 2013 [11:04:46] RECOVERY - Puppet freshness on srv277 is OK: puppet ran at Tue Jul 16 11:04:40 UTC 2013 [11:04:46] RECOVERY - Puppet freshness on db1054 is OK: puppet ran at Tue Jul 16 11:04:40 UTC 2013 [11:04:54] RECOVERY - Puppet freshness on labsdb1002 is OK: puppet ran at Tue Jul 16 11:04:45 UTC 2013 [11:04:54] RECOVERY - Puppet freshness on labsdb1003 is OK: puppet ran at Tue Jul 16 11:04:45 UTC 2013 [11:04:54] RECOVERY - Puppet freshness on cp1056 is OK: puppet ran at Tue Jul 16 11:04:50 UTC 2013 [11:04:54] RECOVERY - Puppet freshness on mw11 is OK: puppet ran at Tue Jul 16 11:04:50 UTC 2013 [11:04:54] RECOVERY - Puppet freshness on cp1061 is OK: puppet ran at Tue Jul 16 11:04:50 UTC 2013 [11:05:00] apergos: you fixed something? or is it the daily spam? [11:05:04] RECOVERY - Puppet freshness on fenari is OK: puppet ran at Tue Jul 16 11:04:55 UTC 2013 [11:05:04] RECOVERY - Puppet freshness on mw122 is OK: puppet ran at Tue Jul 16 11:04:55 UTC 2013 [11:05:04] RECOVERY - Puppet freshness on mw20 is OK: puppet ran at Tue Jul 16 11:04:55 UTC 2013 [11:05:04] RECOVERY - Puppet freshness on mw82 is OK: puppet ran at Tue Jul 16 11:04:55 UTC 2013 [11:05:04] RECOVERY - Puppet freshness on srv238 is OK: puppet ran at Tue Jul 16 11:04:55 UTC 2013 [11:05:04] RECOVERY - Puppet freshness on ms-be8 is OK: puppet ran at Tue Jul 16 11:04:55 UTC 2013 [11:05:04] RECOVERY - Puppet freshness on mw34 is OK: puppet ran at Tue Jul 16 11:04:55 UTC 2013 [11:05:05] RECOVERY - Puppet freshness on srv297 is OK: puppet ran at Tue Jul 16 11:04:55 UTC 2013 [11:05:05] RECOVERY - Puppet freshness on mw66 is OK: puppet ran at Tue Jul 16 11:04:55 UTC 2013 [11:05:06] RECOVERY - Puppet freshness on search1019 is OK: puppet ran at Tue Jul 16 11:05:00 UTC 2013 [11:05:06] RECOVERY - Puppet freshness on search1008 is OK: puppet ran at Tue Jul 16 11:05:00 UTC 2013 [11:05:06] fixed [11:05:07] RECOVERY - Puppet freshness on mw1065 is OK: puppet ran at Tue Jul 16 11:05:00 UTC 2013 [11:05:07] RECOVERY - Puppet freshness on mw1101 is OK: puppet ran at Tue Jul 16 11:05:00 UTC 2013 [11:05:08] RECOVERY - Puppet freshness on mw1052 is OK: puppet ran at Tue Jul 16 11:05:00 UTC 2013 [11:05:08] RECOVERY - Puppet freshness on mw1095 is OK: puppet ran at Tue Jul 16 11:05:00 UTC 2013 [11:05:09] RECOVERY - Puppet freshness on mw1144 is OK: puppet ran at Tue Jul 16 11:05:00 UTC 2013 [11:05:09] RECOVERY - Puppet freshness on mw103 is OK: puppet ran at Tue Jul 16 11:05:00 UTC 2013 [11:05:10] RECOVERY - Puppet freshness on db40 is OK: puppet ran at Tue Jul 16 11:05:00 UTC 2013 [11:05:14] RECOVERY - Puppet freshness on mw1042 is OK: puppet ran at Tue Jul 16 11:05:05 UTC 2013 [11:05:14] RECOVERY - Puppet freshness on cp3007 is OK: puppet ran at Tue Jul 16 11:05:05 UTC 2013 [11:05:14] RECOVERY - Puppet freshness on mw46 is OK: puppet ran at Tue Jul 16 11:05:05 UTC 2013 [11:05:14] RECOVERY - Puppet freshness on search17 is OK: puppet ran at Tue Jul 16 11:05:10 UTC 2013 [11:05:14] RECOVERY - Puppet freshness on mw1015 is OK: puppet ran at Tue Jul 16 11:05:10 UTC 2013 [11:05:14] RECOVERY - Puppet freshness on search1018 is OK: puppet ran at Tue Jul 16 11:05:10 UTC 2013 [11:05:23] we already had the daily spam [11:05:24] RECOVERY - Puppet freshness on mw1123 is OK: puppet ran at Tue Jul 16 11:05:15 UTC 2013 [11:05:24] RECOVERY - Puppet freshness on williams is OK: puppet ran at Tue Jul 16 11:05:21 UTC 2013 [11:05:28] heh [11:05:34] RECOVERY - Puppet freshness on mw1017 is OK: puppet ran at Tue Jul 16 11:05:26 UTC 2013 [11:05:34] RECOVERY - Puppet freshness on sq65 is OK: puppet ran at Tue Jul 16 11:05:26 UTC 2013 [11:05:34] RECOVERY - Puppet freshness on amssq55 is OK: puppet ran at Tue Jul 16 11:05:26 UTC 2013 [11:05:34] RECOVERY - Puppet freshness on es1003 is OK: puppet ran at Tue Jul 16 11:05:31 UTC 2013 [11:05:34] RECOVERY - Puppet freshness on db1019 is OK: puppet ran at Tue Jul 16 11:05:31 UTC 2013 [11:05:54] RECOVERY - Puppet freshness on cp1012 is OK: puppet ran at Tue Jul 16 11:05:46 UTC 2013 [11:05:54] RECOVERY - Puppet freshness on es1005 is OK: puppet ran at Tue Jul 16 11:05:46 UTC 2013 [11:05:54] RECOVERY - Puppet freshness on mw1099 is OK: puppet ran at Tue Jul 16 11:05:51 UTC 2013 [11:05:54] RECOVERY - Puppet freshness on mw125 is OK: puppet ran at Tue Jul 16 11:05:51 UTC 2013 [11:05:54] RECOVERY - Puppet freshness on mw14 is OK: puppet ran at Tue Jul 16 11:05:51 UTC 2013 [11:05:54] RECOVERY - Puppet freshness on analytics1010 is OK: puppet ran at Tue Jul 16 11:05:51 UTC 2013 [11:05:54] RECOVERY - Puppet freshness on srv237 is OK: puppet ran at Tue Jul 16 11:05:51 UTC 2013 [11:05:55] RECOVERY - Puppet freshness on srv286 is OK: puppet ran at Tue Jul 16 11:05:51 UTC 2013 [11:06:04] RECOVERY - Puppet freshness on mw49 is OK: puppet ran at Tue Jul 16 11:05:56 UTC 2013 [11:06:04] RECOVERY - Puppet freshness on mw62 is OK: puppet ran at Tue Jul 16 11:05:56 UTC 2013 [11:06:04] RECOVERY - Puppet freshness on mw1214 is OK: puppet ran at Tue Jul 16 11:05:56 UTC 2013 [11:06:04] RECOVERY - Puppet freshness on mw47 is OK: puppet ran at Tue Jul 16 11:05:56 UTC 2013 [11:06:04] RECOVERY - Puppet freshness on search22 is OK: puppet ran at Tue Jul 16 11:05:56 UTC 2013 [11:06:04] RECOVERY - Puppet freshness on mw1182 is OK: puppet ran at Tue Jul 16 11:05:56 UTC 2013 [11:06:04] RECOVERY - Puppet freshness on srv258 is OK: puppet ran at Tue Jul 16 11:05:56 UTC 2013 [11:06:05] RECOVERY - Puppet freshness on ms-be1 is OK: puppet ran at Tue Jul 16 11:06:01 UTC 2013 [11:06:05] RECOVERY - Puppet freshness on db1023 is OK: puppet ran at Tue Jul 16 11:06:01 UTC 2013 [11:06:06] RECOVERY - Puppet freshness on mw1127 is OK: puppet ran at Tue Jul 16 11:06:01 UTC 2013 [11:06:06] RECOVERY - Puppet freshness on mw1114 is OK: puppet ran at Tue Jul 16 11:06:01 UTC 2013 [11:06:07] RECOVERY - Puppet freshness on mw1126 is OK: puppet ran at Tue Jul 16 11:06:01 UTC 2013 [11:06:07] RECOVERY - Puppet freshness on srv250 is OK: puppet ran at Tue Jul 16 11:06:01 UTC 2013 [11:06:14] RECOVERY - Puppet freshness on mw1175 is OK: puppet ran at Tue Jul 16 11:06:06 UTC 2013 [11:06:14] RECOVERY - Puppet freshness on mw9 is OK: puppet ran at Tue Jul 16 11:06:06 UTC 2013 [11:06:24] RECOVERY - Puppet freshness on dobson is OK: puppet ran at Tue Jul 16 11:06:21 UTC 2013 [11:06:34] RECOVERY - Puppet freshness on tin is OK: puppet ran at Tue Jul 16 11:06:31 UTC 2013 [11:06:34] RECOVERY - Puppet freshness on sq73 is OK: puppet ran at Tue Jul 16 11:06:31 UTC 2013 [11:06:34] RECOVERY - Puppet freshness on sq72 is OK: puppet ran at Tue Jul 16 11:06:31 UTC 2013 [11:06:34] RECOVERY - Puppet freshness on gurvin is OK: puppet ran at Tue Jul 16 11:06:31 UTC 2013 [11:06:34] RECOVERY - Puppet freshness on sq45 is OK: puppet ran at Tue Jul 16 11:06:31 UTC 2013 [11:06:34] RECOVERY - Puppet freshness on es4 is OK: puppet ran at Tue Jul 16 11:06:31 UTC 2013 [11:06:34] RECOVERY - Puppet freshness on db53 is OK: puppet ran at Tue Jul 16 11:06:31 UTC 2013 [11:06:35] RECOVERY - Puppet freshness on amssq57 is OK: puppet ran at Tue Jul 16 11:06:31 UTC 2013 [11:06:35] RECOVERY - Puppet freshness on es1002 is OK: puppet ran at Tue Jul 16 11:06:31 UTC 2013 [11:06:36] RECOVERY - Puppet freshness on cp1013 is OK: puppet ran at Tue Jul 16 11:06:31 UTC 2013 [11:06:44] RECOVERY - Puppet freshness on db1052 is OK: puppet ran at Tue Jul 16 11:06:36 UTC 2013 [11:06:44] RECOVERY - Puppet freshness on pc1003 is OK: puppet ran at Tue Jul 16 11:06:41 UTC 2013 [11:06:44] RECOVERY - Puppet freshness on pc1002 is OK: puppet ran at Tue Jul 16 11:06:41 UTC 2013 [11:06:44] RECOVERY - Puppet freshness on nescio is OK: puppet ran at Tue Jul 16 11:06:41 UTC 2013 [11:06:51] apergos: what was the problem? [11:06:51] Change merged: Mark Bergsma; [operations/debs/varnish] (testing/3.0.3plus-rc1) - https://gerrit.wikimedia.org/r/73145 [11:06:54] RECOVERY - Puppet freshness on cp1051 is OK: puppet ran at Tue Jul 16 11:06:46 UTC 2013 [11:06:54] RECOVERY - Puppet freshness on mw81 is OK: puppet ran at Tue Jul 16 11:06:47 UTC 2013 [11:06:54] RECOVERY - Puppet freshness on srv275 is OK: puppet ran at Tue Jul 16 11:06:47 UTC 2013 [11:06:54] RECOVERY - Puppet freshness on mw90 is OK: puppet ran at Tue Jul 16 11:06:47 UTC 2013 [11:06:54] RECOVERY - Puppet freshness on mw100 is OK: puppet ran at Tue Jul 16 11:06:47 UTC 2013 [11:06:54] RECOVERY - Puppet freshness on search14 is OK: puppet ran at Tue Jul 16 11:06:52 UTC 2013 [11:06:54] RECOVERY - Puppet freshness on search21 is OK: puppet ran at Tue Jul 16 11:06:52 UTC 2013 [11:06:55] RECOVERY - Puppet freshness on zirconium is OK: puppet ran at Tue Jul 16 11:06:52 UTC 2013 [11:06:55] RECOVERY - Puppet freshness on amssq34 is OK: puppet ran at Tue Jul 16 11:06:52 UTC 2013 [11:06:56] RECOVERY - Puppet freshness on mw1162 is OK: puppet ran at Tue Jul 16 11:06:52 UTC 2013 [11:07:04] Change merged: Mark Bergsma; [operations/debs/varnish] (testing/3.0.3plus-rc1) - https://gerrit.wikimedia.org/r/73146 [11:07:04] RECOVERY - Puppet freshness on cp3010 is OK: puppet ran at Tue Jul 16 11:06:57 UTC 2013 [11:07:04] RECOVERY - Puppet freshness on search1014 is OK: puppet ran at Tue Jul 16 11:06:57 UTC 2013 [11:07:04] RECOVERY - Puppet freshness on es10 is OK: puppet ran at Tue Jul 16 11:06:57 UTC 2013 [11:07:04] RECOVERY - Puppet freshness on ms5 is OK: puppet ran at Tue Jul 16 11:06:57 UTC 2013 [11:07:04] RECOVERY - Puppet freshness on cp1069 is OK: puppet ran at Tue Jul 16 11:07:02 UTC 2013 [11:07:14] RECOVERY - Puppet freshness on mw1044 is OK: puppet ran at Tue Jul 16 11:07:07 UTC 2013 [11:07:14] RECOVERY - Puppet freshness on mw5 is OK: puppet ran at Tue Jul 16 11:07:12 UTC 2013 [11:07:14] RECOVERY - Puppet freshness on mw41 is OK: puppet ran at Tue Jul 16 11:07:13 UTC 2013 [11:07:24] RECOVERY - Puppet freshness on mw1196 is OK: puppet ran at Tue Jul 16 11:07:18 UTC 2013 [11:07:33] AzaToth: https://gerrit.wikimedia.org/r/#/c/73946/ that [11:07:34] RECOVERY - Puppet freshness on sq79 is OK: puppet ran at Tue Jul 16 11:07:23 UTC 2013 [11:07:34] RECOVERY - Puppet freshness on cp1020 is OK: puppet ran at Tue Jul 16 11:07:29 UTC 2013 [11:07:34] RECOVERY - Puppet freshness on db1048 is OK: puppet ran at Tue Jul 16 11:07:29 UTC 2013 [11:07:43] one lousy character [11:07:44] RECOVERY - Puppet freshness on mc1005 is OK: puppet ran at Tue Jul 16 11:07:39 UTC 2013 [11:07:44] RECOVERY - Puppet freshness on db1036 is OK: puppet ran at Tue Jul 16 11:07:39 UTC 2013 [11:07:44] RECOVERY - Puppet freshness on wtp1024 is OK: puppet ran at Tue Jul 16 11:07:39 UTC 2013 [11:07:54] RECOVERY - Puppet freshness on cp1050 is OK: puppet ran at Tue Jul 16 11:07:44 UTC 2013 [11:07:54] RECOVERY - Puppet freshness on cp1046 is OK: puppet ran at Tue Jul 16 11:07:44 UTC 2013 [11:07:54] RECOVERY - Puppet freshness on cp1006 is OK: puppet ran at Tue Jul 16 11:07:44 UTC 2013 [11:07:54] RECOVERY - Puppet freshness on mw119 is OK: puppet ran at Tue Jul 16 11:07:44 UTC 2013 [11:07:54] RECOVERY - Puppet freshness on gadolinium is OK: puppet ran at Tue Jul 16 11:07:44 UTC 2013 [11:07:54] RECOVERY - Puppet freshness on mw118 is OK: puppet ran at Tue Jul 16 11:07:49 UTC 2013 [11:07:54] RECOVERY - Puppet freshness on mw83 is OK: puppet ran at Tue Jul 16 11:07:49 UTC 2013 [11:07:55] RECOVERY - Puppet freshness on search36 is OK: puppet ran at Tue Jul 16 11:07:49 UTC 2013 [11:07:55] RECOVERY - Puppet freshness on srv284 is OK: puppet ran at Tue Jul 16 11:07:49 UTC 2013 [11:07:56] RECOVERY - Puppet freshness on srv290 is OK: puppet ran at Tue Jul 16 11:07:49 UTC 2013 [11:07:56] RECOVERY - Puppet freshness on srv301 is OK: puppet ran at Tue Jul 16 11:07:49 UTC 2013 [11:08:04] RECOVERY - Puppet freshness on mw1161 is OK: puppet ran at Tue Jul 16 11:07:54 UTC 2013 [11:08:04] RECOVERY - Puppet freshness on mw1130 is OK: puppet ran at Tue Jul 16 11:07:54 UTC 2013 [11:08:04] RECOVERY - Puppet freshness on lvs5 is OK: puppet ran at Tue Jul 16 11:07:54 UTC 2013 [11:08:04] RECOVERY - Puppet freshness on mw1190 is OK: puppet ran at Tue Jul 16 11:07:54 UTC 2013 [11:08:04] RECOVERY - Puppet freshness on mw1124 is OK: puppet ran at Tue Jul 16 11:07:54 UTC 2013 [11:08:04] RECOVERY - Puppet freshness on mw1147 is OK: puppet ran at Tue Jul 16 11:07:54 UTC 2013 [11:08:04] RECOVERY - Puppet freshness on mw1125 is OK: puppet ran at Tue Jul 16 11:07:54 UTC 2013 [11:08:05] RECOVERY - Puppet freshness on mw38 is OK: puppet ran at Tue Jul 16 11:07:59 UTC 2013 [11:08:29] haha [11:08:44] RECOVERY - Puppet freshness on lvs1 is OK: puppet ran at Tue Jul 16 11:08:39 UTC 2013 [11:08:44] RECOVERY - Puppet freshness on sq51 is OK: puppet ran at Tue Jul 16 11:08:39 UTC 2013 [11:08:44] RECOVERY - Puppet freshness on rubidium is OK: puppet ran at Tue Jul 16 11:08:39 UTC 2013 [11:08:44] RECOVERY - Puppet freshness on cp1003 is OK: puppet ran at Tue Jul 16 11:08:39 UTC 2013 [11:08:44] RECOVERY - Puppet freshness on db45 is OK: puppet ran at Tue Jul 16 11:08:39 UTC 2013 [11:08:44] RECOVERY - Puppet freshness on amssq44 is OK: puppet ran at Tue Jul 16 11:08:40 UTC 2013 [11:08:45] RECOVERY - Puppet freshness on ms-be2 is OK: puppet ran at Tue Jul 16 11:08:40 UTC 2013 [11:08:45] RECOVERY - Puppet freshness on wtp1003 is OK: puppet ran at Tue Jul 16 11:08:40 UTC 2013 [11:08:46] RECOVERY - Puppet freshness on wtp1018 is OK: puppet ran at Tue Jul 16 11:08:40 UTC 2013 [11:08:54] RECOVERY - Puppet freshness on cp1063 is OK: puppet ran at Tue Jul 16 11:08:45 UTC 2013 [11:08:54] RECOVERY - Puppet freshness on wtp1011 is OK: puppet ran at Tue Jul 16 11:08:45 UTC 2013 [11:08:54] RECOVERY - Puppet freshness on srv249 is OK: puppet ran at Tue Jul 16 11:08:45 UTC 2013 [11:08:54] RECOVERY - Puppet freshness on analytics1023 is OK: puppet ran at Tue Jul 16 11:08:45 UTC 2013 [11:08:54] RECOVERY - Puppet freshness on srv289 is OK: puppet ran at Tue Jul 16 11:08:45 UTC 2013 [11:08:54] RECOVERY - Puppet freshness on srv253 is OK: puppet ran at Tue Jul 16 11:08:45 UTC 2013 [11:08:54] RECOVERY - Puppet freshness on srv245 is OK: puppet ran at Tue Jul 16 11:08:45 UTC 2013 [11:08:55] RECOVERY - Puppet freshness on ms-fe3 is OK: puppet ran at Tue Jul 16 11:08:50 UTC 2013 [11:08:55] RECOVERY - Puppet freshness on search19 is OK: puppet ran at Tue Jul 16 11:08:50 UTC 2013 [11:08:56] RECOVERY - Puppet freshness on mc1009 is OK: puppet ran at Tue Jul 16 11:08:50 UTC 2013 [11:09:04] RECOVERY - Puppet freshness on mw1004 is OK: puppet ran at Tue Jul 16 11:08:55 UTC 2013 [11:09:04] RECOVERY - Puppet freshness on mw1146 is OK: puppet ran at Tue Jul 16 11:08:55 UTC 2013 [11:09:04] RECOVERY - Puppet freshness on ms-fe3001 is OK: puppet ran at Tue Jul 16 11:08:55 UTC 2013 [11:09:04] RECOVERY - Puppet freshness on srv244 is OK: puppet ran at Tue Jul 16 11:09:00 UTC 2013 [11:09:14] RECOVERY - Puppet freshness on mw1056 is OK: puppet ran at Tue Jul 16 11:09:05 UTC 2013 [11:09:14] RECOVERY - Puppet freshness on mw1115 is OK: puppet ran at Tue Jul 16 11:09:05 UTC 2013 [11:09:36] RECOVERY - Puppet freshness on db43 is OK: puppet ran at Tue Jul 16 11:09:25 UTC 2013 [11:09:44] RECOVERY - Puppet freshness on mw1081 is OK: puppet ran at Tue Jul 16 11:09:40 UTC 2013 [11:09:54] RECOVERY - Puppet freshness on vanadium is OK: puppet ran at Tue Jul 16 11:09:45 UTC 2013 [11:09:54] RECOVERY - Puppet freshness on solr2 is OK: puppet ran at Tue Jul 16 11:09:45 UTC 2013 [11:09:54] RECOVERY - Puppet freshness on sq74 is OK: puppet ran at Tue Jul 16 11:09:45 UTC 2013 [11:09:54] RECOVERY - Puppet freshness on sq48 is OK: puppet ran at Tue Jul 16 11:09:45 UTC 2013 [11:09:54] RECOVERY - Puppet freshness on sq43 is OK: puppet ran at Tue Jul 16 11:09:45 UTC 2013 [11:09:54] RECOVERY - Puppet freshness on es6 is OK: puppet ran at Tue Jul 16 11:09:45 UTC 2013 [11:09:54] RECOVERY - Puppet freshness on ms-be1012 is OK: puppet ran at Tue Jul 16 11:09:45 UTC 2013 [11:09:55] RECOVERY - Puppet freshness on sq85 is OK: puppet ran at Tue Jul 16 11:09:45 UTC 2013 [11:09:55] RECOVERY - Puppet freshness on ms-be4 is OK: puppet ran at Tue Jul 16 11:09:50 UTC 2013 [11:09:56] RECOVERY - Puppet freshness on sq42 is OK: puppet ran at Tue Jul 16 11:09:50 UTC 2013 [11:09:56] RECOVERY - Puppet freshness on db1004 is OK: puppet ran at Tue Jul 16 11:09:50 UTC 2013 [11:09:57] RECOVERY - Puppet freshness on mw1048 is OK: puppet ran at Tue Jul 16 11:09:50 UTC 2013 [11:09:57] RECOVERY - Puppet freshness on mw1057 is OK: puppet ran at Tue Jul 16 11:09:50 UTC 2013 [11:09:58] RECOVERY - Puppet freshness on db58 is OK: puppet ran at Tue Jul 16 11:09:50 UTC 2013 [11:09:58] RECOVERY - Puppet freshness on mw30 is OK: puppet ran at Tue Jul 16 11:09:50 UTC 2013 [11:09:59] RECOVERY - Puppet freshness on snapshot1 is OK: puppet ran at Tue Jul 16 11:09:50 UTC 2013 [11:09:59] RECOVERY - Puppet freshness on wtp1004 is OK: puppet ran at Tue Jul 16 11:09:50 UTC 2013 [11:10:04] RECOVERY - Puppet freshness on mw1111 is OK: puppet ran at Tue Jul 16 11:09:55 UTC 2013 [11:10:04] RECOVERY - Puppet freshness on mw76 is OK: puppet ran at Tue Jul 16 11:09:55 UTC 2013 [11:10:04] RECOVERY - Puppet freshness on mw1038 is OK: puppet ran at Tue Jul 16 11:10:00 UTC 2013 [11:10:04] RECOVERY - Puppet freshness on srv276 is OK: puppet ran at Tue Jul 16 11:10:00 UTC 2013 [11:10:04] RECOVERY - Puppet freshness on mw1159 is OK: puppet ran at Tue Jul 16 11:10:00 UTC 2013 [11:10:04] RECOVERY - Puppet freshness on mw1186 is OK: puppet ran at Tue Jul 16 11:10:00 UTC 2013 [11:10:14] RECOVERY - Puppet freshness on srv292 is OK: puppet ran at Tue Jul 16 11:10:06 UTC 2013 [11:10:14] RECOVERY - Puppet freshness on mw1007 is OK: puppet ran at Tue Jul 16 11:10:06 UTC 2013 [11:10:14] RECOVERY - Puppet freshness on mw121 is OK: puppet ran at Tue Jul 16 11:10:06 UTC 2013 [11:10:14] RECOVERY - Puppet freshness on mw1041 is OK: puppet ran at Tue Jul 16 11:10:06 UTC 2013 [11:10:14] RECOVERY - Puppet freshness on search1024 is OK: puppet ran at Tue Jul 16 11:10:06 UTC 2013 [11:10:14] RECOVERY - Puppet freshness on srv294 is OK: puppet ran at Tue Jul 16 11:10:06 UTC 2013 [11:10:15] RECOVERY - Puppet freshness on mw1188 is OK: puppet ran at Tue Jul 16 11:10:06 UTC 2013 [11:10:15] RECOVERY - Puppet freshness on mw1210 is OK: puppet ran at Tue Jul 16 11:10:06 UTC 2013 [11:10:16] RECOVERY - Puppet freshness on mw1043 is OK: puppet ran at Tue Jul 16 11:10:06 UTC 2013 [11:10:16] RECOVERY - Puppet freshness on mw1140 is OK: puppet ran at Tue Jul 16 11:10:12 UTC 2013 [11:10:17] RECOVERY - Puppet freshness on solr1003 is OK: puppet ran at Tue Jul 16 11:10:12 UTC 2013 [11:10:17] RECOVERY - Puppet freshness on manganese is OK: puppet ran at Tue Jul 16 11:10:12 UTC 2013 [11:10:24] RECOVERY - Puppet freshness on mc1001 is OK: puppet ran at Tue Jul 16 11:10:17 UTC 2013 [11:10:24] RECOVERY - Puppet freshness on ms-be9 is OK: puppet ran at Tue Jul 16 11:10:17 UTC 2013 [11:10:25] RECOVERY - Puppet freshness on db1029 is OK: puppet ran at Tue Jul 16 11:10:22 UTC 2013 [11:10:25] RECOVERY - Puppet freshness on mw8 is OK: puppet ran at Tue Jul 16 11:10:22 UTC 2013 [11:10:34] RECOVERY - Puppet freshness on mw1197 is OK: puppet ran at Tue Jul 16 11:10:27 UTC 2013 [11:10:34] RECOVERY - Puppet freshness on search18 is OK: puppet ran at Tue Jul 16 11:10:27 UTC 2013 [11:10:34] RECOVERY - Puppet freshness on cp1049 is OK: puppet ran at Tue Jul 16 11:10:27 UTC 2013 [11:10:34] RECOVERY - Puppet freshness on amssq53 is OK: puppet ran at Tue Jul 16 11:10:27 UTC 2013 [11:10:34] RECOVERY - Puppet freshness on ekrem is OK: puppet ran at Tue Jul 16 11:10:27 UTC 2013 [11:10:34] RECOVERY - Puppet freshness on mw1171 is OK: puppet ran at Tue Jul 16 11:10:27 UTC 2013 [11:10:34] RECOVERY - Puppet freshness on mw1087 is OK: puppet ran at Tue Jul 16 11:10:27 UTC 2013 [11:10:35] RECOVERY - Puppet freshness on mw1106 is OK: puppet ran at Tue Jul 16 11:10:27 UTC 2013 [11:10:35] RECOVERY - Puppet freshness on ms-be3 is OK: puppet ran at Tue Jul 16 11:10:27 UTC 2013 [11:10:36] RECOVERY - Puppet freshness on mw1063 is OK: puppet ran at Tue Jul 16 11:10:33 UTC 2013 [11:10:36] RECOVERY - Puppet freshness on titanium is OK: puppet ran at Tue Jul 16 11:10:33 UTC 2013 [11:10:37] RECOVERY - Puppet freshness on sq76 is OK: puppet ran at Tue Jul 16 11:10:33 UTC 2013 [11:10:37] RECOVERY - Puppet freshness on ms10 is OK: puppet ran at Tue Jul 16 11:10:33 UTC 2013 [11:10:38] RECOVERY - Puppet freshness on cp1005 is OK: puppet ran at Tue Jul 16 11:10:33 UTC 2013 [11:10:44] RECOVERY - Puppet freshness on cp1002 is OK: puppet ran at Tue Jul 16 11:10:38 UTC 2013 [11:10:44] RECOVERY - Puppet freshness on cp3012 is OK: puppet ran at Tue Jul 16 11:10:38 UTC 2013 [11:10:44] RECOVERY - Puppet freshness on praseodymium is OK: puppet ran at Tue Jul 16 11:10:38 UTC 2013 [11:10:44] RECOVERY - Puppet freshness on rdb1002 is OK: puppet ran at Tue Jul 16 11:10:43 UTC 2013 [11:10:44] RECOVERY - Puppet freshness on potassium is OK: puppet ran at Tue Jul 16 11:10:43 UTC 2013 [11:10:44] RECOVERY - Puppet freshness on mc1007 is OK: puppet ran at Tue Jul 16 11:10:43 UTC 2013 [11:10:44] RECOVERY - Puppet freshness on analytics1015 is OK: puppet ran at Tue Jul 16 11:10:43 UTC 2013 [11:10:54] RECOVERY - Puppet freshness on ms-be1006 is OK: puppet ran at Tue Jul 16 11:10:48 UTC 2013 [11:10:54] RECOVERY - Puppet freshness on wtp1015 is OK: puppet ran at Tue Jul 16 11:10:48 UTC 2013 [11:10:54] RECOVERY - Puppet freshness on db1001 is OK: puppet ran at Tue Jul 16 11:10:48 UTC 2013 [11:10:54] RECOVERY - Puppet freshness on helium is OK: puppet ran at Tue Jul 16 11:10:48 UTC 2013 [11:10:54] RECOVERY - Puppet freshness on db39 is OK: puppet ran at Tue Jul 16 11:10:48 UTC 2013 [11:10:54] RECOVERY - Puppet freshness on db1031 is OK: puppet ran at Tue Jul 16 11:10:48 UTC 2013 [11:10:54] RECOVERY - Puppet freshness on srv273 is OK: puppet ran at Tue Jul 16 11:10:53 UTC 2013 [11:10:55] RECOVERY - Puppet freshness on mw58 is OK: puppet ran at Tue Jul 16 11:10:53 UTC 2013 [11:10:55] RECOVERY - Puppet freshness on db34 is OK: puppet ran at Tue Jul 16 11:10:53 UTC 2013 [11:10:56] RECOVERY - Puppet freshness on srv255 is OK: puppet ran at Tue Jul 16 11:10:53 UTC 2013 [11:10:56] RECOVERY - Puppet freshness on pc1 is OK: puppet ran at Tue Jul 16 11:10:53 UTC 2013 [11:11:04] RECOVERY - Puppet freshness on cp3009 is OK: puppet ran at Tue Jul 16 11:10:58 UTC 2013 [11:11:04] RECOVERY - Puppet freshness on mw1032 is OK: puppet ran at Tue Jul 16 11:10:58 UTC 2013 [11:11:04] RECOVERY - Puppet freshness on searchidx1001 is OK: puppet ran at Tue Jul 16 11:11:03 UTC 2013 [11:11:04] RECOVERY - Puppet freshness on stat1 is OK: puppet ran at Tue Jul 16 11:11:03 UTC 2013 [11:11:14] RECOVERY - Puppet freshness on srv246 is OK: puppet ran at Tue Jul 16 11:11:08 UTC 2013 [11:11:14] RECOVERY - Puppet freshness on mw57 is OK: puppet ran at Tue Jul 16 11:11:08 UTC 2013 [11:11:14] RECOVERY - Puppet freshness on mw43 is OK: puppet ran at Tue Jul 16 11:11:08 UTC 2013 [11:11:24] RECOVERY - Puppet freshness on sq54 is OK: puppet ran at Tue Jul 16 11:11:23 UTC 2013 [11:11:34] RECOVERY - Puppet freshness on sq58 is OK: puppet ran at Tue Jul 16 11:11:28 UTC 2013 [11:11:34] RECOVERY - Puppet freshness on db1044 is OK: puppet ran at Tue Jul 16 11:11:33 UTC 2013 [11:11:38] apergos, wouldn't it make more sense to monitor puppet run failures instead?:P [11:11:44] RECOVERY - Puppet freshness on db1022 is OK: puppet ran at Tue Jul 16 11:11:38 UTC 2013 [11:11:54] RECOVERY - Puppet freshness on cp1010 is OK: puppet ran at Tue Jul 16 11:11:44 UTC 2013 [11:11:54] RECOVERY - Puppet freshness on rdb1003 is OK: puppet ran at Tue Jul 16 11:11:44 UTC 2013 [11:11:54] RECOVERY - Puppet freshness on ms-fe1001 is OK: puppet ran at Tue Jul 16 11:11:44 UTC 2013 [11:11:54] RECOVERY - Puppet freshness on wtp1007 is OK: puppet ran at Tue Jul 16 11:11:49 UTC 2013 [11:12:04] RECOVERY - Puppet freshness on mw106 is OK: puppet ran at Tue Jul 16 11:11:59 UTC 2013 [11:12:04] RECOVERY - Puppet freshness on mw1003 is OK: puppet ran at Tue Jul 16 11:11:59 UTC 2013 [11:12:04] RECOVERY - Puppet freshness on mw98 is OK: puppet ran at Tue Jul 16 11:11:59 UTC 2013 [11:12:04] RECOVERY - Puppet freshness on db1020 is OK: puppet ran at Tue Jul 16 11:11:59 UTC 2013 [11:12:04] RECOVERY - Puppet freshness on mw2 is OK: puppet ran at Tue Jul 16 11:11:59 UTC 2013 [11:12:04] RECOVERY - Puppet freshness on search33 is OK: puppet ran at Tue Jul 16 11:11:59 UTC 2013 [11:12:04] RECOVERY - Puppet freshness on srv193 is OK: puppet ran at Tue Jul 16 11:11:59 UTC 2013 [11:12:05] RECOVERY - Puppet freshness on mw79 is OK: puppet ran at Tue Jul 16 11:11:59 UTC 2013 [11:12:05] RECOVERY - Puppet freshness on mw35 is OK: puppet ran at Tue Jul 16 11:11:59 UTC 2013 [11:12:06] RECOVERY - Puppet freshness on mw1205 is OK: puppet ran at Tue Jul 16 11:11:59 UTC 2013 [11:12:11] we rpeort the failures [11:12:14] RECOVERY - Puppet freshness on mw1173 is OK: puppet ran at Tue Jul 16 11:12:04 UTC 2013 [11:12:14] RECOVERY - Puppet freshness on db63 is OK: puppet ran at Tue Jul 16 11:12:04 UTC 2013 [11:12:14] RECOVERY - Puppet freshness on mw1068 is OK: puppet ran at Tue Jul 16 11:12:04 UTC 2013 [11:12:14] RECOVERY - Puppet freshness on mw1150 is OK: puppet ran at Tue Jul 16 11:12:04 UTC 2013 [11:12:14] RECOVERY - Puppet freshness on mw1189 is OK: puppet ran at Tue Jul 16 11:12:04 UTC 2013 [11:12:14] RECOVERY - Puppet freshness on mw1033 is OK: puppet ran at Tue Jul 16 11:12:04 UTC 2013 [11:12:14] RECOVERY - Puppet freshness on solr3 is OK: puppet ran at Tue Jul 16 11:12:04 UTC 2013 [11:12:15] RECOVERY - Puppet freshness on mw1201 is OK: puppet ran at Tue Jul 16 11:12:04 UTC 2013 [11:12:15] RECOVERY - Puppet freshness on mw1024 is OK: puppet ran at Tue Jul 16 11:12:04 UTC 2013 [11:12:16] RECOVERY - Puppet freshness on mw1122 is OK: puppet ran at Tue Jul 16 11:12:04 UTC 2013 [11:12:16] RECOVERY - Puppet freshness on mw1046 is OK: puppet ran at Tue Jul 16 11:12:04 UTC 2013 [11:12:17] but we also report recovries after failures [11:12:36] if puppet always runs on time you won't hear about it [11:12:54] RECOVERY - Puppet freshness on mw1069 is OK: puppet ran at Tue Jul 16 11:12:44 UTC 2013 [11:12:54] RECOVERY - Puppet freshness on calcium is OK: puppet ran at Tue Jul 16 11:12:44 UTC 2013 [11:12:54] RECOVERY - Puppet freshness on solr1002 is OK: puppet ran at Tue Jul 16 11:12:44 UTC 2013 [11:12:54] RECOVERY - Puppet freshness on mw1153 is OK: puppet ran at Tue Jul 16 11:12:44 UTC 2013 [11:12:54] RECOVERY - Puppet freshness on sq63 is OK: puppet ran at Tue Jul 16 11:12:44 UTC 2013 [11:12:54] RECOVERY - Puppet freshness on db48 is OK: puppet ran at Tue Jul 16 11:12:44 UTC 2013 [11:12:54] RECOVERY - Puppet freshness on db1040 is OK: puppet ran at Tue Jul 16 11:12:44 UTC 2013 [11:12:55] RECOVERY - Puppet freshness on db1002 is OK: puppet ran at Tue Jul 16 11:12:44 UTC 2013 [11:12:55] RECOVERY - Puppet freshness on mw124 is OK: puppet ran at Tue Jul 16 11:12:44 UTC 2013 [11:12:56] RECOVERY - Puppet freshness on db1033 is OK: puppet ran at Tue Jul 16 11:12:44 UTC 2013 [11:12:56] RECOVERY - Puppet freshness on virt1000 is OK: puppet ran at Tue Jul 16 11:12:44 UTC 2013 [11:12:57] RECOVERY - Puppet freshness on ms-be11 is OK: puppet ran at Tue Jul 16 11:12:49 UTC 2013 [11:13:04] RECOVERY - Puppet freshness on ssl1001 is OK: puppet ran at Tue Jul 16 11:12:54 UTC 2013 [11:13:04] RECOVERY - Puppet freshness on srv267 is OK: puppet ran at Tue Jul 16 11:12:54 UTC 2013 [11:13:04] RECOVERY - Puppet freshness on mw112 is OK: puppet ran at Tue Jul 16 11:12:54 UTC 2013 [11:13:04] RECOVERY - Puppet freshness on mw104 is OK: puppet ran at Tue Jul 16 11:12:54 UTC 2013 [11:13:04] RECOVERY - Puppet freshness on srv260 is OK: puppet ran at Tue Jul 16 11:12:54 UTC 2013 [11:13:04] RECOVERY - Puppet freshness on srv262 is OK: puppet ran at Tue Jul 16 11:12:54 UTC 2013 [11:13:04] RECOVERY - Puppet freshness on srv269 is OK: puppet ran at Tue Jul 16 11:12:54 UTC 2013 [11:13:05] RECOVERY - Puppet freshness on analytics1004 is OK: puppet ran at Tue Jul 16 11:12:59 UTC 2013 [11:13:05] RECOVERY - Puppet freshness on search25 is OK: puppet ran at Tue Jul 16 11:13:00 UTC 2013 [11:13:06] RECOVERY - Puppet freshness on search16 is OK: puppet ran at Tue Jul 16 11:13:00 UTC 2013 [11:13:06] RECOVERY - Puppet freshness on mw68 is OK: puppet ran at Tue Jul 16 11:13:00 UTC 2013 [11:13:14] RECOVERY - Puppet freshness on mw37 is OK: puppet ran at Tue Jul 16 11:13:05 UTC 2013 [11:13:14] RECOVERY - Puppet freshness on mw1118 is OK: puppet ran at Tue Jul 16 11:13:05 UTC 2013 [11:13:14] RECOVERY - Puppet freshness on mw1142 is OK: puppet ran at Tue Jul 16 11:13:05 UTC 2013 [11:13:14] RECOVERY - Puppet freshness on mw1166 is OK: puppet ran at Tue Jul 16 11:13:05 UTC 2013 [11:13:14] RECOVERY - Puppet freshness on virt11 is OK: puppet ran at Tue Jul 16 11:13:05 UTC 2013 [11:13:14] RECOVERY - Puppet freshness on search1001 is OK: puppet ran at Tue Jul 16 11:13:05 UTC 2013 [11:13:14] RECOVERY - Puppet freshness on mw1010 is OK: puppet ran at Tue Jul 16 11:13:05 UTC 2013 [11:13:15] RECOVERY - Puppet freshness on fluorine is OK: puppet ran at Tue Jul 16 11:13:05 UTC 2013 [11:13:15] RECOVERY - Puppet freshness on search1022 is OK: puppet ran at Tue Jul 16 11:13:05 UTC 2013 [11:13:16] RECOVERY - Puppet freshness on mw1091 is OK: puppet ran at Tue Jul 16 11:13:05 UTC 2013 [11:13:16] RECOVERY - Puppet freshness on mw1092 is OK: puppet ran at Tue Jul 16 11:13:10 UTC 2013 [11:13:24] RECOVERY - Puppet freshness on analytics1014 is OK: puppet ran at Tue Jul 16 11:13:16 UTC 2013 [11:13:34] RECOVERY - Puppet freshness on amssq48 is OK: puppet ran at Tue Jul 16 11:13:26 UTC 2013 [11:13:44] RECOVERY - Puppet freshness on db1034 is OK: puppet ran at Tue Jul 16 11:13:36 UTC 2013 [11:13:44] RECOVERY - Puppet freshness on db66 is OK: puppet ran at Tue Jul 16 11:13:41 UTC 2013 [11:13:44] RECOVERY - Puppet freshness on db1051 is OK: puppet ran at Tue Jul 16 11:13:41 UTC 2013 [11:13:45] apergos: I assume all blame is on LeslieCarr [11:13:54] RECOVERY - Puppet freshness on db1042 is OK: puppet ran at Tue Jul 16 11:13:46 UTC 2013 [11:13:54] RECOVERY - Puppet freshness on db54 is OK: puppet ran at Tue Jul 16 11:13:51 UTC 2013 [11:14:03] *shrug* [11:14:04] RECOVERY - Puppet freshness on search30 is OK: puppet ran at Tue Jul 16 11:13:56 UTC 2013 [11:14:04] RECOVERY - Puppet freshness on mw60 is OK: puppet ran at Tue Jul 16 11:13:56 UTC 2013 [11:14:04] RECOVERY - Puppet freshness on mw1011 is OK: puppet ran at Tue Jul 16 11:13:56 UTC 2013 [11:14:04] RECOVERY - Puppet freshness on mw1027 is OK: puppet ran at Tue Jul 16 11:13:56 UTC 2013 [11:14:04] RECOVERY - Puppet freshness on amssq47 is OK: puppet ran at Tue Jul 16 11:13:56 UTC 2013 [11:14:04] RECOVERY - Puppet freshness on mw10 is OK: puppet ran at Tue Jul 16 11:13:56 UTC 2013 [11:14:04] RECOVERY - Puppet freshness on mw1143 is OK: puppet ran at Tue Jul 16 11:13:56 UTC 2013 [11:14:05] RECOVERY - Puppet freshness on mw1054 is OK: puppet ran at Tue Jul 16 11:14:01 UTC 2013 [11:14:05] RECOVERY - Puppet freshness on sockpuppet is OK: puppet ran at Tue Jul 16 11:14:01 UTC 2013 [11:14:06] RECOVERY - Puppet freshness on search29 is OK: puppet ran at Tue Jul 16 11:14:01 UTC 2013 [11:14:06] RECOVERY - Puppet freshness on mw1107 is OK: puppet ran at Tue Jul 16 11:14:01 UTC 2013 [11:14:07] RECOVERY - Puppet freshness on mw1066 is OK: puppet ran at Tue Jul 16 11:14:01 UTC 2013 [11:14:08] whatever, typos happen [11:14:14] RECOVERY - Puppet freshness on amslvs4 is OK: puppet ran at Tue Jul 16 11:14:07 UTC 2013 [11:14:14] RECOVERY - Puppet freshness on search1007 is OK: puppet ran at Tue Jul 16 11:14:07 UTC 2013 [11:14:19] I mean, "Puppet failure: syntax error on line 123 in foo.pp" immediately after the failure, not that thing we see 10 hours later:) [11:14:21] apergos: always need a scapehoat [11:14:33] lemme see who the official scapegoat is [11:14:34] RECOVERY - Puppet freshness on mw116 is OK: puppet ran at Tue Jul 16 11:14:32 UTC 2013 [11:14:34] RECOVERY - Puppet freshness on mw28 is OK: puppet ran at Tue Jul 16 11:14:32 UTC 2013 [11:14:34] RECOVERY - Puppet freshness on analytics1003 is OK: puppet ran at Tue Jul 16 11:14:32 UTC 2013 [11:14:44] RECOVERY - Puppet freshness on mw23 is OK: puppet ran at Tue Jul 16 11:14:37 UTC 2013 [11:14:44] RECOVERY - Puppet freshness on mw1213 is OK: puppet ran at Tue Jul 16 11:14:37 UTC 2013 [11:14:44] RECOVERY - Puppet freshness on search1012 is OK: puppet ran at Tue Jul 16 11:14:37 UTC 2013 [11:14:44] RECOVERY - Puppet freshness on sq52 is OK: puppet ran at Tue Jul 16 11:14:37 UTC 2013 [11:14:44] RECOVERY - Puppet freshness on labstore3 is OK: puppet ran at Tue Jul 16 11:14:37 UTC 2013 [11:14:44] RECOVERY - Puppet freshness on mw1204 is OK: puppet ran at Tue Jul 16 11:14:37 UTC 2013 [11:14:45] RECOVERY - Puppet freshness on sq41 is OK: puppet ran at Tue Jul 16 11:14:37 UTC 2013 [11:14:45] RECOVERY - Puppet freshness on mw1129 is OK: puppet ran at Tue Jul 16 11:14:37 UTC 2013 [11:14:46] RECOVERY - Puppet freshness on cp3011 is OK: puppet ran at Tue Jul 16 11:14:37 UTC 2013 [11:14:46] RECOVERY - Puppet freshness on labstore1001 is OK: puppet ran at Tue Jul 16 11:14:37 UTC 2013 [11:14:47] RECOVERY - Puppet freshness on db1014 is OK: puppet ran at Tue Jul 16 11:14:42 UTC 2013 [11:14:47] RECOVERY - Puppet freshness on antimony is OK: puppet ran at Tue Jul 16 11:14:42 UTC 2013 [11:14:48] RECOVERY - Puppet freshness on solr1 is OK: puppet ran at Tue Jul 16 11:14:42 UTC 2013 [11:14:48] RECOVERY - Puppet freshness on dataset1001 is OK: puppet ran at Tue Jul 16 11:14:42 UTC 2013 [11:14:54] RECOVERY - Puppet freshness on wtp1001 is OK: puppet ran at Tue Jul 16 11:14:47 UTC 2013 [11:14:54] RECOVERY - Puppet freshness on cp1058 is OK: puppet ran at Tue Jul 16 11:14:47 UTC 2013 [11:14:54] RECOVERY - Puppet freshness on ms-be12 is OK: puppet ran at Tue Jul 16 11:14:47 UTC 2013 [11:14:54] RECOVERY - Puppet freshness on ms-fe1002 is OK: puppet ran at Tue Jul 16 11:14:47 UTC 2013 [11:14:54] RECOVERY - Puppet freshness on srv242 is OK: puppet ran at Tue Jul 16 11:14:52 UTC 2013 [11:14:54] RECOVERY - Puppet freshness on mw55 is OK: puppet ran at Tue Jul 16 11:14:52 UTC 2013 [11:14:54] RECOVERY - Puppet freshness on mw1208 is OK: puppet ran at Tue Jul 16 11:14:52 UTC 2013 [11:15:08] I assume it's the one who is on RT duty [11:15:08] RECOVERY - Puppet freshness on mw42 is OK: puppet ran at Tue Jul 16 11:14:57 UTC 2013 [11:15:08] RECOVERY - Puppet freshness on mw75 is OK: puppet ran at Tue Jul 16 11:14:57 UTC 2013 [11:15:08] RECOVERY - Puppet freshness on mw1206 is OK: puppet ran at Tue Jul 16 11:14:57 UTC 2013 [11:15:08] RECOVERY - Puppet freshness on mw1131 is OK: puppet ran at Tue Jul 16 11:14:57 UTC 2013 [11:15:08] RECOVERY - Puppet freshness on mw1021 is OK: puppet ran at Tue Jul 16 11:14:57 UTC 2013 [11:15:08] RECOVERY - Puppet freshness on mw1155 is OK: puppet ran at Tue Jul 16 11:15:02 UTC 2013 [11:15:08] RECOVERY - Puppet freshness on mw1104 is OK: puppet ran at Tue Jul 16 11:15:02 UTC 2013 [11:15:08] RECOVERY - Puppet freshness on ssl1 is OK: puppet ran at Tue Jul 16 11:15:02 UTC 2013 [11:15:09] http://en.wikipedia.org/wiki/User:MZMcBride/Blame_wheel hahaha it says domas [11:15:11] I buy that [11:15:27] the one on rt duty fixed it so they are off the hook for sure [11:15:44] RECOVERY - Puppet freshness on cp1037 is OK: puppet ran at Tue Jul 16 11:15:42 UTC 2013 [11:15:47] hehe [11:15:54] RECOVERY - Puppet freshness on mw1154 is OK: puppet ran at Tue Jul 16 11:15:47 UTC 2013 [11:15:54] RECOVERY - Puppet freshness on bast1001 is OK: puppet ran at Tue Jul 16 11:15:47 UTC 2013 [11:15:54] RECOVERY - Puppet freshness on sq84 is OK: puppet ran at Tue Jul 16 11:15:47 UTC 2013 [11:15:54] RECOVERY - Puppet freshness on db1026 is OK: puppet ran at Tue Jul 16 11:15:47 UTC 2013 [11:15:54] RECOVERY - Puppet freshness on es1004 is OK: puppet ran at Tue Jul 16 11:15:47 UTC 2013 [11:15:54] RECOVERY - Puppet freshness on cp1018 is OK: puppet ran at Tue Jul 16 11:15:47 UTC 2013 [11:15:54] RECOVERY - Puppet freshness on cp1017 is OK: puppet ran at Tue Jul 16 11:15:47 UTC 2013 [11:15:55] RECOVERY - Puppet freshness on amssq51 is OK: puppet ran at Tue Jul 16 11:15:47 UTC 2013 [11:15:55] RECOVERY - Puppet freshness on amssq56 is OK: puppet ran at Tue Jul 16 11:15:47 UTC 2013 [11:15:56] RECOVERY - Puppet freshness on sodium is OK: puppet ran at Tue Jul 16 11:15:47 UTC 2013 [11:15:56] RECOVERY - Puppet freshness on srv261 is OK: puppet ran at Tue Jul 16 11:15:53 UTC 2013 [11:16:04] RECOVERY - Puppet freshness on analytics1002 is OK: puppet ran at Tue Jul 16 11:15:58 UTC 2013 [11:16:04] RECOVERY - Puppet freshness on mw1049 is OK: puppet ran at Tue Jul 16 11:15:58 UTC 2013 [11:16:04] RECOVERY - Puppet freshness on mw1194 is OK: puppet ran at Tue Jul 16 11:16:03 UTC 2013 [11:16:04] RECOVERY - Puppet freshness on analytics1022 is OK: puppet ran at Tue Jul 16 11:16:03 UTC 2013 [11:16:04] RECOVERY - Puppet freshness on mw1128 is OK: puppet ran at Tue Jul 16 11:16:03 UTC 2013 [11:16:04] RECOVERY - Puppet freshness on tmh1001 is OK: puppet ran at Tue Jul 16 11:16:03 UTC 2013 [11:16:04] RECOVERY - Puppet freshness on search1003 is OK: puppet ran at Tue Jul 16 11:16:03 UTC 2013 [11:16:05] RECOVERY - Puppet freshness on mw1047 is OK: puppet ran at Tue Jul 16 11:16:03 UTC 2013 [11:16:14] RECOVERY - Puppet freshness on mw1084 is OK: puppet ran at Tue Jul 16 11:16:08 UTC 2013 [11:16:14] RECOVERY - Puppet freshness on cp1066 is OK: puppet ran at Tue Jul 16 11:16:08 UTC 2013 [11:16:14] RECOVERY - Puppet freshness on mw1137 is OK: puppet ran at Tue Jul 16 11:16:08 UTC 2013 [11:16:14] RECOVERY - Puppet freshness on hume is OK: puppet ran at Tue Jul 16 11:16:08 UTC 2013 [11:16:14] RECOVERY - Puppet freshness on ms-be5 is OK: puppet ran at Tue Jul 16 11:16:13 UTC 2013 [11:16:14] RECOVERY - Puppet freshness on db9 is OK: puppet ran at Tue Jul 16 11:16:13 UTC 2013 [11:16:15] RECOVERY - Puppet freshness on db1027 is OK: puppet ran at Tue Jul 16 11:16:13 UTC 2013 [11:16:15] RECOVERY - Puppet freshness on ssl1003 is OK: puppet ran at Tue Jul 16 11:16:13 UTC 2013 [11:16:15] RECOVERY - Puppet freshness on cp1060 is OK: puppet ran at Tue Jul 16 11:16:13 UTC 2013 [11:16:16] RECOVERY - Puppet freshness on ms6 is OK: puppet ran at Tue Jul 16 11:16:13 UTC 2013 [11:16:22] MaxSem: it's assumed you check on a host or two puppet changes you push out [11:16:24] RECOVERY - Puppet freshness on srv283 is OK: puppet ran at Tue Jul 16 11:16:18 UTC 2013 [11:16:24] RECOVERY - Puppet freshness on mw93 is OK: puppet ran at Tue Jul 16 11:16:18 UTC 2013 [11:16:24] RECOVERY - Puppet freshness on capella is OK: puppet ran at Tue Jul 16 11:16:18 UTC 2013 [11:16:24] RECOVERY - Puppet freshness on snapshot1002 is OK: puppet ran at Tue Jul 16 11:16:18 UTC 2013 [11:16:24] RECOVERY - Puppet freshness on mw61 is OK: puppet ran at Tue Jul 16 11:16:23 UTC 2013 [11:16:25] RECOVERY - Puppet freshness on ssl4 is OK: puppet ran at Tue Jul 16 11:16:23 UTC 2013 [11:16:25] RECOVERY - Puppet freshness on srv299 is OK: puppet ran at Tue Jul 16 11:16:23 UTC 2013 [11:16:25] RECOVERY - Puppet freshness on mw1180 is OK: puppet ran at Tue Jul 16 11:16:23 UTC 2013 [11:16:26] RECOVERY - Puppet freshness on search1005 is OK: puppet ran at Tue Jul 16 11:16:23 UTC 2013 [11:16:26] RECOVERY - Puppet freshness on sq62 is OK: puppet ran at Tue Jul 16 11:16:23 UTC 2013 [11:16:34] RECOVERY - Puppet freshness on mw1211 is OK: puppet ran at Tue Jul 16 11:16:28 UTC 2013 [11:16:34] RECOVERY - Puppet freshness on mw1105 is OK: puppet ran at Tue Jul 16 11:16:28 UTC 2013 [11:16:34] RECOVERY - Puppet freshness on sanger is OK: puppet ran at Tue Jul 16 11:16:28 UTC 2013 [11:16:44] RECOVERY - Puppet freshness on cp1016 is OK: puppet ran at Tue Jul 16 11:16:33 UTC 2013 [11:16:44] RECOVERY - Puppet freshness on pdf2 is OK: puppet ran at Tue Jul 16 11:16:33 UTC 2013 [11:16:45] RECOVERY - Puppet freshness on solr1001 is OK: puppet ran at Tue Jul 16 11:16:38 UTC 2013 [11:16:45] RECOVERY - Puppet freshness on es9 is OK: puppet ran at Tue Jul 16 11:16:38 UTC 2013 [11:16:45] RECOVERY - Puppet freshness on db69 is OK: puppet ran at Tue Jul 16 11:16:38 UTC 2013 [11:16:45] RECOVERY - Puppet freshness on labstore1 is OK: puppet ran at Tue Jul 16 11:16:38 UTC 2013 [11:16:54] RECOVERY - Puppet freshness on mc1012 is OK: puppet ran at Tue Jul 16 11:16:43 UTC 2013 [11:16:54] RECOVERY - Puppet freshness on labstore2 is OK: puppet ran at Tue Jul 16 11:16:44 UTC 2013 [11:16:54] RECOVERY - Puppet freshness on amslvs1 is OK: puppet ran at Tue Jul 16 11:16:44 UTC 2013 [11:16:54] RECOVERY - Puppet freshness on db1005 is OK: puppet ran at Tue Jul 16 11:16:44 UTC 2013 [11:16:54] RECOVERY - Puppet freshness on cp1062 is OK: puppet ran at Tue Jul 16 11:16:44 UTC 2013 [11:16:54] RECOVERY - Puppet freshness on mw13 is OK: puppet ran at Tue Jul 16 11:16:49 UTC 2013 [11:17:04] RECOVERY - Puppet freshness on search31 is OK: puppet ran at Tue Jul 16 11:16:54 UTC 2013 [11:17:04] RECOVERY - Puppet freshness on mw1181 is OK: puppet ran at Tue Jul 16 11:16:55 UTC 2013 [11:17:04] RECOVERY - Puppet freshness on srv291 is OK: puppet ran at Tue Jul 16 11:16:55 UTC 2013 [11:17:04] RECOVERY - Puppet freshness on mw45 is OK: puppet ran at Tue Jul 16 11:16:55 UTC 2013 [11:17:04] RECOVERY - Puppet freshness on db32 is OK: puppet ran at Tue Jul 16 11:17:00 UTC 2013 [11:17:04] RECOVERY - Puppet freshness on db1009 is OK: puppet ran at Tue Jul 16 11:17:00 UTC 2013 [11:17:04] RECOVERY - Puppet freshness on mw1075 is OK: puppet ran at Tue Jul 16 11:17:00 UTC 2013 [11:17:05] RECOVERY - Puppet freshness on db67 is OK: puppet ran at Tue Jul 16 11:17:00 UTC 2013 [11:17:05] RECOVERY - Puppet freshness on mw1165 is OK: puppet ran at Tue Jul 16 11:17:00 UTC 2013 [11:17:06] RECOVERY - Puppet freshness on search1015 is OK: puppet ran at Tue Jul 16 11:17:00 UTC 2013 [11:17:06] RECOVERY - Puppet freshness on mw1030 is OK: puppet ran at Tue Jul 16 11:17:00 UTC 2013 [11:17:07] RECOVERY - Puppet freshness on mw1078 is OK: puppet ran at Tue Jul 16 11:17:00 UTC 2013 [11:17:14] RECOVERY - Puppet freshness on mw1020 is OK: puppet ran at Tue Jul 16 11:17:10 UTC 2013 [11:17:24] RECOVERY - Puppet freshness on lvs1002 is OK: puppet ran at Tue Jul 16 11:17:15 UTC 2013 [11:17:34] RECOVERY - Puppet freshness on nickel is OK: puppet ran at Tue Jul 16 11:17:30 UTC 2013 [11:17:34] RECOVERY - Puppet freshness on amslvs2 is OK: puppet ran at Tue Jul 16 11:17:30 UTC 2013 [11:17:44] RECOVERY - Puppet freshness on amssq37 is OK: puppet ran at Tue Jul 16 11:17:35 UTC 2013 [11:17:44] RECOVERY - Puppet freshness on db77 is OK: puppet ran at Tue Jul 16 11:17:35 UTC 2013 [11:17:44] RECOVERY - Puppet freshness on ms-fe2 is OK: puppet ran at Tue Jul 16 11:17:40 UTC 2013 [11:17:48] 319 to go [11:17:54] RECOVERY - Puppet freshness on ms-be1007 is OK: puppet ran at Tue Jul 16 11:17:45 UTC 2013 [11:17:54] RECOVERY - Puppet freshness on db65 is OK: puppet ran at Tue Jul 16 11:17:45 UTC 2013 [11:17:54] RECOVERY - Puppet freshness on db1058 is OK: puppet ran at Tue Jul 16 11:17:45 UTC 2013 [11:17:54] RECOVERY - Puppet freshness on wtp1002 is OK: puppet ran at Tue Jul 16 11:17:45 UTC 2013 [11:17:54] RECOVERY - Puppet freshness on ms-be1009 is OK: puppet ran at Tue Jul 16 11:17:50 UTC 2013 [11:17:54] RECOVERY - Puppet freshness on cp1038 is OK: puppet ran at Tue Jul 16 11:17:50 UTC 2013 [11:17:54] RECOVERY - Puppet freshness on amssq31 is OK: puppet ran at Tue Jul 16 11:17:50 UTC 2013 [11:17:55] RECOVERY - Puppet freshness on amssq35 is OK: puppet ran at Tue Jul 16 11:17:50 UTC 2013 [11:18:04] RECOVERY - Puppet freshness on mw95 is OK: puppet ran at Tue Jul 16 11:17:55 UTC 2013 [11:18:04] RECOVERY - Puppet freshness on srv264 is OK: puppet ran at Tue Jul 16 11:17:55 UTC 2013 [11:18:04] RECOVERY - Puppet freshness on mw99 is OK: puppet ran at Tue Jul 16 11:17:55 UTC 2013 [11:18:04] RECOVERY - Puppet freshness on mw39 is OK: puppet ran at Tue Jul 16 11:18:00 UTC 2013 [11:18:04] RECOVERY - Puppet freshness on mw1198 is OK: puppet ran at Tue Jul 16 11:18:00 UTC 2013 [11:18:04] RECOVERY - Puppet freshness on mw1169 is OK: puppet ran at Tue Jul 16 11:18:00 UTC 2013 [11:18:04] RECOVERY - Puppet freshness on mw1053 is OK: puppet ran at Tue Jul 16 11:18:00 UTC 2013 [11:18:05] RECOVERY - Puppet freshness on search1023 is OK: puppet ran at Tue Jul 16 11:18:00 UTC 2013 [11:18:05] RECOVERY - Puppet freshness on mw1116 is OK: puppet ran at Tue Jul 16 11:18:00 UTC 2013 [11:18:14] RECOVERY - Puppet freshness on mw1094 is OK: puppet ran at Tue Jul 16 11:18:05 UTC 2013 [11:18:14] RECOVERY - Puppet freshness on mw1074 is OK: puppet ran at Tue Jul 16 11:18:05 UTC 2013 [11:18:14] RECOVERY - Puppet freshness on mw1083 is OK: puppet ran at Tue Jul 16 11:18:05 UTC 2013 [11:18:14] RECOVERY - Puppet freshness on search1021 is OK: puppet ran at Tue Jul 16 11:18:05 UTC 2013 [11:18:14] RECOVERY - Puppet freshness on mw1136 is OK: puppet ran at Tue Jul 16 11:18:11 UTC 2013 [11:18:24] RECOVERY - Puppet freshness on mw74 is OK: puppet ran at Tue Jul 16 11:18:21 UTC 2013 [11:18:34] RECOVERY - Puppet freshness on brewster is OK: puppet ran at Tue Jul 16 11:18:26 UTC 2013 [11:18:34] RECOVERY - Puppet freshness on db57 is OK: puppet ran at Tue Jul 16 11:18:26 UTC 2013 [11:18:34] RECOVERY - Puppet freshness on sq50 is OK: puppet ran at Tue Jul 16 11:18:26 UTC 2013 [11:18:34] RECOVERY - Puppet freshness on stat1002 is OK: puppet ran at Tue Jul 16 11:18:31 UTC 2013 [11:18:34] RECOVERY - Puppet freshness on db1024 is OK: puppet ran at Tue Jul 16 11:18:31 UTC 2013 [11:18:44] RECOVERY - Puppet freshness on dataset2 is OK: puppet ran at Tue Jul 16 11:18:36 UTC 2013 [11:18:44] RECOVERY - Puppet freshness on db51 is OK: puppet ran at Tue Jul 16 11:18:36 UTC 2013 [11:18:44] RECOVERY - Puppet freshness on cp1065 is OK: puppet ran at Tue Jul 16 11:18:36 UTC 2013 [11:18:44] RECOVERY - Puppet freshness on es5 is OK: puppet ran at Tue Jul 16 11:18:36 UTC 2013 [11:18:44] RECOVERY - Puppet freshness on wtp1013 is OK: puppet ran at Tue Jul 16 11:18:41 UTC 2013 [11:18:44] RECOVERY - Puppet freshness on mc1008 is OK: puppet ran at Tue Jul 16 11:18:41 UTC 2013 [11:18:54] RECOVERY - Puppet freshness on mw33 is OK: puppet ran at Tue Jul 16 11:18:46 UTC 2013 [11:18:54] RECOVERY - Puppet freshness on mw36 is OK: puppet ran at Tue Jul 16 11:18:46 UTC 2013 [11:18:54] RECOVERY - Puppet freshness on mw26 is OK: puppet ran at Tue Jul 16 11:18:46 UTC 2013 [11:18:54] RECOVERY - Puppet freshness on mw1179 is OK: puppet ran at Tue Jul 16 11:18:51 UTC 2013 [11:18:54] RECOVERY - Puppet freshness on mw64 is OK: puppet ran at Tue Jul 16 11:18:51 UTC 2013 [11:18:54] RECOVERY - Puppet freshness on mw1138 is OK: puppet ran at Tue Jul 16 11:18:51 UTC 2013 [11:18:54] RECOVERY - Puppet freshness on analytics1011 is OK: puppet ran at Tue Jul 16 11:18:51 UTC 2013 [11:19:04] RECOVERY - Puppet freshness on analytics1009 is OK: puppet ran at Tue Jul 16 11:19:01 UTC 2013 [11:19:34] RECOVERY - Puppet freshness on db46 is OK: puppet ran at Tue Jul 16 11:19:26 UTC 2013 [11:19:34] RECOVERY - Puppet freshness on sq68 is OK: puppet ran at Tue Jul 16 11:19:26 UTC 2013 [11:19:34] RECOVERY - Puppet freshness on tarin is OK: puppet ran at Tue Jul 16 11:19:26 UTC 2013 [11:19:34] RECOVERY - Puppet freshness on labsdb1001 is OK: puppet ran at Tue Jul 16 11:19:31 UTC 2013 [11:19:34] RECOVERY - Puppet freshness on db55 is OK: puppet ran at Tue Jul 16 11:19:31 UTC 2013 [11:19:34] RECOVERY - Puppet freshness on analytics1008 is OK: puppet ran at Tue Jul 16 11:19:31 UTC 2013 [11:19:34] RECOVERY - Puppet freshness on wtp1021 is OK: puppet ran at Tue Jul 16 11:19:31 UTC 2013 [11:19:44] RECOVERY - Puppet freshness on mw107 is OK: puppet ran at Tue Jul 16 11:19:41 UTC 2013 [11:19:44] RECOVERY - Puppet freshness on srv272 is OK: puppet ran at Tue Jul 16 11:19:41 UTC 2013 [11:19:44] RECOVERY - Puppet freshness on mw1218 is OK: puppet ran at Tue Jul 16 11:19:41 UTC 2013 [11:19:44] RECOVERY - Puppet freshness on srv296 is OK: puppet ran at Tue Jul 16 11:19:41 UTC 2013 [11:19:54] RECOVERY - Puppet freshness on mw53 is OK: puppet ran at Tue Jul 16 11:19:46 UTC 2013 [11:19:54] RECOVERY - Puppet freshness on mw40 is OK: puppet ran at Tue Jul 16 11:19:46 UTC 2013 [11:19:54] RECOVERY - Puppet freshness on es1009 is OK: puppet ran at Tue Jul 16 11:19:46 UTC 2013 [11:19:54] RECOVERY - Puppet freshness on mw1001 is OK: puppet ran at Tue Jul 16 11:19:51 UTC 2013 [11:19:54] RECOVERY - Puppet freshness on mw1062 is OK: puppet ran at Tue Jul 16 11:19:51 UTC 2013 [11:19:54] RECOVERY - Puppet freshness on snapshot4 is OK: puppet ran at Tue Jul 16 11:19:51 UTC 2013 [11:19:54] RECOVERY - Puppet freshness on mw109 is OK: puppet ran at Tue Jul 16 11:19:51 UTC 2013 [11:20:04] RECOVERY - Puppet freshness on db1010 is OK: puppet ran at Tue Jul 16 11:19:56 UTC 2013 [11:20:04] RECOVERY - Puppet freshness on mw70 is OK: puppet ran at Tue Jul 16 11:20:02 UTC 2013 [11:20:04] RECOVERY - Puppet freshness on mw1185 is OK: puppet ran at Tue Jul 16 11:20:02 UTC 2013 [11:20:14] RECOVERY - Puppet freshness on mw1040 is OK: puppet ran at Tue Jul 16 11:20:07 UTC 2013 [11:20:14] RECOVERY - Puppet freshness on mw1022 is OK: puppet ran at Tue Jul 16 11:20:07 UTC 2013 [11:20:14] RECOVERY - Puppet freshness on mw1132 is OK: puppet ran at Tue Jul 16 11:20:07 UTC 2013 [11:20:39] RECOVERY - Puppet freshness on pdf3 is OK: puppet ran at Tue Jul 16 11:20:27 UTC 2013 [11:20:39] RECOVERY - Puppet freshness on sq64 is OK: puppet ran at Tue Jul 16 11:20:32 UTC 2013 [11:20:39] RECOVERY - Puppet freshness on mc1011 is OK: puppet ran at Tue Jul 16 11:20:32 UTC 2013 [11:20:39] RECOVERY - Puppet freshness on colby is OK: puppet ran at Tue Jul 16 11:20:32 UTC 2013 [11:20:39] RECOVERY - Puppet freshness on mw117 is OK: puppet ran at Tue Jul 16 11:20:32 UTC 2013 [11:20:39] RECOVERY - Puppet freshness on sq81 is OK: puppet ran at Tue Jul 16 11:20:32 UTC 2013 [11:20:39] RECOVERY - Puppet freshness on aluminium is OK: puppet ran at Tue Jul 16 11:20:32 UTC 2013 [11:20:39] RECOVERY - Puppet freshness on sq55 is OK: puppet ran at Tue Jul 16 11:20:32 UTC 2013 [11:20:44] RECOVERY - Puppet freshness on amssq59 is OK: puppet ran at Tue Jul 16 11:20:37 UTC 2013 [11:20:44] RECOVERY - Puppet freshness on analytics1001 is OK: puppet ran at Tue Jul 16 11:20:37 UTC 2013 [11:20:44] RECOVERY - Puppet freshness on srv285 is OK: puppet ran at Tue Jul 16 11:20:42 UTC 2013 [11:20:44] RECOVERY - Puppet freshness on mw87 is OK: puppet ran at Tue Jul 16 11:20:42 UTC 2013 [11:20:44] RECOVERY - Puppet freshness on mw111 is OK: puppet ran at Tue Jul 16 11:20:42 UTC 2013 [11:20:44] RECOVERY - Puppet freshness on professor is OK: puppet ran at Tue Jul 16 11:20:42 UTC 2013 [11:20:44] RECOVERY - Puppet freshness on search27 is OK: puppet ran at Tue Jul 16 11:20:42 UTC 2013 [11:20:54] RECOVERY - Puppet freshness on mw1072 is OK: puppet ran at Tue Jul 16 11:20:47 UTC 2013 [11:20:54] RECOVERY - Puppet freshness on mw1178 is OK: puppet ran at Tue Jul 16 11:20:47 UTC 2013 [11:20:54] RECOVERY - Puppet freshness on lvs6 is OK: puppet ran at Tue Jul 16 11:20:52 UTC 2013 [11:20:54] RECOVERY - Puppet freshness on search1011 is OK: puppet ran at Tue Jul 16 11:20:53 UTC 2013 [11:21:04] RECOVERY - Puppet freshness on mw1219 is OK: puppet ran at Tue Jul 16 11:21:03 UTC 2013 [11:21:04] RECOVERY - Puppet freshness on search1006 is OK: puppet ran at Tue Jul 16 11:21:03 UTC 2013 [11:21:04] RECOVERY - Puppet freshness on db31 is OK: puppet ran at Tue Jul 16 11:21:03 UTC 2013 [11:21:04] RECOVERY - Puppet freshness on virt2 is OK: puppet ran at Tue Jul 16 11:21:03 UTC 2013 [11:21:14] RECOVERY - Puppet freshness on cp3006 is OK: puppet ran at Tue Jul 16 11:21:08 UTC 2013 [11:21:44] RECOVERY - Puppet freshness on db1011 is OK: puppet ran at Tue Jul 16 11:21:43 UTC 2013 [11:21:54] RECOVERY - Puppet freshness on mw1139 is OK: puppet ran at Tue Jul 16 11:21:48 UTC 2013 [11:21:54] RECOVERY - Puppet freshness on db60 is OK: puppet ran at Tue Jul 16 11:21:48 UTC 2013 [11:21:54] RECOVERY - Puppet freshness on db59 is OK: puppet ran at Tue Jul 16 11:21:48 UTC 2013 [11:21:54] RECOVERY - Puppet freshness on mw1031 is OK: puppet ran at Tue Jul 16 11:21:48 UTC 2013 [11:21:54] RECOVERY - Puppet freshness on mw1093 is OK: puppet ran at Tue Jul 16 11:21:48 UTC 2013 [11:21:54] RECOVERY - Puppet freshness on mw1134 is OK: puppet ran at Tue Jul 16 11:21:48 UTC 2013 [11:21:54] RECOVERY - Puppet freshness on wtp1010 is OK: puppet ran at Tue Jul 16 11:21:48 UTC 2013 [11:21:55] RECOVERY - Puppet freshness on sq78 is OK: puppet ran at Tue Jul 16 11:21:48 UTC 2013 [11:21:55] RECOVERY - Puppet freshness on mchenry is OK: puppet ran at Tue Jul 16 11:21:53 UTC 2013 [11:21:56] RECOVERY - Puppet freshness on db73 is OK: puppet ran at Tue Jul 16 11:21:53 UTC 2013 [11:21:56] RECOVERY - Puppet freshness on amssq43 is OK: puppet ran at Tue Jul 16 11:21:53 UTC 2013 [11:21:57] RECOVERY - Puppet freshness on mw1080 is OK: puppet ran at Tue Jul 16 11:21:53 UTC 2013 [11:21:57] RECOVERY - Puppet freshness on srv270 is OK: puppet ran at Tue Jul 16 11:21:53 UTC 2013 [11:21:58] RECOVERY - Puppet freshness on srv282 is OK: puppet ran at Tue Jul 16 11:21:53 UTC 2013 [11:21:58] RECOVERY - Puppet freshness on mw114 is OK: puppet ran at Tue Jul 16 11:21:53 UTC 2013 [11:21:59] RECOVERY - Puppet freshness on mw115 is OK: puppet ran at Tue Jul 16 11:21:53 UTC 2013 [11:21:59] RECOVERY - Puppet freshness on sq82 is OK: puppet ran at Tue Jul 16 11:21:53 UTC 2013 [11:22:00] RECOVERY - Puppet freshness on db36 is OK: puppet ran at Tue Jul 16 11:21:53 UTC 2013 [11:22:00] RECOVERY - Puppet freshness on amssq32 is OK: puppet ran at Tue Jul 16 11:21:53 UTC 2013 [11:22:01] RECOVERY - Puppet freshness on mc1013 is OK: puppet ran at Tue Jul 16 11:21:53 UTC 2013 [11:22:01] RECOVERY - Puppet freshness on magnesium is OK: puppet ran at Tue Jul 16 11:21:53 UTC 2013 [11:22:02] RECOVERY - Puppet freshness on db29 is OK: puppet ran at Tue Jul 16 11:21:53 UTC 2013 [11:22:04] RECOVERY - Puppet freshness on srv259 is OK: puppet ran at Tue Jul 16 11:21:58 UTC 2013 [11:22:04] RECOVERY - Puppet freshness on cp1015 is OK: puppet ran at Tue Jul 16 11:21:58 UTC 2013 [11:22:04] RECOVERY - Puppet freshness on srv280 is OK: puppet ran at Tue Jul 16 11:21:58 UTC 2013 [11:22:04] RECOVERY - Puppet freshness on tmh1 is OK: puppet ran at Tue Jul 16 11:21:58 UTC 2013 [11:22:04] RECOVERY - Puppet freshness on mw52 is OK: puppet ran at Tue Jul 16 11:21:58 UTC 2013 [11:22:14] RECOVERY - Puppet freshness on terbium is OK: puppet ran at Tue Jul 16 11:22:03 UTC 2013 [11:22:14] RECOVERY - Puppet freshness on mw1200 is OK: puppet ran at Tue Jul 16 11:22:03 UTC 2013 [11:22:14] RECOVERY - Puppet freshness on mw1203 is OK: puppet ran at Tue Jul 16 11:22:03 UTC 2013 [11:22:14] RECOVERY - Puppet freshness on search1004 is OK: puppet ran at Tue Jul 16 11:22:03 UTC 2013 [11:22:14] RECOVERY - Puppet freshness on ms-be1004 is OK: puppet ran at Tue Jul 16 11:22:08 UTC 2013 [11:22:24] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:22:54] RECOVERY - Puppet freshness on mw1141 is OK: puppet ran at Tue Jul 16 11:22:44 UTC 2013 [11:22:54] RECOVERY - Puppet freshness on sq59 is OK: puppet ran at Tue Jul 16 11:22:44 UTC 2013 [11:22:54] RECOVERY - Puppet freshness on mw1215 is OK: puppet ran at Tue Jul 16 11:22:49 UTC 2013 [11:23:04] RECOVERY - Puppet freshness on ms-be1002 is OK: puppet ran at Tue Jul 16 11:22:54 UTC 2013 [11:23:04] RECOVERY - Puppet freshness on srv240 is OK: puppet ran at Tue Jul 16 11:22:54 UTC 2013 [11:23:04] RECOVERY - Puppet freshness on mw1045 is OK: puppet ran at Tue Jul 16 11:22:59 UTC 2013 [11:23:04] RECOVERY - Puppet freshness on mw1220 is OK: puppet ran at Tue Jul 16 11:22:59 UTC 2013 [11:23:04] RECOVERY - Puppet freshness on wtp1020 is OK: puppet ran at Tue Jul 16 11:22:59 UTC 2013 [11:23:04] RECOVERY - Puppet freshness on linne is OK: puppet ran at Tue Jul 16 11:22:59 UTC 2013 [11:23:04] RECOVERY - Puppet freshness on ms-fe1003 is OK: puppet ran at Tue Jul 16 11:22:59 UTC 2013 [11:23:05] RECOVERY - Puppet freshness on db56 is OK: puppet ran at Tue Jul 16 11:22:59 UTC 2013 [11:23:05] RECOVERY - Puppet freshness on es1008 is OK: puppet ran at Tue Jul 16 11:22:59 UTC 2013 [11:23:06] RECOVERY - Puppet freshness on db1056 is OK: puppet ran at Tue Jul 16 11:22:59 UTC 2013 [11:23:06] RECOVERY - Puppet freshness on wtp1008 is OK: puppet ran at Tue Jul 16 11:22:59 UTC 2013 [11:23:07] RECOVERY - Puppet freshness on sq37 is OK: puppet ran at Tue Jul 16 11:22:59 UTC 2013 [11:23:07] RECOVERY - Puppet freshness on ms-be1003 is OK: puppet ran at Tue Jul 16 11:22:59 UTC 2013 [11:23:08] RECOVERY - Puppet freshness on mw1112 is OK: puppet ran at Tue Jul 16 11:22:59 UTC 2013 [11:23:14] RECOVERY - Puppet freshness on analytics1017 is OK: puppet ran at Tue Jul 16 11:23:04 UTC 2013 [11:23:14] RECOVERY - Puppet freshness on cp1054 is OK: puppet ran at Tue Jul 16 11:23:04 UTC 2013 [11:23:14] RECOVERY - Puppet freshness on stat1001 is OK: puppet ran at Tue Jul 16 11:23:04 UTC 2013 [11:23:14] RECOVERY - Puppet freshness on mc1006 is OK: puppet ran at Tue Jul 16 11:23:04 UTC 2013 [11:23:14] RECOVERY - Puppet freshness on wtp1006 is OK: puppet ran at Tue Jul 16 11:23:04 UTC 2013 [11:23:14] RECOVERY - Puppet freshness on db1057 is OK: puppet ran at Tue Jul 16 11:23:04 UTC 2013 [11:23:14] RECOVERY - Puppet freshness on amssq40 is OK: puppet ran at Tue Jul 16 11:23:04 UTC 2013 [11:23:15] RECOVERY - Puppet freshness on cp1039 is OK: puppet ran at Tue Jul 16 11:23:04 UTC 2013 [11:23:15] RECOVERY - Puppet freshness on ssl1002 is OK: puppet ran at Tue Jul 16 11:23:04 UTC 2013 [11:23:16] RECOVERY - Puppet freshness on mw91 is OK: puppet ran at Tue Jul 16 11:23:09 UTC 2013 [11:23:16] RECOVERY - Puppet freshness on srv271 is OK: puppet ran at Tue Jul 16 11:23:09 UTC 2013 [11:23:17] RECOVERY - Puppet freshness on mw92 is OK: puppet ran at Tue Jul 16 11:23:09 UTC 2013 [11:23:17] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.125 second response time [11:23:24] RECOVERY - Puppet freshness on search35 is OK: puppet ran at Tue Jul 16 11:23:14 UTC 2013 [11:23:24] RECOVERY - Puppet freshness on mw1174 is OK: puppet ran at Tue Jul 16 11:23:14 UTC 2013 [11:23:24] RECOVERY - Puppet freshness on mw1135 is OK: puppet ran at Tue Jul 16 11:23:14 UTC 2013 [11:23:24] RECOVERY - Puppet freshness on emery is OK: puppet ran at Tue Jul 16 11:23:14 UTC 2013 [11:23:24] RECOVERY - Puppet freshness on mw1121 is OK: puppet ran at Tue Jul 16 11:23:14 UTC 2013 [11:23:24] RECOVERY - Puppet freshness on srv268 is OK: puppet ran at Tue Jul 16 11:23:14 UTC 2013 [11:23:24] RECOVERY - Puppet freshness on srv298 is OK: puppet ran at Tue Jul 16 11:23:14 UTC 2013 [11:23:25] RECOVERY - Puppet freshness on mw65 is OK: puppet ran at Tue Jul 16 11:23:19 UTC 2013 [11:23:25] RECOVERY - Puppet freshness on chromium is OK: puppet ran at Tue Jul 16 11:23:19 UTC 2013 [11:23:26] RECOVERY - Puppet freshness on mw15 is OK: puppet ran at Tue Jul 16 11:23:19 UTC 2013 [11:23:26] RECOVERY - Puppet freshness on mw1158 is OK: puppet ran at Tue Jul 16 11:23:19 UTC 2013 [11:23:34] RECOVERY - Puppet freshness on cp1068 is OK: puppet ran at Tue Jul 16 11:23:24 UTC 2013 [11:23:34] RECOVERY - Puppet freshness on mw1090 is OK: puppet ran at Tue Jul 16 11:23:24 UTC 2013 [11:23:34] RECOVERY - Puppet freshness on cp1053 is OK: puppet ran at Tue Jul 16 11:23:24 UTC 2013 [11:23:34] RECOVERY - Puppet freshness on db44 is OK: puppet ran at Tue Jul 16 11:23:24 UTC 2013 [11:23:34] RECOVERY - Puppet freshness on mw1059 is OK: puppet ran at Tue Jul 16 11:23:24 UTC 2013 [11:23:34] RECOVERY - Puppet freshness on mw1086 is OK: puppet ran at Tue Jul 16 11:23:24 UTC 2013 [11:23:34] RECOVERY - Puppet freshness on mw1145 is OK: puppet ran at Tue Jul 16 11:23:24 UTC 2013 [11:23:35] RECOVERY - Puppet freshness on lvs2 is OK: puppet ran at Tue Jul 16 11:23:24 UTC 2013 [11:23:35] RECOVERY - Puppet freshness on amssq50 is OK: puppet ran at Tue Jul 16 11:23:29 UTC 2013 [11:23:44] RECOVERY - Puppet freshness on sq49 is OK: puppet ran at Tue Jul 16 11:23:36 UTC 2013 [11:23:44] RECOVERY - Puppet freshness on es2 is OK: puppet ran at Tue Jul 16 11:23:41 UTC 2013 [11:23:44] RECOVERY - Puppet freshness on cp3021 is OK: puppet ran at Tue Jul 16 11:23:41 UTC 2013 [11:23:54] RECOVERY - Puppet freshness on mc1002 is OK: puppet ran at Tue Jul 16 11:23:46 UTC 2013 [11:23:54] RECOVERY - Puppet freshness on virt1005 is OK: puppet ran at Tue Jul 16 11:23:51 UTC 2013 [11:23:54] RECOVERY - Puppet freshness on virt8 is OK: puppet ran at Tue Jul 16 11:23:51 UTC 2013 [11:23:54] RECOVERY - Puppet freshness on db1045 is OK: puppet ran at Tue Jul 16 11:23:51 UTC 2013 [11:24:04] RECOVERY - Puppet freshness on wtp1016 is OK: puppet ran at Tue Jul 16 11:23:56 UTC 2013 [11:24:04] RECOVERY - Puppet freshness on ms-be1010 is OK: puppet ran at Tue Jul 16 11:23:56 UTC 2013 [11:24:04] RECOVERY - Puppet freshness on mw113 is OK: puppet ran at Tue Jul 16 11:23:56 UTC 2013 [11:24:04] RECOVERY - Puppet freshness on mw1082 is OK: puppet ran at Tue Jul 16 11:23:56 UTC 2013 [11:24:04] RECOVERY - Puppet freshness on mw1026 is OK: puppet ran at Tue Jul 16 11:24:01 UTC 2013 [11:24:04] RECOVERY - Puppet freshness on harmon is OK: puppet ran at Tue Jul 16 11:24:01 UTC 2013 [11:24:14] RECOVERY - Puppet freshness on mw1110 is OK: puppet ran at Tue Jul 16 11:24:06 UTC 2013 [11:24:14] RECOVERY - Puppet freshness on sq77 is OK: puppet ran at Tue Jul 16 11:24:06 UTC 2013 [11:24:14] RECOVERY - Puppet freshness on ms1004 is OK: puppet ran at Tue Jul 16 11:24:06 UTC 2013 [11:24:14] RECOVERY - Puppet freshness on cp1001 is OK: puppet ran at Tue Jul 16 11:24:06 UTC 2013 [11:24:14] RECOVERY - Puppet freshness on srv251 is OK: puppet ran at Tue Jul 16 11:24:06 UTC 2013 [11:24:14] RECOVERY - Puppet freshness on srv243 is OK: puppet ran at Tue Jul 16 11:24:06 UTC 2013 [11:24:14] RECOVERY - Puppet freshness on mw3 is OK: puppet ran at Tue Jul 16 11:24:11 UTC 2013 [11:24:15] RECOVERY - Puppet freshness on mw1120 is OK: puppet ran at Tue Jul 16 11:24:11 UTC 2013 [11:24:15] RECOVERY - Puppet freshness on mw7 is OK: puppet ran at Tue Jul 16 11:24:11 UTC 2013 [11:24:16] RECOVERY - Puppet freshness on mw25 is OK: puppet ran at Tue Jul 16 11:24:11 UTC 2013 [11:24:16] RECOVERY - Puppet freshness on cp3019 is OK: puppet ran at Tue Jul 16 11:24:11 UTC 2013 [11:24:17] RECOVERY - Puppet freshness on analytics1021 is OK: puppet ran at Tue Jul 16 11:24:11 UTC 2013 [11:24:17] RECOVERY - Puppet freshness on mw1008 is OK: puppet ran at Tue Jul 16 11:24:11 UTC 2013 [11:24:18] RECOVERY - Puppet freshness on search23 is OK: puppet ran at Tue Jul 16 11:24:11 UTC 2013 [11:24:18] RECOVERY - Puppet freshness on mw1009 is OK: puppet ran at Tue Jul 16 11:24:11 UTC 2013 [11:24:19] RECOVERY - Puppet freshness on srv293 is OK: puppet ran at Tue Jul 16 11:24:11 UTC 2013 [11:24:24] RECOVERY - Puppet freshness on mw1199 is OK: puppet ran at Tue Jul 16 11:24:16 UTC 2013 [11:24:24] RECOVERY - Puppet freshness on mw1148 is OK: puppet ran at Tue Jul 16 11:24:16 UTC 2013 [11:24:24] RECOVERY - Puppet freshness on mw59 is OK: puppet ran at Tue Jul 16 11:24:21 UTC 2013 [11:24:24] RECOVERY - Puppet freshness on mw1108 is OK: puppet ran at Tue Jul 16 11:24:21 UTC 2013 [11:24:24] RECOVERY - Puppet freshness on mw1 is OK: puppet ran at Tue Jul 16 11:24:21 UTC 2013 [11:24:34] RECOVERY - Puppet freshness on sq66 is OK: puppet ran at Tue Jul 16 11:24:26 UTC 2013 [11:24:34] RECOVERY - Puppet freshness on mw1073 is OK: puppet ran at Tue Jul 16 11:24:26 UTC 2013 [11:24:34] RECOVERY - Puppet freshness on mw1060 is OK: puppet ran at Tue Jul 16 11:24:26 UTC 2013 [11:24:44] RECOVERY - Puppet freshness on sq75 is OK: puppet ran at Tue Jul 16 11:24:36 UTC 2013 [11:24:44] RECOVERY - Puppet freshness on cp1009 is OK: puppet ran at Tue Jul 16 11:24:41 UTC 2013 [11:24:44] RECOVERY - Puppet freshness on cp1008 is OK: puppet ran at Tue Jul 16 11:24:41 UTC 2013 [11:24:44] RECOVERY - Puppet freshness on hydrogen is OK: puppet ran at Tue Jul 16 11:24:41 UTC 2013 [11:24:44] RECOVERY - Puppet freshness on nitrogen is OK: puppet ran at Tue Jul 16 11:24:42 UTC 2013 [11:24:44] RECOVERY - Puppet freshness on db1018 is OK: puppet ran at Tue Jul 16 11:24:42 UTC 2013 [11:24:54] RECOVERY - Puppet freshness on db1046 is OK: puppet ran at Tue Jul 16 11:24:47 UTC 2013 [11:24:54] RECOVERY - Puppet freshness on db1015 is OK: puppet ran at Tue Jul 16 11:24:47 UTC 2013 [11:24:54] RECOVERY - Puppet freshness on mc1016 is OK: puppet ran at Tue Jul 16 11:24:47 UTC 2013 [11:24:54] RECOVERY - Puppet freshness on ms-be10 is OK: puppet ran at Tue Jul 16 11:24:47 UTC 2013 [11:24:54] RECOVERY - Puppet freshness on db1049 is OK: puppet ran at Tue Jul 16 11:24:52 UTC 2013 [11:24:54] RECOVERY - Puppet freshness on db64 is OK: puppet ran at Tue Jul 16 11:24:52 UTC 2013 [11:24:54] RECOVERY - Puppet freshness on snapshot3 is OK: puppet ran at Tue Jul 16 11:24:52 UTC 2013 [11:24:55] RECOVERY - Puppet freshness on ms-fe1004 is OK: puppet ran at Tue Jul 16 11:24:52 UTC 2013 [11:25:04] RECOVERY - Puppet freshness on holmium is OK: puppet ran at Tue Jul 16 11:24:57 UTC 2013 [11:25:04] RECOVERY - Puppet freshness on cp1064 is OK: puppet ran at Tue Jul 16 11:24:57 UTC 2013 [11:25:04] RECOVERY - Puppet freshness on mw1170 is OK: puppet ran at Tue Jul 16 11:25:02 UTC 2013 [11:25:04] RECOVERY - Puppet freshness on srv257 is OK: puppet ran at Tue Jul 16 11:25:02 UTC 2013 [11:25:04] RECOVERY - Puppet freshness on srv274 is OK: puppet ran at Tue Jul 16 11:25:02 UTC 2013 [11:25:04] RECOVERY - Puppet freshness on mw56 is OK: puppet ran at Tue Jul 16 11:25:02 UTC 2013 [11:25:04] RECOVERY - Puppet freshness on mw48 is OK: puppet ran at Tue Jul 16 11:25:02 UTC 2013 [11:25:05] RECOVERY - Puppet freshness on srv235 is OK: puppet ran at Tue Jul 16 11:25:02 UTC 2013 [11:25:14] RECOVERY - Puppet freshness on mw12 is OK: puppet ran at Tue Jul 16 11:25:07 UTC 2013 [11:25:14] RECOVERY - Puppet freshness on mw1058 is OK: puppet ran at Tue Jul 16 11:25:07 UTC 2013 [11:25:14] RECOVERY - Puppet freshness on mw1019 is OK: puppet ran at Tue Jul 16 11:25:07 UTC 2013 [11:25:14] RECOVERY - Puppet freshness on mw1070 is OK: puppet ran at Tue Jul 16 11:25:07 UTC 2013 [11:25:14] RECOVERY - Puppet freshness on mw1157 is OK: puppet ran at Tue Jul 16 11:25:07 UTC 2013 [11:25:14] RECOVERY - Puppet freshness on mw1061 is OK: puppet ran at Tue Jul 16 11:25:07 UTC 2013 [11:25:14] RECOVERY - Puppet freshness on mw1119 is OK: puppet ran at Tue Jul 16 11:25:12 UTC 2013 [11:25:15] RECOVERY - Puppet freshness on mw1102 is OK: puppet ran at Tue Jul 16 11:25:12 UTC 2013 [11:25:15] RECOVERY - Puppet freshness on mw1085 is OK: puppet ran at Tue Jul 16 11:25:12 UTC 2013 [11:25:16] RECOVERY - Puppet freshness on virt0 is OK: puppet ran at Tue Jul 16 11:25:12 UTC 2013 [11:25:44] RECOVERY - Puppet freshness on sq56 is OK: puppet ran at Tue Jul 16 11:25:42 UTC 2013 [11:25:44] RECOVERY - Puppet freshness on ersch is OK: puppet ran at Tue Jul 16 11:25:42 UTC 2013 [11:25:44] RECOVERY - Puppet freshness on amssq60 is OK: puppet ran at Tue Jul 16 11:25:42 UTC 2013 [11:25:44] RECOVERY - Puppet freshness on amssq36 is OK: puppet ran at Tue Jul 16 11:25:42 UTC 2013 [11:25:45] RECOVERY - Puppet freshness on amssq41 is OK: puppet ran at Tue Jul 16 11:25:42 UTC 2013 [11:25:45] RECOVERY - Puppet freshness on wtp1014 is OK: puppet ran at Tue Jul 16 11:25:42 UTC 2013 [11:25:54] RECOVERY - Puppet freshness on ms-be1005 is OK: puppet ran at Tue Jul 16 11:25:48 UTC 2013 [11:25:54] RECOVERY - Puppet freshness on db68 is OK: puppet ran at Tue Jul 16 11:25:48 UTC 2013 [11:25:54] RECOVERY - Puppet freshness on db1028 is OK: puppet ran at Tue Jul 16 11:25:48 UTC 2013 [11:25:54] RECOVERY - Puppet freshness on db38 is OK: puppet ran at Tue Jul 16 11:25:48 UTC 2013 [11:26:04] RECOVERY - Puppet freshness on cp1057 is OK: puppet ran at Tue Jul 16 11:25:53 UTC 2013 [11:26:04] RECOVERY - Puppet freshness on mw120 is OK: puppet ran at Tue Jul 16 11:25:53 UTC 2013 [11:26:04] RECOVERY - Puppet freshness on analytics1024 is OK: puppet ran at Tue Jul 16 11:25:54 UTC 2013 [11:26:04] RECOVERY - Puppet freshness on srv252 is OK: puppet ran at Tue Jul 16 11:25:54 UTC 2013 [11:26:04] RECOVERY - Puppet freshness on gallium is OK: puppet ran at Tue Jul 16 11:25:59 UTC 2013 [11:26:04] RECOVERY - Puppet freshness on mw96 is OK: puppet ran at Tue Jul 16 11:25:59 UTC 2013 [11:26:04] RECOVERY - Puppet freshness on search1013 is OK: puppet ran at Tue Jul 16 11:25:59 UTC 2013 [11:26:05] RECOVERY - Puppet freshness on lvs3 is OK: puppet ran at Tue Jul 16 11:25:59 UTC 2013 [11:26:05] RECOVERY - Puppet freshness on srv247 is OK: puppet ran at Tue Jul 16 11:25:59 UTC 2013 [11:26:06] RECOVERY - Puppet freshness on srv239 is OK: puppet ran at Tue Jul 16 11:25:59 UTC 2013 [11:26:15] RECOVERY - Puppet freshness on mw1172 is OK: puppet ran at Tue Jul 16 11:26:04 UTC 2013 [11:26:15] RECOVERY - Puppet freshness on mw1191 is OK: puppet ran at Tue Jul 16 11:26:04 UTC 2013 [11:26:15] RECOVERY - Puppet freshness on mw1184 is OK: puppet ran at Tue Jul 16 11:26:04 UTC 2013 [11:26:15] RECOVERY - Puppet freshness on mw1177 is OK: puppet ran at Tue Jul 16 11:26:04 UTC 2013 [11:26:15] RECOVERY - Puppet freshness on mw1039 is OK: puppet ran at Tue Jul 16 11:26:04 UTC 2013 [11:26:15] RECOVERY - Puppet freshness on search1009 is OK: puppet ran at Tue Jul 16 11:26:04 UTC 2013 [11:26:15] RECOVERY - Puppet freshness on tmh1002 is OK: puppet ran at Tue Jul 16 11:26:09 UTC 2013 [11:26:16] RECOVERY - Puppet freshness on analytics1006 is OK: puppet ran at Tue Jul 16 11:26:09 UTC 2013 [11:26:16] RECOVERY - Puppet freshness on ssl3001 is OK: puppet ran at Tue Jul 16 11:26:09 UTC 2013 [11:26:44] RECOVERY - Puppet freshness on mw67 is OK: puppet ran at Tue Jul 16 11:26:39 UTC 2013 [11:26:54] RECOVERY - Puppet freshness on lvs4 is OK: puppet ran at Tue Jul 16 11:26:44 UTC 2013 [11:26:54] RECOVERY - Puppet freshness on ms-be7 is OK: puppet ran at Tue Jul 16 11:26:44 UTC 2013 [11:26:54] RECOVERY - Puppet freshness on sq57 is OK: puppet ran at Tue Jul 16 11:26:44 UTC 2013 [11:26:54] RECOVERY - Puppet freshness on lvs1001 is OK: puppet ran at Tue Jul 16 11:26:44 UTC 2013 [11:26:54] RECOVERY - Puppet freshness on wtp1005 is OK: puppet ran at Tue Jul 16 11:26:44 UTC 2013 [11:26:54] RECOVERY - Puppet freshness on cp1042 is OK: puppet ran at Tue Jul 16 11:26:44 UTC 2013 [11:26:54] RECOVERY - Puppet freshness on es7 is OK: puppet ran at Tue Jul 16 11:26:44 UTC 2013 [11:26:55] RECOVERY - Puppet freshness on hooper is OK: puppet ran at Tue Jul 16 11:26:44 UTC 2013 [11:26:55] RECOVERY - Puppet freshness on db1003 is OK: puppet ran at Tue Jul 16 11:26:44 UTC 2013 [11:26:56] PROBLEM - Puppet freshness on cp1042 is CRITICAL: No successful Puppet run in the last 10 hours [11:26:56] RECOVERY - Puppet freshness on db33 is OK: puppet ran at Tue Jul 16 11:26:49 UTC 2013 [11:26:57] RECOVERY - Puppet freshness on cp1042 is OK: puppet ran at Tue Jul 16 11:26:49 UTC 2013 [11:26:57] RECOVERY - Puppet freshness on wtp1012 is OK: puppet ran at Tue Jul 16 11:26:49 UTC 2013 [11:26:58] RECOVERY - Puppet freshness on db1041 is OK: puppet ran at Tue Jul 16 11:26:49 UTC 2013 [11:26:58] RECOVERY - Puppet freshness on amssq58 is OK: puppet ran at Tue Jul 16 11:26:50 UTC 2013 [11:26:59] RECOVERY - Puppet freshness on db37 is OK: puppet ran at Tue Jul 16 11:26:50 UTC 2013 [11:26:59] RECOVERY - Puppet freshness on sq61 is OK: puppet ran at Tue Jul 16 11:26:50 UTC 2013 [11:27:00] RECOVERY - Puppet freshness on mw84 is OK: puppet ran at Tue Jul 16 11:26:50 UTC 2013 [11:27:04] RECOVERY - Puppet freshness on cp1019 is OK: puppet ran at Tue Jul 16 11:26:55 UTC 2013 [11:27:04] RECOVERY - Puppet freshness on mw71 is OK: puppet ran at Tue Jul 16 11:26:55 UTC 2013 [11:27:04] RECOVERY - Puppet freshness on mw1076 is OK: puppet ran at Tue Jul 16 11:27:00 UTC 2013 [11:27:04] RECOVERY - Puppet freshness on mw1195 is OK: puppet ran at Tue Jul 16 11:27:00 UTC 2013 [11:27:04] RECOVERY - Puppet freshness on mw1035 is OK: puppet ran at Tue Jul 16 11:27:00 UTC 2013 [11:27:04] RECOVERY - Puppet freshness on ms-fe4 is OK: puppet ran at Tue Jul 16 11:27:00 UTC 2013 [11:27:04] RECOVERY - Puppet freshness on mw80 is OK: puppet ran at Tue Jul 16 11:27:00 UTC 2013 [11:27:14] RECOVERY - Puppet freshness on es1007 is OK: puppet ran at Tue Jul 16 11:27:05 UTC 2013 [11:27:14] RECOVERY - Puppet freshness on mw1149 is OK: puppet ran at Tue Jul 16 11:27:05 UTC 2013 [11:27:14] RECOVERY - Puppet freshness on srv254 is OK: puppet ran at Tue Jul 16 11:27:05 UTC 2013 [11:27:14] RECOVERY - Puppet freshness on db1043 is OK: puppet ran at Tue Jul 16 11:27:05 UTC 2013 [11:27:14] RECOVERY - Puppet freshness on mw69 is OK: puppet ran at Tue Jul 16 11:27:05 UTC 2013 [11:27:24] RECOVERY - Puppet freshness on mw1036 is OK: puppet ran at Tue Jul 16 11:27:15 UTC 2013 [11:27:34] RECOVERY - Puppet freshness on silver is OK: puppet ran at Tue Jul 16 11:27:25 UTC 2013 [11:27:34] RECOVERY - Puppet freshness on analytics1016 is OK: puppet ran at Tue Jul 16 11:27:30 UTC 2013 [11:27:44] RECOVERY - Puppet freshness on cp1044 is OK: puppet ran at Tue Jul 16 11:27:40 UTC 2013 [11:27:54] PROBLEM - Puppet freshness on cp1042 is CRITICAL: No successful Puppet run in the last 10 hours [11:28:24] PROBLEM - Puppet freshness on cp1044 is CRITICAL: No successful Puppet run in the last 10 hours [11:28:54] RECOVERY - Puppet freshness on cp1041 is OK: puppet ran at Tue Jul 16 11:28:52 UTC 2013 [11:28:54] RECOVERY - Puppet freshness on nfs2 is OK: puppet ran at Tue Jul 16 11:28:52 UTC 2013 [11:29:44] PROBLEM - Puppet freshness on cp1041 is CRITICAL: No successful Puppet run in the last 10 hours [11:29:44] RECOVERY - Puppet freshness on search26 is OK: puppet ran at Tue Jul 16 11:29:38 UTC 2013 [11:32:20] well let's see how that goes [11:33:14] RECOVERY - Puppet freshness on cp1043 is OK: puppet ran at Tue Jul 16 11:33:12 UTC 2013 [11:33:24] PROBLEM - Puppet freshness on cp1043 is CRITICAL: No successful Puppet run in the last 10 hours [11:33:44] RECOVERY - Puppet freshness on neon is OK: puppet ran at Tue Jul 16 11:33:39 UTC 2013 [11:52:24] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:53:14] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.128 second response time [11:56:14] RECOVERY - Puppet freshness on ssl1004 is OK: puppet ran at Tue Jul 16 11:56:08 UTC 2013 [11:58:24] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:59:04] RECOVERY - Puppet freshness on cp1044 is OK: puppet ran at Tue Jul 16 11:59:02 UTC 2013 [11:59:14] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.153 second response time [11:59:24] PROBLEM - Puppet freshness on cp1044 is CRITICAL: No successful Puppet run in the last 10 hours [11:59:54] RECOVERY - Puppet freshness on cp1042 is OK: puppet ran at Tue Jul 16 11:59:53 UTC 2013 [12:00:24] RECOVERY - Puppet freshness on cp1041 is OK: puppet ran at Tue Jul 16 12:00:19 UTC 2013 [12:00:44] PROBLEM - Puppet freshness on cp1041 is CRITICAL: No successful Puppet run in the last 10 hours [12:00:54] PROBLEM - Puppet freshness on cp1042 is CRITICAL: No successful Puppet run in the last 10 hours [12:01:30] weird [12:02:44] RECOVERY - Puppet freshness on cp1043 is OK: puppet ran at Tue Jul 16 12:02:43 UTC 2013 [12:03:24] PROBLEM - Puppet freshness on cp1043 is CRITICAL: No successful Puppet run in the last 10 hours [12:03:33] and it's just those [12:04:48] !log Created georecord m.wikimedia.org [12:04:59] Logged the message, Master [12:07:27] PROBLEM - Auth DNS on ns1.wikimedia.org is CRITICAL: CRITICAL - Plugin timed out while executing system call [12:08:47] RECOVERY - Auth DNS on ns1.wikimedia.org is OK: DNS OK: 0.033 seconds response time. www.wikipedia.org returns 208.80.154.225 [12:20:04] !log Sending dutch mobile traffic to esams [12:20:13] Logged the message, Master [12:22:27] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:23:17] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.135 second response time [12:25:07] RECOVERY - Puppet freshness on cp1042 is OK: puppet ran at Tue Jul 16 12:25:00 UTC 2013 [12:25:47] PROBLEM - Puppet freshness on cp1042 is CRITICAL: No successful Puppet run in the last 10 hours [12:28:07] RECOVERY - Puppet freshness on cp1044 is OK: puppet ran at Tue Jul 16 12:28:04 UTC 2013 [12:28:27] PROBLEM - Puppet freshness on cp1044 is CRITICAL: No successful Puppet run in the last 10 hours [12:28:57] RECOVERY - Puppet freshness on cp1041 is OK: puppet ran at Tue Jul 16 12:28:49 UTC 2013 [12:29:47] PROBLEM - Puppet freshness on cp1041 is CRITICAL: No successful Puppet run in the last 10 hours [12:31:37] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:32:17] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.126 second response time [12:32:48] RECOVERY - Puppet freshness on cp1043 is OK: puppet ran at Tue Jul 16 12:32:42 UTC 2013 [12:33:17] PROBLEM - Puppet freshness on cp1043 is CRITICAL: No successful Puppet run in the last 10 hours [12:43:27] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:44:17] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.132 second response time [12:52:27] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:53:17] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.125 second response time [12:55:07] RECOVERY - Puppet freshness on cp1042 is OK: puppet ran at Tue Jul 16 12:55:00 UTC 2013 [12:55:47] PROBLEM - Puppet freshness on cp1042 is CRITICAL: No successful Puppet run in the last 10 hours [12:56:51] !log Sending all European mobile traffic to esams [12:57:03] Logged the message, Master [12:57:47] RECOVERY - Puppet freshness on cp1044 is OK: puppet ran at Tue Jul 16 12:57:43 UTC 2013 [12:58:27] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:58:27] PROBLEM - Puppet freshness on cp1044 is CRITICAL: No successful Puppet run in the last 10 hours [12:59:17] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 3.561 second response time [12:59:37] RECOVERY - Puppet freshness on cp1041 is OK: puppet ran at Tue Jul 16 12:59:29 UTC 2013 [12:59:47] PROBLEM - Puppet freshness on cp1041 is CRITICAL: No successful Puppet run in the last 10 hours [13:01:08] New patchset: Mark Bergsma; "Add mobile caches esams group to gmetad" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/73957 [13:02:27] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:02:33] Change merged: Mark Bergsma; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/73957 [13:02:57] RECOVERY - Puppet freshness on cp1043 is OK: puppet ran at Tue Jul 16 13:02:53 UTC 2013 [13:03:17] PROBLEM - Puppet freshness on cp1043 is CRITICAL: No successful Puppet run in the last 10 hours [13:03:17] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.128 second response time [13:06:27] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:07:00] New review: Nemo bis; "(1 comment)" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/69982 [13:07:18] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.169 second response time [13:07:47] New patchset: Mark Bergsma; "Add LVS monitoring for mobile in esams" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/73958 [13:09:19] New patchset: Mark Bergsma; "Add LVS monitoring for mobile in esams" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/73958 [13:11:34] Change merged: Mark Bergsma; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/73958 [13:15:06] apergos: ↑ [13:15:21] apergos: stafford is doing the usual [13:15:30] and cp104x [13:16:21] stafford is overloaded [13:16:30] look at its cpu [13:17:03] it's been that way for some time [13:17:56] apergos: There was an error collecting ganglia data (127.0.0.1:8654): fsockopen error: Connection refused [13:18:23] just retry [13:19:33] ok, looks a bit choked [13:19:55] apergos: anyway, cp104{1..4} is doing the puppet freshness dance [13:19:55] yes, those four have something else going on [13:20:42] hmm, cp104{1..4} isn't shown in ganglia, only cp104{5..9} [13:21:30] cp1041 is a Wikimedia DECOMMISSIONED server (base::decommissioned). [13:22:11] same for the other 3 [13:22:14] ok [13:22:27] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:22:40] is decommissioned related to hardware or just virtually? [13:23:17] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.143 second response time [13:23:23] https://wikitech.wikimedia.org/wiki/Server_Lifecycle [13:23:26] here ya go [13:23:33] ty [13:23:59] (sometimes I've difficult to trust wikitech pages, becuase often they are outdated) [13:24:13] if we want to reuse it with a new hostname we go through th steps in 'reclaim or decommision' [13:24:13] yeah rob has been working on these [13:24:46] is there any list of all hardware you have? [13:24:58] racktables maybe [13:25:17] RECOVERY - Puppet freshness on cp1042 is OK: puppet ran at Tue Jul 16 13:25:08 UTC 2013 [13:25:17] searching for hardware related queries only ends up at the 2007 rack image :-P [13:25:46] helpful isn't it :-P [13:25:47] PROBLEM - Puppet freshness on cp1042 is CRITICAL: No successful Puppet run in the last 10 hours [13:25:52] hehe [13:26:09] and it all boild down to brion and some waffles [13:27:27] apergos: if they are "decommishioned", are they stilml running then? [13:28:17] RECOVERY - Puppet freshness on cp1044 is OK: puppet ran at Tue Jul 16 13:28:07 UTC 2013 [13:28:26] apergos: afaik, "Puppet freshness on cp1044 is OK" doesn't look it's offline [13:28:27] PROBLEM - Puppet freshness on cp1044 is CRITICAL: No successful Puppet run in the last 10 hours [13:28:53] well in theory they should get pulled from icinga [13:28:57] RECOVERY - Puppet freshness on cp1041 is OK: puppet ran at Tue Jul 16 13:28:53 UTC 2013 [13:29:46] they might be powered on (til someone does something with them) but shouldn't get checked etc [13:29:47] PROBLEM - Puppet freshness on cp1041 is CRITICAL: No successful Puppet run in the last 10 hours [13:30:40] apergos: last entry in server admin logs are from May 20: 17:48 mutante: upgrading/rebooting mobile varnish cache cp1041-cp1044 [13:32:01] apergos: where is mobile varnish cache now? [13:32:19] so the parson that would know is mark [13:32:32] maybe these are meant to go back in at some point, I don't know that either [13:32:41] but right now they are marked as decomissioned [13:32:45] no, they'll get wiped and repurposed soon [13:32:48] RECOVERY - Puppet freshness on cp1043 is OK: puppet ran at Tue Jul 16 13:32:42 UTC 2013 [13:32:56] cool [13:33:17] PROBLEM - Puppet freshness on cp1043 is CRITICAL: No successful Puppet run in the last 10 hours [13:35:47] RECOVERY - Host mw1163 is UP: PING OK - Packet loss = 0%, RTA = 0.47 ms [13:37:37] RECOVERY - Puppet freshness on mw1163 is OK: puppet ran at Tue Jul 16 13:37:36 UTC 2013 [13:38:48] PROBLEM - Apache HTTP on mw1163 is CRITICAL: Connection refused [13:39:47] RECOVERY - Apache HTTP on mw1163 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 747 bytes in 0.110 second response time [13:43:57] PROBLEM - Host mw1163 is DOWN: PING CRITICAL - Packet loss = 100% [13:49:07] RECOVERY - Host mw1163 is UP: PING OK - Packet loss = 0%, RTA = 0.22 ms [13:51:17] PROBLEM - SSH on mw1163 is CRITICAL: Connection refused [13:51:27] PROBLEM - Disk space on mw1163 is CRITICAL: Connection refused by host [13:51:37] PROBLEM - twemproxy process on mw1163 is CRITICAL: Connection refused by host [13:51:37] PROBLEM - RAID on mw1163 is CRITICAL: Connection refused by host [13:51:54] PROBLEM - Apache HTTP on mw1163 is CRITICAL: Connection refused [13:51:57] PROBLEM - DPKG on mw1163 is CRITICAL: Connection refused by host [13:52:13] that's me updating bios ^ [13:52:42] good, cause I was wondering why I couldn't get on [13:52:47] PROBLEM - Host carbon is DOWN: PING CRITICAL - Packet loss = 100% [13:53:34] carbon is me to...doing a reinstall [13:54:17] RECOVERY - Host carbon is UP: PING OK - Packet loss = 0%, RTA = 4.88 ms [13:55:07] RECOVERY - Puppet freshness on cp1042 is OK: puppet ran at Tue Jul 16 13:55:00 UTC 2013 [13:55:47] PROBLEM - Host mw1163 is DOWN: PING CRITICAL - Packet loss = 100% [13:55:47] PROBLEM - Puppet freshness on cp1042 is CRITICAL: No successful Puppet run in the last 10 hours [13:57:57] RECOVERY - Puppet freshness on cp1044 is OK: puppet ran at Tue Jul 16 13:57:52 UTC 2013 [13:58:27] PROBLEM - Puppet freshness on cp1044 is CRITICAL: No successful Puppet run in the last 10 hours [13:58:47] RECOVERY - Puppet freshness on cp1041 is OK: puppet ran at Tue Jul 16 13:58:43 UTC 2013 [13:59:47] PROBLEM - Puppet freshness on cp1041 is CRITICAL: No successful Puppet run in the last 10 hours [13:59:57] RECOVERY - DPKG on mw1163 is OK: All packages OK [14:00:07] RECOVERY - Host mw1163 is UP: PING OK - Packet loss = 0%, RTA = 0.29 ms [14:00:17] RECOVERY - SSH on mw1163 is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1.1 (protocol 2.0) [14:00:27] RECOVERY - Disk space on mw1163 is OK: DISK OK [14:00:37] RECOVERY - twemproxy process on mw1163 is OK: PROCS OK: 1 process with UID = 65534 (nobody), command name nutcracker [14:00:37] RECOVERY - RAID on mw1163 is OK: OK: no RAID installed [14:02:49] RECOVERY - Puppet freshness on cp1043 is OK: puppet ran at Tue Jul 16 14:02:41 UTC 2013 [14:03:17] PROBLEM - Puppet freshness on cp1043 is CRITICAL: No successful Puppet run in the last 10 hours [14:06:47] PROBLEM - Host carbon is DOWN: CRITICAL - Host Unreachable (208.80.154.10) [14:07:55] icinga-wm should notice if the host is rebooting/reinstalling [14:08:29] AzaToth: you can set a maintenance window in icinga that disable notifications [14:12:55] hashar: which apergos should have done ツ [14:14:00] I should? Im not rebooting it :-P [14:14:15] was referring to the freshness ツ [14:15:17] RECOVERY - Host carbon is UP: PING OK - Packet loss = 0%, RTA = 0.42 ms [14:17:51] I think hashar was not :-P [14:18:27] PROBLEM - Host carbon is DOWN: CRITICAL - Host Unreachable (208.80.154.10) [14:18:28] New patchset: Hashar; "mobile $wgLoadScript now uses $wmfHostnames['bits']" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/71779 [14:19:01] New review: Hashar; "rebased" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/71779 [14:19:07] RECOVERY - Host carbon is UP: PING OK - Packet loss = 0%, RTA = 0.25 ms [14:22:27] PROBLEM - Host carbon is DOWN: CRITICAL - Host Unreachable (208.80.154.10) [14:22:27] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:23:17] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.140 second response time [14:24:57] RECOVERY - Puppet freshness on cp1042 is OK: puppet ran at Tue Jul 16 14:24:53 UTC 2013 [14:25:07] PROBLEM - Puppet freshness on manutius is CRITICAL: No successful Puppet run in the last 10 hours [14:25:47] PROBLEM - Puppet freshness on cp1042 is CRITICAL: No successful Puppet run in the last 10 hours [14:27:07] New review: MaxSem; "The code in question will die in a couple of days anyway." [operations/mediawiki-config] (master) C: -2; - https://gerrit.wikimedia.org/r/71779 [14:27:16] hashar, ^^ :) [14:27:30] MaxSem: great :-] [14:27:57] PROBLEM - SSH on pdf2 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:28:27] RECOVERY - Puppet freshness on cp1044 is OK: puppet ran at Tue Jul 16 14:28:17 UTC 2013 [14:28:27] PROBLEM - Puppet freshness on cp1044 is CRITICAL: No successful Puppet run in the last 10 hours [14:28:57] RECOVERY - Puppet freshness on cp1041 is OK: puppet ran at Tue Jul 16 14:28:52 UTC 2013 [14:29:00] New review: Hashar; "Great! Thank you Max. I am keeping the change around to remember myself to double check the bits.wm..." [operations/mediawiki-config] (master) C: -2; - https://gerrit.wikimedia.org/r/71779 [14:29:19] MaxSem: I am keeping the change open with a -2 by me to remember about it [14:29:26] MaxSem: feel free to remove yourself [14:29:47] PROBLEM - Puppet freshness on cp1041 is CRITICAL: No successful Puppet run in the last 10 hours [14:30:47] RECOVERY - SSH on pdf2 is OK: SSH OK - OpenSSH_4.7p1 Debian-8ubuntu3 (protocol 2.0) [14:30:51] cool [14:31:22] as the matter of fact, it is possible to remove it since the last week, but I'm being overcautious [14:32:48] RECOVERY - Puppet freshness on cp1043 is OK: puppet ran at Tue Jul 16 14:32:41 UTC 2013 [14:33:17] PROBLEM - Puppet freshness on cp1043 is CRITICAL: No successful Puppet run in the last 10 hours [14:35:38] mark: do you have a few minutes...have a problem trying to get carbon to install [14:43:27] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:44:17] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.129 second response time [14:52:27] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:52:46] !log zuul / jenkins -merge jobs are no more triggered by Zuul since it detects whether a patch apply cleanly on the tip of the branch. That saves up a bunch of overhead {{bug|51395}} and {{gerrit|73947}} [14:52:57] Logged the message, Master [14:53:17] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.132 second response time [14:54:57] RECOVERY - Puppet freshness on cp1042 is OK: puppet ran at Tue Jul 16 14:54:49 UTC 2013 [14:55:47] PROBLEM - Puppet freshness on cp1042 is CRITICAL: No successful Puppet run in the last 10 hours [14:57:57] RECOVERY - Puppet freshness on cp1044 is OK: puppet ran at Tue Jul 16 14:57:46 UTC 2013 [14:58:27] PROBLEM - Puppet freshness on cp1044 is CRITICAL: No successful Puppet run in the last 10 hours [14:58:57] RECOVERY - Puppet freshness on cp1041 is OK: puppet ran at Tue Jul 16 14:58:53 UTC 2013 [14:59:47] PROBLEM - Puppet freshness on cp1041 is CRITICAL: No successful Puppet run in the last 10 hours [15:02:47] RECOVERY - Puppet freshness on cp1043 is OK: puppet ran at Tue Jul 16 15:02:40 UTC 2013 [15:03:17] PROBLEM - Puppet freshness on cp1043 is CRITICAL: No successful Puppet run in the last 10 hours [15:22:26] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:23:16] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.132 second response time [15:24:56] RECOVERY - Puppet freshness on cp1042 is OK: puppet ran at Tue Jul 16 15:24:52 UTC 2013 [15:25:46] PROBLEM - Puppet freshness on cp1042 is CRITICAL: No successful Puppet run in the last 10 hours [15:27:56] RECOVERY - Puppet freshness on cp1044 is OK: puppet ran at Tue Jul 16 15:27:50 UTC 2013 [15:28:26] PROBLEM - Puppet freshness on cp1044 is CRITICAL: No successful Puppet run in the last 10 hours [15:29:06] RECOVERY - Puppet freshness on cp1041 is OK: puppet ran at Tue Jul 16 15:28:56 UTC 2013 [15:29:26] PROBLEM - Puppet freshness on cp1041 is CRITICAL: No successful Puppet run in the last 10 hours [15:33:00] !log reedy synchronized php-1.22wmf10/extensions/Wikibase [15:33:09] Logged the message, Master [15:33:12] thanks Reedy :) [15:33:26] RECOVERY - Puppet freshness on cp1043 is OK: puppet ran at Tue Jul 16 15:33:17 UTC 2013 [15:34:16] PROBLEM - Puppet freshness on cp1043 is CRITICAL: No successful Puppet run in the last 10 hours [15:46:24] andrewbogott: there is Jenkins plugin that rely on JClouds which apparently supports openstack api :-) [15:46:24] http://jclouds.incubator.apache.org/documentation/reference/supported-providers/ [15:46:40] at the bottom OpenStack openstack-nova is listed :-] [15:46:53] hm… ok [15:48:48] but it might be easier to just use a VagrantFile [15:50:26] though that is apparently used to create Jenkins slaves [15:54:56] RECOVERY - Puppet freshness on cp1042 is OK: puppet ran at Tue Jul 16 15:54:54 UTC 2013 [15:55:47] PROBLEM - Puppet freshness on cp1042 is CRITICAL: No successful Puppet run in the last 10 hours [15:58:36] RECOVERY - Puppet freshness on cp1044 is OK: puppet ran at Tue Jul 16 15:58:35 UTC 2013 [15:59:06] RECOVERY - Puppet freshness on cp1041 is OK: puppet ran at Tue Jul 16 15:59:00 UTC 2013 [15:59:26] PROBLEM - Puppet freshness on cp1041 is CRITICAL: No successful Puppet run in the last 10 hours [15:59:26] PROBLEM - Puppet freshness on cp1044 is CRITICAL: No successful Puppet run in the last 10 hours [16:02:46] RECOVERY - Puppet freshness on cp1043 is OK: puppet ran at Tue Jul 16 16:02:39 UTC 2013 [16:03:16] PROBLEM - Puppet freshness on cp1043 is CRITICAL: No successful Puppet run in the last 10 hours [16:22:26] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:23:17] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.135 second response time [16:27:16] RECOVERY - Puppet freshness on cp1042 is OK: puppet ran at Tue Jul 16 16:27:11 UTC 2013 [16:27:46] PROBLEM - Puppet freshness on cp1042 is CRITICAL: No successful Puppet run in the last 10 hours [16:27:56] RECOVERY - Puppet freshness on cp1044 is OK: puppet ran at Tue Jul 16 16:27:51 UTC 2013 [16:28:27] PROBLEM - Puppet freshness on cp1044 is CRITICAL: No successful Puppet run in the last 10 hours [16:28:56] RECOVERY - Puppet freshness on cp1041 is OK: puppet ran at Tue Jul 16 16:28:47 UTC 2013 [16:29:26] PROBLEM - Puppet freshness on cp1041 is CRITICAL: No successful Puppet run in the last 10 hours [16:29:44] !log swapping sdb1 on manganese (software raid1) rt5442 [16:30:25] Logged the message, Master [16:30:56] !log that's sdb2 on manganese [16:33:12] Logged the message, Master [16:33:16] RECOVERY - Puppet freshness on cp1043 is OK: puppet ran at Tue Jul 16 16:33:14 UTC 2013 [16:34:16] PROBLEM - Puppet freshness on cp1043 is CRITICAL: No successful Puppet run in the last 10 hours [16:39:36] RECOVERY - Host carbon is UP: PING OK - Packet loss = 0%, RTA = 0.54 ms [16:52:26] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:53:22] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.129 second response time [16:54:56] RECOVERY - Puppet freshness on cp1042 is OK: puppet ran at Tue Jul 16 16:54:52 UTC 2013 [16:55:46] PROBLEM - Puppet freshness on cp1042 is CRITICAL: No successful Puppet run in the last 10 hours [16:58:06] RECOVERY - Puppet freshness on cp1044 is OK: puppet ran at Tue Jul 16 16:58:00 UTC 2013 [16:58:26] PROBLEM - Puppet freshness on cp1044 is CRITICAL: No successful Puppet run in the last 10 hours [17:00:36] RECOVERY - Puppet freshness on cp1041 is OK: puppet ran at Tue Jul 16 17:00:34 UTC 2013 [17:01:26] PROBLEM - Puppet freshness on cp1041 is CRITICAL: No successful Puppet run in the last 10 hours [17:01:26] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:02:17] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.128 second response time [17:02:46] RECOVERY - Puppet freshness on cp1043 is OK: puppet ran at Tue Jul 16 17:02:37 UTC 2013 [17:03:16] PROBLEM - Puppet freshness on cp1043 is CRITICAL: No successful Puppet run in the last 10 hours [17:06:29] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:07:18] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.129 second response time [17:09:02] <^demon> !log gitblit restarted to grab a bugfix [17:09:11] Logged the message, Master [17:22:28] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:23:18] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.128 second response time [17:24:58] RECOVERY - Puppet freshness on cp1042 is OK: puppet ran at Tue Jul 16 17:24:52 UTC 2013 [17:25:48] PROBLEM - Puppet freshness on cp1042 is CRITICAL: No successful Puppet run in the last 10 hours [17:27:48] RECOVERY - Puppet freshness on cp1044 is OK: puppet ran at Tue Jul 16 17:27:41 UTC 2013 [17:28:28] PROBLEM - Puppet freshness on cp1044 is CRITICAL: No successful Puppet run in the last 10 hours [17:28:58] RECOVERY - Puppet freshness on cp1041 is OK: puppet ran at Tue Jul 16 17:28:53 UTC 2013 [17:29:28] PROBLEM - Puppet freshness on cp1041 is CRITICAL: No successful Puppet run in the last 10 hours [17:32:48] RECOVERY - Puppet freshness on cp1043 is OK: puppet ran at Tue Jul 16 17:32:46 UTC 2013 [17:33:18] PROBLEM - Puppet freshness on cp1043 is CRITICAL: No successful Puppet run in the last 10 hours [17:45:47] Ryan_Lane, when you have a moment… can I get a review of https://gerrit.wikimedia.org/r/73521 and can you sign off on the overall concept of https://gerrit.wikimedia.org/r/73888? [17:46:49] Change merged: Ryan Lane; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/73521 [17:47:56] thanks! [17:47:58] <^demon> Ooh, me next! https://gerrit.wikimedia.org/r/#/c/70762/ [17:50:21] New review: Ryan Lane; "Concept looks good to me." [operations/puppet] (production) C: 1; - https://gerrit.wikimedia.org/r/73888 [17:52:28] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:53:18] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.128 second response time [17:53:27] !log deployed LdapAuthentication fdb01877 to virt0 and purged memcache to remove bad memcache entries for users [17:53:37] Logged the message, Master [17:54:58] RECOVERY - Puppet freshness on cp1042 is OK: puppet ran at Tue Jul 16 17:54:49 UTC 2013 [17:55:48] PROBLEM - Puppet freshness on cp1042 is CRITICAL: No successful Puppet run in the last 10 hours [17:57:38] RECOVERY - Puppet freshness on cp1044 is OK: puppet ran at Tue Jul 16 17:57:35 UTC 2013 [17:58:28] PROBLEM - Puppet freshness on cp1044 is CRITICAL: No successful Puppet run in the last 10 hours [17:58:48] RECOVERY - Puppet freshness on cp1041 is OK: puppet ran at Tue Jul 16 17:58:41 UTC 2013 [17:58:53] ori-l: thanks for the merge! [17:59:28] PROBLEM - Puppet freshness on cp1041 is CRITICAL: No successful Puppet run in the last 10 hours [18:01:28] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:02:19] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.131 second response time [18:02:38] RECOVERY - Puppet freshness on cp1043 is OK: puppet ran at Tue Jul 16 18:02:37 UTC 2013 [18:03:18] PROBLEM - Puppet freshness on cp1043 is CRITICAL: No successful Puppet run in the last 10 hours [18:03:30] !log rechecking tag_summary enwiki re bug 51254 comment 11 [18:03:40] Logged the message, Master [18:13:32] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:14:22] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.126 second response time [18:15:00] New review: Ori.livneh; "Don't make the resource title the sysctl key name; Puppet resource titles and sysctl keys are subjec..." [operations/puppet] (production) C: -1; - https://gerrit.wikimedia.org/r/73888 [18:22:34] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:23:22] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.126 second response time [18:25:02] RECOVERY - Puppet freshness on cp1042 is OK: puppet ran at Tue Jul 16 18:24:52 UTC 2013 [18:25:56] PROBLEM - Puppet freshness on cp1042 is CRITICAL: No successful Puppet run in the last 10 hours [18:28:42] RECOVERY - Puppet freshness on cp1044 is OK: puppet ran at Tue Jul 16 18:28:41 UTC 2013 [18:29:02] RECOVERY - Puppet freshness on cp1041 is OK: puppet ran at Tue Jul 16 18:28:57 UTC 2013 [18:29:24] PROBLEM - Puppet freshness on cp1041 is CRITICAL: No successful Puppet run in the last 10 hours [18:29:25] PROBLEM - Puppet freshness on cp1044 is CRITICAL: No successful Puppet run in the last 10 hours [18:34:52] RECOVERY - Puppet freshness on cp1043 is OK: puppet ran at Tue Jul 16 18:34:44 UTC 2013 [18:35:12] PROBLEM - Puppet freshness on cp1043 is CRITICAL: No successful Puppet run in the last 10 hours [18:39:19] New patchset: Pyoungmeister; "enabling virt1005 and virt1007 as ganglia aggregators" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/73992 [18:43:28] PROBLEM - SSH on pdf2 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:43:50] Change merged: Pyoungmeister; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/73992 [18:45:12] RECOVERY - SSH on pdf2 is OK: SSH OK - OpenSSH_4.7p1 Debian-8ubuntu3 (protocol 2.0) [18:45:33] andrewbogott: mark I'm merging your stuff on sockpuppet [18:46:09] 'k [18:51:05] New review: Andrew Bogott; "Incoming patch addresses all comments except for the 'use a hash' one -- the 'content' pathway large..." [operations/puppet] (production) - https://gerrit.wikimedia.org/r/73888 [18:51:42] PROBLEM - check google safe browsing for wikisource.org on google is CRITICAL: Connection timed out [18:52:32] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:52:32] RECOVERY - check google safe browsing for wikisource.org on google is OK: HTTP OK: HTTP/1.1 200 OK - 3818 bytes in 0.175 second response time [18:53:22] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.130 second response time [18:53:52] PROBLEM - Puppet freshness on analytics1019 is CRITICAL: No successful Puppet run in the last 10 hours [18:54:52] RECOVERY - Puppet freshness on cp1042 is OK: puppet ran at Tue Jul 16 18:54:48 UTC 2013 [18:55:52] PROBLEM - Puppet freshness on cp1042 is CRITICAL: No successful Puppet run in the last 10 hours [18:57:32] RECOVERY - Puppet freshness on cp1044 is OK: puppet ran at Tue Jul 16 18:57:26 UTC 2013 [18:57:52] PROBLEM - Puppet freshness on analytics1018 is CRITICAL: No successful Puppet run in the last 10 hours [18:58:22] PROBLEM - Puppet freshness on cp1044 is CRITICAL: No successful Puppet run in the last 10 hours [18:58:52] PROBLEM - Puppet freshness on analytics1020 is CRITICAL: No successful Puppet run in the last 10 hours [19:00:10] New patchset: Andrew Bogott; "Add sysctlfile module and one use case." [operations/puppet] (production) - https://gerrit.wikimedia.org/r/73888 [19:02:12] RECOVERY - Puppet freshness on cp1041 is OK: puppet ran at Tue Jul 16 19:02:08 UTC 2013 [19:02:22] PROBLEM - Puppet freshness on cp1041 is CRITICAL: No successful Puppet run in the last 10 hours [19:02:52] RECOVERY - Puppet freshness on cp1043 is OK: puppet ran at Tue Jul 16 19:02:49 UTC 2013 [19:03:12] PROBLEM - Puppet freshness on cp1043 is CRITICAL: No successful Puppet run in the last 10 hours [19:22:48] @notify orenwold [19:22:48] I'll let you know when I see orenwold around here [19:22:49] @notify orenwolf [19:22:50] I'll let you know when I see orenwolf around here [19:25:02] RECOVERY - Puppet freshness on cp1042 is OK: puppet ran at Tue Jul 16 19:24:54 UTC 2013 [19:25:52] PROBLEM - Puppet freshness on cp1042 is CRITICAL: No successful Puppet run in the last 10 hours [19:27:52] RECOVERY - Puppet freshness on cp1044 is OK: puppet ran at Tue Jul 16 19:27:43 UTC 2013 [19:28:13] New patchset: Ori.livneh; "Document 'eventlogging' role" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/73997 [19:28:22] PROBLEM - Puppet freshness on cp1044 is CRITICAL: No successful Puppet run in the last 10 hours [19:28:52] RECOVERY - Puppet freshness on cp1041 is OK: puppet ran at Tue Jul 16 19:28:51 UTC 2013 [19:29:22] PROBLEM - Puppet freshness on cp1041 is CRITICAL: No successful Puppet run in the last 10 hours [19:32:25] paravoid: as promised :) [19:32:52] RECOVERY - Puppet freshness on cp1043 is OK: puppet ran at Tue Jul 16 19:32:44 UTC 2013 [19:33:12] PROBLEM - Puppet freshness on cp1043 is CRITICAL: No successful Puppet run in the last 10 hours [19:34:52] PROBLEM - Puppet freshness on erzurumi is CRITICAL: No successful Puppet run in the last 10 hours [19:34:52] PROBLEM - Puppet freshness on lvs1004 is CRITICAL: No successful Puppet run in the last 10 hours [19:34:52] PROBLEM - Puppet freshness on lvs1005 is CRITICAL: No successful Puppet run in the last 10 hours [19:34:52] PROBLEM - Puppet freshness on lvs1006 is CRITICAL: No successful Puppet run in the last 10 hours [19:34:52] PROBLEM - Puppet freshness on virt1 is CRITICAL: No successful Puppet run in the last 10 hours [19:34:52] PROBLEM - Puppet freshness on virt3 is CRITICAL: No successful Puppet run in the last 10 hours [19:34:52] PROBLEM - Puppet freshness on virt4 is CRITICAL: No successful Puppet run in the last 10 hours [19:37:43] New review: Andrew Bogott; "This looks totally fine -- do you mind cleaning up the tab/space mix and going with just spaces?" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/73997 [19:48:16] New patchset: Ori.livneh; "Document 'eventlogging' role" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/73997 [19:49:06] New review: Ori.livneh; "PS2 fixes the whitespace issue." [operations/puppet] (production) - https://gerrit.wikimedia.org/r/73997 [19:51:46] Change merged: Andrew Bogott; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/73997 [19:51:51] thanks andrewbogott [19:52:02] np [19:53:56] Hm… how is our current puppet repo licensed? [19:54:13] I want to import a lib from puppet labs which is GPL… surely I'm not the first. [19:54:52] RECOVERY - Puppet freshness on cp1042 is OK: puppet ran at Tue Jul 16 19:54:51 UTC 2013 [19:54:58] * YuviPanda roots for WTFPL [19:55:11] Ryan_Lane, thoughts? ^^ [19:55:52] PROBLEM - Puppet freshness on cp1042 is CRITICAL: No successful Puppet run in the last 10 hours [19:58:02] RECOVERY - Puppet freshness on cp1044 is OK: puppet ran at Tue Jul 16 19:57:54 UTC 2013 [19:58:22] PROBLEM - Puppet freshness on cp1044 is CRITICAL: No successful Puppet run in the last 10 hours [19:58:52] RECOVERY - Puppet freshness on cp1041 is OK: puppet ran at Tue Jul 16 19:58:45 UTC 2013 [19:59:23] PROBLEM - Puppet freshness on cp1041 is CRITICAL: No successful Puppet run in the last 10 hours [20:01:32] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:02:23] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.155 second response time [20:02:52] RECOVERY - Puppet freshness on cp1043 is OK: puppet ran at Tue Jul 16 20:02:49 UTC 2013 [20:03:12] PROBLEM - Puppet freshness on cp1043 is CRITICAL: No successful Puppet run in the last 10 hours [20:06:24] Change merged: jenkins-bot; [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/73488 [20:10:37] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:11:27] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.133 second response time [20:12:16] New patchset: MaxSem; "Remove last vestiges of $wgMFLogEvents" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/74045 [20:12:43] Change merged: jenkins-bot; [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/74045 [20:16:08] Change merged: jenkins-bot; [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/73332 [20:16:19] Change merged: jenkins-bot; [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/72983 [20:22:37] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:23:27] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.138 second response time [20:24:14] hashar: I notice jshint is non-voting on jenkins; Is there a reason for that? [20:25:07] RECOVERY - Puppet freshness on cp1042 is OK: puppet ran at Tue Jul 16 20:24:58 UTC 2013 [20:25:47] PROBLEM - Puppet freshness on cp1042 is CRITICAL: No successful Puppet run in the last 10 hours [20:26:37] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:27:27] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.211 second response time [20:28:07] RECOVERY - Puppet freshness on cp1044 is OK: puppet ran at Tue Jul 16 20:28:05 UTC 2013 [20:28:27] PROBLEM - Puppet freshness on cp1044 is CRITICAL: No successful Puppet run in the last 10 hours [20:29:07] RECOVERY - Puppet freshness on cp1041 is OK: puppet ran at Tue Jul 16 20:29:00 UTC 2013 [20:29:27] PROBLEM - Puppet freshness on cp1041 is CRITICAL: No successful Puppet run in the last 10 hours [20:30:07] PROBLEM - check_mysql on db78 is CRITICAL: Access denied for user nagios@localhost (using password: NO) [20:30:08] PROBLEM - check_raid on db78 is CRITICAL: CRITICAL: Degraded [20:32:33] most projects don't pass jshint :P [20:32:37] I think? [20:32:57] RECOVERY - Puppet freshness on cp1043 is OK: puppet ran at Tue Jul 16 20:32:50 UTC 2013 [20:33:19] PROBLEM - Puppet freshness on cp1043 is CRITICAL: No successful Puppet run in the last 10 hours [20:34:47] RECOVERY - check_mysql on db78 is OK: Uptime: 541366 Threads: 1 Questions: 2582627 Slow queries: 3744 Opens: 11581 Flush tables: 2 Open tables: 64 Queries per second avg: 4.770 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 0 [20:34:48] PROBLEM - check_raid on db78 is CRITICAL: CRITICAL: Degraded [20:35:07] PROBLEM - check_raid on db78 is CRITICAL: CRITICAL: Degraded [20:39:02] scapping [20:41:42] AzaToth: jshint is non voting by default cause it fails on most repositories [20:41:53] AzaToth: so once the git repo has fixed up its jshint we manually enable it [20:42:01] AzaToth: ideally we would make jshint voting by default :] [20:48:04] !log maxsem Started syncing Wikimedia installation... : Weekly mobile deployment [20:48:14] Logged the message, Master [20:54:47] RECOVERY - Puppet freshness on cp1042 is OK: puppet ran at Tue Jul 16 20:54:42 UTC 2013 [20:54:47] PROBLEM - Puppet freshness on cp1042 is CRITICAL: No successful Puppet run in the last 10 hours [20:57:11] !log finished rechecking tag_summary enwiki re bug 51254 comment 11 [20:57:21] Logged the message, Master [20:57:37] RECOVERY - Puppet freshness on cp1044 is OK: puppet ran at Tue Jul 16 20:57:29 UTC 2013 [20:58:27] PROBLEM - Puppet freshness on cp1044 is CRITICAL: No successful Puppet run in the last 10 hours [20:58:47] RECOVERY - Puppet freshness on cp1041 is OK: puppet ran at Tue Jul 16 20:58:46 UTC 2013 [20:58:55] !log maxsem Finished syncing Wikimedia installation... : Weekly mobile deployment [20:59:05] Logged the message, Master [20:59:27] PROBLEM - Puppet freshness on cp1041 is CRITICAL: No successful Puppet run in the last 10 hours [21:00:40] <^demon> Ryan_Lane: pingity ping ping. [21:00:48] not here [21:00:51] :) [21:00:54] ^demon: what's up? [21:01:03] <^demon> :) https://gerrit.wikimedia.org/r/#/c/70762/ [21:01:32] Change merged: Ryan Lane; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/70762 [21:03:17] RECOVERY - Puppet freshness on cp1043 is OK: puppet ran at Tue Jul 16 21:03:11 UTC 2013 [21:04:17] PROBLEM - Puppet freshness on cp1043 is CRITICAL: No successful Puppet run in the last 10 hours [21:17:13] New patchset: MaxSem; "Remove device detection from bits" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/73342 [21:19:06] New review: MaxSem; "Can be deployed now:)" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/73342 [21:20:51] btw, only $162 until the git-annex crowd funding hits $10k. who wants to get joey on our deploy sprint for $300? ;) [21:22:31] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:22:50] <^demon> greg-g: Oh, I hadn't donated yet. Will do now. [21:23:10] I will too [21:23:21] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.133 second response time [21:23:23] :) [21:23:26] greg-g: I donated on 05/30/2012 originally :p [21:23:36] I've got some stickers upstairs [21:23:56] Reedy: ditto, well, at home :) [21:25:01] RECOVERY - Puppet freshness on cp1042 is OK: puppet ran at Tue Jul 16 21:24:54 UTC 2013 [21:25:51] PROBLEM - Puppet freshness on cp1042 is CRITICAL: No successful Puppet run in the last 10 hours [21:28:11] RECOVERY - Puppet freshness on cp1044 is OK: puppet ran at Tue Jul 16 21:28:07 UTC 2013 [21:28:21] PROBLEM - Puppet freshness on cp1044 is CRITICAL: No successful Puppet run in the last 10 hours [21:29:11] RECOVERY - Puppet freshness on cp1041 is OK: puppet ran at Tue Jul 16 21:29:08 UTC 2013 [21:29:21] PROBLEM - Puppet freshness on cp1041 is CRITICAL: No successful Puppet run in the last 10 hours [21:32:17] ok, i have a stupid question [21:32:29] like, really stupid [21:32:50] ori-l: yessir? [21:33:00] is connecting to localhost using the loopback adapter at all faster than using an actual local ethernet interface? [21:33:36] hm. [21:34:07] that's a good question, I honestly don't know [21:34:36] beware oris pretending to ask stupid questions... [21:34:45] like maybe the signal doesn't have to travel to the card and back or something, i don't know [21:34:58] indeed. he says it's a stupid question, but it actually requires me to go look ;) [21:35:18] though I guess maybe someone has already looked this up and knows off the top of their head [21:35:31] RECOVERY - Puppet freshness on cp1043 is OK: puppet ran at Tue Jul 16 21:35:30 UTC 2013 [21:35:41] PROBLEM - Puppet freshness on cp1043 is CRITICAL: No successful Puppet run in the last 10 hours [21:36:00] whee: http://serverfault.com/questions/234223/how-fast-is-127-0-0-1 [21:37:46] so, it's faster. crazy. [21:39:07] 127.0.0.1 ? [21:39:13] yes [21:39:15] that should be pure software / out of nic [21:40:03] yeah, i wasn't sure if the OS is smart enough to do the same for packets that are sent to an IP that is currently bound to a local ethernet adapter [21:41:24] dig in https://github.com/torvalds/linux/blob/master/drivers/net/loopback.c :D [21:41:35] Larry McVoy : Tiny tweak to double performance [21:41:36] ahah [21:43:31] ori-l: also faster than referencing 'localhost' which needs a name -> Ip lookup [21:45:37] New patchset: Ottomata; "Puppetizing oozie client and server" [operations/puppet/cdh4] (master) - https://gerrit.wikimedia.org/r/69804 [21:46:07] hashar: is localhost actually looked up? [21:46:38] via gethostbyname() I guess [21:47:39] New patchset: Ottomata; "Puppetizing oozie client and server" [operations/puppet/cdh4] (master) - https://gerrit.wikimedia.org/r/69804 [21:48:09] New patchset: Ottomata; "Adding missing jdbc_port parameter in hive class" [operations/puppet/cdh4] (master) - https://gerrit.wikimedia.org/r/74069 [21:48:13] ori-l: you should be able to point 'localhost' to another ip such as 127.0.0.51 [21:49:10] New review: Odder; "There is community consensus now, looks OK to me." [operations/mediawiki-config] (master) C: 1; - https://gerrit.wikimedia.org/r/73578 [21:50:18] Change merged: Ottomata; [operations/puppet/cdh4] (master) - https://gerrit.wikimedia.org/r/74069 [21:50:44] New patchset: Ottomata; "Puppetizing oozie client and server" [operations/puppet/cdh4] (master) - https://gerrit.wikimedia.org/r/69804 [21:51:29] sleeping [21:54:51] RECOVERY - Puppet freshness on cp1042 is OK: puppet ran at Tue Jul 16 21:54:49 UTC 2013 [21:55:51] PROBLEM - Puppet freshness on cp1042 is CRITICAL: No successful Puppet run in the last 10 hours [21:58:51] RECOVERY - Puppet freshness on cp1044 is OK: puppet ran at Tue Jul 16 21:58:47 UTC 2013 [21:59:01] RECOVERY - Puppet freshness on cp1041 is OK: puppet ran at Tue Jul 16 21:58:57 UTC 2013 [21:59:21] PROBLEM - Puppet freshness on cp1041 is CRITICAL: No successful Puppet run in the last 10 hours [21:59:22] PROBLEM - Puppet freshness on cp1044 is CRITICAL: No successful Puppet run in the last 10 hours [22:01:11] ori-l: yeah. localhost is definitely looked up [22:01:15] it has to be [22:02:51] RECOVERY - Puppet freshness on cp1043 is OK: puppet ran at Tue Jul 16 22:02:42 UTC 2013 [22:03:41] PROBLEM - Puppet freshness on cp1043 is CRITICAL: No successful Puppet run in the last 10 hours [22:13:16] OS X comes with a warning about removing the localhost entry. [22:15:32] <^demon> Elsie: Have they put a release date on the new OSX yet? [22:15:43] <^demon> I want my better dual monitor support like yesterday. [22:15:50] Haha [22:16:21] https://en.wikipedia.org/wiki/OS_X#Versions [22:16:24] Looks like it's still TBA. [22:16:47] By the end of the ear, for sure. [22:16:48] year [22:18:22] <^demon> Also, during the entire OSX part of the keynote...every time they said "Maverick" I had flashbacks to the '08 election. [22:18:52] I hope it's a free update. They really haven't done anything major in ages. [22:19:13] "Subtly changed the platform for the Dock icons." Greaaaat. [22:25:02] RECOVERY - Puppet freshness on cp1042 is OK: puppet ran at Tue Jul 16 22:24:52 UTC 2013 [22:25:52] PROBLEM - Puppet freshness on cp1042 is CRITICAL: No successful Puppet run in the last 10 hours [22:27:42] RECOVERY - Puppet freshness on cp1044 is OK: puppet ran at Tue Jul 16 22:27:40 UTC 2013 [22:28:22] PROBLEM - Puppet freshness on cp1044 is CRITICAL: No successful Puppet run in the last 10 hours [22:28:58] ori-l: whoah, that's awesome [22:29:02] RECOVERY - Puppet freshness on cp1041 is OK: puppet ran at Tue Jul 16 22:29:01 UTC 2013 [22:29:23] PROBLEM - Puppet freshness on cp1041 is CR [22:48:58] New review: Faidon; "Looks good. the password thing is the only thing that troubles me a bit." [operations/puppet/cdh4] (master) C: 1; - https://gerrit.wikimedia.org/r/69804 [22:53:45] New patchset: Ori.livneh; "Make role::eventlogging realm-aware" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/74085 [22:54:44] New patchset: Ori.livneh; "Make role::eventlogging realm-aware" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/74085 [22:56:11] New patchset: Ori.livneh; "Make role::eventlogging realm-aware" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/74085 [23:10:14] bsitu: ping [23:10:33] Krinkle: I am almost done [23:10:40] New review: CSteipp; "Sounds good to me" [operations/mediawiki-config] (master) C: 1; - https://gerrit.wikimedia.org/r/73603 [23:10:43] bsitu: Do you need scap? [23:10:47] Kinkle: no [23:10:49] I do, so we could do it together [23:10:51] OK [23:11:01] Kindle: just sync-dir [23:13:39] !log bsitu synchronized php-1.22wmf10/extensions/Echo 'Update Echo to master' [23:13:49] Logged the message, Master [23:13:52] New review: MaxSem; "w00t!" [operations/puppet] (production) C: 1; - https://gerrit.wikimedia.org/r/74085 [23:14:10] !log bsitu synchronized php-1.22wmf9/extensions/Echo 'Update Echo to master' [23:14:21] Logged the message, Master [23:14:44] Krinkle: I am done [23:14:53] Thx [23:15:12] <^demon> manybubbles: Is solr-mw2 already running on the ES branch? [23:16:02] Ryan_Lane: when specifying regexes for labs deployment hosts, is there a reason to prefer I-XXXXXXXX.pmtpa.wmflabs rather than a more human-readable hostname assigned to the instance? [23:16:46] i'm setting it up for EL, wondering if I should use 'deployment-eventlogging.pmtpa.wmflabs' or 'I-00000733.pmtpa.wmflabs'; they both resolve to the same instance. [23:18:30] ori-l: salt and puppet in labs use the i-xxx name for ids [23:18:54] I'd like to switch everything to the human readable name, but that requires us to rename all the old instances [23:19:00] ah, makes sense [23:19:02] and for us to purge keys when an instance is deleted [23:19:11] it's doable, but hasn't been a huge priority [23:19:17] ori-l: I'm working on using grains right now [23:19:23] so the regex will die [23:20:36] AaronSchulz: http://gdash.wikimedia.org/dashboards/pcache/ [23:20:48] see the 1 week latency graph [23:21:00] New patchset: Ori.livneh; "Make role::eventlogging realm-aware" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/74085 [23:21:00] New patchset: Ori.livneh; "Specify I-00000733 as labs git-deploy target for EventLogging" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/74089 [23:21:15] I see [23:21:25] Ryan_Lane: oh. should I abandon the latter change, then? [23:22:04] ori-l: yeah, i would. we don't really have labs set up for this right now [23:22:22] I probably do have this set up somewhere, but I'd need to figure out where :) [23:22:47] AaronSchulz: i have a feeling this is due to tim pointing tampa systems to eqiad memcached [23:22:52] to test the changes I'm making right now I'l going to need to set it up again somewhere [23:22:56] I'll probably use the sartoris project [23:23:18] AaronSchulz: which makes the data useless in terms of assessing real prod [23:23:24] Running scap now to deply 'Ifeea7885d67af6e53bf5c19ccaa231f86c7ead90' [23:23:37] yeah p99 can easily be made useless by a few stupid things [23:23:48] binasher: what things are affected? [23:24:19] Why doesn't scap do the log before it starts to do all the slow stuff. This isn't helpful to avoid issues with other people deploying stuff [23:24:34] AaronSchulz: the pretty graph things [23:24:51] tp90 impact is only ~1ms [23:25:03] I mean there most be some maintenance script or something [23:25:06] http://bit.ly/16HIQau [23:25:09] s/most/must [23:25:51] Ryan_Lane: k, abandoned [23:26:06] most profiled requests from tampa are going to be main page reqs from pybal and icinga [23:26:17] would be thankful for a merge of https://gerrit.wikimedia.org/r/#/c/74085/ [23:26:27] i think there's one set of crons running in pmtpa [23:26:34] there could be things i don't know about though [23:27:15] i should change gdash to show p90 for pcache latency [23:27:20] but.. my pretty graphs :( [23:27:30] binasher: btw, totally unrelated, but i noticed this yesterday: https://ganglia.wikimedia.org/latest/graph.php?r=month&z=xlarge&c=Bits+caches+esams&m=network_report&s=by+name&mc=2&g=network_report [23:29:40] same with eqiad - https://ganglia.wikimedia.org/latest/graph.php?r=month&z=large&c=Bits+caches+eqiad&m=cpu_report&s=by+name&mc=2&g=network_report [23:30:39] !log krinkle Started syncing Wikimedia installation... : [23:30:49] Logged the message, Master [23:32:55] class { misc::maintenance::refreshlinks: enabled => false } [23:33:00] * AaronSchulz wonders wtf that is [23:34:08] Change merged: Ryan Lane; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/74085 [23:34:22] ori-l: merged [23:34:24] Ryan_Lane: :D thanks! [23:34:27] yw [23:36:37] i like how every enwiki hit gets modules=ext.articleFeedbackv5.startup from resourceloader [23:37:06] !log krinkle Finished syncing Wikimedia installation... : [23:37:17] Logged the message, Master [23:42:49] binasher: wow. [23:48:06] the bits request that includes ext.centralNotice.bannerController is 102k gzip'd, >300k not [23:48:28] i wouldn't be surprised if turning on fundraising bloated bits traffic [23:48:37] that stuff isn't very efficient [23:49:41] most of it happens even on requests where a fr banner isn't displayed [23:51:26] still wish we'd build an ad server that isn't based on mediawiki for fundraising [23:51:44] "just use doubleclick" [23:55:00] jimbo pop-unders = $$$