[00:08:27] RECOVERY - Puppet freshness on ms2 is OK: puppet ran at Fri May 24 00:08:18 UTC 2013 [00:09:07] PROBLEM - Puppet freshness on ms2 is CRITICAL: No successful Puppet run in the last 10 hours [00:10:07] PROBLEM - Puppet freshness on lvs1004 is CRITICAL: No successful Puppet run in the last 10 hours [00:10:07] PROBLEM - Puppet freshness on lvs1005 is CRITICAL: No successful Puppet run in the last 10 hours [00:10:07] PROBLEM - Puppet freshness on lvs1006 is CRITICAL: No successful Puppet run in the last 10 hours [00:10:28] RECOVERY - Puppet freshness on ms2 is OK: puppet ran at Fri May 24 00:10:25 UTC 2013 [00:11:07] PROBLEM - Puppet freshness on ms2 is CRITICAL: No successful Puppet run in the last 10 hours [00:12:07] PROBLEM - Puppet freshness on mc15 is CRITICAL: No successful Puppet run in the last 10 hours [00:12:37] RECOVERY - Puppet freshness on ms2 is OK: puppet ran at Fri May 24 00:12:27 UTC 2013 [00:13:09] PROBLEM - Puppet freshness on ms2 is CRITICAL: No successful Puppet run in the last 10 hours [00:14:37] RECOVERY - Puppet freshness on ms2 is OK: puppet ran at Fri May 24 00:14:34 UTC 2013 [00:15:09] PROBLEM - Puppet freshness on ms2 is CRITICAL: No successful Puppet run in the last 10 hours [00:16:39] RECOVERY - Puppet freshness on ms2 is OK: puppet ran at Fri May 24 00:16:28 UTC 2013 [00:17:09] PROBLEM - Puppet freshness on ms2 is CRITICAL: No successful Puppet run in the last 10 hours [00:18:18] RECOVERY - Puppet freshness on ms2 is OK: puppet ran at Fri May 24 00:18:10 UTC 2013 [00:19:09] PROBLEM - Puppet freshness on ms2 is CRITICAL: No successful Puppet run in the last 10 hours [00:19:40] RECOVERY - Puppet freshness on ms2 is OK: puppet ran at Fri May 24 00:19:29 UTC 2013 [00:20:07] PROBLEM - Puppet freshness on ms2 is CRITICAL: No successful Puppet run in the last 10 hours [00:20:20] RECOVERY - Puppet freshness on ms2 is OK: puppet ran at Fri May 24 00:20:13 UTC 2013 [00:21:07] PROBLEM - Puppet freshness on ms2 is CRITICAL: No successful Puppet run in the last 10 hours [00:24:14] PROBLEM - Puppet freshness on db1017 is CRITICAL: No successful Puppet run in the last 10 hours [01:01:47] RECOVERY - NTP on ssl3003 is OK: NTP OK: Offset -0.0002281665802 secs [01:02:27] RECOVERY - NTP on ssl3002 is OK: NTP OK: Offset -0.004601716995 secs [02:07:17] !log LocalisationUpdate completed (1.22wmf4) at Fri May 24 02:07:17 UTC 2013 [02:07:27] Logged the message, Master [02:28:20] !log LocalisationUpdate ResourceLoader cache refresh completed at Fri May 24 02:28:20 UTC 2013 [02:28:29] Logged the message, Master [04:08:44] RECOVERY - Puppet freshness on ms2 is OK: puppet ran at Fri May 24 04:08:35 UTC 2013 [04:09:33] PROBLEM - Puppet freshness on ms2 is CRITICAL: No successful Puppet run in the last 10 hours [04:11:13] RECOVERY - Puppet freshness on ms2 is OK: puppet ran at Fri May 24 04:11:11 UTC 2013 [04:11:34] PROBLEM - Puppet freshness on ms2 is CRITICAL: No successful Puppet run in the last 10 hours [04:13:34] RECOVERY - Puppet freshness on ms2 is OK: puppet ran at Fri May 24 04:13:23 UTC 2013 [04:13:34] PROBLEM - Puppet freshness on ms2 is CRITICAL: No successful Puppet run in the last 10 hours [04:15:43] RECOVERY - Puppet freshness on ms2 is OK: puppet ran at Fri May 24 04:15:35 UTC 2013 [04:16:34] PROBLEM - Puppet freshness on ms2 is CRITICAL: No successful Puppet run in the last 10 hours [04:17:44] RECOVERY - Puppet freshness on ms2 is OK: puppet ran at Fri May 24 04:17:42 UTC 2013 [04:18:03] PROBLEM - Puppet freshness on virt1 is CRITICAL: No successful Puppet run in the last 10 hours [04:18:03] PROBLEM - Puppet freshness on virt3 is CRITICAL: No successful Puppet run in the last 10 hours [04:18:03] PROBLEM - Puppet freshness on virt4 is CRITICAL: No successful Puppet run in the last 10 hours [04:18:33] PROBLEM - Puppet freshness on ms2 is CRITICAL: No successful Puppet run in the last 10 hours [04:19:23] RECOVERY - Puppet freshness on ms2 is OK: puppet ran at Fri May 24 04:19:14 UTC 2013 [04:19:33] PROBLEM - Puppet freshness on ms2 is CRITICAL: No successful Puppet run in the last 10 hours [04:20:13] RECOVERY - Puppet freshness on ms2 is OK: puppet ran at Fri May 24 04:20:04 UTC 2013 [04:20:33] PROBLEM - Puppet freshness on ms2 is CRITICAL: No successful Puppet run in the last 10 hours [04:20:53] RECOVERY - Puppet freshness on ms2 is OK: puppet ran at Fri May 24 04:20:47 UTC 2013 [04:21:03] PROBLEM - Puppet freshness on pdf1 is CRITICAL: No successful Puppet run in the last 10 hours [04:21:03] PROBLEM - Puppet freshness on pdf2 is CRITICAL: No successful Puppet run in the last 10 hours [04:21:37] PROBLEM - Puppet freshness on ms2 is CRITICAL: No successful Puppet run in the last 10 hours [04:45:14] PROBLEM - Puppet freshness on db45 is CRITICAL: No successful Puppet run in the last 10 hours [04:50:46] PROBLEM - NTP on ssl3002 is CRITICAL: NTP CRITICAL: No response from NTP server [05:02:04] PROBLEM - NTP on ssl3003 is CRITICAL: NTP CRITICAL: No response from NTP server [06:11:22] PROBLEM - Puppet freshness on ms-fe3001 is CRITICAL: No successful Puppet run in the last 10 hours [06:28:52] RECOVERY - Puppet freshness on ms2 is OK: puppet ran at Fri May 24 06:28:49 UTC 2013 [06:29:11] PROBLEM - Puppet freshness on ms2 is CRITICAL: No successful Puppet run in the last 10 hours [06:30:01] RECOVERY - Puppet freshness on ms2 is OK: puppet ran at Fri May 24 06:29:53 UTC 2013 [06:30:12] PROBLEM - Puppet freshness on ms2 is CRITICAL: No successful Puppet run in the last 10 hours [06:30:41] RECOVERY - Puppet freshness on ms2 is OK: puppet ran at Fri May 24 06:30:40 UTC 2013 [06:31:12] PROBLEM - Puppet freshness on ms2 is CRITICAL: No successful Puppet run in the last 10 hours [06:31:21] RECOVERY - Puppet freshness on ms2 is OK: puppet ran at Fri May 24 06:31:19 UTC 2013 [06:32:11] PROBLEM - Puppet freshness on ms2 is CRITICAL: No successful Puppet run in the last 10 hours [06:45:00] RECOVERY - Puppet freshness on ms2 is OK: puppet ran at Fri May 24 06:44:52 UTC 2013 [06:45:10] PROBLEM - Puppet freshness on ms2 is CRITICAL: No successful Puppet run in the last 10 hours [07:02:00] PROBLEM - Puppet freshness on db44 is CRITICAL: No successful Puppet run in the last 10 hours [07:37:14] New patchset: Dereckson; "(bug 48763) Logo for el.wikivoyage" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/65175 [08:02:30] RECOVERY - NTP on ssl3003 is OK: NTP OK: Offset -0.002872109413 secs [08:05:29] RECOVERY - NTP on ssl3002 is OK: NTP OK: Offset -0.004616141319 secs [08:08:03] RECOVERY - Puppet freshness on ms2 is OK: puppet ran at Fri May 24 08:07:53 UTC 2013 [08:08:45] PROBLEM - Puppet freshness on ms2 is CRITICAL: No successful Puppet run in the last 10 hours [08:14:03] PROBLEM - Puppet freshness on erzurumi is CRITICAL: No successful Puppet run in the last 10 hours [08:14:53] RECOVERY - Puppet freshness on ms2 is OK: puppet ran at Fri May 24 08:14:51 UTC 2013 [08:15:43] PROBLEM - Puppet freshness on ms2 is CRITICAL: No successful Puppet run in the last 10 hours [08:36:23] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:38:00] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.122 second response time [09:00:17] New review: BBlack; "Patchset 8 (incoming) has multiple multicast groups, s/sender/purger/, and a single multi-headed que..." [operations/software/varnish/vhtcpd] (master) - https://gerrit.wikimedia.org/r/60390 [09:17:59] New patchset: BBlack; "vhtcpd - basically complete" [operations/software/varnish/vhtcpd] (master) - https://gerrit.wikimedia.org/r/60390 [09:29:44] New review: Faidon; "Thanks!" [operations/puppet] (production) C: 2; - https://gerrit.wikimedia.org/r/65117 [09:29:45] Change merged: Faidon; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/65117 [09:31:05] Change merged: Faidon; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/65156 [09:34:21] PROBLEM - DPKG on sockpuppet is CRITICAL: DPKG CRITICAL dpkg reports broken packages [09:35:21] RECOVERY - DPKG on sockpuppet is OK: All packages OK [09:43:19] #wmhack [09:45:23] New patchset: Pyoungmeister; "cking out testing stuffs for db77" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/65237 [09:46:47] Change merged: Pyoungmeister; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/65237 [09:47:18] #whack ... just one letter difference [09:47:50] heh [09:49:01] "Irssi: #whack: Total of 1 nicks" [09:49:02] sad [09:50:39] binasher, mark, MaxSem, yurik: MaxSem dropped and by and said he recommends we talk Varnish cache optimization after lunch. at last week's ops meeting binasher and Ryan_Lane suggested use of an ESI fragment for Zero banner rendering. see you at the ops table later, okay? [09:51:27] New patchset: Krinkle; "Fix various path inflexibilities and inconsistencies" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/62923 [09:51:38] New review: Krinkle; "Rebased" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/62923 [09:51:52] New patchset: Krinkle; "Fix various path inflexibilities and inconsistencies" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/62923 [09:52:43] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:52:50] PROBLEM - Host db77 is DOWN: PING CRITICAL - Packet loss = 100% [09:53:30] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.128 second response time [09:54:38] New patchset: Hashar; "lanthanum as a jenkins slave" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/64601 [09:54:43] New patchset: Krinkle; "Fix various path inflexibilities and inconsistencies" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/62923 [09:58:00] RECOVERY - Host db77 is UP: PING OK - Packet loss = 0%, RTA = 26.55 ms [10:00:20] PROBLEM - MySQL Recent Restart on db77 is CRITICAL: Connection refused by host [10:00:20] PROBLEM - SSH on db77 is CRITICAL: Connection refused [10:00:20] PROBLEM - MySQL Slave Running on db77 is CRITICAL: Connection refused by host [10:00:40] PROBLEM - MySQL Slave Delay on db77 is CRITICAL: Connection refused by host [10:00:40] PROBLEM - MySQL Idle Transactions on db77 is CRITICAL: Connection refused by host [10:00:41] PROBLEM - MySQL disk space on db77 is CRITICAL: Connection refused by host [10:00:41] PROBLEM - RAID on db77 is CRITICAL: Connection refused by host [10:00:41] PROBLEM - Disk space on db77 is CRITICAL: Connection refused by host [10:00:41] PROBLEM - Full LVS Snapshot on db77 is CRITICAL: Connection refused by host [10:01:00] PROBLEM - DPKG on db77 is CRITICAL: Connection refused by host [10:04:40] PROBLEM - Host db77 is DOWN: PING CRITICAL - Packet loss = 100% [10:05:20] RECOVERY - SSH on db77 is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1.1 (protocol 2.0) [10:05:30] RECOVERY - Host db77 is UP: PING OK - Packet loss = 0%, RTA = 26.59 ms [10:05:57] PROBLEM - NTP on db77 is CRITICAL: NTP CRITICAL: No response from NTP server [10:07:05] !log tridge upgrading, will require reboot [10:07:13] Logged the message, RobH [10:10:39] New patchset: Pyoungmeister; "comment for bookkeeping" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/65240 [10:10:56] PROBLEM - Puppet freshness on lvs1004 is CRITICAL: No successful Puppet run in the last 10 hours [10:10:56] PROBLEM - Puppet freshness on lvs1005 is CRITICAL: No successful Puppet run in the last 10 hours [10:10:56] PROBLEM - Puppet freshness on lvs1006 is CRITICAL: No successful Puppet run in the last 10 hours [10:12:56] PROBLEM - Puppet freshness on mc15 is CRITICAL: No successful Puppet run in the last 10 hours [10:16:56] RECOVERY - DPKG on db77 is OK: All packages OK [10:17:06] RECOVERY - MySQL disk space on db77 is OK: DISK OK [10:17:26] RECOVERY - RAID on db77 is OK: OK: State is Optimal, checked 6 logical device(s) [10:17:36] RECOVERY - Disk space on db77 is OK: DISK OK [10:17:58] New patchset: Dereckson; "(bug 48682) Enable flood group on es.wikipedia" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/65241 [10:21:04] !log installing upgrades on fenari [10:21:10] !log authdns-update to add fundraisingdb service aliases [10:21:16] Logged the message, Master [10:21:24] Logged the message, Master [10:24:56] PROBLEM - Puppet freshness on db1017 is CRITICAL: No successful Puppet run in the last 10 hours [10:27:26] New patchset: Reedy; "Wikimedia pakistan chapter redirect" [operations/apache-config] (master) - https://gerrit.wikimedia.org/r/65242 [10:27:32] Change merged: Pyoungmeister; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/65240 [10:27:40] !log jenkins: updating all phplint jobs to get rid of the old ant building script [10:27:45] Change merged: Asher; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/64834 [10:27:49] Logged the message, Master [10:28:46] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:28:50] New patchset: Mark Bergsma; "Decommission old knsq* servers, and amssq* upload squids" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/65243 [10:29:36] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.137 second response time [10:32:11] New patchset: Hashar; "lanthanum as a jenkins slave" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/64601 [10:32:24] New patchset: Asher; "typo fix" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/65244 [10:32:45] Change merged: Asher; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/65244 [10:34:16] RECOVERY - NTP on db77 is OK: NTP OK: Offset -0.0002356767654 secs [10:36:57] New patchset: Mark Bergsma; "Decommission old knsq* servers, and amssq* upload squids" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/65243 [10:37:36] Change merged: Mark Bergsma; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/65243 [10:38:55] MariaDB 5.5.31! [10:41:21] New patchset: Mark Bergsma; "I screwed that up didn't I?" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/65245 [10:41:44] !log authdns-update again... [10:41:54] Logged the message, Master [10:41:55] New patchset: Mark Bergsma; "I screwed that up didn't I?" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/65245 [10:42:41] Change merged: Mark Bergsma; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/65245 [10:44:45] PROBLEM - Auth DNS on ns0.wikimedia.org is CRITICAL: CRITICAL - Plugin timed out while executing system call [10:45:36] RECOVERY - Auth DNS on ns0.wikimedia.org is OK: DNS OK: 0.031 seconds response time. www.wikipedia.org returns 208.80.154.225 [10:45:42] restarted pdns on dobson [10:56:17] New patchset: Ottomata; "Puppetizing Hadoop for CDH4." [operations/puppet/cdh4] (master) - https://gerrit.wikimedia.org/r/61710 [11:04:57] New patchset: Ottomata; "Puppetizing Hadoop for CDH4." [operations/puppet/cdh4] (master) - https://gerrit.wikimedia.org/r/61710 [11:06:13] PROBLEM - Puppet freshness on amssq50 is CRITICAL: No successful Puppet run in the last 10 hours [11:06:13] PROBLEM - Puppet freshness on amssq56 is CRITICAL: No successful Puppet run in the last 10 hours [11:06:23] PROBLEM - Puppet freshness on amssq51 is CRITICAL: No successful Puppet run in the last 10 hours [11:06:23] PROBLEM - Puppet freshness on knsq17 is CRITICAL: No successful Puppet run in the last 10 hours [11:06:23] PROBLEM - Puppet freshness on knsq21 is CRITICAL: No successful Puppet run in the last 10 hours [11:06:23] PROBLEM - Puppet freshness on knsq28 is CRITICAL: No successful Puppet run in the last 10 hours [11:06:43] PROBLEM - Puppet freshness on amssq47 is CRITICAL: No successful Puppet run in the last 10 hours [11:06:43] PROBLEM - Puppet freshness on amssq59 is CRITICAL: No successful Puppet run in the last 10 hours [11:06:43] PROBLEM - Puppet freshness on amssq53 is CRITICAL: No successful Puppet run in the last 10 hours [11:06:43] PROBLEM - Puppet freshness on knsq18 is CRITICAL: No successful Puppet run in the last 10 hours [11:06:43] PROBLEM - Puppet freshness on knsq22 is CRITICAL: No successful Puppet run in the last 10 hours [11:06:53] PROBLEM - Puppet freshness on amssq48 is CRITICAL: No successful Puppet run in the last 10 hours [11:06:53] PROBLEM - Puppet freshness on amssq60 is CRITICAL: No successful Puppet run in the last 10 hours [11:06:53] PROBLEM - Puppet freshness on knsq19 is CRITICAL: No successful Puppet run in the last 10 hours [11:06:53] PROBLEM - Puppet freshness on knsq26 is CRITICAL: No successful Puppet run in the last 10 hours [11:07:03] PROBLEM - Puppet freshness on knsq16 is CRITICAL: No successful Puppet run in the last 10 hours [11:07:03] PROBLEM - Puppet freshness on knsq27 is CRITICAL: No successful Puppet run in the last 10 hours [11:21:18] New review: Matthias Mullie; "Is anything else needed to get this one merged?" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/62602 [11:35:32] New review: Faidon; "Can we use 3128 instead of 81? Otherwise it looks good I think, in combination with r64008 to actual..." [operations/puppet] (production) - https://gerrit.wikimedia.org/r/63890 [11:36:04] PROBLEM - Puppet freshness on amssq61 is CRITICAL: No successful Puppet run in the last 10 hours [11:36:05] PROBLEM - Puppet freshness on amssq54 is CRITICAL: No successful Puppet run in the last 10 hours [11:36:05] PROBLEM - Puppet freshness on knsq24 is CRITICAL: No successful Puppet run in the last 10 hours [11:36:14] PROBLEM - Puppet freshness on amssq49 is CRITICAL: No successful Puppet run in the last 10 hours [11:36:14] PROBLEM - Puppet freshness on amssq55 is CRITICAL: No successful Puppet run in the last 10 hours [11:36:24] PROBLEM - Puppet freshness on amssq57 is CRITICAL: No successful Puppet run in the last 10 hours [11:36:24] PROBLEM - Puppet freshness on amssq62 is CRITICAL: No successful Puppet run in the last 10 hours [11:36:24] PROBLEM - Puppet freshness on knsq29 is CRITICAL: No successful Puppet run in the last 10 hours [11:36:34] PROBLEM - Puppet freshness on amssq52 is CRITICAL: No successful Puppet run in the last 10 hours [11:36:34] PROBLEM - Puppet freshness on amssq58 is CRITICAL: No successful Puppet run in the last 10 hours [11:36:34] PROBLEM - Puppet freshness on knsq20 is CRITICAL: No successful Puppet run in the last 10 hours [11:36:54] PROBLEM - Puppet freshness on knsq23 is CRITICAL: No successful Puppet run in the last 10 hours [11:37:31] New patchset: Hashar; "beta: configuration for Wikidata" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/61428 [11:37:40] New review: Hashar; "(1 comment)" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/61428 [11:39:29] Change merged: jenkins-bot; [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/61428 [11:43:08] in about half an hour I will be heading out, and then on a rain for 4 hours so not reachable in irc, and probably spotty access all weekend [11:43:59] Is it a big rain? [11:44:46] apergos doesn't just run the cloud, they're IN the cloud [11:45:09] New review: Faidon; "I don't feel sufficiently informed to say if it's sane from a Hadoop PoV, but I don't see any obviou..." [operations/puppet/cdh4] (master) C: 2; - https://gerrit.wikimedia.org/r/61710 [11:45:37] train [11:45:42] my typos are getting dangerous [11:49:10] Change merged: Ottomata; [operations/puppet/cdh4] (master) - https://gerrit.wikimedia.org/r/61710 [12:01:36] paravoid: do you have that squid conf list? :) [12:01:44] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:01:49] yes [12:02:06] I need to do another pass or two, but I have an initial todo list [12:02:34] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.124 second response time [12:02:40] ok [12:02:55] New patchset: BBlack; "vhtcpd - basically complete" [operations/software/varnish/vhtcpd] (master) - https://gerrit.wikimedia.org/r/60390 [12:03:03] http://etherpad.wikimedia.org/SquidToVarnish [12:03:13] New patchset: Ottomata; "Adding git submodule operations/pupppet/cdh4 at modules/cdh4" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/65247 [12:03:49] New review: BBlack; "Patchset 9 just adds very very basic stats output so that ganglia will be able to see some useful r..." [operations/software/varnish/vhtcpd] (master) - https://gerrit.wikimedia.org/r/60390 [12:04:17] mark: close to what you wanted? [12:04:53] perfect [12:07:42] paravoid, I just did the first git submodule commit. Want to review that and try puppet-merge with me? or should I just do it? [12:07:42] RECOVERY - Puppet freshness on knsq20 is OK: puppet ran at Fri May 24 12:07:35 UTC 2013 [12:07:42] RECOVERY - Puppet freshness on knsq24 is OK: puppet ran at Fri May 24 12:07:35 UTC 2013 [12:07:42] RECOVERY - Puppet freshness on knsq28 is OK: puppet ran at Fri May 24 12:07:40 UTC 2013 [12:07:52] RECOVERY - Puppet freshness on knsq16 is OK: puppet ran at Fri May 24 12:07:49 UTC 2013 [12:07:52] RECOVERY - Puppet freshness on amssq53 is OK: puppet ran at Fri May 24 12:07:49 UTC 2013 [12:08:02] RECOVERY - Puppet freshness on knsq18 is OK: puppet ran at Fri May 24 12:07:51 UTC 2013 [12:08:02] PROBLEM - Puppet freshness on knsq16 is CRITICAL: No successful Puppet run in the last 10 hours [12:08:02] PROBLEM - Puppet freshness on knsq24 is CRITICAL: No successful Puppet run in the last 10 hours [12:08:02] RECOVERY - Puppet freshness on amssq48 is OK: puppet ran at Fri May 24 12:07:57 UTC 2013 [12:08:02] RECOVERY - Puppet freshness on ms2 is OK: puppet ran at Fri May 24 12:07:59 UTC 2013 [12:08:03] RECOVERY - Puppet freshness on amssq47 is OK: puppet ran at Fri May 24 12:08:01 UTC 2013 [12:08:13] RECOVERY - Puppet freshness on amssq56 is OK: puppet ran at Fri May 24 12:08:02 UTC 2013 [12:08:14] RECOVERY - Puppet freshness on amssq51 is OK: puppet ran at Fri May 24 12:08:02 UTC 2013 [12:08:14] PROBLEM - Puppet freshness on amssq47 is CRITICAL: No successful Puppet run in the last 10 hours [12:08:14] PROBLEM - Puppet freshness on amssq56 is CRITICAL: No successful Puppet run in the last 10 hours [12:08:14] PROBLEM - Puppet freshness on knsq18 is CRITICAL: No successful Puppet run in the last 10 hours [12:08:24] PROBLEM - Puppet freshness on amssq53 is CRITICAL: No successful Puppet run in the last 10 hours [12:08:24] PROBLEM - Puppet freshness on amssq51 is CRITICAL: No successful Puppet run in the last 10 hours [12:08:24] PROBLEM - Puppet freshness on knsq28 is CRITICAL: No successful Puppet run in the last 10 hours [12:08:32] RECOVERY - Puppet freshness on knsq27 is OK: puppet ran at Fri May 24 12:08:22 UTC 2013 [12:08:32] RECOVERY - Puppet freshness on knsq26 is OK: puppet ran at Fri May 24 12:08:23 UTC 2013 [12:08:32] PROBLEM - Puppet freshness on knsq20 is CRITICAL: No successful Puppet run in the last 10 hours [12:08:32] PROBLEM - Puppet freshness on ms2 is CRITICAL: No successful Puppet run in the last 10 hours [12:08:42] RECOVERY - Puppet freshness on amssq59 is OK: puppet ran at Fri May 24 12:08:32 UTC 2013 [12:08:42] RECOVERY - Puppet freshness on knsq21 is OK: puppet ran at Fri May 24 12:08:40 UTC 2013 [12:08:57] RECOVERY - Puppet freshness on knsq22 is OK: puppet ran at Fri May 24 12:08:43 UTC 2013 [12:08:57] RECOVERY - Puppet freshness on amssq50 is OK: puppet ran at Fri May 24 12:08:43 UTC 2013 [12:09:04] RECOVERY - Puppet freshness on amssq60 is OK: puppet ran at Fri May 24 12:08:52 UTC 2013 [12:09:04] PROBLEM - Puppet freshness on knsq27 is CRITICAL: No successful Puppet run in the last 10 hours [12:09:04] PROBLEM - Puppet freshness on amssq48 is CRITICAL: No successful Puppet run in the last 10 hours [12:09:12] RECOVERY - Puppet freshness on knsq17 is OK: puppet ran at Fri May 24 12:09:02 UTC 2013 [12:09:12] RECOVERY - Puppet freshness on amssq58 is OK: puppet ran at Fri May 24 12:09:02 UTC 2013 [12:09:12] RECOVERY - Puppet freshness on knsq19 is OK: puppet ran at Fri May 24 12:09:02 UTC 2013 [12:09:12] PROBLEM - Puppet freshness on amssq50 is CRITICAL: No successful Puppet run in the last 10 hours [12:09:12] PROBLEM - Puppet freshness on knsq26 is CRITICAL: No successful Puppet run in the last 10 hours [12:09:13] RECOVERY - Puppet freshness on knsq23 is OK: puppet ran at Fri May 24 12:09:10 UTC 2013 [12:09:22] RECOVERY - Puppet freshness on amssq62 is OK: puppet ran at Fri May 24 12:09:16 UTC 2013 [12:09:23] PROBLEM - Puppet freshness on amssq62 is CRITICAL: No successful Puppet run in the last 10 hours [12:09:23] PROBLEM - Puppet freshness on knsq17 is CRITICAL: No successful Puppet run in the last 10 hours [12:09:23] PROBLEM - Puppet freshness on knsq21 is CRITICAL: No successful Puppet run in the last 10 hours [12:09:23] PROBLEM - Puppet freshness on knsq19 is CRITICAL: No successful Puppet run in the last 10 hours [12:09:23] RECOVERY - Puppet freshness on amssq49 is OK: puppet ran at Fri May 24 12:09:21 UTC 2013 [12:09:23] RECOVERY - Puppet freshness on amssq54 is OK: puppet ran at Fri May 24 12:09:21 UTC 2013 [12:09:32] RECOVERY - Puppet freshness on amssq61 is OK: puppet ran at Fri May 24 12:09:24 UTC 2013 [12:09:32] RECOVERY - Puppet freshness on knsq29 is OK: puppet ran at Fri May 24 12:09:24 UTC 2013 [12:09:32] PROBLEM - Puppet freshness on amssq59 is CRITICAL: No successful Puppet run in the last 10 hours [12:09:32] PROBLEM - Puppet freshness on amssq58 is CRITICAL: No successful Puppet run in the last 10 hours [12:09:33] RECOVERY - Puppet freshness on knsq20 is OK: puppet ran at Fri May 24 12:09:28 UTC 2013 [12:09:33] RECOVERY - Puppet freshness on amssq52 is OK: puppet ran at Fri May 24 12:09:29 UTC 2013 [12:09:33] RECOVERY - Puppet freshness on amssq55 is OK: puppet ran at Fri May 24 12:09:30 UTC 2013 [12:09:38] ottomata: go ahed [12:09:41] *ahead [12:09:42] RECOVERY - Puppet freshness on amssq57 is OK: puppet ran at Fri May 24 12:09:34 UTC 2013 [12:09:42] PROBLEM - Puppet freshness on knsq22 is CRITICAL: No successful Puppet run in the last 10 hours [12:09:42] RECOVERY - Puppet freshness on knsq24 is OK: puppet ran at Fri May 24 12:09:37 UTC 2013 [12:09:43] RECOVERY - Puppet freshness on knsq28 is OK: puppet ran at Fri May 24 12:09:40 UTC 2013 [12:09:52] PROBLEM - Puppet freshness on amssq60 is CRITICAL: No successful Puppet run in the last 10 hours [12:09:52] PROBLEM - Puppet freshness on knsq23 is CRITICAL: No successful Puppet run in the last 10 hours [12:09:52] RECOVERY - Puppet freshness on knsq16 is OK: puppet ran at Fri May 24 12:09:47 UTC 2013 [12:09:52] RECOVERY - Puppet freshness on amssq53 is OK: puppet ran at Fri May 24 12:09:47 UTC 2013 [12:09:52] RECOVERY - Puppet freshness on knsq18 is OK: puppet ran at Fri May 24 12:09:49 UTC 2013 [12:10:02] PROBLEM - Puppet freshness on knsq16 is CRITICAL: No successful Puppet run in the last 10 hours [12:10:02] PROBLEM - Puppet freshness on amssq61 is CRITICAL: No successful Puppet run in the last 10 hours [12:10:02] PROBLEM - Puppet freshness on knsq24 is CRITICAL: No successful Puppet run in the last 10 hours [12:10:02] PROBLEM - Puppet freshness on amssq54 is CRITICAL: No successful Puppet run in the last 10 hours [12:10:02] RECOVERY - Puppet freshness on amssq47 is OK: puppet ran at Fri May 24 12:09:56 UTC 2013 [12:10:03] RECOVERY - Puppet freshness on amssq48 is OK: puppet ran at Fri May 24 12:09:57 UTC 2013 [12:10:03] RECOVERY - Puppet freshness on ms2 is OK: puppet ran at Fri May 24 12:09:59 UTC 2013 [12:10:04] RECOVERY - Puppet freshness on amssq56 is OK: puppet ran at Fri May 24 12:10:00 UTC 2013 [12:10:04] RECOVERY - Puppet freshness on amssq51 is OK: puppet ran at Fri May 24 12:10:00 UTC 2013 [12:10:08] Change merged: Ottomata; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/65247 [12:10:12] RECOVERY - Puppet freshness on knsq27 is OK: puppet ran at Fri May 24 12:10:04 UTC 2013 [12:10:12] PROBLEM - Puppet freshness on amssq56 is CRITICAL: No successful Puppet run in the last 10 hours [12:10:12] PROBLEM - Puppet freshness on amssq47 is CRITICAL: No successful Puppet run in the last 10 hours [12:10:12] RECOVERY - Puppet freshness on knsq26 is OK: puppet ran at Fri May 24 12:10:05 UTC 2013 [12:10:12] PROBLEM - Puppet freshness on amssq49 is CRITICAL: No successful Puppet run in the last 10 hours [12:10:13] PROBLEM - Puppet freshness on knsq18 is CRITICAL: No successful Puppet run in the last 10 hours [12:10:13] PROBLEM - Puppet freshness on amssq55 is CRITICAL: No successful Puppet run in the last 10 hours [12:10:14] PROBLEM - Puppet freshness on knsq26 is CRITICAL: No successful Puppet run in the last 10 hours [12:10:24] RECOVERY - Puppet freshness on amssq59 is OK: puppet ran at Fri May 24 12:10:12 UTC 2013 [12:10:24] PROBLEM - Puppet freshness on amssq53 is CRITICAL: No successful Puppet run in the last 10 hours [12:10:24] RECOVERY - Puppet freshness on knsq21 is OK: puppet ran at Fri May 24 12:10:16 UTC 2013 [12:10:24] PROBLEM - Puppet freshness on amssq57 is CRITICAL: No successful Puppet run in the last 10 hours [12:10:24] PROBLEM - Puppet freshness on amssq51 is CRITICAL: No successful Puppet run in the last 10 hours [12:10:24] PROBLEM - Puppet freshness on knsq28 is CRITICAL: No successful Puppet run in the last 10 hours [12:10:25] PROBLEM - Puppet freshness on knsq21 is CRITICAL: No successful Puppet run in the last 10 hours [12:10:25] PROBLEM - Puppet freshness on knsq29 is CRITICAL: No successful Puppet run in the last 10 hours [12:10:32] RECOVERY - Puppet freshness on knsq22 is OK: puppet ran at Fri May 24 12:10:24 UTC 2013 [12:10:32] RECOVERY - Puppet freshness on amssq50 is OK: puppet ran at Fri May 24 12:10:24 UTC 2013 [12:10:32] PROBLEM - Puppet freshness on amssq59 is CRITICAL: No successful Puppet run in the last 10 hours [12:10:32] PROBLEM - Puppet freshness on amssq52 is CRITICAL: No successful Puppet run in the last 10 hours [12:10:32] PROBLEM - Puppet freshness on knsq20 is CRITICAL: No successful Puppet run in the last 10 hours [12:10:33] PROBLEM - Puppet freshness on ms2 is CRITICAL: No successful Puppet run in the last 10 hours [12:10:33] RECOVERY - Puppet freshness on amssq60 is OK: puppet ran at Fri May 24 12:10:30 UTC 2013 [12:10:42] RECOVERY - Puppet freshness on knsq17 is OK: puppet ran at Fri May 24 12:10:33 UTC 2013 [12:10:42] RECOVERY - Puppet freshness on knsq19 is OK: puppet ran at Fri May 24 12:10:33 UTC 2013 [12:10:42] RECOVERY - Puppet freshness on amssq58 is OK: puppet ran at Fri May 24 12:10:33 UTC 2013 [12:10:42] PROBLEM - Puppet freshness on knsq22 is CRITICAL: No successful Puppet run in the last 10 hours [12:10:42] RECOVERY - Puppet freshness on knsq23 is OK: puppet ran at Fri May 24 12:10:35 UTC 2013 [12:10:52] RECOVERY - Puppet freshness on amssq62 is OK: puppet ran at Fri May 24 12:10:42 UTC 2013 [12:10:52] RECOVERY - Puppet freshness on amssq54 is OK: puppet ran at Fri May 24 12:10:45 UTC 2013 [12:10:52] PROBLEM - Puppet freshness on amssq60 is CRITICAL: No successful Puppet run in the last 10 hours [12:10:52] PROBLEM - Puppet freshness on knsq23 is CRITICAL: No successful Puppet run in the last 10 hours [12:10:52] RECOVERY - Puppet freshness on amssq49 is OK: puppet ran at Fri May 24 12:10:46 UTC 2013 [12:10:53] RECOVERY - Puppet freshness on amssq61 is OK: puppet ran at Fri May 24 12:10:47 UTC 2013 [12:10:53] RECOVERY - Puppet freshness on knsq29 is OK: puppet ran at Fri May 24 12:10:47 UTC 2013 [12:11:02] RECOVERY - Puppet freshness on amssq52 is OK: puppet ran at Fri May 24 12:10:53 UTC 2013 [12:11:02] RECOVERY - Puppet freshness on knsq20 is OK: puppet ran at Fri May 24 12:10:53 UTC 2013 [12:11:02] PROBLEM - Puppet freshness on knsq27 is CRITICAL: No successful Puppet run in the last 10 hours [12:11:02] PROBLEM - Puppet freshness on amssq48 is CRITICAL: No successful Puppet run in the last 10 hours [12:11:02] PROBLEM - Puppet freshness on amssq61 is CRITICAL: No successful Puppet run in the last 10 hours [12:11:03] PROBLEM - Puppet freshness on amssq54 is CRITICAL: No successful Puppet run in the last 10 hours [12:11:03] RECOVERY - Puppet freshness on amssq55 is OK: puppet ran at Fri May 24 12:10:55 UTC 2013 [12:11:04] RECOVERY - Puppet freshness on amssq57 is OK: puppet ran at Fri May 24 12:10:57 UTC 2013 [12:11:04] RECOVERY - Puppet freshness on knsq24 is OK: puppet ran at Fri May 24 12:10:59 UTC 2013 [12:11:05] RECOVERY - Puppet freshness on knsq28 is OK: puppet ran at Fri May 24 12:11:01 UTC 2013 [12:11:12] PROBLEM - Puppet freshness on amssq55 is CRITICAL: No successful Puppet run in the last 10 hours [12:11:12] PROBLEM - Puppet freshness on amssq50 is CRITICAL: No successful Puppet run in the last 10 hours [12:11:12] PROBLEM - Puppet freshness on amssq49 is CRITICAL: No successful Puppet run in the last 10 hours [12:11:12] RECOVERY - Puppet freshness on knsq16 is OK: puppet ran at Fri May 24 12:11:11 UTC 2013 [12:11:25] RECOVERY - Puppet freshness on knsq18 is OK: puppet ran at Fri May 24 12:11:12 UTC 2013 [12:11:25] PROBLEM - Puppet freshness on amssq62 is CRITICAL: No successful Puppet run in the last 10 hours [12:11:25] PROBLEM - Puppet freshness on amssq57 is CRITICAL: No successful Puppet run in the last 10 hours [12:11:25] PROBLEM - Puppet freshness on knsq17 is CRITICAL: No successful Puppet run in the last 10 hours [12:11:25] PROBLEM - Puppet freshness on knsq19 is CRITICAL: No successful Puppet run in the last 10 hours [12:11:26] PROBLEM - Puppet freshness on knsq28 is CRITICAL: No successful Puppet run in the last 10 hours [12:11:26] PROBLEM - Puppet freshness on knsq29 is CRITICAL: No successful Puppet run in the last 10 hours [12:11:27] RECOVERY - Puppet freshness on amssq53 is OK: puppet ran at Fri May 24 12:11:17 UTC 2013 [12:11:27] RECOVERY - Puppet freshness on amssq48 is OK: puppet ran at Fri May 24 12:11:17 UTC 2013 [12:11:28] RECOVERY - Puppet freshness on amssq47 is OK: puppet ran at Fri May 24 12:11:17 UTC 2013 [12:11:32] RECOVERY - Puppet freshness on ms2 is OK: puppet ran at Fri May 24 12:11:24 UTC 2013 [12:11:32] RECOVERY - Puppet freshness on amssq56 is OK: puppet ran at Fri May 24 12:11:25 UTC 2013 [12:11:32] RECOVERY - Puppet freshness on amssq51 is OK: puppet ran at Fri May 24 12:11:25 UTC 2013 [12:11:32] PROBLEM - Puppet freshness on amssq58 is CRITICAL: No successful Puppet run in the last 10 hours [12:11:32] PROBLEM - Puppet freshness on amssq52 is CRITICAL: No successful Puppet run in the last 10 hours [12:11:33] PROBLEM - Puppet freshness on knsq20 is CRITICAL: No successful Puppet run in the last 10 hours [12:11:33] PROBLEM - Puppet freshness on ms2 is CRITICAL: No successful Puppet run in the last 10 hours [12:11:34] RECOVERY - Puppet freshness on knsq27 is OK: puppet ran at Fri May 24 12:11:31 UTC 2013 [12:11:34] RECOVERY - Puppet freshness on knsq26 is OK: puppet ran at Fri May 24 12:11:31 UTC 2013 [12:11:42] RECOVERY - Puppet freshness on amssq59 is OK: puppet ran at Fri May 24 12:11:35 UTC 2013 [12:11:42] RECOVERY - Puppet freshness on knsq21 is OK: puppet ran at Fri May 24 12:11:41 UTC 2013 [12:11:52] RECOVERY - Puppet freshness on amssq50 is OK: puppet ran at Fri May 24 12:11:42 UTC 2013 [12:11:52] RECOVERY - Puppet freshness on knsq22 is OK: puppet ran at Fri May 24 12:11:42 UTC 2013 [12:11:52] RECOVERY - Puppet freshness on amssq60 is OK: puppet ran at Fri May 24 12:11:46 UTC 2013 [12:11:52] RECOVERY - Puppet freshness on amssq58 is OK: puppet ran at Fri May 24 12:11:51 UTC 2013 [12:11:52] RECOVERY - Puppet freshness on knsq19 is OK: puppet ran at Fri May 24 12:11:51 UTC 2013 [12:12:02] RECOVERY - Puppet freshness on knsq17 is OK: puppet ran at Fri May 24 12:11:52 UTC 2013 [12:12:03] RECOVERY - Puppet freshness on knsq23 is OK: puppet ran at Fri May 24 12:11:54 UTC 2013 [12:12:03] PROBLEM - Puppet freshness on knsq27 is CRITICAL: No successful Puppet run in the last 10 hours [12:12:03] PROBLEM - Puppet freshness on knsq16 is CRITICAL: No successful Puppet run in the last 10 hours [12:12:03] PROBLEM - Puppet freshness on amssq48 is CRITICAL: No successful Puppet run in the last 10 hours [12:12:03] PROBLEM - Puppet freshness on knsq24 is CRITICAL: No successful Puppet run in the last 10 hours [12:12:12] RECOVERY - Puppet freshness on amssq62 is OK: puppet ran at Fri May 24 12:12:02 UTC 2013 [12:12:12] PROBLEM - Puppet freshness on amssq56 is CRITICAL: No successful Puppet run in the last 10 hours [12:12:12] PROBLEM - Puppet freshness on amssq47 is CRITICAL: No successful Puppet run in the last 10 hours [12:12:12] PROBLEM - Puppet freshness on knsq26 is CRITICAL: No successful Puppet run in the last 10 hours [12:12:12] PROBLEM - Puppet freshness on amssq50 is CRITICAL: No successful Puppet run in the last 10 hours [12:12:13] PROBLEM - Puppet freshness on knsq18 is CRITICAL: No successful Puppet run in the last 10 hours [12:12:13] RECOVERY - Puppet freshness on amssq54 is OK: puppet ran at Fri May 24 12:12:10 UTC 2013 [12:12:14] RECOVERY - Puppet freshness on amssq49 is OK: puppet ran at Fri May 24 12:12:10 UTC 2013 [12:12:14] yeahhhhh paravoid, puppet-merge worked just like we wanted! [12:12:18] yay [12:12:22] RECOVERY - Puppet freshness on amssq61 is OK: puppet ran at Fri May 24 12:12:12 UTC 2013 [12:12:22] RECOVERY - Puppet freshness on knsq29 is OK: puppet ran at Fri May 24 12:12:12 UTC 2013 [12:12:22] PROBLEM - Puppet freshness on amssq53 is CRITICAL: No successful Puppet run in the last 10 hours [12:12:23] RECOVERY - Puppet freshness on amssq52 is OK: puppet ran at Fri May 24 12:12:16 UTC 2013 [12:12:23] PROBLEM - Puppet freshness on knsq21 is CRITICAL: No successful Puppet run in the last 10 hours [12:12:23] PROBLEM - Puppet freshness on amssq51 is CRITICAL: No successful Puppet run in the last 10 hours [12:12:23] PROBLEM - Puppet freshness on amssq62 is CRITICAL: No successful Puppet run in the last 10 hours [12:12:24] PROBLEM - Puppet freshness on knsq17 is CRITICAL: No successful Puppet run in the last 10 hours [12:12:24] PROBLEM - Puppet freshness on knsq19 is CRITICAL: No successful Puppet run in the last 10 hours [12:12:25] PROBLEM - Puppet freshness on knsq29 is CRITICAL: No successful Puppet run in the last 10 hours [12:12:25] RECOVERY - Puppet freshness on knsq20 is OK: puppet ran at Fri May 24 12:12:17 UTC 2013 [12:12:26] RECOVERY - Puppet freshness on amssq55 is OK: puppet ran at Fri May 24 12:12:19 UTC 2013 [12:12:26] RECOVERY - Puppet freshness on amssq57 is OK: puppet ran at Fri May 24 12:12:20 UTC 2013 [12:12:26] good job :) [12:12:32] RECOVERY - Puppet freshness on knsq24 is OK: puppet ran at Fri May 24 12:12:22 UTC 2013 [12:12:32] RECOVERY - Puppet freshness on knsq28 is OK: puppet ran at Fri May 24 12:12:23 UTC 2013 [12:12:32] PROBLEM - Puppet freshness on amssq59 is CRITICAL: No successful Puppet run in the last 10 hours [12:12:32] PROBLEM - Puppet freshness on amssq58 is CRITICAL: No successful Puppet run in the last 10 hours [12:12:32] PROBLEM - Puppet freshness on amssq52 is CRITICAL: No successful Puppet run in the last 10 hours [12:12:33] PROBLEM - Puppet freshness on knsq20 is CRITICAL: No successful Puppet run in the last 10 hours [12:12:33] RECOVERY - Puppet freshness on amssq53 is OK: puppet ran at Fri May 24 12:12:28 UTC 2013 [12:12:34] RECOVERY - Puppet freshness on knsq16 is OK: puppet ran at Fri May 24 12:12:29 UTC 2013 [12:12:34] RECOVERY - Puppet freshness on knsq18 is OK: puppet ran at Fri May 24 12:12:30 UTC 2013 [12:12:42] PROBLEM - Puppet freshness on knsq22 is CRITICAL: No successful Puppet run in the last 10 hours [12:12:42] RECOVERY - Puppet freshness on amssq56 is OK: puppet ran at Fri May 24 12:12:37 UTC 2013 [12:12:42] RECOVERY - Puppet freshness on amssq51 is OK: puppet ran at Fri May 24 12:12:37 UTC 2013 [12:12:42] RECOVERY - Puppet freshness on ms2 is OK: puppet ran at Fri May 24 12:12:37 UTC 2013 [12:12:42] RECOVERY - Puppet freshness on amssq48 is OK: puppet ran at Fri May 24 12:12:39 UTC 2013 [12:12:43] RECOVERY - Puppet freshness on amssq47 is OK: puppet ran at Fri May 24 12:12:39 UTC 2013 [12:12:52] RECOVERY - Puppet freshness on knsq27 is OK: puppet ran at Fri May 24 12:12:43 UTC 2013 [12:12:52] RECOVERY - Puppet freshness on knsq26 is OK: puppet ran at Fri May 24 12:12:43 UTC 2013 [12:12:52] PROBLEM - Puppet freshness on amssq60 is CRITICAL: No successful Puppet run in the last 10 hours [12:12:52] PROBLEM - Puppet freshness on knsq23 is CRITICAL: No successful Puppet run in the last 10 hours [12:13:02] RECOVERY - Puppet freshness on amssq59 is OK: puppet ran at Fri May 24 12:12:52 UTC 2013 [12:13:02] PROBLEM - Puppet freshness on knsq27 is CRITICAL: No successful Puppet run in the last 10 hours [12:13:02] PROBLEM - Puppet freshness on knsq16 is CRITICAL: No successful Puppet run in the last 10 hours [12:13:02] PROBLEM - Puppet freshness on amssq61 is CRITICAL: No successful Puppet run in the last 10 hours [12:13:02] PROBLEM - Puppet freshness on knsq24 is CRITICAL: No successful Puppet run in the last 10 hours [12:13:03] PROBLEM - Puppet freshness on amssq54 is CRITICAL: No successful Puppet run in the last 10 hours [12:13:03] PROBLEM - Puppet freshness on amssq48 is CRITICAL: No successful Puppet run in the last 10 hours [12:13:04] RECOVERY - Puppet freshness on knsq21 is OK: puppet ran at Fri May 24 12:13:00 UTC 2013 [12:13:12] RECOVERY - Puppet freshness on knsq22 is OK: puppet ran at Fri May 24 12:13:02 UTC 2013 [12:13:12] RECOVERY - Puppet freshness on amssq50 is OK: puppet ran at Fri May 24 12:13:02 UTC 2013 [12:13:12] PROBLEM - Puppet freshness on amssq56 is CRITICAL: No successful Puppet run in the last 10 hours [12:13:12] PROBLEM - Puppet freshness on amssq47 is CRITICAL: No successful Puppet run in the last 10 hours [12:13:12] PROBLEM - Puppet freshness on knsq18 is CRITICAL: No successful Puppet run in the last 10 hours [12:13:13] PROBLEM - Puppet freshness on amssq55 is CRITICAL: No successful Puppet run in the last 10 hours [12:13:13] PROBLEM - Puppet freshness on knsq26 is CRITICAL: No successful Puppet run in the last 10 hours [12:13:14] PROBLEM - Puppet freshness on amssq49 is CRITICAL: No successful Puppet run in the last 10 hours [12:13:14] PROBLEM - Puppet freshness on amssq50 is CRITICAL: No successful Puppet run in the last 10 hours [12:13:15] RECOVERY - Puppet freshness on amssq60 is OK: puppet ran at Fri May 24 12:13:07 UTC 2013 [12:13:15] RECOVERY - Puppet freshness on knsq17 is OK: puppet ran at Fri May 24 12:13:08 UTC 2013 [12:13:16] RECOVERY - Puppet freshness on amssq58 is OK: puppet ran at Fri May 24 12:13:08 UTC 2013 [12:13:16] RECOVERY - Puppet freshness on knsq19 is OK: puppet ran at Fri May 24 12:13:08 UTC 2013 [12:13:17] RECOVERY - Puppet freshness on knsq23 is OK: puppet ran at Fri May 24 12:13:10 UTC 2013 [12:13:25] RECOVERY - Puppet freshness on amssq62 is OK: puppet ran at Fri May 24 12:13:13 UTC 2013 [12:13:25] PROBLEM - Puppet freshness on amssq53 is CRITICAL: No successful Puppet run in the last 10 hours [12:13:25] RECOVERY - Puppet freshness on amssq49 is OK: puppet ran at Fri May 24 12:13:16 UTC 2013 [12:13:25] RECOVERY - Puppet freshness on amssq54 is OK: puppet ran at Fri May 24 12:13:16 UTC 2013 [12:13:25] PROBLEM - Puppet freshness on knsq21 is CRITICAL: No successful Puppet run in the last 10 hours [12:13:25] PROBLEM - Puppet freshness on amssq62 is CRITICAL: No successful Puppet run in the last 10 hours [12:13:25] PROBLEM - Puppet freshness on amssq51 is CRITICAL: No successful Puppet run in the last 10 hours [12:13:26] PROBLEM - Puppet freshness on knsq17 is CRITICAL: No successful Puppet run in the last 10 hours [12:13:26] PROBLEM - Puppet freshness on amssq57 is CRITICAL: No successful Puppet run in the last 10 hours [12:13:27] PROBLEM - Puppet freshness on knsq19 is CRITICAL: No successful Puppet run in the last 10 hours [12:13:27] PROBLEM - Puppet freshness on knsq28 is CRITICAL: No successful Puppet run in the last 10 hours [12:13:28] RECOVERY - Puppet freshness on knsq29 is OK: puppet ran at Fri May 24 12:13:20 UTC 2013 [12:13:28] RECOVERY - Puppet freshness on amssq61 is OK: puppet ran at Fri May 24 12:13:20 UTC 2013 [12:13:33] RECOVERY - Puppet freshness on amssq52 is OK: puppet ran at Fri May 24 12:13:25 UTC 2013 [12:13:36] RECOVERY - Puppet freshness on knsq20 is OK: puppet ran at Fri May 24 12:13:25 UTC 2013 [12:13:36] PROBLEM - Puppet freshness on ms2 is CRITICAL: No successful Puppet run in the last 10 hours [12:13:36] PROBLEM - Puppet freshness on amssq59 is CRITICAL: No successful Puppet run in the last 10 hours [12:13:36] PROBLEM - Puppet freshness on amssq58 is CRITICAL: No successful Puppet run in the last 10 hours [12:13:36] PROBLEM - Puppet freshness on amssq52 is CRITICAL: No successful Puppet run in the last 10 hours [12:13:36] PROBLEM - Puppet freshness on knsq20 is CRITICAL: No successful Puppet run in the last 10 hours [12:13:36] RECOVERY - Puppet freshness on amssq55 is OK: puppet ran at Fri May 24 12:13:29 UTC 2013 [12:13:37] RECOVERY - Puppet freshness on amssq57 is OK: puppet ran at Fri May 24 12:13:31 UTC 2013 [12:13:42] RECOVERY - Puppet freshness on knsq24 is OK: puppet ran at Fri May 24 12:13:33 UTC 2013 [12:13:42] RECOVERY - Puppet freshness on knsq28 is OK: puppet ran at Fri May 24 12:13:36 UTC 2013 [12:13:42] PROBLEM - Puppet freshness on knsq22 is CRITICAL: No successful Puppet run in the last 10 hours [12:13:42] RECOVERY - Puppet freshness on knsq16 is OK: puppet ran at Fri May 24 12:13:37 UTC 2013 [12:13:42] RECOVERY - Puppet freshness on amssq53 is OK: puppet ran at Fri May 24 12:13:41 UTC 2013 [12:13:52] RECOVERY - Puppet freshness on knsq18 is OK: puppet ran at Fri May 24 12:13:43 UTC 2013 [12:13:52] PROBLEM - Puppet freshness on amssq60 is CRITICAL: No successful Puppet run in the last 10 hours [12:13:52] PROBLEM - Puppet freshness on knsq23 is CRITICAL: No successful Puppet run in the last 10 hours [12:13:52] RECOVERY - Puppet freshness on amssq48 is OK: puppet ran at Fri May 24 12:13:46 UTC 2013 [12:13:52] RECOVERY - Puppet freshness on amssq47 is OK: puppet ran at Fri May 24 12:13:46 UTC 2013 [12:13:53] RECOVERY - Puppet freshness on amssq51 is OK: puppet ran at Fri May 24 12:13:50 UTC 2013 [12:13:53] RECOVERY - Puppet freshness on amssq56 is OK: puppet ran at Fri May 24 12:13:50 UTC 2013 [12:14:02] RECOVERY - Puppet freshness on knsq27 is OK: puppet ran at Fri May 24 12:13:53 UTC 2013 [12:14:02] RECOVERY - Puppet freshness on knsq26 is OK: puppet ran at Fri May 24 12:13:53 UTC 2013 [12:14:02] RECOVERY - Puppet freshness on ms2 is OK: puppet ran at Fri May 24 12:13:54 UTC 2013 [12:14:02] PROBLEM - Puppet freshness on knsq27 is CRITICAL: No successful Puppet run in the last 10 hours [12:14:02] PROBLEM - Puppet freshness on knsq16 is CRITICAL: No successful Puppet run in the last 10 hours [12:14:03] PROBLEM - Puppet freshness on amssq61 is CRITICAL: No successful Puppet run in the last 10 hours [12:14:03] PROBLEM - Puppet freshness on knsq24 is CRITICAL: No successful Puppet run in the last 10 hours [12:14:04] PROBLEM - Puppet freshness on amssq54 is CRITICAL: No successful Puppet run in the last 10 hours [12:14:04] PROBLEM - Puppet freshness on amssq48 is CRITICAL: No successful Puppet run in the last 10 hours [12:14:05] RECOVERY - Puppet freshness on amssq59 is OK: puppet ran at Fri May 24 12:13:58 UTC 2013 [12:14:05] RECOVERY - Puppet freshness on knsq21 is OK: puppet ran at Fri May 24 12:14:00 UTC 2013 [12:14:13] RECOVERY - Puppet freshness on amssq50 is OK: puppet ran at Fri May 24 12:14:01 UTC 2013 [12:14:13] RECOVERY - Puppet freshness on knsq22 is OK: puppet ran at Fri May 24 12:14:02 UTC 2013 [12:14:13] PROBLEM - Puppet freshness on amssq50 is CRITICAL: No successful Puppet run in the last 10 hours [12:14:13] PROBLEM - Puppet freshness on amssq56 is CRITICAL: No successful Puppet run in the last 10 hours [12:14:13] PROBLEM - Puppet freshness on amssq47 is CRITICAL: No successful Puppet run in the last 10 hours [12:14:13] PROBLEM - Puppet freshness on knsq18 is CRITICAL: No successful Puppet run in the last 10 hours [12:14:13] PROBLEM - Puppet freshness on amssq55 is CRITICAL: No successful Puppet run in the last 10 hours [12:14:14] PROBLEM - Puppet freshness on knsq26 is CRITICAL: No successful Puppet run in the last 10 hours [12:14:14] PROBLEM - Puppet freshness on amssq49 is CRITICAL: No successful Puppet run in the last 10 hours [12:14:15] RECOVERY - Puppet freshness on amssq60 is OK: puppet ran at Fri May 24 12:14:10 UTC 2013 [12:14:27] RECOVERY - Puppet freshness on knsq17 is OK: puppet ran at Fri May 24 12:14:12 UTC 2013 [12:14:27] RECOVERY - Puppet freshness on amssq58 is OK: puppet ran at Fri May 24 12:14:12 UTC 2013 [12:14:27] RECOVERY - Puppet freshness on knsq19 is OK: puppet ran at Fri May 24 12:14:13 UTC 2013 [12:14:27] PROBLEM - Puppet freshness on knsq29 is CRITICAL: No successful Puppet run in the last 10 hours [12:14:27] PROBLEM - Puppet freshness on amssq53 is CRITICAL: No successful Puppet run in the last 10 hours [12:14:27] RECOVERY - Puppet freshness on knsq23 is OK: puppet ran at Fri May 24 12:14:16 UTC 2013 [12:14:27] PROBLEM - Puppet freshness on knsq21 is CRITICAL: No successful Puppet run in the last 10 hours [12:14:27] PROBLEM - Puppet freshness on amssq51 is CRITICAL: No successful Puppet run in the last 10 hours [12:14:27] PROBLEM - Puppet freshness on knsq17 is CRITICAL: No successful Puppet run in the last 10 hours [12:14:27] PROBLEM - Puppet freshness on amssq57 is CRITICAL: No successful Puppet run in the last 10 hours [12:14:27] PROBLEM - Puppet freshness on knsq28 is CRITICAL: No successful Puppet run in the last 10 hours [12:14:27] PROBLEM - Puppet freshness on knsq19 is CRITICAL: No successful Puppet run in the last 10 hours [12:14:32] RECOVERY - Puppet freshness on amssq62 is OK: puppet ran at Fri May 24 12:14:21 UTC 2013 [12:14:32] RECOVERY - Puppet freshness on amssq54 is OK: puppet ran at Fri May 24 12:14:25 UTC 2013 [12:14:32] RECOVERY - Puppet freshness on amssq49 is OK: puppet ran at Fri May 24 12:14:25 UTC 2013 [12:14:32] PROBLEM - Puppet freshness on ms2 is CRITICAL: No successful Puppet run in the last 10 hours [12:14:32] RECOVERY - Puppet freshness on amssq61 is OK: puppet ran at Fri May 24 12:14:26 UTC 2013 [12:14:33] PROBLEM - Puppet freshness on amssq59 is CRITICAL: No successful Puppet run in the last 10 hours [12:14:33] PROBLEM - Puppet freshness on amssq58 is CRITICAL: No successful Puppet run in the last 10 hours [12:14:34] RECOVERY - Puppet freshness on knsq29 is OK: puppet ran at Fri May 24 12:14:27 UTC 2013 [12:14:34] RECOVERY - Puppet freshness on knsq20 is OK: puppet ran at Fri May 24 12:14:30 UTC 2013 [12:14:35] RECOVERY - Puppet freshness on amssq52 is OK: puppet ran at Fri May 24 12:14:30 UTC 2013 [12:14:42] RECOVERY - Puppet freshness on amssq55 is OK: puppet ran at Fri May 24 12:14:32 UTC 2013 [12:14:42] RECOVERY - Puppet freshness on amssq57 is OK: puppet ran at Fri May 24 12:14:33 UTC 2013 [12:14:42] RECOVERY - Puppet freshness on knsq24 is OK: puppet ran at Fri May 24 12:14:35 UTC 2013 [12:14:42] PROBLEM - Puppet freshness on knsq22 is CRITICAL: No successful Puppet run in the last 10 hours [12:14:42] RECOVERY - Puppet freshness on knsq28 is OK: puppet ran at Fri May 24 12:14:37 UTC 2013 [12:14:43] RECOVERY - Puppet freshness on amssq53 is OK: puppet ran at Fri May 24 12:14:39 UTC 2013 [12:14:43] RECOVERY - Puppet freshness on knsq16 is OK: puppet ran at Fri May 24 12:14:40 UTC 2013 [12:14:44] RECOVERY - Puppet freshness on knsq18 is OK: puppet ran at Fri May 24 12:14:41 UTC 2013 [12:14:52] RECOVERY - Puppet freshness on amssq47 is OK: puppet ran at Fri May 24 12:14:43 UTC 2013 [12:14:52] RECOVERY - Puppet freshness on amssq48 is OK: puppet ran at Fri May 24 12:14:43 UTC 2013 [12:14:52] RECOVERY - Puppet freshness on ms2 is OK: puppet ran at Fri May 24 12:14:45 UTC 2013 [12:14:52] RECOVERY - Puppet freshness on amssq56 is OK: puppet ran at Fri May 24 12:14:45 UTC 2013 [12:14:52] PROBLEM - Puppet freshness on amssq60 is CRITICAL: No successful Puppet run in the last 10 hours [12:14:53] PROBLEM - Puppet freshness on knsq23 is CRITICAL: No successful Puppet run in the last 10 hours [12:14:53] RECOVERY - Puppet freshness on amssq51 is OK: puppet ran at Fri May 24 12:14:46 UTC 2013 [12:14:54] RECOVERY - Puppet freshness on knsq27 is OK: puppet ran at Fri May 24 12:14:48 UTC 2013 [12:14:54] RECOVERY - Puppet freshness on knsq26 is OK: puppet ran at Fri May 24 12:14:48 UTC 2013 [12:14:55] RECOVERY - Puppet freshness on amssq59 is OK: puppet ran at Fri May 24 12:14:51 UTC 2013 [12:15:02] PROBLEM - Puppet freshness on knsq27 is CRITICAL: No successful Puppet run in the last 10 hours [12:15:02] RECOVERY - Puppet freshness on knsq21 is OK: puppet ran at Fri May 24 12:14:54 UTC 2013 [12:15:02] RECOVERY - Puppet freshness on amssq50 is OK: puppet ran at Fri May 24 12:14:55 UTC 2013 [12:15:02] RECOVERY - Puppet freshness on knsq22 is OK: puppet ran at Fri May 24 12:14:55 UTC 2013 [12:15:02] PROBLEM - Puppet freshness on knsq16 is CRITICAL: No successful Puppet run in the last 10 hours [12:15:03] PROBLEM - Puppet freshness on amssq61 is CRITICAL: No successful Puppet run in the last 10 hours [12:15:03] PROBLEM - Puppet freshness on knsq24 is CRITICAL: No successful Puppet run in the last 10 hours [12:15:04] PROBLEM - Puppet freshness on amssq48 is CRITICAL: No successful Puppet run in the last 10 hours [12:15:04] PROBLEM - Puppet freshness on amssq54 is CRITICAL: No successful Puppet run in the last 10 hours [12:15:05] RECOVERY - Puppet freshness on amssq60 is OK: puppet ran at Fri May 24 12:15:01 UTC 2013 [12:15:05] PROBLEM - Puppet freshness on amssq50 is CRITICAL: No successful Puppet run in the last 10 hours [12:15:13] RECOVERY - Puppet freshness on knsq17 is OK: puppet ran at Fri May 24 12:15:02 UTC 2013 [12:15:13] RECOVERY - Puppet freshness on amssq58 is OK: puppet ran at Fri May 24 12:15:02 UTC 2013 [12:15:13] RECOVERY - Puppet freshness on knsq19 is OK: puppet ran at Fri May 24 12:15:03 UTC 2013 [12:15:13] PROBLEM - Puppet freshness on amssq56 is CRITICAL: No successful Puppet run in the last 10 hours [12:15:13] PROBLEM - Puppet freshness on amssq47 is CRITICAL: No successful Puppet run in the last 10 hours [12:15:14] PROBLEM - Puppet freshness on knsq18 is CRITICAL: No successful Puppet run in the last 10 hours [12:15:14] PROBLEM - Puppet freshness on amssq55 is CRITICAL: No successful Puppet run in the last 10 hours [12:15:15] PROBLEM - Puppet freshness on knsq26 is CRITICAL: No successful Puppet run in the last 10 hours [12:15:15] PROBLEM - Puppet freshness on amssq49 is CRITICAL: No successful Puppet run in the last 10 hours [12:15:16] RECOVERY - Puppet freshness on knsq23 is OK: puppet ran at Fri May 24 12:15:07 UTC 2013 [12:15:22] RECOVERY - Puppet freshness on amssq54 is OK: puppet ran at Fri May 24 12:15:13 UTC 2013 [12:15:22] RECOVERY - Puppet freshness on amssq49 is OK: puppet ran at Fri May 24 12:15:13 UTC 2013 [12:15:22] PROBLEM - Puppet freshness on knsq17 is CRITICAL: No successful Puppet run in the last 10 hours [12:15:22] PROBLEM - Puppet freshness on knsq19 is CRITICAL: No successful Puppet run in the last 10 hours [12:15:22] PROBLEM - Puppet freshness on knsq29 is CRITICAL: No successful Puppet run in the last 10 hours [12:15:23] PROBLEM - Puppet freshness on amssq62 is CRITICAL: No successful Puppet run in the last 10 hours [12:15:23] PROBLEM - Puppet freshness on amssq53 is CRITICAL: No successful Puppet run in the last 10 hours [12:15:24] RECOVERY - Puppet freshness on knsq29 is OK: puppet ran at Fri May 24 12:15:16 UTC 2013 [12:15:24] RECOVERY - Puppet freshness on amssq61 is OK: puppet ran at Fri May 24 12:15:16 UTC 2013 [12:15:25] PROBLEM - Puppet freshness on knsq21 is CRITICAL: No successful Puppet run in the last 10 hours [12:15:25] PROBLEM - Puppet freshness on amssq51 is CRITICAL: No successful Puppet run in the last 10 hours [12:15:26] PROBLEM - Puppet freshness on amssq57 is CRITICAL: No successful Puppet run in the last 10 hours [12:15:26] PROBLEM - Puppet freshness on knsq28 is CRITICAL: No successful Puppet run in the last 10 hours [12:15:34] RECOVERY - Puppet freshness on amssq55 is OK: puppet ran at Fri May 24 12:15:22 UTC 2013 [12:15:36] RECOVERY - Puppet freshness on amssq57 is OK: puppet ran at Fri May 24 12:15:23 UTC 2013 [12:15:36] RECOVERY - Puppet freshness on knsq24 is OK: puppet ran at Fri May 24 12:15:24 UTC 2013 [12:15:36] RECOVERY - Puppet freshness on knsq28 is OK: puppet ran at Fri May 24 12:15:25 UTC 2013 [12:15:36] PROBLEM - Puppet freshness on amssq52 is CRITICAL: No successful Puppet run in the last 10 hours [12:15:36] PROBLEM - Puppet freshness on knsq20 is CRITICAL: No successful Puppet run in the last 10 hours [12:15:37] PROBLEM - Puppet freshness on ms2 is CRITICAL: No successful Puppet run in the last 10 hours [12:15:37] PROBLEM - Puppet freshness on amssq59 is CRITICAL: No successful Puppet run in the last 10 hours [12:15:38] PROBLEM - Puppet freshness on amssq58 is CRITICAL: No successful Puppet run in the last 10 hours [12:15:38] RECOVERY - Puppet freshness on amssq53 is OK: puppet ran at Fri May 24 12:15:28 UTC 2013 [12:15:39] RECOVERY - Puppet freshness on knsq16 is OK: puppet ran at Fri May 24 12:15:28 UTC 2013 [12:15:39] RECOVERY - Puppet freshness on knsq18 is OK: puppet ran at Fri May 24 12:15:29 UTC 2013 [12:15:42] RECOVERY - Puppet freshness on amssq47 is OK: puppet ran at Fri May 24 12:15:31 UTC 2013 [12:15:42] RECOVERY - Puppet freshness on amssq48 is OK: puppet ran at Fri May 24 12:15:32 UTC 2013 [12:15:42] RECOVERY - Puppet freshness on ms2 is OK: puppet ran at Fri May 24 12:15:33 UTC 2013 [12:15:42] RECOVERY - Puppet freshness on amssq51 is OK: puppet ran at Fri May 24 12:15:36 UTC 2013 [12:15:42] PROBLEM - Puppet freshness on knsq22 is CRITICAL: No successful Puppet run in the last 10 hours [12:15:43] RECOVERY - Puppet freshness on knsq26 is OK: puppet ran at Fri May 24 12:15:38 UTC 2013 [12:15:43] RECOVERY - Puppet freshness on knsq27 is OK: puppet ran at Fri May 24 12:15:38 UTC 2013 [12:15:52] RECOVERY - Puppet freshness on amssq56 is OK: puppet ran at Fri May 24 12:15:42 UTC 2013 [12:15:52] RECOVERY - Puppet freshness on amssq59 is OK: puppet ran at Fri May 24 12:15:45 UTC 2013 [12:15:52] PROBLEM - Puppet freshness on amssq60 is CRITICAL: No successful Puppet run in the last 10 hours [12:15:52] PROBLEM - Puppet freshness on knsq23 is CRITICAL: No successful Puppet run in the last 10 hours [12:15:52] RECOVERY - Puppet freshness on knsq21 is OK: puppet ran at Fri May 24 12:15:47 UTC 2013 [12:15:53] RECOVERY - Puppet freshness on amssq50 is OK: puppet ran at Fri May 24 12:15:48 UTC 2013 [12:15:53] RECOVERY - Puppet freshness on knsq22 is OK: puppet ran at Fri May 24 12:15:48 UTC 2013 [12:16:02] RECOVERY - Puppet freshness on amssq60 is OK: puppet ran at Fri May 24 12:15:52 UTC 2013 [12:16:02] RECOVERY - Puppet freshness on knsq17 is OK: puppet ran at Fri May 24 12:15:53 UTC 2013 [12:16:02] RECOVERY - Puppet freshness on knsq19 is OK: puppet ran at Fri May 24 12:15:53 UTC 2013 [12:16:03] RECOVERY - Puppet freshness on amssq58 is OK: puppet ran at Fri May 24 12:15:53 UTC 2013 [12:16:03] PROBLEM - Puppet freshness on knsq27 is CRITICAL: No successful Puppet run in the last 10 hours [12:16:03] RECOVERY - Puppet freshness on knsq23 is OK: puppet ran at Fri May 24 12:15:54 UTC 2013 [12:16:03] PROBLEM - Puppet freshness on amssq50 is CRITICAL: No successful Puppet run in the last 10 hours [12:16:04] PROBLEM - Puppet freshness on amssq61 is CRITICAL: No successful Puppet run in the last 10 hours [12:16:04] PROBLEM - Puppet freshness on knsq24 is CRITICAL: No successful Puppet run in the last 10 hours [12:16:05] PROBLEM - Puppet freshness on knsq16 is CRITICAL: No successful Puppet run in the last 10 hours [12:16:05] PROBLEM - Puppet freshness on amssq48 is CRITICAL: No successful Puppet run in the last 10 hours [12:16:06] PROBLEM - Puppet freshness on amssq54 is CRITICAL: No successful Puppet run in the last 10 hours [12:16:06] RECOVERY - Puppet freshness on amssq62 is OK: puppet ran at Fri May 24 12:15:56 UTC 2013 [12:16:07] RECOVERY - Puppet freshness on amssq54 is OK: puppet ran at Fri May 24 12:15:59 UTC 2013 [12:16:07] RECOVERY - Puppet freshness on amssq61 is OK: puppet ran at Fri May 24 12:16:00 UTC 2013 [12:16:12] RECOVERY - Puppet freshness on knsq20 is OK: puppet ran at Fri May 24 12:16:02 UTC 2013 [12:16:12] RECOVERY - Puppet freshness on amssq52 is OK: puppet ran at Fri May 24 12:16:02 UTC 2013 [12:16:12] RECOVERY - Puppet freshness on knsq24 is OK: puppet ran at Fri May 24 12:16:05 UTC 2013 [12:16:12] PROBLEM - Puppet freshness on amssq56 is CRITICAL: No successful Puppet run in the last 10 hours [12:16:12] PROBLEM - Puppet freshness on amssq47 is CRITICAL: No successful Puppet run in the last 10 hours [12:16:13] PROBLEM - Puppet freshness on knsq18 is CRITICAL: No successful Puppet run in the last 10 hours [12:16:13] PROBLEM - Puppet freshness on knsq26 is CRITICAL: No successful Puppet run in the last 10 hours [12:16:14] PROBLEM - Puppet freshness on amssq49 is CRITICAL: No successful Puppet run in the last 10 hours [12:16:14] PROBLEM - Puppet freshness on amssq55 is CRITICAL: No successful Puppet run in the last 10 hours [12:16:23] RECOVERY - Puppet freshness on knsq18 is OK: puppet ran at Fri May 24 12:16:13 UTC 2013 [12:16:23] RECOVERY - Puppet freshness on knsq16 is OK: puppet ran at Fri May 24 12:16:13 UTC 2013 [12:16:23] PROBLEM - Puppet freshness on knsq17 is CRITICAL: No successful Puppet run in the last 10 hours [12:16:23] PROBLEM - Puppet freshness on knsq19 is CRITICAL: No successful Puppet run in the last 10 hours [12:16:23] PROBLEM - Puppet freshness on knsq29 is CRITICAL: No successful Puppet run in the last 10 hours [12:16:24] PROBLEM - Puppet freshness on amssq62 is CRITICAL: No successful Puppet run in the last 10 hours [12:16:24] PROBLEM - Puppet freshness on amssq53 is CRITICAL: No successful Puppet run in the last 10 hours [12:16:25] RECOVERY - Puppet freshness on amssq48 is OK: puppet ran at Fri May 24 12:16:16 UTC 2013 [12:16:25] RECOVERY - Puppet freshness on amssq47 is OK: puppet ran at Fri May 24 12:16:16 UTC 2013 [12:16:26] PROBLEM - Puppet freshness on knsq21 is CRITICAL: No successful Puppet run in the last 10 hours [12:16:26] PROBLEM - Puppet freshness on amssq51 is CRITICAL: No successful Puppet run in the last 10 hours [12:16:27] PROBLEM - Puppet freshness on knsq28 is CRITICAL: No successful Puppet run in the last 10 hours [12:16:27] PROBLEM - Puppet freshness on amssq57 is CRITICAL: No successful Puppet run in the last 10 hours [12:16:32] RECOVERY - Puppet freshness on amssq56 is OK: puppet ran at Fri May 24 12:16:23 UTC 2013 [12:16:32] RECOVERY - Puppet freshness on amssq51 is OK: puppet ran at Fri May 24 12:16:23 UTC 2013 [12:16:32] PROBLEM - Puppet freshness on knsq20 is CRITICAL: No successful Puppet run in the last 10 hours [12:16:32] PROBLEM - Puppet freshness on ms2 is CRITICAL: No successful Puppet run in the last 10 hours [12:16:32] PROBLEM - Puppet freshness on amssq52 is CRITICAL: No successful Puppet run in the last 10 hours [12:16:33] PROBLEM - Puppet freshness on amssq59 is CRITICAL: No successful Puppet run in the last 10 hours [12:16:33] PROBLEM - Puppet freshness on amssq58 is CRITICAL: No successful Puppet run in the last 10 hours [12:16:42] PROBLEM - Puppet freshness on knsq22 is CRITICAL: No successful Puppet run in the last 10 hours [12:16:52] PROBLEM - Puppet freshness on amssq60 is CRITICAL: No successful Puppet run in the last 10 hours [12:16:52] PROBLEM - Puppet freshness on knsq23 is CRITICAL: No successful Puppet run in the last 10 hours [12:17:02] PROBLEM - Puppet freshness on amssq61 is CRITICAL: No successful Puppet run in the last 10 hours [12:17:02] PROBLEM - Puppet freshness on knsq24 is CRITICAL: No successful Puppet run in the last 10 hours [12:17:02] PROBLEM - Puppet freshness on knsq16 is CRITICAL: No successful Puppet run in the last 10 hours [12:17:02] PROBLEM - Puppet freshness on amssq54 is CRITICAL: No successful Puppet run in the last 10 hours [12:17:02] PROBLEM - Puppet freshness on amssq48 is CRITICAL: No successful Puppet run in the last 10 hours [12:17:13] PROBLEM - Puppet freshness on amssq56 is CRITICAL: No successful Puppet run in the last 10 hours [12:17:14] PROBLEM - Puppet freshness on amssq47 is CRITICAL: No successful Puppet run in the last 10 hours [12:17:14] PROBLEM - Puppet freshness on knsq18 is CRITICAL: No successful Puppet run in the last 10 hours [12:17:22] PROBLEM - Puppet freshness on amssq51 is CRITICAL: No successful Puppet run in the last 10 hours [12:17:32] RECOVERY - Puppet freshness on knsq26 is OK: puppet ran at Fri May 24 12:17:29 UTC 2013 [12:17:32] RECOVERY - Puppet freshness on knsq27 is OK: puppet ran at Fri May 24 12:17:30 UTC 2013 [12:18:02] PROBLEM - Puppet freshness on knsq27 is CRITICAL: No successful Puppet run in the last 10 hours [12:18:13] PROBLEM - Puppet freshness on knsq26 is CRITICAL: No successful Puppet run in the last 10 hours [12:20:32] RECOVERY - Puppet freshness on amssq59 is OK: puppet ran at Fri May 24 12:20:30 UTC 2013 [12:21:32] PROBLEM - Puppet freshness on amssq59 is CRITICAL: No successful Puppet run in the last 10 hours [12:22:42] RECOVERY - Puppet freshness on knsq21 is OK: puppet ran at Fri May 24 12:22:32 UTC 2013 [12:23:23] PROBLEM - Puppet freshness on knsq21 is CRITICAL: No successful Puppet run in the last 10 hours [12:24:02] RECOVERY - Puppet freshness on amssq50 is OK: puppet ran at Fri May 24 12:23:54 UTC 2013 [12:24:02] RECOVERY - Puppet freshness on knsq22 is OK: puppet ran at Fri May 24 12:23:54 UTC 2013 [12:24:02] PROBLEM - Puppet freshness on amssq50 is CRITICAL: No successful Puppet run in the last 10 hours [12:24:42] PROBLEM - Puppet freshness on knsq22 is CRITICAL: No successful Puppet run in the last 10 hours [12:25:32] RECOVERY - Puppet freshness on amssq60 is OK: puppet ran at Fri May 24 12:25:29 UTC 2013 [12:25:52] PROBLEM - Puppet freshness on amssq60 is CRITICAL: No successful Puppet run in the last 10 hours [12:25:56] New patchset: Mark Bergsma; "Add role::cache::text for Varnish" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/65252 [12:25:56] New patchset: Mark Bergsma; "Add Text Varnish VCL include boilerplate" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/65253 [12:27:02] RECOVERY - Puppet freshness on amssq58 is OK: puppet ran at Fri May 24 12:26:52 UTC 2013 [12:27:02] RECOVERY - Puppet freshness on knsq19 is OK: puppet ran at Fri May 24 12:26:58 UTC 2013 [12:27:02] RECOVERY - Puppet freshness on knsq17 is OK: puppet ran at Fri May 24 12:26:58 UTC 2013 [12:27:22] PROBLEM - Puppet freshness on knsq17 is CRITICAL: No successful Puppet run in the last 10 hours [12:27:22] PROBLEM - Puppet freshness on knsq19 is CRITICAL: No successful Puppet run in the last 10 hours [12:27:32] PROBLEM - Puppet freshness on amssq58 is CRITICAL: No successful Puppet run in the last 10 hours [12:27:32] RECOVERY - Puppet freshness on knsq23 is OK: puppet ran at Fri May 24 12:27:28 UTC 2013 [12:27:52] PROBLEM - Puppet freshness on knsq23 is CRITICAL: No successful Puppet run in the last 10 hours [12:29:33] RECOVERY - Puppet freshness on amssq62 is OK: puppet ran at Fri May 24 12:29:23 UTC 2013 [12:29:59] New patchset: Hashar; "beta: symlink /a/common" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/65254 [12:30:22] PROBLEM - Puppet freshness on amssq62 is CRITICAL: No successful Puppet run in the last 10 hours [12:31:05] New review: Hashar; "That is related to the tin migration." [operations/puppet] (production) - https://gerrit.wikimedia.org/r/65254 [12:31:53] RECOVERY - Puppet freshness on amssq49 is OK: puppet ran at Fri May 24 12:31:44 UTC 2013 [12:31:54] RECOVERY - Puppet freshness on amssq54 is OK: puppet ran at Fri May 24 12:31:44 UTC 2013 [12:32:02] PROBLEM - Puppet freshness on amssq54 is CRITICAL: No successful Puppet run in the last 10 hours [12:32:13] PROBLEM - Puppet freshness on amssq49 is CRITICAL: No successful Puppet run in the last 10 hours [12:32:33] RECOVERY - Puppet freshness on amssq61 is OK: puppet ran at Fri May 24 12:32:30 UTC 2013 [12:32:52] RECOVERY - Puppet freshness on knsq29 is OK: puppet ran at Fri May 24 12:32:51 UTC 2013 [12:33:02] PROBLEM - Puppet freshness on amssq61 is CRITICAL: No successful Puppet run in the last 10 hours [12:33:24] PROBLEM - Puppet freshness on knsq29 is CRITICAL: No successful Puppet run in the last 10 hours [12:33:46] New patchset: Reedy; "Disable Narayam on commons now that they have ULS" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/64884 [12:34:02] Change merged: jenkins-bot; [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/64884 [12:34:27] RobH: https://gerrit.wikimedia.org/r/#/c/64601/ [12:34:42] RECOVERY - Puppet freshness on knsq20 is OK: puppet ran at Fri May 24 12:34:34 UTC 2013 [12:34:43] RECOVERY - Puppet freshness on amssq52 is OK: puppet ran at Fri May 24 12:34:39 UTC 2013 [12:35:28] PROBLEM - Puppet freshness on amssq52 is CRITICAL: No successful Puppet run in the last 10 hours [12:35:28] PROBLEM - Puppet freshness on knsq20 is CRITICAL: No successful Puppet run in the last 10 hours [12:36:53] New patchset: Reedy; "(bug 40341) Enable translation import on wikis with Translate extension" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/64919 [12:37:19] Change merged: jenkins-bot; [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/64919 [12:37:53] New patchset: Aklapper; "Weekly Bugzilla Report mail: Fix wrong SQL query on bug resolutions" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/63998 [12:39:06] Change merged: Dzahn; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/63998 [12:39:43] New patchset: BBlack; "vhtcpd - basically complete" [operations/software/varnish/vhtcpd] (master) - https://gerrit.wikimedia.org/r/60390 [12:40:03] New patchset: Mark Bergsma; "Add role::cache::text for Varnish" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/65252 [12:40:04] New patchset: Mark Bergsma; "Add Text Varnish VCL include boilerplate" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/65253 [12:40:22] New review: BBlack; "Now with basic GPL copyright stuff, and some whitespace fixes. Still needs debian-ization." [operations/software/varnish/vhtcpd] (master) - https://gerrit.wikimedia.org/r/60390 [12:41:00] New patchset: Hashar; "lanthanum as a jenkins slave" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/64601 [12:41:00] New patchset: Hashar; "lanthanum is no more meant to be an IRC relay/server" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/65256 [12:41:06] cc2e56e lanthanum is no more meant to be an IRC relay/server [12:41:07] RobH: https://gerrit.wikimedia.org/r/65256 [12:41:08] :D [12:41:50] New review: BBlack; "Approving this myself so it appears in the real "master" and can move on from there..." [operations/software/varnish/vhtcpd] (master); V: 2 C: 2; - https://gerrit.wikimedia.org/r/60390 [12:41:50] Change merged: BBlack; [operations/software/varnish/vhtcpd] (master) - https://gerrit.wikimedia.org/r/60390 [12:42:26] New patchset: Reedy; "(bug 48075) Localise $wgSiteName, $wgMetaNamespace for dvwiki" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/64798 [12:42:45] Change merged: jenkins-bot; [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/64798 [12:44:02] New patchset: Reedy; "(bug 48763) Logo for el.wikivoyage" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/65175 [12:44:18] Change merged: RobH; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/65256 [12:44:23] Change merged: jenkins-bot; [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/65175 [12:45:15] Reedy: https://gerrit.wikimedia.org/r/#/c/64635/ was merely a test commit—I was helping out HazardSJ set up his git configuration properly [12:45:21] New patchset: Reedy; "(bug 48682) Enable flood group on es.wikipedia" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/65241 [12:45:30] Reedy: I guess he should simply abandon it? [12:45:32] So it was a test review ;) [12:45:55] Change merged: jenkins-bot; [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/65241 [12:46:47] !log reedy synchronized wmf-config/ [12:46:56] Logged the message, Master [12:50:07] PROBLEM - check_raid on samarium is CRITICAL: CRITICAL: None [12:50:13] RobH: lanthanum.wikimedia.org [12:51:27] PROBLEM - check_raid on samarium is CRITICAL: CRITICAL: None [12:51:37] PROBLEM - check_raid on samarium is CRITICAL: CRITICAL: None [12:52:21] New review: Reedy; "(1 comment)" [operations/mediawiki-config] (master) C: -1; - https://gerrit.wikimedia.org/r/62923 [12:54:17] RECOVERY - check_raid on samarium is OK: OK: State is Optimal, checked 2 logical device(s) [12:55:21] New patchset: Tim Starling; "Disable create page captcha on testwiki" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/65264 [12:56:20] Change merged: Tim Starling; [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/65264 [12:57:09] New patchset: Mark Bergsma; "Add role::cache::text for Varnish" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/65252 [12:57:09] New patchset: Mark Bergsma; "Add Text Varnish VCL include boilerplate" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/65253 [13:00:47] !log tstarling synchronized wmf-config/CommonSettings.php 'testwiki captcha' [13:00:56] Logged the message, Master [13:05:13] Change merged: Mark Bergsma; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/65252 [13:05:25] Change merged: Mark Bergsma; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/65253 [13:08:18] New patchset: Dzahn; "remove one space that broke the mysql query for bugzilla report" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/65265 [13:08:51] Published patchset: Dereckson; "Test SearchExtraNS extension on .beta.wmflabs.org" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/43387 [13:08:55] Change abandoned: Dereckson; "(no reason)" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/43387 [13:09:04] Change merged: Dzahn; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/65265 [13:20:37] PROBLEM - Host labs-ns1.wikimedia.org is DOWN: CRITICAL - Host Unreachable (208.80.154.19) [13:20:57] PROBLEM - Host virt1000 is DOWN: CRITICAL - Host Unreachable (208.80.154.18) [13:22:07] New patchset: Ottomata; "Adding classes to install hive, pig and sqoop. (very simple!)" [operations/puppet/cdh4] (master) - https://gerrit.wikimedia.org/r/65267 [13:22:29] PROBLEM - Host mw72 is DOWN: PING CRITICAL - Packet loss = 100% [13:23:48] RECOVERY - Host mw72 is UP: PING OK - Packet loss = 0%, RTA = 26.88 ms [13:23:48] New patchset: Ottomata; "Adding classes to install hive, pig and sqoop. (very simple!)" [operations/puppet/cdh4] (master) - https://gerrit.wikimedia.org/r/65267 [13:35:14] New patchset: Mark Bergsma; "Hack for getting Text Varnish in Beta Labs" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/65268 [13:35:52] Change merged: Mark Bergsma; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/65268 [13:39:46] hrmm [13:39:54] can the video scaler job runners reboot like normal job runners? [13:40:21] a mw dev may know... [13:40:28] TimStarling or Reedy ^ ? [13:40:48] I think so.. [13:40:50] (i assume it just treats those like normal job runners, and anything transcoding when i reboot has to restart) [13:40:59] i have kernel upgrades to roll out. [13:41:02] should work [13:41:12] cool two 'should be ok' is as good as it gets =] [13:41:13] yeah, I noticed the USN today [13:42:18] !log tmh1 rebooting for kernel updates [13:42:27] Logged the message, RobH [13:47:00] New patchset: Mark Bergsma; "Use the mediawiki primary as the default backend" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/65272 [13:47:40] Change merged: Mark Bergsma; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/65272 [13:49:38] PROBLEM - Host tmh1 is DOWN: PING CRITICAL - Packet loss = 100% [13:50:56] RECOVERY - Host tmh1 is UP: PING OK - Packet loss = 0%, RTA = 26.52 ms [13:51:47] PROBLEM - search indices - check lucene status page on search19 is CRITICAL: HTTP CRITICAL: HTTP/1.1 200 OK - pattern found - 60051 bytes in 0.119 second response time [13:52:33] Change merged: Dzahn; [operations/apache-config] (master) - https://gerrit.wikimedia.org/r/64225 [13:55:50] !log tmh1 back online, tmh2 rebooting for kernel updates [13:55:58] Logged the message, RobH [13:56:36] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:57:26] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.120 second response time [13:59:51] New review: Dzahn; "dzahn@fenari:~$ apache-fast-test be-x-old.url mw1044" [operations/apache-config] (master) - https://gerrit.wikimedia.org/r/64225 [14:02:46] RECOVERY - Apache HTTP on mw1023 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 747 bytes in 0.069 second response time [14:03:13] New patchset: Hashar; "beta: wikidata $wgServer and $wgCanonicalServer" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/65276 [14:03:18] New patchset: Ottomata; "Fixing kafka_broker_host_regex for ganglia view" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/65277 [14:03:27] Change merged: Ottomata; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/65277 [14:04:06] Change merged: jenkins-bot; [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/65276 [14:04:33] New patchset: Asher; "run twemproxy daemon on mw servers" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/65278 [14:04:48] !log dzahn graceful'ed apaches and api servers , add redirect for be-x-old.wikisource [14:04:56] Logged the message, Master [14:05:26] Reedy: http://be-x-old.wikisource.org/ and guess what.. apache-graceful-all also restarted API servers [14:05:50] Woooo [14:06:37] was that a bug? [14:06:45] be-x-old [14:07:05] got it [14:07:53] New patchset: Hashar; "beta: wikidata -> wikidatawiki" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/65280 [14:08:18] New patchset: Faidon; "Make check-raid.py compatible with newer MegaCli" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/65281 [14:08:38] Change merged: Faidon; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/65281 [14:09:24] New patchset: Mark Bergsma; "Install amssq47..62 as text varnish" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/65282 [14:10:13] Change merged: Mark Bergsma; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/65282 [14:10:58] Change merged: jenkins-bot; [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/65280 [14:14:47] New patchset: Asher; "run twemproxy daemon on mw servers" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/65278 [14:15:29] PROBLEM - Host amssq47 is DOWN: PING CRITICAL - Packet loss = 100% [14:18:39] PROBLEM - Puppet freshness on virt4 is CRITICAL: No successful Puppet run in the last 10 hours [14:18:39] PROBLEM - Puppet freshness on virt1 is CRITICAL: No successful Puppet run in the last 10 hours [14:18:39] PROBLEM - Puppet freshness on virt3 is CRITICAL: No successful Puppet run in the last 10 hours [14:20:40] RECOVERY - Host amssq47 is UP: PING OK - Packet loss = 0%, RTA = 89.63 ms [14:21:39] PROBLEM - Puppet freshness on pdf1 is CRITICAL: No successful Puppet run in the last 10 hours [14:21:39] PROBLEM - Puppet freshness on pdf2 is CRITICAL: No successful Puppet run in the last 10 hours [14:22:39] PROBLEM - SSH on amssq47 is CRITICAL: Connection refused [14:31:40] RECOVERY - SSH on amssq47 is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1.1 (protocol 2.0) [14:32:24] New patchset: Mark Bergsma; "Add amssq47 as text varnish server" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/65286 [14:35:43] PROBLEM - NTP on amssq47 is CRITICAL: NTP CRITICAL: No response from NTP server [14:40:08] PROBLEM - mysqld processes on db1001 is CRITICAL: PROCS CRITICAL: 2 processes with command name mysqld [14:42:00] Change merged: Mark Bergsma; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/65286 [14:44:28] RECOVERY - NTP on amssq47 is OK: NTP OK: Offset 0.0009235143661 secs [14:45:57] PROBLEM - Puppet freshness on db45 is CRITICAL: No successful Puppet run in the last 10 hours [14:52:07] RECOVERY - mysqld processes on db1001 is OK: PROCS OK: 1 process with command name mysqld [14:59:38] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:00:27] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.130 second response time [15:18:52] New patchset: Ottomata; "Adding classes to install hive, pig and sqoop. (very simple!)" [operations/puppet/cdh4] (master) - https://gerrit.wikimedia.org/r/65267 [15:27:36] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:30:26] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 1.151 second response time [15:40:31] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:41:21] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.123 second response time [15:57:31] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:58:21] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.124 second response time [16:07:50] RECOVERY - Puppet freshness on amssq55 is OK: puppet ran at Fri May 24 16:07:38 UTC 2013 [16:07:51] RECOVERY - Puppet freshness on amssq57 is OK: puppet ran at Fri May 24 16:07:44 UTC 2013 [16:07:51] RECOVERY - Puppet freshness on knsq24 is OK: puppet ran at Fri May 24 16:07:45 UTC 2013 [16:07:59] RECOVERY - Puppet freshness on knsq28 is OK: puppet ran at Fri May 24 16:07:50 UTC 2013 [16:07:59] PROBLEM - Puppet freshness on knsq24 is CRITICAL: No successful Puppet run in the last 10 hours [16:08:18] RECOVERY - Puppet freshness on knsq16 is OK: puppet ran at Fri May 24 16:08:07 UTC 2013 [16:08:18] RECOVERY - Puppet freshness on amssq53 is OK: puppet ran at Fri May 24 16:08:07 UTC 2013 [16:08:18] RECOVERY - Puppet freshness on knsq18 is OK: puppet ran at Fri May 24 16:08:09 UTC 2013 [16:08:18] PROBLEM - Puppet freshness on knsq18 is CRITICAL: No successful Puppet run in the last 10 hours [16:08:18] PROBLEM - Puppet freshness on knsq16 is CRITICAL: No successful Puppet run in the last 10 hours [16:08:28] RECOVERY - Puppet freshness on amssq48 is OK: puppet ran at Fri May 24 16:08:18 UTC 2013 [16:08:28] RECOVERY - Puppet freshness on amssq47 is OK: puppet ran at Fri May 24 16:08:18 UTC 2013 [16:08:28] PROBLEM - Puppet freshness on knsq28 is CRITICAL: No successful Puppet run in the last 10 hours [16:08:28] RECOVERY - Puppet freshness on ms2 is OK: puppet ran at Fri May 24 16:08:22 UTC 2013 [16:08:28] PROBLEM - Puppet freshness on amssq47 is CRITICAL: No successful Puppet run in the last 10 hours [16:08:29] RECOVERY - Puppet freshness on amssq56 is OK: puppet ran at Fri May 24 16:08:24 UTC 2013 [16:08:29] RECOVERY - Puppet freshness on amssq51 is OK: puppet ran at Fri May 24 16:08:24 UTC 2013 [16:08:39] RECOVERY - Puppet freshness on knsq27 is OK: puppet ran at Fri May 24 16:08:28 UTC 2013 [16:08:39] RECOVERY - Puppet freshness on knsq26 is OK: puppet ran at Fri May 24 16:08:28 UTC 2013 [16:08:39] PROBLEM - Puppet freshness on amssq55 is CRITICAL: No successful Puppet run in the last 10 hours [16:08:39] PROBLEM - Puppet freshness on amssq53 is CRITICAL: No successful Puppet run in the last 10 hours [16:08:39] PROBLEM - Puppet freshness on amssq56 is CRITICAL: No successful Puppet run in the last 10 hours [16:08:40] PROBLEM - Puppet freshness on amssq51 is CRITICAL: No successful Puppet run in the last 10 hours [16:08:40] PROBLEM - Puppet freshness on knsq26 is CRITICAL: No successful Puppet run in the last 10 hours [16:08:41] PROBLEM - Puppet freshness on ms2 is CRITICAL: No successful Puppet run in the last 10 hours [16:08:54] PROBLEM - Puppet freshness on amssq48 is CRITICAL: No successful Puppet run in the last 10 hours [16:08:54] PROBLEM - Puppet freshness on amssq57 is CRITICAL: No successful Puppet run in the last 10 hours [16:08:54] RECOVERY - Puppet freshness on amssq59 is OK: puppet ran at Fri May 24 16:08:45 UTC 2013 [16:08:58] RECOVERY - Puppet freshness on knsq21 is OK: puppet ran at Fri May 24 16:08:56 UTC 2013 [16:09:08] RECOVERY - Puppet freshness on knsq22 is OK: puppet ran at Fri May 24 16:09:00 UTC 2013 [16:09:08] RECOVERY - Puppet freshness on amssq50 is OK: puppet ran at Fri May 24 16:09:00 UTC 2013 [16:09:19] PROBLEM - Puppet freshness on amssq59 is CRITICAL: No successful Puppet run in the last 10 hours [16:09:19] RECOVERY - Puppet freshness on amssq60 is OK: puppet ran at Fri May 24 16:09:10 UTC 2013 [16:09:19] RECOVERY - Puppet freshness on knsq19 is OK: puppet ran at Fri May 24 16:09:12 UTC 2013 [16:09:19] RECOVERY - Puppet freshness on knsq17 is OK: puppet ran at Fri May 24 16:09:12 UTC 2013 [16:09:19] RECOVERY - Puppet freshness on amssq58 is OK: puppet ran at Fri May 24 16:09:12 UTC 2013 [16:09:20] RECOVERY - Puppet freshness on knsq23 is OK: puppet ran at Fri May 24 16:09:15 UTC 2013 [16:09:29] PROBLEM - Puppet freshness on amssq50 is CRITICAL: No successful Puppet run in the last 10 hours [16:09:29] PROBLEM - Puppet freshness on knsq22 is CRITICAL: No successful Puppet run in the last 10 hours [16:09:29] PROBLEM - Puppet freshness on knsq27 is CRITICAL: No successful Puppet run in the last 10 hours [16:09:29] RECOVERY - Puppet freshness on amssq62 is OK: puppet ran at Fri May 24 16:09:27 UTC 2013 [16:09:38] PROBLEM - Puppet freshness on knsq21 is CRITICAL: No successful Puppet run in the last 10 hours [16:09:38] PROBLEM - Puppet freshness on amssq62 is CRITICAL: No successful Puppet run in the last 10 hours [16:09:38] PROBLEM - Puppet freshness on knsq19 is CRITICAL: No successful Puppet run in the last 10 hours [16:09:38] PROBLEM - Puppet freshness on knsq23 is CRITICAL: No successful Puppet run in the last 10 hours [16:09:38] PROBLEM - Puppet freshness on knsq17 is CRITICAL: No successful Puppet run in the last 10 hours [16:09:39] RECOVERY - Puppet freshness on amssq54 is OK: puppet ran at Fri May 24 16:09:37 UTC 2013 [16:09:39] RECOVERY - Puppet freshness on amssq49 is OK: puppet ran at Fri May 24 16:09:37 UTC 2013 [16:09:49] RECOVERY - Puppet freshness on knsq29 is OK: puppet ran at Fri May 24 16:09:40 UTC 2013 [16:09:49] RECOVERY - Puppet freshness on amssq61 is OK: puppet ran at Fri May 24 16:09:40 UTC 2013 [16:09:58] RECOVERY - Puppet freshness on knsq20 is OK: puppet ran at Fri May 24 16:09:50 UTC 2013 [16:09:58] RECOVERY - Puppet freshness on amssq52 is OK: puppet ran at Fri May 24 16:09:50 UTC 2013 [16:09:58] PROBLEM - Puppet freshness on amssq49 is CRITICAL: No successful Puppet run in the last 10 hours [16:09:58] RECOVERY - Puppet freshness on amssq55 is OK: puppet ran at Fri May 24 16:09:52 UTC 2013 [16:09:58] RECOVERY - Puppet freshness on amssq57 is OK: puppet ran at Fri May 24 16:09:55 UTC 2013 [16:09:59] RECOVERY - Puppet freshness on knsq24 is OK: puppet ran at Fri May 24 16:09:57 UTC 2013 [16:10:08] RECOVERY - Puppet freshness on knsq28 is OK: puppet ran at Fri May 24 16:09:59 UTC 2013 [16:10:09] PROBLEM - Puppet freshness on amssq58 is CRITICAL: No successful Puppet run in the last 10 hours [16:10:09] PROBLEM - Puppet freshness on amssq60 is CRITICAL: No successful Puppet run in the last 10 hours [16:10:28] PROBLEM - Puppet freshness on knsq28 is CRITICAL: No successful Puppet run in the last 10 hours [16:10:28] RECOVERY - Puppet freshness on knsq16 is OK: puppet ran at Fri May 24 16:10:21 UTC 2013 [16:10:28] RECOVERY - Puppet freshness on amssq53 is OK: puppet ran at Fri May 24 16:10:21 UTC 2013 [16:10:28] PROBLEM - Puppet freshness on amssq61 is CRITICAL: No successful Puppet run in the last 10 hours [16:10:28] PROBLEM - Puppet freshness on amssq54 is CRITICAL: No successful Puppet run in the last 10 hours [16:10:29] RECOVERY - Puppet freshness on knsq18 is OK: puppet ran at Fri May 24 16:10:26 UTC 2013 [16:10:38] PROBLEM - Puppet freshness on amssq55 is CRITICAL: No successful Puppet run in the last 10 hours [16:10:38] PROBLEM - Puppet freshness on amssq53 is CRITICAL: No successful Puppet run in the last 10 hours [16:10:38] RECOVERY - Puppet freshness on amssq47 is OK: puppet ran at Fri May 24 16:10:34 UTC 2013 [16:10:38] PROBLEM - Puppet freshness on knsq29 is CRITICAL: No successful Puppet run in the last 10 hours [16:10:38] RECOVERY - Puppet freshness on amssq48 is OK: puppet ran at Fri May 24 16:10:35 UTC 2013 [16:10:39] PROBLEM - Puppet freshness on amssq52 is CRITICAL: No successful Puppet run in the last 10 hours [16:10:39] PROBLEM - Puppet freshness on knsq20 is CRITICAL: No successful Puppet run in the last 10 hours [16:10:49] RECOVERY - Puppet freshness on ms2 is OK: puppet ran at Fri May 24 16:10:38 UTC 2013 [16:10:49] PROBLEM - Puppet freshness on amssq48 is CRITICAL: No successful Puppet run in the last 10 hours [16:10:49] PROBLEM - Puppet freshness on amssq57 is CRITICAL: No successful Puppet run in the last 10 hours [16:10:49] RECOVERY - Puppet freshness on amssq56 is OK: puppet ran at Fri May 24 16:10:40 UTC 2013 [16:10:49] RECOVERY - Puppet freshness on knsq27 is OK: puppet ran at Fri May 24 16:10:45 UTC 2013 [16:10:49] RECOVERY - Puppet freshness on knsq26 is OK: puppet ran at Fri May 24 16:10:45 UTC 2013 [16:10:49] RECOVERY - Puppet freshness on amssq51 is OK: puppet ran at Fri May 24 16:10:45 UTC 2013 [16:10:58] PROBLEM - Puppet freshness on knsq24 is CRITICAL: No successful Puppet run in the last 10 hours [16:11:09] PROBLEM - Puppet freshness on knsq16 is CRITICAL: No successful Puppet run in the last 10 hours [16:11:19] RECOVERY - Puppet freshness on amssq59 is OK: puppet ran at Fri May 24 16:11:07 UTC 2013 [16:11:19] PROBLEM - Puppet freshness on amssq59 is CRITICAL: No successful Puppet run in the last 10 hours [16:11:19] PROBLEM - Puppet freshness on knsq18 is CRITICAL: No successful Puppet run in the last 10 hours [16:11:20] RECOVERY - Puppet freshness on knsq21 is OK: puppet ran at Fri May 24 16:11:15 UTC 2013 [16:11:28] PROBLEM - Puppet freshness on amssq47 is CRITICAL: No successful Puppet run in the last 10 hours [16:11:28] RECOVERY - Puppet freshness on amssq50 is OK: puppet ran at Fri May 24 16:11:22 UTC 2013 [16:11:28] RECOVERY - Puppet freshness on knsq22 is OK: puppet ran at Fri May 24 16:11:22 UTC 2013 [16:11:28] PROBLEM - Puppet freshness on amssq50 is CRITICAL: No successful Puppet run in the last 10 hours [16:11:28] PROBLEM - Puppet freshness on knsq22 is CRITICAL: No successful Puppet run in the last 10 hours [16:11:29] PROBLEM - Puppet freshness on knsq27 is CRITICAL: No successful Puppet run in the last 10 hours [16:11:39] RECOVERY - Puppet freshness on amssq60 is OK: puppet ran at Fri May 24 16:11:28 UTC 2013 [16:11:39] PROBLEM - Puppet freshness on knsq26 is CRITICAL: No successful Puppet run in the last 10 hours [16:11:39] PROBLEM - Puppet freshness on knsq21 is CRITICAL: No successful Puppet run in the last 10 hours [16:11:39] RECOVERY - Puppet freshness on knsq19 is OK: puppet ran at Fri May 24 16:11:31 UTC 2013 [16:11:39] RECOVERY - Puppet freshness on amssq58 is OK: puppet ran at Fri May 24 16:11:31 UTC 2013 [16:11:40] RECOVERY - Puppet freshness on knsq17 is OK: puppet ran at Fri May 24 16:11:31 UTC 2013 [16:11:40] PROBLEM - Puppet freshness on amssq56 is CRITICAL: No successful Puppet run in the last 10 hours [16:11:41] PROBLEM - Puppet freshness on amssq51 is CRITICAL: No successful Puppet run in the last 10 hours [16:11:41] PROBLEM - Puppet freshness on knsq19 is CRITICAL: No successful Puppet run in the last 10 hours [16:11:42] PROBLEM - Puppet freshness on knsq17 is CRITICAL: No successful Puppet run in the last 10 hours [16:11:50] PROBLEM - Puppet freshness on ms2 is CRITICAL: No successful Puppet run in the last 10 hours [16:11:50] RECOVERY - Puppet freshness on knsq23 is OK: puppet ran at Fri May 24 16:11:40 UTC 2013 [16:11:58] RECOVERY - Puppet freshness on amssq62 is OK: puppet ran at Fri May 24 16:11:57 UTC 2013 [16:12:08] PROBLEM - Puppet freshness on ms-fe3001 is CRITICAL: No successful Puppet run in the last 10 hours [16:12:08] PROBLEM - Puppet freshness on amssq58 is CRITICAL: No successful Puppet run in the last 10 hours [16:12:08] PROBLEM - Puppet freshness on amssq60 is CRITICAL: No successful Puppet run in the last 10 hours [16:12:08] RECOVERY - Puppet freshness on amssq54 is OK: puppet ran at Fri May 24 16:12:05 UTC 2013 [16:12:08] RECOVERY - Puppet freshness on amssq49 is OK: puppet ran at Fri May 24 16:12:05 UTC 2013 [16:12:19] RECOVERY - Puppet freshness on amssq61 is OK: puppet ran at Fri May 24 16:12:08 UTC 2013 [16:12:19] RECOVERY - Puppet freshness on knsq29 is OK: puppet ran at Fri May 24 16:12:08 UTC 2013 [16:12:19] RECOVERY - Puppet freshness on amssq52 is OK: puppet ran at Fri May 24 16:12:12 UTC 2013 [16:12:19] RECOVERY - Puppet freshness on amssq55 is OK: puppet ran at Fri May 24 16:12:15 UTC 2013 [16:12:29] RECOVERY - Puppet freshness on amssq57 is OK: puppet ran at Fri May 24 16:12:21 UTC 2013 [16:12:30] PROBLEM - Puppet freshness on amssq61 is CRITICAL: No successful Puppet run in the last 10 hours [16:12:30] PROBLEM - Puppet freshness on amssq54 is CRITICAL: No successful Puppet run in the last 10 hours [16:12:30] RECOVERY - Puppet freshness on knsq20 is OK: puppet ran at Fri May 24 16:12:26 UTC 2013 [16:12:39] RECOVERY - Puppet freshness on knsq24 is OK: puppet ran at Fri May 24 16:12:27 UTC 2013 [16:12:39] PROBLEM - Puppet freshness on amssq55 is CRITICAL: No successful Puppet run in the last 10 hours [16:12:39] RECOVERY - Puppet freshness on knsq28 is OK: puppet ran at Fri May 24 16:12:34 UTC 2013 [16:12:39] PROBLEM - Puppet freshness on amssq62 is CRITICAL: No successful Puppet run in the last 10 hours [16:12:39] PROBLEM - Puppet freshness on knsq29 is CRITICAL: No successful Puppet run in the last 10 hours [16:12:39] PROBLEM - Puppet freshness on knsq23 is CRITICAL: No successful Puppet run in the last 10 hours [16:12:39] PROBLEM - Puppet freshness on amssq52 is CRITICAL: No successful Puppet run in the last 10 hours [16:12:49] PROBLEM - Puppet freshness on knsq20 is CRITICAL: No successful Puppet run in the last 10 hours [16:12:49] PROBLEM - Puppet freshness on amssq57 is CRITICAL: No successful Puppet run in the last 10 hours [16:12:49] RECOVERY - Puppet freshness on amssq53 is OK: puppet ran at Fri May 24 16:12:43 UTC 2013 [16:12:49] RECOVERY - Puppet freshness on knsq16 is OK: puppet ran at Fri May 24 16:12:44 UTC 2013 [16:12:58] RECOVERY - Puppet freshness on knsq18 is OK: puppet ran at Fri May 24 16:12:49 UTC 2013 [16:12:58] PROBLEM - Puppet freshness on amssq49 is CRITICAL: No successful Puppet run in the last 10 hours [16:12:58] PROBLEM - Puppet freshness on knsq24 is CRITICAL: No successful Puppet run in the last 10 hours [16:12:58] RECOVERY - Puppet freshness on amssq48 is OK: puppet ran at Fri May 24 16:12:54 UTC 2013 [16:12:58] RECOVERY - Puppet freshness on amssq47 is OK: puppet ran at Fri May 24 16:12:54 UTC 2013 [16:12:59] RECOVERY - Puppet freshness on ms2 is OK: puppet ran at Fri May 24 16:12:57 UTC 2013 [16:13:10] RECOVERY - Puppet freshness on amssq56 is OK: puppet ran at Fri May 24 16:12:59 UTC 2013 [16:13:10] RECOVERY - Puppet freshness on amssq51 is OK: puppet ran at Fri May 24 16:12:59 UTC 2013 [16:13:10] RECOVERY - Puppet freshness on knsq26 is OK: puppet ran at Fri May 24 16:13:07 UTC 2013 [16:13:11] RECOVERY - Puppet freshness on knsq27 is OK: puppet ran at Fri May 24 16:13:07 UTC 2013 [16:13:11] PROBLEM - Puppet freshness on knsq16 is CRITICAL: No successful Puppet run in the last 10 hours [16:13:19] PROBLEM - Puppet freshness on knsq18 is CRITICAL: No successful Puppet run in the last 10 hours [16:13:28] PROBLEM - Puppet freshness on amssq47 is CRITICAL: No successful Puppet run in the last 10 hours [16:13:28] PROBLEM - Puppet freshness on knsq28 is CRITICAL: No successful Puppet run in the last 10 hours [16:13:28] PROBLEM - Puppet freshness on knsq27 is CRITICAL: No successful Puppet run in the last 10 hours [16:13:28] RECOVERY - Puppet freshness on amssq59 is OK: puppet ran at Fri May 24 16:13:27 UTC 2013 [16:13:38] PROBLEM - Puppet freshness on knsq26 is CRITICAL: No successful Puppet run in the last 10 hours [16:13:38] RECOVERY - Puppet freshness on knsq21 is OK: puppet ran at Fri May 24 16:13:33 UTC 2013 [16:13:38] RECOVERY - Puppet freshness on amssq50 is OK: puppet ran at Fri May 24 16:13:35 UTC 2013 [16:13:38] RECOVERY - Puppet freshness on knsq22 is OK: puppet ran at Fri May 24 16:13:36 UTC 2013 [16:13:38] PROBLEM - Puppet freshness on amssq53 is CRITICAL: No successful Puppet run in the last 10 hours [16:13:39] PROBLEM - Puppet freshness on amssq56 is CRITICAL: No successful Puppet run in the last 10 hours [16:13:39] PROBLEM - Puppet freshness on amssq51 is CRITICAL: No successful Puppet run in the last 10 hours [16:13:48] PROBLEM - Puppet freshness on ms2 is CRITICAL: No successful Puppet run in the last 10 hours [16:13:48] PROBLEM - Puppet freshness on amssq48 is CRITICAL: No successful Puppet run in the last 10 hours [16:13:48] RECOVERY - Puppet freshness on amssq60 is OK: puppet ran at Fri May 24 16:13:41 UTC 2013 [16:13:48] RECOVERY - Puppet freshness on knsq19 is OK: puppet ran at Fri May 24 16:13:43 UTC 2013 [16:13:49] RECOVERY - Puppet freshness on knsq17 is OK: puppet ran at Fri May 24 16:13:44 UTC 2013 [16:13:58] RECOVERY - Puppet freshness on knsq23 is OK: puppet ran at Fri May 24 16:13:52 UTC 2013 [16:13:58] RECOVERY - Puppet freshness on amssq58 is OK: puppet ran at Fri May 24 16:13:57 UTC 2013 [16:14:08] PROBLEM - Puppet freshness on amssq58 is CRITICAL: No successful Puppet run in the last 10 hours [16:14:08] RECOVERY - Puppet freshness on amssq62 is OK: puppet ran at Fri May 24 16:14:03 UTC 2013 [16:14:08] PROBLEM - Puppet freshness on amssq60 is CRITICAL: No successful Puppet run in the last 10 hours [16:14:08] PROBLEM - Puppet freshness on amssq59 is CRITICAL: No successful Puppet run in the last 10 hours [16:14:22] RECOVERY - Puppet freshness on amssq54 is OK: puppet ran at Fri May 24 16:14:10 UTC 2013 [16:14:22] RECOVERY - Puppet freshness on amssq49 is OK: puppet ran at Fri May 24 16:14:10 UTC 2013 [16:14:22] RECOVERY - Puppet freshness on amssq61 is OK: puppet ran at Fri May 24 16:14:13 UTC 2013 [16:14:22] RECOVERY - Puppet freshness on knsq29 is OK: puppet ran at Fri May 24 16:14:13 UTC 2013 [16:14:28] RECOVERY - Puppet freshness on amssq52 is OK: puppet ran at Fri May 24 16:14:18 UTC 2013 [16:14:28] RECOVERY - Puppet freshness on knsq20 is OK: puppet ran at Fri May 24 16:14:18 UTC 2013 [16:14:28] RECOVERY - Puppet freshness on amssq55 is OK: puppet ran at Fri May 24 16:14:21 UTC 2013 [16:14:28] PROBLEM - Puppet freshness on amssq50 is CRITICAL: No successful Puppet run in the last 10 hours [16:14:28] PROBLEM - Puppet freshness on knsq22 is CRITICAL: No successful Puppet run in the last 10 hours [16:14:29] PROBLEM - Puppet freshness on amssq54 is CRITICAL: No successful Puppet run in the last 10 hours [16:14:29] PROBLEM - Puppet freshness on amssq61 is CRITICAL: No successful Puppet run in the last 10 hours [16:14:30] RECOVERY - Puppet freshness on amssq57 is OK: puppet ran at Fri May 24 16:14:25 UTC 2013 [16:14:30] RECOVERY - Puppet freshness on knsq24 is OK: puppet ran at Fri May 24 16:14:26 UTC 2013 [16:14:38] RECOVERY - Puppet freshness on knsq28 is OK: puppet ran at Fri May 24 16:14:28 UTC 2013 [16:14:38] PROBLEM - Puppet freshness on amssq55 is CRITICAL: No successful Puppet run in the last 10 hours [16:14:38] PROBLEM - Puppet freshness on knsq21 is CRITICAL: No successful Puppet run in the last 10 hours [16:14:38] PROBLEM - Puppet freshness on knsq19 is CRITICAL: No successful Puppet run in the last 10 hours [16:14:38] PROBLEM - Puppet freshness on knsq17 is CRITICAL: No successful Puppet run in the last 10 hours [16:14:39] RECOVERY - Puppet freshness on amssq53 is OK: puppet ran at Fri May 24 16:14:36 UTC 2013 [16:14:39] PROBLEM - Puppet freshness on amssq53 is CRITICAL: No successful Puppet run in the last 10 hours [16:14:40] PROBLEM - Puppet freshness on amssq62 is CRITICAL: No successful Puppet run in the last 10 hours [16:14:40] PROBLEM - Puppet freshness on knsq29 is CRITICAL: No successful Puppet run in the last 10 hours [16:14:41] PROBLEM - Puppet freshness on knsq23 is CRITICAL: No successful Puppet run in the last 10 hours [16:14:49] RECOVERY - Puppet freshness on knsq16 is OK: puppet ran at Fri May 24 16:14:36 UTC 2013 [16:14:49] PROBLEM - Puppet freshness on amssq52 is CRITICAL: No successful Puppet run in the last 10 hours [16:14:49] PROBLEM - Puppet freshness on knsq20 is CRITICAL: No successful Puppet run in the last 10 hours [16:14:49] PROBLEM - Puppet freshness on amssq57 is CRITICAL: No successful Puppet run in the last 10 hours [16:14:50] RECOVERY - Puppet freshness on knsq18 is OK: puppet ran at Fri May 24 16:14:47 UTC 2013 [16:14:58] PROBLEM - Puppet freshness on amssq49 is CRITICAL: No successful Puppet run in the last 10 hours [16:14:58] PROBLEM - Puppet freshness on knsq24 is CRITICAL: No successful Puppet run in the last 10 hours [16:14:58] RECOVERY - Puppet freshness on amssq48 is OK: puppet ran at Fri May 24 16:14:56 UTC 2013 [16:15:08] RECOVERY - Puppet freshness on ms2 is OK: puppet ran at Fri May 24 16:14:58 UTC 2013 [16:15:08] RECOVERY - Puppet freshness on amssq56 is OK: puppet ran at Fri May 24 16:15:00 UTC 2013 [16:15:08] RECOVERY - Puppet freshness on amssq51 is OK: puppet ran at Fri May 24 16:15:00 UTC 2013 [16:15:08] RECOVERY - Puppet freshness on knsq27 is OK: puppet ran at Fri May 24 16:15:07 UTC 2013 [16:15:08] PROBLEM - Puppet freshness on knsq16 is CRITICAL: No successful Puppet run in the last 10 hours [16:15:18] RECOVERY - Puppet freshness on knsq26 is OK: puppet ran at Fri May 24 16:15:10 UTC 2013 [16:15:18] PROBLEM - Puppet freshness on knsq18 is CRITICAL: No successful Puppet run in the last 10 hours [16:15:18] RECOVERY - Puppet freshness on amssq59 is OK: puppet ran at Fri May 24 16:15:11 UTC 2013 [16:15:18] RECOVERY - Puppet freshness on knsq21 is OK: puppet ran at Fri May 24 16:15:15 UTC 2013 [16:15:28] PROBLEM - Puppet freshness on knsq28 is CRITICAL: No successful Puppet run in the last 10 hours [16:15:28] RECOVERY - Puppet freshness on knsq22 is OK: puppet ran at Fri May 24 16:15:22 UTC 2013 [16:15:28] RECOVERY - Puppet freshness on amssq50 is OK: puppet ran at Fri May 24 16:15:22 UTC 2013 [16:15:28] PROBLEM - Puppet freshness on amssq50 is CRITICAL: No successful Puppet run in the last 10 hours [16:15:28] PROBLEM - Puppet freshness on knsq22 is CRITICAL: No successful Puppet run in the last 10 hours [16:15:29] PROBLEM - Puppet freshness on knsq27 is CRITICAL: No successful Puppet run in the last 10 hours [16:15:29] RECOVERY - Puppet freshness on knsq22 is OK: puppet ran at Fri May 24 16:15:25 UTC 2013 [16:15:30] RECOVERY - Puppet freshness on amssq50 is OK: puppet ran at Fri May 24 16:15:25 UTC 2013 [16:15:38] PROBLEM - Puppet freshness on knsq26 is CRITICAL: No successful Puppet run in the last 10 hours [16:15:38] PROBLEM - Puppet freshness on knsq21 is CRITICAL: No successful Puppet run in the last 10 hours [16:15:38] RECOVERY - Puppet freshness on amssq60 is OK: puppet ran at Fri May 24 16:15:33 UTC 2013 [16:15:38] PROBLEM - Puppet freshness on amssq56 is CRITICAL: No successful Puppet run in the last 10 hours [16:15:38] PROBLEM - Puppet freshness on amssq51 is CRITICAL: No successful Puppet run in the last 10 hours [16:15:49] PROBLEM - Puppet freshness on ms2 is CRITICAL: No successful Puppet run in the last 10 hours [16:15:49] PROBLEM - Puppet freshness on amssq48 is CRITICAL: No successful Puppet run in the last 10 hours [16:15:49] RECOVERY - Puppet freshness on knsq19 is OK: puppet ran at Fri May 24 16:15:42 UTC 2013 [16:15:49] RECOVERY - Puppet freshness on knsq17 is OK: puppet ran at Fri May 24 16:15:42 UTC 2013 [16:15:49] RECOVERY - Puppet freshness on amssq58 is OK: puppet ran at Fri May 24 16:15:42 UTC 2013 [16:15:50] RECOVERY - Puppet freshness on knsq23 is OK: puppet ran at Fri May 24 16:15:45 UTC 2013 [16:15:58] RECOVERY - Puppet freshness on amssq62 is OK: puppet ran at Fri May 24 16:15:50 UTC 2013 [16:15:58] RECOVERY - Puppet freshness on amssq54 is OK: puppet ran at Fri May 24 16:15:54 UTC 2013 [16:15:58] RECOVERY - Puppet freshness on amssq49 is OK: puppet ran at Fri May 24 16:15:54 UTC 2013 [16:15:58] RECOVERY - Puppet freshness on amssq61 is OK: puppet ran at Fri May 24 16:15:55 UTC 2013 [16:15:58] RECOVERY - Puppet freshness on knsq29 is OK: puppet ran at Fri May 24 16:15:55 UTC 2013 [16:16:08] PROBLEM - Puppet freshness on amssq58 is CRITICAL: No successful Puppet run in the last 10 hours [16:16:09] RECOVERY - Puppet freshness on amssq52 is OK: puppet ran at Fri May 24 16:16:00 UTC 2013 [16:16:09] RECOVERY - Puppet freshness on knsq20 is OK: puppet ran at Fri May 24 16:16:00 UTC 2013 [16:16:09] RECOVERY - Puppet freshness on amssq55 is OK: puppet ran at Fri May 24 16:16:01 UTC 2013 [16:16:09] RECOVERY - Puppet freshness on amssq57 is OK: puppet ran at Fri May 24 16:16:03 UTC 2013 [16:16:09] PROBLEM - Puppet freshness on amssq60 is CRITICAL: No successful Puppet run in the last 10 hours [16:16:09] RECOVERY - Puppet freshness on knsq28 is OK: puppet ran at Fri May 24 16:16:07 UTC 2013 [16:16:10] PROBLEM - Puppet freshness on amssq59 is CRITICAL: No successful Puppet run in the last 10 hours [16:16:18] RECOVERY - Puppet freshness on amssq47 is OK: puppet ran at Fri May 24 16:16:12 UTC 2013 [16:16:19] RECOVERY - Puppet freshness on knsq24 is OK: puppet ran at Fri May 24 16:16:12 UTC 2013 [16:16:19] RECOVERY - Puppet freshness on amssq53 is OK: puppet ran at Fri May 24 16:16:13 UTC 2013 [16:16:19] RECOVERY - Puppet freshness on knsq16 is OK: puppet ran at Fri May 24 16:16:16 UTC 2013 [16:16:28] RECOVERY - Puppet freshness on knsq18 is OK: puppet ran at Fri May 24 16:16:18 UTC 2013 [16:16:28] PROBLEM - Puppet freshness on amssq47 is CRITICAL: No successful Puppet run in the last 10 hours [16:16:29] PROBLEM - Puppet freshness on knsq28 is CRITICAL: No successful Puppet run in the last 10 hours [16:16:29] RECOVERY - Puppet freshness on amssq47 is OK: puppet ran at Fri May 24 16:16:22 UTC 2013 [16:16:29] RECOVERY - Puppet freshness on amssq48 is OK: puppet ran at Fri May 24 16:16:22 UTC 2013 [16:16:29] PROBLEM - Puppet freshness on knsq22 is CRITICAL: No successful Puppet run in the last 10 hours [16:16:29] PROBLEM - Puppet freshness on amssq50 is CRITICAL: No successful Puppet run in the last 10 hours [16:16:30] PROBLEM - Puppet freshness on amssq54 is CRITICAL: No successful Puppet run in the last 10 hours [16:16:30] PROBLEM - Puppet freshness on amssq61 is CRITICAL: No successful Puppet run in the last 10 hours [16:16:31] RECOVERY - Puppet freshness on amssq56 is OK: puppet ran at Fri May 24 16:16:26 UTC 2013 [16:16:31] RECOVERY - Puppet freshness on amssq51 is OK: puppet ran at Fri May 24 16:16:26 UTC 2013 [16:16:38] PROBLEM - Puppet freshness on amssq55 is CRITICAL: No successful Puppet run in the last 10 hours [16:16:38] PROBLEM - Puppet freshness on knsq19 is CRITICAL: No successful Puppet run in the last 10 hours [16:16:38] PROBLEM - Puppet freshness on knsq17 is CRITICAL: No successful Puppet run in the last 10 hours [16:16:38] RECOVERY - Puppet freshness on knsq27 is OK: puppet ran at Fri May 24 16:16:32 UTC 2013 [16:16:38] RECOVERY - Puppet freshness on knsq26 is OK: puppet ran at Fri May 24 16:16:32 UTC 2013 [16:16:39] RECOVERY - Puppet freshness on ms2 is OK: puppet ran at Fri May 24 16:16:34 UTC 2013 [16:16:39] RECOVERY - Puppet freshness on amssq59 is OK: puppet ran at Fri May 24 16:16:36 UTC 2013 [16:16:40] PROBLEM - Puppet freshness on amssq53 is CRITICAL: No successful Puppet run in the last 10 hours [16:16:40] PROBLEM - Puppet freshness on amssq56 is CRITICAL: No successful Puppet run in the last 10 hours [16:16:41] PROBLEM - Puppet freshness on amssq62 is CRITICAL: No successful Puppet run in the last 10 hours [16:16:41] PROBLEM - Puppet freshness on amssq51 is CRITICAL: No successful Puppet run in the last 10 hours [16:16:42] PROBLEM - Puppet freshness on knsq29 is CRITICAL: No successful Puppet run in the last 10 hours [16:16:42] PROBLEM - Puppet freshness on knsq23 is CRITICAL: No successful Puppet run in the last 10 hours [16:16:48] RECOVERY - Puppet freshness on knsq21 is OK: puppet ran at Fri May 24 16:16:39 UTC 2013 [16:16:48] PROBLEM - Puppet freshness on amssq52 is CRITICAL: No successful Puppet run in the last 10 hours [16:16:48] PROBLEM - Puppet freshness on knsq20 is CRITICAL: No successful Puppet run in the last 10 hours [16:16:48] PROBLEM - Puppet freshness on ms2 is CRITICAL: No successful Puppet run in the last 10 hours [16:16:48] PROBLEM - Puppet freshness on amssq48 is CRITICAL: No successful Puppet run in the last 10 hours [16:16:49] PROBLEM - Puppet freshness on amssq57 is CRITICAL: No successful Puppet run in the last 10 hours [16:16:49] RECOVERY - Puppet freshness on knsq22 is OK: puppet ran at Fri May 24 16:16:40 UTC 2013 [16:16:50] RECOVERY - Puppet freshness on amssq50 is OK: puppet ran at Fri May 24 16:16:41 UTC 2013 [16:16:50] RECOVERY - Puppet freshness on amssq60 is OK: puppet ran at Fri May 24 16:16:44 UTC 2013 [16:16:51] RECOVERY - Puppet freshness on knsq17 is OK: puppet ran at Fri May 24 16:16:46 UTC 2013 [16:16:51] RECOVERY - Puppet freshness on knsq19 is OK: puppet ran at Fri May 24 16:16:47 UTC 2013 [16:16:52] RECOVERY - Puppet freshness on amssq58 is OK: puppet ran at Fri May 24 16:16:47 UTC 2013 [16:16:58] RECOVERY - Puppet freshness on knsq23 is OK: puppet ran at Fri May 24 16:16:48 UTC 2013 [16:16:58] PROBLEM - Puppet freshness on amssq49 is CRITICAL: No successful Puppet run in the last 10 hours [16:16:58] PROBLEM - Puppet freshness on knsq24 is CRITICAL: No successful Puppet run in the last 10 hours [16:16:58] RECOVERY - Puppet freshness on amssq62 is OK: puppet ran at Fri May 24 16:16:53 UTC 2013 [16:17:11] RECOVERY - Puppet freshness on amssq54 is OK: puppet ran at Fri May 24 16:16:58 UTC 2013 [16:17:11] PROBLEM - Puppet freshness on amssq58 is CRITICAL: No successful Puppet run in the last 10 hours [16:17:11] RECOVERY - Puppet freshness on knsq29 is OK: puppet ran at Fri May 24 16:17:00 UTC 2013 [16:17:11] RECOVERY - Puppet freshness on amssq61 is OK: puppet ran at Fri May 24 16:17:00 UTC 2013 [16:17:11] RECOVERY - Puppet freshness on amssq49 is OK: puppet ran at Fri May 24 16:17:02 UTC 2013 [16:17:11] RECOVERY - Puppet freshness on amssq52 is OK: puppet ran at Fri May 24 16:17:02 UTC 2013 [16:17:11] RECOVERY - Puppet freshness on knsq20 is OK: puppet ran at Fri May 24 16:17:03 UTC 2013 [16:17:12] PROBLEM - Puppet freshness on amssq60 is CRITICAL: No successful Puppet run in the last 10 hours [16:17:12] RECOVERY - Puppet freshness on amssq55 is OK: puppet ran at Fri May 24 16:17:06 UTC 2013 [16:17:13] PROBLEM - Puppet freshness on knsq16 is CRITICAL: No successful Puppet run in the last 10 hours [16:17:13] PROBLEM - Puppet freshness on amssq59 is CRITICAL: No successful Puppet run in the last 10 hours [16:17:20] RECOVERY - Puppet freshness on amssq57 is OK: puppet ran at Fri May 24 16:17:08 UTC 2013 [16:17:20] RECOVERY - Puppet freshness on knsq24 is OK: puppet ran at Fri May 24 16:17:10 UTC 2013 [16:17:20] PROBLEM - Puppet freshness on knsq18 is CRITICAL: No successful Puppet run in the last 10 hours [16:17:20] RECOVERY - Puppet freshness on knsq28 is OK: puppet ran at Fri May 24 16:17:12 UTC 2013 [16:17:20] RECOVERY - Puppet freshness on knsq16 is OK: puppet ran at Fri May 24 16:17:16 UTC 2013 [16:17:21] RECOVERY - Puppet freshness on amssq53 is OK: puppet ran at Fri May 24 16:17:16 UTC 2013 [16:17:28] RECOVERY - Puppet freshness on knsq18 is OK: puppet ran at Fri May 24 16:17:20 UTC 2013 [16:17:28] PROBLEM - Puppet freshness on amssq47 is CRITICAL: No successful Puppet run in the last 10 hours [16:17:28] PROBLEM - Puppet freshness on knsq28 is CRITICAL: No successful Puppet run in the last 10 hours [16:17:28] PROBLEM - Puppet freshness on knsq22 is CRITICAL: No successful Puppet run in the last 10 hours [16:17:28] PROBLEM - Puppet freshness on amssq50 is CRITICAL: No successful Puppet run in the last 10 hours [16:17:29] PROBLEM - Puppet freshness on amssq54 is CRITICAL: No successful Puppet run in the last 10 hours [16:17:29] PROBLEM - Puppet freshness on amssq61 is CRITICAL: No successful Puppet run in the last 10 hours [16:17:30] PROBLEM - Puppet freshness on knsq27 is CRITICAL: No successful Puppet run in the last 10 hours [16:17:30] RECOVERY - Puppet freshness on amssq48 is OK: puppet ran at Fri May 24 16:17:23 UTC 2013 [16:17:31] RECOVERY - Puppet freshness on amssq47 is OK: puppet ran at Fri May 24 16:17:23 UTC 2013 [16:17:31] RECOVERY - Puppet freshness on amssq56 is OK: puppet ran at Fri May 24 16:17:25 UTC 2013 [16:17:32] RECOVERY - Puppet freshness on amssq51 is OK: puppet ran at Fri May 24 16:17:25 UTC 2013 [16:17:32] RECOVERY - Puppet freshness on ms2 is OK: puppet ran at Fri May 24 16:17:25 UTC 2013 [16:17:39] RECOVERY - Puppet freshness on knsq27 is OK: puppet ran at Fri May 24 16:17:28 UTC 2013 [16:17:39] PROBLEM - Puppet freshness on amssq55 is CRITICAL: No successful Puppet run in the last 10 hours [16:17:39] PROBLEM - Puppet freshness on knsq26 is CRITICAL: No successful Puppet run in the last 10 hours [16:17:39] PROBLEM - Puppet freshness on knsq21 is CRITICAL: No successful Puppet run in the last 10 hours [16:17:39] RECOVERY - Puppet freshness on amssq59 is OK: puppet ran at Fri May 24 16:17:31 UTC 2013 [16:17:39] PROBLEM - Puppet freshness on knsq19 is CRITICAL: No successful Puppet run in the last 10 hours [16:17:39] PROBLEM - Puppet freshness on knsq17 is CRITICAL: No successful Puppet run in the last 10 hours [16:17:40] RECOVERY - Puppet freshness on knsq21 is OK: puppet ran at Fri May 24 16:17:33 UTC 2013 [16:17:40] PROBLEM - Puppet freshness on amssq53 is CRITICAL: No successful Puppet run in the last 10 hours [16:17:41] RECOVERY - Puppet freshness on amssq50 is OK: puppet ran at Fri May 24 16:17:37 UTC 2013 [16:17:41] RECOVERY - Puppet freshness on knsq22 is OK: puppet ran at Fri May 24 16:17:37 UTC 2013 [16:17:42] PROBLEM - Puppet freshness on amssq56 is CRITICAL: No successful Puppet run in the last 10 hours [16:17:42] PROBLEM - Puppet freshness on amssq62 is CRITICAL: No successful Puppet run in the last 10 hours [16:17:43] PROBLEM - Puppet freshness on amssq51 is CRITICAL: No successful Puppet run in the last 10 hours [16:17:43] PROBLEM - Puppet freshness on knsq29 is CRITICAL: No successful Puppet run in the last 10 hours [16:17:44] PROBLEM - Puppet freshness on knsq23 is CRITICAL: No successful Puppet run in the last 10 hours [16:17:51] PROBLEM - Puppet freshness on amssq52 is CRITICAL: No successful Puppet run in the last 10 hours [16:17:51] PROBLEM - Puppet freshness on knsq20 is CRITICAL: No successful Puppet run in the last 10 hours [16:17:52] PROBLEM - Puppet freshness on ms2 is CRITICAL: No successful Puppet run in the last 10 hours [16:17:52] PROBLEM - Puppet freshness on amssq48 is CRITICAL: No successful Puppet run in the last 10 hours [16:17:52] PROBLEM - Puppet freshness on amssq57 is CRITICAL: No successful Puppet run in the last 10 hours [16:17:52] RECOVERY - Puppet freshness on amssq60 is OK: puppet ran at Fri May 24 16:17:42 UTC 2013 [16:17:52] RECOVERY - Puppet freshness on knsq17 is OK: puppet ran at Fri May 24 16:17:44 UTC 2013 [16:17:52] RECOVERY - Puppet freshness on knsq19 is OK: puppet ran at Fri May 24 16:17:44 UTC 2013 [16:17:53] RECOVERY - Puppet freshness on amssq58 is OK: puppet ran at Fri May 24 16:17:44 UTC 2013 [16:17:53] RECOVERY - Puppet freshness on knsq23 is OK: puppet ran at Fri May 24 16:17:46 UTC 2013 [16:17:58] RECOVERY - Puppet freshness on amssq62 is OK: puppet ran at Fri May 24 16:17:48 UTC 2013 [16:17:58] PROBLEM - Puppet freshness on amssq49 is CRITICAL: No successful Puppet run in the last 10 hours [16:17:58] PROBLEM - Puppet freshness on knsq24 is CRITICAL: No successful Puppet run in the last 10 hours [16:17:58] RECOVERY - Puppet freshness on amssq49 is OK: puppet ran at Fri May 24 16:17:54 UTC 2013 [16:17:58] RECOVERY - Puppet freshness on amssq54 is OK: puppet ran at Fri May 24 16:17:54 UTC 2013 [16:17:59] RECOVERY - Puppet freshness on amssq61 is OK: puppet ran at Fri May 24 16:17:57 UTC 2013 [16:17:59] RECOVERY - Puppet freshness on knsq29 is OK: puppet ran at Fri May 24 16:17:57 UTC 2013 [16:18:08] PROBLEM - Puppet freshness on amssq58 is CRITICAL: No successful Puppet run in the last 10 hours [16:18:08] RECOVERY - Puppet freshness on knsq20 is OK: puppet ran at Fri May 24 16:18:01 UTC 2013 [16:18:08] RECOVERY - Puppet freshness on amssq52 is OK: puppet ran at Fri May 24 16:18:01 UTC 2013 [16:18:08] RECOVERY - Puppet freshness on amssq55 is OK: puppet ran at Fri May 24 16:18:02 UTC 2013 [16:18:08] RECOVERY - Puppet freshness on amssq57 is OK: puppet ran at Fri May 24 16:18:03 UTC 2013 [16:18:09] PROBLEM - Puppet freshness on amssq60 is CRITICAL: No successful Puppet run in the last 10 hours [16:18:09] RECOVERY - Puppet freshness on knsq28 is OK: puppet ran at Fri May 24 16:18:05 UTC 2013 [16:18:10] RECOVERY - Puppet freshness on knsq24 is OK: puppet ran at Fri May 24 16:18:05 UTC 2013 [16:18:10] PROBLEM - Puppet freshness on amssq59 is CRITICAL: No successful Puppet run in the last 10 hours [16:18:11] PROBLEM - Puppet freshness on knsq16 is CRITICAL: No successful Puppet run in the last 10 hours [16:18:18] RECOVERY - Puppet freshness on knsq16 is OK: puppet ran at Fri May 24 16:18:07 UTC 2013 [16:18:18] RECOVERY - Puppet freshness on amssq53 is OK: puppet ran at Fri May 24 16:18:08 UTC 2013 [16:18:18] RECOVERY - Puppet freshness on amssq48 is OK: puppet ran at Fri May 24 16:18:11 UTC 2013 [16:18:18] PROBLEM - Puppet freshness on knsq18 is CRITICAL: No successful Puppet run in the last 10 hours [16:18:18] RECOVERY - Puppet freshness on ms2 is OK: puppet ran at Fri May 24 16:18:12 UTC 2013 [16:18:19] RECOVERY - Puppet freshness on amssq51 is OK: puppet ran at Fri May 24 16:18:13 UTC 2013 [16:18:19] RECOVERY - Puppet freshness on amssq56 is OK: puppet ran at Fri May 24 16:18:13 UTC 2013 [16:18:20] RECOVERY - Puppet freshness on knsq26 is OK: puppet ran at Fri May 24 16:18:15 UTC 2013 [16:18:29] RECOVERY - Puppet freshness on amssq59 is OK: puppet ran at Fri May 24 16:18:20 UTC 2013 [16:18:29] PROBLEM - Puppet freshness on amssq47 is CRITICAL: No successful Puppet run in the last 10 hours [16:18:29] PROBLEM - Puppet freshness on knsq28 is CRITICAL: No successful Puppet run in the last 10 hours [16:18:29] PROBLEM - Puppet freshness on knsq22 is CRITICAL: No successful Puppet run in the last 10 hours [16:18:29] PROBLEM - Puppet freshness on amssq50 is CRITICAL: No successful Puppet run in the last 10 hours [16:18:29] PROBLEM - Puppet freshness on amssq54 is CRITICAL: No successful Puppet run in the last 10 hours [16:18:29] PROBLEM - Puppet freshness on knsq27 is CRITICAL: No successful Puppet run in the last 10 hours [16:18:30] PROBLEM - Puppet freshness on amssq61 is CRITICAL: No successful Puppet run in the last 10 hours [16:18:30] RECOVERY - Puppet freshness on amssq60 is OK: puppet ran at Fri May 24 16:18:25 UTC 2013 [16:18:38] RECOVERY - Puppet freshness on amssq58 is OK: puppet ran at Fri May 24 16:18:28 UTC 2013 [16:18:38] PROBLEM - Puppet freshness on amssq55 is CRITICAL: No successful Puppet run in the last 10 hours [16:18:38] PROBLEM - Puppet freshness on knsq26 is CRITICAL: No successful Puppet run in the last 10 hours [16:18:38] PROBLEM - Puppet freshness on knsq21 is CRITICAL: No successful Puppet run in the last 10 hours [16:18:38] PROBLEM - Puppet freshness on knsq19 is CRITICAL: No successful Puppet run in the last 10 hours [16:18:39] PROBLEM - Puppet freshness on knsq17 is CRITICAL: No successful Puppet run in the last 10 hours [16:18:39] RECOVERY - Puppet freshness on amssq54 is OK: puppet ran at Fri May 24 16:18:36 UTC 2013 [16:18:40] PROBLEM - Puppet freshness on amssq53 is CRITICAL: No successful Puppet run in the last 10 hours [16:18:40] RECOVERY - Puppet freshness on amssq61 is OK: puppet ran at Fri May 24 16:18:37 UTC 2013 [16:18:41] PROBLEM - Puppet freshness on amssq56 is CRITICAL: No successful Puppet run in the last 10 hours [16:18:41] PROBLEM - Puppet freshness on amssq62 is CRITICAL: No successful Puppet run in the last 10 hours [16:18:42] PROBLEM - Puppet freshness on amssq51 is CRITICAL: No successful Puppet run in the last 10 hours [16:18:42] PROBLEM - Puppet freshness on knsq29 is CRITICAL: No successful Puppet run in the last 10 hours [16:18:43] PROBLEM - Puppet freshness on knsq23 is CRITICAL: No successful Puppet run in the last 10 hours [16:18:48] RECOVERY - Puppet freshness on knsq29 is OK: puppet ran at Fri May 24 16:18:38 UTC 2013 [16:18:48] PROBLEM - Puppet freshness on amssq52 is CRITICAL: No successful Puppet run in the last 10 hours [16:18:48] PROBLEM - Puppet freshness on knsq20 is CRITICAL: No successful Puppet run in the last 10 hours [16:18:48] PROBLEM - Puppet freshness on ms2 is CRITICAL: No successful Puppet run in the last 10 hours [16:18:48] PROBLEM - Puppet freshness on amssq48 is CRITICAL: No successful Puppet run in the last 10 hours [16:18:49] PROBLEM - Puppet freshness on amssq57 is CRITICAL: No successful Puppet run in the last 10 hours [16:18:49] RECOVERY - Puppet freshness on amssq55 is OK: puppet ran at Fri May 24 16:18:42 UTC 2013 [16:18:50] RECOVERY - Puppet freshness on amssq57 is OK: puppet ran at Fri May 24 16:18:43 UTC 2013 [16:18:58] RECOVERY - Puppet freshness on knsq28 is OK: puppet ran at Fri May 24 16:18:50 UTC 2013 [16:18:58] RECOVERY - Puppet freshness on amssq53 is OK: puppet ran at Fri May 24 16:18:52 UTC 2013 [16:18:58] PROBLEM - Puppet freshness on amssq49 is CRITICAL: No successful Puppet run in the last 10 hours [16:18:58] PROBLEM - Puppet freshness on knsq24 is CRITICAL: No successful Puppet run in the last 10 hours [16:18:58] RECOVERY - Puppet freshness on knsq18 is OK: puppet ran at Fri May 24 16:18:53 UTC 2013 [16:18:59] RECOVERY - Puppet freshness on amssq48 is OK: puppet ran at Fri May 24 16:18:55 UTC 2013 [16:18:59] RECOVERY - Puppet freshness on amssq47 is OK: puppet ran at Fri May 24 16:18:55 UTC 2013 [16:19:00] RECOVERY - Puppet freshness on ms2 is OK: puppet ran at Fri May 24 16:18:55 UTC 2013 [16:19:00] RECOVERY - Puppet freshness on amssq56 is OK: puppet ran at Fri May 24 16:18:56 UTC 2013 [16:19:01] RECOVERY - Puppet freshness on amssq51 is OK: puppet ran at Fri May 24 16:18:56 UTC 2013 [16:19:01] RECOVERY - Puppet freshness on knsq26 is OK: puppet ran at Fri May 24 16:18:57 UTC 2013 [16:19:08] RECOVERY - Puppet freshness on knsq27 is OK: puppet ran at Fri May 24 16:18:58 UTC 2013 [16:19:08] PROBLEM - Puppet freshness on amssq58 is CRITICAL: No successful Puppet run in the last 10 hours [16:19:08] PROBLEM - Puppet freshness on amssq60 is CRITICAL: No successful Puppet run in the last 10 hours [16:19:08] PROBLEM - Puppet freshness on knsq16 is CRITICAL: No successful Puppet run in the last 10 hours [16:19:08] PROBLEM - Puppet freshness on amssq59 is CRITICAL: No successful Puppet run in the last 10 hours [16:19:18] PROBLEM - Puppet freshness on knsq18 is CRITICAL: No successful Puppet run in the last 10 hours [16:19:28] PROBLEM - Puppet freshness on amssq47 is CRITICAL: No successful Puppet run in the last 10 hours [16:19:28] PROBLEM - Puppet freshness on knsq28 is CRITICAL: No successful Puppet run in the last 10 hours [16:19:28] PROBLEM - Puppet freshness on knsq27 is CRITICAL: No successful Puppet run in the last 10 hours [16:19:29] PROBLEM - Puppet freshness on amssq54 is CRITICAL: No successful Puppet run in the last 10 hours [16:19:29] PROBLEM - Puppet freshness on amssq61 is CRITICAL: No successful Puppet run in the last 10 hours [16:19:38] PROBLEM - Puppet freshness on amssq55 is CRITICAL: No successful Puppet run in the last 10 hours [16:19:38] PROBLEM - Puppet freshness on knsq26 is CRITICAL: No successful Puppet run in the last 10 hours [16:19:38] PROBLEM - Puppet freshness on amssq53 is CRITICAL: No successful Puppet run in the last 10 hours [16:19:38] PROBLEM - Puppet freshness on amssq56 is CRITICAL: No successful Puppet run in the last 10 hours [16:19:38] PROBLEM - Puppet freshness on amssq51 is CRITICAL: No successful Puppet run in the last 10 hours [16:19:39] PROBLEM - Puppet freshness on knsq29 is CRITICAL: No successful Puppet run in the last 10 hours [16:19:48] PROBLEM - Puppet freshness on ms2 is CRITICAL: No successful Puppet run in the last 10 hours [16:19:48] PROBLEM - Puppet freshness on amssq48 is CRITICAL: No successful Puppet run in the last 10 hours [16:19:48] PROBLEM - Puppet freshness on amssq57 is CRITICAL: No successful Puppet run in the last 10 hours [16:20:28] RECOVERY - Puppet freshness on amssq59 is OK: puppet ran at Fri May 24 16:20:26 UTC 2013 [16:21:08] PROBLEM - Puppet freshness on amssq59 is CRITICAL: No successful Puppet run in the last 10 hours [16:23:08] RECOVERY - Puppet freshness on knsq21 is OK: puppet ran at Fri May 24 16:23:00 UTC 2013 [16:23:39] PROBLEM - Puppet freshness on knsq21 is CRITICAL: No successful Puppet run in the last 10 hours [16:24:08] RECOVERY - Puppet freshness on amssq50 is OK: puppet ran at Fri May 24 16:24:00 UTC 2013 [16:24:08] RECOVERY - Puppet freshness on knsq22 is OK: puppet ran at Fri May 24 16:24:05 UTC 2013 [16:24:28] PROBLEM - Puppet freshness on knsq22 is CRITICAL: No successful Puppet run in the last 10 hours [16:24:28] PROBLEM - Puppet freshness on amssq50 is CRITICAL: No successful Puppet run in the last 10 hours [16:25:21] New patchset: Mark Bergsma; "Redirect mobile clients to the mobile sites" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/65306 [16:25:38] RECOVERY - Puppet freshness on amssq60 is OK: puppet ran at Fri May 24 16:25:29 UTC 2013 [16:26:11] PROBLEM - Puppet freshness on amssq60 is CRITICAL: No successful Puppet run in the last 10 hours [16:26:58] RECOVERY - Puppet freshness on knsq17 is OK: puppet ran at Fri May 24 16:26:53 UTC 2013 [16:26:58] RECOVERY - Puppet freshness on amssq58 is OK: puppet ran at Fri May 24 16:26:53 UTC 2013 [16:26:58] RECOVERY - Puppet freshness on knsq19 is OK: puppet ran at Fri May 24 16:26:53 UTC 2013 [16:27:08] PROBLEM - Puppet freshness on amssq58 is CRITICAL: No successful Puppet run in the last 10 hours [16:27:38] PROBLEM - Puppet freshness on knsq19 is CRITICAL: No successful Puppet run in the last 10 hours [16:27:38] PROBLEM - Puppet freshness on knsq17 is CRITICAL: No successful Puppet run in the last 10 hours [16:27:58] RECOVERY - Puppet freshness on knsq23 is OK: puppet ran at Fri May 24 16:27:54 UTC 2013 [16:28:38] PROBLEM - Puppet freshness on knsq23 is CRITICAL: No successful Puppet run in the last 10 hours [16:29:48] RECOVERY - Puppet freshness on amssq62 is OK: puppet ran at Fri May 24 16:29:39 UTC 2013 [16:30:38] PROBLEM - Puppet freshness on amssq62 is CRITICAL: No successful Puppet run in the last 10 hours [16:31:58] RECOVERY - Puppet freshness on amssq54 is OK: puppet ran at Fri May 24 16:31:54 UTC 2013 [16:31:58] RECOVERY - Puppet freshness on amssq49 is OK: puppet ran at Fri May 24 16:31:54 UTC 2013 [16:32:28] PROBLEM - Puppet freshness on amssq54 is CRITICAL: No successful Puppet run in the last 10 hours [16:32:38] RECOVERY - Puppet freshness on amssq61 is OK: puppet ran at Fri May 24 16:32:35 UTC 2013 [16:32:58] PROBLEM - Puppet freshness on amssq49 is CRITICAL: No successful Puppet run in the last 10 hours [16:33:28] PROBLEM - Puppet freshness on amssq61 is CRITICAL: No successful Puppet run in the last 10 hours [16:33:38] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:33:50] RECOVERY - Puppet freshness on knsq29 is OK: puppet ran at Fri May 24 16:33:45 UTC 2013 [16:34:38] PROBLEM - Puppet freshness on knsq29 is CRITICAL: No successful Puppet run in the last 10 hours [16:34:58] RECOVERY - Puppet freshness on amssq52 is OK: puppet ran at Fri May 24 16:34:52 UTC 2013 [16:35:28] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.125 second response time [16:35:38] RECOVERY - Puppet freshness on amssq55 is OK: puppet ran at Fri May 24 16:35:29 UTC 2013 [16:35:39] PROBLEM - Puppet freshness on amssq55 is CRITICAL: No successful Puppet run in the last 10 hours [16:35:48] PROBLEM - Puppet freshness on amssq52 is CRITICAL: No successful Puppet run in the last 10 hours [16:36:38] RECOVERY - Puppet freshness on amssq57 is OK: puppet ran at Fri May 24 16:36:30 UTC 2013 [16:36:48] PROBLEM - Puppet freshness on amssq57 is CRITICAL: No successful Puppet run in the last 10 hours [16:43:37] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:44:25] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.124 second response time [16:49:56] PROBLEM - SSH on amslvs1 is CRITICAL: Server answer: [16:50:55] RECOVERY - SSH on amslvs1 is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1.1 (protocol 2.0) [17:02:26] PROBLEM - Puppet freshness on db44 is CRITICAL: No successful Puppet run in the last 10 hours [18:14:47] PROBLEM - Puppet freshness on erzurumi is CRITICAL: No successful Puppet run in the last 10 hours [19:45:45] New patchset: Andrew Bogott; "Several minor changes to the openstack manifest (take two):" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/65327 [19:47:30] New review: Andrew Bogott; "The previous version of this patch failed on systems that installed some but not all nova roles. Th..." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/65327 [20:07:41] RECOVERY - Puppet freshness on knsq20 is OK: puppet ran at Fri May 24 20:07:34 UTC 2013 [20:07:41] RECOVERY - Puppet freshness on knsq24 is OK: puppet ran at Fri May 24 20:07:34 UTC 2013 [20:07:41] RECOVERY - Puppet freshness on knsq28 is OK: puppet ran at Fri May 24 20:07:37 UTC 2013 [20:07:51] RECOVERY - Puppet freshness on knsq16 is OK: puppet ran at Fri May 24 20:07:43 UTC 2013 [20:07:52] RECOVERY - Puppet freshness on knsq18 is OK: puppet ran at Fri May 24 20:07:45 UTC 2013 [20:07:52] RECOVERY - Puppet freshness on amssq47 is OK: puppet ran at Fri May 24 20:07:49 UTC 2013 [20:07:52] RECOVERY - Puppet freshness on amssq48 is OK: puppet ran at Fri May 24 20:07:50 UTC 2013 [20:07:52] PROBLEM - Puppet freshness on amssq47 is CRITICAL: No successful Puppet run in the last 10 hours [20:08:01] RECOVERY - Puppet freshness on amssq56 is OK: puppet ran at Fri May 24 20:07:54 UTC 2013 [20:08:01] RECOVERY - Puppet freshness on amssq51 is OK: puppet ran at Fri May 24 20:07:54 UTC 2013 [20:08:01] RECOVERY - Puppet freshness on ms2 is OK: puppet ran at Fri May 24 20:07:54 UTC 2013 [20:08:01] RECOVERY - Puppet freshness on knsq26 is OK: puppet ran at Fri May 24 20:08:00 UTC 2013 [20:08:01] RECOVERY - Puppet freshness on knsq27 is OK: puppet ran at Fri May 24 20:08:00 UTC 2013 [20:08:12] RECOVERY - Puppet freshness on amssq59 is OK: puppet ran at Fri May 24 20:08:07 UTC 2013 [20:08:12] PROBLEM - Puppet freshness on knsq18 is CRITICAL: No successful Puppet run in the last 10 hours [20:08:12] PROBLEM - Puppet freshness on knsq24 is CRITICAL: No successful Puppet run in the last 10 hours [20:08:12] PROBLEM - Puppet freshness on amssq51 is CRITICAL: No successful Puppet run in the last 10 hours [20:08:12] PROBLEM - Puppet freshness on amssq59 is CRITICAL: No successful Puppet run in the last 10 hours [20:08:13] PROBLEM - Puppet freshness on knsq26 is CRITICAL: No successful Puppet run in the last 10 hours [20:08:23] RECOVERY - Puppet freshness on knsq21 is OK: puppet ran at Fri May 24 20:08:12 UTC 2013 [20:08:23] RECOVERY - Puppet freshness on knsq22 is OK: puppet ran at Fri May 24 20:08:12 UTC 2013 [20:08:23] RECOVERY - Puppet freshness on amssq50 is OK: puppet ran at Fri May 24 20:08:13 UTC 2013 [20:08:25] PROBLEM - Puppet freshness on knsq16 is CRITICAL: No successful Puppet run in the last 10 hours [20:08:31] RECOVERY - Puppet freshness on amssq60 is OK: puppet ran at Fri May 24 20:08:21 UTC 2013 [20:08:31] PROBLEM - Puppet freshness on knsq20 is CRITICAL: No successful Puppet run in the last 10 hours [20:08:31] RECOVERY - Puppet freshness on knsq19 is OK: puppet ran at Fri May 24 20:08:27 UTC 2013 [20:08:31] RECOVERY - Puppet freshness on knsq17 is OK: puppet ran at Fri May 24 20:08:27 UTC 2013 [20:08:31] RECOVERY - Puppet freshness on amssq58 is OK: puppet ran at Fri May 24 20:08:27 UTC 2013 [20:08:41] RECOVERY - Puppet freshness on knsq23 is OK: puppet ran at Fri May 24 20:08:32 UTC 2013 [20:08:41] PROBLEM - Puppet freshness on amssq58 is CRITICAL: No successful Puppet run in the last 10 hours [20:08:41] PROBLEM - Puppet freshness on knsq28 is CRITICAL: No successful Puppet run in the last 10 hours [20:08:41] RECOVERY - Puppet freshness on amssq62 is OK: puppet ran at Fri May 24 20:08:35 UTC 2013 [20:08:41] PROBLEM - Puppet freshness on knsq22 is CRITICAL: No successful Puppet run in the last 10 hours [20:08:51] PROBLEM - Puppet freshness on amssq48 is CRITICAL: No successful Puppet run in the last 10 hours [20:08:51] RECOVERY - Puppet freshness on amssq54 is OK: puppet ran at Fri May 24 20:08:42 UTC 2013 [20:08:51] RECOVERY - Puppet freshness on amssq49 is OK: puppet ran at Fri May 24 20:08:42 UTC 2013 [20:08:51] PROBLEM - Puppet freshness on knsq21 is CRITICAL: No successful Puppet run in the last 10 hours [20:08:51] PROBLEM - Puppet freshness on knsq17 is CRITICAL: No successful Puppet run in the last 10 hours [20:08:52] PROBLEM - Puppet freshness on knsq19 is CRITICAL: No successful Puppet run in the last 10 hours [20:08:52] PROBLEM - Puppet freshness on ms2 is CRITICAL: No successful Puppet run in the last 10 hours [20:08:53] RECOVERY - Puppet freshness on knsq29 is OK: puppet ran at Fri May 24 20:08:44 UTC 2013 [20:08:53] RECOVERY - Puppet freshness on amssq61 is OK: puppet ran at Fri May 24 20:08:45 UTC 2013 [20:08:54] RECOVERY - Puppet freshness on knsq20 is OK: puppet ran at Fri May 24 20:08:48 UTC 2013 [20:08:54] RECOVERY - Puppet freshness on amssq52 is OK: puppet ran at Fri May 24 20:08:48 UTC 2013 [20:09:01] RECOVERY - Puppet freshness on amssq55 is OK: puppet ran at Fri May 24 20:08:51 UTC 2013 [20:09:01] PROBLEM - Puppet freshness on amssq49 is CRITICAL: No successful Puppet run in the last 10 hours [20:09:01] PROBLEM - Puppet freshness on amssq56 is CRITICAL: No successful Puppet run in the last 10 hours [20:09:01] PROBLEM - Puppet freshness on knsq27 is CRITICAL: No successful Puppet run in the last 10 hours [20:09:01] RECOVERY - Puppet freshness on amssq57 is OK: puppet ran at Fri May 24 20:08:55 UTC 2013 [20:09:02] RECOVERY - Puppet freshness on knsq24 is OK: puppet ran at Fri May 24 20:08:58 UTC 2013 [20:09:02] RECOVERY - Puppet freshness on knsq28 is OK: puppet ran at Fri May 24 20:09:00 UTC 2013 [20:09:11] PROBLEM - Puppet freshness on amssq54 is CRITICAL: No successful Puppet run in the last 10 hours [20:09:11] RECOVERY - Puppet freshness on amssq53 is OK: puppet ran at Fri May 24 20:09:05 UTC 2013 [20:09:12] RECOVERY - Puppet freshness on knsq16 is OK: puppet ran at Fri May 24 20:09:05 UTC 2013 [20:09:12] PROBLEM - Puppet freshness on knsq24 is CRITICAL: No successful Puppet run in the last 10 hours [20:09:12] PROBLEM - Puppet freshness on amssq62 is CRITICAL: No successful Puppet run in the last 10 hours [20:09:12] RECOVERY - Puppet freshness on knsq18 is OK: puppet ran at Fri May 24 20:09:10 UTC 2013 [20:09:21] PROBLEM - Puppet freshness on amssq50 is CRITICAL: No successful Puppet run in the last 10 hours [20:09:21] PROBLEM - Puppet freshness on knsq23 is CRITICAL: No successful Puppet run in the last 10 hours [20:09:21] PROBLEM - Puppet freshness on amssq60 is CRITICAL: No successful Puppet run in the last 10 hours [20:09:21] RECOVERY - Puppet freshness on amssq48 is OK: puppet ran at Fri May 24 20:09:16 UTC 2013 [20:09:21] RECOVERY - Puppet freshness on amssq47 is OK: puppet ran at Fri May 24 20:09:17 UTC 2013 [20:09:22] PROBLEM - Puppet freshness on amssq55 is CRITICAL: No successful Puppet run in the last 10 hours [20:09:22] PROBLEM - Puppet freshness on knsq16 is CRITICAL: No successful Puppet run in the last 10 hours [20:09:23] RECOVERY - Puppet freshness on ms2 is OK: puppet ran at Fri May 24 20:09:19 UTC 2013 [20:09:31] RECOVERY - Puppet freshness on amssq56 is OK: puppet ran at Fri May 24 20:09:21 UTC 2013 [20:09:31] RECOVERY - Puppet freshness on amssq51 is OK: puppet ran at Fri May 24 20:09:21 UTC 2013 [20:09:31] PROBLEM - Puppet freshness on knsq20 is CRITICAL: No successful Puppet run in the last 10 hours [20:09:31] RECOVERY - Puppet freshness on knsq27 is OK: puppet ran at Fri May 24 20:09:26 UTC 2013 [20:09:31] RECOVERY - Puppet freshness on knsq26 is OK: puppet ran at Fri May 24 20:09:26 UTC 2013 [20:09:41] RECOVERY - Puppet freshness on amssq59 is OK: puppet ran at Fri May 24 20:09:31 UTC 2013 [20:09:41] PROBLEM - Puppet freshness on knsq28 is CRITICAL: No successful Puppet run in the last 10 hours [20:09:41] RECOVERY - Puppet freshness on knsq21 is OK: puppet ran at Fri May 24 20:09:38 UTC 2013 [20:09:41] PROBLEM - Puppet freshness on amssq53 is CRITICAL: No successful Puppet run in the last 10 hours [20:09:41] RECOVERY - Puppet freshness on knsq22 is OK: puppet ran at Fri May 24 20:09:40 UTC 2013 [20:09:42] PROBLEM - Puppet freshness on amssq61 is CRITICAL: No successful Puppet run in the last 10 hours [20:09:42] PROBLEM - Puppet freshness on amssq57 is CRITICAL: No successful Puppet run in the last 10 hours [20:09:43] PROBLEM - Puppet freshness on knsq22 is CRITICAL: No successful Puppet run in the last 10 hours [20:09:51] RECOVERY - Puppet freshness on amssq50 is OK: puppet ran at Fri May 24 20:09:40 UTC 2013 [20:09:51] PROBLEM - Puppet freshness on amssq48 is CRITICAL: No successful Puppet run in the last 10 hours [20:09:51] PROBLEM - Puppet freshness on amssq52 is CRITICAL: No successful Puppet run in the last 10 hours [20:09:51] PROBLEM - Puppet freshness on knsq21 is CRITICAL: No successful Puppet run in the last 10 hours [20:09:51] PROBLEM - Puppet freshness on knsq29 is CRITICAL: No successful Puppet run in the last 10 hours [20:09:52] PROBLEM - Puppet freshness on ms2 is CRITICAL: No successful Puppet run in the last 10 hours [20:09:52] RECOVERY - Puppet freshness on amssq60 is OK: puppet ran at Fri May 24 20:09:46 UTC 2013 [20:09:53] RECOVERY - Puppet freshness on knsq19 is OK: puppet ran at Fri May 24 20:09:47 UTC 2013 [20:09:53] RECOVERY - Puppet freshness on amssq58 is OK: puppet ran at Fri May 24 20:09:48 UTC 2013 [20:09:54] PROBLEM - Puppet freshness on amssq47 is CRITICAL: No successful Puppet run in the last 10 hours [20:09:54] RECOVERY - Puppet freshness on knsq23 is OK: puppet ran at Fri May 24 20:09:50 UTC 2013 [20:10:01] PROBLEM - Puppet freshness on amssq56 is CRITICAL: No successful Puppet run in the last 10 hours [20:10:01] PROBLEM - Puppet freshness on knsq27 is CRITICAL: No successful Puppet run in the last 10 hours [20:10:01] RECOVERY - Puppet freshness on amssq62 is OK: puppet ran at Fri May 24 20:09:56 UTC 2013 [20:10:01] RECOVERY - Puppet freshness on knsq17 is OK: puppet ran at Fri May 24 20:09:56 UTC 2013 [20:10:11] RECOVERY - Puppet freshness on amssq49 is OK: puppet ran at Fri May 24 20:10:04 UTC 2013 [20:10:11] RECOVERY - Puppet freshness on amssq54 is OK: puppet ran at Fri May 24 20:10:05 UTC 2013 [20:10:11] RECOVERY - Puppet freshness on knsq29 is OK: puppet ran at Fri May 24 20:10:06 UTC 2013 [20:10:12] RECOVERY - Puppet freshness on amssq61 is OK: puppet ran at Fri May 24 20:10:07 UTC 2013 [20:10:12] PROBLEM - Puppet freshness on amssq59 is CRITICAL: No successful Puppet run in the last 10 hours [20:10:12] PROBLEM - Puppet freshness on knsq18 is CRITICAL: No successful Puppet run in the last 10 hours [20:10:12] PROBLEM - Puppet freshness on amssq51 is CRITICAL: No successful Puppet run in the last 10 hours [20:10:13] PROBLEM - Puppet freshness on amssq62 is CRITICAL: No successful Puppet run in the last 10 hours [20:10:21] PROBLEM - Puppet freshness on knsq26 is CRITICAL: No successful Puppet run in the last 10 hours [20:10:21] RECOVERY - Puppet freshness on knsq20 is OK: puppet ran at Fri May 24 20:10:12 UTC 2013 [20:10:21] PROBLEM - Puppet freshness on amssq50 is CRITICAL: No successful Puppet run in the last 10 hours [20:10:21] PROBLEM - Puppet freshness on knsq23 is CRITICAL: No successful Puppet run in the last 10 hours [20:10:21] RECOVERY - Puppet freshness on amssq52 is OK: puppet ran at Fri May 24 20:10:13 UTC 2013 [20:10:22] PROBLEM - Puppet freshness on amssq60 is CRITICAL: No successful Puppet run in the last 10 hours [20:10:22] RECOVERY - Puppet freshness on amssq55 is OK: puppet ran at Fri May 24 20:10:17 UTC 2013 [20:10:23] RECOVERY - Puppet freshness on amssq57 is OK: puppet ran at Fri May 24 20:10:18 UTC 2013 [20:10:23] PROBLEM - Puppet freshness on amssq55 is CRITICAL: No successful Puppet run in the last 10 hours [20:10:24] RECOVERY - Puppet freshness on knsq24 is OK: puppet ran at Fri May 24 20:10:20 UTC 2013 [20:10:31] RECOVERY - Puppet freshness on knsq28 is OK: puppet ran at Fri May 24 20:10:22 UTC 2013 [20:10:31] RECOVERY - Puppet freshness on knsq16 is OK: puppet ran at Fri May 24 20:10:25 UTC 2013 [20:10:32] RECOVERY - Puppet freshness on amssq53 is OK: puppet ran at Fri May 24 20:10:25 UTC 2013 [20:10:32] PROBLEM - Puppet freshness on knsq20 is CRITICAL: No successful Puppet run in the last 10 hours [20:10:32] RECOVERY - Puppet freshness on knsq18 is OK: puppet ran at Fri May 24 20:10:28 UTC 2013 [20:10:41] PROBLEM - Puppet freshness on amssq58 is CRITICAL: No successful Puppet run in the last 10 hours [20:10:41] PROBLEM - Puppet freshness on knsq28 is CRITICAL: No successful Puppet run in the last 10 hours [20:10:41] RECOVERY - Puppet freshness on amssq48 is OK: puppet ran at Fri May 24 20:10:37 UTC 2013 [20:10:41] PROBLEM - Puppet freshness on amssq53 is CRITICAL: No successful Puppet run in the last 10 hours [20:10:41] RECOVERY - Puppet freshness on amssq47 is OK: puppet ran at Fri May 24 20:10:39 UTC 2013 [20:10:42] PROBLEM - Puppet freshness on amssq61 is CRITICAL: No successful Puppet run in the last 10 hours [20:10:42] PROBLEM - Puppet freshness on amssq57 is CRITICAL: No successful Puppet run in the last 10 hours [20:10:51] PROBLEM - Puppet freshness on amssq48 is CRITICAL: No successful Puppet run in the last 10 hours [20:10:51] PROBLEM - Puppet freshness on amssq52 is CRITICAL: No successful Puppet run in the last 10 hours [20:10:51] RECOVERY - Puppet freshness on ms2 is OK: puppet ran at Fri May 24 20:10:42 UTC 2013 [20:10:51] PROBLEM - Puppet freshness on knsq17 is CRITICAL: No successful Puppet run in the last 10 hours [20:10:51] PROBLEM - Puppet freshness on knsq19 is CRITICAL: No successful Puppet run in the last 10 hours [20:10:52] PROBLEM - Puppet freshness on knsq29 is CRITICAL: No successful Puppet run in the last 10 hours [20:10:52] PROBLEM - Puppet freshness on ms2 is CRITICAL: No successful Puppet run in the last 10 hours [20:10:53] RECOVERY - Puppet freshness on amssq51 is OK: puppet ran at Fri May 24 20:10:43 UTC 2013 [20:10:53] RECOVERY - Puppet freshness on amssq56 is OK: puppet ran at Fri May 24 20:10:43 UTC 2013 [20:10:54] RECOVERY - Puppet freshness on knsq27 is OK: puppet ran at Fri May 24 20:10:46 UTC 2013 [20:10:54] RECOVERY - Puppet freshness on knsq26 is OK: puppet ran at Fri May 24 20:10:46 UTC 2013 [20:10:55] PROBLEM - Puppet freshness on amssq47 is CRITICAL: No successful Puppet run in the last 10 hours [20:11:01] RECOVERY - Puppet freshness on amssq59 is OK: puppet ran at Fri May 24 20:10:52 UTC 2013 [20:11:01] PROBLEM - Puppet freshness on amssq49 is CRITICAL: No successful Puppet run in the last 10 hours [20:11:01] PROBLEM - Puppet freshness on amssq56 is CRITICAL: No successful Puppet run in the last 10 hours [20:11:01] PROBLEM - Puppet freshness on knsq27 is CRITICAL: No successful Puppet run in the last 10 hours [20:11:01] RECOVERY - Puppet freshness on knsq21 is OK: puppet ran at Fri May 24 20:10:58 UTC 2013 [20:11:02] RECOVERY - Puppet freshness on knsq22 is OK: puppet ran at Fri May 24 20:11:00 UTC 2013 [20:11:11] RECOVERY - Puppet freshness on amssq50 is OK: puppet ran at Fri May 24 20:11:00 UTC 2013 [20:11:11] PROBLEM - Puppet freshness on amssq54 is CRITICAL: No successful Puppet run in the last 10 hours [20:11:11] RECOVERY - Puppet freshness on amssq60 is OK: puppet ran at Fri May 24 20:11:05 UTC 2013 [20:11:12] RECOVERY - Puppet freshness on knsq19 is OK: puppet ran at Fri May 24 20:11:07 UTC 2013 [20:11:12] RECOVERY - Puppet freshness on knsq17 is OK: puppet ran at Fri May 24 20:11:07 UTC 2013 [20:11:12] PROBLEM - Puppet freshness on amssq59 is CRITICAL: No successful Puppet run in the last 10 hours [20:11:12] PROBLEM - Puppet freshness on knsq18 is CRITICAL: No successful Puppet run in the last 10 hours [20:11:13] PROBLEM - Puppet freshness on knsq24 is CRITICAL: No successful Puppet run in the last 10 hours [20:11:13] RECOVERY - Puppet freshness on amssq58 is OK: puppet ran at Fri May 24 20:11:10 UTC 2013 [20:11:14] PROBLEM - Puppet freshness on amssq51 is CRITICAL: No successful Puppet run in the last 10 hours [20:11:14] PROBLEM - Puppet freshness on knsq26 is CRITICAL: No successful Puppet run in the last 10 hours [20:11:21] RECOVERY - Puppet freshness on knsq23 is OK: puppet ran at Fri May 24 20:11:12 UTC 2013 [20:11:21] PROBLEM - Puppet freshness on amssq50 is CRITICAL: No successful Puppet run in the last 10 hours [20:11:21] PROBLEM - Puppet freshness on knsq23 is CRITICAL: No successful Puppet run in the last 10 hours [20:11:21] PROBLEM - Puppet freshness on amssq60 is CRITICAL: No successful Puppet run in the last 10 hours [20:11:21] RECOVERY - Puppet freshness on amssq62 is OK: puppet ran at Fri May 24 20:11:15 UTC 2013 [20:11:22] PROBLEM - Puppet freshness on knsq16 is CRITICAL: No successful Puppet run in the last 10 hours [20:11:31] RECOVERY - Puppet freshness on amssq49 is OK: puppet ran at Fri May 24 20:11:22 UTC 2013 [20:11:31] RECOVERY - Puppet freshness on amssq54 is OK: puppet ran at Fri May 24 20:11:22 UTC 2013 [20:11:31] RECOVERY - Puppet freshness on knsq29 is OK: puppet ran at Fri May 24 20:11:23 UTC 2013 [20:11:31] RECOVERY - Puppet freshness on amssq61 is OK: puppet ran at Fri May 24 20:11:23 UTC 2013 [20:11:31] RECOVERY - Puppet freshness on knsq20 is OK: puppet ran at Fri May 24 20:11:29 UTC 2013 [20:11:32] RECOVERY - Puppet freshness on amssq52 is OK: puppet ran at Fri May 24 20:11:29 UTC 2013 [20:11:41] PROBLEM - Puppet freshness on lvs1006 is CRITICAL: No successful Puppet run in the last 10 hours [20:11:41] PROBLEM - Puppet freshness on lvs1004 is CRITICAL: No successful Puppet run in the last 10 hours [20:11:41] PROBLEM - Puppet freshness on lvs1005 is CRITICAL: No successful Puppet run in the last 10 hours [20:11:41] RECOVERY - Puppet freshness on amssq55 is OK: puppet ran at Fri May 24 20:11:32 UTC 2013 [20:11:41] PROBLEM - Puppet freshness on amssq58 is CRITICAL: No successful Puppet run in the last 10 hours [20:11:42] RECOVERY - Puppet freshness on amssq57 is OK: puppet ran at Fri May 24 20:11:35 UTC 2013 [20:11:42] RECOVERY - Puppet freshness on knsq24 is OK: puppet ran at Fri May 24 20:11:36 UTC 2013 [20:11:43] RECOVERY - Puppet freshness on knsq28 is OK: puppet ran at Fri May 24 20:11:38 UTC 2013 [20:11:43] PROBLEM - Puppet freshness on knsq22 is CRITICAL: No successful Puppet run in the last 10 hours [20:11:44] PROBLEM - Puppet freshness on amssq61 is CRITICAL: No successful Puppet run in the last 10 hours [20:11:51] RECOVERY - Puppet freshness on knsq16 is OK: puppet ran at Fri May 24 20:11:41 UTC 2013 [20:11:51] RECOVERY - Puppet freshness on amssq53 is OK: puppet ran at Fri May 24 20:11:41 UTC 2013 [20:11:51] PROBLEM - Puppet freshness on amssq52 is CRITICAL: No successful Puppet run in the last 10 hours [20:11:51] PROBLEM - Puppet freshness on knsq21 is CRITICAL: No successful Puppet run in the last 10 hours [20:11:51] PROBLEM - Puppet freshness on knsq17 is CRITICAL: No successful Puppet run in the last 10 hours [20:11:52] PROBLEM - Puppet freshness on knsq19 is CRITICAL: No successful Puppet run in the last 10 hours [20:11:52] PROBLEM - Puppet freshness on knsq29 is CRITICAL: No successful Puppet run in the last 10 hours [20:11:53] RECOVERY - Puppet freshness on knsq18 is OK: puppet ran at Fri May 24 20:11:44 UTC 2013 [20:11:53] RECOVERY - Puppet freshness on amssq48 is OK: puppet ran at Fri May 24 20:11:47 UTC 2013 [20:11:54] RECOVERY - Puppet freshness on amssq47 is OK: puppet ran at Fri May 24 20:11:47 UTC 2013 [20:11:54] RECOVERY - Puppet freshness on ms2 is OK: puppet ran at Fri May 24 20:11:49 UTC 2013 [20:11:55] PROBLEM - Puppet freshness on amssq47 is CRITICAL: No successful Puppet run in the last 10 hours [20:12:01] RECOVERY - Puppet freshness on amssq56 is OK: puppet ran at Fri May 24 20:11:50 UTC 2013 [20:12:01] RECOVERY - Puppet freshness on amssq51 is OK: puppet ran at Fri May 24 20:11:50 UTC 2013 [20:12:01] PROBLEM - Puppet freshness on amssq49 is CRITICAL: No successful Puppet run in the last 10 hours [20:12:01] PROBLEM - Puppet freshness on amssq56 is CRITICAL: No successful Puppet run in the last 10 hours [20:12:01] RECOVERY - Puppet freshness on knsq27 is OK: puppet ran at Fri May 24 20:11:55 UTC 2013 [20:12:02] RECOVERY - Puppet freshness on knsq26 is OK: puppet ran at Fri May 24 20:11:55 UTC 2013 [20:12:02] RECOVERY - Puppet freshness on amssq59 is OK: puppet ran at Fri May 24 20:11:59 UTC 2013 [20:12:11] PROBLEM - Puppet freshness on amssq54 is CRITICAL: No successful Puppet run in the last 10 hours [20:12:11] RECOVERY - Puppet freshness on knsq21 is OK: puppet ran at Fri May 24 20:12:03 UTC 2013 [20:12:11] RECOVERY - Puppet freshness on amssq50 is OK: puppet ran at Fri May 24 20:12:04 UTC 2013 [20:12:11] RECOVERY - Puppet freshness on knsq22 is OK: puppet ran at Fri May 24 20:12:06 UTC 2013 [20:12:11] PROBLEM - Puppet freshness on amssq59 is CRITICAL: No successful Puppet run in the last 10 hours [20:12:12] PROBLEM - Puppet freshness on knsq18 is CRITICAL: No successful Puppet run in the last 10 hours [20:12:12] PROBLEM - Puppet freshness on knsq24 is CRITICAL: No successful Puppet run in the last 10 hours [20:12:21] PROBLEM - Puppet freshness on amssq51 is CRITICAL: No successful Puppet run in the last 10 hours [20:12:21] PROBLEM - Puppet freshness on amssq62 is CRITICAL: No successful Puppet run in the last 10 hours [20:12:21] PROBLEM - Puppet freshness on knsq26 is CRITICAL: No successful Puppet run in the last 10 hours [20:12:21] PROBLEM - Puppet freshness on amssq50 is CRITICAL: No successful Puppet run in the last 10 hours [20:12:21] RECOVERY - Puppet freshness on knsq19 is OK: puppet ran at Fri May 24 20:12:15 UTC 2013 [20:12:22] RECOVERY - Puppet freshness on amssq58 is OK: puppet ran at Fri May 24 20:12:15 UTC 2013 [20:12:22] RECOVERY - Puppet freshness on knsq17 is OK: puppet ran at Fri May 24 20:12:15 UTC 2013 [20:12:23] RECOVERY - Puppet freshness on knsq23 is OK: puppet ran at Fri May 24 20:12:16 UTC 2013 [20:12:23] RECOVERY - Puppet freshness on amssq60 is OK: puppet ran at Fri May 24 20:12:16 UTC 2013 [20:12:24] PROBLEM - Puppet freshness on amssq55 is CRITICAL: No successful Puppet run in the last 10 hours [20:12:24] PROBLEM - Puppet freshness on knsq16 is CRITICAL: No successful Puppet run in the last 10 hours [20:12:31] RECOVERY - Puppet freshness on amssq62 is OK: puppet ran at Fri May 24 20:12:21 UTC 2013 [20:12:31] RECOVERY - Puppet freshness on amssq54 is OK: puppet ran at Fri May 24 20:12:24 UTC 2013 [20:12:31] RECOVERY - Puppet freshness on amssq49 is OK: puppet ran at Fri May 24 20:12:24 UTC 2013 [20:12:31] PROBLEM - Puppet freshness on knsq20 is CRITICAL: No successful Puppet run in the last 10 hours [20:12:31] RECOVERY - Puppet freshness on amssq61 is OK: puppet ran at Fri May 24 20:12:26 UTC 2013 [20:12:32] RECOVERY - Puppet freshness on knsq20 is OK: puppet ran at Fri May 24 20:12:29 UTC 2013 [20:12:32] RECOVERY - Puppet freshness on knsq29 is OK: puppet ran at Fri May 24 20:12:29 UTC 2013 [20:12:33] RECOVERY - Puppet freshness on amssq52 is OK: puppet ran at Fri May 24 20:12:30 UTC 2013 [20:12:41] RECOVERY - Puppet freshness on amssq55 is OK: puppet ran at Fri May 24 20:12:31 UTC 2013 [20:12:41] PROBLEM - Puppet freshness on amssq58 is CRITICAL: No successful Puppet run in the last 10 hours [20:12:41] PROBLEM - Puppet freshness on knsq28 is CRITICAL: No successful Puppet run in the last 10 hours [20:12:41] RECOVERY - Puppet freshness on amssq57 is OK: puppet ran at Fri May 24 20:12:33 UTC 2013 [20:12:41] RECOVERY - Puppet freshness on knsq24 is OK: puppet ran at Fri May 24 20:12:35 UTC 2013 [20:12:42] PROBLEM - Puppet freshness on amssq57 is CRITICAL: No successful Puppet run in the last 10 hours [20:12:42] RECOVERY - Puppet freshness on knsq28 is OK: puppet ran at Fri May 24 20:12:36 UTC 2013 [20:12:43] PROBLEM - Puppet freshness on amssq53 is CRITICAL: No successful Puppet run in the last 10 hours [20:12:43] RECOVERY - Puppet freshness on knsq16 is OK: puppet ran at Fri May 24 20:12:39 UTC 2013 [20:12:44] RECOVERY - Puppet freshness on amssq53 is OK: puppet ran at Fri May 24 20:12:40 UTC 2013 [20:12:44] PROBLEM - Puppet freshness on knsq22 is CRITICAL: No successful Puppet run in the last 10 hours [20:12:45] PROBLEM - Puppet freshness on amssq61 is CRITICAL: No successful Puppet run in the last 10 hours [20:12:51] RECOVERY - Puppet freshness on knsq18 is OK: puppet ran at Fri May 24 20:12:41 UTC 2013 [20:12:51] PROBLEM - Puppet freshness on amssq48 is CRITICAL: No successful Puppet run in the last 10 hours [20:12:51] PROBLEM - Puppet freshness on amssq52 is CRITICAL: No successful Puppet run in the last 10 hours [20:12:51] PROBLEM - Puppet freshness on ms2 is CRITICAL: No successful Puppet run in the last 10 hours [20:12:51] PROBLEM - Puppet freshness on knsq21 is CRITICAL: No successful Puppet run in the last 10 hours [20:12:52] PROBLEM - Puppet freshness on knsq17 is CRITICAL: No successful Puppet run in the last 10 hours [20:12:52] PROBLEM - Puppet freshness on knsq29 is CRITICAL: No successful Puppet run in the last 10 hours [20:12:53] PROBLEM - Puppet freshness on knsq19 is CRITICAL: No successful Puppet run in the last 10 hours [20:12:53] RECOVERY - Puppet freshness on amssq47 is OK: puppet ran at Fri May 24 20:12:46 UTC 2013 [20:12:54] RECOVERY - Puppet freshness on amssq48 is OK: puppet ran at Fri May 24 20:12:46 UTC 2013 [20:12:54] PROBLEM - Puppet freshness on amssq47 is CRITICAL: No successful Puppet run in the last 10 hours [20:12:55] RECOVERY - Puppet freshness on ms2 is OK: puppet ran at Fri May 24 20:12:48 UTC 2013 [20:12:55] RECOVERY - Puppet freshness on amssq56 is OK: puppet ran at Fri May 24 20:12:49 UTC 2013 [20:12:56] RECOVERY - Puppet freshness on amssq51 is OK: puppet ran at Fri May 24 20:12:49 UTC 2013 [20:12:56] PROBLEM - Puppet freshness on amssq56 is CRITICAL: No successful Puppet run in the last 10 hours [20:13:01] RECOVERY - Puppet freshness on knsq26 is OK: puppet ran at Fri May 24 20:12:51 UTC 2013 [20:13:01] PROBLEM - Puppet freshness on amssq49 is CRITICAL: No successful Puppet run in the last 10 hours [20:13:01] PROBLEM - Puppet freshness on knsq27 is CRITICAL: No successful Puppet run in the last 10 hours [20:13:01] RECOVERY - Puppet freshness on amssq59 is OK: puppet ran at Fri May 24 20:12:56 UTC 2013 [20:13:01] RECOVERY - Puppet freshness on knsq21 is OK: puppet ran at Fri May 24 20:13:00 UTC 2013 [20:13:11] PROBLEM - Puppet freshness on amssq54 is CRITICAL: No successful Puppet run in the last 10 hours [20:13:11] RECOVERY - Puppet freshness on knsq22 is OK: puppet ran at Fri May 24 20:13:07 UTC 2013 [20:13:11] RECOVERY - Puppet freshness on amssq50 is OK: puppet ran at Fri May 24 20:13:07 UTC 2013 [20:13:11] PROBLEM - Puppet freshness on amssq59 is CRITICAL: No successful Puppet run in the last 10 hours [20:13:12] PROBLEM - Puppet freshness on knsq18 is CRITICAL: No successful Puppet run in the last 10 hours [20:13:12] PROBLEM - Puppet freshness on knsq24 is CRITICAL: No successful Puppet run in the last 10 hours [20:13:12] PROBLEM - Puppet freshness on amssq62 is CRITICAL: No successful Puppet run in the last 10 hours [20:13:13] PROBLEM - Puppet freshness on amssq51 is CRITICAL: No successful Puppet run in the last 10 hours [20:13:13] PROBLEM - Puppet freshness on knsq26 is CRITICAL: No successful Puppet run in the last 10 hours [20:13:21] PROBLEM - Puppet freshness on knsq23 is CRITICAL: No successful Puppet run in the last 10 hours [20:13:21] PROBLEM - Puppet freshness on amssq50 is CRITICAL: No successful Puppet run in the last 10 hours [20:13:21] PROBLEM - Puppet freshness on amssq60 is CRITICAL: No successful Puppet run in the last 10 hours [20:13:21] RECOVERY - Puppet freshness on knsq17 is OK: puppet ran at Fri May 24 20:13:16 UTC 2013 [20:13:21] RECOVERY - Puppet freshness on amssq58 is OK: puppet ran at Fri May 24 20:13:16 UTC 2013 [20:13:22] RECOVERY - Puppet freshness on knsq19 is OK: puppet ran at Fri May 24 20:13:16 UTC 2013 [20:13:22] RECOVERY - Puppet freshness on knsq23 is OK: puppet ran at Fri May 24 20:13:17 UTC 2013 [20:13:23] PROBLEM - Puppet freshness on amssq55 is CRITICAL: No successful Puppet run in the last 10 hours [20:13:23] PROBLEM - Puppet freshness on knsq16 is CRITICAL: No successful Puppet run in the last 10 hours [20:13:24] RECOVERY - Puppet freshness on amssq62 is OK: puppet ran at Fri May 24 20:13:20 UTC 2013 [20:13:31] RECOVERY - Puppet freshness on amssq54 is OK: puppet ran at Fri May 24 20:13:23 UTC 2013 [20:13:31] RECOVERY - Puppet freshness on amssq49 is OK: puppet ran at Fri May 24 20:13:23 UTC 2013 [20:13:31] RECOVERY - Puppet freshness on amssq61 is OK: puppet ran at Fri May 24 20:13:24 UTC 2013 [20:13:31] RECOVERY - Puppet freshness on knsq29 is OK: puppet ran at Fri May 24 20:13:24 UTC 2013 [20:13:31] PROBLEM - Puppet freshness on knsq20 is CRITICAL: No successful Puppet run in the last 10 hours [20:13:32] RECOVERY - Puppet freshness on knsq20 is OK: puppet ran at Fri May 24 20:13:26 UTC 2013 [20:13:32] RECOVERY - Puppet freshness on amssq52 is OK: puppet ran at Fri May 24 20:13:26 UTC 2013 [20:13:33] RECOVERY - Puppet freshness on amssq57 is OK: puppet ran at Fri May 24 20:13:30 UTC 2013 [20:13:41] RECOVERY - Puppet freshness on amssq55 is OK: puppet ran at Fri May 24 20:13:31 UTC 2013 [20:13:41] RECOVERY - Puppet freshness on knsq24 is OK: puppet ran at Fri May 24 20:13:31 UTC 2013 [20:13:41] PROBLEM - Puppet freshness on mc15 is CRITICAL: No successful Puppet run in the last 10 hours [20:13:41] PROBLEM - Puppet freshness on amssq58 is CRITICAL: No successful Puppet run in the last 10 hours [20:13:41] PROBLEM - Puppet freshness on knsq28 is CRITICAL: No successful Puppet run in the last 10 hours [20:13:42] PROBLEM - Puppet freshness on amssq57 is CRITICAL: No successful Puppet run in the last 10 hours [20:13:42] RECOVERY - Puppet freshness on knsq16 is OK: puppet ran at Fri May 24 20:13:35 UTC 2013 [20:13:43] RECOVERY - Puppet freshness on knsq18 is OK: puppet ran at Fri May 24 20:13:37 UTC 2013 [20:13:43] PROBLEM - Puppet freshness on amssq53 is CRITICAL: No successful Puppet run in the last 10 hours [20:13:44] PROBLEM - Puppet freshness on knsq22 is CRITICAL: No successful Puppet run in the last 10 hours [20:13:44] PROBLEM - Puppet freshness on amssq61 is CRITICAL: No successful Puppet run in the last 10 hours [20:13:51] PROBLEM - Puppet freshness on amssq48 is CRITICAL: No successful Puppet run in the last 10 hours [20:13:51] PROBLEM - Puppet freshness on amssq52 is CRITICAL: No successful Puppet run in the last 10 hours [20:13:51] RECOVERY - Puppet freshness on amssq47 is OK: puppet ran at Fri May 24 20:13:43 UTC 2013 [20:13:51] PROBLEM - Puppet freshness on ms2 is CRITICAL: No successful Puppet run in the last 10 hours [20:13:51] PROBLEM - Puppet freshness on knsq21 is CRITICAL: No successful Puppet run in the last 10 hours [20:13:52] PROBLEM - Puppet freshness on knsq29 is CRITICAL: No successful Puppet run in the last 10 hours [20:13:52] PROBLEM - Puppet freshness on knsq17 is CRITICAL: No successful Puppet run in the last 10 hours [20:13:53] PROBLEM - Puppet freshness on knsq19 is CRITICAL: No successful Puppet run in the last 10 hours [20:13:53] RECOVERY - Puppet freshness on ms2 is OK: puppet ran at Fri May 24 20:13:46 UTC 2013 [20:13:54] PROBLEM - Puppet freshness on amssq47 is CRITICAL: No successful Puppet run in the last 10 hours [20:13:54] RECOVERY - Puppet freshness on amssq56 is OK: puppet ran at Fri May 24 20:13:47 UTC 2013 [20:13:55] RECOVERY - Puppet freshness on amssq51 is OK: puppet ran at Fri May 24 20:13:47 UTC 2013 [20:13:55] PROBLEM - Puppet freshness on amssq56 is CRITICAL: No successful Puppet run in the last 10 hours [20:14:01] RECOVERY - Puppet freshness on knsq27 is OK: puppet ran at Fri May 24 20:13:51 UTC 2013 [20:14:01] RECOVERY - Puppet freshness on knsq26 is OK: puppet ran at Fri May 24 20:13:51 UTC 2013 [20:14:01] PROBLEM - Puppet freshness on amssq49 is CRITICAL: No successful Puppet run in the last 10 hours [20:14:01] PROBLEM - Puppet freshness on knsq27 is CRITICAL: No successful Puppet run in the last 10 hours [20:14:01] RECOVERY - Puppet freshness on amssq59 is OK: puppet ran at Fri May 24 20:13:55 UTC 2013 [20:14:02] RECOVERY - Puppet freshness on knsq21 is OK: puppet ran at Fri May 24 20:13:58 UTC 2013 [20:14:02] RECOVERY - Puppet freshness on knsq22 is OK: puppet ran at Fri May 24 20:13:59 UTC 2013 [20:14:03] RECOVERY - Puppet freshness on amssq50 is OK: puppet ran at Fri May 24 20:13:59 UTC 2013 [20:14:11] RECOVERY - Puppet freshness on amssq60 is OK: puppet ran at Fri May 24 20:14:02 UTC 2013 [20:14:11] PROBLEM - Puppet freshness on amssq54 is CRITICAL: No successful Puppet run in the last 10 hours [20:14:11] RECOVERY - Puppet freshness on knsq17 is OK: puppet ran at Fri May 24 20:14:03 UTC 2013 [20:14:11] RECOVERY - Puppet freshness on knsq19 is OK: puppet ran at Fri May 24 20:14:03 UTC 2013 [20:14:11] RECOVERY - Puppet freshness on amssq58 is OK: puppet ran at Fri May 24 20:14:03 UTC 2013 [20:14:12] PROBLEM - Puppet freshness on amssq59 is CRITICAL: No successful Puppet run in the last 10 hours [20:14:12] PROBLEM - Puppet freshness on knsq18 is CRITICAL: No successful Puppet run in the last 10 hours [20:14:13] PROBLEM - Puppet freshness on knsq24 is CRITICAL: No successful Puppet run in the last 10 hours [20:14:13] PROBLEM - Puppet freshness on amssq62 is CRITICAL: No successful Puppet run in the last 10 hours [20:14:14] PROBLEM - Puppet freshness on amssq51 is CRITICAL: No successful Puppet run in the last 10 hours [20:14:14] PROBLEM - Puppet freshness on knsq26 is CRITICAL: No successful Puppet run in the last 10 hours [20:14:21] RECOVERY - Puppet freshness on amssq49 is OK: puppet ran at Fri May 24 20:14:10 UTC 2013 [20:14:21] RECOVERY - Puppet freshness on amssq54 is OK: puppet ran at Fri May 24 20:14:10 UTC 2013 [20:14:21] RECOVERY - Puppet freshness on amssq61 is OK: puppet ran at Fri May 24 20:14:11 UTC 2013 [20:14:21] RECOVERY - Puppet freshness on knsq29 is OK: puppet ran at Fri May 24 20:14:12 UTC 2013 [20:14:21] PROBLEM - Puppet freshness on knsq23 is CRITICAL: No successful Puppet run in the last 10 hours [20:14:22] PROBLEM - Puppet freshness on amssq50 is CRITICAL: No successful Puppet run in the last 10 hours [20:14:22] PROBLEM - Puppet freshness on amssq60 is CRITICAL: No successful Puppet run in the last 10 hours [20:14:23] RECOVERY - Puppet freshness on amssq52 is OK: puppet ran at Fri May 24 20:14:14 UTC 2013 [20:14:23] RECOVERY - Puppet freshness on amssq57 is OK: puppet ran at Fri May 24 20:14:17 UTC 2013 [20:14:24] PROBLEM - Puppet freshness on amssq55 is CRITICAL: No successful Puppet run in the last 10 hours [20:14:24] RECOVERY - Puppet freshness on knsq24 is OK: puppet ran at Fri May 24 20:14:18 UTC 2013 [20:14:25] PROBLEM - Puppet freshness on knsq16 is CRITICAL: No successful Puppet run in the last 10 hours [20:14:25] RECOVERY - Puppet freshness on knsq28 is OK: puppet ran at Fri May 24 20:14:20 UTC 2013 [20:14:31] RECOVERY - Puppet freshness on amssq53 is OK: puppet ran at Fri May 24 20:14:22 UTC 2013 [20:14:31] RECOVERY - Puppet freshness on knsq16 is OK: puppet ran at Fri May 24 20:14:22 UTC 2013 [20:14:31] RECOVERY - Puppet freshness on knsq18 is OK: puppet ran at Fri May 24 20:14:23 UTC 2013 [20:14:31] RECOVERY - Puppet freshness on amssq47 is OK: puppet ran at Fri May 24 20:14:25 UTC 2013 [20:14:31] RECOVERY - Puppet freshness on amssq48 is OK: puppet ran at Fri May 24 20:14:25 UTC 2013 [20:14:32] PROBLEM - Puppet freshness on knsq20 is CRITICAL: No successful Puppet run in the last 10 hours [20:14:32] RECOVERY - Puppet freshness on amssq56 is OK: puppet ran at Fri May 24 20:14:27 UTC 2013 [20:14:41] RECOVERY - Puppet freshness on knsq26 is OK: puppet ran at Fri May 24 20:14:31 UTC 2013 [20:14:41] RECOVERY - Puppet freshness on knsq27 is OK: puppet ran at Fri May 24 20:14:31 UTC 2013 [20:14:41] RECOVERY - Puppet freshness on amssq51 is OK: puppet ran at Fri May 24 20:14:31 UTC 2013 [20:14:41] PROBLEM - Puppet freshness on amssq58 is CRITICAL: No successful Puppet run in the last 10 hours [20:14:41] PROBLEM - Puppet freshness on knsq28 is CRITICAL: No successful Puppet run in the last 10 hours [20:14:42] PROBLEM - Puppet freshness on amssq57 is CRITICAL: No successful Puppet run in the last 10 hours [20:14:42] RECOVERY - Puppet freshness on amssq59 is OK: puppet ran at Fri May 24 20:14:35 UTC 2013 [20:14:43] RECOVERY - Puppet freshness on amssq50 is OK: puppet ran at Fri May 24 20:14:38 UTC 2013 [20:14:43] PROBLEM - Puppet freshness on amssq53 is CRITICAL: No successful Puppet run in the last 10 hours [20:14:44] PROBLEM - Puppet freshness on knsq22 is CRITICAL: No successful Puppet run in the last 10 hours [20:14:44] PROBLEM - Puppet freshness on amssq61 is CRITICAL: No successful Puppet run in the last 10 hours [20:14:51] RECOVERY - Puppet freshness on knsq22 is OK: puppet ran at Fri May 24 20:14:41 UTC 2013 [20:14:51] RECOVERY - Puppet freshness on amssq58 is OK: puppet ran at Fri May 24 20:14:42 UTC 2013 [20:14:51] RECOVERY - Puppet freshness on amssq60 is OK: puppet ran at Fri May 24 20:14:42 UTC 2013 [20:14:51] PROBLEM - Puppet freshness on amssq48 is CRITICAL: No successful Puppet run in the last 10 hours [20:14:51] PROBLEM - Puppet freshness on amssq52 is CRITICAL: No successful Puppet run in the last 10 hours [20:14:52] PROBLEM - Puppet freshness on amssq47 is CRITICAL: No successful Puppet run in the last 10 hours [20:14:52] PROBLEM - Puppet freshness on knsq21 is CRITICAL: No successful Puppet run in the last 10 hours [20:14:53] PROBLEM - Puppet freshness on ms2 is CRITICAL: No successful Puppet run in the last 10 hours [20:14:53] PROBLEM - Puppet freshness on knsq29 is CRITICAL: No successful Puppet run in the last 10 hours [20:14:54] PROBLEM - Puppet freshness on knsq17 is CRITICAL: No successful Puppet run in the last 10 hours [20:14:54] PROBLEM - Puppet freshness on knsq19 is CRITICAL: No successful Puppet run in the last 10 hours [20:14:55] RECOVERY - Puppet freshness on knsq23 is OK: puppet ran at Fri May 24 20:14:44 UTC 2013 [20:14:55] RECOVERY - Puppet freshness on amssq62 is OK: puppet ran at Fri May 24 20:14:46 UTC 2013 [20:14:56] PROBLEM - Puppet freshness on amssq56 is CRITICAL: No successful Puppet run in the last 10 hours [20:14:56] RECOVERY - Puppet freshness on knsq29 is OK: puppet ran at Fri May 24 20:14:49 UTC 2013 [20:14:57] RECOVERY - Puppet freshness on amssq61 is OK: puppet ran at Fri May 24 20:14:49 UTC 2013 [20:15:01] RECOVERY - Puppet freshness on amssq55 is OK: puppet ran at Fri May 24 20:14:51 UTC 2013 [20:15:01] RECOVERY - Puppet freshness on knsq20 is OK: puppet ran at Fri May 24 20:14:51 UTC 2013 [20:15:01] RECOVERY - Puppet freshness on amssq52 is OK: puppet ran at Fri May 24 20:14:51 UTC 2013 [20:15:01] PROBLEM - Puppet freshness on knsq27 is CRITICAL: No successful Puppet run in the last 10 hours [20:15:01] RECOVERY - Puppet freshness on amssq57 is OK: puppet ran at Fri May 24 20:14:52 UTC 2013 [20:15:02] PROBLEM - Puppet freshness on amssq49 is CRITICAL: No successful Puppet run in the last 10 hours [20:15:02] RECOVERY - Puppet freshness on knsq28 is OK: puppet ran at Fri May 24 20:14:59 UTC 2013 [20:15:11] RECOVERY - Puppet freshness on amssq53 is OK: puppet ran at Fri May 24 20:15:01 UTC 2013 [20:15:11] PROBLEM - Puppet freshness on amssq54 is CRITICAL: No successful Puppet run in the last 10 hours [20:15:11] RECOVERY - Puppet freshness on amssq47 is OK: puppet ran at Fri May 24 20:15:04 UTC 2013 [20:15:11] RECOVERY - Puppet freshness on amssq48 is OK: puppet ran at Fri May 24 20:15:04 UTC 2013 [20:15:11] PROBLEM - Puppet freshness on amssq59 is CRITICAL: No successful Puppet run in the last 10 hours [20:15:12] PROBLEM - Puppet freshness on knsq18 is CRITICAL: No successful Puppet run in the last 10 hours [20:15:12] PROBLEM - Puppet freshness on knsq24 is CRITICAL: No successful Puppet run in the last 10 hours [20:15:13] PROBLEM - Puppet freshness on amssq51 is CRITICAL: No successful Puppet run in the last 10 hours [20:15:13] PROBLEM - Puppet freshness on amssq62 is CRITICAL: No successful Puppet run in the last 10 hours [20:15:14] PROBLEM - Puppet freshness on knsq26 is CRITICAL: No successful Puppet run in the last 10 hours [20:15:21] RECOVERY - Puppet freshness on ms2 is OK: puppet ran at Fri May 24 20:15:11 UTC 2013 [20:15:21] PROBLEM - Puppet freshness on knsq23 is CRITICAL: No successful Puppet run in the last 10 hours [20:15:21] PROBLEM - Puppet freshness on amssq50 is CRITICAL: No successful Puppet run in the last 10 hours [20:15:21] PROBLEM - Puppet freshness on amssq60 is CRITICAL: No successful Puppet run in the last 10 hours [20:15:21] PROBLEM - Puppet freshness on amssq55 is CRITICAL: No successful Puppet run in the last 10 hours [20:15:22] PROBLEM - Puppet freshness on knsq16 is CRITICAL: No successful Puppet run in the last 10 hours [20:15:31] PROBLEM - Puppet freshness on knsq20 is CRITICAL: No successful Puppet run in the last 10 hours [20:15:41] PROBLEM - Puppet freshness on amssq58 is CRITICAL: No successful Puppet run in the last 10 hours [20:15:41] PROBLEM - Puppet freshness on knsq28 is CRITICAL: No successful Puppet run in the last 10 hours [20:15:41] PROBLEM - Puppet freshness on amssq57 is CRITICAL: No successful Puppet run in the last 10 hours [20:15:41] PROBLEM - Puppet freshness on amssq53 is CRITICAL: No successful Puppet run in the last 10 hours [20:15:41] PROBLEM - Puppet freshness on knsq22 is CRITICAL: No successful Puppet run in the last 10 hours [20:15:42] PROBLEM - Puppet freshness on amssq61 is CRITICAL: No successful Puppet run in the last 10 hours [20:15:51] RECOVERY - Puppet freshness on amssq56 is OK: puppet ran at Fri May 24 20:15:41 UTC 2013 [20:15:51] RECOVERY - Puppet freshness on amssq51 is OK: puppet ran at Fri May 24 20:15:41 UTC 2013 [20:15:51] PROBLEM - Puppet freshness on amssq48 is CRITICAL: No successful Puppet run in the last 10 hours [20:15:51] PROBLEM - Puppet freshness on amssq52 is CRITICAL: No successful Puppet run in the last 10 hours [20:15:51] PROBLEM - Puppet freshness on amssq47 is CRITICAL: No successful Puppet run in the last 10 hours [20:15:52] PROBLEM - Puppet freshness on ms2 is CRITICAL: No successful Puppet run in the last 10 hours [20:15:52] PROBLEM - Puppet freshness on knsq29 is CRITICAL: No successful Puppet run in the last 10 hours [20:15:53] PROBLEM - Puppet freshness on amssq56 is CRITICAL: No successful Puppet run in the last 10 hours [20:16:12] PROBLEM - Puppet freshness on amssq51 is CRITICAL: No successful Puppet run in the last 10 hours [20:17:52] RECOVERY - Puppet freshness on knsq26 is OK: puppet ran at Fri May 24 20:17:45 UTC 2013 [20:18:12] PROBLEM - Puppet freshness on knsq26 is CRITICAL: No successful Puppet run in the last 10 hours [20:19:52] RECOVERY - Puppet freshness on knsq27 is OK: puppet ran at Fri May 24 20:19:47 UTC 2013 [20:20:01] PROBLEM - Puppet freshness on knsq27 is CRITICAL: No successful Puppet run in the last 10 hours [20:20:31] RECOVERY - Puppet freshness on amssq59 is OK: puppet ran at Fri May 24 20:20:23 UTC 2013 [20:21:11] PROBLEM - Puppet freshness on amssq59 is CRITICAL: No successful Puppet run in the last 10 hours [20:22:51] RECOVERY - Puppet freshness on knsq21 is OK: puppet ran at Fri May 24 20:22:47 UTC 2013 [20:23:51] PROBLEM - Puppet freshness on knsq21 is CRITICAL: No successful Puppet run in the last 10 hours [20:23:51] RECOVERY - Puppet freshness on amssq50 is OK: puppet ran at Fri May 24 20:23:49 UTC 2013 [20:23:51] RECOVERY - Puppet freshness on knsq22 is OK: puppet ran at Fri May 24 20:23:49 UTC 2013 [20:24:22] PROBLEM - Puppet freshness on amssq50 is CRITICAL: No successful Puppet run in the last 10 hours [20:24:41] PROBLEM - Puppet freshness on knsq22 is CRITICAL: No successful Puppet run in the last 10 hours [20:25:31] RECOVERY - Puppet freshness on amssq60 is OK: puppet ran at Fri May 24 20:25:29 UTC 2013 [20:25:41] PROBLEM - Puppet freshness on db1017 is CRITICAL: No successful Puppet run in the last 10 hours [20:26:21] PROBLEM - Puppet freshness on amssq60 is CRITICAL: No successful Puppet run in the last 10 hours [20:26:51] RECOVERY - Puppet freshness on knsq17 is OK: puppet ran at Fri May 24 20:26:42 UTC 2013 [20:26:51] RECOVERY - Puppet freshness on knsq19 is OK: puppet ran at Fri May 24 20:26:42 UTC 2013 [20:26:51] RECOVERY - Puppet freshness on amssq58 is OK: puppet ran at Fri May 24 20:26:42 UTC 2013 [20:26:51] PROBLEM - Puppet freshness on knsq17 is CRITICAL: No successful Puppet run in the last 10 hours [20:26:51] PROBLEM - Puppet freshness on knsq19 is CRITICAL: No successful Puppet run in the last 10 hours [20:27:31] RECOVERY - Puppet freshness on knsq23 is OK: puppet ran at Fri May 24 20:27:29 UTC 2013 [20:27:41] PROBLEM - Puppet freshness on amssq58 is CRITICAL: No successful Puppet run in the last 10 hours [20:28:21] PROBLEM - Puppet freshness on knsq23 is CRITICAL: No successful Puppet run in the last 10 hours [20:29:31] RECOVERY - Puppet freshness on amssq62 is OK: puppet ran at Fri May 24 20:29:24 UTC 2013 [20:30:11] PROBLEM - Puppet freshness on amssq62 is CRITICAL: No successful Puppet run in the last 10 hours [20:32:01] RECOVERY - Puppet freshness on amssq54 is OK: puppet ran at Fri May 24 20:31:51 UTC 2013 [20:32:01] RECOVERY - Puppet freshness on amssq49 is OK: puppet ran at Fri May 24 20:31:51 UTC 2013 [20:32:01] PROBLEM - Puppet freshness on amssq49 is CRITICAL: No successful Puppet run in the last 10 hours [20:32:11] PROBLEM - Puppet freshness on amssq54 is CRITICAL: No successful Puppet run in the last 10 hours [20:32:33] RECOVERY - Puppet freshness on amssq61 is OK: puppet ran at Fri May 24 20:32:27 UTC 2013 [20:32:33] RECOVERY - Puppet freshness on knsq29 is OK: puppet ran at Fri May 24 20:32:27 UTC 2013 [20:32:43] PROBLEM - Puppet freshness on amssq61 is CRITICAL: No successful Puppet run in the last 10 hours [20:32:51] PROBLEM - Puppet freshness on knsq29 is CRITICAL: No successful Puppet run in the last 10 hours [20:34:41] RECOVERY - Puppet freshness on knsq20 is OK: puppet ran at Fri May 24 20:34:31 UTC 2013 [20:34:41] RECOVERY - Puppet freshness on amssq52 is OK: puppet ran at Fri May 24 20:34:31 UTC 2013 [20:34:51] PROBLEM - Puppet freshness on amssq52 is CRITICAL: No successful Puppet run in the last 10 hours [20:35:30] PROBLEM - Puppet freshness on knsq20 is CRITICAL: No successful Puppet run in the last 10 hours [20:38:49] !log aaron synchronized php-1.22wmf4/includes/job/jobs/RefreshLinksJob.php '88ffcabd2b0039d18b805bd1e5d8d350c26588e1' [20:38:58] Logged the message, Master [20:51:20] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:52:11] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.125 second response time [21:06:26] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:07:16] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.128 second response time [21:31:26] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:32:16] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.127 second response time [21:40:30] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:41:20] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.125 second response time [21:56:30] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:59:20] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.169 second response time [22:12:59] New patchset: Spage; "Default to new login and create account forms" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/65340 [22:15:25] ohno [22:15:29] someone -1 it please [22:22:28] New patchset: Mattflaschen; "Default to new login and create account forms" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/65340 [22:23:09] New patchset: Mattflaschen; "Default to new login and create account forms" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/65340 [22:28:32] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:29:22] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.125 second response time [22:30:15] New patchset: Petrb; "inserted phpunit on exec nodes because they are needed for development tests" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/65341 [22:31:55] Coren can u pls merge this https://gerrit.wikimedia.org/r/#/c/65341/ [22:53:39] PROBLEM - NTP on ssl3003 is CRITICAL: NTP CRITICAL: No response from NTP server [22:54:09] Yo. [22:54:19] xff@wikimedia.org routes to whom? [22:59:16] https://noc.wikimedia.org/conf/ [22:59:21] "CDB" there refers to ... [22:59:47] https://en.wikipedia.org/wiki/Cdb_(software) ? [23:01:10] New patchset: Reedy; "Updated trusted-xff.cdb" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/65347 [23:01:31] Change merged: Reedy; [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/65347 [23:02:45] !log reedy synchronized wmf-config/trusted-xff.cdb [23:02:52] Logged the message, Master [23:09:30] PROBLEM - NTP on ssl3002 is CRITICAL: NTP CRITICAL: No response from NTP server [23:18:03] New review: Mattflaschen; "Looks good. Will +2 when we're ready to deploy this config change." [operations/mediawiki-config] (master) C: 1; - https://gerrit.wikimedia.org/r/65340 [23:36:30] !log aaron synchronized php-1.22wmf4/includes/profiler/Profiler.php '3d47dae94d7c416a1e2d4acc506c5e6a212592af' [23:36:39] Logged the message, Master [23:37:10] !log aaron synchronized php-1.22wmf4/maintenance '3d47dae94d7c416a1e2d4acc506c5e6a212592af' [23:37:19] Logged the message, Master