[00:00:26] PROBLEM - Apache HTTP on srv198 is CRITICAL: Connection refused [00:07:39] RECOVERY - LDAP on nfs1 is OK: TCP OK - 0.001 second response time on port 389 [00:08:07] RECOVERY - LDAPS on nfs1 is OK: TCP OK - 0.002 second response time on port 636 [00:08:24] RECOVERY - LDAPS on nfs2 is OK: TCP OK - 0.002 second response time on port 636 [00:09:36] RECOVERY - LDAP on nfs2 is OK: TCP OK - 0.001 second response time on port 389 [00:14:51] RECOVERY - NTP on srv198 is OK: NTP OK: Offset -0.03795826435 secs [01:41:06] PROBLEM - MySQL Slave Delay on db1025 is CRITICAL: CRIT replication delay 269 seconds [01:41:06] PROBLEM - MySQL Slave Delay on storage3 is CRITICAL: CRIT replication delay 271 seconds [01:45:36] RECOVERY - MySQL Slave Delay on db1025 is OK: OK replication delay 2 seconds [01:45:45] RECOVERY - MySQL Slave Delay on storage3 is OK: OK replication delay 4 seconds [02:56:06] PROBLEM - Puppet freshness on ocg3 is CRITICAL: Puppet has not run in the last 10 hours [02:56:06] PROBLEM - Puppet freshness on singer is CRITICAL: Puppet has not run in the last 10 hours [02:56:06] PROBLEM - Puppet freshness on virt1002 is CRITICAL: Puppet has not run in the last 10 hours [02:56:06] PROBLEM - Puppet freshness on virt1003 is CRITICAL: Puppet has not run in the last 10 hours [02:56:06] PROBLEM - Puppet freshness on virt1001 is CRITICAL: Puppet has not run in the last 10 hours [02:56:07] PROBLEM - Puppet freshness on virt1004 is CRITICAL: Puppet has not run in the last 10 hours [03:35:15] PROBLEM - Puppet freshness on ms-be10 is CRITICAL: Puppet has not run in the last 10 hours [04:04:57] PROBLEM - udp2log log age for locke on locke is CRITICAL: CRITICAL: log files /a/squid/fundraising/logs/bannerImpressions-sampled100.log, have not been written in a critical amount of time. For most logs, this is 4 hours. For slow logs, this is 4 days. [04:30:45] RECOVERY - udp2log log age for locke on locke is OK: OK: all log files active [04:52:12] PROBLEM - Puppet freshness on zhen is CRITICAL: Puppet has not run in the last 10 hours [05:42:09] PROBLEM - Puppet freshness on ms-be7 is CRITICAL: Puppet has not run in the last 10 hours [05:49:35] yeah and it's not going to run on ms be7, I disabled it so there :-P [05:49:39] * apergos goes to bed [05:52:30] PROBLEM - Apache HTTP on srv221 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [05:56:51] RECOVERY - Apache HTTP on srv221 is OK: HTTP OK - HTTP/1.1 301 Moved Permanently - 0.027 second response time [07:01:31] PROBLEM - Puppet freshness on neon is CRITICAL: Puppet has not run in the last 10 hours [08:16:53] New review: Dereckson; "The following comment has been received on Bugzilla:" [operations/mediawiki-config] (master) C: -1; - https://gerrit.wikimedia.org/r/23222 [08:20:33] New patchset: Dereckson; "(bug 39206) Namespaces configuration for se.wikipedia" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/23222 [08:21:38] New review: Dereckson; "PS3: Port?ls?gastallan -> Port?las?gastallan (per comment bug requesting change)" [operations/mediawiki-config] (master) C: 0; - https://gerrit.wikimedia.org/r/23222 [08:27:28] PROBLEM - Puppet freshness on zinc is CRITICAL: Puppet has not run in the last 10 hours [08:27:28] PROBLEM - Puppet freshness on magnesium is CRITICAL: Puppet has not run in the last 10 hours [09:09:28] PROBLEM - Puppet freshness on knsq21 is CRITICAL: Puppet has not run in the last 10 hours [09:11:34] PROBLEM - Puppet freshness on srv195 is CRITICAL: Puppet has not run in the last 10 hours [09:13:31] PROBLEM - Puppet freshness on amssq35 is CRITICAL: Puppet has not run in the last 10 hours [09:19:31] PROBLEM - Puppet freshness on srv196 is CRITICAL: Puppet has not run in the last 10 hours [09:27:28] PROBLEM - Puppet freshness on srv197 is CRITICAL: Puppet has not run in the last 10 hours [09:29:34] PROBLEM - Puppet freshness on manganese is CRITICAL: Puppet has not run in the last 10 hours [09:31:31] PROBLEM - Puppet freshness on amslvs1 is CRITICAL: Puppet has not run in the last 10 hours [09:31:31] PROBLEM - Puppet freshness on amslvs2 is CRITICAL: Puppet has not run in the last 10 hours [09:31:31] PROBLEM - Puppet freshness on amssq32 is CRITICAL: Puppet has not run in the last 10 hours [09:31:31] PROBLEM - Puppet freshness on amssq33 is CRITICAL: Puppet has not run in the last 10 hours [09:31:31] PROBLEM - Puppet freshness on amssq36 is CRITICAL: Puppet has not run in the last 10 hours [09:32:39] PROBLEM - Puppet freshness on ms6 is CRITICAL: Puppet has not run in the last 10 hours [09:32:39] PROBLEM - Puppet freshness on ssl3002 is CRITICAL: Puppet has not run in the last 10 hours [09:32:40] PROBLEM - Puppet freshness on ssl3001 is CRITICAL: Puppet has not run in the last 10 hours [09:32:40] PROBLEM - Puppet freshness on ssl3003 is CRITICAL: Puppet has not run in the last 10 hours [09:35:34] PROBLEM - Puppet freshness on srv198 is CRITICAL: Puppet has not run in the last 10 hours [09:36:37] PROBLEM - Puppet freshness on amssq43 is CRITICAL: Puppet has not run in the last 10 hours [09:36:37] PROBLEM - Puppet freshness on amssq46 is CRITICAL: Puppet has not run in the last 10 hours [09:37:31] PROBLEM - Puppet freshness on sq60 is CRITICAL: Puppet has not run in the last 10 hours [09:38:30] New patchset: Dereckson; "(bug 40173) Set wgArticleCountMethod for gu.wikisource" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/23490 [09:38:34] PROBLEM - Puppet freshness on amssq59 is CRITICAL: Puppet has not run in the last 10 hours [09:38:34] PROBLEM - Puppet freshness on analytics1008 is CRITICAL: Puppet has not run in the last 10 hours [09:39:28] PROBLEM - Puppet freshness on erzurumi is CRITICAL: Puppet has not run in the last 10 hours [09:39:49] New review: Dereckson; "Shellpolicy. A local consensus is required before merge this change." [operations/mediawiki-config] (master) C: -1; - https://gerrit.wikimedia.org/r/23490 [09:40:31] PROBLEM - Puppet freshness on dobson is CRITICAL: Puppet has not run in the last 10 hours [09:40:31] PROBLEM - Puppet freshness on pdf1 is CRITICAL: Puppet has not run in the last 10 hours [09:40:31] PROBLEM - Puppet freshness on ocg1 is CRITICAL: Puppet has not run in the last 10 hours [09:41:34] PROBLEM - Puppet freshness on chromium is CRITICAL: Puppet has not run in the last 10 hours [09:41:34] PROBLEM - Puppet freshness on ms10 is CRITICAL: Puppet has not run in the last 10 hours [09:42:37] PROBLEM - Puppet freshness on ms-be6 is CRITICAL: Puppet has not run in the last 10 hours [09:43:31] PROBLEM - Puppet freshness on pdf2 is CRITICAL: Puppet has not run in the last 10 hours [09:43:31] PROBLEM - Puppet freshness on yvon is CRITICAL: Puppet has not run in the last 10 hours [09:44:34] PROBLEM - Puppet freshness on cp1029 is CRITICAL: Puppet has not run in the last 10 hours [09:44:34] PROBLEM - Puppet freshness on analytics1001 is CRITICAL: Puppet has not run in the last 10 hours [09:44:34] PROBLEM - Puppet freshness on cp1033 is CRITICAL: Puppet has not run in the last 10 hours [09:44:34] PROBLEM - Puppet freshness on db9 is CRITICAL: Puppet has not run in the last 10 hours [09:44:34] PROBLEM - Puppet freshness on db10 is CRITICAL: Puppet has not run in the last 10 hours [09:46:31] PROBLEM - Puppet freshness on ms1001 is CRITICAL: Puppet has not run in the last 10 hours [09:46:31] PROBLEM - Puppet freshness on tarin is CRITICAL: Puppet has not run in the last 10 hours [09:49:31] PROBLEM - Puppet freshness on cp1031 is CRITICAL: Puppet has not run in the last 10 hours [09:49:31] PROBLEM - Puppet freshness on ersch is CRITICAL: Puppet has not run in the last 10 hours [09:52:31] PROBLEM - Puppet freshness on analytics1011 is CRITICAL: Puppet has not run in the last 10 hours [09:52:31] PROBLEM - Puppet freshness on analytics1012 is CRITICAL: Puppet has not run in the last 10 hours [09:52:31] PROBLEM - Puppet freshness on analytics1018 is CRITICAL: Puppet has not run in the last 10 hours [09:52:31] PROBLEM - Puppet freshness on db29 is CRITICAL: Puppet has not run in the last 10 hours [09:52:31] PROBLEM - Puppet freshness on analytics1022 is CRITICAL: Puppet has not run in the last 10 hours [09:52:31] PROBLEM - Puppet freshness on search15 is CRITICAL: Puppet has not run in the last 10 hours [09:52:31] PROBLEM - Puppet freshness on nitrogen is CRITICAL: Puppet has not run in the last 10 hours [09:52:32] PROBLEM - Puppet freshness on search20 is CRITICAL: Puppet has not run in the last 10 hours [09:52:33] PROBLEM - Puppet freshness on search32 is CRITICAL: Puppet has not run in the last 10 hours [09:52:33] PROBLEM - Puppet freshness on sq63 is CRITICAL: Puppet has not run in the last 10 hours [09:52:33] PROBLEM - Puppet freshness on virt3 is CRITICAL: Puppet has not run in the last 10 hours [09:52:34] PROBLEM - Puppet freshness on virt1008 is CRITICAL: Puppet has not run in the last 10 hours [09:53:34] PROBLEM - Puppet freshness on amssq31 is CRITICAL: Puppet has not run in the last 10 hours [09:53:34] PROBLEM - Puppet freshness on amslvs3 is CRITICAL: Puppet has not run in the last 10 hours [09:53:34] PROBLEM - Puppet freshness on analytics1002 is CRITICAL: Puppet has not run in the last 10 hours [09:53:34] PROBLEM - Puppet freshness on analytics1003 is CRITICAL: Puppet has not run in the last 10 hours [09:53:34] PROBLEM - Puppet freshness on amssq40 is CRITICAL: Puppet has not run in the last 10 hours [14:30:18] I [14:30:23] I'm looking at the swift outage [14:30:29] (fwiw) [14:46:43] !log restarted swift on ms-fe1-4, proxy-server was leaking memory and had been swapping [14:46:53] Logged the message, Master [14:48:22] restarted the silly bot too [14:48:28] RECOVERY - Swift HTTP on ms-fe2 is OK: HTTP OK HTTP/1.1 200 OK - 397 bytes in 0.008 seconds [14:48:37] what swift outage? (I slept through whatever happened) [14:48:46] RECOVERY - Apache HTTP on srv221 is OK: HTTP OK - HTTP/1.1 301 Moved Permanently - 0.030 second response time [14:51:16] apergos: Good to see your /swift/ actions [14:52:21] well more to the point I see no pages or email [14:53:34] PROBLEM - Puppet freshness on zhen is CRITICAL: Puppet has not run in the last 10 hours [14:54:13] I guess I will find out later today [15:20:52] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK HTTP/1.1 400 Bad Request - 336 bytes in 0.028 seconds [15:21:01] RECOVERY - Puppet freshness on cp1016 is OK: puppet ran at Wed Sep 12 15:20:25 UTC 2012 [15:21:01] RECOVERY - Puppet freshness on sq76 is OK: puppet ran at Wed Sep 12 15:20:26 UTC 2012 [15:21:01] RECOVERY - Puppet freshness on iron is OK: puppet ran at Wed Sep 12 15:20:32 UTC 2012 [15:21:01] RECOVERY - Puppet freshness on search32 is OK: puppet ran at Wed Sep 12 15:20:37 UTC 2012 [15:21:01] RECOVERY - Puppet freshness on search31 is OK: puppet ran at Wed Sep 12 15:20:42 UTC 2012 [15:21:02] RECOVERY - Puppet freshness on mw56 is OK: puppet ran at Wed Sep 12 15:20:43 UTC 2012 [15:21:02] RECOVERY - Puppet freshness on mw5 is OK: puppet ran at Wed Sep 12 15:20:46 UTC 2012 [15:21:02] RECOVERY - Puppet freshness on mw64 is OK: puppet ran at Wed Sep 12 15:20:46 UTC 2012 [15:21:03] RECOVERY - Puppet freshness on zirconium is OK: puppet ran at Wed Sep 12 15:20:48 UTC 2012 [15:21:03] RECOVERY - Puppet freshness on snapshot1004 is OK: puppet ran at Wed Sep 12 15:20:51 UTC 2012 [15:21:04] RECOVERY - Puppet freshness on es1 is OK: puppet ran at Wed Sep 12 15:20:53 UTC 2012 [15:21:10] RECOVERY - Puppet freshness on ms1002 is OK: puppet ran at Wed Sep 12 15:20:58 UTC 2012 [15:21:10] RECOVERY - Puppet freshness on srv194 is OK: puppet ran at Wed Sep 12 15:21:00 UTC 2012 [15:21:19] RECOVERY - Puppet freshness on srv270 is OK: puppet ran at Wed Sep 12 15:21:01 UTC 2012 [15:21:19] RECOVERY - Puppet freshness on analytics1020 is OK: puppet ran at Wed Sep 12 15:21:05 UTC 2012 [15:21:19] RECOVERY - Puppet freshness on db46 is OK: puppet ran at Wed Sep 12 15:21:08 UTC 2012 [15:21:19] RECOVERY - Puppet freshness on srv244 is OK: puppet ran at Wed Sep 12 15:21:08 UTC 2012 [15:21:29] RECOVERY - Puppet freshness on srv195 is OK: puppet ran at Wed Sep 12 15:21:12 UTC 2012 [15:21:29] RECOVERY - Puppet freshness on db44 is OK: puppet ran at Wed Sep 12 15:21:17 UTC 2012 [15:21:29] RECOVERY - Puppet freshness on mw72 is OK: puppet ran at Wed Sep 12 15:21:17 UTC 2012 [15:21:37] RECOVERY - Puppet freshness on db60 is OK: puppet ran at Wed Sep 12 15:21:28 UTC 2012 [15:21:37] RECOVERY - Puppet freshness on mw25 is OK: puppet ran at Wed Sep 12 15:21:29 UTC 2012 [15:21:46] RECOVERY - Puppet freshness on srv232 is OK: puppet ran at Wed Sep 12 15:21:30 UTC 2012 [15:21:46] RECOVERY - Puppet freshness on srv257 is OK: puppet ran at Wed Sep 12 15:21:31 UTC 2012 [15:21:55] RECOVERY - Puppet freshness on mw17 is OK: puppet ran at Wed Sep 12 15:21:37 UTC 2012 [15:21:55] RECOVERY - Puppet freshness on silver is OK: puppet ran at Wed Sep 12 15:21:39 UTC 2012 [15:21:55] RECOVERY - Puppet freshness on palladium is OK: puppet ran at Wed Sep 12 15:21:40 UTC 2012 [15:21:55] RECOVERY - Puppet freshness on analytics1024 is OK: puppet ran at Wed Sep 12 15:21:47 UTC 2012 [15:21:55] RECOVERY - Puppet freshness on mw11 is OK: puppet ran at Wed Sep 12 15:21:52 UTC 2012 [15:22:31] RECOVERY - Puppet freshness on nickel is OK: puppet ran at Wed Sep 12 15:21:58 UTC 2012 [15:22:31] RECOVERY - Puppet freshness on cp1024 is OK: puppet ran at Wed Sep 12 15:22:00 UTC 2012 [15:22:31] RECOVERY - Puppet freshness on mw60 is OK: puppet ran at Wed Sep 12 15:22:02 UTC 2012 [15:22:31] RECOVERY - Puppet freshness on search21 is OK: puppet ran at Wed Sep 12 15:22:06 UTC 2012 [15:22:31] RECOVERY - Puppet freshness on amssq36 is OK: puppet ran at Wed Sep 12 15:22:07 UTC 2012 [15:22:32] RECOVERY - Puppet freshness on cp1007 is OK: puppet ran at Wed Sep 12 15:22:15 UTC 2012 [15:22:32] RECOVERY - Puppet freshness on ms-be4 is OK: puppet ran at Wed Sep 12 15:22:16 UTC 2012 [15:22:33] RECOVERY - Puppet freshness on cp1019 is OK: puppet ran at Wed Sep 12 15:22:20 UTC 2012 [15:22:33] RECOVERY - Puppet freshness on sq54 is OK: puppet ran at Wed Sep 12 15:22:21 UTC 2012 [15:22:33] RECOVERY - Puppet freshness on nitrogen is OK: puppet ran at Wed Sep 12 15:22:22 UTC 2012 [15:22:34] RECOVERY - Puppet freshness on snapshot1001 is OK: puppet ran at Wed Sep 12 15:22:22 UTC 2012 [15:22:34] RECOVERY - Puppet freshness on ms-be1011 is OK: puppet ran at Wed Sep 12 15:22:23 UTC 2012 [15:22:40] RECOVERY - Puppet freshness on search27 is OK: puppet ran at Wed Sep 12 15:22:25 UTC 2012 [15:22:40] RECOVERY - Puppet freshness on db9 is OK: puppet ran at Wed Sep 12 15:22:25 UTC 2012 [15:22:40] RECOVERY - Puppet freshness on search1013 is OK: puppet ran at Wed Sep 12 15:22:28 UTC 2012 [15:22:40] RECOVERY - Puppet freshness on stat1001 is OK: puppet ran at Wed Sep 12 15:22:30 UTC 2012 [15:22:40] RECOVERY - Puppet freshness on fenari is OK: puppet ran at Wed Sep 12 15:22:30 UTC 2012 [15:22:49] RECOVERY - Puppet freshness on es1006 is OK: puppet ran at Wed Sep 12 15:22:31 UTC 2012 [15:22:49] RECOVERY - Puppet freshness on sq66 is OK: puppet ran at Wed Sep 12 15:22:31 UTC 2012 [15:22:49] RECOVERY - Puppet freshness on analytics1011 is OK: puppet ran at Wed Sep 12 15:22:32 UTC 2012 [15:22:49] RECOVERY - Puppet freshness on ms10 is OK: puppet ran at Wed Sep 12 15:22:36 UTC 2012 [15:22:49] RECOVERY - Puppet freshness on calcium is OK: puppet ran at Wed Sep 12 15:22:39 UTC 2012 [15:22:58] RECOVERY - Puppet freshness on search1024 is OK: puppet ran at Wed Sep 12 15:22:42 UTC 2012 [15:22:58] RECOVERY - Puppet freshness on srv229 is OK: puppet ran at Wed Sep 12 15:22:44 UTC 2012 [15:22:58] RECOVERY - Puppet freshness on amssq34 is OK: puppet ran at Wed Sep 12 15:22:45 UTC 2012 [15:22:58] RECOVERY - Puppet freshness on mw28 is OK: puppet ran at Wed Sep 12 15:22:52 UTC 2012 [15:23:16] RECOVERY - Puppet freshness on db27 is OK: puppet ran at Wed Sep 12 15:23:00 UTC 2012 [15:23:16] RECOVERY - Puppet freshness on amssq54 is OK: puppet ran at Wed Sep 12 15:23:03 UTC 2012 [15:23:16] RECOVERY - Puppet freshness on srv283 is OK: puppet ran at Wed Sep 12 15:23:03 UTC 2012 [15:23:25] RECOVERY - Puppet freshness on srv205 is OK: puppet ran at Wed Sep 12 15:23:09 UTC 2012 [15:23:25] RECOVERY - Puppet freshness on sq67 is OK: puppet ran at Wed Sep 12 15:23:11 UTC 2012 [15:23:25] RECOVERY - Puppet freshness on mw15 is OK: puppet ran at Wed Sep 12 15:23:11 UTC 2012 [15:23:25] RECOVERY - Puppet freshness on srv233 is OK: puppet ran at Wed Sep 12 15:23:13 UTC 2012 [15:23:25] RECOVERY - Puppet freshness on srv285 is OK: puppet ran at Wed Sep 12 15:23:16 UTC 2012 [15:23:25] RECOVERY - Puppet freshness on srv295 is OK: puppet ran at Wed Sep 12 15:23:19 UTC 2012 [15:23:26] RECOVERY - Puppet freshness on copper is OK: puppet ran at Wed Sep 12 15:23:20 UTC 2012 [15:24:01] RECOVERY - Puppet freshness on db1033 is OK: puppet ran at Wed Sep 12 15:23:26 UTC 2012 [15:24:01] RECOVERY - Puppet freshness on srv243 is OK: puppet ran at Wed Sep 12 15:23:27 UTC 2012 [15:24:01] RECOVERY - Puppet freshness on storage3 is OK: puppet ran at Wed Sep 12 15:23:27 UTC 2012 [15:24:01] RECOVERY - Puppet freshness on mw52 is OK: puppet ran at Wed Sep 12 15:23:28 UTC 2012 [15:24:01] RECOVERY - Puppet freshness on srv212 is OK: puppet ran at Wed Sep 12 15:23:28 UTC 2012 [15:24:02] RECOVERY - Puppet freshness on analytics1013 is OK: puppet ran at Wed Sep 12 15:23:31 UTC 2012 [15:24:02] RECOVERY - Puppet freshness on db54 is OK: puppet ran at Wed Sep 12 15:23:33 UTC 2012 [15:24:02] RECOVERY - Puppet freshness on search1011 is OK: puppet ran at Wed Sep 12 15:23:38 UTC 2012 [15:24:03] RECOVERY - Puppet freshness on cp1042 is OK: puppet ran at Wed Sep 12 15:23:38 UTC 2012 [15:24:04] RECOVERY - Puppet freshness on amssq46 is OK: puppet ran at Wed Sep 12 15:23:39 UTC 2012 [15:24:04] RECOVERY - Puppet freshness on search24 is OK: puppet ran at Wed Sep 12 15:23:40 UTC 2012 [15:24:04] RECOVERY - Puppet freshness on es1008 is OK: puppet ran at Wed Sep 12 15:23:42 UTC 2012 [15:24:05] RECOVERY - Puppet freshness on cp3001 is OK: puppet ran at Wed Sep 12 15:23:42 UTC 2012 [15:24:05] RECOVERY - Puppet freshness on search30 is OK: puppet ran at Wed Sep 12 15:23:42 UTC 2012 [15:24:06] RECOVERY - Puppet freshness on sq55 is OK: puppet ran at Wed Sep 12 15:23:44 UTC 2012 [15:24:06] RECOVERY - Puppet freshness on sq63 is OK: puppet ran at Wed Sep 12 15:23:45 UTC 2012 [15:24:07] RECOVERY - Puppet freshness on ocg2 is OK: puppet ran at Wed Sep 12 15:23:47 UTC 2012 [15:24:07] RECOVERY - Puppet freshness on amssq38 is OK: puppet ran at Wed Sep 12 15:23:49 UTC 2012 [15:24:08] RECOVERY - Puppet freshness on linne is OK: puppet ran at Wed Sep 12 15:23:50 UTC 2012 [15:24:10] RECOVERY - Puppet freshness on knsq18 is OK: puppet ran at Wed Sep 12 15:23:56 UTC 2012 [15:24:10] RECOVERY - Puppet freshness on formey is OK: puppet ran at Wed Sep 12 15:23:59 UTC 2012 [15:24:10] RECOVERY - Puppet freshness on analytics1009 is OK: puppet ran at Wed Sep 12 15:24:01 UTC 2012 [15:24:19] RECOVERY - Puppet freshness on ms-fe1 is OK: puppet ran at Wed Sep 12 15:24:03 UTC 2012 [15:24:19] RECOVERY - Puppet freshness on db1038 is OK: puppet ran at Wed Sep 12 15:24:04 UTC 2012 [15:24:19] RECOVERY - Puppet freshness on cp1022 is OK: puppet ran at Wed Sep 12 15:24:07 UTC 2012 [15:24:19] RECOVERY - Puppet freshness on search15 is OK: puppet ran at Wed Sep 12 15:24:09 UTC 2012 [15:24:28] RECOVERY - Puppet freshness on srv267 is OK: puppet ran at Wed Sep 12 15:24:13 UTC 2012 [15:24:28] RECOVERY - Puppet freshness on mw46 is OK: puppet ran at Wed Sep 12 15:24:23 UTC 2012 [15:24:28] RECOVERY - Puppet freshness on sq80 is OK: puppet ran at Wed Sep 12 15:24:23 UTC 2012 [15:24:28] RECOVERY - Puppet freshness on es7 is OK: puppet ran at Wed Sep 12 15:24:23 UTC 2012 [15:24:37] RECOVERY - Puppet freshness on search1019 is OK: puppet ran at Wed Sep 12 15:24:28 UTC 2012 [15:24:46] RECOVERY - Puppet freshness on search1022 is OK: puppet ran at Wed Sep 12 15:24:35 UTC 2012 [15:24:55] RECOVERY - Puppet freshness on mw33 is OK: puppet ran at Wed Sep 12 15:24:49 UTC 2012 [15:25:31] RECOVERY - Puppet freshness on labstore2 is OK: puppet ran at Wed Sep 12 15:24:56 UTC 2012 [15:25:31] RECOVERY - Puppet freshness on srv275 is OK: puppet ran at Wed Sep 12 15:24:58 UTC 2012 [15:25:31] RECOVERY - Puppet freshness on srv284 is OK: puppet ran at Wed Sep 12 15:25:00 UTC 2012 [15:25:31] RECOVERY - Puppet freshness on sq60 is OK: puppet ran at Wed Sep 12 15:25:05 UTC 2012 [15:25:31] RECOVERY - Puppet freshness on sq64 is OK: puppet ran at Wed Sep 12 15:25:10 UTC 2012 [15:25:31] RECOVERY - Puppet freshness on brewster is OK: puppet ran at Wed Sep 12 15:25:11 UTC 2012 [15:25:32] RECOVERY - Puppet freshness on mw34 is OK: puppet ran at Wed Sep 12 15:25:11 UTC 2012 [15:25:32] RECOVERY - Puppet freshness on virt1007 is OK: puppet ran at Wed Sep 12 15:25:19 UTC 2012 [15:25:33] RECOVERY - Puppet freshness on db1011 is OK: puppet ran at Wed Sep 12 15:25:19 UTC 2012 [15:25:33] RECOVERY - Puppet freshness on db55 is OK: puppet ran at Wed Sep 12 15:25:21 UTC 2012 [15:25:40] RECOVERY - Puppet freshness on search1002 is OK: puppet ran at Wed Sep 12 15:25:30 UTC 2012 [15:25:49] RECOVERY - Puppet freshness on db1047 is OK: puppet ran at Wed Sep 12 15:25:32 UTC 2012 [15:25:49] RECOVERY - Puppet freshness on srv262 is OK: puppet ran at Wed Sep 12 15:25:35 UTC 2012 [15:25:49] RECOVERY - Puppet freshness on analytics1001 is OK: puppet ran at Wed Sep 12 15:25:36 UTC 2012 [15:25:58] RECOVERY - Puppet freshness on sq45 is OK: puppet ran at Wed Sep 12 15:25:42 UTC 2012 [15:25:58] RECOVERY - Puppet freshness on search25 is OK: puppet ran at Wed Sep 12 15:25:46 UTC 2012 [15:25:58] RECOVERY - Puppet freshness on es1001 is OK: puppet ran at Wed Sep 12 15:25:52 UTC 2012 [15:25:58] RECOVERY - Puppet freshness on search1016 is OK: puppet ran at Wed Sep 12 15:25:53 UTC 2012 [15:25:58] RECOVERY - Puppet freshness on db63 is OK: puppet ran at Wed Sep 12 15:25:55 UTC 2012 [15:26:07] RECOVERY - Puppet freshness on knsq23 is OK: puppet ran at Wed Sep 12 15:25:55 UTC 2012 [15:26:07] RECOVERY - Puppet freshness on cp1026 is OK: puppet ran at Wed Sep 12 15:25:55 UTC 2012 [15:26:07] RECOVERY - Puppet freshness on srv196 is OK: puppet ran at Wed Sep 12 15:25:55 UTC 2012 [15:26:16] RECOVERY - Puppet freshness on mchenry is OK: puppet ran at Wed Sep 12 15:25:58 UTC 2012 [15:26:16] RECOVERY - Puppet freshness on ssl1002 is OK: puppet ran at Wed Sep 12 15:25:59 UTC 2012 [15:26:16] RECOVERY - Puppet freshness on mw32 is OK: puppet ran at Wed Sep 12 15:26:00 UTC 2012 [15:26:16] RECOVERY - Puppet freshness on cadmium is OK: puppet ran at Wed Sep 12 15:26:01 UTC 2012 [15:26:25] RECOVERY - Puppet freshness on vanadium is OK: puppet ran at Wed Sep 12 15:26:08 UTC 2012 [15:26:25] RECOVERY - Puppet freshness on lvs1004 is OK: puppet ran at Wed Sep 12 15:26:09 UTC 2012 [15:26:25] RECOVERY - Puppet freshness on cp1021 is OK: puppet ran at Wed Sep 12 15:26:11 UTC 2012 [15:26:25] RECOVERY - Puppet freshness on argon is OK: puppet ran at Wed Sep 12 15:26:12 UTC 2012 [15:26:25] RECOVERY - Puppet freshness on ms-fe1002 is OK: puppet ran at Wed Sep 12 15:26:14 UTC 2012 [15:26:25] RECOVERY - Puppet freshness on sq68 is OK: puppet ran at Wed Sep 12 15:26:14 UTC 2012 [15:26:26] RECOVERY - Puppet freshness on analytics1016 is OK: puppet ran at Wed Sep 12 15:26:15 UTC 2012 [15:26:26] RECOVERY - Puppet freshness on virt4 is OK: puppet ran at Wed Sep 12 15:26:18 UTC 2012 [15:26:27] RECOVERY - Puppet freshness on db25 is OK: puppet ran at Wed Sep 12 15:26:19 UTC 2012 [15:26:27] RECOVERY - Puppet freshness on srv204 is OK: puppet ran at Wed Sep 12 15:26:21 UTC 2012 [15:26:28] RECOVERY - Puppet freshness on mw18 is OK: puppet ran at Wed Sep 12 15:26:23 UTC 2012 [15:26:28] RECOVERY - Puppet freshness on amssq41 is OK: puppet ran at Wed Sep 12 15:26:24 UTC 2012 [15:27:01] RECOVERY - Puppet freshness on amssq44 is OK: puppet ran at Wed Sep 12 15:26:30 UTC 2012 [15:27:01] RECOVERY - Puppet freshness on ssl3003 is OK: puppet ran at Wed Sep 12 15:26:36 UTC 2012 [15:27:01] RECOVERY - Puppet freshness on db1030 is OK: puppet ran at Wed Sep 12 15:26:39 UTC 2012 [15:27:01] RECOVERY - Puppet freshness on analytics1007 is OK: puppet ran at Wed Sep 12 15:26:39 UTC 2012 [15:27:01] RECOVERY - Puppet freshness on amssq51 is OK: puppet ran at Wed Sep 12 15:26:42 UTC 2012 [15:27:01] RECOVERY - Puppet freshness on srv246 is OK: puppet ran at Wed Sep 12 15:26:44 UTC 2012 [15:27:02] RECOVERY - Puppet freshness on db1006 is OK: puppet ran at Wed Sep 12 15:26:53 UTC 2012 [15:27:03] RECOVERY - Puppet freshness on db39 is OK: puppet ran at Wed Sep 12 15:26:53 UTC 2012 [15:27:10] RECOVERY - Puppet freshness on srv299 is OK: puppet ran at Wed Sep 12 15:26:56 UTC 2012 [15:27:10] RECOVERY - Puppet freshness on srv269 is OK: puppet ran at Wed Sep 12 15:26:57 UTC 2012 [15:27:19] RECOVERY - Puppet freshness on pdf1 is OK: puppet ran at Wed Sep 12 15:27:02 UTC 2012 [15:27:19] RECOVERY - Puppet freshness on sq83 is OK: puppet ran at Wed Sep 12 15:27:08 UTC 2012 [15:27:19] RECOVERY - Puppet freshness on niobium is OK: puppet ran at Wed Sep 12 15:27:08 UTC 2012 [15:27:28] RECOVERY - Puppet freshness on mw35 is OK: puppet ran at Wed Sep 12 15:27:11 UTC 2012 [15:27:29] RECOVERY - Puppet freshness on amslvs2 is OK: puppet ran at Wed Sep 12 15:27:11 UTC 2012 [15:27:29] RECOVERY - Puppet freshness on mw22 is OK: puppet ran at Wed Sep 12 15:27:13 UTC 2012 [15:27:29] RECOVERY - Puppet freshness on sq65 is OK: puppet ran at Wed Sep 12 15:27:13 UTC 2012 [15:27:29] RECOVERY - Puppet freshness on ssl1004 is OK: puppet ran at Wed Sep 12 15:27:13 UTC 2012 [15:27:29] RECOVERY - Puppet freshness on cp1008 is OK: puppet ran at Wed Sep 12 15:27:14 UTC 2012 [15:27:37] RECOVERY - Puppet freshness on srv253 is OK: puppet ran at Wed Sep 12 15:27:27 UTC 2012 [15:27:46] RECOVERY - Puppet freshness on srv222 is OK: puppet ran at Wed Sep 12 15:27:31 UTC 2012 [15:27:46] RECOVERY - Puppet freshness on search28 is OK: puppet ran at Wed Sep 12 15:27:36 UTC 2012 [15:27:55] RECOVERY - Puppet freshness on db1024 is OK: puppet ran at Wed Sep 12 15:27:39 UTC 2012 [15:27:55] RECOVERY - Puppet freshness on es1007 is OK: puppet ran at Wed Sep 12 15:27:48 UTC 2012 [15:27:55] RECOVERY - Puppet freshness on analytics1002 is OK: puppet ran at Wed Sep 12 15:27:51 UTC 2012 [15:27:55] RECOVERY - Puppet freshness on chromium is OK: puppet ran at Wed Sep 12 15:27:55 UTC 2012 [15:28:31] RECOVERY - Puppet freshness on virt3 is OK: puppet ran at Wed Sep 12 15:28:02 UTC 2012 [15:28:31] RECOVERY - Puppet freshness on ms-fe1001 is OK: puppet ran at Wed Sep 12 15:28:04 UTC 2012 [15:28:31] RECOVERY - Puppet freshness on mw73 is OK: puppet ran at Wed Sep 12 15:28:07 UTC 2012 [15:28:31] RECOVERY - Puppet freshness on srv202 is OK: puppet ran at Wed Sep 12 15:28:08 UTC 2012 [15:28:31] RECOVERY - Puppet freshness on mw63 is OK: puppet ran at Wed Sep 12 15:28:11 UTC 2012 [15:28:31] RECOVERY - Puppet freshness on db1035 is OK: puppet ran at Wed Sep 12 15:28:18 UTC 2012 [15:28:32] RECOVERY - Puppet freshness on srv203 is OK: puppet ran at Wed Sep 12 15:28:24 UTC 2012 [15:28:40] RECOVERY - Puppet freshness on mw36 is OK: puppet ran at Wed Sep 12 15:28:27 UTC 2012 [15:28:49] RECOVERY - Puppet freshness on mw68 is OK: puppet ran at Wed Sep 12 15:28:36 UTC 2012 [15:28:49] RECOVERY - Puppet freshness on srv249 is OK: puppet ran at Wed Sep 12 15:28:37 UTC 2012 [15:28:49] RECOVERY - Puppet freshness on db56 is OK: puppet ran at Wed Sep 12 15:28:38 UTC 2012 [15:28:58] RECOVERY - Puppet freshness on db51 is OK: puppet ran at Wed Sep 12 15:28:43 UTC 2012 [15:28:58] RECOVERY - Puppet freshness on srv280 is OK: puppet ran at Wed Sep 12 15:28:53 UTC 2012 [15:29:07] RECOVERY - Puppet freshness on nescio is OK: puppet ran at Wed Sep 12 15:28:56 UTC 2012 [15:29:16] RECOVERY - Puppet freshness on mw4 is OK: puppet ran at Wed Sep 12 15:29:00 UTC 2012 [15:29:25] RECOVERY - Puppet freshness on es1005 is OK: puppet ran at Wed Sep 12 15:29:08 UTC 2012 [15:29:25] RECOVERY - Puppet freshness on srv190 is OK: puppet ran at Wed Sep 12 15:29:09 UTC 2012 [15:29:25] RECOVERY - Puppet freshness on amssq59 is OK: puppet ran at Wed Sep 12 15:29:10 UTC 2012 [15:29:25] RECOVERY - Puppet freshness on arsenic is OK: puppet ran at Wed Sep 12 15:29:10 UTC 2012 [15:29:25] RECOVERY - Puppet freshness on ms-be5 is OK: puppet ran at Wed Sep 12 15:29:24 UTC 2012 [15:30:01] RECOVERY - Puppet freshness on search14 is OK: puppet ran at Wed Sep 12 15:29:25 UTC 2012 [15:30:01] RECOVERY - Puppet freshness on virt1 is OK: puppet ran at Wed Sep 12 15:29:26 UTC 2012 [15:30:01] RECOVERY - Puppet freshness on search26 is OK: puppet ran at Wed Sep 12 15:29:28 UTC 2012 [15:30:01] RECOVERY - Puppet freshness on search36 is OK: puppet ran at Wed Sep 12 15:29:30 UTC 2012 [15:30:01] RECOVERY - Puppet freshness on cp1035 is OK: puppet ran at Wed Sep 12 15:29:35 UTC 2012 [15:30:01] RECOVERY - Puppet freshness on wtp1 is OK: puppet ran at Wed Sep 12 15:29:37 UTC 2012 [15:30:02] RECOVERY - Puppet freshness on srv197 is OK: puppet ran at Wed Sep 12 15:29:38 UTC 2012 [15:30:02] RECOVERY - Puppet freshness on db1048 is OK: puppet ran at Wed Sep 12 15:29:40 UTC 2012 [15:30:03] RECOVERY - Puppet freshness on bellin is OK: puppet ran at Wed Sep 12 15:29:43 UTC 2012 [15:30:03] RECOVERY - Puppet freshness on sq48 is OK: puppet ran at Wed Sep 12 15:29:45 UTC 2012 [15:30:04] RECOVERY - Puppet freshness on sq33 is OK: puppet ran at Wed Sep 12 15:29:46 UTC 2012 [15:30:04] RECOVERY - Puppet freshness on mw26 is OK: puppet ran at Wed Sep 12 15:29:51 UTC 2012 [15:30:05] RECOVERY - Puppet freshness on ssl4 is OK: puppet ran at Wed Sep 12 15:29:52 UTC 2012 [15:30:28] RECOVERY - Puppet freshness on srv236 is OK: puppet ran at Wed Sep 12 15:30:04 UTC 2012 [15:30:28] RECOVERY - Puppet freshness on cp1001 is OK: puppet ran at Wed Sep 12 15:30:05 UTC 2012 [15:30:28] RECOVERY - Puppet freshness on mw66 is OK: puppet ran at Wed Sep 12 15:30:07 UTC 2012 [15:30:28] RECOVERY - Puppet freshness on sq43 is OK: puppet ran at Wed Sep 12 15:30:11 UTC 2012 [15:30:28] RECOVERY - Puppet freshness on cp1025 is OK: puppet ran at Wed Sep 12 15:30:16 UTC 2012 [15:30:28] RECOVERY - Puppet freshness on emery is OK: puppet ran at Wed Sep 12 15:30:22 UTC 2012 [15:30:29] RECOVERY - Puppet freshness on srv258 is OK: puppet ran at Wed Sep 12 15:30:22 UTC 2012 [15:30:29] RECOVERY - Puppet freshness on knsq24 is OK: puppet ran at Wed Sep 12 15:30:23 UTC 2012 [15:30:37] RECOVERY - Puppet freshness on search1004 is OK: puppet ran at Wed Sep 12 15:30:25 UTC 2012 [15:30:37] RECOVERY - Puppet freshness on analytics1005 is OK: puppet ran at Wed Sep 12 15:30:27 UTC 2012 [15:30:46] RECOVERY - Puppet freshness on sq86 is OK: puppet ran at Wed Sep 12 15:30:29 UTC 2012 [15:30:46] RECOVERY - Puppet freshness on search1001 is OK: puppet ran at Wed Sep 12 15:30:30 UTC 2012 [15:30:46] RECOVERY - Puppet freshness on es4 is OK: puppet ran at Wed Sep 12 15:30:31 UTC 2012 [15:30:46] RECOVERY - Puppet freshness on mw40 is OK: puppet ran at Wed Sep 12 15:30:33 UTC 2012 [15:30:55] RECOVERY - Puppet freshness on pdf3 is OK: puppet ran at Wed Sep 12 15:30:38 UTC 2012 [15:30:55] RECOVERY - Puppet freshness on dataset2 is OK: puppet ran at Wed Sep 12 15:30:42 UTC 2012 [15:30:55] RECOVERY - Puppet freshness on es2 is OK: puppet ran at Wed Sep 12 15:30:42 UTC 2012 [15:30:55] RECOVERY - Puppet freshness on srv290 is OK: puppet ran at Wed Sep 12 15:30:46 UTC 2012 [15:30:55] RECOVERY - Puppet freshness on analytics1026 is OK: puppet ran at Wed Sep 12 15:30:46 UTC 2012 [15:30:55] RECOVERY - Puppet freshness on hooft is OK: puppet ran at Wed Sep 12 15:30:47 UTC 2012 [15:30:56] RECOVERY - Puppet freshness on knsq22 is OK: puppet ran at Wed Sep 12 15:30:48 UTC 2012 [15:30:56] RECOVERY - Puppet freshness on mw69 is OK: puppet ran at Wed Sep 12 15:30:48 UTC 2012 [15:31:31] RECOVERY - Puppet freshness on cp1010 is OK: puppet ran at Wed Sep 12 15:30:57 UTC 2012 [15:31:31] RECOVERY - Puppet freshness on knsq28 is OK: puppet ran at Wed Sep 12 15:31:02 UTC 2012 [15:31:31] RECOVERY - Puppet freshness on amssq33 is OK: puppet ran at Wed Sep 12 15:31:05 UTC 2012 [15:31:31] RECOVERY - Puppet freshness on srv247 is OK: puppet ran at Wed Sep 12 15:31:16 UTC 2012 [15:31:31] RECOVERY - Puppet freshness on mw1 is OK: puppet ran at Wed Sep 12 15:31:17 UTC 2012 [15:31:31] RECOVERY - Puppet freshness on cp1005 is OK: puppet ran at Wed Sep 12 15:31:23 UTC 2012 [15:31:40] RECOVERY - Puppet freshness on ms-be3 is OK: puppet ran at Wed Sep 12 15:31:27 UTC 2012 [15:31:40] RECOVERY - Puppet freshness on srv234 is OK: puppet ran at Wed Sep 12 15:31:27 UTC 2012 [15:31:40] RECOVERY - Puppet freshness on lvs3 is OK: puppet ran at Wed Sep 12 15:31:29 UTC 2012 [15:31:49] RECOVERY - Puppet freshness on es8 is OK: puppet ran at Wed Sep 12 15:31:32 UTC 2012 [15:31:58] RECOVERY - Puppet freshness on es5 is OK: puppet ran at Wed Sep 12 15:31:41 UTC 2012 [15:31:58] RECOVERY - Puppet freshness on knsq17 is OK: puppet ran at Wed Sep 12 15:31:47 UTC 2012 [15:31:58] RECOVERY - Puppet freshness on cp1011 is OK: puppet ran at Wed Sep 12 15:31:52 UTC 2012 [15:31:58] RECOVERY - Puppet freshness on search1021 is OK: puppet ran at Wed Sep 12 15:31:52 UTC 2012 [15:32:07] RECOVERY - Puppet freshness on amssq32 is OK: puppet ran at Wed Sep 12 15:31:57 UTC 2012 [15:32:07] RECOVERY - Puppet freshness on es1009 is OK: puppet ran at Wed Sep 12 15:31:57 UTC 2012 [15:32:07] RECOVERY - Puppet freshness on ms-be12 is OK: puppet ran at Wed Sep 12 15:31:57 UTC 2012 [15:32:16] RECOVERY - Puppet freshness on sanger is OK: puppet ran at Wed Sep 12 15:32:03 UTC 2012 [15:32:25] RECOVERY - Puppet freshness on srv291 is OK: puppet ran at Wed Sep 12 15:32:11 UTC 2012 [15:32:25] RECOVERY - Puppet freshness on ms1004 is OK: puppet ran at Wed Sep 12 15:32:16 UTC 2012 [15:32:25] RECOVERY - Puppet freshness on mw67 is OK: puppet ran at Wed Sep 12 15:32:18 UTC 2012 [15:32:25] RECOVERY - Puppet freshness on cp1027 is OK: puppet ran at Wed Sep 12 15:32:19 UTC 2012 [15:32:25] RECOVERY - Puppet freshness on knsq16 is OK: puppet ran at Wed Sep 12 15:32:24 UTC 2012 [15:33:01] RECOVERY - Puppet freshness on srv242 is OK: puppet ran at Wed Sep 12 15:32:26 UTC 2012 [15:33:01] RECOVERY - Puppet freshness on srv223 is OK: puppet ran at Wed Sep 12 15:32:29 UTC 2012 [15:33:01] RECOVERY - Puppet freshness on db1007 is OK: puppet ran at Wed Sep 12 15:32:30 UTC 2012 [15:33:01] RECOVERY - Puppet freshness on srv209 is OK: puppet ran at Wed Sep 12 15:32:32 UTC 2012 [15:33:01] RECOVERY - Puppet freshness on es1004 is OK: puppet ran at Wed Sep 12 15:32:33 UTC 2012 [15:33:01] RECOVERY - Puppet freshness on aluminium is OK: puppet ran at Wed Sep 12 15:32:39 UTC 2012 [15:33:02] RECOVERY - Puppet freshness on amslvs4 is OK: puppet ran at Wed Sep 12 15:32:42 UTC 2012 [15:33:02] RECOVERY - Puppet freshness on mw41 is OK: puppet ran at Wed Sep 12 15:32:44 UTC 2012 [15:33:03] RECOVERY - Puppet freshness on search1007 is OK: puppet ran at Wed Sep 12 15:32:46 UTC 2012 [15:33:03] RECOVERY - Puppet freshness on search1008 is OK: puppet ran at Wed Sep 12 15:32:50 UTC 2012 [15:33:04] RECOVERY - Puppet freshness on amssq52 is OK: puppet ran at Wed Sep 12 15:32:51 UTC 2012 [15:33:04] RECOVERY - Puppet freshness on search19 is OK: puppet ran at Wed Sep 12 15:32:51 UTC 2012 [15:33:10] RECOVERY - Puppet freshness on srv218 is OK: puppet ran at Wed Sep 12 15:32:55 UTC 2012 [15:33:10] RECOVERY - Puppet freshness on pc1 is OK: puppet ran at Wed Sep 12 15:32:55 UTC 2012 [15:33:10] RECOVERY - Puppet freshness on db1005 is OK: puppet ran at Wed Sep 12 15:32:57 UTC 2012 [15:33:10] RECOVERY - Puppet freshness on mw30 is OK: puppet ran at Wed Sep 12 15:32:59 UTC 2012 [15:33:10] RECOVERY - Puppet freshness on sodium is OK: puppet ran at Wed Sep 12 15:33:00 UTC 2012 [15:33:19] RECOVERY - Puppet freshness on db1044 is OK: puppet ran at Wed Sep 12 15:33:02 UTC 2012 [15:33:19] RECOVERY - Puppet freshness on db1040 is OK: puppet ran at Wed Sep 12 15:33:03 UTC 2012 [15:33:19] RECOVERY - Puppet freshness on srv279 is OK: puppet ran at Wed Sep 12 15:33:07 UTC 2012 [15:33:28] RECOVERY - Puppet freshness on mw74 is OK: puppet ran at Wed Sep 12 15:33:15 UTC 2012 [15:33:28] RECOVERY - Puppet freshness on db33 is OK: puppet ran at Wed Sep 12 15:33:22 UTC 2012 [15:33:28] RECOVERY - Puppet freshness on srv277 is OK: puppet ran at Wed Sep 12 15:33:22 UTC 2012 [15:33:28] RECOVERY - Puppet freshness on srv216 is OK: puppet ran at Wed Sep 12 15:33:23 UTC 2012 [15:33:37] RECOVERY - Puppet freshness on srv250 is OK: puppet ran at Wed Sep 12 15:33:25 UTC 2012 [15:33:37] RECOVERY - Puppet freshness on sq77 is OK: puppet ran at Wed Sep 12 15:33:27 UTC 2012 [15:33:46] RECOVERY - Puppet freshness on gurvin is OK: puppet ran at Wed Sep 12 15:33:33 UTC 2012 [15:33:46] RECOVERY - Puppet freshness on srv251 is OK: puppet ran at Wed Sep 12 15:33:33 UTC 2012 [15:33:55] RECOVERY - Puppet freshness on srv200 is OK: puppet ran at Wed Sep 12 15:33:38 UTC 2012 [15:33:55] RECOVERY - Puppet freshness on mw31 is OK: puppet ran at Wed Sep 12 15:33:43 UTC 2012 [15:33:55] RECOVERY - Puppet freshness on manutius is OK: puppet ran at Wed Sep 12 15:33:43 UTC 2012 [15:33:55] RECOVERY - Puppet freshness on srv296 is OK: puppet ran at Wed Sep 12 15:33:46 UTC 2012 [15:33:55] RECOVERY - Puppet freshness on tarin is OK: puppet ran at Wed Sep 12 15:33:47 UTC 2012 [15:33:55] RECOVERY - Puppet freshness on amssq35 is OK: puppet ran at Wed Sep 12 15:33:51 UTC 2012 [15:33:56] RECOVERY - Puppet freshness on amslvs1 is OK: puppet ran at Wed Sep 12 15:33:52 UTC 2012 [15:33:57] RECOVERY - Puppet freshness on db50 is OK: puppet ran at Wed Sep 12 15:33:53 UTC 2012 [15:33:57] RECOVERY - Puppet freshness on srv198 is OK: puppet ran at Wed Sep 12 15:33:54 UTC 2012 [15:34:04] RECOVERY - Puppet freshness on cp1006 is OK: puppet ran at Wed Sep 12 15:33:55 UTC 2012 [15:34:13] RECOVERY - Puppet freshness on yvon is OK: puppet ran at Wed Sep 12 15:33:56 UTC 2012 [15:34:22] RECOVERY - Puppet freshness on srv255 is OK: puppet ran at Wed Sep 12 15:34:07 UTC 2012 [15:34:22] RECOVERY - Puppet freshness on db1042 is OK: puppet ran at Wed Sep 12 15:34:12 UTC 2012 [15:34:31] RECOVERY - Puppet freshness on mw38 is OK: puppet ran at Wed Sep 12 15:34:16 UTC 2012 [15:34:31] RECOVERY - Puppet freshness on analytics1003 is OK: puppet ran at Wed Sep 12 15:34:20 UTC 2012 [15:34:31] RECOVERY - Puppet freshness on search1017 is OK: puppet ran at Wed Sep 12 15:34:21 UTC 2012 [15:34:31] RECOVERY - Puppet freshness on sq78 is OK: puppet ran at Wed Sep 12 15:34:21 UTC 2012 [15:34:31] RECOVERY - Puppet freshness on db1010 is OK: puppet ran at Wed Sep 12 15:34:24 UTC 2012 [15:34:58] RECOVERY - Puppet freshness on amssq48 is OK: puppet ran at Wed Sep 12 15:34:34 UTC 2012 [15:34:58] RECOVERY - Puppet freshness on mw43 is OK: puppet ran at Wed Sep 12 15:34:39 UTC 2012 [15:34:58] RECOVERY - Puppet freshness on mw29 is OK: puppet ran at Wed Sep 12 15:34:42 UTC 2012 [15:34:58] RECOVERY - Puppet freshness on lvs1001 is OK: puppet ran at Wed Sep 12 15:34:42 UTC 2012 [15:34:58] RECOVERY - Puppet freshness on srv282 is OK: puppet ran at Wed Sep 12 15:34:45 UTC 2012 [15:35:07] RECOVERY - Puppet freshness on analytics1019 is OK: puppet ran at Wed Sep 12 15:34:57 UTC 2012 [15:35:16] RECOVERY - Puppet freshness on sockpuppet is OK: puppet ran at Wed Sep 12 15:35:02 UTC 2012 [15:35:16] RECOVERY - Puppet freshness on mw37 is OK: puppet ran at Wed Sep 12 15:35:04 UTC 2012 [15:35:16] RECOVERY - Puppet freshness on amssq62 is OK: puppet ran at Wed Sep 12 15:35:04 UTC 2012 [15:35:16] RECOVERY - Puppet freshness on search13 is OK: puppet ran at Wed Sep 12 15:35:07 UTC 2012 [15:35:25] RECOVERY - Puppet freshness on amssq43 is OK: puppet ran at Wed Sep 12 15:35:11 UTC 2012 [15:35:25] RECOVERY - Puppet freshness on amssq45 is OK: puppet ran at Wed Sep 12 15:35:20 UTC 2012 [15:35:25] RECOVERY - Puppet freshness on db58 is OK: puppet ran at Wed Sep 12 15:35:21 UTC 2012 [15:35:25] RECOVERY - Puppet freshness on mw51 is OK: puppet ran at Wed Sep 12 15:35:21 UTC 2012 [15:35:25] RECOVERY - Puppet freshness on db29 is OK: puppet ran at Wed Sep 12 15:35:23 UTC 2012 [15:35:25] RECOVERY - Puppet freshness on cp1030 is OK: puppet ran at Wed Sep 12 15:35:23 UTC 2012 [15:35:26] RECOVERY - Puppet freshness on srv278 is OK: puppet ran at Wed Sep 12 15:35:24 UTC 2012 [15:36:01] RECOVERY - Puppet freshness on db47 is OK: puppet ran at Wed Sep 12 15:35:26 UTC 2012 [15:36:01] RECOVERY - Puppet freshness on mw71 is OK: puppet ran at Wed Sep 12 15:35:27 UTC 2012 [15:36:01] RECOVERY - Puppet freshness on ms-fe1004 is OK: puppet ran at Wed Sep 12 15:35:27 UTC 2012 [15:36:01] RECOVERY - Puppet freshness on db1001 is OK: puppet ran at Wed Sep 12 15:35:35 UTC 2012 [15:36:01] RECOVERY - Puppet freshness on snapshot1 is OK: puppet ran at Wed Sep 12 15:35:37 UTC 2012 [15:36:01] RECOVERY - Puppet freshness on mw70 is OK: puppet ran at Wed Sep 12 15:35:37 UTC 2012 [15:36:02] RECOVERY - Puppet freshness on mw55 is OK: puppet ran at Wed Sep 12 15:35:46 UTC 2012 [15:36:02] RECOVERY - Puppet freshness on mw8 is OK: puppet ran at Wed Sep 12 15:35:47 UTC 2012 [15:36:03] RECOVERY - Puppet freshness on srv230 is OK: puppet ran at Wed Sep 12 15:35:49 UTC 2012 [15:36:10] RECOVERY - Puppet freshness on sq57 is OK: puppet ran at Wed Sep 12 15:35:56 UTC 2012 [15:36:10] RECOVERY - Puppet freshness on srv274 is OK: puppet ran at Wed Sep 12 15:36:00 UTC 2012 [15:36:19] RECOVERY - Puppet freshness on srv235 is OK: puppet ran at Wed Sep 12 15:36:04 UTC 2012 [15:36:19] RECOVERY - Puppet freshness on srv276 is OK: puppet ran at Wed Sep 12 15:36:04 UTC 2012 [15:36:19] RECOVERY - Puppet freshness on labstore3 is OK: puppet ran at Wed Sep 12 15:36:05 UTC 2012 [15:36:19] RECOVERY - Puppet freshness on mw13 is OK: puppet ran at Wed Sep 12 15:36:07 UTC 2012 [15:36:19] RECOVERY - Puppet freshness on srv254 is OK: puppet ran at Wed Sep 12 15:36:08 UTC 2012 [15:36:19] RECOVERY - Puppet freshness on srv227 is OK: puppet ran at Wed Sep 12 15:36:08 UTC 2012 [15:36:28] RECOVERY - Puppet freshness on srv300 is OK: puppet ran at Wed Sep 12 15:36:14 UTC 2012 [15:36:28] RECOVERY - Puppet freshness on bast1001 is OK: puppet ran at Wed Sep 12 15:36:14 UTC 2012 [15:36:28] RECOVERY - Puppet freshness on db17 is OK: puppet ran at Wed Sep 12 15:36:14 UTC 2012 [15:36:28] RECOVERY - Puppet freshness on virt8 is OK: puppet ran at Wed Sep 12 15:36:18 UTC 2012 [15:36:28] RECOVERY - Puppet freshness on srv213 is OK: puppet ran at Wed Sep 12 15:36:20 UTC 2012 [15:36:28] RECOVERY - Puppet freshness on db59 is OK: puppet ran at Wed Sep 12 15:36:24 UTC 2012 [15:36:37] RECOVERY - Puppet freshness on kaulen is OK: puppet ran at Wed Sep 12 15:36:25 UTC 2012 [15:36:46] RECOVERY - Puppet freshness on srv248 is OK: puppet ran at Wed Sep 12 15:36:29 UTC 2012 [15:36:46] RECOVERY - Puppet freshness on lvs6 is OK: puppet ran at Wed Sep 12 15:36:29 UTC 2012 [15:36:46] RECOVERY - Puppet freshness on sq81 is OK: puppet ran at Wed Sep 12 15:36:32 UTC 2012 [15:36:46] RECOVERY - Puppet freshness on ms5 is OK: puppet ran at Wed Sep 12 15:36:34 UTC 2012 [15:36:46] RECOVERY - Puppet freshness on cp1003 is OK: puppet ran at Wed Sep 12 15:36:36 UTC 2012 [15:36:55] RECOVERY - Puppet freshness on sq61 is OK: puppet ran at Wed Sep 12 15:36:37 UTC 2012 [15:36:55] RECOVERY - Puppet freshness on amssq60 is OK: puppet ran at Wed Sep 12 15:36:38 UTC 2012 [15:36:55] RECOVERY - Puppet freshness on locke is OK: puppet ran at Wed Sep 12 15:36:40 UTC 2012 [15:36:55] RECOVERY - Puppet freshness on nfs2 is OK: puppet ran at Wed Sep 12 15:36:40 UTC 2012 [15:36:55] RECOVERY - Puppet freshness on srv221 is OK: puppet ran at Wed Sep 12 15:36:42 UTC 2012 [15:36:55] RECOVERY - Puppet freshness on db26 is OK: puppet ran at Wed Sep 12 15:36:45 UTC 2012 [15:36:56] RECOVERY - Puppet freshness on marmontel is OK: puppet ran at Wed Sep 12 15:36:50 UTC 2012 [15:37:31] RECOVERY - Puppet freshness on es1010 is OK: puppet ran at Wed Sep 12 15:36:56 UTC 2012 [15:37:31] RECOVERY - Puppet freshness on sq41 is OK: puppet ran at Wed Sep 12 15:37:00 UTC 2012 [15:37:31] RECOVERY - Puppet freshness on sq79 is OK: puppet ran at Wed Sep 12 15:37:07 UTC 2012 [15:37:31] RECOVERY - Puppet freshness on analytics1015 is OK: puppet ran at Wed Sep 12 15:37:07 UTC 2012 [15:37:31] RECOVERY - Puppet freshness on amssq42 is OK: puppet ran at Wed Sep 12 15:37:08 UTC 2012 [15:37:31] RECOVERY - Puppet freshness on ekrem is OK: puppet ran at Wed Sep 12 15:37:16 UTC 2012 [15:37:32] RECOVERY - Puppet freshness on sq72 is OK: puppet ran at Wed Sep 12 15:37:17 UTC 2012 [15:37:32] RECOVERY - Puppet freshness on snapshot1003 is OK: puppet ran at Wed Sep 12 15:37:20 UTC 2012 [15:37:33] RECOVERY - Puppet freshness on knsq26 is OK: puppet ran at Wed Sep 12 15:37:22 UTC 2012 [15:37:33] RECOVERY - Puppet freshness on search1006 is OK: puppet ran at Wed Sep 12 15:37:25 UTC 2012 [15:37:40] RECOVERY - Puppet freshness on db1029 is OK: puppet ran at Wed Sep 12 15:37:26 UTC 2012 [15:37:49] RECOVERY - Puppet freshness on ms-fe1003 is OK: puppet ran at Wed Sep 12 15:37:34 UTC 2012 [15:37:49] RECOVERY - Puppet freshness on mw61 is OK: puppet ran at Wed Sep 12 15:37:35 UTC 2012 [15:37:49] RECOVERY - Puppet freshness on srv191 is OK: puppet ran at Wed Sep 12 15:37:39 UTC 2012 [15:37:49] RECOVERY - Puppet freshness on search1020 is OK: puppet ran at Wed Sep 12 15:37:40 UTC 2012 [15:37:58] RECOVERY - Puppet freshness on ssl1001 is OK: puppet ran at Wed Sep 12 15:37:49 UTC 2012 [15:37:58] RECOVERY - Puppet freshness on search1012 is OK: puppet ran at Wed Sep 12 15:37:53 UTC 2012 [15:37:58] RECOVERY - Puppet freshness on lvs1005 is OK: puppet ran at Wed Sep 12 15:37:55 UTC 2012 [15:38:07] RECOVERY - Puppet freshness on erzurumi is OK: puppet ran at Wed Sep 12 15:37:56 UTC 2012 [15:38:07] RECOVERY - Puppet freshness on lvs2 is OK: puppet ran at Wed Sep 12 15:37:56 UTC 2012 [15:38:16] RECOVERY - Puppet freshness on virt1000 is OK: puppet ran at Wed Sep 12 15:37:58 UTC 2012 [15:38:16] RECOVERY - Puppet freshness on mw16 is OK: puppet ran at Wed Sep 12 15:38:06 UTC 2012 [15:38:25] RECOVERY - Puppet freshness on virt5 is OK: puppet ran at Wed Sep 12 15:38:09 UTC 2012 [15:38:25] RECOVERY - Puppet freshness on srv238 is OK: puppet ran at Wed Sep 12 15:38:10 UTC 2012 [15:38:25] RECOVERY - Puppet freshness on stat1 is OK: puppet ran at Wed Sep 12 15:38:14 UTC 2012 [15:38:25] RECOVERY - Puppet freshness on srv259 is OK: puppet ran at Wed Sep 12 15:38:18 UTC 2012 [15:38:25] RECOVERY - Puppet freshness on ms-be1 is OK: puppet ran at Wed Sep 12 15:38:20 UTC 2012 [15:38:26] RECOVERY - Puppet freshness on ms-be1002 is OK: puppet ran at Wed Sep 12 15:38:22 UTC 2012 [15:39:01] RECOVERY - Puppet freshness on search1014 is OK: puppet ran at Wed Sep 12 15:38:29 UTC 2012 [15:39:01] RECOVERY - Puppet freshness on srv273 is OK: puppet ran at Wed Sep 12 15:38:31 UTC 2012 [15:39:01] RECOVERY - Puppet freshness on db38 is OK: puppet ran at Wed Sep 12 15:38:38 UTC 2012 [15:39:01] RECOVERY - Puppet freshness on search23 is OK: puppet ran at Wed Sep 12 15:38:43 UTC 2012 [15:39:01] RECOVERY - Puppet freshness on mw53 is OK: puppet ran at Wed Sep 12 15:38:45 UTC 2012 [15:39:01] RECOVERY - Puppet freshness on db35 is OK: puppet ran at Wed Sep 12 15:38:46 UTC 2012 [15:39:02] RECOVERY - Puppet freshness on sq85 is OK: puppet ran at Wed Sep 12 15:38:50 UTC 2012 [15:39:02] RECOVERY - Puppet freshness on snapshot2 is OK: puppet ran at Wed Sep 12 15:38:54 UTC 2012 [15:39:10] RECOVERY - Puppet freshness on lvs1006 is OK: puppet ran at Wed Sep 12 15:38:55 UTC 2012 [15:39:19] RECOVERY - Puppet freshness on db1045 is OK: puppet ran at Wed Sep 12 15:39:02 UTC 2012 [15:39:19] RECOVERY - Puppet freshness on db10 is OK: puppet ran at Wed Sep 12 15:39:03 UTC 2012 [15:39:19] RECOVERY - Puppet freshness on amssq39 is OK: puppet ran at Wed Sep 12 15:39:04 UTC 2012 [15:39:19] RECOVERY - Puppet freshness on ssl3 is OK: puppet ran at Wed Sep 12 15:39:04 UTC 2012 [15:39:19] RECOVERY - Puppet freshness on sq70 is OK: puppet ran at Wed Sep 12 15:39:06 UTC 2012 [15:39:19] RECOVERY - Puppet freshness on srv263 is OK: puppet ran at Wed Sep 12 15:39:07 UTC 2012 [15:39:29] RECOVERY - Puppet freshness on srv271 is OK: puppet ran at Wed Sep 12 15:39:12 UTC 2012 [15:39:29] RECOVERY - Puppet freshness on virt6 is OK: puppet ran at Wed Sep 12 15:39:14 UTC 2012 [15:39:29] RECOVERY - Puppet freshness on sq42 is OK: puppet ran at Wed Sep 12 15:39:22 UTC 2012 [15:39:29] RECOVERY - Puppet freshness on srv272 is OK: puppet ran at Wed Sep 12 15:39:23 UTC 2012 [15:39:29] RECOVERY - Puppet freshness on ms-be2 is OK: puppet ran at Wed Sep 12 15:39:23 UTC 2012 [15:39:29] RECOVERY - Puppet freshness on db42 is OK: puppet ran at Wed Sep 12 15:39:23 UTC 2012 [15:39:46] RECOVERY - Puppet freshness on labstore1 is OK: puppet ran at Wed Sep 12 15:39:35 UTC 2012 [15:39:55] RECOVERY - Puppet freshness on williams is OK: puppet ran at Wed Sep 12 15:39:39 UTC 2012 [15:39:55] RECOVERY - Puppet freshness on mw44 is OK: puppet ran at Wed Sep 12 15:39:42 UTC 2012 [15:39:55] RECOVERY - Puppet freshness on sq75 is OK: puppet ran at Wed Sep 12 15:39:43 UTC 2012 [15:39:55] RECOVERY - Puppet freshness on virt2 is OK: puppet ran at Wed Sep 12 15:39:51 UTC 2012 [15:39:55] RECOVERY - Puppet freshness on db53 is OK: puppet ran at Wed Sep 12 15:39:52 UTC 2012 [15:39:55] RECOVERY - Puppet freshness on virt1005 is OK: puppet ran at Wed Sep 12 15:39:52 UTC 2012 [15:39:56] RECOVERY - Puppet freshness on ms-be1007 is OK: puppet ran at Wed Sep 12 15:39:52 UTC 2012 [15:39:56] RECOVERY - Puppet freshness on hooper is OK: puppet ran at Wed Sep 12 15:39:54 UTC 2012 [15:40:31] RECOVERY - Puppet freshness on searchidx1001 is OK: puppet ran at Wed Sep 12 15:39:55 UTC 2012 [15:40:31] RECOVERY - Puppet freshness on db1049 is OK: puppet ran at Wed Sep 12 15:40:00 UTC 2012 [15:40:31] RECOVERY - Puppet freshness on mw14 is OK: puppet ran at Wed Sep 12 15:40:00 UTC 2012 [15:40:31] RECOVERY - Puppet freshness on lvs5 is OK: puppet ran at Wed Sep 12 15:40:01 UTC 2012 [15:40:31] RECOVERY - Puppet freshness on srv261 is OK: puppet ran at Wed Sep 12 15:40:06 UTC 2012 [15:40:31] RECOVERY - Puppet freshness on analytics1023 is OK: puppet ran at Wed Sep 12 15:40:07 UTC 2012 [15:40:32] RECOVERY - Puppet freshness on db36 is OK: puppet ran at Wed Sep 12 15:40:07 UTC 2012 [15:40:32] RECOVERY - Puppet freshness on ssl1003 is OK: puppet ran at Wed Sep 12 15:40:09 UTC 2012 [15:40:33] RECOVERY - Puppet freshness on mw3 is OK: puppet ran at Wed Sep 12 15:40:09 UTC 2012 [15:40:33] RECOVERY - Puppet freshness on stafford is OK: puppet ran at Wed Sep 12 15:40:14 UTC 2012 [15:40:34] RECOVERY - Puppet freshness on srv294 is OK: puppet ran at Wed Sep 12 15:40:16 UTC 2012 [15:40:34] RECOVERY - Puppet freshness on db11 is OK: puppet ran at Wed Sep 12 15:40:20 UTC 2012 [15:40:35] RECOVERY - Puppet freshness on db1022 is OK: puppet ran at Wed Sep 12 15:40:21 UTC 2012 [15:40:35] RECOVERY - Puppet freshness on cp1034 is OK: puppet ran at Wed Sep 12 15:40:24 UTC 2012 [15:40:36] RECOVERY - Puppet freshness on amssq55 is OK: puppet ran at Wed Sep 12 15:40:25 UTC 2012 [15:40:40] RECOVERY - Puppet freshness on srv264 is OK: puppet ran at Wed Sep 12 15:40:30 UTC 2012 [15:40:49] RECOVERY - Puppet freshness on dobson is OK: puppet ran at Wed Sep 12 15:40:34 UTC 2012 [15:40:49] RECOVERY - Puppet freshness on srv210 is OK: puppet ran at Wed Sep 12 15:40:36 UTC 2012 [15:40:49] RECOVERY - Puppet freshness on mw12 is OK: puppet ran at Wed Sep 12 15:40:37 UTC 2012 [15:40:49] RECOVERY - Puppet freshness on srv219 is OK: puppet ran at Wed Sep 12 15:40:39 UTC 2012 [15:40:58] RECOVERY - Puppet freshness on srv214 is OK: puppet ran at Wed Sep 12 15:40:40 UTC 2012 [15:40:58] RECOVERY - Puppet freshness on cp1036 is OK: puppet ran at Wed Sep 12 15:40:48 UTC 2012 [15:40:58] RECOVERY - Puppet freshness on lvs1 is OK: puppet ran at Wed Sep 12 15:40:49 UTC 2012 [15:40:58] RECOVERY - Puppet freshness on srv289 is OK: puppet ran at Wed Sep 12 15:40:50 UTC 2012 [15:41:07] RECOVERY - Puppet freshness on db22 is OK: puppet ran at Wed Sep 12 15:40:55 UTC 2012 [15:41:16] RECOVERY - Puppet freshness on srv226 is OK: puppet ran at Wed Sep 12 15:40:58 UTC 2012 [15:41:16] RECOVERY - Puppet freshness on mw27 is OK: puppet ran at Wed Sep 12 15:41:01 UTC 2012 [15:41:25] RECOVERY - Puppet freshness on db1003 is OK: puppet ran at Wed Sep 12 15:41:08 UTC 2012 [15:41:25] RECOVERY - Puppet freshness on amssq58 is OK: puppet ran at Wed Sep 12 15:41:10 UTC 2012 [15:41:25] RECOVERY - Puppet freshness on db1012 is OK: puppet ran at Wed Sep 12 15:41:12 UTC 2012 [15:41:25] RECOVERY - Puppet freshness on cp1020 is OK: puppet ran at Wed Sep 12 15:41:17 UTC 2012 [15:41:25] RECOVERY - Puppet freshness on srv225 is OK: puppet ran at Wed Sep 12 15:41:19 UTC 2012 [15:41:25] RECOVERY - Puppet freshness on sq82 is OK: puppet ran at Wed Sep 12 15:41:24 UTC 2012 [15:42:01] RECOVERY - Puppet freshness on mw57 is OK: puppet ran at Wed Sep 12 15:41:25 UTC 2012 [15:42:01] RECOVERY - Puppet freshness on ms1001 is OK: puppet ran at Wed Sep 12 15:41:28 UTC 2012 [15:42:01] RECOVERY - Puppet freshness on cp1023 is OK: puppet ran at Wed Sep 12 15:41:29 UTC 2012 [15:42:01] RECOVERY - Puppet freshness on lvs1002 is OK: puppet ran at Wed Sep 12 15:41:30 UTC 2012 [15:42:01] RECOVERY - Puppet freshness on srv286 is OK: puppet ran at Wed Sep 12 15:41:30 UTC 2012 [15:42:01] RECOVERY - Puppet freshness on db43 is OK: puppet ran at Wed Sep 12 15:41:31 UTC 2012 [15:42:02] RECOVERY - Puppet freshness on sq53 is OK: puppet ran at Wed Sep 12 15:41:36 UTC 2012 [15:42:02] RECOVERY - Puppet freshness on srv208 is OK: puppet ran at Wed Sep 12 15:41:37 UTC 2012 [15:42:03] RECOVERY - Puppet freshness on ms-fe3 is OK: puppet ran at Wed Sep 12 15:41:37 UTC 2012 [15:42:03] RECOVERY - Puppet freshness on sq49 is OK: puppet ran at Wed Sep 12 15:41:41 UTC 2012 [15:42:04] RECOVERY - Puppet freshness on srv240 is OK: puppet ran at Wed Sep 12 15:41:43 UTC 2012 [15:42:04] RECOVERY - Puppet freshness on mw9 is OK: puppet ran at Wed Sep 12 15:41:44 UTC 2012 [15:42:05] RECOVERY - Puppet freshness on sq74 is OK: puppet ran at Wed Sep 12 15:41:47 UTC 2012 [15:42:05] RECOVERY - Puppet freshness on srv265 is OK: puppet ran at Wed Sep 12 15:41:48 UTC 2012 [15:42:06] RECOVERY - Puppet freshness on knsq20 is OK: puppet ran at Wed Sep 12 15:41:48 UTC 2012 [15:42:06] RECOVERY - Puppet freshness on knsq29 is OK: puppet ran at Wed Sep 12 15:41:48 UTC 2012 [15:42:07] RECOVERY - Puppet freshness on srv252 is OK: puppet ran at Wed Sep 12 15:41:49 UTC 2012 [15:42:07] RECOVERY - Puppet freshness on knsq21 is OK: puppet ran at Wed Sep 12 15:41:52 UTC 2012 [15:42:08] RECOVERY - Puppet freshness on analytics1008 is OK: puppet ran at Wed Sep 12 15:41:53 UTC 2012 [15:42:08] RECOVERY - Puppet freshness on mw39 is OK: puppet ran at Wed Sep 12 15:41:54 UTC 2012 [15:42:28] RECOVERY - Puppet freshness on db1021 is OK: puppet ran at Wed Sep 12 15:42:04 UTC 2012 [15:42:28] RECOVERY - Puppet freshness on amssq53 is OK: puppet ran at Wed Sep 12 15:42:05 UTC 2012 [15:42:28] RECOVERY - Puppet freshness on cp1018 is OK: puppet ran at Wed Sep 12 15:42:06 UTC 2012 [15:42:28] RECOVERY - Puppet freshness on sq58 is OK: puppet ran at Wed Sep 12 15:42:08 UTC 2012 [15:42:28] RECOVERY - Puppet freshness on srv231 is OK: puppet ran at Wed Sep 12 15:42:08 UTC 2012 [15:42:28] RECOVERY - Puppet freshness on oxygen is OK: puppet ran at Wed Sep 12 15:42:09 UTC 2012 [15:42:29] RECOVERY - Puppet freshness on search22 is OK: puppet ran at Wed Sep 12 15:42:16 UTC 2012 [15:42:29] RECOVERY - Puppet freshness on analytics1004 is OK: puppet ran at Wed Sep 12 15:42:18 UTC 2012 [15:42:30] RECOVERY - Puppet freshness on mw45 is OK: puppet ran at Wed Sep 12 15:42:21 UTC 2012 [15:42:31] RECOVERY - Puppet freshness on mw2 is OK: puppet ran at Wed Sep 12 15:42:21 UTC 2012 [15:42:31] RECOVERY - Puppet freshness on srv224 is OK: puppet ran at Wed Sep 12 15:42:22 UTC 2012 [15:42:37] RECOVERY - Puppet freshness on sq73 is OK: puppet ran at Wed Sep 12 15:42:26 UTC 2012 [15:42:46] RECOVERY - Puppet freshness on sq84 is OK: puppet ran at Wed Sep 12 15:42:31 UTC 2012 [15:42:46] RECOVERY - Puppet freshness on mw58 is OK: puppet ran at Wed Sep 12 15:42:31 UTC 2012 [15:42:46] RECOVERY - Puppet freshness on search33 is OK: puppet ran at Wed Sep 12 15:42:32 UTC 2012 [15:42:55] RECOVERY - Puppet freshness on analytics1006 is OK: puppet ran at Wed Sep 12 15:42:42 UTC 2012 [15:42:55] RECOVERY - Puppet freshness on db1015 is OK: puppet ran at Wed Sep 12 15:42:43 UTC 2012 [15:42:55] RECOVERY - Puppet freshness on srv245 is OK: puppet ran at Wed Sep 12 15:42:47 UTC 2012 [15:42:56] RECOVERY - Puppet freshness on db1020 is OK: puppet ran at Wed Sep 12 15:42:50 UTC 2012 [15:43:31] RECOVERY - Puppet freshness on mw24 is OK: puppet ran at Wed Sep 12 15:42:59 UTC 2012 [15:43:31] RECOVERY - Puppet freshness on cp1014 is OK: puppet ran at Wed Sep 12 15:43:03 UTC 2012 [15:43:31] RECOVERY - Puppet freshness on helium is OK: puppet ran at Wed Sep 12 15:43:06 UTC 2012 [15:43:31] RECOVERY - Puppet freshness on srv192 is OK: puppet ran at Wed Sep 12 15:43:07 UTC 2012 [15:43:31] RECOVERY - Puppet freshness on nfs1 is OK: puppet ran at Wed Sep 12 15:43:09 UTC 2012 [15:43:31] RECOVERY - Puppet freshness on analytics1022 is OK: puppet ran at Wed Sep 12 15:43:09 UTC 2012 [15:43:32] RECOVERY - Puppet freshness on pdf2 is OK: puppet ran at Wed Sep 12 15:43:13 UTC 2012 [15:43:32] RECOVERY - Puppet freshness on sq59 is OK: puppet ran at Wed Sep 12 15:43:21 UTC 2012 [15:43:33] RECOVERY - Puppet freshness on mw20 is OK: puppet ran at Wed Sep 12 15:43:23 UTC 2012 [15:43:33] RECOVERY - Puppet freshness on cp1044 is OK: puppet ran at Wed Sep 12 15:43:24 UTC 2012 [15:43:34] PROBLEM - Puppet freshness on ms-be7 is CRITICAL: Puppet has not run in the last 10 hours [15:43:40] RECOVERY - Puppet freshness on mw48 is OK: puppet ran at Wed Sep 12 15:43:25 UTC 2012 [15:43:49] RECOVERY - Puppet freshness on ms-fe2 is OK: puppet ran at Wed Sep 12 15:43:33 UTC 2012 [15:43:49] RECOVERY - Puppet freshness on search1015 is OK: puppet ran at Wed Sep 12 15:43:34 UTC 2012 [15:43:49] RECOVERY - Puppet freshness on ms-be1006 is OK: puppet ran at Wed Sep 12 15:43:35 UTC 2012 [15:43:49] RECOVERY - Puppet freshness on cp1017 is OK: puppet ran at Wed Sep 12 15:43:35 UTC 2012 [15:43:49] RECOVERY - Puppet freshness on db48 is OK: puppet ran at Wed Sep 12 15:43:38 UTC 2012 [15:43:49] RECOVERY - Puppet freshness on db1046 is OK: puppet ran at Wed Sep 12 15:43:39 UTC 2012 [15:43:58] RECOVERY - Puppet freshness on search1023 is OK: puppet ran at Wed Sep 12 15:43:41 UTC 2012 [15:43:58] RECOVERY - Puppet freshness on db1019 is OK: puppet ran at Wed Sep 12 15:43:44 UTC 2012 [15:43:58] RECOVERY - Puppet freshness on ocg1 is OK: puppet ran at Wed Sep 12 15:43:46 UTC 2012 [15:43:58] RECOVERY - Puppet freshness on db1041 is OK: puppet ran at Wed Sep 12 15:43:46 UTC 2012 [15:43:58] RECOVERY - Puppet freshness on cp1013 is OK: puppet ran at Wed Sep 12 15:43:52 UTC 2012 [15:44:07] RECOVERY - Puppet freshness on search1003 is OK: puppet ran at Wed Sep 12 15:43:55 UTC 2012 [15:44:07] RECOVERY - Puppet freshness on cp1031 is OK: puppet ran at Wed Sep 12 15:43:58 UTC 2012 [15:44:16] RECOVERY - Puppet freshness on mw65 is OK: puppet ran at Wed Sep 12 15:43:59 UTC 2012 [15:44:16] RECOVERY - Puppet freshness on carbon is OK: puppet ran at Wed Sep 12 15:44:02 UTC 2012 [15:44:16] RECOVERY - Puppet freshness on ms-be1005 is OK: puppet ran at Wed Sep 12 15:44:07 UTC 2012 [15:44:25] RECOVERY - Puppet freshness on db1004 is OK: puppet ran at Wed Sep 12 15:44:10 UTC 2012 [15:44:25] RECOVERY - Puppet freshness on db1002 is OK: puppet ran at Wed Sep 12 15:44:11 UTC 2012 [15:44:25] RECOVERY - Puppet freshness on db1050 is OK: puppet ran at Wed Sep 12 15:44:14 UTC 2012 [15:44:25] RECOVERY - Puppet freshness on knsq27 is OK: puppet ran at Wed Sep 12 15:44:17 UTC 2012 [15:44:25] RECOVERY - Puppet freshness on sq71 is OK: puppet ran at Wed Sep 12 15:44:18 UTC 2012 [15:44:25] RECOVERY - Puppet freshness on srv292 is OK: puppet ran at Wed Sep 12 15:44:20 UTC 2012 [15:44:26] RECOVERY - Puppet freshness on amssq31 is OK: puppet ran at Wed Sep 12 15:44:22 UTC 2012 [15:45:01] RECOVERY - Puppet freshness on sq37 is OK: puppet ran at Wed Sep 12 15:44:26 UTC 2012 [15:45:01] RECOVERY - Puppet freshness on db1025 is OK: puppet ran at Wed Sep 12 15:44:28 UTC 2012 [15:45:01] RECOVERY - Puppet freshness on db1031 is OK: puppet ran at Wed Sep 12 15:44:30 UTC 2012 [15:45:01] RECOVERY - Puppet freshness on srv220 is OK: puppet ran at Wed Sep 12 15:44:30 UTC 2012 [15:45:01] RECOVERY - Puppet freshness on amssq40 is OK: puppet ran at Wed Sep 12 15:44:32 UTC 2012 [15:45:01] RECOVERY - Puppet freshness on srv301 is OK: puppet ran at Wed Sep 12 15:44:33 UTC 2012 [15:45:02] RECOVERY - Puppet freshness on mw21 is OK: puppet ran at Wed Sep 12 15:44:34 UTC 2012 [15:45:02] RECOVERY - Puppet freshness on amssq56 is OK: puppet ran at Wed Sep 12 15:44:36 UTC 2012 [15:45:03] RECOVERY - Puppet freshness on ersch is OK: puppet ran at Wed Sep 12 15:44:36 UTC 2012 [15:45:03] RECOVERY - Puppet freshness on db1013 is OK: puppet ran at Wed Sep 12 15:44:37 UTC 2012 [15:45:04] RECOVERY - Puppet freshness on searchidx2 is OK: puppet ran at Wed Sep 12 15:44:40 UTC 2012 [15:45:04] RECOVERY - Puppet freshness on srv215 is OK: puppet ran at Wed Sep 12 15:44:43 UTC 2012 [15:45:05] RECOVERY - Puppet freshness on fluorine is OK: puppet ran at Wed Sep 12 15:44:44 UTC 2012 [15:45:05] RECOVERY - Puppet freshness on analytics1012 is OK: puppet ran at Wed Sep 12 15:44:45 UTC 2012 [15:45:06] RECOVERY - Puppet freshness on srv268 is OK: puppet ran at Wed Sep 12 15:44:49 UTC 2012 [15:45:06] RECOVERY - Puppet freshness on hume is OK: puppet ran at Wed Sep 12 15:44:51 UTC 2012 [15:45:07] RECOVERY - Puppet freshness on es10 is OK: puppet ran at Wed Sep 12 15:44:53 UTC 2012 [15:45:10] RECOVERY - Puppet freshness on db1008 is OK: puppet ran at Wed Sep 12 15:45:01 UTC 2012 [15:45:19] RECOVERY - Puppet freshness on amssq49 is OK: puppet ran at Wed Sep 12 15:45:06 UTC 2012 [15:45:28] RECOVERY - Puppet freshness on ms-fe4 is OK: puppet ran at Wed Sep 12 15:45:10 UTC 2012 [15:45:28] RECOVERY - Puppet freshness on db1028 is OK: puppet ran at Wed Sep 12 15:45:11 UTC 2012 [15:45:28] RECOVERY - Puppet freshness on ms-be11 is OK: puppet ran at Wed Sep 12 15:45:12 UTC 2012 [15:45:28] RECOVERY - Puppet freshness on ssl1 is OK: puppet ran at Wed Sep 12 15:45:14 UTC 2012 [15:45:28] RECOVERY - Puppet freshness on amslvs3 is OK: puppet ran at Wed Sep 12 15:45:20 UTC 2012 [15:45:29] RECOVERY - Puppet freshness on cp1043 is OK: puppet ran at Wed Sep 12 15:45:20 UTC 2012 [15:45:37] RECOVERY - Puppet freshness on snapshot3 is OK: puppet ran at Wed Sep 12 15:45:26 UTC 2012 [15:45:37] RECOVERY - Puppet freshness on gallium is OK: puppet ran at Wed Sep 12 15:45:26 UTC 2012 [15:45:37] RECOVERY - Puppet freshness on ssl3001 is OK: puppet ran at Wed Sep 12 15:45:27 UTC 2012 [15:45:46] RECOVERY - Puppet freshness on es9 is OK: puppet ran at Wed Sep 12 15:45:29 UTC 2012 [15:45:46] RECOVERY - Puppet freshness on db1017 is OK: puppet ran at Wed Sep 12 15:45:33 UTC 2012 [15:45:46] RECOVERY - Puppet freshness on sq69 is OK: puppet ran at Wed Sep 12 15:45:33 UTC 2012 [15:45:46] RECOVERY - Puppet freshness on sq62 is OK: puppet ran at Wed Sep 12 15:45:36 UTC 2012 [15:45:55] RECOVERY - Puppet freshness on amssq50 is OK: puppet ran at Wed Sep 12 15:45:41 UTC 2012 [15:45:55] RECOVERY - Puppet freshness on cp1009 is OK: puppet ran at Wed Sep 12 15:45:42 UTC 2012 [15:45:55] RECOVERY - Puppet freshness on srv298 is OK: puppet ran at Wed Sep 12 15:45:43 UTC 2012 [15:45:55] RECOVERY - Puppet freshness on snapshot4 is OK: puppet ran at Wed Sep 12 15:45:43 UTC 2012 [15:45:55] RECOVERY - Puppet freshness on db1014 is OK: puppet ran at Wed Sep 12 15:45:44 UTC 2012 [15:45:56] RECOVERY - Puppet freshness on es6 is OK: puppet ran at Wed Sep 12 15:45:50 UTC 2012 [15:45:56] RECOVERY - Puppet freshness on db37 is OK: puppet ran at Wed Sep 12 15:45:50 UTC 2012 [15:45:56] RECOVERY - Puppet freshness on cp1004 is OK: puppet ran at Wed Sep 12 15:45:51 UTC 2012 [15:45:57] RECOVERY - Puppet freshness on db57 is OK: puppet ran at Wed Sep 12 15:45:51 UTC 2012 [15:46:31] RECOVERY - Puppet freshness on analytics1025 is OK: puppet ran at Wed Sep 12 15:46:02 UTC 2012 [15:46:31] RECOVERY - Puppet freshness on capella is OK: puppet ran at Wed Sep 12 15:46:02 UTC 2012 [15:46:31] RECOVERY - Puppet freshness on analytics1017 is OK: puppet ran at Wed Sep 12 15:46:09 UTC 2012 [15:46:31] RECOVERY - Puppet freshness on ms-be1003 is OK: puppet ran at Wed Sep 12 15:46:09 UTC 2012 [15:46:31] RECOVERY - Puppet freshness on ms-be1009 is OK: puppet ran at Wed Sep 12 15:46:13 UTC 2012 [15:46:31] RECOVERY - Puppet freshness on analytics1014 is OK: puppet ran at Wed Sep 12 15:46:17 UTC 2012 [15:46:32] RECOVERY - Puppet freshness on cp1028 is OK: puppet ran at Wed Sep 12 15:46:18 UTC 2012 [15:46:40] RECOVERY - Puppet freshness on db1034 is OK: puppet ran at Wed Sep 12 15:46:27 UTC 2012 [15:46:40] RECOVERY - Puppet freshness on srv288 is OK: puppet ran at Wed Sep 12 15:46:28 UTC 2012 [15:46:40] RECOVERY - Puppet freshness on analytics1018 is OK: puppet ran at Wed Sep 12 15:46:30 UTC 2012 [15:46:49] RECOVERY - Puppet freshness on virt1008 is OK: puppet ran at Wed Sep 12 15:46:36 UTC 2012 [15:46:49] RECOVERY - Puppet freshness on srv241 is OK: puppet ran at Wed Sep 12 15:46:37 UTC 2012 [15:46:49] RECOVERY - Puppet freshness on db1009 is OK: puppet ran at Wed Sep 12 15:46:39 UTC 2012 [15:46:49] RECOVERY - Puppet freshness on spence is OK: puppet ran at Wed Sep 12 15:46:39 UTC 2012 [15:46:49] RECOVERY - Puppet freshness on professor is OK: puppet ran at Wed Sep 12 15:46:39 UTC 2012 [15:46:58] RECOVERY - Puppet freshness on es1002 is OK: puppet ran at Wed Sep 12 15:46:43 UTC 2012 [15:46:58] RECOVERY - Puppet freshness on srv228 is OK: puppet ran at Wed Sep 12 15:46:44 UTC 2012 [15:46:58] RECOVERY - Puppet freshness on ms-be1012 is OK: puppet ran at Wed Sep 12 15:46:49 UTC 2012 [15:46:58] RECOVERY - Puppet freshness on cp1041 is OK: puppet ran at Wed Sep 12 15:46:53 UTC 2012 [15:47:07] RECOVERY - Puppet freshness on search18 is OK: puppet ran at Wed Sep 12 15:46:57 UTC 2012 [15:47:16] RECOVERY - Puppet freshness on yttrium is OK: puppet ran at Wed Sep 12 15:47:03 UTC 2012 [15:47:16] RECOVERY - Puppet freshness on sq52 is OK: puppet ran at Wed Sep 12 15:47:04 UTC 2012 [15:47:16] RECOVERY - Puppet freshness on db40 is OK: puppet ran at Wed Sep 12 15:47:04 UTC 2012 [15:47:25] RECOVERY - Puppet freshness on mw59 is OK: puppet ran at Wed Sep 12 15:47:09 UTC 2012 [15:47:25] RECOVERY - Puppet freshness on db1018 is OK: puppet ran at Wed Sep 12 15:47:16 UTC 2012 [15:47:25] RECOVERY - Puppet freshness on cp1032 is OK: puppet ran at Wed Sep 12 15:47:18 UTC 2012 [15:47:25] RECOVERY - Puppet freshness on ms-be1010 is OK: puppet ran at Wed Sep 12 15:47:18 UTC 2012 [15:47:25] RECOVERY - Puppet freshness on lvs4 is OK: puppet ran at Wed Sep 12 15:47:20 UTC 2012 [15:47:25] RECOVERY - Puppet freshness on mw19 is OK: puppet ran at Wed Sep 12 15:47:24 UTC 2012 [15:48:01] RECOVERY - Puppet freshness on ssl3002 is OK: puppet ran at Wed Sep 12 15:47:27 UTC 2012 [15:48:01] RECOVERY - Puppet freshness on cp1002 is OK: puppet ran at Wed Sep 12 15:47:36 UTC 2012 [15:48:01] RECOVERY - Puppet freshness on strontium is OK: puppet ran at Wed Sep 12 15:47:36 UTC 2012 [15:48:01] RECOVERY - Puppet freshness on srv297 is OK: puppet ran at Wed Sep 12 15:47:38 UTC 2012 [15:48:01] RECOVERY - Puppet freshness on hydrogen is OK: puppet ran at Wed Sep 12 15:47:39 UTC 2012 [15:48:01] RECOVERY - Puppet freshness on ms-be1008 is OK: puppet ran at Wed Sep 12 15:47:47 UTC 2012 [15:48:02] RECOVERY - Puppet freshness on cp1029 is OK: puppet ran at Wed Sep 12 15:47:48 UTC 2012 [15:48:02] RECOVERY - Puppet freshness on amssq47 is OK: puppet ran at Wed Sep 12 15:47:48 UTC 2012 [15:48:03] RECOVERY - Puppet freshness on snapshot1002 is OK: puppet ran at Wed Sep 12 15:47:50 UTC 2012 [15:48:10] RECOVERY - Puppet freshness on ms-be1001 is OK: puppet ran at Wed Sep 12 15:47:59 UTC 2012 [15:48:10] RECOVERY - Puppet freshness on db45 is OK: puppet ran at Wed Sep 12 15:48:00 UTC 2012 [15:48:21] RECOVERY - Puppet freshness on search34 is OK: puppet ran at Wed Sep 12 15:48:01 UTC 2012 [15:48:29] RECOVERY - Puppet freshness on lvs1003 is OK: puppet ran at Wed Sep 12 15:48:10 UTC 2012 [15:48:29] RECOVERY - Puppet freshness on srv237 is OK: puppet ran at Wed Sep 12 15:48:14 UTC 2012 [15:48:29] RECOVERY - Puppet freshness on mw6 is OK: puppet ran at Wed Sep 12 15:48:16 UTC 2012 [15:48:37] RECOVERY - Puppet freshness on srv287 is OK: puppet ran at Wed Sep 12 15:48:27 UTC 2012 [15:48:37] RECOVERY - Puppet freshness on analytics1027 is OK: puppet ran at Wed Sep 12 15:48:27 UTC 2012 [15:48:37] RECOVERY - Puppet freshness on sq51 is OK: puppet ran at Wed Sep 12 15:48:28 UTC 2012 [15:48:37] RECOVERY - Puppet freshness on db1026 is OK: puppet ran at Wed Sep 12 15:48:28 UTC 2012 [15:48:46] RECOVERY - Puppet freshness on ms-be9 is OK: puppet ran at Wed Sep 12 15:48:32 UTC 2012 [15:48:55] RECOVERY - Puppet freshness on db1043 is OK: puppet ran at Wed Sep 12 15:48:38 UTC 2012 [15:48:55] RECOVERY - Puppet freshness on cp1033 is OK: puppet ran at Wed Sep 12 15:48:41 UTC 2012 [15:48:55] RECOVERY - Puppet freshness on grosley is OK: puppet ran at Wed Sep 12 15:48:45 UTC 2012 [15:48:55] RECOVERY - Puppet freshness on sq44 is OK: puppet ran at Wed Sep 12 15:48:47 UTC 2012 [15:48:55] RECOVERY - Puppet freshness on db1027 is OK: puppet ran at Wed Sep 12 15:48:49 UTC 2012 [15:48:55] RECOVERY - Puppet freshness on srv193 is OK: puppet ran at Wed Sep 12 15:48:49 UTC 2012 [15:48:56] RECOVERY - Puppet freshness on labstore4 is OK: puppet ran at Wed Sep 12 15:48:52 UTC 2012 [15:48:56] RECOVERY - Puppet freshness on virt7 is OK: puppet ran at Wed Sep 12 15:48:52 UTC 2012 [15:48:57] RECOVERY - Puppet freshness on sq56 is OK: puppet ran at Wed Sep 12 15:48:53 UTC 2012 [15:48:57] RECOVERY - Puppet freshness on mw42 is OK: puppet ran at Wed Sep 12 15:48:53 UTC 2012 [15:49:22] RECOVERY - Puppet freshness on db52 is OK: puppet ran at Wed Sep 12 15:48:56 UTC 2012 [15:49:22] RECOVERY - Puppet freshness on search1005 is OK: puppet ran at Wed Sep 12 15:48:58 UTC 2012 [15:49:22] RECOVERY - Puppet freshness on dataset1001 is OK: puppet ran at Wed Sep 12 15:48:59 UTC 2012 [15:49:22] RECOVERY - Puppet freshness on search1018 is OK: puppet ran at Wed Sep 12 15:49:00 UTC 2012 [15:49:22] RECOVERY - Puppet freshness on cp3002 is OK: puppet ran at Wed Sep 12 15:49:02 UTC 2012 [15:49:22] RECOVERY - Puppet freshness on db34 is OK: puppet ran at Wed Sep 12 15:49:05 UTC 2012 [15:49:23] RECOVERY - Puppet freshness on amssq61 is OK: puppet ran at Wed Sep 12 15:49:12 UTC 2012 [15:49:23] RECOVERY - Puppet freshness on srv256 is OK: puppet ran at Wed Sep 12 15:49:12 UTC 2012 [15:49:24] RECOVERY - Puppet freshness on analytics1021 is OK: puppet ran at Wed Sep 12 15:49:15 UTC 2012 [15:49:31] RECOVERY - Puppet freshness on srv260 is OK: puppet ran at Wed Sep 12 15:49:15 UTC 2012 [15:49:31] RECOVERY - Puppet freshness on mw10 is OK: puppet ran at Wed Sep 12 15:49:16 UTC 2012 [15:49:31] RECOVERY - Puppet freshness on mw54 is OK: puppet ran at Wed Sep 12 15:49:16 UTC 2012 [15:49:31] RECOVERY - Puppet freshness on search29 is OK: puppet ran at Wed Sep 12 15:49:17 UTC 2012 [15:49:31] RECOVERY - Puppet freshness on search1010 is OK: puppet ran at Wed Sep 12 15:49:20 UTC 2012 [15:49:40] RECOVERY - Puppet freshness on db1039 is OK: puppet ran at Wed Sep 12 15:49:25 UTC 2012 [15:49:40] RECOVERY - Puppet freshness on search35 is OK: puppet ran at Wed Sep 12 15:49:25 UTC 2012 [15:49:40] RECOVERY - Puppet freshness on tridge is OK: puppet ran at Wed Sep 12 15:49:26 UTC 2012 [15:49:51] RECOVERY - Puppet freshness on srv201 is OK: puppet ran at Wed Sep 12 15:49:34 UTC 2012 [15:49:51] RECOVERY - Puppet freshness on cp1012 is OK: puppet ran at Wed Sep 12 15:49:34 UTC 2012 [15:49:51] RECOVERY - Puppet freshness on ssl2 is OK: puppet ran at Wed Sep 12 15:49:36 UTC 2012 [15:49:59] RECOVERY - Puppet freshness on srv239 is OK: puppet ran at Wed Sep 12 15:49:41 UTC 2012 [15:49:59] RECOVERY - Puppet freshness on sq50 is OK: puppet ran at Wed Sep 12 15:49:42 UTC 2012 [15:49:59] RECOVERY - Puppet freshness on search20 is OK: puppet ran at Wed Sep 12 15:49:47 UTC 2012 [15:49:59] RECOVERY - Puppet freshness on search17 is OK: puppet ran at Wed Sep 12 15:49:48 UTC 2012 [15:49:59] RECOVERY - Puppet freshness on search1009 is OK: puppet ran at Wed Sep 12 15:49:52 UTC 2012 [15:49:59] RECOVERY - Puppet freshness on mw7 is OK: puppet ran at Wed Sep 12 15:49:54 UTC 2012 [15:50:07] RECOVERY - Puppet freshness on mw49 is OK: puppet ran at Wed Sep 12 15:49:55 UTC 2012 [15:50:07] RECOVERY - Puppet freshness on cp1015 is OK: puppet ran at Wed Sep 12 15:49:57 UTC 2012 [15:50:07] RECOVERY - Puppet freshness on amssq57 is OK: puppet ran at Wed Sep 12 15:49:57 UTC 2012 [15:50:07] RECOVERY - Puppet freshness on analytics1010 is OK: puppet ran at Wed Sep 12 15:49:58 UTC 2012 [15:50:07] RECOVERY - Puppet freshness on virt0 is OK: puppet ran at Wed Sep 12 15:49:58 UTC 2012 [15:50:16] RECOVERY - Puppet freshness on search16 is OK: puppet ran at Wed Sep 12 15:50:06 UTC 2012 [15:50:16] RECOVERY - Puppet freshness on sq36 is OK: puppet ran at Wed Sep 12 15:50:06 UTC 2012 [15:50:25] RECOVERY - Puppet freshness on db1016 is OK: puppet ran at Wed Sep 12 15:50:10 UTC 2012 [15:50:25] RECOVERY - Puppet freshness on srv207 is OK: puppet ran at Wed Sep 12 15:50:11 UTC 2012 [15:50:25] RECOVERY - Puppet freshness on ms6 is OK: puppet ran at Wed Sep 12 15:50:13 UTC 2012 [15:50:25] RECOVERY - Puppet freshness on srv293 is OK: puppet ran at Wed Sep 12 15:50:14 UTC 2012 [15:50:25] RECOVERY - Puppet freshness on amssq37 is OK: puppet ran at Wed Sep 12 15:50:16 UTC 2012 [15:50:26] RECOVERY - Puppet freshness on srv211 is OK: puppet ran at Wed Sep 12 15:50:17 UTC 2012 [15:50:28] RECOVERY - Puppet freshness on knsq19 is OK: puppet ran at Wed Sep 12 15:50:18 UTC 2012 [15:50:28] RECOVERY - Puppet freshness on mw47 is OK: puppet ran at Wed Sep 12 15:50:19 UTC 2012 [15:50:34] RECOVERY - Puppet freshness on mw62 is OK: puppet ran at Wed Sep 12 15:50:27 UTC 2012 [16:51:38] RECOVERY - Apache HTTP on srv195 is OK: HTTP OK - HTTP/1.1 301 Moved Permanently - 0.037 second response time [16:51:47] RECOVERY - Apache HTTP on srv198 is OK: HTTP OK - HTTP/1.1 301 Moved Permanently - 0.040 second response time [16:52:32] RECOVERY - Apache HTTP on srv196 is OK: HTTP OK - HTTP/1.1 301 Moved Permanently - 0.313 second response time [16:54:02] RECOVERY - Apache HTTP on srv197 is OK: HTTP OK - HTTP/1.1 301 Moved Permanently - 0.044 second response time [17:02:35] PROBLEM - Puppet freshness on neon is CRITICAL: Puppet has not run in the last 10 hours [17:25:09] New patchset: Diederik; "Migrate bi_filter and lp_filter to udp-filter" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/23518 [17:26:02] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/23518 [17:38:24] !log removing mw1-16 from apaches pool to reimage them as precise jobrunner boxes [17:38:33] Logged the message, notpeter [17:53:22] !log re-adding srv194-199 to apaches pool [17:53:31] Logged the message, notpeter [18:09:03] !log adding mw59 to apaches pool [18:09:12] Logged the message, notpeter [18:19:06] PROBLEM - Packetloss_Average on oxygen is CRITICAL: CRITICAL: packet_loss_average is 14.3658198319 (gt 8.0) [18:28:06] PROBLEM - Puppet freshness on magnesium is CRITICAL: Puppet has not run in the last 10 hours [18:28:06] PROBLEM - Puppet freshness on zinc is CRITICAL: Puppet has not run in the last 10 hours [18:32:27] RECOVERY - Packetloss_Average on oxygen is OK: OK: packet_loss_average is -0.0383338983051 [18:33:30] PROBLEM - Apache HTTP on mw1 is CRITICAL: Connection refused [18:33:30] PROBLEM - Apache HTTP on mw10 is CRITICAL: Connection refused [18:33:31] PROBLEM - Apache HTTP on mw4 is CRITICAL: Connection refused [18:33:31] PROBLEM - Apache HTTP on mw2 is CRITICAL: Connection refused [18:33:31] PROBLEM - Apache HTTP on mw5 is CRITICAL: Connection refused [18:33:31] PROBLEM - Apache HTTP on mw3 is CRITICAL: Connection refused [18:33:31] PROBLEM - Apache HTTP on mw6 is CRITICAL: Connection refused [18:33:39] PROBLEM - Apache HTTP on mw16 is CRITICAL: Connection refused [18:34:06] PROBLEM - Apache HTTP on mw9 is CRITICAL: Connection refused [18:34:06] PROBLEM - Apache HTTP on mw7 is CRITICAL: Connection refused [18:34:36] New patchset: Pyoungmeister; "making mw1-16 jobrunner only boxes. also, upgrading to precise" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/23536 [18:34:51] PROBLEM - Apache HTTP on mw15 is CRITICAL: Connection refused [18:35:00] PROBLEM - Apache HTTP on mw14 is CRITICAL: Connection refused [18:35:00] PROBLEM - Apache HTTP on mw12 is CRITICAL: Connection refused [18:35:00] PROBLEM - Apache HTTP on mw11 is CRITICAL: Connection refused [18:35:00] PROBLEM - Apache HTTP on mw13 is CRITICAL: Connection refused [18:35:38] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/23536 [18:37:06] PROBLEM - Apache HTTP on mw8 is CRITICAL: Connection refused [18:39:05] Change merged: Pyoungmeister; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/23536 [18:51:25] New patchset: Dzahn; "wikistats - adjust mw versions for stats table colors" [operations/debs/wikistats] (master) - https://gerrit.wikimedia.org/r/23538 [18:52:56] Change merged: Dzahn; [operations/debs/wikistats] (master) - https://gerrit.wikimedia.org/r/23538 [19:00:01] !log jenkins: hashar installed a snapshot of Gerrit Trigger plugin [19:00:11] Logged the message, Master [19:04:51] PROBLEM - SSH on mw1 is CRITICAL: Connection refused [19:05:18] PROBLEM - Memcached on mw1 is CRITICAL: Connection refused [19:05:23] !log reimaging mw1-16 [19:05:32] Logged the message, notpeter [19:11:53] PROBLEM - SSH on mw2 is CRITICAL: Connection refused [19:13:23] RECOVERY - SSH on mw1 is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1 (protocol 2.0) [19:14:26] PROBLEM - Memcached on mw2 is CRITICAL: Connection refused [19:22:23] PROBLEM - Host mw6 is DOWN: PING CRITICAL - Packet loss = 100% [19:22:23] PROBLEM - Host mw7 is DOWN: PING CRITICAL - Packet loss = 100% [19:22:32] PROBLEM - Host mw4 is DOWN: PING CRITICAL - Packet loss = 100% [19:22:32] PROBLEM - Host mw5 is DOWN: PING CRITICAL - Packet loss = 100% [19:22:32] PROBLEM - Host mw10 is DOWN: PING CRITICAL - Packet loss = 100% [19:22:41] PROBLEM - Host mw11 is DOWN: PING CRITICAL - Packet loss = 100% [19:22:41] PROBLEM - Host mw13 is DOWN: PING CRITICAL - Packet loss = 100% [19:22:50] PROBLEM - Host mw9 is DOWN: PING CRITICAL - Packet loss = 100% [19:23:10] that's me. that's ok [19:26:08] PROBLEM - SSH on mw1 is CRITICAL: Connection refused [19:26:44] PROBLEM - Memcached on mw12 is CRITICAL: Connection refused [19:26:44] PROBLEM - Memcached on mw15 is CRITICAL: Connection refused [19:26:53] PROBLEM - Memcached on mw3 is CRITICAL: Connection refused [19:27:02] PROBLEM - Memcached on mw16 is CRITICAL: Connection refused [19:27:11] PROBLEM - SSH on mw3 is CRITICAL: Connection refused [19:27:20] PROBLEM - SSH on mw12 is CRITICAL: Connection refused [19:27:20] PROBLEM - SSH on mw14 is CRITICAL: Connection refused [19:27:29] PROBLEM - Memcached on mw14 is CRITICAL: Connection refused [19:27:29] PROBLEM - SSH on mw15 is CRITICAL: Connection refused [19:27:38] PROBLEM - SSH on mw8 is CRITICAL: Connection refused [19:27:38] PROBLEM - SSH on mw16 is CRITICAL: Connection refused [19:27:56] RECOVERY - Host mw7 is UP: PING OK - Packet loss = 0%, RTA = 0.26 ms [19:27:56] RECOVERY - Host mw6 is UP: PING OK - Packet loss = 0%, RTA = 0.29 ms [19:28:05] RECOVERY - Host mw5 is UP: PING OK - Packet loss = 0%, RTA = 0.33 ms [19:28:05] RECOVERY - Host mw4 is UP: PING OK - Packet loss = 0%, RTA = 0.33 ms [19:28:05] RECOVERY - Host mw10 is UP: PING OK - Packet loss = 0%, RTA = 0.23 ms [19:28:14] RECOVERY - Host mw11 is UP: PING OK - Packet loss = 0%, RTA = 0.23 ms [19:28:14] RECOVERY - Host mw13 is UP: PING OK - Packet loss = 0%, RTA = 0.21 ms [19:28:23] RECOVERY - Host mw9 is UP: PING OK - Packet loss = 0%, RTA = 1.99 ms [19:31:05] PROBLEM - Puppet freshness on manganese is CRITICAL: Puppet has not run in the last 10 hours [19:31:14] PROBLEM - Memcached on mw10 is CRITICAL: Connection refused [19:31:14] PROBLEM - SSH on mw7 is CRITICAL: Connection refused [19:31:14] PROBLEM - Memcached on mw6 is CRITICAL: Connection refused [19:31:23] PROBLEM - Memcached on mw11 is CRITICAL: Connection refused [19:31:32] PROBLEM - Memcached on mw5 is CRITICAL: Connection refused [19:31:32] PROBLEM - Memcached on mw4 is CRITICAL: Connection refused [19:31:32] PROBLEM - Memcached on mw13 is CRITICAL: Connection refused [19:31:50] PROBLEM - SSH on mw11 is CRITICAL: Connection refused [19:31:59] PROBLEM - SSH on mw10 is CRITICAL: Connection refused [19:31:59] PROBLEM - SSH on mw5 is CRITICAL: Connection refused [19:32:08] PROBLEM - SSH on mw4 is CRITICAL: Connection refused [19:32:08] PROBLEM - SSH on mw6 is CRITICAL: Connection refused [19:32:08] PROBLEM - Memcached on mw7 is CRITICAL: Connection refused [19:32:08] PROBLEM - SSH on mw9 is CRITICAL: Connection refused [19:32:26] PROBLEM - SSH on mw13 is CRITICAL: Connection refused [19:33:02] PROBLEM - Memcached on mw9 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:33:47] RECOVERY - SSH on mw2 is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1 (protocol 2.0) [19:33:56] RECOVERY - SSH on mw1 is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1 (protocol 2.0) [19:34:50] RECOVERY - SSH on mw11 is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1 (protocol 2.0) [19:34:59] RECOVERY - SSH on mw3 is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1 (protocol 2.0) [19:34:59] RECOVERY - SSH on mw14 is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1 (protocol 2.0) [19:34:59] RECOVERY - SSH on mw12 is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1 (protocol 2.0) [19:35:08] RECOVERY - SSH on mw10 is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1 (protocol 2.0) [19:35:08] RECOVERY - SSH on mw5 is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1 (protocol 2.0) [19:35:17] RECOVERY - SSH on mw8 is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1 (protocol 2.0) [19:35:17] RECOVERY - SSH on mw9 is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1 (protocol 2.0) [19:35:17] RECOVERY - SSH on mw16 is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1 (protocol 2.0) [19:35:26] RECOVERY - SSH on mw6 is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1 (protocol 2.0) [19:35:26] RECOVERY - SSH on mw4 is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1 (protocol 2.0) [19:35:35] RECOVERY - SSH on mw15 is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1 (protocol 2.0) [19:35:35] RECOVERY - SSH on mw13 is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1 (protocol 2.0) [19:36:02] RECOVERY - SSH on mw7 is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1 (protocol 2.0) [19:36:29] PROBLEM - Packetloss_Average on oxygen is CRITICAL: CRITICAL: packet_loss_average is 62.255705678 (gt 8.0) [19:41:45] RECOVERY - Packetloss_Average on oxygen is OK: OK: packet_loss_average is 0.0 [19:43:50] PROBLEM - NTP on mw1 is CRITICAL: NTP CRITICAL: No response from NTP server [19:43:59] PROBLEM - Puppet freshness on ms-be6 is CRITICAL: Puppet has not run in the last 10 hours [19:44:35] PROBLEM - NTP on mw15 is CRITICAL: NTP CRITICAL: No response from NTP server [19:45:20] PROBLEM - NTP on mw16 is CRITICAL: NTP CRITICAL: No response from NTP server [19:45:20] PROBLEM - NTP on mw12 is CRITICAL: NTP CRITICAL: No response from NTP server [19:46:05] PROBLEM - NTP on mw8 is CRITICAL: NTP CRITICAL: No response from NTP server [19:50:35] PROBLEM - NTP on mw9 is CRITICAL: NTP CRITICAL: No response from NTP server [19:55:41] PROBLEM - NTP on mw2 is CRITICAL: NTP CRITICAL: No response from NTP server [19:56:08] PROBLEM - NTP on mw3 is CRITICAL: NTP CRITICAL: No response from NTP server [19:57:20] PROBLEM - NTP on mw14 is CRITICAL: NTP CRITICAL: No response from NTP server [19:57:20] PROBLEM - NTP on mw13 is CRITICAL: NTP CRITICAL: No response from NTP server [19:57:20] PROBLEM - NTP on mw10 is CRITICAL: NTP CRITICAL: No response from NTP server [19:57:20] PROBLEM - NTP on mw5 is CRITICAL: NTP CRITICAL: No response from NTP server [19:57:20] PROBLEM - NTP on mw6 is CRITICAL: NTP CRITICAL: No response from NTP server [19:57:20] PROBLEM - NTP on mw11 is CRITICAL: NTP CRITICAL: No response from NTP server [19:57:20] PROBLEM - NTP on mw4 is CRITICAL: NTP CRITICAL: No response from NTP server [19:58:14] PROBLEM - NTP on mw7 is CRITICAL: NTP CRITICAL: No response from NTP server [21:06:08] New patchset: Diederik; "Filter creditcard numbers, email addresses and social security numbers from searchterms Provide unittest case for said functionality" [operations/debs/lucene-search-2] (master) - https://gerrit.wikimedia.org/r/23583 [21:12:21] Change merged: Jgreen; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/23518 [21:20:33] New patchset: Ryan Lane; "Initial checkin of salt puppet module" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/23587 [21:21:18] PROBLEM - Varnish traffic logger on cp1027 is CRITICAL: PROCS CRITICAL: 2 processes with command name varnishncsa [21:21:18] PROBLEM - Varnish traffic logger on cp1021 is CRITICAL: PROCS CRITICAL: 2 processes with command name varnishncsa [21:21:18] PROBLEM - Varnish traffic logger on cp1025 is CRITICAL: PROCS CRITICAL: 2 processes with command name varnishncsa [21:21:18] PROBLEM - Varnish traffic logger on cp1023 is CRITICAL: PROCS CRITICAL: 2 processes with command name varnishncsa [21:21:18] PROBLEM - Varnish traffic logger on cp1043 is CRITICAL: PROCS CRITICAL: 2 processes with command name varnishncsa [21:21:18] PROBLEM - Varnish traffic logger on cp1041 is CRITICAL: PROCS CRITICAL: 2 processes with command name varnishncsa [21:21:31] Change merged: Reedy; [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/23287 [21:21:31] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/23587 [21:22:03] PROBLEM - Varnish traffic logger on cp1024 is CRITICAL: PROCS CRITICAL: 2 processes with command name varnishncsa [21:22:03] PROBLEM - Varnish traffic logger on cp1028 is CRITICAL: PROCS CRITICAL: 2 processes with command name varnishncsa [21:22:03] PROBLEM - Varnish traffic logger on cp1022 is CRITICAL: PROCS CRITICAL: 2 processes with command name varnishncsa [21:22:03] PROBLEM - Varnish traffic logger on cp1026 is CRITICAL: PROCS CRITICAL: 2 processes with command name varnishncsa [21:22:03] PROBLEM - Varnish traffic logger on cp1042 is CRITICAL: PROCS CRITICAL: 2 processes with command name varnishncsa [21:22:48] RECOVERY - Varnish traffic logger on cp1023 is OK: PROCS OK: 3 processes with command name varnishncsa [21:25:48] RECOVERY - Varnish traffic logger on cp1043 is OK: PROCS OK: 3 processes with command name varnishncsa [21:26:27] Hi. [21:26:33] RECOVERY - Varnish traffic logger on cp1028 is OK: PROCS OK: 3 processes with command name varnishncsa [21:26:36] Should uploads be disabled on Commons until this inconsistent state issue is resolved? [21:27:08] https://bugzilla.wikimedia.org/show_bug.cgi?id=39221 [21:29:15] RECOVERY - Varnish traffic logger on cp1041 is OK: PROCS OK: 3 processes with command name varnishncsa [21:29:28] paravoid: ^^ [21:29:42] PROBLEM - Packetloss_Average on oxygen is CRITICAL: CRITICAL: packet_loss_average is 17.7038214286 (gt 8.0) [21:29:54] Brooke: i haven't seen aaron all day. (although i've not been paying attention consistently). had an unrelated (i think) question about [21:30:16] New patchset: Pyoungmeister; "correctly check for undef var" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/23588 [21:31:13] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/23588 [21:31:20] Change merged: Pyoungmeister; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/23588 [21:34:03] RECOVERY - Varnish traffic logger on cp1024 is OK: PROCS OK: 3 processes with command name varnishncsa [21:35:33] RECOVERY - Varnish traffic logger on cp1022 is OK: PROCS OK: 3 processes with command name varnishncsa [21:35:51] RECOVERY - Varnish traffic logger on cp1042 is OK: PROCS OK: 3 processes with command name varnishncsa [21:36:23] New patchset: Pyoungmeister; "need class default as well." [operations/puppet] (production) - https://gerrit.wikimedia.org/r/23589 [21:37:12] RECOVERY - Varnish traffic logger on cp1026 is OK: PROCS OK: 3 processes with command name varnishncsa [21:37:19] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/23589 [21:37:48] RECOVERY - Varnish traffic logger on cp1021 is OK: PROCS OK: 3 processes with command name varnishncsa [21:37:55] Change merged: Pyoungmeister; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/23589 [21:40:48] RECOVERY - Varnish traffic logger on cp1025 is OK: PROCS OK: 3 processes with command name varnishncsa [21:41:55] New patchset: Ryan Lane; "Initial checkin of salt puppet module" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/23587 [21:42:36] RECOVERY - Packetloss_Average on oxygen is OK: OK: packet_loss_average is 0.0 [21:42:59] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/23587 [21:43:48] RECOVERY - Varnish traffic logger on cp1027 is OK: PROCS OK: 3 processes with command name varnishncsa [21:44:13] 15 packets transmitted, 8 packets received, 46.7% packet loss [21:44:14] round-trip min/avg/max/stddev = 38.737/861.890/1615.757/584.689 ms [21:44:23] we did it.. we found wifi that makes the office look good. [21:46:31] New patchset: Ryan Lane; "Add salt roles for master and minion" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/23591 [21:47:28] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/23591 [21:48:27] RECOVERY - mysqld processes on es5 is OK: PROCS OK: 1 process with command name mysqld [21:51:33] New patchset: Pyoungmeister; "defining ganglia group/octet for dedicated jobrunner cluster" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/23593 [21:52:35] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/23593 [21:52:54] Change merged: Pyoungmeister; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/23593 [21:54:34] New patchset: Ryan Lane; "Add salt roles for master and minion" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/23591 [21:55:37] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/23591 [22:00:38] New patchset: Pyoungmeister; "jobrunner.pp: hack to deal with migration to mediawiki module" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/23597 [22:01:30] New patchset: RobH; "Added in pmtpa row C and eqiad row C" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/23598 [22:02:23] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/23597 [22:02:24] Change merged: Pyoungmeister; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/23597 [22:02:24] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/23598 [22:08:56] RECOVERY - Apache HTTP on mw1 is OK: HTTP OK HTTP/1.1 200 OK - 454 bytes in 0.001 seconds [22:12:13] New patchset: Asher; "10g mac for mc2" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/23602 [22:13:18] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/23602 [22:18:02] oh yay, the 10g nic comes up as eth2 and the installer fails [22:18:05] RECOVERY - Host pay-lvs1001 is UP: PING WARNING - Packet loss = 50%, RTA = 26.93 ms [22:18:41] Change merged: Asher; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/23602 [22:21:08] New patchset: Pyoungmeister; "disable apache on jobrunner boxes" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/23603 [22:22:05] New patchset: Hashar; "install puppet-lint gem on contint host" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/23604 [22:22:26] Ryan_Lane: https://gerrit.wikimedia.org/r/23604 :) [22:23:00] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/23603 [22:23:01] New patchset: Pyoungmeister; "disable apache on jobrunner boxes" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/23603 [22:24:01] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/23604 [22:24:01] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/23603 [22:24:43] Change merged: Pyoungmeister; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/23603 [22:25:19] New review: Ryan Lane; "Please no gems." [operations/puppet] (production); V: 0 C: -2; - https://gerrit.wikimedia.org/r/23604 [22:28:26] Change abandoned: Hashar; "per discussion with Ryan :-D puppet-lint is not going to be used anytime soon by ops :-D" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/23604 [22:28:57] New patchset: awjrichards; "WLM banner" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/23021 [22:30:14] PROBLEM - Apache HTTP on mw1 is CRITICAL: Connection refused [22:31:08] RECOVERY - NTP on mw1 is OK: NTP OK: Offset -0.01274323463 secs [22:31:13] Oh my god, They killed mw1! You bastards! [22:31:52] it's just not serving pages anymore :) [22:32:35] LeslieCarr: do the mc hosts in tampa still have link on their 1gb eth0 nic? [22:33:09] i'm not sure, i can check [22:33:13] their 10g should be working well [22:33:46] yes, but we preseed with "netcfg/choose_interface select auto" [22:34:02] which selects the first interface with physical link [22:34:31] binasher: the links are down [22:34:46] the host i'm trying pxeboots fine off the 10gb, then the installer tries to dhcp over eth0 which is the onboard 1gb [22:34:52] weird .... [22:35:39] it theoretically shouldn't do that unless there's some amount of link.. maybe portdown still results in voltage on a 2 wire pair [22:35:40] just double checked with cmjohnson1 - the eth0's are physically uncabled [22:35:54] ok, good to know [22:35:59] ubuntu fail [22:36:06] Change merged: MaxSem; [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/23021 [22:36:12] we should proably use freebsd instead [22:36:19] are they on their own subnet? [22:36:39] i'll need some way to give them a unique preseed cfg file [22:36:42] with netcfg/choose_interface select eth2 [22:39:07] binasher: it may not be eth2...there are 4 onboard nics [22:39:25] and we are using the 10G card [22:40:07] binasher: they are not on their own subnet [22:40:14] it's in the mega internal subnet of tampa [22:40:44] RECOVERY - Host pay-lvs1002 is UP: PING OK - Packet loss = 0%, RTA = 26.97 ms [22:42:47] robh: what is the hot key for f2 w/the mac [22:47:10] New patchset: Pyoungmeister; "include php ini files on jobrunners" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/23608 [22:48:20] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/23608 [22:48:23] PROBLEM - udp2log log age for locke on locke is CRITICAL: CRITICAL: log files /a/squid/fundraising/logs/bannerImpressions-sampled100.log, have not been written in a critical amount of time. For most logs, this is 4 hours. For slow logs, this is 4 days. [22:48:30] Change merged: Pyoungmeister; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/23608 [22:49:06] Change merged: Hashar; [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/23077 [22:49:06] Change merged: Hashar; [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/23074 [22:49:33] dunno, its on the wikitech platform specific page [22:49:36] cmjohnson1: ^ [22:52:26] paravoid: shit like this is why I hate puppet: http://projects.puppetlabs.com/issues/5489 [22:52:46] I have absolutely no clue which one of the parameters in my module is fucking me [22:53:16] New patchset: Dereckson; "(bug 40186) Favicon for es.wiktionary" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/23610 [22:54:47] New patchset: Pyoungmeister; "adding other required classes to jobrunners" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/23611 [22:55:47] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/23611 [22:55:56] Change merged: Pyoungmeister; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/23611 [22:58:08] PROBLEM - Puppet freshness on virt1002 is CRITICAL: Puppet has not run in the last 10 hours [22:58:08] PROBLEM - Puppet freshness on singer is CRITICAL: Puppet has not run in the last 10 hours [22:58:08] PROBLEM - Puppet freshness on virt1001 is CRITICAL: Puppet has not run in the last 10 hours [22:58:08] PROBLEM - Puppet freshness on virt1003 is CRITICAL: Puppet has not run in the last 10 hours [22:58:08] PROBLEM - Puppet freshness on ocg3 is CRITICAL: Puppet has not run in the last 10 hours [22:58:08] PROBLEM - Puppet freshness on virt1004 is CRITICAL: Puppet has not run in the last 10 hours [23:00:20] New patchset: Pyoungmeister; "yet more classes for job runners" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/23613 [23:00:23] RECOVERY - udp2log log age for locke on locke is OK: OK: all log files active [23:01:16] folks who are watching the ms-fe proxies might be interested in this graph: [23:01:20] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/23613 [23:01:38] Change merged: Pyoungmeister; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/23613 [23:01:45] http://ganglia.wikimedia.org/latest/graph.php?h=ms-fe1.pmtpa.wmnet&m=load_one&r=custom&s=by%20name&hc=4&mc=2&cs=8%2F29%2F2012%202%3A33&ce=9%2F2%2F2012%2018%3A33&st=1347490749&g=mem_report&z=medium&c=Swift%20pmtpa [23:02:03] it looks very much to me like we start entering swap in less than two days [23:02:27] maybe we want to restart the proxies (staggered) once a day til we know the cause [23:02:31] (and have a fix) [23:02:51] That only sounds like partially a bad idea [23:03:10] waiting until they swap is deadly [23:04:06] Yeah, that would end up like the awesome apach/memcache issue where it just takes the site down in one go... finding the issue while restarting to avoid it is probably tricky unless it's known upstream. [23:04:10] the last restart was much longer than two days, but I expect there was an impact on response time for some users well before that [23:05:14] upstream doesn't know about existing memory leaks for this version [23:05:48] we'll upgrade of course, we're waiting on a specific patch that we want to grab for another issue [23:05:58] but no guarantee that will address the memory issue [23:10:38] New review: Vogone; "Yes, wikipedia:lt: instead of wp:lt: is necessary because no betawikipedia exists. wikipedia:lt: red..." [operations/mediawiki-config] (master) C: 1; - https://gerrit.wikimedia.org/r/23419 [23:13:25] apergos: I have migrated some favicons out of upload to bits.wm.org. That was listed as a pending issue following the migration to swift :) [23:13:39] yay! [23:13:45] I saw you were doing that [23:14:01] if you want to mark it as done on the cruft page that is great! [23:14:20] http://wikitech.wikimedia.org/view/Swift/Open_Issues_Aug_-_Sept_2012/Cruft_on_ms7 [23:15:20] https://gerrit.wikimedia.org/r/#q,23074,n,z and https://gerrit.wikimedia.org/r/#q,23077,n,z [23:15:32] PROBLEM - check_mysql on payments1001 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) [23:17:58] LeslieCarr: [ 5.668349] ixgbe 0000:04:00.0: failed to load because an unsupported SFP+ module type was detected. [23:18:04] :D [23:18:12] binasher: why did you do that ? [23:18:16] :( [23:18:35] interesting though [23:19:00] so, is that mc16 ? [23:19:03] LeslieCarr: mc2 [23:19:04] if it's any other host than mc16, shit [23:19:06] weird [23:19:10] so they are showing link [23:19:13] from the switch side [23:19:16] they're fine [23:19:17] they pxe boot [23:19:23] cool [23:19:34] um [23:19:43] they're unusable [23:19:48] y u hate us SFP+ ? [23:19:53] eep [23:20:01] um, they pxe boot and then fail to continue ? [23:20:20] PROBLEM - check_mysql on payments1001 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) [23:20:56] the card pxeboots fine, but intel's linux 10g kernel module won't work with that sfp :( [23:21:44] we're using something else in eqiad, right? [23:22:13] New patchset: Ryan Lane; "Initial checkin of salt puppet module" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/23587 [23:22:39] LeslieCarr: there's a list of supported sfp+ modules at http://www.mjmwired.net/kernel/Documentation/networking/ixgbe.txt [23:22:49] binasher: the first 8 mc's are using something different [23:23:02] 1001-1008? [23:23:06] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/23587 [23:23:32] binasher: we're using one of the approved ones [23:23:37] the FTLX1471D3BCL [23:23:42] in tampa? [23:23:45] yes [23:23:48] and for mc1009-1016 [23:23:51] wtf [23:24:56] hrm, this could be informational - "- Intel(R) Network Adapters that support removable optical modules only support [23:24:56] 82 their original module type (i.e., the Intel(R) 10 Gigabit SR Dual Port [23:24:57] 83 Express Module only supports SR optical modules). If you plug in a different [23:24:57] 84 type of module, the driver will not load." [23:25:09] do we know if the cards use SR optical modules ? [23:25:14] RobH: do we have a hardware manifest ? [23:25:17] New patchset: Ryan Lane; "Add salt roles for master and minion" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/23591 [23:25:35] PROBLEM - check_mysql on payments1001 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) [23:26:07] LeslieCarr: actually, the list with the kernel driver i sent you isn't right, different cards support different combos [23:26:10] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/23591 [23:26:21] binasher: know which card we have ? [23:26:41] this might apply http://www.intel.com/support/network/adapter/pro100/sb/CS-030612.htm [23:27:05] please tell me these are not fucking vendor locked sfp's [23:27:10] i mean vendor locked cards [23:27:22] do we know htat's the version ? [23:28:00] not sure [23:29:02] RobH: ? it looks like these were ordered before i started/before my email searching abilities [23:29:30] parm: allow_unsupported_sfp:Allow unsupported and untested SFP+ modules on 82599-based adapters (uint) [23:29:44] Change merged: Ryan Lane; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/23587 [23:30:01] LeslieCarr: a hardware manifest for? [23:30:21] the mc1001 - mc1008 machines ? [23:30:28] or wait no [23:30:32] i mean the mc1-mc16 machines [23:30:32] PROBLEM - check_mysql on payments1001 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) [23:30:48] Change merged: Ryan Lane; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/23591 [23:30:54] !!! [23:30:59] I get root key :-]]]]]]]]]]]]] [23:31:07] hashar: eh? [23:31:09] got some op laptop hehe [23:31:21] hashar: who's? :) [23:31:21] i have the quote we ordered from [23:31:39] mark handled the proposal for the mc1001-1008 as they were initial ashburn order [23:31:44] LeslieCarr: im emailing you the quote [23:31:48] thank you [23:32:15] Ryan_Lane: was mostly kidding. I am just holding the locked laptop to make sure nobody is going to pick it :D [23:32:32] so we just orderd ten DAC cables via dell for the mc1011+ [23:32:36] now i wonder if they will work [23:32:51] they are juniper cables, specifically since we had issues with the others, direct from dell resale [23:32:52] New patchset: Ryan Lane; "Adding salt master role to virt0" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/23619 [23:32:57] going to be interesting on monday. [23:33:45] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/23619 [23:33:45] RobH: can you send me the mc1-mc16 one as well ? [23:35:03] and of course, the quote doesn't specify which card version it actually is [23:35:38] PROBLEM - check_mysql on payments1001 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) [23:35:44] LeslieCarr: https://rt.wikimedia.org/Ticket/Display.html?id=2854 for mc1011+ and https://rt.wikimedia.org/Ticket/Display.html?id=2882 for mc1-mc16 [23:36:05] binasher: does the card self-identify during bootup/bios loading process ? [23:36:55] ok, they specify the X520 in that quote [23:37:08] PROBLEM - Puppet freshness on ms-be10 is CRITICAL: Puppet has not run in the last 10 hours [23:37:14] RT HAS ALL THE ANSWERS [23:37:18] ALL HAIL RT [23:37:23] good and bad, do we think we can use paravoid's hack to get around the stupid "check the programming of the sfp" ? [23:37:32] i didnt bring up in ops meeting [23:37:33] thanks RobH and RT [23:37:39] that i think we should stop running both BZ and RT [23:37:49] New patchset: Ryan Lane; "Add iptables info for salt ports" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/23620 [23:37:51] and combine them and ditch the volunteer full admins, demoting them to queue admins only.... [23:38:09] because having two ticketing systems when you happen to work on public facing bugs succccks. [23:38:41] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/23620 [23:38:43] fwiw, there is a Mediawiki extension for RT, to display RT stuff like tickets/description/status in MW pages [23:39:28] binasher: http://lists.openwall.net/netdev/2012/01/19/63 [23:40:07] PROBLEM - check_mysql on payments1001 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) [23:40:21] Yes Seconds? hmm:) [23:40:47] New patchset: Ryan Lane; "Limit access to salt master" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/23621 [23:40:52] mutante: No, it's null seconds behind master [23:41:38] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/23621 [23:41:42] Change merged: Ryan Lane; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/23620 [23:41:55] Change merged: Ryan Lane; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/23621 [23:43:04] LeslieCarr: any idea how much intel sfp's cost? [23:43:44] RobH: I'm all with you [23:44:26] binasher: well on the good side, we would only need them on the one side (the memcache side) .. one site has them for $500 [23:44:42] binasher: list price is 1000 [23:44:45] 32 of those.. :( [23:45:04] PROBLEM - check_mysql on payments1001 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) [23:45:30] well we have at least 8 working machines, so 24 of them... [23:45:32] but yeah [23:45:46] damn, the normal site requires a quote for these (though they say they have 63 in stock) [23:46:59] what are we using for 1001-8? [23:47:11] an old version of the copper direct attach cables [23:47:20] the new versions don't work (they changed what they flashed them as) [23:48:13] which is bullshit. [23:48:23] Change merged: Ryan Lane; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/23619 [23:49:47] robh lesliecarr and binasher maybe we should order 2 for tampa and see if they work before investing in them [23:50:04] yes, definitely order small amounts before the whole sheband this time [23:50:10] on the good side, we can reuse the sfp [23:50:10] PROBLEM - check_mysql on payments1001 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) [23:50:11] +'s [23:51:13] RECOVERY - Host srv281 is UP: PING OK - Packet loss = 0%, RTA = 0.27 ms [23:51:52] i saw we wait until we try the DAC cables in eqiad. [23:51:55] they may work [23:51:58] and they are cheap. [23:52:07] binasher: setup eqiad mc ;] [23:52:32] RobH: configure the switches [23:53:10] binasher: sounds like a LeslieCarr or mark job. [23:53:17] if only they were around to do it... [23:53:58] * LeslieCarr hides under the table [23:54:31] PROBLEM - Apache HTTP on srv281 is CRITICAL: Connection refused [23:55:07] PROBLEM - check_mysql on payments1001 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) [23:58:50] New patchset: Andrew Bogott; "Update wiki instance-status pages automatically." [operations/puppet] (production) - https://gerrit.wikimedia.org/r/23155 [23:59:43] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/23155