[00:05:29] PROBLEM - Host wtp1008 is DOWN: PING CRITICAL - Packet loss = 100% [00:06:00] RECOVERY - Host wtp1008 is UP: PING OK - Packet loss = 0%, RTA = 0.59 ms [01:02:20] RECOVERY - NTP on ssl3003 is OK: NTP OK: Offset 0.001989603043 secs [01:02:30] RECOVERY - NTP on ssl3002 is OK: NTP OK: Offset 0.005225539207 secs [02:07:00] !log LocalisationUpdate completed (1.22wmf5) at Wed Jun 5 02:06:59 UTC 2013 [02:09:11] PROBLEM - Puppet freshness on mw1102 is CRITICAL: No successful Puppet run in the last 10 hours [02:09:11] PROBLEM - Puppet freshness on mw1073 is CRITICAL: No successful Puppet run in the last 10 hours [02:09:26] morebots died? [02:12:38] !log LocalisationUpdate completed (1.22wmf4) at Wed Jun 5 02:12:37 UTC 2013 [02:20:19] !log LocalisationUpdate ResourceLoader cache refresh completed at Wed Jun 5 02:20:18 UTC 2013 [02:22:10] PROBLEM - Puppet freshness on db1011 is CRITICAL: No successful Puppet run in the last 10 hours [02:22:10] PROBLEM - Puppet freshness on lvs2 is CRITICAL: No successful Puppet run in the last 10 hours [02:22:10] PROBLEM - Puppet freshness on mc1013 is CRITICAL: No successful Puppet run in the last 10 hours [02:22:10] PROBLEM - Puppet freshness on mc8 is CRITICAL: No successful Puppet run in the last 10 hours [02:22:10] PROBLEM - Puppet freshness on magnesium is CRITICAL: No successful Puppet run in the last 10 hours [02:23:10] PROBLEM - Puppet freshness on amssq32 is CRITICAL: No successful Puppet run in the last 10 hours [02:23:10] PROBLEM - Puppet freshness on amssq43 is CRITICAL: No successful Puppet run in the last 10 hours [02:23:10] PROBLEM - Puppet freshness on analytics1018 is CRITICAL: No successful Puppet run in the last 10 hours [02:23:10] PROBLEM - Puppet freshness on arsenic is CRITICAL: No successful Puppet run in the last 10 hours [02:23:10] PROBLEM - Puppet freshness on chromium is CRITICAL: No successful Puppet run in the last 10 hours [02:25:12] PROBLEM - Puppet freshness on amssq41 is CRITICAL: No successful Puppet run in the last 10 hours [02:25:12] PROBLEM - Puppet freshness on amssq36 is CRITICAL: No successful Puppet run in the last 10 hours [02:25:12] PROBLEM - Puppet freshness on amssq58 is CRITICAL: No successful Puppet run in the last 10 hours [02:25:12] PROBLEM - Puppet freshness on amssq60 is CRITICAL: No successful Puppet run in the last 10 hours [02:25:12] PROBLEM - Puppet freshness on analytics1006 is CRITICAL: No successful Puppet run in the last 10 hours [02:27:11] PROBLEM - Puppet freshness on amssq46 is CRITICAL: No successful Puppet run in the last 10 hours [02:27:12] PROBLEM - Puppet freshness on amssq62 is CRITICAL: No successful Puppet run in the last 10 hours [02:27:12] PROBLEM - Puppet freshness on analytics1013 is CRITICAL: No successful Puppet run in the last 10 hours [02:27:12] PROBLEM - Puppet freshness on cp1004 is CRITICAL: No successful Puppet run in the last 10 hours [02:27:12] PROBLEM - Puppet freshness on cp1008 is CRITICAL: No successful Puppet run in the last 10 hours [02:28:10] PROBLEM - Puppet freshness on amssq33 is CRITICAL: No successful Puppet run in the last 10 hours [02:28:10] PROBLEM - Puppet freshness on amssq38 is CRITICAL: No successful Puppet run in the last 10 hours [02:28:10] PROBLEM - Puppet freshness on amssq42 is CRITICAL: No successful Puppet run in the last 10 hours [02:28:10] PROBLEM - Puppet freshness on amssq54 is CRITICAL: No successful Puppet run in the last 10 hours [02:28:10] PROBLEM - Puppet freshness on carbon is CRITICAL: No successful Puppet run in the last 10 hours [02:29:10] PROBLEM - Puppet freshness on amssq39 is CRITICAL: No successful Puppet run in the last 10 hours [02:29:10] PROBLEM - Puppet freshness on amssq45 is CRITICAL: No successful Puppet run in the last 10 hours [02:29:10] PROBLEM - Puppet freshness on amssq61 is CRITICAL: No successful Puppet run in the last 10 hours [02:29:10] PROBLEM - Puppet freshness on analytics1005 is CRITICAL: No successful Puppet run in the last 10 hours [02:29:10] PROBLEM - Puppet freshness on analytics1025 is CRITICAL: No successful Puppet run in the last 10 hours [02:31:10] PROBLEM - Puppet freshness on amssq34 is CRITICAL: No successful Puppet run in the last 10 hours [02:31:10] PROBLEM - Puppet freshness on amssq57 is CRITICAL: No successful Puppet run in the last 10 hours [02:31:10] PROBLEM - Puppet freshness on analytics1010 is CRITICAL: No successful Puppet run in the last 10 hours [02:31:10] PROBLEM - Puppet freshness on cp1003 is CRITICAL: No successful Puppet run in the last 10 hours [02:31:10] PROBLEM - Puppet freshness on cp1006 is CRITICAL: No successful Puppet run in the last 10 hours [02:32:10] PROBLEM - Puppet freshness on amssq44 is CRITICAL: No successful Puppet run in the last 10 hours [02:32:11] PROBLEM - Puppet freshness on analytics1015 is CRITICAL: No successful Puppet run in the last 10 hours [02:32:11] PROBLEM - Puppet freshness on analytics1023 is CRITICAL: No successful Puppet run in the last 10 hours [02:32:11] PROBLEM - Puppet freshness on cp1013 is CRITICAL: No successful Puppet run in the last 10 hours [02:32:11] PROBLEM - Puppet freshness on db1004 is CRITICAL: No successful Puppet run in the last 10 hours [02:32:11] PROBLEM - Puppet freshness on db43 is CRITICAL: No successful Puppet run in the last 10 hours [02:34:10] PROBLEM - Puppet freshness on amslvs1 is CRITICAL: No successful Puppet run in the last 10 hours [02:34:10] PROBLEM - Puppet freshness on amslvs4 is CRITICAL: No successful Puppet run in the last 10 hours [02:34:10] PROBLEM - Puppet freshness on analytics1004 is CRITICAL: No successful Puppet run in the last 10 hours [02:34:10] PROBLEM - Puppet freshness on amssq53 is CRITICAL: No successful Puppet run in the last 10 hours [02:34:10] PROBLEM - Puppet freshness on cp1026 is CRITICAL: No successful Puppet run in the last 10 hours [02:35:10] PROBLEM - Puppet freshness on amssq47 is CRITICAL: No successful Puppet run in the last 10 hours [02:35:10] PROBLEM - Puppet freshness on amssq48 is CRITICAL: No successful Puppet run in the last 10 hours [02:35:10] PROBLEM - Puppet freshness on analytics1003 is CRITICAL: No successful Puppet run in the last 10 hours [02:35:10] PROBLEM - Puppet freshness on antimony is CRITICAL: No successful Puppet run in the last 10 hours [02:35:10] PROBLEM - Puppet freshness on db1013 is CRITICAL: No successful Puppet run in the last 10 hours [02:37:03] PROBLEM - Puppet freshness on amslvs2 is CRITICAL: No successful Puppet run in the last 10 hours [02:37:05] PROBLEM - Puppet freshness on amssq56 is CRITICAL: No successful Puppet run in the last 10 hours [02:37:05] PROBLEM - Puppet freshness on analytics1002 is CRITICAL: No successful Puppet run in the last 10 hours [02:37:05] PROBLEM - Puppet freshness on analytics1009 is CRITICAL: No successful Puppet run in the last 10 hours [02:37:05] PROBLEM - Puppet freshness on cp1017 is CRITICAL: No successful Puppet run in the last 10 hours [02:37:05] PROBLEM - Puppet freshness on analytics1022 is CRITICAL: No successful Puppet run in the last 10 hours [02:51:35] * Susan eyes ChanServ. [02:51:59] legoktm: https://bugzilla.wikimedia.org/show_bug.cgi?id=47228 [03:57:49] !log restarted morebots [03:57:57] Logged the message, Master [04:58:23] New patchset: ArielGlenn; "dump sitestable for wikibase repos" [operations/dumps] (ariel) - https://gerrit.wikimedia.org/r/67052 [05:00:21] Change merged: ArielGlenn; [operations/dumps] (ariel) - https://gerrit.wikimedia.org/r/67052 [06:49:42] New review: Hazard-SJ; "I was standardizing the indentation, and btw, the coding conventions doesn't specify about this, but..." [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/65860 [07:13:16] New patchset: Akosiaris; "Add hadoop-0.20-mapreduce packages from cloudera" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/67054 [07:22:09] Change merged: Akosiaris; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/67054 [08:02:25] New patchset: Petrb; "inserted several missing packages" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/67055 [08:08:42] New patchset: Petrb; "inserted several missing packages" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/67055 [08:16:13] New patchset: Petrb; "inserted several missing packages" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/67055 [08:18:17] New patchset: Petrb; "inserted several missing packages" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/67055 [08:26:29] New patchset: Petrb; "inserted several missing packages" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/67055 [08:27:53] New patchset: Petrb; "inserted several missing packages" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/67055 [10:23:20] RECOVERY - Puppet freshness on mc1007 is OK: puppet ran at Wed Jun 5 10:23:09 UTC 2013 [10:23:20] RECOVERY - Puppet freshness on mw1122 is OK: puppet ran at Wed Jun 5 10:23:09 UTC 2013 [10:23:20] RECOVERY - Puppet freshness on sq63 is OK: puppet ran at Wed Jun 5 10:23:10 UTC 2013 [10:23:20] RECOVERY - Puppet freshness on db1002 is OK: puppet ran at Wed Jun 5 10:23:10 UTC 2013 [10:23:20] RECOVERY - Puppet freshness on db1033 is OK: puppet ran at Wed Jun 5 10:23:12 UTC 2013 [10:23:20] RECOVERY - Puppet freshness on mw2 is OK: puppet ran at Wed Jun 5 10:23:12 UTC 2013 [10:23:21] RECOVERY - Puppet freshness on mw1003 is OK: puppet ran at Wed Jun 5 10:23:13 UTC 2013 [10:23:21] RECOVERY - Puppet freshness on virt1000 is OK: puppet ran at Wed Jun 5 10:23:15 UTC 2013 [10:23:22] RECOVERY - Puppet freshness on mw1033 is OK: puppet ran at Wed Jun 5 10:23:15 UTC 2013 [10:23:22] RECOVERY - Puppet freshness on mw1153 is OK: puppet ran at Wed Jun 5 10:23:16 UTC 2013 [10:23:23] RECOVERY - Puppet freshness on ms-be11 is OK: puppet ran at Wed Jun 5 10:23:17 UTC 2013 [10:23:23] RECOVERY - Puppet freshness on ssl1001 is OK: puppet ran at Wed Jun 5 10:23:18 UTC 2013 [10:23:24] RECOVERY - Puppet freshness on db48 is OK: puppet ran at Wed Jun 5 10:23:19 UTC 2013 [10:23:30] RECOVERY - Puppet freshness on db1040 is OK: puppet ran at Wed Jun 5 10:23:19 UTC 2013 [10:23:30] RECOVERY - Puppet freshness on srv267 is OK: puppet ran at Wed Jun 5 10:23:20 UTC 2013 [10:23:30] RECOVERY - Puppet freshness on mw112 is OK: puppet ran at Wed Jun 5 10:23:21 UTC 2013 [10:23:30] RECOVERY - Puppet freshness on mw104 is OK: puppet ran at Wed Jun 5 10:23:22 UTC 2013 [10:23:30] RECOVERY - Puppet freshness on srv262 is OK: puppet ran at Wed Jun 5 10:23:22 UTC 2013 [10:23:30] RECOVERY - Puppet freshness on search25 is OK: puppet ran at Wed Jun 5 10:23:24 UTC 2013 [10:23:31] RECOVERY - Puppet freshness on srv260 is OK: puppet ran at Wed Jun 5 10:23:24 UTC 2013 [10:23:31] RECOVERY - Puppet freshness on srv269 is OK: puppet ran at Wed Jun 5 10:23:25 UTC 2013 [10:23:32] RECOVERY - Puppet freshness on search16 is OK: puppet ran at Wed Jun 5 10:23:28 UTC 2013 [10:23:32] RECOVERY - Puppet freshness on fluorine is OK: puppet ran at Wed Jun 5 10:23:28 UTC 2013 [10:23:40] RECOVERY - Puppet freshness on mw68 is OK: puppet ran at Wed Jun 5 10:23:29 UTC 2013 [10:23:40] RECOVERY - Puppet freshness on solr1002 is OK: puppet ran at Wed Jun 5 10:23:30 UTC 2013 [10:23:40] RECOVERY - Puppet freshness on mw1010 is OK: puppet ran at Wed Jun 5 10:23:33 UTC 2013 [10:23:40] RECOVERY - Puppet freshness on mw1166 is OK: puppet ran at Wed Jun 5 10:23:34 UTC 2013 [10:23:40] RECOVERY - Puppet freshness on search1001 is OK: puppet ran at Wed Jun 5 10:23:34 UTC 2013 [10:23:40] RECOVERY - Puppet freshness on cp1037 is OK: puppet ran at Wed Jun 5 10:23:36 UTC 2013 [10:23:41] RECOVERY - Puppet freshness on mw1118 is OK: puppet ran at Wed Jun 5 10:23:36 UTC 2013 [10:23:41] RECOVERY - Puppet freshness on mw1092 is OK: puppet ran at Wed Jun 5 10:23:37 UTC 2013 [10:23:42] RECOVERY - Puppet freshness on mw1142 is OK: puppet ran at Wed Jun 5 10:23:38 UTC 2013 [10:23:48] !log starting icinga [10:23:50] RECOVERY - Puppet freshness on virt11 is OK: puppet ran at Wed Jun 5 10:23:39 UTC 2013 [10:23:50] RECOVERY - Puppet freshness on mw1091 is OK: puppet ran at Wed Jun 5 10:23:40 UTC 2013 [10:23:50] RECOVERY - Puppet freshness on amslvs4 is OK: puppet ran at Wed Jun 5 10:23:41 UTC 2013 [10:23:50] RECOVERY - Puppet freshness on mw1025 is OK: puppet ran at Wed Jun 5 10:23:41 UTC 2013 [10:23:50] RECOVERY - Puppet freshness on cp1026 is OK: puppet ran at Wed Jun 5 10:23:42 UTC 2013 [10:23:50] RECOVERY - Puppet freshness on search1022 is OK: puppet ran at Wed Jun 5 10:23:43 UTC 2013 [10:23:51] RECOVERY - Puppet freshness on analytics1004 is OK: puppet ran at Wed Jun 5 10:23:44 UTC 2013 [10:23:51] RECOVERY - Puppet freshness on amssq48 is OK: puppet ran at Wed Jun 5 10:23:45 UTC 2013 [10:23:52] RECOVERY - Puppet freshness on mw37 is OK: puppet ran at Wed Jun 5 10:23:48 UTC 2013 [10:23:52] RECOVERY - Puppet freshness on db54 is OK: puppet ran at Wed Jun 5 10:23:49 UTC 2013 [10:23:56] Logged the message, Master [10:24:00] RECOVERY - Puppet freshness on db1034 is OK: puppet ran at Wed Jun 5 10:23:50 UTC 2013 [10:24:00] RECOVERY - Puppet freshness on db1051 is OK: puppet ran at Wed Jun 5 10:23:53 UTC 2013 [10:24:00] RECOVERY - Puppet freshness on db1013 is OK: puppet ran at Wed Jun 5 10:23:54 UTC 2013 [10:24:00] RECOVERY - Puppet freshness on analytics1003 is OK: puppet ran at Wed Jun 5 10:23:55 UTC 2013 [10:24:00] RECOVERY - Puppet freshness on amssq47 is OK: puppet ran at Wed Jun 5 10:23:58 UTC 2013 [10:24:10] RECOVERY - Puppet freshness on search29 is OK: puppet ran at Wed Jun 5 10:23:59 UTC 2013 [10:24:10] RECOVERY - Puppet freshness on mw60 is OK: puppet ran at Wed Jun 5 10:24:00 UTC 2013 [10:24:10] RECOVERY - Puppet freshness on mw23 is OK: puppet ran at Wed Jun 5 10:24:01 UTC 2013 [10:24:10] RECOVERY - Puppet freshness on mw1213 is OK: puppet ran at Wed Jun 5 10:24:01 UTC 2013 [10:24:10] RECOVERY - Puppet freshness on search30 is OK: puppet ran at Wed Jun 5 10:24:02 UTC 2013 [10:24:10] RECOVERY - Puppet freshness on mw1011 is OK: puppet ran at Wed Jun 5 10:24:03 UTC 2013 [10:24:11] RECOVERY - Puppet freshness on mw28 is OK: puppet ran at Wed Jun 5 10:24:04 UTC 2013 [10:24:11] RECOVERY - Puppet freshness on sockpuppet is OK: puppet ran at Wed Jun 5 10:24:05 UTC 2013 [10:24:12] RECOVERY - Puppet freshness on search1007 is OK: puppet ran at Wed Jun 5 10:24:06 UTC 2013 [10:24:12] RECOVERY - Puppet freshness on db66 is OK: puppet ran at Wed Jun 5 10:24:08 UTC 2013 [10:24:13] RECOVERY - Puppet freshness on search1012 is OK: puppet ran at Wed Jun 5 10:24:09 UTC 2013 [10:24:20] RECOVERY - Puppet freshness on mw1129 is OK: puppet ran at Wed Jun 5 10:24:09 UTC 2013 [10:24:20] RECOVERY - Puppet freshness on mw1107 is OK: puppet ran at Wed Jun 5 10:24:10 UTC 2013 [10:24:20] PROBLEM - Puppet freshness on aluminium is CRITICAL: No successful Puppet run in the last 10 hours [10:24:20] PROBLEM - Puppet freshness on amssq59 is CRITICAL: No successful Puppet run in the last 10 hours [10:24:20] PROBLEM - Puppet freshness on amssq35 is CRITICAL: No successful Puppet run in the last 10 hours [10:25:31] RECOVERY - Puppet freshness on mw42 is OK: puppet ran at Wed Jun 5 10:25:19 UTC 2013 [10:25:31] RECOVERY - Puppet freshness on cp1016 is OK: puppet ran at Wed Jun 5 10:25:20 UTC 2013 [10:25:32] RECOVERY - Puppet freshness on mc12 is OK: puppet ran at Wed Jun 5 10:25:22 UTC 2013 [10:25:32] RECOVERY - Puppet freshness on solr1001 is OK: puppet ran at Wed Jun 5 10:25:23 UTC 2013 [10:25:33] RECOVERY - Puppet freshness on grosley is OK: puppet ran at Wed Jun 5 10:25:24 UTC 2013 [10:25:33] RECOVERY - Puppet freshness on analytics1022 is OK: puppet ran at Wed Jun 5 10:25:25 UTC 2013 [10:25:34] RECOVERY - Puppet freshness on db69 is OK: puppet ran at Wed Jun 5 10:25:27 UTC 2013 [10:25:34] RECOVERY - Puppet freshness on labstore2 is OK: puppet ran at Wed Jun 5 10:25:28 UTC 2013 [10:25:35] RECOVERY - Puppet freshness on es9 is OK: puppet ran at Wed Jun 5 10:25:29 UTC 2013 [10:25:40] RECOVERY - Puppet freshness on mw61 is OK: puppet ran at Wed Jun 5 10:25:29 UTC 2013 [10:25:40] RECOVERY - Puppet freshness on db67 is OK: puppet ran at Wed Jun 5 10:25:31 UTC 2013 [10:25:40] RECOVERY - Puppet freshness on analytics1002 is OK: puppet ran at Wed Jun 5 10:25:33 UTC 2013 [10:25:40] RECOVERY - Puppet freshness on analytics1009 is OK: puppet ran at Wed Jun 5 10:25:34 UTC 2013 [10:25:40] RECOVERY - Puppet freshness on db1009 is OK: puppet ran at Wed Jun 5 10:25:35 UTC 2013 [10:25:41] RECOVERY - Puppet freshness on db1005 is OK: puppet ran at Wed Jun 5 10:25:36 UTC 2013 [10:25:41] RECOVERY - Puppet freshness on mw74 is OK: puppet ran at Wed Jun 5 10:25:36 UTC 2013 [10:25:42] RECOVERY - Puppet freshness on search1003 is OK: puppet ran at Wed Jun 5 10:25:37 UTC 2013 [10:25:42] RECOVERY - Puppet freshness on mc11 is OK: puppet ran at Wed Jun 5 10:25:38 UTC 2013 [10:25:50] RECOVERY - Puppet freshness on mw1049 is OK: puppet ran at Wed Jun 5 10:25:39 UTC 2013 [10:25:50] RECOVERY - Puppet freshness on mw1047 is OK: puppet ran at Wed Jun 5 10:25:40 UTC 2013 [10:25:50] RECOVERY - Puppet freshness on sanger is OK: puppet ran at Wed Jun 5 10:25:41 UTC 2013 [10:25:50] RECOVERY - Puppet freshness on mw1105 is OK: puppet ran at Wed Jun 5 10:25:44 UTC 2013 [10:25:50] RECOVERY - Puppet freshness on search1005 is OK: puppet ran at Wed Jun 5 10:25:44 UTC 2013 [10:25:50] RECOVERY - Puppet freshness on mw1084 is OK: puppet ran at Wed Jun 5 10:25:45 UTC 2013 [10:25:51] RECOVERY - Puppet freshness on search31 is OK: puppet ran at Wed Jun 5 10:25:46 UTC 2013 [10:25:51] RECOVERY - Puppet freshness on hume is OK: puppet ran at Wed Jun 5 10:25:47 UTC 2013 [10:25:52] RECOVERY - Puppet freshness on mw45 is OK: puppet ran at Wed Jun 5 10:25:48 UTC 2013 [10:25:52] RECOVERY - Puppet freshness on srv291 is OK: puppet ran at Wed Jun 5 10:25:49 UTC 2013 [10:26:00] RECOVERY - Puppet freshness on mw1179 is OK: puppet ran at Wed Jun 5 10:25:49 UTC 2013 [10:26:01] RECOVERY - Puppet freshness on db32 is OK: puppet ran at Wed Jun 5 10:25:50 UTC 2013 [10:26:01] RECOVERY - Puppet freshness on mw1030 is OK: puppet ran at Wed Jun 5 10:25:51 UTC 2013 [10:26:01] RECOVERY - Puppet freshness on mw1075 is OK: puppet ran at Wed Jun 5 10:25:52 UTC 2013 [10:26:01] RECOVERY - Puppet freshness on mw1020 is OK: puppet ran at Wed Jun 5 10:25:53 UTC 2013 [10:26:01] RECOVERY - Puppet freshness on mw1078 is OK: puppet ran at Wed Jun 5 10:25:54 UTC 2013 [10:26:01] RECOVERY - Puppet freshness on nickel is OK: puppet ran at Wed Jun 5 10:25:55 UTC 2013 [10:26:02] RECOVERY - Puppet freshness on mc4 is OK: puppet ran at Wed Jun 5 10:25:55 UTC 2013 [10:26:02] RECOVERY - Puppet freshness on ms-fe2 is OK: puppet ran at Wed Jun 5 10:25:57 UTC 2013 [10:26:03] RECOVERY - Puppet freshness on niobium is OK: puppet ran at Wed Jun 5 10:25:58 UTC 2013 [10:26:03] RECOVERY - Puppet freshness on amssq35 is OK: puppet ran at Wed Jun 5 10:25:58 UTC 2013 [10:26:04] RECOVERY - Puppet freshness on amssq31 is OK: puppet ran at Wed Jun 5 10:25:59 UTC 2013 [10:26:10] RECOVERY - Puppet freshness on db77 is OK: puppet ran at Wed Jun 5 10:26:00 UTC 2013 [10:26:10] RECOVERY - Puppet freshness on db1058 is OK: puppet ran at Wed Jun 5 10:26:04 UTC 2013 [10:26:11] RECOVERY - Puppet freshness on sq68 is OK: puppet ran at Wed Jun 5 10:26:05 UTC 2013 [10:26:11] RECOVERY - Puppet freshness on mw1165 is OK: puppet ran at Wed Jun 5 10:26:08 UTC 2013 [10:26:11] RECOVERY - Puppet freshness on mw1181 is OK: puppet ran at Wed Jun 5 10:26:09 UTC 2013 [10:26:20] RECOVERY - Puppet freshness on db65 is OK: puppet ran at Wed Jun 5 10:26:10 UTC 2013 [10:26:20] RECOVERY - Puppet freshness on ms-be1009 is OK: puppet ran at Wed Jun 5 10:26:12 UTC 2013 [10:26:20] RECOVERY - Puppet freshness on wtp1002 is OK: puppet ran at Wed Jun 5 10:26:13 UTC 2013 [10:26:20] RECOVERY - Puppet freshness on cp1038 is OK: puppet ran at Wed Jun 5 10:26:14 UTC 2013 [10:26:20] RECOVERY - Puppet freshness on srv264 is OK: puppet ran at Wed Jun 5 10:26:15 UTC 2013 [10:26:20] RECOVERY - Puppet freshness on mw99 is OK: puppet ran at Wed Jun 5 10:26:16 UTC 2013 [10:26:21] RECOVERY - Puppet freshness on analytics1008 is OK: puppet ran at Wed Jun 5 10:26:17 UTC 2013 [10:26:21] RECOVERY - Puppet freshness on mw95 is OK: puppet ran at Wed Jun 5 10:26:18 UTC 2013 [10:26:30] RECOVERY - Puppet freshness on mw39 is OK: puppet ran at Wed Jun 5 10:26:19 UTC 2013 [10:26:30] RECOVERY - Puppet freshness on search1023 is OK: puppet ran at Wed Jun 5 10:26:20 UTC 2013 [10:26:30] RECOVERY - Puppet freshness on search1021 is OK: puppet ran at Wed Jun 5 10:26:21 UTC 2013 [10:26:31] RECOVERY - Puppet freshness on mw1198 is OK: puppet ran at Wed Jun 5 10:26:21 UTC 2013 [10:26:31] RECOVERY - Puppet freshness on mw1136 is OK: puppet ran at Wed Jun 5 10:26:23 UTC 2013 [10:26:31] RECOVERY - Puppet freshness on mw1053 is OK: puppet ran at Wed Jun 5 10:26:23 UTC 2013 [10:26:31] RECOVERY - Puppet freshness on mw1169 is OK: puppet ran at Wed Jun 5 10:26:24 UTC 2013 [10:26:31] RECOVERY - Puppet freshness on mw1094 is OK: puppet ran at Wed Jun 5 10:26:25 UTC 2013 [10:26:32] RECOVERY - Puppet freshness on mw1116 is OK: puppet ran at Wed Jun 5 10:26:26 UTC 2013 [10:26:32] RECOVERY - Puppet freshness on brewster is OK: puppet ran at Wed Jun 5 10:26:27 UTC 2013 [10:26:33] RECOVERY - Puppet freshness on mw1074 is OK: puppet ran at Wed Jun 5 10:26:27 UTC 2013 [10:26:33] RECOVERY - Puppet freshness on db57 is OK: puppet ran at Wed Jun 5 10:26:28 UTC 2013 [10:26:40] RECOVERY - Puppet freshness on sq62 is OK: puppet ran at Wed Jun 5 10:26:29 UTC 2013 [10:26:40] RECOVERY - Puppet freshness on es5 is OK: puppet ran at Wed Jun 5 10:26:30 UTC 2013 [10:26:40] RECOVERY - Puppet freshness on dataset2 is OK: puppet ran at Wed Jun 5 10:26:31 UTC 2013 [10:26:40] RECOVERY - Puppet freshness on stat1002 is OK: puppet ran at Wed Jun 5 10:26:32 UTC 2013 [10:26:40] RECOVERY - Puppet freshness on sq50 is OK: puppet ran at Wed Jun 5 10:26:32 UTC 2013 [10:26:41] RECOVERY - Puppet freshness on db1024 is OK: puppet ran at Wed Jun 5 10:26:33 UTC 2013 [10:26:41] RECOVERY - Puppet freshness on db51 is OK: puppet ran at Wed Jun 5 10:26:34 UTC 2013 [10:26:41] RECOVERY - Puppet freshness on mw36 is OK: puppet ran at Wed Jun 5 10:26:35 UTC 2013 [10:26:42] RECOVERY - Puppet freshness on mw33 is OK: puppet ran at Wed Jun 5 10:26:35 UTC 2013 [10:26:42] RECOVERY - Puppet freshness on cp1025 is OK: puppet ran at Wed Jun 5 10:26:37 UTC 2013 [10:26:43] RECOVERY - Puppet freshness on mw64 is OK: puppet ran at Wed Jun 5 10:26:38 UTC 2013 [10:26:43] RECOVERY - Puppet freshness on wtp1013 is OK: puppet ran at Wed Jun 5 10:26:39 UTC 2013 [10:26:50] RECOVERY - Puppet freshness on mc1008 is OK: puppet ran at Wed Jun 5 10:26:40 UTC 2013 [10:26:50] RECOVERY - Puppet freshness on analytics1019 is OK: puppet ran at Wed Jun 5 10:26:40 UTC 2013 [10:26:50] RECOVERY - Puppet freshness on mw26 is OK: puppet ran at Wed Jun 5 10:26:43 UTC 2013 [10:26:51] RECOVERY - Puppet freshness on mw117 is OK: puppet ran at Wed Jun 5 10:26:45 UTC 2013 [10:26:51] RECOVERY - Puppet freshness on search1015 is OK: puppet ran at Wed Jun 5 10:26:46 UTC 2013 [10:26:51] RECOVERY - Puppet freshness on mw1138 is OK: puppet ran at Wed Jun 5 10:26:49 UTC 2013 [10:27:01] RECOVERY - Puppet freshness on analytics1011 is OK: puppet ran at Wed Jun 5 10:26:50 UTC 2013 [10:27:01] RECOVERY - Puppet freshness on amssq37 is OK: puppet ran at Wed Jun 5 10:26:51 UTC 2013 [10:27:01] RECOVERY - Puppet freshness on db46 is OK: puppet ran at Wed Jun 5 10:26:53 UTC 2013 [10:27:01] RECOVERY - Puppet freshness on tarin is OK: puppet ran at Wed Jun 5 10:26:54 UTC 2013 [10:27:01] RECOVERY - Puppet freshness on mc16 is OK: puppet ran at Wed Jun 5 10:26:55 UTC 2013 [10:27:02] RECOVERY - Puppet freshness on db55 is OK: puppet ran at Wed Jun 5 10:26:56 UTC 2013 [10:27:02] RECOVERY - Puppet freshness on labsdb1001 is OK: puppet ran at Wed Jun 5 10:26:58 UTC 2013 [10:27:10] RECOVERY - Puppet freshness on ms-be1007 is OK: puppet ran at Wed Jun 5 10:26:59 UTC 2013 [10:27:10] RECOVERY - Puppet freshness on srv272 is OK: puppet ran at Wed Jun 5 10:27:00 UTC 2013 [10:27:10] RECOVERY - Puppet freshness on db1010 is OK: puppet ran at Wed Jun 5 10:27:01 UTC 2013 [10:27:10] RECOVERY - Puppet freshness on snapshot4 is OK: puppet ran at Wed Jun 5 10:27:02 UTC 2013 [10:27:10] RECOVERY - Puppet freshness on wtp1021 is OK: puppet ran at Wed Jun 5 10:27:05 UTC 2013 [10:27:11] RECOVERY - Puppet freshness on es1009 is OK: puppet ran at Wed Jun 5 10:27:06 UTC 2013 [10:27:11] RECOVERY - Puppet freshness on lvs6 is OK: puppet ran at Wed Jun 5 10:27:06 UTC 2013 [10:27:12] RECOVERY - Puppet freshness on mw109 is OK: puppet ran at Wed Jun 5 10:27:08 UTC 2013 [10:27:12] RECOVERY - Puppet freshness on mw53 is OK: puppet ran at Wed Jun 5 10:27:09 UTC 2013 [10:27:20] RECOVERY - Puppet freshness on mw40 is OK: puppet ran at Wed Jun 5 10:27:09 UTC 2013 [10:27:20] RECOVERY - Puppet freshness on mw1218 is OK: puppet ran at Wed Jun 5 10:27:10 UTC 2013 [10:27:20] RECOVERY - Puppet freshness on srv296 is OK: puppet ran at Wed Jun 5 10:27:11 UTC 2013 [10:27:20] RECOVERY - Puppet freshness on mw70 is OK: puppet ran at Wed Jun 5 10:27:12 UTC 2013 [10:27:20] RECOVERY - Puppet freshness on cp1023 is OK: puppet ran at Wed Jun 5 10:27:13 UTC 2013 [10:27:21] RECOVERY - Puppet freshness on mw1185 is OK: puppet ran at Wed Jun 5 10:27:14 UTC 2013 [10:27:21] RECOVERY - Puppet freshness on mw1001 is OK: puppet ran at Wed Jun 5 10:27:15 UTC 2013 [10:27:22] RECOVERY - Puppet freshness on lvs2 is OK: puppet ran at Wed Jun 5 10:27:16 UTC 2013 [10:27:22] RECOVERY - Puppet freshness on mw1040 is OK: puppet ran at Wed Jun 5 10:27:17 UTC 2013 [10:27:23] RECOVERY - Puppet freshness on mw1062 is OK: puppet ran at Wed Jun 5 10:27:18 UTC 2013 [10:27:30] RECOVERY - Puppet freshness on lvs1001 is OK: puppet ran at Wed Jun 5 10:27:19 UTC 2013 [10:27:30] RECOVERY - Puppet freshness on mw1132 is OK: puppet ran at Wed Jun 5 10:27:20 UTC 2013 [10:27:30] RECOVERY - Puppet freshness on sq81 is OK: puppet ran at Wed Jun 5 10:27:21 UTC 2013 [10:27:30] RECOVERY - Puppet freshness on sq55 is OK: puppet ran at Wed Jun 5 10:27:21 UTC 2013 [10:27:30] RECOVERY - Puppet freshness on sq64 is OK: puppet ran at Wed Jun 5 10:27:21 UTC 2013 [10:27:30] RECOVERY - Puppet freshness on mc2 is OK: puppet ran at Wed Jun 5 10:27:22 UTC 2013 [10:27:31] RECOVERY - Puppet freshness on amssq59 is OK: puppet ran at Wed Jun 5 10:27:23 UTC 2013 [10:27:31] RECOVERY - Puppet freshness on amssq40 is OK: puppet ran at Wed Jun 5 10:27:25 UTC 2013 [10:27:32] RECOVERY - Puppet freshness on aluminium is OK: puppet ran at Wed Jun 5 10:27:26 UTC 2013 [10:27:32] RECOVERY - Puppet freshness on ms-fe1003 is OK: puppet ran at Wed Jun 5 10:27:27 UTC 2013 [10:27:44] RECOVERY - Puppet freshness on mc1011 is OK: puppet ran at Wed Jun 5 10:27:30 UTC 2013 [10:27:44] RECOVERY - Puppet freshness on analytics1001 is OK: puppet ran at Wed Jun 5 10:27:31 UTC 2013 [10:27:44] RECOVERY - Puppet freshness on srv285 is OK: puppet ran at Wed Jun 5 10:27:33 UTC 2013 [10:27:44] RECOVERY - Puppet freshness on virt2 is OK: puppet ran at Wed Jun 5 10:27:34 UTC 2013 [10:27:44] RECOVERY - Puppet freshness on mw111 is OK: puppet ran at Wed Jun 5 10:27:36 UTC 2013 [10:27:50] RECOVERY - Puppet freshness on mw87 is OK: puppet ran at Wed Jun 5 10:27:41 UTC 2013 [10:27:50] RECOVERY - Puppet freshness on manutius is OK: puppet ran at Wed Jun 5 10:27:43 UTC 2013 [10:27:50] RECOVERY - Puppet freshness on mw1178 is OK: puppet ran at Wed Jun 5 10:27:44 UTC 2013 [10:27:50] RECOVERY - Puppet freshness on mw1219 is OK: puppet ran at Wed Jun 5 10:27:45 UTC 2013 [10:27:50] RECOVERY - Puppet freshness on professor is OK: puppet ran at Wed Jun 5 10:27:46 UTC 2013 [10:27:50] RECOVERY - Puppet freshness on mw1072 is OK: puppet ran at Wed Jun 5 10:27:46 UTC 2013 [10:28:00] RECOVERY - Puppet freshness on db31 is OK: puppet ran at Wed Jun 5 10:27:49 UTC 2013 [10:28:00] RECOVERY - Puppet freshness on cp3006 is OK: puppet ran at Wed Jun 5 10:27:50 UTC 2013 [10:28:01] RECOVERY - Puppet freshness on db1011 is OK: puppet ran at Wed Jun 5 10:27:51 UTC 2013 [10:28:01] RECOVERY - Puppet freshness on mw1031 is OK: puppet ran at Wed Jun 5 10:27:51 UTC 2013 [10:28:01] RECOVERY - Puppet freshness on mw1139 is OK: puppet ran at Wed Jun 5 10:27:52 UTC 2013 [10:28:01] RECOVERY - Puppet freshness on search1006 is OK: puppet ran at Wed Jun 5 10:27:53 UTC 2013 [10:28:01] RECOVERY - Puppet freshness on ms-be10 is OK: puppet ran at Wed Jun 5 10:27:54 UTC 2013 [10:28:01] RECOVERY - Puppet freshness on db29 is OK: puppet ran at Wed Jun 5 10:27:55 UTC 2013 [10:28:02] RECOVERY - Puppet freshness on search27 is OK: puppet ran at Wed Jun 5 10:27:56 UTC 2013 [10:28:10] RECOVERY - Puppet freshness on search1011 is OK: puppet ran at Wed Jun 5 10:27:59 UTC 2013 [10:28:10] RECOVERY - Puppet freshness on magnesium is OK: puppet ran at Wed Jun 5 10:28:00 UTC 2013 [10:28:10] RECOVERY - Puppet freshness on wtp1010 is OK: puppet ran at Wed Jun 5 10:28:01 UTC 2013 [10:28:10] RECOVERY - Puppet freshness on mc1013 is OK: puppet ran at Wed Jun 5 10:28:01 UTC 2013 [10:28:10] RECOVERY - Puppet freshness on sq82 is OK: puppet ran at Wed Jun 5 10:28:01 UTC 2013 [10:28:10] RECOVERY - Puppet freshness on sq78 is OK: puppet ran at Wed Jun 5 10:28:02 UTC 2013 [10:28:10] RECOVERY - Puppet freshness on mc8 is OK: puppet ran at Wed Jun 5 10:28:02 UTC 2013 [10:28:11] RECOVERY - Puppet freshness on mchenry is OK: puppet ran at Wed Jun 5 10:28:03 UTC 2013 [10:28:11] RECOVERY - Puppet freshness on ms-be1002 is OK: puppet ran at Wed Jun 5 10:28:05 UTC 2013 [10:28:12] RECOVERY - Puppet freshness on amssq32 is OK: puppet ran at Wed Jun 5 10:28:05 UTC 2013 [10:28:12] RECOVERY - Puppet freshness on db60 is OK: puppet ran at Wed Jun 5 10:28:06 UTC 2013 [10:28:13] RECOVERY - Puppet freshness on amssq43 is OK: puppet ran at Wed Jun 5 10:28:07 UTC 2013 [10:28:13] RECOVERY - Puppet freshness on db73 is OK: puppet ran at Wed Jun 5 10:28:08 UTC 2013 [10:28:14] RECOVERY - Puppet freshness on db36 is OK: puppet ran at Wed Jun 5 10:28:08 UTC 2013 [10:28:21] RECOVERY - Puppet freshness on mw1080 is OK: puppet ran at Wed Jun 5 10:28:12 UTC 2013 [10:28:21] RECOVERY - Puppet freshness on mw107 is OK: puppet ran at Wed Jun 5 10:28:12 UTC 2013 [10:28:21] RECOVERY - Puppet freshness on mw1134 is OK: puppet ran at Wed Jun 5 10:28:13 UTC 2013 [10:28:21] RECOVERY - Puppet freshness on srv282 is OK: puppet ran at Wed Jun 5 10:28:14 UTC 2013 [10:28:21] RECOVERY - Puppet freshness on cp1015 is OK: puppet ran at Wed Jun 5 10:28:15 UTC 2013 [10:28:21] RECOVERY - Puppet freshness on mw114 is OK: puppet ran at Wed Jun 5 10:28:16 UTC 2013 [10:28:22] RECOVERY - Puppet freshness on srv280 is OK: puppet ran at Wed Jun 5 10:28:17 UTC 2013 [10:28:31] RECOVERY - Puppet freshness on mw1022 is OK: puppet ran at Wed Jun 5 10:28:20 UTC 2013 [10:28:31] RECOVERY - Puppet freshness on tmh1 is OK: puppet ran at Wed Jun 5 10:28:21 UTC 2013 [10:28:31] RECOVERY - Puppet freshness on mw115 is OK: puppet ran at Wed Jun 5 10:28:22 UTC 2013 [10:28:31] RECOVERY - Puppet freshness on srv270 is OK: puppet ran at Wed Jun 5 10:28:23 UTC 2013 [10:28:31] RECOVERY - Puppet freshness on mw65 is OK: puppet ran at Wed Jun 5 10:28:24 UTC 2013 [10:28:32] RECOVERY - Puppet freshness on srv259 is OK: puppet ran at Wed Jun 5 10:28:25 UTC 2013 [10:28:32] RECOVERY - Puppet freshness on mw15 is OK: puppet ran at Wed Jun 5 10:28:26 UTC 2013 [10:28:33] RECOVERY - Puppet freshness on db59 is OK: puppet ran at Wed Jun 5 10:28:29 UTC 2013 [10:28:40] RECOVERY - Puppet freshness on mw1215 is OK: puppet ran at Wed Jun 5 10:28:30 UTC 2013 [10:28:40] RECOVERY - Puppet freshness on mw1090 is OK: puppet ran at Wed Jun 5 10:28:31 UTC 2013 [10:28:40] RECOVERY - Puppet freshness on srv240 is OK: puppet ran at Wed Jun 5 10:28:32 UTC 2013 [10:28:40] RECOVERY - Puppet freshness on mw1059 is OK: puppet ran at Wed Jun 5 10:28:34 UTC 2013 [10:28:40] RECOVERY - Puppet freshness on ms-be1004 is OK: puppet ran at Wed Jun 5 10:28:35 UTC 2013 [10:28:41] RECOVERY - Puppet freshness on mc1016 is OK: puppet ran at Wed Jun 5 10:28:36 UTC 2013 [10:28:41] RECOVERY - Puppet freshness on mw52 is OK: puppet ran at Wed Jun 5 10:28:37 UTC 2013 [10:28:50] RECOVERY - Puppet freshness on stat1001 is OK: puppet ran at Wed Jun 5 10:28:39 UTC 2013 [10:28:50] RECOVERY - Puppet freshness on mw1220 is OK: puppet ran at Wed Jun 5 10:28:40 UTC 2013 [10:28:50] RECOVERY - Puppet freshness on mw1045 is OK: puppet ran at Wed Jun 5 10:28:41 UTC 2013 [10:28:50] RECOVERY - Puppet freshness on mw1200 is OK: puppet ran at Wed Jun 5 10:28:42 UTC 2013 [10:28:50] RECOVERY - Puppet freshness on terbium is OK: puppet ran at Wed Jun 5 10:28:43 UTC 2013 [10:28:51] RECOVERY - Puppet freshness on sq59 is OK: puppet ran at Wed Jun 5 10:28:43 UTC 2013 [10:28:51] RECOVERY - Puppet freshness on mw1203 is OK: puppet ran at Wed Jun 5 10:28:44 UTC 2013 [10:28:52] RECOVERY - Puppet freshness on ssl3001 is OK: puppet ran at Wed Jun 5 10:28:45 UTC 2013 [10:28:52] RECOVERY - Puppet freshness on mw1141 is OK: puppet ran at Wed Jun 5 10:28:46 UTC 2013 [10:28:53] RECOVERY - Puppet freshness on sq37 is OK: puppet ran at Wed Jun 5 10:28:47 UTC 2013 [10:28:53] RECOVERY - Puppet freshness on mw1145 is OK: puppet ran at Wed Jun 5 10:28:48 UTC 2013 [10:28:54] RECOVERY - Puppet freshness on mw1112 is OK: puppet ran at Wed Jun 5 10:28:49 UTC 2013 [10:29:01] RECOVERY - Puppet freshness on arsenic is OK: puppet ran at Wed Jun 5 10:28:49 UTC 2013 [10:29:01] RECOVERY - Puppet freshness on chromium is OK: puppet ran at Wed Jun 5 10:28:51 UTC 2013 [10:29:02] RECOVERY - Puppet freshness on cp1021 is OK: puppet ran at Wed Jun 5 10:28:51 UTC 2013 [10:29:02] RECOVERY - Puppet freshness on db56 is OK: puppet ran at Wed Jun 5 10:28:52 UTC 2013 [10:29:02] RECOVERY - Puppet freshness on db1045 is OK: puppet ran at Wed Jun 5 10:28:55 UTC 2013 [10:29:02] RECOVERY - Puppet freshness on db44 is OK: puppet ran at Wed Jun 5 10:28:56 UTC 2013 [10:29:02] RECOVERY - Puppet freshness on mc1006 is OK: puppet ran at Wed Jun 5 10:28:57 UTC 2013 [10:29:11] RECOVERY - Puppet freshness on es1008 is OK: puppet ran at Wed Jun 5 10:28:59 UTC 2013 [10:29:11] RECOVERY - Puppet freshness on wtp1008 is OK: puppet ran at Wed Jun 5 10:29:00 UTC 2013 [10:29:11] RECOVERY - Puppet freshness on db1057 is OK: puppet ran at Wed Jun 5 10:29:02 UTC 2013 [10:29:11] RECOVERY - Puppet freshness on mw1086 is OK: puppet ran at Wed Jun 5 10:29:04 UTC 2013 [10:29:11] RECOVERY - Puppet freshness on cp1039 is OK: puppet ran at Wed Jun 5 10:29:04 UTC 2013 [10:29:11] RECOVERY - Puppet freshness on linne is OK: puppet ran at Wed Jun 5 10:29:06 UTC 2013 [10:29:11] RECOVERY - Puppet freshness on srv271 is OK: puppet ran at Wed Jun 5 10:29:07 UTC 2013 [10:29:20] RECOVERY - Puppet freshness on search1004 is OK: puppet ran at Wed Jun 5 10:29:09 UTC 2013 [10:29:20] RECOVERY - Puppet freshness on cp1035 is OK: puppet ran at Wed Jun 5 10:29:10 UTC 2013 [10:29:20] RECOVERY - Puppet freshness on mw91 is OK: puppet ran at Wed Jun 5 10:29:11 UTC 2013 [10:29:20] RECOVERY - Puppet freshness on lvs4 is OK: puppet ran at Wed Jun 5 10:29:12 UTC 2013 [10:29:20] RECOVERY - Puppet freshness on mw59 is OK: puppet ran at Wed Jun 5 10:29:13 UTC 2013 [10:29:20] RECOVERY - Puppet freshness on emery is OK: puppet ran at Wed Jun 5 10:29:15 UTC 2013 [10:29:21] RECOVERY - Puppet freshness on ms-be1003 is OK: puppet ran at Wed Jun 5 10:29:16 UTC 2013 [10:29:21] RECOVERY - Puppet freshness on srv298 is OK: puppet ran at Wed Jun 5 10:29:16 UTC 2013 [10:29:22] RECOVERY - Puppet freshness on analytics1020 is OK: puppet ran at Wed Jun 5 10:29:17 UTC 2013 [10:29:22] RECOVERY - Puppet freshness on mw1174 is OK: puppet ran at Wed Jun 5 10:29:18 UTC 2013 [10:29:31] RECOVERY - Puppet freshness on lvs3 is OK: puppet ran at Wed Jun 5 10:29:19 UTC 2013 [10:29:31] RECOVERY - Puppet freshness on mw1110 is OK: puppet ran at Wed Jun 5 10:29:20 UTC 2013 [10:29:31] RECOVERY - Puppet freshness on db1056 is OK: puppet ran at Wed Jun 5 10:29:21 UTC 2013 [10:29:31] RECOVERY - Puppet freshness on wtp1006 is OK: puppet ran at Wed Jun 5 10:29:21 UTC 2013 [10:29:31] RECOVERY - Puppet freshness on ssl1002 is OK: puppet ran at Wed Jun 5 10:29:22 UTC 2013 [10:29:31] RECOVERY - Puppet freshness on mw1135 is OK: puppet ran at Wed Jun 5 10:29:24 UTC 2013 [10:29:31] RECOVERY - Puppet freshness on mw1082 is OK: puppet ran at Wed Jun 5 10:29:24 UTC 2013 [10:29:32] RECOVERY - Puppet freshness on analytics1018 is OK: puppet ran at Wed Jun 5 10:29:25 UTC 2013 [10:29:32] RECOVERY - Puppet freshness on search35 is OK: puppet ran at Wed Jun 5 10:29:26 UTC 2013 [10:29:33] RECOVERY - Puppet freshness on mw1158 is OK: puppet ran at Wed Jun 5 10:29:27 UTC 2013 [10:29:33] RECOVERY - Puppet freshness on wtp1020 is OK: puppet ran at Wed Jun 5 10:29:27 UTC 2013 [10:29:40] RECOVERY - Puppet freshness on mw1026 is OK: puppet ran at Wed Jun 5 10:29:30 UTC 2013 [10:29:40] RECOVERY - Puppet freshness on srv268 is OK: puppet ran at Wed Jun 5 10:29:31 UTC 2013 [10:29:41] RECOVERY - Puppet freshness on mw92 is OK: puppet ran at Wed Jun 5 10:29:31 UTC 2013 [10:29:41] RECOVERY - Puppet freshness on mw113 is OK: puppet ran at Wed Jun 5 10:29:32 UTC 2013 [10:29:41] RECOVERY - Puppet freshness on mw1 is OK: puppet ran at Wed Jun 5 10:29:34 UTC 2013 [10:29:41] RECOVERY - Puppet freshness on analytics1017 is OK: puppet ran at Wed Jun 5 10:29:35 UTC 2013 [10:29:41] RECOVERY - Puppet freshness on amssq50 is OK: puppet ran at Wed Jun 5 10:29:36 UTC 2013 [10:29:41] RECOVERY - Puppet freshness on mw1121 is OK: puppet ran at Wed Jun 5 10:29:37 UTC 2013 [10:29:42] RECOVERY - Puppet freshness on ms1004 is OK: puppet ran at Wed Jun 5 10:29:38 UTC 2013 [10:29:42] RECOVERY - Puppet freshness on sq49 is OK: puppet ran at Wed Jun 5 10:29:38 UTC 2013 [10:29:43] RECOVERY - Puppet freshness on sq77 is OK: puppet ran at Wed Jun 5 10:29:39 UTC 2013 [10:29:50] RECOVERY - Puppet freshness on mc13 is OK: puppet ran at Wed Jun 5 10:29:40 UTC 2013 [10:29:50] RECOVERY - Puppet freshness on es2 is OK: puppet ran at Wed Jun 5 10:29:41 UTC 2013 [10:29:50] RECOVERY - Puppet freshness on cp1001 is OK: puppet ran at Wed Jun 5 10:29:42 UTC 2013 [10:29:50] RECOVERY - Puppet freshness on cp3021 is OK: puppet ran at Wed Jun 5 10:29:42 UTC 2013 [10:29:50] RECOVERY - Puppet freshness on cp3019 is OK: puppet ran at Wed Jun 5 10:29:45 UTC 2013 [10:29:50] RECOVERY - Puppet freshness on virt1005 is OK: puppet ran at Wed Jun 5 10:29:46 UTC 2013 [10:29:50] RECOVERY - Puppet freshness on harmon is OK: puppet ran at Wed Jun 5 10:29:47 UTC 2013 [10:30:00] RECOVERY - Puppet freshness on wtp1016 is OK: puppet ran at Wed Jun 5 10:29:50 UTC 2013 [10:30:00] RECOVERY - Puppet freshness on ms-be1010 is OK: puppet ran at Wed Jun 5 10:29:51 UTC 2013 [10:30:00] RECOVERY - Puppet freshness on search23 is OK: puppet ran at Wed Jun 5 10:29:52 UTC 2013 [10:30:00] RECOVERY - Puppet freshness on srv243 is OK: puppet ran at Wed Jun 5 10:29:52 UTC 2013 [10:30:00] RECOVERY - Puppet freshness on srv251 is OK: puppet ran at Wed Jun 5 10:29:54 UTC 2013 [10:30:00] RECOVERY - Puppet freshness on virt8 is OK: puppet ran at Wed Jun 5 10:29:54 UTC 2013 [10:30:00] RECOVERY - Puppet freshness on mw25 is OK: puppet ran at Wed Jun 5 10:29:55 UTC 2013 [10:30:01] RECOVERY - Puppet freshness on mw3 is OK: puppet ran at Wed Jun 5 10:29:56 UTC 2013 [10:30:02] RECOVERY - Puppet freshness on mw7 is OK: puppet ran at Wed Jun 5 10:29:57 UTC 2013 [10:30:02] RECOVERY - Puppet freshness on srv293 is OK: puppet ran at Wed Jun 5 10:29:58 UTC 2013 [10:30:10] RECOVERY - Puppet freshness on mw1199 is OK: puppet ran at Wed Jun 5 10:29:59 UTC 2013 [10:30:10] RECOVERY - Puppet freshness on mw1108 is OK: puppet ran at Wed Jun 5 10:30:00 UTC 2013 [10:30:10] RECOVERY - Puppet freshness on mw1008 is OK: puppet ran at Wed Jun 5 10:30:02 UTC 2013 [10:30:10] RECOVERY - Puppet freshness on mw1009 is OK: puppet ran at Wed Jun 5 10:30:03 UTC 2013 [10:30:11] RECOVERY - Puppet freshness on mw1148 is OK: puppet ran at Wed Jun 5 10:30:04 UTC 2013 [10:30:11] RECOVERY - Puppet freshness on mw1073 is OK: puppet ran at Wed Jun 5 10:30:05 UTC 2013 [10:30:11] RECOVERY - Puppet freshness on analytics1021 is OK: puppet ran at Wed Jun 5 10:30:05 UTC 2013 [10:30:11] RECOVERY - Puppet freshness on mw1093 is OK: puppet ran at Wed Jun 5 10:30:06 UTC 2013 [10:30:12] RECOVERY - Puppet freshness on mw1060 is OK: puppet ran at Wed Jun 5 10:30:07 UTC 2013 [10:30:12] RECOVERY - Puppet freshness on mw1120 is OK: puppet ran at Wed Jun 5 10:30:08 UTC 2013 [10:30:20] RECOVERY - Puppet freshness on sq66 is OK: puppet ran at Wed Jun 5 10:30:09 UTC 2013 [10:30:20] RECOVERY - Puppet freshness on sq75 is OK: puppet ran at Wed Jun 5 10:30:09 UTC 2013 [10:30:20] RECOVERY - Puppet freshness on nitrogen is OK: puppet ran at Wed Jun 5 10:30:10 UTC 2013 [10:30:20] RECOVERY - Puppet freshness on sq36 is OK: puppet ran at Wed Jun 5 10:30:11 UTC 2013 [10:30:20] RECOVERY - Puppet freshness on db1046 is OK: puppet ran at Wed Jun 5 10:30:11 UTC 2013 [10:30:21] RECOVERY - Puppet freshness on cp1008 is OK: puppet ran at Wed Jun 5 10:30:13 UTC 2013 [10:30:21] RECOVERY - Puppet freshness on hydrogen is OK: puppet ran at Wed Jun 5 10:30:14 UTC 2013 [10:30:22] RECOVERY - Puppet freshness on db1018 is OK: puppet ran at Wed Jun 5 10:30:15 UTC 2013 [10:30:22] RECOVERY - Puppet freshness on cp1009 is OK: puppet ran at Wed Jun 5 10:30:16 UTC 2013 [10:30:23] RECOVERY - Puppet freshness on snapshot3 is OK: puppet ran at Wed Jun 5 10:30:17 UTC 2013 [10:30:23] RECOVERY - Puppet freshness on db1015 is OK: puppet ran at Wed Jun 5 10:30:18 UTC 2013 [10:30:30] RECOVERY - Puppet freshness on ms-fe1004 is OK: puppet ran at Wed Jun 5 10:30:20 UTC 2013 [10:30:30] RECOVERY - Puppet freshness on db64 is OK: puppet ran at Wed Jun 5 10:30:21 UTC 2013 [10:30:30] RECOVERY - Puppet freshness on constable is OK: puppet ran at Wed Jun 5 10:30:22 UTC 2013 [10:30:30] RECOVERY - Puppet freshness on db1049 is OK: puppet ran at Wed Jun 5 10:30:23 UTC 2013 [10:30:30] RECOVERY - Puppet freshness on holmium is OK: puppet ran at Wed Jun 5 10:30:23 UTC 2013 [10:30:31] RECOVERY - Puppet freshness on srv274 is OK: puppet ran at Wed Jun 5 10:30:24 UTC 2013 [10:30:31] RECOVERY - Puppet freshness on srv257 is OK: puppet ran at Wed Jun 5 10:30:25 UTC 2013 [10:30:32] RECOVERY - Puppet freshness on srv235 is OK: puppet ran at Wed Jun 5 10:30:26 UTC 2013 [10:30:32] RECOVERY - Puppet freshness on mw1170 is OK: puppet ran at Wed Jun 5 10:30:27 UTC 2013 [10:30:33] RECOVERY - Puppet freshness on mw48 is OK: puppet ran at Wed Jun 5 10:30:28 UTC 2013 [10:30:40] RECOVERY - Puppet freshness on mw1061 is OK: puppet ran at Wed Jun 5 10:30:30 UTC 2013 [10:30:40] RECOVERY - Puppet freshness on mw56 is OK: puppet ran at Wed Jun 5 10:30:30 UTC 2013 [10:30:40] RECOVERY - Puppet freshness on virt0 is OK: puppet ran at Wed Jun 5 10:30:31 UTC 2013 [10:30:40] RECOVERY - Puppet freshness on mw1085 is OK: puppet ran at Wed Jun 5 10:30:32 UTC 2013 [10:30:40] RECOVERY - Puppet freshness on mw1119 is OK: puppet ran at Wed Jun 5 10:30:33 UTC 2013 [10:30:41] RECOVERY - Puppet freshness on mw1058 is OK: puppet ran at Wed Jun 5 10:30:35 UTC 2013 [10:30:41] RECOVERY - Puppet freshness on cp1042 is OK: puppet ran at Wed Jun 5 10:30:36 UTC 2013 [10:30:42] RECOVERY - Puppet freshness on mw1070 is OK: puppet ran at Wed Jun 5 10:30:37 UTC 2013 [10:30:42] RECOVERY - Puppet freshness on mw1157 is OK: puppet ran at Wed Jun 5 10:30:38 UTC 2013 [10:30:50] RECOVERY - Puppet freshness on mw12 is OK: puppet ran at Wed Jun 5 10:30:39 UTC 2013 [10:30:50] RECOVERY - Puppet freshness on ersch is OK: puppet ran at Wed Jun 5 10:30:40 UTC 2013 [10:30:50] RECOVERY - Puppet freshness on amssq60 is OK: puppet ran at Wed Jun 5 10:30:40 UTC 2013 [10:30:50] RECOVERY - Puppet freshness on amssq36 is OK: puppet ran at Wed Jun 5 10:30:41 UTC 2013 [10:30:50] RECOVERY - Puppet freshness on mc5 is OK: puppet ran at Wed Jun 5 10:30:42 UTC 2013 [10:30:50] RECOVERY - Puppet freshness on amssq41 is OK: puppet ran at Wed Jun 5 10:30:43 UTC 2013 [10:30:51] RECOVERY - Puppet freshness on spence is OK: puppet ran at Wed Jun 5 10:30:45 UTC 2013 [10:30:51] RECOVERY - Puppet freshness on mw1019 is OK: puppet ran at Wed Jun 5 10:30:46 UTC 2013 [10:30:52] RECOVERY - Puppet freshness on mc1002 is OK: puppet ran at Wed Jun 5 10:30:47 UTC 2013 [10:30:52] RECOVERY - Puppet freshness on db1028 is OK: puppet ran at Wed Jun 5 10:30:47 UTC 2013 [10:30:53] RECOVERY - Puppet freshness on sq56 is OK: puppet ran at Wed Jun 5 10:30:48 UTC 2013 [10:30:53] RECOVERY - Puppet freshness on wtp1014 is OK: puppet ran at Wed Jun 5 10:30:49 UTC 2013 [10:31:00] RECOVERY - Puppet freshness on ms-be7 is OK: puppet ran at Wed Jun 5 10:30:50 UTC 2013 [10:31:00] RECOVERY - Puppet freshness on mw96 is OK: puppet ran at Wed Jun 5 10:30:50 UTC 2013 [10:31:00] RECOVERY - Puppet freshness on gallium is OK: puppet ran at Wed Jun 5 10:30:52 UTC 2013 [10:31:00] RECOVERY - Puppet freshness on srv252 is OK: puppet ran at Wed Jun 5 10:30:52 UTC 2013 [10:31:00] RECOVERY - Puppet freshness on mw1177 is OK: puppet ran at Wed Jun 5 10:30:53 UTC 2013 [10:31:01] RECOVERY - Puppet freshness on mw67 is OK: puppet ran at Wed Jun 5 10:30:57 UTC 2013 [10:31:01] RECOVERY - Puppet freshness on mw1184 is OK: puppet ran at Wed Jun 5 10:30:57 UTC 2013 [10:31:02] RECOVERY - Puppet freshness on mw1172 is OK: puppet ran at Wed Jun 5 10:30:58 UTC 2013 [10:31:12] RECOVERY - Puppet freshness on mw1191 is OK: puppet ran at Wed Jun 5 10:30:59 UTC 2013 [10:31:12] RECOVERY - Puppet freshness on ssl1004 is OK: puppet ran at Wed Jun 5 10:31:00 UTC 2013 [10:31:12] RECOVERY - Puppet freshness on mw1039 is OK: puppet ran at Wed Jun 5 10:31:01 UTC 2013 [10:31:12] RECOVERY - Puppet freshness on analytics1006 is OK: puppet ran at Wed Jun 5 10:31:02 UTC 2013 [10:31:12] RECOVERY - Puppet freshness on tmh1002 is OK: puppet ran at Wed Jun 5 10:31:03 UTC 2013 [10:31:12] RECOVERY - Puppet freshness on mw120 is OK: puppet ran at Wed Jun 5 10:31:06 UTC 2013 [10:31:13] RECOVERY - Puppet freshness on srv247 is OK: puppet ran at Wed Jun 5 10:31:06 UTC 2013 [10:31:13] RECOVERY - Puppet freshness on srv239 is OK: puppet ran at Wed Jun 5 10:31:08 UTC 2013 [10:31:14] RECOVERY - Puppet freshness on analytics1024 is OK: puppet ran at Wed Jun 5 10:31:08 UTC 2013 [10:31:22] RECOVERY - Puppet freshness on mw78 is OK: puppet ran at Wed Jun 5 10:31:12 UTC 2013 [10:31:22] RECOVERY - Puppet freshness on ms-be1005 is OK: puppet ran at Wed Jun 5 10:31:12 UTC 2013 [10:31:22] RECOVERY - Puppet freshness on db38 is OK: puppet ran at Wed Jun 5 10:31:13 UTC 2013 [10:31:22] RECOVERY - Puppet freshness on sq57 is OK: puppet ran at Wed Jun 5 10:31:14 UTC 2013 [10:31:22] RECOVERY - Puppet freshness on search1009 is OK: puppet ran at Wed Jun 5 10:31:15 UTC 2013 [10:31:22] RECOVERY - Puppet freshness on sq61 is OK: puppet ran at Wed Jun 5 10:31:16 UTC 2013 [10:31:22] RECOVERY - Puppet freshness on es7 is OK: puppet ran at Wed Jun 5 10:31:16 UTC 2013 [10:31:23] RECOVERY - Puppet freshness on hooper is OK: puppet ran at Wed Jun 5 10:31:17 UTC 2013 [10:31:23] RECOVERY - Puppet freshness on db33 is OK: puppet ran at Wed Jun 5 10:31:17 UTC 2013 [10:31:24] RECOVERY - Puppet freshness on amssq58 is OK: puppet ran at Wed Jun 5 10:31:18 UTC 2013 [10:31:24] RECOVERY - Puppet freshness on db1041 is OK: puppet ran at Wed Jun 5 10:31:18 UTC 2013 [10:31:30] RECOVERY - Puppet freshness on cp1019 is OK: puppet ran at Wed Jun 5 10:31:22 UTC 2013 [10:31:31] RECOVERY - Puppet freshness on nfs2 is OK: puppet ran at Wed Jun 5 10:31:26 UTC 2013 [10:31:31] RECOVERY - Puppet freshness on ms-fe4 is OK: puppet ran at Wed Jun 5 10:31:27 UTC 2013 [10:31:40] RECOVERY - Puppet freshness on db1043 is OK: puppet ran at Wed Jun 5 10:31:30 UTC 2013 [10:31:40] RECOVERY - Puppet freshness on es1007 is OK: puppet ran at Wed Jun 5 10:31:31 UTC 2013 [10:31:40] RECOVERY - Puppet freshness on wtp1012 is OK: puppet ran at Wed Jun 5 10:31:32 UTC 2013 [10:31:40] RECOVERY - Puppet freshness on wtp1005 is OK: puppet ran at Wed Jun 5 10:31:33 UTC 2013 [10:31:40] RECOVERY - Puppet freshness on srv254 is OK: puppet ran at Wed Jun 5 10:31:34 UTC 2013 [10:31:40] RECOVERY - Puppet freshness on mw84 is OK: puppet ran at Wed Jun 5 10:31:35 UTC 2013 [10:31:40] RECOVERY - Puppet freshness on search26 is OK: puppet ran at Wed Jun 5 10:31:35 UTC 2013 [10:31:41] RECOVERY - Puppet freshness on db1003 is OK: puppet ran at Wed Jun 5 10:31:37 UTC 2013 [10:31:42] RECOVERY - Puppet freshness on mw63 is OK: puppet ran at Wed Jun 5 10:31:38 UTC 2013 [10:31:42] RECOVERY - Puppet freshness on mw1013 is OK: puppet ran at Wed Jun 5 10:31:39 UTC 2013 [10:31:51] RECOVERY - Puppet freshness on mw1195 is OK: puppet ran at Wed Jun 5 10:31:39 UTC 2013 [10:31:51] RECOVERY - Puppet freshness on analytics1016 is OK: puppet ran at Wed Jun 5 10:31:40 UTC 2013 [10:31:51] RECOVERY - Puppet freshness on mw1035 is OK: puppet ran at Wed Jun 5 10:31:41 UTC 2013 [10:31:51] RECOVERY - Puppet freshness on db37 is OK: puppet ran at Wed Jun 5 10:31:42 UTC 2013 [10:31:51] RECOVERY - Puppet freshness on mw1149 is OK: puppet ran at Wed Jun 5 10:31:43 UTC 2013 [10:31:52] RECOVERY - Puppet freshness on mw1076 is OK: puppet ran at Wed Jun 5 10:31:45 UTC 2013 [10:31:52] RECOVERY - Puppet freshness on sq53 is OK: puppet ran at Wed Jun 5 10:31:46 UTC 2013 [10:31:52] RECOVERY - Puppet freshness on lvs1003 is OK: puppet ran at Wed Jun 5 10:31:47 UTC 2013 [10:31:53] RECOVERY - Puppet freshness on virt10 is OK: puppet ran at Wed Jun 5 10:31:47 UTC 2013 [10:32:00] RECOVERY - Puppet freshness on mw1036 is OK: puppet ran at Wed Jun 5 10:31:51 UTC 2013 [10:32:00] RECOVERY - Puppet freshness on mw69 is OK: puppet ran at Wed Jun 5 10:31:51 UTC 2013 [10:32:01] RECOVERY - Puppet freshness on mw17 is OK: puppet ran at Wed Jun 5 10:31:52 UTC 2013 [10:32:01] RECOVERY - Puppet freshness on mw1167 is OK: puppet ran at Wed Jun 5 10:31:53 UTC 2013 [10:32:01] RECOVERY - Puppet freshness on silver is OK: puppet ran at Wed Jun 5 10:31:54 UTC 2013 [10:32:01] RECOVERY - Puppet freshness on mw1055 is OK: puppet ran at Wed Jun 5 10:31:55 UTC 2013 [10:32:01] RECOVERY - Puppet freshness on db50 is OK: puppet ran at Wed Jun 5 10:31:56 UTC 2013 [10:32:02] RECOVERY - Puppet freshness on amslvs3 is OK: puppet ran at Wed Jun 5 10:31:57 UTC 2013 [10:32:02] RECOVERY - Puppet freshness on es3 is OK: puppet ran at Wed Jun 5 10:31:57 UTC 2013 [10:32:03] RECOVERY - Puppet freshness on db1039 is OK: puppet ran at Wed Jun 5 10:31:58 UTC 2013 [10:32:10] RECOVERY - Puppet freshness on db1007 is OK: puppet ran at Wed Jun 5 10:32:00 UTC 2013 [10:32:10] RECOVERY - Puppet freshness on mc1014 is OK: puppet ran at Wed Jun 5 10:32:00 UTC 2013 [10:32:10] RECOVERY - Puppet freshness on mc1010 is OK: puppet ran at Wed Jun 5 10:32:02 UTC 2013 [10:32:10] RECOVERY - Puppet freshness on cp3022 is OK: puppet ran at Wed Jun 5 10:32:03 UTC 2013 [10:32:10] RECOVERY - Puppet freshness on sq69 is OK: puppet ran at Wed Jun 5 10:32:06 UTC 2013 [10:32:11] RECOVERY - Puppet freshness on ssl3003 is OK: puppet ran at Wed Jun 5 10:32:07 UTC 2013 [10:32:11] RECOVERY - Puppet freshness on cp1044 is OK: puppet ran at Wed Jun 5 10:32:08 UTC 2013 [10:32:20] RECOVERY - Puppet freshness on db68 is OK: puppet ran at Wed Jun 5 10:32:09 UTC 2013 [10:32:20] RECOVERY - Puppet freshness on snapshot1001 is OK: puppet ran at Wed Jun 5 10:32:09 UTC 2013 [10:32:20] RECOVERY - Puppet freshness on analytics1026 is OK: puppet ran at Wed Jun 5 10:32:11 UTC 2013 [10:32:20] RECOVERY - Puppet freshness on search24 is OK: puppet ran at Wed Jun 5 10:32:11 UTC 2013 [10:32:20] RECOVERY - Puppet freshness on nfs1 is OK: puppet ran at Wed Jun 5 10:32:12 UTC 2013 [10:32:21] RECOVERY - Puppet freshness on sq44 is OK: puppet ran at Wed Jun 5 10:32:13 UTC 2013 [10:32:21] RECOVERY - Puppet freshness on mw32 is OK: puppet ran at Wed Jun 5 10:32:16 UTC 2013 [10:32:22] RECOVERY - Puppet freshness on mw88 is OK: puppet ran at Wed Jun 5 10:32:16 UTC 2013 [10:32:22] RECOVERY - Puppet freshness on mw1216 is OK: puppet ran at Wed Jun 5 10:32:17 UTC 2013 [10:32:23] RECOVERY - Puppet freshness on search1002 is OK: puppet ran at Wed Jun 5 10:32:18 UTC 2013 [10:32:23] RECOVERY - Puppet freshness on analytics1013 is OK: puppet ran at Wed Jun 5 10:32:19 UTC 2013 [10:32:30] RECOVERY - Puppet freshness on mw1192 is OK: puppet ran at Wed Jun 5 10:32:19 UTC 2013 [10:32:30] RECOVERY - Puppet freshness on mw18 is OK: puppet ran at Wed Jun 5 10:32:20 UTC 2013 [10:32:31] RECOVERY - Puppet freshness on mw1014 is OK: puppet ran at Wed Jun 5 10:32:21 UTC 2013 [10:32:31] RECOVERY - Puppet freshness on es1006 is OK: puppet ran at Wed Jun 5 10:32:21 UTC 2013 [10:32:31] RECOVERY - Puppet freshness on mw1051 is OK: puppet ran at Wed Jun 5 10:32:23 UTC 2013 [10:32:31] RECOVERY - Puppet freshness on mw1168 is OK: puppet ran at Wed Jun 5 10:32:24 UTC 2013 [10:32:31] RECOVERY - Puppet freshness on mw1133 is OK: puppet ran at Wed Jun 5 10:32:25 UTC 2013 [10:32:32] RECOVERY - Puppet freshness on mw1079 is OK: puppet ran at Wed Jun 5 10:32:26 UTC 2013 [10:32:32] RECOVERY - Puppet freshness on iron is OK: puppet ran at Wed Jun 5 10:32:27 UTC 2013 [10:32:33] RECOVERY - Puppet freshness on cp1011 is OK: puppet ran at Wed Jun 5 10:32:27 UTC 2013 [10:32:33] RECOVERY - Puppet freshness on db49 is OK: puppet ran at Wed Jun 5 10:32:28 UTC 2013 [10:32:40] RECOVERY - Puppet freshness on mw86 is OK: puppet ran at Wed Jun 5 10:32:30 UTC 2013 [10:32:40] RECOVERY - Puppet freshness on cp1004 is OK: puppet ran at Wed Jun 5 10:32:31 UTC 2013 [10:32:40] RECOVERY - Puppet freshness on rdb1001 is OK: puppet ran at Wed Jun 5 10:32:32 UTC 2013 [10:32:40] RECOVERY - Puppet freshness on formey is OK: puppet ran at Wed Jun 5 10:32:34 UTC 2013 [10:32:40] RECOVERY - Puppet freshness on mw1202 is OK: puppet ran at Wed Jun 5 10:32:36 UTC 2013 [10:32:41] RECOVERY - Puppet freshness on wtp1 is OK: puppet ran at Wed Jun 5 10:32:36 UTC 2013 [10:32:41] RECOVERY - Puppet freshness on mw1151 is OK: puppet ran at Wed Jun 5 10:32:37 UTC 2013 [10:32:42] RECOVERY - Puppet freshness on mw1098 is OK: puppet ran at Wed Jun 5 10:32:38 UTC 2013 [10:32:42] RECOVERY - Puppet freshness on ms-be1008 is OK: puppet ran at Wed Jun 5 10:32:39 UTC 2013 [10:32:50] RECOVERY - Puppet freshness on db74 is OK: puppet ran at Wed Jun 5 10:32:40 UTC 2013 [10:32:50] RECOVERY - Puppet freshness on mc1004 is OK: puppet ran at Wed Jun 5 10:32:41 UTC 2013 [10:32:50] RECOVERY - Puppet freshness on db72 is OK: puppet ran at Wed Jun 5 10:32:42 UTC 2013 [10:32:50] RECOVERY - Puppet freshness on mexia is OK: puppet ran at Wed Jun 5 10:32:42 UTC 2013 [10:32:50] RECOVERY - Puppet freshness on sq70 is OK: puppet ran at Wed Jun 5 10:32:43 UTC 2013 [10:32:51] RECOVERY - Puppet freshness on search20 is OK: puppet ran at Wed Jun 5 10:32:45 UTC 2013 [10:32:51] RECOVERY - Puppet freshness on db1016 is OK: puppet ran at Wed Jun 5 10:32:46 UTC 2013 [10:32:51] RECOVERY - Puppet freshness on oxygen is OK: puppet ran at Wed Jun 5 10:32:47 UTC 2013 [10:32:52] RECOVERY - Puppet freshness on db1030 is OK: puppet ran at Wed Jun 5 10:32:48 UTC 2013 [10:32:52] RECOVERY - Puppet freshness on hooft is OK: puppet ran at Wed Jun 5 10:32:49 UTC 2013 [10:33:01] RECOVERY - Puppet freshness on cp1027 is OK: puppet ran at Wed Jun 5 10:32:49 UTC 2013 [10:33:01] RECOVERY - Puppet freshness on mw85 is OK: puppet ran at Wed Jun 5 10:32:50 UTC 2013 [10:33:01] RECOVERY - Puppet freshness on mw97 is OK: puppet ran at Wed Jun 5 10:32:51 UTC 2013 [10:33:01] RECOVERY - Puppet freshness on srv263 is OK: puppet ran at Wed Jun 5 10:32:52 UTC 2013 [10:33:01] RECOVERY - Puppet freshness on search32 is OK: puppet ran at Wed Jun 5 10:32:53 UTC 2013 [10:33:01] RECOVERY - Puppet freshness on mw123 is OK: puppet ran at Wed Jun 5 10:32:55 UTC 2013 [10:33:01] RECOVERY - Puppet freshness on mw72 is OK: puppet ran at Wed Jun 5 10:32:56 UTC 2013 [10:33:02] RECOVERY - Puppet freshness on virt1007 is OK: puppet ran at Wed Jun 5 10:32:57 UTC 2013 [10:33:02] RECOVERY - Puppet freshness on ssl3002 is OK: puppet ran at Wed Jun 5 10:32:58 UTC 2013 [10:33:03] RECOVERY - Puppet freshness on searchidx2 is OK: puppet ran at Wed Jun 5 10:32:59 UTC 2013 [10:33:12] RECOVERY - Puppet freshness on mw4 is OK: puppet ran at Wed Jun 5 10:33:01 UTC 2013 [10:33:12] RECOVERY - Puppet freshness on mw1034 is OK: puppet ran at Wed Jun 5 10:33:02 UTC 2013 [10:33:12] RECOVERY - Puppet freshness on mw1183 is OK: puppet ran at Wed Jun 5 10:33:03 UTC 2013 [10:33:12] RECOVERY - Puppet freshness on mw1050 is OK: puppet ran at Wed Jun 5 10:33:04 UTC 2013 [10:33:12] RECOVERY - Puppet freshness on virt7 is OK: puppet ran at Wed Jun 5 10:33:05 UTC 2013 [10:33:12] RECOVERY - Puppet freshness on cp1041 is OK: puppet ran at Wed Jun 5 10:33:06 UTC 2013 [10:33:12] RECOVERY - Puppet freshness on search1013 is OK: puppet ran at Wed Jun 5 10:33:07 UTC 2013 [10:33:12] RECOVERY - Puppet freshness on ms1002 is OK: puppet ran at Wed Jun 5 10:33:08 UTC 2013 [10:33:12] RECOVERY - Puppet freshness on amssq46 is OK: puppet ran at Wed Jun 5 10:33:08 UTC 2013 [10:33:20] RECOVERY - Puppet freshness on db1059 is OK: puppet ran at Wed Jun 5 10:33:09 UTC 2013 [10:33:21] RECOVERY - Puppet freshness on db78 is OK: puppet ran at Wed Jun 5 10:33:10 UTC 2013 [10:33:21] RECOVERY - Puppet freshness on es8 is OK: puppet ran at Wed Jun 5 10:33:11 UTC 2013 [10:33:21] RECOVERY - Puppet freshness on virt5 is OK: puppet ran at Wed Jun 5 10:33:12 UTC 2013 [10:33:21] RECOVERY - Puppet freshness on wtp1022 is OK: puppet ran at Wed Jun 5 10:33:13 UTC 2013 [10:33:21] RECOVERY - Puppet freshness on lardner is OK: puppet ran at Wed Jun 5 10:33:14 UTC 2013 [10:33:21] RECOVERY - Puppet freshness on wtp1023 is OK: puppet ran at Wed Jun 5 10:33:16 UTC 2013 [10:33:22] RECOVERY - Puppet freshness on mw80 is OK: puppet ran at Wed Jun 5 10:33:17 UTC 2013 [10:33:22] RECOVERY - Puppet freshness on ssl3 is OK: puppet ran at Wed Jun 5 10:33:18 UTC 2013 [10:33:23] RECOVERY - Puppet freshness on dysprosium is OK: puppet ran at Wed Jun 5 10:33:19 UTC 2013 [10:33:31] RECOVERY - Puppet freshness on srv279 is OK: puppet ran at Wed Jun 5 10:33:21 UTC 2013 [10:33:31] RECOVERY - Puppet freshness on mw6 is OK: puppet ran at Wed Jun 5 10:33:22 UTC 2013 [10:33:32] RECOVERY - Puppet freshness on mw1163 is OK: puppet ran at Wed Jun 5 10:33:23 UTC 2013 [10:33:32] RECOVERY - Puppet freshness on snapshot1004 is OK: puppet ran at Wed Jun 5 10:33:24 UTC 2013 [10:33:32] RECOVERY - Puppet freshness on search1016 is OK: puppet ran at Wed Jun 5 10:33:25 UTC 2013 [10:33:32] RECOVERY - Puppet freshness on cp1033 is OK: puppet ran at Wed Jun 5 10:33:25 UTC 2013 [10:33:32] RECOVERY - Puppet freshness on mw1212 is OK: puppet ran at Wed Jun 5 10:33:26 UTC 2013 [10:33:32] RECOVERY - Puppet freshness on mw1096 is OK: puppet ran at Wed Jun 5 10:33:27 UTC 2013 [10:33:33] RECOVERY - Puppet freshness on search1017 is OK: puppet ran at Wed Jun 5 10:33:28 UTC 2013 [10:33:41] RECOVERY - Puppet freshness on mw71 is OK: puppet ran at Wed Jun 5 10:33:31 UTC 2013 [10:33:41] RECOVERY - Puppet freshness on db1053 is OK: puppet ran at Wed Jun 5 10:33:32 UTC 2013 [10:33:41] RECOVERY - Puppet freshness on mw1002 is OK: puppet ran at Wed Jun 5 10:33:33 UTC 2013 [10:33:41] RECOVERY - Puppet freshness on mw102 is OK: puppet ran at Wed Jun 5 10:33:35 UTC 2013 [10:33:41] RECOVERY - Puppet freshness on srv288 is OK: puppet ran at Wed Jun 5 10:33:36 UTC 2013 [10:33:42] RECOVERY - Puppet freshness on srv241 is OK: puppet ran at Wed Jun 5 10:33:37 UTC 2013 [10:33:42] RECOVERY - Puppet freshness on sq80 is OK: puppet ran at Wed Jun 5 10:33:38 UTC 2013 [10:33:43] RECOVERY - Puppet freshness on db1008 is OK: puppet ran at Wed Jun 5 10:33:39 UTC 2013 [10:33:50] RECOVERY - Puppet freshness on db1038 is OK: puppet ran at Wed Jun 5 10:33:41 UTC 2013 [10:33:50] RECOVERY - Puppet freshness on db1006 is OK: puppet ran at Wed Jun 5 10:33:40 UTC 2013 [10:33:50] RECOVERY - Puppet freshness on mw44 is OK: puppet ran at Wed Jun 5 10:33:42 UTC 2013 [10:33:50] RECOVERY - Puppet freshness on db35 is OK: puppet ran at Wed Jun 5 10:33:42 UTC 2013 [10:33:50] RECOVERY - Puppet freshness on amssq38 is OK: puppet ran at Wed Jun 5 10:33:42 UTC 2013 [10:33:51] RECOVERY - Puppet freshness on pc3 is OK: puppet ran at Wed Jun 5 10:33:43 UTC 2013 [10:33:51] RECOVERY - Puppet freshness on mw27 is OK: puppet ran at Wed Jun 5 10:33:43 UTC 2013 [10:33:52] RECOVERY - Puppet freshness on mw1097 is OK: puppet ran at Wed Jun 5 10:33:44 UTC 2013 [10:33:52] RECOVERY - Puppet freshness on db47 is OK: puppet ran at Wed Jun 5 10:33:46 UTC 2013 [10:33:53] RECOVERY - Puppet freshness on db62 is OK: puppet ran at Wed Jun 5 10:33:46 UTC 2013 [10:33:53] RECOVERY - Puppet freshness on mw1029 is OK: puppet ran at Wed Jun 5 10:33:47 UTC 2013 [10:34:00] RECOVERY - Puppet freshness on ms1001 is OK: puppet ran at Wed Jun 5 10:33:50 UTC 2013 [10:34:01] RECOVERY - Puppet freshness on wtp1017 is OK: puppet ran at Wed Jun 5 10:33:53 UTC 2013 [10:34:01] RECOVERY - Puppet freshness on mw1077 is OK: puppet ran at Wed Jun 5 10:33:53 UTC 2013 [10:34:01] RECOVERY - Puppet freshness on wtp1019 is OK: puppet ran at Wed Jun 5 10:33:55 UTC 2013 [10:34:01] RECOVERY - Puppet freshness on ms-be1011 is OK: puppet ran at Wed Jun 5 10:33:56 UTC 2013 [10:34:01] RECOVERY - Puppet freshness on es1010 is OK: puppet ran at Wed Jun 5 10:33:56 UTC 2013 [10:34:01] RECOVERY - Puppet freshness on mw1023 is OK: puppet ran at Wed Jun 5 10:33:57 UTC 2013 [10:34:02] RECOVERY - Puppet freshness on locke is OK: puppet ran at Wed Jun 5 10:33:58 UTC 2013 [10:34:10] RECOVERY - Puppet freshness on srv281 is OK: puppet ran at Wed Jun 5 10:33:59 UTC 2013 [10:34:10] RECOVERY - Puppet freshness on analytics1027 is OK: puppet ran at Wed Jun 5 10:34:01 UTC 2013 [10:34:10] RECOVERY - Puppet freshness on mw89 is OK: puppet ran at Wed Jun 5 10:34:02 UTC 2013 [10:34:10] RECOVERY - Puppet freshness on cp1031 is OK: puppet ran at Wed Jun 5 10:34:02 UTC 2013 [10:34:10] RECOVERY - Puppet freshness on mw21 is OK: puppet ran at Wed Jun 5 10:34:03 UTC 2013 [10:34:11] RECOVERY - Puppet freshness on mw1209 is OK: puppet ran at Wed Jun 5 10:34:07 UTC 2013 [10:34:11] RECOVERY - Puppet freshness on mw1005 is OK: puppet ran at Wed Jun 5 10:34:07 UTC 2013 [10:34:12] RECOVERY - Puppet freshness on pc1001 is OK: puppet ran at Wed Jun 5 10:34:08 UTC 2013 [10:34:12] RECOVERY - Puppet freshness on search1020 is OK: puppet ran at Wed Jun 5 10:34:09 UTC 2013 [10:34:20] RECOVERY - Puppet freshness on mw1067 is OK: puppet ran at Wed Jun 5 10:34:10 UTC 2013 [10:34:20] RECOVERY - Puppet freshness on mw1152 is OK: puppet ran at Wed Jun 5 10:34:11 UTC 2013 [10:34:20] RECOVERY - Puppet freshness on db1047 is OK: puppet ran at Wed Jun 5 10:34:12 UTC 2013 [10:34:20] RECOVERY - Puppet freshness on ms-be6 is OK: puppet ran at Wed Jun 5 10:34:14 UTC 2013 [10:34:20] RECOVERY - Puppet freshness on sq60 is OK: puppet ran at Wed Jun 5 10:34:14 UTC 2013 [10:34:21] RECOVERY - Puppet freshness on cp1022 is OK: puppet ran at Wed Jun 5 10:34:17 UTC 2013 [10:34:22] RECOVERY - Puppet freshness on cp1029 is OK: puppet ran at Wed Jun 5 10:34:18 UTC 2013 [10:34:22] RECOVERY - Puppet freshness on mw1016 is OK: puppet ran at Wed Jun 5 10:34:18 UTC 2013 [10:34:30] RECOVERY - Puppet freshness on amssq54 is OK: puppet ran at Wed Jun 5 10:34:22 UTC 2013 [10:34:30] RECOVERY - Puppet freshness on amssq42 is OK: puppet ran at Wed Jun 5 10:34:22 UTC 2013 [10:34:30] RECOVERY - Puppet freshness on amssq62 is OK: puppet ran at Wed Jun 5 10:34:23 UTC 2013 [10:34:30] RECOVERY - Puppet freshness on carbon is OK: puppet ran at Wed Jun 5 10:34:23 UTC 2013 [10:34:30] RECOVERY - Puppet freshness on es1001 is OK: puppet ran at Wed Jun 5 10:34:27 UTC 2013 [10:34:30] RECOVERY - Puppet freshness on kaulen is OK: puppet ran at Wed Jun 5 10:34:28 UTC 2013 [10:34:31] RECOVERY - Puppet freshness on mc9 is OK: puppet ran at Wed Jun 5 10:34:29 UTC 2013 [10:34:40] RECOVERY - Puppet freshness on zhen is OK: puppet ran at Wed Jun 5 10:34:30 UTC 2013 [10:34:40] RECOVERY - Puppet freshness on cp3020 is OK: puppet ran at Wed Jun 5 10:34:31 UTC 2013 [10:34:40] RECOVERY - Puppet freshness on mw1028 is OK: puppet ran at Wed Jun 5 10:34:32 UTC 2013 [10:34:40] RECOVERY - Puppet freshness on virt6 is OK: puppet ran at Wed Jun 5 10:34:33 UTC 2013 [10:34:40] RECOVERY - Puppet freshness on ms-fe1 is OK: puppet ran at Wed Jun 5 10:34:33 UTC 2013 [10:34:41] RECOVERY - Puppet freshness on db71 is OK: puppet ran at Wed Jun 5 10:34:35 UTC 2013 [10:34:41] RECOVERY - Puppet freshness on mw1156 is OK: puppet ran at Wed Jun 5 10:34:37 UTC 2013 [10:34:41] RECOVERY - Puppet freshness on db1017 is OK: puppet ran at Wed Jun 5 10:34:37 UTC 2013 [10:34:42] RECOVERY - Puppet freshness on amssq33 is OK: puppet ran at Wed Jun 5 10:34:38 UTC 2013 [10:34:52] RECOVERY - Puppet freshness on amssq45 is OK: puppet ran at Wed Jun 5 10:34:39 UTC 2013 [10:34:52] RECOVERY - Puppet freshness on amssq49 is OK: puppet ran at Wed Jun 5 10:34:40 UTC 2013 [10:34:52] RECOVERY - Puppet freshness on cp1032 is OK: puppet ran at Wed Jun 5 10:34:43 UTC 2013 [10:34:52] RECOVERY - Puppet freshness on search34 is OK: puppet ran at Wed Jun 5 10:34:44 UTC 2013 [10:34:52] RECOVERY - Puppet freshness on mw31 is OK: puppet ran at Wed Jun 5 10:34:45 UTC 2013 [10:34:52] RECOVERY - Puppet freshness on srv248 is OK: puppet ran at Wed Jun 5 10:34:45 UTC 2013 [10:34:52] RECOVERY - Puppet freshness on mw11 is OK: puppet ran at Wed Jun 5 10:34:48 UTC 2013 [10:35:00] RECOVERY - Puppet freshness on srv277 is OK: puppet ran at Wed Jun 5 10:34:49 UTC 2013 [10:35:00] RECOVERY - Puppet freshness on srv265 is OK: puppet ran at Wed Jun 5 10:34:50 UTC 2013 [10:35:01] RECOVERY - Puppet freshness on cp1014 is OK: puppet ran at Wed Jun 5 10:34:51 UTC 2013 [10:35:01] RECOVERY - Puppet freshness on cp3008 is OK: puppet ran at Wed Jun 5 10:34:52 UTC 2013 [10:35:01] RECOVERY - Puppet freshness on mw22 is OK: puppet ran at Wed Jun 5 10:34:53 UTC 2013 [10:35:01] RECOVERY - Puppet freshness on mw19 is OK: puppet ran at Wed Jun 5 10:34:54 UTC 2013 [10:35:01] RECOVERY - Puppet freshness on mw24 is OK: puppet ran at Wed Jun 5 10:34:55 UTC 2013 [10:35:02] RECOVERY - Puppet freshness on mw1193 is OK: puppet ran at Wed Jun 5 10:34:58 UTC 2013 [10:35:02] RECOVERY - Puppet freshness on search15 is OK: puppet ran at Wed Jun 5 10:34:58 UTC 2013 [10:35:11] RECOVERY - Puppet freshness on snapshot2 is OK: puppet ran at Wed Jun 5 10:35:00 UTC 2013 [10:35:11] RECOVERY - Puppet freshness on mw1064 is OK: puppet ran at Wed Jun 5 10:35:00 UTC 2013 [10:35:12] RECOVERY - Puppet freshness on wtp1009 is OK: puppet ran at Wed Jun 5 10:35:01 UTC 2013 [10:35:12] RECOVERY - Puppet freshness on mw1012 is OK: puppet ran at Wed Jun 5 10:35:02 UTC 2013 [10:35:12] RECOVERY - Puppet freshness on mw1071 is OK: puppet ran at Wed Jun 5 10:35:03 UTC 2013 [10:35:12] RECOVERY - Puppet freshness on ssl2 is OK: puppet ran at Wed Jun 5 10:35:04 UTC 2013 [10:35:12] RECOVERY - Puppet freshness on mw77 is OK: puppet ran at Wed Jun 5 10:35:04 UTC 2013 [10:35:12] RECOVERY - Puppet freshness on yvon is OK: puppet ran at Wed Jun 5 10:35:07 UTC 2013 [10:35:13] RECOVERY - Puppet freshness on srv256 is OK: puppet ran at Wed Jun 5 10:35:08 UTC 2013 [10:35:20] RECOVERY - Puppet freshness on tridge is OK: puppet ran at Wed Jun 5 10:35:09 UTC 2013 [10:35:20] RECOVERY - Puppet freshness on cp1040 is OK: puppet ran at Wed Jun 5 10:35:10 UTC 2013 [10:35:20] RECOVERY - Puppet freshness on amssq39 is OK: puppet ran at Wed Jun 5 10:35:11 UTC 2013 [10:35:20] RECOVERY - Puppet freshness on amssq61 is OK: puppet ran at Wed Jun 5 10:35:12 UTC 2013 [10:35:20] RECOVERY - Puppet freshness on mw1006 is OK: puppet ran at Wed Jun 5 10:35:13 UTC 2013 [10:35:21] RECOVERY - Puppet freshness on db1035 is OK: puppet ran at Wed Jun 5 10:35:14 UTC 2013 [10:35:21] RECOVERY - Puppet freshness on strontium is OK: puppet ran at Wed Jun 5 10:35:15 UTC 2013 [10:35:22] RECOVERY - Puppet freshness on cp3004 is OK: puppet ran at Wed Jun 5 10:35:16 UTC 2013 [10:35:22] RECOVERY - Puppet freshness on cp3005 is OK: puppet ran at Wed Jun 5 10:35:17 UTC 2013 [10:35:23] RECOVERY - Puppet freshness on db1037 is OK: puppet ran at Wed Jun 5 10:35:18 UTC 2013 [10:35:23] RECOVERY - Puppet freshness on tola is OK: puppet ran at Wed Jun 5 10:35:19 UTC 2013 [10:35:30] RECOVERY - Puppet freshness on cp1043 is OK: puppet ran at Wed Jun 5 10:35:19 UTC 2013 [10:35:31] RECOVERY - Puppet freshness on analytics1025 is OK: puppet ran at Wed Jun 5 10:35:20 UTC 2013 [10:35:31] RECOVERY - Puppet freshness on mw103 is OK: puppet ran at Wed Jun 5 10:35:21 UTC 2013 [10:35:31] RECOVERY - Puppet freshness on mw101 is OK: puppet ran at Wed Jun 5 10:35:22 UTC 2013 [10:35:31] RECOVERY - Puppet freshness on snapshot1003 is OK: puppet ran at Wed Jun 5 10:35:23 UTC 2013 [10:35:31] RECOVERY - Puppet freshness on analytics1005 is OK: puppet ran at Wed Jun 5 10:35:24 UTC 2013 [10:35:31] RECOVERY - Puppet freshness on mw108 is OK: puppet ran at Wed Jun 5 10:35:24 UTC 2013 [10:35:32] RECOVERY - Puppet freshness on search28 is OK: puppet ran at Wed Jun 5 10:35:25 UTC 2013 [10:35:32] RECOVERY - Puppet freshness on mw94 is OK: puppet ran at Wed Jun 5 10:35:26 UTC 2013 [10:35:40] RECOVERY - Puppet freshness on mw105 is OK: puppet ran at Wed Jun 5 10:35:29 UTC 2013 [10:35:40] RECOVERY - Puppet freshness on virt9 is OK: puppet ran at Wed Jun 5 10:35:30 UTC 2013 [10:35:40] RECOVERY - Puppet freshness on mw54 is OK: puppet ran at Wed Jun 5 10:35:31 UTC 2013 [10:35:40] RECOVERY - Puppet freshness on mw29 is OK: puppet ran at Wed Jun 5 10:35:33 UTC 2013 [10:35:40] RECOVERY - Puppet freshness on srv295 is OK: puppet ran at Wed Jun 5 10:35:34 UTC 2013 [10:35:41] RECOVERY - Puppet freshness on mw1037 is OK: puppet ran at Wed Jun 5 10:35:35 UTC 2013 [10:35:41] RECOVERY - Puppet freshness on mw1207 is OK: puppet ran at Wed Jun 5 10:35:36 UTC 2013 [10:35:42] RECOVERY - Puppet freshness on mw1187 is OK: puppet ran at Wed Jun 5 10:35:37 UTC 2013 [10:35:42] RECOVERY - Puppet freshness on ms-be1001 is OK: puppet ran at Wed Jun 5 10:35:39 UTC 2013 [10:35:50] RECOVERY - Puppet freshness on mw1113 is OK: puppet ran at Wed Jun 5 10:35:40 UTC 2013 [10:35:50] RECOVERY - Puppet freshness on mw1160 is OK: puppet ran at Wed Jun 5 10:35:40 UTC 2013 [10:35:50] RECOVERY - Puppet freshness on maerlant is OK: puppet ran at Wed Jun 5 10:35:40 UTC 2013 [10:35:50] RECOVERY - Puppet freshness on cp3003 is OK: puppet ran at Wed Jun 5 10:35:42 UTC 2013 [10:35:50] RECOVERY - Puppet freshness on mc1015 is OK: puppet ran at Wed Jun 5 10:35:43 UTC 2013 [10:35:51] RECOVERY - Puppet freshness on sq67 is OK: puppet ran at Wed Jun 5 10:35:43 UTC 2013 [10:35:51] RECOVERY - Puppet freshness on db52 is OK: puppet ran at Wed Jun 5 10:35:44 UTC 2013 [10:35:52] RECOVERY - Puppet freshness on mc1003 is OK: puppet ran at Wed Jun 5 10:35:45 UTC 2013 [10:35:52] RECOVERY - Puppet freshness on nescio is OK: puppet ran at Wed Jun 5 10:35:46 UTC 2013 [10:35:53] RECOVERY - Puppet freshness on mw51 is OK: puppet ran at Wed Jun 5 10:35:47 UTC 2013 [10:35:53] RECOVERY - Puppet freshness on srv287 is OK: puppet ran at Wed Jun 5 10:35:48 UTC 2013 [10:35:54] RECOVERY - Puppet freshness on stafford is OK: puppet ran at Wed Jun 5 10:35:49 UTC 2013 [10:36:00] RECOVERY - Puppet freshness on tmh2 is OK: puppet ran at Wed Jun 5 10:35:49 UTC 2013 [10:36:00] RECOVERY - Puppet freshness on search13 is OK: puppet ran at Wed Jun 5 10:35:50 UTC 2013 [10:36:00] RECOVERY - Puppet freshness on srv300 is OK: puppet ran at Wed Jun 5 10:35:51 UTC 2013 [10:36:00] RECOVERY - Puppet freshness on mw16 is OK: puppet ran at Wed Jun 5 10:35:54 UTC 2013 [10:36:00] RECOVERY - Puppet freshness on search1010 is OK: puppet ran at Wed Jun 5 10:35:55 UTC 2013 [10:36:01] RECOVERY - Puppet freshness on mw1176 is OK: puppet ran at Wed Jun 5 10:35:55 UTC 2013 [10:36:01] RECOVERY - Puppet freshness on mw1217 is OK: puppet ran at Wed Jun 5 10:35:57 UTC 2013 [10:36:19] RECOVERY - Puppet freshness on mw1164 is OK: puppet ran at Wed Jun 5 10:36:00 UTC 2013 [10:36:19] RECOVERY - Puppet freshness on mw1117 is OK: puppet ran at Wed Jun 5 10:36:00 UTC 2013 [10:36:19] RECOVERY - Puppet freshness on mw1103 is OK: puppet ran at Wed Jun 5 10:36:02 UTC 2013 [10:42:29] hi [10:58:28] New review: Faidon; "Nice work, see inline for a few comments." [operations/puppet/zookeeper] (master) - https://gerrit.wikimedia.org/r/66882 [12:07:45] RECOVERY - Puppet freshness on mw1100 is OK: puppet ran at Wed Jun 5 12:07:35 UTC 2013 [12:07:46] RECOVERY - Puppet freshness on mw1088 is OK: puppet ran at Wed Jun 5 12:07:36 UTC 2013 [12:07:46] RECOVERY - Puppet freshness on mw1018 is OK: puppet ran at Wed Jun 5 12:07:37 UTC 2013 [12:07:46] RECOVERY - Puppet freshness on sq83 is OK: puppet ran at Wed Jun 5 12:07:39 UTC 2013 [12:07:46] RECOVERY - Puppet freshness on kuo is OK: puppet ran at Wed Jun 5 12:07:39 UTC 2013 [12:07:46] RECOVERY - Puppet freshness on sq71 is OK: puppet ran at Wed Jun 5 12:07:40 UTC 2013 [12:07:46] RECOVERY - Puppet freshness on labsdb1003 is OK: puppet ran at Wed Jun 5 12:07:41 UTC 2013 [12:07:47] RECOVERY - Puppet freshness on db1054 is OK: puppet ran at Wed Jun 5 12:07:41 UTC 2013 [12:07:47] RECOVERY - Puppet freshness on neon is OK: puppet ran at Wed Jun 5 12:07:42 UTC 2013 [12:07:53] RECOVERY - Puppet freshness on es1 is OK: puppet ran at Wed Jun 5 12:07:42 UTC 2013 [12:07:53] RECOVERY - Puppet freshness on cp1007 is OK: puppet ran at Wed Jun 5 12:07:43 UTC 2013 [12:07:53] RECOVERY - Puppet freshness on amssq52 is OK: puppet ran at Wed Jun 5 12:07:48 UTC 2013 [12:07:53] RECOVERY - Puppet freshness on labsdb1002 is OK: puppet ran at Wed Jun 5 12:07:52 UTC 2013 [12:08:03] RECOVERY - Puppet freshness on cerium is OK: puppet ran at Wed Jun 5 12:07:53 UTC 2013 [12:08:03] RECOVERY - Puppet freshness on analytics1012 is OK: puppet ran at Wed Jun 5 12:07:55 UTC 2013 [12:08:03] RECOVERY - Puppet freshness on ms-be8 is OK: puppet ran at Wed Jun 5 12:07:56 UTC 2013 [12:08:03] RECOVERY - Puppet freshness on fenari is OK: puppet ran at Wed Jun 5 12:07:57 UTC 2013 [12:08:03] RECOVERY - Puppet freshness on mw73 is OK: puppet ran at Wed Jun 5 12:07:58 UTC 2013 [12:08:04] RECOVERY - Puppet freshness on mw122 is OK: puppet ran at Wed Jun 5 12:08:00 UTC 2013 [12:08:04] RECOVERY - Puppet freshness on cp3007 is OK: puppet ran at Wed Jun 5 12:08:01 UTC 2013 [12:08:13] RECOVERY - Puppet freshness on srv236 is OK: puppet ran at Wed Jun 5 12:08:02 UTC 2013 [12:08:13] RECOVERY - Puppet freshness on mw20 is OK: puppet ran at Wed Jun 5 12:08:03 UTC 2013 [12:08:13] RECOVERY - Puppet freshness on mw82 is OK: puppet ran at Wed Jun 5 12:08:03 UTC 2013 [12:08:13] RECOVERY - Puppet freshness on srv238 is OK: puppet ran at Wed Jun 5 12:08:05 UTC 2013 [12:08:13] RECOVERY - Puppet freshness on search17 is OK: puppet ran at Wed Jun 5 12:08:06 UTC 2013 [12:08:14] RECOVERY - Puppet freshness on mw66 is OK: puppet ran at Wed Jun 5 12:08:07 UTC 2013 [12:08:14] RECOVERY - Puppet freshness on mw1144 is OK: puppet ran at Wed Jun 5 12:08:07 UTC 2013 [12:08:15] RECOVERY - Puppet freshness on mw1015 is OK: puppet ran at Wed Jun 5 12:08:08 UTC 2013 [12:08:15] RECOVERY - Puppet freshness on mw1042 is OK: puppet ran at Wed Jun 5 12:08:09 UTC 2013 [12:08:16] RECOVERY - Puppet freshness on db40 is OK: puppet ran at Wed Jun 5 12:08:10 UTC 2013 [12:08:16] RECOVERY - Puppet freshness on mw34 is OK: puppet ran at Wed Jun 5 12:08:11 UTC 2013 [12:08:23] RECOVERY - Puppet freshness on srv297 is OK: puppet ran at Wed Jun 5 12:08:12 UTC 2013 [12:08:23] RECOVERY - Puppet freshness on mw46 is OK: puppet ran at Wed Jun 5 12:08:13 UTC 2013 [12:08:23] RECOVERY - Puppet freshness on search1008 is OK: puppet ran at Wed Jun 5 12:08:14 UTC 2013 [12:08:23] RECOVERY - Puppet freshness on mw1095 is OK: puppet ran at Wed Jun 5 12:08:15 UTC 2013 [12:08:23] RECOVERY - Puppet freshness on mw1101 is OK: puppet ran at Wed Jun 5 12:08:16 UTC 2013 [12:08:23] RECOVERY - Puppet freshness on amssq55 is OK: puppet ran at Wed Jun 5 12:08:16 UTC 2013 [12:08:24] RECOVERY - Puppet freshness on cp1012 is OK: puppet ran at Wed Jun 5 12:08:18 UTC 2013 [12:08:24] RECOVERY - Puppet freshness on cp1020 is OK: puppet ran at Wed Jun 5 12:08:19 UTC 2013 [12:08:25] RECOVERY - Puppet freshness on mw1065 is OK: puppet ran at Wed Jun 5 12:08:20 UTC 2013 [12:08:25] RECOVERY - Puppet freshness on mw1123 is OK: puppet ran at Wed Jun 5 12:08:20 UTC 2013 [12:08:26] RECOVERY - Puppet freshness on search1018 is OK: puppet ran at Wed Jun 5 12:08:21 UTC 2013 [12:08:26] RECOVERY - Puppet freshness on mw1052 is OK: puppet ran at Wed Jun 5 12:08:22 UTC 2013 [12:08:33] RECOVERY - Puppet freshness on db1023 is OK: puppet ran at Wed Jun 5 12:08:23 UTC 2013 [12:08:33] RECOVERY - Puppet freshness on pc2 is OK: puppet ran at Wed Jun 5 12:08:23 UTC 2013 [12:08:34] RECOVERY - Puppet freshness on es1005 is OK: puppet ran at Wed Jun 5 12:08:24 UTC 2013 [12:08:34] RECOVERY - Puppet freshness on williams is OK: puppet ran at Wed Jun 5 12:08:25 UTC 2013 [12:08:34] RECOVERY - Puppet freshness on lvs1 is OK: puppet ran at Wed Jun 5 12:08:26 UTC 2013 [12:08:34] RECOVERY - Puppet freshness on sq65 is OK: puppet ran at Wed Jun 5 12:08:27 UTC 2013 [12:08:34] RECOVERY - Puppet freshness on mw62 is OK: puppet ran at Wed Jun 5 12:08:28 UTC 2013 [12:08:35] RECOVERY - Puppet freshness on srv286 is OK: puppet ran at Wed Jun 5 12:08:29 UTC 2013 [12:08:35] RECOVERY - Puppet freshness on mw9 is OK: puppet ran at Wed Jun 5 12:08:30 UTC 2013 [12:08:36] RECOVERY - Puppet freshness on es1003 is OK: puppet ran at Wed Jun 5 12:08:31 UTC 2013 [12:08:43] RECOVERY - Puppet freshness on mw1099 is OK: puppet ran at Wed Jun 5 12:08:33 UTC 2013 [12:08:44] RECOVERY - Puppet freshness on mc3 is OK: puppet ran at Wed Jun 5 12:08:34 UTC 2013 [12:08:44] RECOVERY - Puppet freshness on search22 is OK: puppet ran at Wed Jun 5 12:08:37 UTC 2013 [12:08:44] RECOVERY - Puppet freshness on mw1214 is OK: puppet ran at Wed Jun 5 12:08:37 UTC 2013 [12:08:44] RECOVERY - Puppet freshness on srv258 is OK: puppet ran at Wed Jun 5 12:08:38 UTC 2013 [12:08:44] RECOVERY - Puppet freshness on mw14 is OK: puppet ran at Wed Jun 5 12:08:42 UTC 2013 [12:08:53] RECOVERY - Puppet freshness on mw47 is OK: puppet ran at Wed Jun 5 12:08:43 UTC 2013 [12:08:53] RECOVERY - Puppet freshness on mw49 is OK: puppet ran at Wed Jun 5 12:08:43 UTC 2013 [12:08:53] RECOVERY - Puppet freshness on mw1175 is OK: puppet ran at Wed Jun 5 12:08:47 UTC 2013 [12:08:53] RECOVERY - Puppet freshness on dobson is OK: puppet ran at Wed Jun 5 12:08:48 UTC 2013 [12:08:53] RECOVERY - Puppet freshness on mw1114 is OK: puppet ran at Wed Jun 5 12:08:51 UTC 2013 [12:08:53] RECOVERY - Puppet freshness on mw1126 is OK: puppet ran at Wed Jun 5 12:08:52 UTC 2013 [12:09:03] RECOVERY - Puppet freshness on cp1028 is OK: puppet ran at Wed Jun 5 12:08:53 UTC 2013 [12:09:03] RECOVERY - Puppet freshness on mw1127 is OK: puppet ran at Wed Jun 5 12:08:53 UTC 2013 [12:09:03] RECOVERY - Puppet freshness on sq73 is OK: puppet ran at Wed Jun 5 12:08:54 UTC 2013 [12:09:03] RECOVERY - Puppet freshness on sq45 is OK: puppet ran at Wed Jun 5 12:08:55 UTC 2013 [12:09:03] RECOVERY - Puppet freshness on gurvin is OK: puppet ran at Wed Jun 5 12:08:55 UTC 2013 [12:09:03] RECOVERY - Puppet freshness on sq72 is OK: puppet ran at Wed Jun 5 12:08:56 UTC 2013 [12:09:03] RECOVERY - Puppet freshness on db53 is OK: puppet ran at Wed Jun 5 12:08:57 UTC 2013 [12:09:04] RECOVERY - Puppet freshness on cp1013 is OK: puppet ran at Wed Jun 5 12:08:57 UTC 2013 [12:09:04] RECOVERY - Puppet freshness on cp1003 is OK: puppet ran at Wed Jun 5 12:08:58 UTC 2013 [12:09:05] RECOVERY - Puppet freshness on es4 is OK: puppet ran at Wed Jun 5 12:08:57 UTC 2013 [12:09:05] RECOVERY - Puppet freshness on es1002 is OK: puppet ran at Wed Jun 5 12:08:58 UTC 2013 [12:09:06] RECOVERY - Puppet freshness on caesium is OK: puppet ran at Wed Jun 5 12:08:59 UTC 2013 [12:09:06] RECOVERY - Puppet freshness on amssq57 is OK: puppet ran at Wed Jun 5 12:09:00 UTC 2013 [12:09:07] RECOVERY - Puppet freshness on srv250 is OK: puppet ran at Wed Jun 5 12:09:01 UTC 2013 [12:09:07] RECOVERY - Puppet freshness on sq86 is OK: puppet ran at Wed Jun 5 12:09:01 UTC 2013 [12:09:15] RECOVERY - Puppet freshness on analytics1010 is OK: puppet ran at Wed Jun 5 12:09:05 UTC 2013 [12:09:16] RECOVERY - Puppet freshness on tin is OK: puppet ran at Wed Jun 5 12:09:08 UTC 2013 [12:09:16] RECOVERY - Puppet freshness on pc1002 is OK: puppet ran at Wed Jun 5 12:09:11 UTC 2013 [12:09:16] RECOVERY - Puppet freshness on srv237 is OK: puppet ran at Wed Jun 5 12:09:12 UTC 2013 [12:09:16] RECOVERY - Puppet freshness on amssq34 is OK: puppet ran at Wed Jun 5 12:09:12 UTC 2013 [12:09:24] RECOVERY - Puppet freshness on mw125 is OK: puppet ran at Wed Jun 5 12:09:13 UTC 2013 [12:09:24] RECOVERY - Puppet freshness on db1052 is OK: puppet ran at Wed Jun 5 12:09:16 UTC 2013 [12:09:24] RECOVERY - Puppet freshness on mw1182 is OK: puppet ran at Wed Jun 5 12:09:17 UTC 2013 [12:09:24] RECOVERY - Puppet freshness on srv284 is OK: puppet ran at Wed Jun 5 12:09:18 UTC 2013 [12:09:24] RECOVERY - Puppet freshness on mw90 is OK: puppet ran at Wed Jun 5 12:09:18 UTC 2013 [12:09:24] RECOVERY - Puppet freshness on srv275 is OK: puppet ran at Wed Jun 5 12:09:20 UTC 2013 [12:09:25] RECOVERY - Puppet freshness on mw41 is OK: puppet ran at Wed Jun 5 12:09:21 UTC 2013 [12:09:33] RECOVERY - Puppet freshness on es10 is OK: puppet ran at Wed Jun 5 12:09:22 UTC 2013 [12:09:33] RECOVERY - Puppet freshness on mw1162 is OK: puppet ran at Wed Jun 5 12:09:24 UTC 2013 [12:09:33] RECOVERY - Puppet freshness on search14 is OK: puppet ran at Wed Jun 5 12:09:27 UTC 2013 [12:09:33] RECOVERY - Puppet freshness on mw81 is OK: puppet ran at Wed Jun 5 12:09:28 UTC 2013 [12:09:33] RECOVERY - Puppet freshness on ms5 is OK: puppet ran at Wed Jun 5 12:09:29 UTC 2013 [12:09:34] RECOVERY - Puppet freshness on zirconium is OK: puppet ran at Wed Jun 5 12:09:30 UTC 2013 [12:09:34] RECOVERY - Puppet freshness on search1014 is OK: puppet ran at Wed Jun 5 12:09:30 UTC 2013 [12:09:35] RECOVERY - Puppet freshness on lvs5 is OK: puppet ran at Wed Jun 5 12:09:31 UTC 2013 [12:09:43] RECOVERY - Puppet freshness on sq79 is OK: puppet ran at Wed Jun 5 12:09:32 UTC 2013 [12:09:43] RECOVERY - Puppet freshness on cp1006 is OK: puppet ran at Wed Jun 5 12:09:33 UTC 2013 [12:09:43] RECOVERY - Puppet freshness on db1048 is OK: puppet ran at Wed Jun 5 12:09:34 UTC 2013 [12:09:43] RECOVERY - Puppet freshness on search21 is OK: puppet ran at Wed Jun 5 12:09:35 UTC 2013 [12:09:43] RECOVERY - Puppet freshness on mw5 is OK: puppet ran at Wed Jun 5 12:09:36 UTC 2013 [12:09:44] RECOVERY - Puppet freshness on db1036 is OK: puppet ran at Wed Jun 5 12:09:36 UTC 2013 [12:09:44] RECOVERY - Puppet freshness on mw1044 is OK: puppet ran at Wed Jun 5 12:09:38 UTC 2013 [12:09:45] RECOVERY - Puppet freshness on mc1005 is OK: puppet ran at Wed Jun 5 12:09:38 UTC 2013 [12:09:45] RECOVERY - Puppet freshness on gadolinium is OK: puppet ran at Wed Jun 5 12:09:39 UTC 2013 [12:09:53] RECOVERY - Puppet freshness on mw83 is OK: puppet ran at Wed Jun 5 12:09:42 UTC 2013 [12:09:53] RECOVERY - Puppet freshness on search36 is OK: puppet ran at Wed Jun 5 12:09:43 UTC 2013 [12:09:53] RECOVERY - Puppet freshness on mw119 is OK: puppet ran at Wed Jun 5 12:09:44 UTC 2013 [12:09:53] RECOVERY - Puppet freshness on mw38 is OK: puppet ran at Wed Jun 5 12:09:44 UTC 2013 [12:09:53] RECOVERY - Puppet freshness on mw1130 is OK: puppet ran at Wed Jun 5 12:09:45 UTC 2013 [12:09:54] RECOVERY - Puppet freshness on analytics1015 is OK: puppet ran at Wed Jun 5 12:09:46 UTC 2013 [12:09:54] RECOVERY - Puppet freshness on ms-be4 is OK: puppet ran at Wed Jun 5 12:09:47 UTC 2013 [12:09:55] RECOVERY - Puppet freshness on mw1124 is OK: puppet ran at Wed Jun 5 12:09:48 UTC 2013 [12:09:55] RECOVERY - Puppet freshness on mw1125 is OK: puppet ran at Wed Jun 5 12:09:49 UTC 2013 [12:09:56] RECOVERY - Puppet freshness on vanadium is OK: puppet ran at Wed Jun 5 12:09:50 UTC 2013 [12:09:56] RECOVERY - Puppet freshness on sq51 is OK: puppet ran at Wed Jun 5 12:09:50 UTC 2013 [12:09:57] RECOVERY - Puppet freshness on search1019 is OK: puppet ran at Wed Jun 5 12:09:50 UTC 2013 [12:09:57] RECOVERY - Puppet freshness on db1019 is OK: puppet ran at Wed Jun 5 12:09:51 UTC 2013 [12:09:58] RECOVERY - Puppet freshness on ms-fe3 is OK: puppet ran at Wed Jun 5 12:09:51 UTC 2013 [12:09:58] RECOVERY - Puppet freshness on wtp1018 is OK: puppet ran at Wed Jun 5 12:09:52 UTC 2013 [12:10:03] RECOVERY - Puppet freshness on db43 is OK: puppet ran at Wed Jun 5 12:09:52 UTC 2013 [12:10:03] RECOVERY - Puppet freshness on srv301 is OK: puppet ran at Wed Jun 5 12:09:53 UTC 2013 [12:10:03] RECOVERY - Puppet freshness on db45 is OK: puppet ran at Wed Jun 5 12:09:53 UTC 2013 [12:10:03] PROBLEM - Puppet freshness on mw1102 is CRITICAL: No successful Puppet run in the last 10 hours [12:10:03] RECOVERY - Puppet freshness on amssq44 is OK: puppet ran at Wed Jun 5 12:09:58 UTC 2013 [12:10:04] RECOVERY - Puppet freshness on srv290 is OK: puppet ran at Wed Jun 5 12:10:00 UTC 2013 [12:10:13] RECOVERY - Puppet freshness on wtp1003 is OK: puppet ran at Wed Jun 5 12:10:02 UTC 2013 [12:10:13] RECOVERY - Puppet freshness on mw1111 is OK: puppet ran at Wed Jun 5 12:10:03 UTC 2013 [12:10:13] RECOVERY - Puppet freshness on mc1009 is OK: puppet ran at Wed Jun 5 12:10:05 UTC 2013 [12:10:13] RECOVERY - Puppet freshness on wtp1011 is OK: puppet ran at Wed Jun 5 12:10:07 UTC 2013 [12:10:13] RECOVERY - Puppet freshness on mw1190 is OK: puppet ran at Wed Jun 5 12:10:09 UTC 2013 [12:10:14] RECOVERY - Puppet freshness on mw1017 is OK: puppet ran at Wed Jun 5 12:10:12 UTC 2013 [12:10:23] RECOVERY - Puppet freshness on mw1147 is OK: puppet ran at Wed Jun 5 12:10:13 UTC 2013 [12:10:23] RECOVERY - Puppet freshness on analytics1023 is OK: puppet ran at Wed Jun 5 12:10:14 UTC 2013 [12:10:23] RECOVERY - Puppet freshness on mw1161 is OK: puppet ran at Wed Jun 5 12:10:16 UTC 2013 [12:10:23] RECOVERY - Puppet freshness on srv249 is OK: puppet ran at Wed Jun 5 12:10:17 UTC 2013 [12:10:23] RECOVERY - Puppet freshness on srv245 is OK: puppet ran at Wed Jun 5 12:10:18 UTC 2013 [12:10:23] RECOVERY - Puppet freshness on srv246 is OK: puppet ran at Wed Jun 5 12:10:19 UTC 2013 [12:10:24] RECOVERY - Puppet freshness on srv289 is OK: puppet ran at Wed Jun 5 12:10:20 UTC 2013 [12:10:24] RECOVERY - Puppet freshness on srv244 is OK: puppet ran at Wed Jun 5 12:10:20 UTC 2013 [12:10:25] RECOVERY - Puppet freshness on search19 is OK: puppet ran at Wed Jun 5 12:10:21 UTC 2013 [12:10:25] RECOVERY - Puppet freshness on mw30 is OK: puppet ran at Wed Jun 5 12:10:22 UTC 2013 [12:10:33] RECOVERY - Puppet freshness on mw1048 is OK: puppet ran at Wed Jun 5 12:10:23 UTC 2013 [12:10:33] RECOVERY - Puppet freshness on sq85 is OK: puppet ran at Wed Jun 5 12:10:24 UTC 2013 [12:10:33] RECOVERY - Puppet freshness on mw1146 is OK: puppet ran at Wed Jun 5 12:10:24 UTC 2013 [12:10:33] RECOVERY - Puppet freshness on sq48 is OK: puppet ran at Wed Jun 5 12:10:24 UTC 2013 [12:10:33] RECOVERY - Puppet freshness on sq43 is OK: puppet ran at Wed Jun 5 12:10:25 UTC 2013 [12:10:34] RECOVERY - Puppet freshness on mw1004 is OK: puppet ran at Wed Jun 5 12:10:26 UTC 2013 [12:10:34] RECOVERY - Puppet freshness on mw1038 is OK: puppet ran at Wed Jun 5 12:10:26 UTC 2013 [12:10:34] RECOVERY - Puppet freshness on mw1089 is OK: puppet ran at Wed Jun 5 12:10:26 UTC 2013 [12:10:35] RECOVERY - Puppet freshness on mw1081 is OK: puppet ran at Wed Jun 5 12:10:28 UTC 2013 [12:10:35] RECOVERY - Puppet freshness on mw76 is OK: puppet ran at Wed Jun 5 12:10:28 UTC 2013 [12:10:36] RECOVERY - Puppet freshness on db1004 is OK: puppet ran at Wed Jun 5 12:10:28 UTC 2013 [12:10:36] RECOVERY - Puppet freshness on solr3 is OK: puppet ran at Wed Jun 5 12:10:30 UTC 2013 [12:10:37] RECOVERY - Puppet freshness on cp3010 is OK: puppet ran at Wed Jun 5 12:10:31 UTC 2013 [12:10:43] RECOVERY - Puppet freshness on srv253 is OK: puppet ran at Wed Jun 5 12:10:35 UTC 2013 [12:10:43] RECOVERY - Puppet freshness on pc1003 is OK: puppet ran at Wed Jun 5 12:10:37 UTC 2013 [12:10:44] RECOVERY - Puppet freshness on es6 is OK: puppet ran at Wed Jun 5 12:10:38 UTC 2013 [12:10:44] RECOVERY - Puppet freshness on mw1056 is OK: puppet ran at Wed Jun 5 12:10:39 UTC 2013 [12:10:44] RECOVERY - Puppet freshness on mw1115 is OK: puppet ran at Wed Jun 5 12:10:40 UTC 2013 [12:10:44] RECOVERY - Puppet freshness on db58 is OK: puppet ran at Wed Jun 5 12:10:40 UTC 2013 [12:10:53] RECOVERY - Puppet freshness on solr2 is OK: puppet ran at Wed Jun 5 12:10:43 UTC 2013 [12:10:53] RECOVERY - Puppet freshness on ms-be1012 is OK: puppet ran at Wed Jun 5 12:10:43 UTC 2013 [12:10:53] RECOVERY - Puppet freshness on mw100 is OK: puppet ran at Wed Jun 5 12:10:44 UTC 2013 [12:10:53] RECOVERY - Puppet freshness on ms-be9 is OK: puppet ran at Wed Jun 5 12:10:45 UTC 2013 [12:10:53] RECOVERY - Puppet freshness on snapshot1 is OK: puppet ran at Wed Jun 5 12:10:48 UTC 2013 [12:10:53] RECOVERY - Puppet freshness on mw1159 is OK: puppet ran at Wed Jun 5 12:10:49 UTC 2013 [12:10:53] RECOVERY - Puppet freshness on db1020 is OK: puppet ran at Wed Jun 5 12:10:50 UTC 2013 [12:10:54] RECOVERY - Puppet freshness on mw1196 is OK: puppet ran at Wed Jun 5 12:10:51 UTC 2013 [12:11:03] RECOVERY - Puppet freshness on ms-be3 is OK: puppet ran at Wed Jun 5 12:10:52 UTC 2013 [12:11:04] RECOVERY - Puppet freshness on sq74 is OK: puppet ran at Wed Jun 5 12:10:53 UTC 2013 [12:11:04] RECOVERY - Puppet freshness on mw8 is OK: puppet ran at Wed Jun 5 12:10:54 UTC 2013 [12:11:04] RECOVERY - Puppet freshness on srv294 is OK: puppet ran at Wed Jun 5 12:10:55 UTC 2013 [12:11:04] RECOVERY - Puppet freshness on mw58 is OK: puppet ran at Wed Jun 5 12:10:58 UTC 2013 [12:11:04] RECOVERY - Puppet freshness on mw1188 is OK: puppet ran at Wed Jun 5 12:10:59 UTC 2013 [12:11:04] RECOVERY - Puppet freshness on search18 is OK: puppet ran at Wed Jun 5 12:10:59 UTC 2013 [12:11:05] RECOVERY - Puppet freshness on mw1197 is OK: puppet ran at Wed Jun 5 12:11:01 UTC 2013 [12:11:13] RECOVERY - Puppet freshness on mw1087 is OK: puppet ran at Wed Jun 5 12:11:04 UTC 2013 [12:11:14] RECOVERY - Puppet freshness on mw1106 is OK: puppet ran at Wed Jun 5 12:11:05 UTC 2013 [12:11:14] RECOVERY - Puppet freshness on solr1003 is OK: puppet ran at Wed Jun 5 12:11:05 UTC 2013 [12:11:14] RECOVERY - Puppet freshness on mw121 is OK: puppet ran at Wed Jun 5 12:11:07 UTC 2013 [12:11:14] RECOVERY - Puppet freshness on mw1140 is OK: puppet ran at Wed Jun 5 12:11:07 UTC 2013 [12:11:14] RECOVERY - Puppet freshness on mw1043 is OK: puppet ran at Wed Jun 5 12:11:08 UTC 2013 [12:11:15] RECOVERY - Puppet freshness on mw1057 is OK: puppet ran at Wed Jun 5 12:11:09 UTC 2013 [12:11:15] RECOVERY - Puppet freshness on manganese is OK: puppet ran at Wed Jun 5 12:11:10 UTC 2013 [12:11:16] RECOVERY - Puppet freshness on mw1041 is OK: puppet ran at Wed Jun 5 12:11:10 UTC 2013 [12:11:23] RECOVERY - Puppet freshness on mw1007 is OK: puppet ran at Wed Jun 5 12:11:14 UTC 2013 [12:11:23] RECOVERY - Puppet freshness on wtp1004 is OK: puppet ran at Wed Jun 5 12:11:15 UTC 2013 [12:11:23] RECOVERY - Puppet freshness on db1029 is OK: puppet ran at Wed Jun 5 12:11:16 UTC 2013 [12:11:23] RECOVERY - Puppet freshness on searchidx1001 is OK: puppet ran at Wed Jun 5 12:11:17 UTC 2013 [12:11:23] RECOVERY - Puppet freshness on srv276 is OK: puppet ran at Wed Jun 5 12:11:18 UTC 2013 [12:11:24] RECOVERY - Puppet freshness on ekrem is OK: puppet ran at Wed Jun 5 12:11:18 UTC 2013 [12:11:24] RECOVERY - Puppet freshness on amssq53 is OK: puppet ran at Wed Jun 5 12:11:19 UTC 2013 [12:11:25] RECOVERY - Puppet freshness on cp1036 is OK: puppet ran at Wed Jun 5 12:11:20 UTC 2013 [12:11:33] RECOVERY - Puppet freshness on stat1 is OK: puppet ran at Wed Jun 5 12:11:22 UTC 2013 [12:11:33] RECOVERY - Puppet freshness on mw1186 is OK: puppet ran at Wed Jun 5 12:11:22 UTC 2013 [12:11:33] RECOVERY - Puppet freshness on srv292 is OK: puppet ran at Wed Jun 5 12:11:24 UTC 2013 [12:11:33] RECOVERY - Puppet freshness on mw1210 is OK: puppet ran at Wed Jun 5 12:11:25 UTC 2013 [12:11:34] RECOVERY - Puppet freshness on sq76 is OK: puppet ran at Wed Jun 5 12:11:26 UTC 2013 [12:11:34] RECOVERY - Puppet freshness on search1024 is OK: puppet ran at Wed Jun 5 12:11:26 UTC 2013 [12:11:34] RECOVERY - Puppet freshness on wtp1024 is OK: puppet ran at Wed Jun 5 12:11:28 UTC 2013 [12:11:35] RECOVERY - Puppet freshness on mw1063 is OK: puppet ran at Wed Jun 5 12:11:30 UTC 2013 [12:11:43] RECOVERY - Puppet freshness on cp1002 is OK: puppet ran at Wed Jun 5 12:11:33 UTC 2013 [12:11:43] RECOVERY - Puppet freshness on ms10 is OK: puppet ran at Wed Jun 5 12:11:34 UTC 2013 [12:11:43] RECOVERY - Puppet freshness on mc7 is OK: puppet ran at Wed Jun 5 12:11:35 UTC 2013 [12:11:43] RECOVERY - Puppet freshness on cp1005 is OK: puppet ran at Wed Jun 5 12:11:37 UTC 2013 [12:11:44] RECOVERY - Puppet freshness on db34 is OK: puppet ran at Wed Jun 5 12:11:38 UTC 2013 [12:11:44] RECOVERY - Puppet freshness on praseodymium is OK: puppet ran at Wed Jun 5 12:11:39 UTC 2013 [12:11:44] RECOVERY - Puppet freshness on titanium is OK: puppet ran at Wed Jun 5 12:11:40 UTC 2013 [12:11:45] RECOVERY - Puppet freshness on potassium is OK: puppet ran at Wed Jun 5 12:11:41 UTC 2013 [12:11:53] RECOVERY - Puppet freshness on rdb1002 is OK: puppet ran at Wed Jun 5 12:11:43 UTC 2013 [12:11:53] RECOVERY - Puppet freshness on helium is OK: puppet ran at Wed Jun 5 12:11:44 UTC 2013 [12:11:53] RECOVERY - Puppet freshness on wtp1015 is OK: puppet ran at Wed Jun 5 12:11:45 UTC 2013 [12:11:53] RECOVERY - Puppet freshness on db1001 is OK: puppet ran at Wed Jun 5 12:11:46 UTC 2013 [12:11:53] RECOVERY - Puppet freshness on db1031 is OK: puppet ran at Wed Jun 5 12:11:49 UTC 2013 [12:11:54] RECOVERY - Puppet freshness on cp3009 is OK: puppet ran at Wed Jun 5 12:11:51 UTC 2013 [12:11:54] RECOVERY - Puppet freshness on pc1 is OK: puppet ran at Wed Jun 5 12:11:51 UTC 2013 [12:12:03] RECOVERY - Puppet freshness on srv273 is OK: puppet ran at Wed Jun 5 12:11:52 UTC 2013 [12:12:03] RECOVERY - Puppet freshness on srv255 is OK: puppet ran at Wed Jun 5 12:11:53 UTC 2013 [12:12:03] RECOVERY - Puppet freshness on mw124 is OK: puppet ran at Wed Jun 5 12:11:54 UTC 2013 [12:12:03] RECOVERY - Puppet freshness on mw43 is OK: puppet ran at Wed Jun 5 12:11:55 UTC 2013 [12:12:03] RECOVERY - Puppet freshness on ms-be1006 is OK: puppet ran at Wed Jun 5 12:11:56 UTC 2013 [12:12:04] RECOVERY - Puppet freshness on db39 is OK: puppet ran at Wed Jun 5 12:11:57 UTC 2013 [12:12:04] RECOVERY - Puppet freshness on analytics1014 is OK: puppet ran at Wed Jun 5 12:11:57 UTC 2013 [12:12:05] RECOVERY - Puppet freshness on mw57 is OK: puppet ran at Wed Jun 5 12:11:58 UTC 2013 [12:12:05] RECOVERY - Puppet freshness on mw1032 is OK: puppet ran at Wed Jun 5 12:11:59 UTC 2013 [12:12:06] RECOVERY - Puppet freshness on mw118 is OK: puppet ran at Wed Jun 5 12:12:00 UTC 2013 [12:12:06] RECOVERY - Puppet freshness on sq58 is OK: puppet ran at Wed Jun 5 12:12:01 UTC 2013 [12:12:07] RECOVERY - Puppet freshness on sq54 is OK: puppet ran at Wed Jun 5 12:12:02 UTC 2013 [12:12:13] RECOVERY - Puppet freshness on db1044 is OK: puppet ran at Wed Jun 5 12:12:03 UTC 2013 [12:12:14] RECOVERY - Puppet freshness on db1022 is OK: puppet ran at Wed Jun 5 12:12:03 UTC 2013 [12:12:14] RECOVERY - Puppet freshness on mc10 is OK: puppet ran at Wed Jun 5 12:12:04 UTC 2013 [12:12:14] RECOVERY - Puppet freshness on mc1 is OK: puppet ran at Wed Jun 5 12:12:05 UTC 2013 [12:12:14] RECOVERY - Puppet freshness on ms-be2 is OK: puppet ran at Wed Jun 5 12:12:06 UTC 2013 [12:12:14] RECOVERY - Puppet freshness on cp1010 is OK: puppet ran at Wed Jun 5 12:12:08 UTC 2013 [12:12:14] RECOVERY - Puppet freshness on ms-fe1001 is OK: puppet ran at Wed Jun 5 12:12:09 UTC 2013 [12:12:15] RECOVERY - Puppet freshness on wtp1007 is OK: puppet ran at Wed Jun 5 12:12:10 UTC 2013 [12:12:23] RECOVERY - Puppet freshness on cp1030 is OK: puppet ran at Wed Jun 5 12:12:13 UTC 2013 [12:12:23] RECOVERY - Puppet freshness on mw98 is OK: puppet ran at Wed Jun 5 12:12:14 UTC 2013 [12:12:24] RECOVERY - Puppet freshness on srv193 is OK: puppet ran at Wed Jun 5 12:12:15 UTC 2013 [12:12:24] RECOVERY - Puppet freshness on mw106 is OK: puppet ran at Wed Jun 5 12:12:16 UTC 2013 [12:12:24] RECOVERY - Puppet freshness on mw1205 is OK: puppet ran at Wed Jun 5 12:12:18 UTC 2013 [12:12:24] RECOVERY - Puppet freshness on db63 is OK: puppet ran at Wed Jun 5 12:12:19 UTC 2013 [12:12:24] RECOVERY - Puppet freshness on mw1189 is OK: puppet ran at Wed Jun 5 12:12:20 UTC 2013 [12:12:24] RECOVERY - Puppet freshness on mw79 is OK: puppet ran at Wed Jun 5 12:12:21 UTC 2013 [12:12:33] RECOVERY - Puppet freshness on mw1201 is OK: puppet ran at Wed Jun 5 12:12:23 UTC 2013 [12:12:33] RECOVERY - Puppet freshness on mw1046 is OK: puppet ran at Wed Jun 5 12:12:25 UTC 2013 [12:12:34] RECOVERY - Puppet freshness on mc1001 is OK: puppet ran at Wed Jun 5 12:12:29 UTC 2013 [12:12:34] RECOVERY - Puppet freshness on search33 is OK: puppet ran at Wed Jun 5 12:12:30 UTC 2013 [12:12:34] RECOVERY - Puppet freshness on mw1150 is OK: puppet ran at Wed Jun 5 12:12:31 UTC 2013 [12:12:34] RECOVERY - Puppet freshness on sq42 is OK: puppet ran at Wed Jun 5 12:12:32 UTC 2013 [12:12:44] RECOVERY - Puppet freshness on pdf3 is OK: puppet ran at Wed Jun 5 12:12:33 UTC 2013 [12:12:44] RECOVERY - Puppet freshness on amslvs1 is OK: puppet ran at Wed Jun 5 12:12:34 UTC 2013 [12:12:44] RECOVERY - Puppet freshness on mw1024 is OK: puppet ran at Wed Jun 5 12:12:35 UTC 2013 [12:12:44] RECOVERY - Puppet freshness on mw1068 is OK: puppet ran at Wed Jun 5 12:12:36 UTC 2013 [12:12:44] RECOVERY - Puppet freshness on calcium is OK: puppet ran at Wed Jun 5 12:12:37 UTC 2013 [12:12:44] RECOVERY - Puppet freshness on mc6 is OK: puppet ran at Wed Jun 5 12:12:38 UTC 2013 [12:13:04] RECOVERY - Puppet freshness on cp1024 is OK: puppet ran at Wed Jun 5 12:12:56 UTC 2013 [12:13:23] RECOVERY - Puppet freshness on mw35 is OK: puppet ran at Wed Jun 5 12:13:21 UTC 2013 [12:13:53] RECOVERY - Puppet freshness on mw1069 is OK: puppet ran at Wed Jun 5 12:13:46 UTC 2013 [12:16:13] RECOVERY - Puppet freshness on mw1083 is OK: puppet ran at Wed Jun 5 12:16:03 UTC 2013 [12:20:13] RECOVERY - Puppet freshness on mw1102 is OK: puppet ran at Wed Jun 5 12:20:12 UTC 2013 [12:22:05] RECOVERY - Puppet freshness on mw1109 is OK: puppet ran at Wed Jun 5 12:22:00 UTC 2013 [13:08:31] hyeaa akosiaris! [13:08:52] got a bit to look at the kafka .deb with me? [13:09:39] hey ottomata [13:09:41] yeah sure [13:09:46] i was waiting for ya [13:10:04] :) [13:10:08] so yeah, first question [13:10:09] how do you build [13:10:10] just [13:10:11] debuild? [13:10:39] git-buildpackage -uc -us [13:11:10] i am almost done btw with the init script [13:11:18] i got a question though [13:11:21] -uc -us ? [13:11:38] all the other bin/* files... do we need the functionality they offer ? [13:11:56] many of them we don't, but some of them are really handy for debugging and testings setups [13:11:57] it tells it not to sign the .deb and .changes [13:12:01] ah ok [13:12:24] don't debian-branch and upstream-branch need to be set? [13:12:31] your branches aren't the defaults [13:12:45] xmm i haven't commited the .gbp.conf ... [13:12:51] ok gimme 5 secs ... [13:12:54] ahh ok, i started my own, but yours is probably better [13:12:54] danke [13:13:35] also, do you need a debian/source/format file? [13:13:42] i got some complaints about that at some point [13:13:47] setting it to quilt (3.0) [13:14:07] no not really at this point. probably later though [13:14:14] hm, ok [13:14:30] for now the changes to the code are only my Makefiles so not quilt needed yet [13:14:37] why? (i'm not entirely sure what that does…just tells deb build tools how to build the source package or something?) [13:14:49] oh quilt is the patch thing [13:14:50] right [13:14:56] no. it keeps the patches we apply to upstream [13:15:00] exactly [13:15:03] aye right ok [13:15:03] cool [13:15:15] hm, i think whatever I was doing was complaining about modified binary files in ext/ and elsewhere [13:15:18] dunno why, but it said to set it [13:15:22] (setting it didn't really help though) [13:15:33] but, i'll wait for your gbp.conf and try again [13:17:51] ready. [13:18:16] I just installed in a labs machine the latest version and figuring out some errors ... [13:20:05] oo nice work with kafka.defaults CLASSAPTH [13:20:21] aawwww, no more upstart? [13:21:07] axaxa we can do upstart... [13:21:19] but i started the debian way to get it working [13:21:30] and we can go the ubuntu way later on [13:21:44] naw its fine, i dont' really care [13:22:04] i just kinda like upstart better than init scripts (from uninformed point of view), but whatever works [13:22:25] probably there are more complicated things you have to do now since the dep .jars aren't all in the /usr/share/kafka dir [13:23:54] ok so, since you've got CLASSPATH all set in /etc/kafka/default [13:24:06] should kafka.sh just source that? [13:24:06] sorry [13:24:15] /etc/default/kafka [13:24:31] yeah that be correct [13:24:38] k cool [13:24:39] oo [13:24:49] shoudl we put .gbp.conf in debian/gpb.conf? [13:25:00] i think its better there because then it isn't a modification to the source dir [13:25:07] i know its only in the debian branch, but still [13:25:55] that won't work [13:26:09] you will get the standard error Use --git-ignore-branch to ignore or --git-debian-branch to set the branch name [13:26:25] CONFIGURATION FILES [13:26:26] Four configuration files are parsed to set defaults for the above commandline arguments: [13:26:26] /etc/git-buildpackage/gbp.conf [13:26:26] system wide configuraton [13:26:26] ~/.gbp.conf [13:26:27] per user configuration [13:26:28] .gbp.conf [13:26:28] per branch configuration, can be published with the repository [13:26:29] debian/gbp.conf [13:26:30] per branch configuration, can be published with the repository [13:26:31] .git/gbp.conf [13:26:31] per repository configuration [13:26:48] i usually put it there: debian/gbp.conf [13:26:57] (also faidon told me to put it there once :p ) [13:28:04] brb [13:28:33] oh my bad... without the dot ... yeah sure... [13:31:28] (back) [13:36:25] hm akosiaris [13:36:34] do we need [13:36:35] upstream-tree=branch [13:36:35] ? [13:36:39] otherwise I get [13:56:49] so sbt/maven/ivy/gradle whatever all do want to download all of their dependencies and the _normal_ way to deploy stuff built with them is to package all their dependencies together [13:56:51] in the artifact [13:57:07] but yeah, manybubbles_1, our biggest problem is that none of us here (i guess until now) have much java packaging and deployment experience [13:57:18] so we are struggling to make somethign work [13:57:27] I can totally help with that [13:57:43] don't we have our own repository? [13:57:50] diedrik set it up someplace, I was trying to get our Android stuff into it [13:57:52] yes [13:57:52] nexus.wmflabs.org [13:57:53] but its in labs [13:57:54] and isn't vetted [13:57:55] manybubbles_1: during the weekly meeting I told you i can help. unfortunately I am having a four days break starting tomorrow. So I am afraid that postpone any input from me to monday :( [13:58:00] hmm [13:58:08] hashar: well, have a good break at least. sorry I didn't get on early yesterday. I think I'm learning some of what I need to know from ^demon and just playing with labs. [13:58:14] i think this is a organizational issue we need to figure out, as more and more people seem to be interested in deploying java stuff [13:58:16] so what exactly is the problem [13:58:22] we do know how to build java packages and how to deploy them, but there is a strong preference to use the debian methodology for deploying apps at WMF [13:58:25] ottomata: so, yeah, if we're going to deploy java stuff we should probably figure out the right way to do it [13:58:31] PROBLEM - Host wtp1008 is DOWN: PING CRITICAL - Packet loss = 100% [13:58:37] i think we already closed that discussion for Kafka, but feel free to reopen it for Solr :) [13:58:58] RECOVERY - Host wtp1008 is UP: PING OK - Packet loss = 0%, RTA = 0.53 ms [13:58:58] we'll need it opened for storm too [13:58:59] <^d> Same discussion is relevant for Gerrit as well. [13:58:59] drdee: so when I've deployed java stuffin the past we'd typically build that java artifact as normal in java and then wrap it in an appropriate package (RPM in my case) with dependencies just on the openjdk [13:59:06] manybubbles_1: we've been told that we need to freeze the debs at buildtime [13:59:07] <^demon> So we've got kafka, storm, solr and gerrit, all non-trivial java applications that need a sane way to build. [13:59:08] so [13:59:09] manybubbles_1: well there is a ton of stuff to learn. We can spend a couple hours monday together if you want and see how it went. If I can't answer myself, I can almost guarantee you I know who you can speak with. Have an happy java dependencies handling. [13:59:09] someone can clone the deb repository and build without having an internet connection [13:59:13] hashar: thanks! [13:59:21] ottomata: java doesn't work like that. no, really, it doesn't. [13:59:23] yes i agree ^demon, but i rather not reopen this discussion again now we are finally making progress [13:59:24] ottomata: well, it hasn't since maven,ivy,etc [13:59:42] it is actually pretty sane even though it still wants to download stuff [13:59:45] manybubbles_1 you might want to talk to Ops, we have made similar arguments as you are making right now [14:00:01] <^demon> drdee: Doing it as debian packages is basically a non-starter for Gerrit right now. GWT has never been debianized and that would be a huge timesink for us to waste time on. [14:00:29] <^demon> drdee: Have we ever considered the possibility that ops is wrong here? :)) [14:00:35] drdee: sometimes I get confused who I'm talking to. [14:00:41] " it is actually pretty sane even though it still wants to download stuff". This guy's pretty nice except that he still wants to strangle kittens. :-) [14:00:47] drdee, ottomata: team analytics [14:00:55] ^demon, hashar: team platform [14:01:06] <^demon> manybubbles_1 is on our team :) [14:01:06] none of us are ops [14:01:26] I know ^demon and hashar already! I've even talked to them on a horribly broken up phone call! It was great! [14:01:35] i'm ops! [14:01:37] + analytics [14:01:38] hehe [14:01:54] drdee, we can move forward with kafka [14:01:58] as is [14:02:03] and still have this discussion [14:02:04] ^demon: yes that thought has crossed my mind but again i am happy to make progress with kafka right now, so i think we need to discuss this on a project by project basis [14:02:15] <^demon> I think that's a mistake tbh. [14:02:19] they aren't exclusive [14:02:24] if we get a working kafka .deb right now [14:02:25] then we can use it [14:02:36] I'm ops-like. :-) [14:02:37] if later we get a vetted java build/deploy system [14:02:41] then we can use it [14:02:43] <^demon> We've got a number of non-trivial java projects, and if we keep treating them as one off rather than dealing with the bigger problem we'll just end up with a bunch of one-off solutions that makes nobody happy. [14:03:02] i've been working on this kafka .deb since february [14:03:04] don't' preach to the converted :D [14:03:06] heheh [14:03:17] i think we have suffered the most of this so far [14:03:43] ja ^demon, i wanted to talk to you about this at the ams hackathon, but then realized you weren't there! [14:03:56] i talked to ryan_lane about this, and it sounded like your gerrit woes weren't quite of the same magnitude as the kafka ones [14:04:01] but ja there is a shared problem for sure [14:04:08] so would it make sense to call some kind of meeting with actual good phone connections and stuff? [14:04:15] <^demon> ottomata: Wish I could've made it this year. [14:04:23] <^demon> ottomata: It also doesn't help that upstream gerrit has decided they don't like maven anymore and are going with $coolNewToy, but that's my own cross to bear ;-) [14:04:26] like with ops and stuff [14:04:30] manybubbles_1: yeah totally [14:04:50] yup [14:05:03] i couldn't do it alone though, i just don't know enough about either way of doing things [14:05:04] drdee: I can serve as an expert witness but I don't know who to invite. [14:05:32] i will create a calendar invite [14:05:43] and add the appropriate folks [14:05:46] can we have a bit of an email thread about htis first? [14:05:59] i think ops people would prefer to get an overview of this via email discussion rather than live meeting [14:06:09] that way we can all do research and understand the issues before meeting [14:06:29] i think the purpose of the meeting is just to surface issues, without looking at solutions [14:06:35] hmm ok [14:06:43] ok ok [14:06:59] New patchset: Odder; "(bug 49176) Increase account creation limit for itwiki GLAM event" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/67087 [14:07:53] I can put together a strawman proposal which totally downloads stuff from the internet and we can pick it appart and see where we should go from there. [14:08:19] <^demon> ottomata: Gerrit's dropped maven for buck (https://github.com/facebook/buck). It still downloads the jars (but can work offline). My main PITA with buck right now is no real packaging (but that's probably easy). [14:08:47] <^demon> A buck package would make my life easier. [14:10:18] hm ,yeah [14:10:30] is buck able to use the internet's existing maven repos? [14:10:54] if so, i think this would be the same problem as we're ahving with sbt [14:11:08] and would be solveable if we had a vetted secure local maven repo of some kind (nexus, artifactory, whatever) [14:11:11] I imagine a java build tool that didn't use maven central would be a pretty hard sell [14:11:15] analagous to how we use apt.wm.org [14:11:18] yeah [14:11:35] I'm a little confused about why we don't think maven central isn't secure. [14:12:05] for kafka, its not just maven central [14:12:18] i think it might be ok, no tsure [14:12:31] i think thus far we just don't know about java world, so nobody knows? [14:12:32] i dunno [14:12:34] most folks want a local reposity for speed reasons and so they can force things to stay in existence forever and so they deploy internal dependencies [14:13:07] yeah, manybubbles_1, i are you on the ops@lists.wikimedia.org mailing list? [14:13:25] kafka probably uses the scala repositories. for some reason the scala folks didn't shove all their things into maven central. When you setup your own repository it tends to proxy both. [14:13:47] ja, it also uses some stupid github hosted things too [14:13:57] sbt-assembly? [14:13:58] something [14:14:13] ottomata: I don't think so. I don't know how to sign myself up for it but I'd like to. [14:14:41] github hosted things: silly silly. I wouldn't want to depend on that. [14:14:49] <^demon> ottomata: Could you look at a puppet change for me? It's pretty simple. [14:14:52] sure [14:15:03] manybubbles_1: uhh, dunno how you sign up, ask robla maybe? :) [14:15:11] but, i will forward you the email thread about the kafka .deb [14:15:14] the github dependency was metrics from coda hale (IIRC) [14:15:22] manybubbles_1: what's your email? [14:15:25] github is great but isn't a release repository [14:15:33] ottomata: neverett@wikimedia.org [14:15:36] <^demon> Wow, crazy lag from freenode. [14:15:43] <^demon> Catching up... [14:16:08] ottomata: I think I should make a manybubbles@wikimedia.org forwarding address at this rate. [14:16:26] <^demon> ottomata: Yes, it should be able to use an arbitrary maven destination (it uses central + some google repos) at the moment. [14:19:47] <^demon> REPO_ROOTS = { [14:19:47] <^demon> 'GERRIT': 'http://gerrit-maven.commondatastorage.googleapis.com', [14:19:47] <^demon> 'MAVEN_CENTRAL': 'http://repo1.maven.org/maven2', [14:19:49] <^demon> } [14:19:51] <^demon> ottomata: ^ [14:20:04] aye cool [14:20:12] yeah that's the type of thing we'll have to solve [14:20:21] we might be able to allow maven central [14:20:29] but probably not google repos [14:20:34] <^demon> Fair enough [14:20:38] so we'll have to do some kind of local repo, like we do for apt.wm.org [14:20:48] <^demon> Yeah, that sounds reasonable. [14:20:52] <^demon> And seems like it'd work for us [14:21:24] if you guys want to setup another repository, make sure to get ops team to agree on it. [14:21:27] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:21:33] <^demon> buck also does checksumming, à la maven, so that argument can be made for "making sure we're downloading what we think we are." [14:21:52] last year someone wanted to setup a maven repo and he got frowned upon because we already had apt [14:22:12] <^demon> Yes, but considering we're doing more java stuff now it's not an unreasonable argument to make. [14:22:34] yup that makes perfect sense nowadays [14:22:56] just warning you that some people could still think otherwise. So they need to be informed via ops list :-D [14:23:02] I am confident the output will be positive. [14:23:17] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.129 second response time [14:23:20] <^demon> Indeed, and that's part of the reason we're talking it now...both analytics and platform want to have the discussion now :) [14:23:24] (then we can host a pip repo haha) [14:23:32] <^demon> pip sucks lol. [14:23:47] or easy_install [14:23:50] <^demon> Plus it's totally not necessary for git-review...gr is a single file and you can drop it anywhere in $PATH [14:28:40] ^demon: you had a puppet chagne you wanted me to look at? [14:29:01] ^d you had a puppet chagne you wanted me to look at [14:29:12] ? [14:29:16] <^d> Ah, yes. https://gerrit.wikimedia.org/r/#/c/61036/ [14:29:47] <^d> The gitblit.properties file is long but it's mostly just copy+pasted from upstream. [14:29:58] <^d> Freenode is killing me today. [14:30:25] Change merged: Ottomata; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/61036 [14:30:35] ^d ^ [14:30:53] <^demon> Sweet, can you merge on sockpuppet? I can run puppet on antimony [14:31:46] done [14:31:52] <^demon> Thanks! [14:32:27] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:33:36] ottomata: I just finished reading the mail. I'm glad folks seem to be coming around tothe twitter guy. His words are sensible. [14:34:17] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.127 second response time [14:34:19] yeah totally. ike I said. i think this is historically a problem of not having the experience on the team [14:34:25] so we can probably reach a sensible conclusion [14:36:38] the internet is exploding [14:37:46] New patchset: Demon; "source not content" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/67091 [14:39:45] heading home from cafe, be back in a bit [14:41:05] New review: Hashar; "files/ldap/scripts/ldaplist output has been slightly altered. An output string contained a leading t..." [operations/puppet] (production) C: -1; - https://gerrit.wikimedia.org/r/67008 [14:44:37] New review: Hashar; "(1 comment)" [operations/puppet] (production) C: -1; - https://gerrit.wikimedia.org/r/67021 [14:47:25] New patchset: Andrew Bogott; "Pep8 cleanups:" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/67008 [14:49:43] New patchset: Andrew Bogott; "Pep8 cleanups; mostly whitespace." [operations/puppet] (production) - https://gerrit.wikimedia.org/r/67021 [14:52:26] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:53:17] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.128 second response time [14:56:08] New review: Hashar; "(1 comment)" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/67022 [14:57:51] New review: Hashar; "nice ! :)" [operations/puppet] (production) C: 1; - https://gerrit.wikimedia.org/r/67008 [14:59:07] New patchset: Andrew Bogott; "Pep8 cleanup stage one: tab purge!" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/67022 [14:59:33] New review: Hashar; "(1 comment)" [operations/puppet] (production) C: 1; - https://gerrit.wikimedia.org/r/67021 [15:00:26] andrewbogott: now submit an octopus merge commit that bring in the 4 changes all together (I am evil). [15:00:51] Really? Isn't sets of smaller patches better? [15:00:57] Esp if one breaks something :) [15:02:35] New patchset: Andrew Bogott; "Pep8 cleanups; mostly whitespace." [operations/puppet] (production) - https://gerrit.wikimedia.org/r/67021 [15:03:14] andrewbogott: I was kidding :D [15:03:37] hashar: Yeah, I figured that out right after I said that [15:03:47] Thanks for reading all these tedious patches. [15:04:01] all goods for me :) [15:08:00] New review: Demon; "noob mistake :(" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/67091 [15:08:58] New patchset: Andrew Bogott; "Suppress a jillion pep8 warnings for these files." [operations/puppet] (production) - https://gerrit.wikimedia.org/r/67030 [15:09:51] New review: Hashar; ":-)" [operations/puppet] (production) C: 1; - https://gerrit.wikimedia.org/r/67091 [15:11:50] andrewbogott: pep8 let you exclude files :-D [15:12:27] hashar, I looked for docs on that but couldn't find anything. What's the syntax? [15:12:55] exclude=* [15:12:56] :-D [15:13:06] New review: Hashar; "(1 comment)" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/67030 [15:13:15] andrewbogott let me check on gallium though [15:13:57] that works on gallium (pep8 v1.3.x [15:14:03] so definitely exclude=* [15:14:04] works [15:15:07] New patchset: Andrew Bogott; "Exclude these files from pep8 tests." [operations/puppet] (production) - https://gerrit.wikimedia.org/r/67030 [15:16:41] <^demon> andrewbogott: While you're on a roll, could you look at https://gerrit.wikimedia.org/r/#/c/67091/? I made a dumb mistake. [15:17:44] New review: Hashar; "confirmed to work on jenkins server." [operations/puppet] (production) C: 1; - https://gerrit.wikimedia.org/r/67030 [15:17:53] andrewbogott: all good to me [15:17:56] Change merged: Andrew Bogott; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/67091 [15:23:28] off to get my daughter back home. [15:23:40] will not show up on irc tonight but will handle my mail backlog instead [15:23:46] *wave* [15:29:26] morebots, everything good? [15:29:28] I am a logbot running on wikitech-static. [15:29:28] Messages are logged to wikitech.wikimedia.org/wiki/Server_Admin_Log. [15:29:28] To log a message, type !log . [15:30:22] !log upgraded adminbot on wikitech-static to 1.7.3. Should be more likely to survive netsplits. [15:30:57] well, that's discouraging [15:33:07] Logged the message, Master [15:33:20] :D [15:33:24] morebots, what the heck? That took you like five minutes to even notice. [15:34:02] I am a logbot running on wikitech-static. [15:34:03] Messages are logged to wikitech.wikimedia.org/wiki/Server_Admin_Log. [15:34:03] To log a message, type !log . [15:34:21] Ryan_Lane, was the bot like that before? Super slow? [15:34:29] shouldn't be [15:34:31] morebots: ... [15:34:31] I am a logbot running on wikitech-static. [15:34:31] Messages are logged to wikitech.wikimedia.org/wiki/Server_Admin_Log. [15:34:31] To log a message, type !log . [15:34:35] morebots: ... [15:34:35] I am a logbot running on wikitech-static. [15:34:36] Messages are logged to wikitech.wikimedia.org/wiki/Server_Admin_Log. [15:34:36] To log a message, type !log . [15:34:39] it's fast now [15:34:44] maybe it just needed some warm up time :) [17:19:07] New patchset: Demon; "Updating gitblit to 1.3.0-SNAPSHOT" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/67110 [17:19:12] New patchset: Andrew Bogott; "pep8 fixes for generate-ganglia-conf.py" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/67115 [17:19:24] New review: Demon; "This just adds the new defaults for 1.3.0, no actual changes in our customizations." [operations/puppet] (production) - https://gerrit.wikimedia.org/r/67110 [17:22:14] mark or paravoid, https://gerrit.wikimedia.org/r/66874 is the performance improvement of device detection we discussed - can you review it please? [17:23:54] New patchset: Andrew Bogott; "Suppress long-to-line warnings for noc files." [operations/puppet] (production) - https://gerrit.wikimedia.org/r/67116 [17:24:53] New patchset: Andrew Bogott; "Suppress long-too-line warnings for noc files." [operations/puppet] (production) - https://gerrit.wikimedia.org/r/67116 [17:25:48] Change merged: Andrew Bogott; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/67116 [17:29:44] New patchset: Andrew Bogott; "Pep8 cleanup for drac.py" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/67117 [17:39:55] New patchset: Krinkle; "wgRC2UDPPrefix: Use hostname-".org" instead of lang.site" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/47307 [17:46:09] New patchset: Jdlrobson; "Enable CORS on beta labs domains" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/67119 [17:48:14] New review: Hashar; "easy one :-)" [operations/puppet] (production) C: 1; - https://gerrit.wikimedia.org/r/67115 [17:48:16] Change merged: jenkins-bot; [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/67119 [17:48:43] New patchset: Demon; "Utility manifest for building hiphop" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/67120 [17:49:11] Change merged: Andrew Bogott; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/67114 [17:49:37] Change merged: Andrew Bogott; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/67115 [17:49:58] Change merged: Andrew Bogott; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/67117 [17:54:27] New patchset: Dzahn; "add virtual language subdomain redirects for wikidata" [operations/apache-config] (master) - https://gerrit.wikimedia.org/r/65443 [18:02:41] New patchset: Krinkle; "wgRC2UDPPrefix: Use hostname-".org" instead of lang.site" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/47307 [18:02:49] New review: Krinkle; "Fix space." [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/47307 [18:05:01] New patchset: Aude; "add wikidata redirector script entry point (DO NOT MERGE)" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/67123 [18:05:47] New review: Aude; "(1 comment)" [operations/mediawiki-config] (master) C: -1; - https://gerrit.wikimedia.org/r/67123 [18:06:30] New patchset: Andrew Bogott; "Pep8 cleanup for modules/deployment" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/67124 [18:09:23] New patchset: Aude; "add wikidata redirector script entry point (DO NOT MERGE)" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/67123 [18:09:48] New review: Aude; "work in progress" [operations/mediawiki-config] (master) C: -1; - https://gerrit.wikimedia.org/r/67123 [18:14:25] New patchset: Andrew Bogott; "Pep8 cleanup in files/misc" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/67126 [18:17:35] PROBLEM - HTTP radosgw on ms-fe1003 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 758 bytes in 0.001 second response time [18:19:59] New patchset: Andrew Bogott; "Looks like jenkins' pep8 tool doesn't observe #noqa" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/67128 [18:21:55] New patchset: Andrew Bogott; "One tiny pep8 fix" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/67129 [18:23:11] Change merged: Andrew Bogott; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/67129 [18:23:46] Change merged: Andrew Bogott; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/67128 [18:24:56] !log reedy rebuilt wikiversions.cdb and synchronized wikiversions files: Everything over to 1.22wmf5 [18:25:04] Logged the message, Master [18:26:35] RECOVERY - HTTP radosgw on ms-fe1003 is OK: HTTP OK: HTTP/1.1 200 OK - 311 bytes in 0.003 second response time [18:32:01] New patchset: Reedy; "Everything to 1.22wmf5" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/67131 [18:34:21] Change merged: Reedy; [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/67131 [18:39:53] New patchset: Andrew Bogott; "pep8 cleanup for icinga files" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/67134 [18:41:21] Gerrit down? [18:42:00] Hm.. no just git.wikimedia.org [18:42:04] HTTP 503 [18:43:45] New review: Dzahn; "(1 comment)" [operations/apache-config] (master) - https://gerrit.wikimedia.org/r/65463 [18:46:34] hashar: hey hey !! [18:46:48] hashar: Antoine can you help me with something please ? [18:47:14] average: hi :D [18:47:56] is it possible to get a newer version of git-buildpackage on jenkins ? [18:48:06] uhm, but that's not critical [18:48:16] wait, nevermind (will come back with details later) [18:48:38] average: we use what ever comes with Ubuntu Precise :D [18:49:02] have you managed to build a package ? The only way I found required root at some point :( [18:49:34] hashar: I was thinking of setting up a job on jenkins to build a package [18:49:43] hashar: but it's not critical [18:49:45] so that might work [18:50:08] but whenever the package has an unsatisfied build dependency it will abort [18:50:27] in some case you could install the build dependencies automatically but that needs root access [18:50:32] maybe we can do that in a vagrant instance. [18:52:33] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:53:51] ori-l: any idea where we can fetch a vagrant image for precise? ;D [18:54:46] we should build our own maybe [18:55:24] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.129 second response time [19:28:50] <^demon> ottomata: So, is there any way for me to install upstream zookeeper yet? I'm kinda stuck :\ [19:30:54] how do i sync a new docroot? fenari tells me to use tin instead and tin doesn't have /h/w/common/ [19:32:25] install? install shoudl be fine, right? or do you mean via puppet? [19:32:51] ok, it's /a/common, thanks reedy [19:33:16] <^demon> ottomata: apt-get install zookeeperd gave me zookeeperd : Depends: zookeeper (= 3.3.5+dfsg1-1ubuntu1) but 3.4.5+16-1.cdh4.2.1.p0.8~precise-cdh4.2.1 is to be installed [19:33:27] ahh yeah [19:33:29] do um [19:34:03] sudo apt-get install zookeeper=3.3.5+dfsg1-1ubuntu1 [19:34:05] and for server [19:34:11] apt-get install zookeeperd=3.3.5+dfsg1-1ubuntu1 [19:34:51] hashar: the vagrant image that mediawiki-vagrant uses runs precise [19:35:18] specifically: https://cloud-images.ubuntu.com/vagrant/precise/current/precise-server-cloudimg-amd64-vagrant-disk1.box [19:35:45] ori-l: I guess I will have to finally fetch that image on gallium and create a job using it :-D [19:36:05] ori-l: average was asking earlier about to build debian packages, and that need root access to install dependencies :D [19:36:36] so many things to do [19:38:19] New patchset: Dzahn; "add document root dir with placeholder file for test.wikidata" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/67139 [19:38:50] using vagrant to build packages is a great idea, could be easily scriptable in jenkins too [19:38:56] that way it can install whatever into the vm [19:38:59] hashar: the current mediawiki-vagrant default puppet role configures a machine for developer productivity, which includes some things you wouldn't want/need on a test instance. i think a good step forward would be to start making a list of what things should go into a test box. [19:39:00] and not affect the actual build server [19:39:14] i've been using a personal 'build' vagrant vm [19:39:18] we could make it more general [19:39:26] puppetize all the usual things that are needed for building packages [19:39:54] ^demon, let me know if that worked [19:40:02] <^demon> It did, thanks much. [19:40:02] ottomata: but isn't there something equivalent that is part of the debian packaging ecosystem? i vaguely recall using something like that [19:40:49] ah yes, https://wikitech.wikimedia.org/wiki/Pbuilder [19:41:33] New patchset: Dzahn; "add VirtualHost for test.wikidata" [operations/apache-config] (master) - https://gerrit.wikimedia.org/r/67140 [19:42:31] Reedy: csteipp https://gerrit.wikimedia.org/r/#/c/67140/1/main.conf + https://gerrit.wikimedia.org/r/#/c/67139/1 and done .. ack? [19:42:35] worksforme [19:43:06] it's a simple copy and all separated that way [19:43:14] aude: ^ [19:43:29] Shouldn't you have copied the skeleton docroot or something? ;) [19:43:50] oh, you're right! yes [19:43:54] i'll fix it [19:44:05] I was slightly confused why you had an empty folder :p [19:44:18] first i wasn't even sure if you want to intsall a wiki in it [19:44:40] and i didnt want to add the Apache config and get warnings for non-existing docroot [19:44:58] ottomata: will look at it eventually :-) thx [19:45:27] ottomata: though there's still the cost of setting up all the pbuilder-related stuff so maybe that makes sense [19:45:54] pbuilder-inside-vagrant: vagrant env configured with all the debian packaging tools, pbuilder env barebone as it should be [19:46:08] mutante: The wikidata tell me it's still MediaWiki in there somewhere... [19:46:17] s/wikidata/wikidata team/ [19:46:36] ori-l: we have puppet classes for pbuilder [19:46:37] New patchset: Dzahn; "add document root dir with placeholder file for test.wikidata, from skeletion" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/67139 [19:47:12] Ryan_Lane: oh right, misc::package-builder [19:47:15] they likely need to be turned into modules, though [19:47:32] it sets up everything necessary for a sane package building environment [19:48:31] Change merged: Dzahn; [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/67139 [19:48:38] ottomata: if you share notes about what sort of stuff you've configured on your package-building vm, i'll try to take that and combine it with the misc::package-builder class and build a portable puppet module [19:48:45] so i pull on fenari and on tin , just in case ... [19:49:01] wha? [19:50:19] mutante: the apache looks good to me [19:50:31] aude: document root for wikidata_test, copy of regular wikidata [19:50:35] good [19:50:55] copy of the wikidata config? [19:51:33] as long as we don't enable any of the wikidata cronjobs yet for the test wikidata, then okay [19:51:47] yes, copy of the Apache VirtualHost [19:51:50] PROBLEM - Host wtp1008 is DOWN: PING CRITICAL - Packet loss = 100% [19:51:53] oh, that is fine [19:51:54] so first there is ServerName test.wikidata.org [19:51:59] yes [19:52:02] and after that the *.wikidata.org ServerName [19:52:09] ok [19:52:09] and that way it just works [19:52:25] https://gerrit.wikimedia.org/r/#/c/67140/1/main.conf [19:52:25] that's what i was thinking [19:52:30] RECOVERY - Host wtp1008 is UP: PING OK - Packet loss = 0%, RTA = 0.67 ms [19:52:48] https://gerrit.wikimedia.org/r/#/c/65602/ [19:53:13] * Reedy fixes the obvious fail [19:54:14] New patchset: Reedy; "testwikidatawiki (is that even going to be the right dbname?) config" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/65602 [19:54:38] New patchset: Reedy; "testwikidatawiki (is that even going to be the right dbname?) config" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/65602 [19:54:39] !log sync_dir docroot/wikidata_test/ on tin [19:54:47] Logged the message, Master [19:54:49] It should log automagically as normal.. [19:55:03] ok [19:55:05] Reedy: looks ok [19:55:28] Still not sure what the dbname will end up being.. [19:55:34] !log dzahn synchronized docroot/wikidata_test/ [19:55:38] key thing to be different is localClientDatabases but that only important for the cronjob [19:55:43] Logged the message, Master [19:56:07] we can have test2wiki be it's client [19:56:39] Change merged: Dzahn; [operations/apache-config] (master) - https://gerrit.wikimedia.org/r/67140 [19:57:07] Reedy: uhm.. and Apache config is still on fenari as before it looks... [19:57:14] Yup [19:57:15] :Dd [19:57:18] k:) [19:58:14] Errors parsing wmf-config/InitialiseSettings.php [19:58:34] expecting ')' in wmf-config/InitialiseSettings.php on line 1161 [19:59:16] dbname, i dunno, could also be wikidatatestwiki [19:59:16] ori-l there's not much [19:59:19] aside from installing debhelper and related things [19:59:21] and git-buildpackage [19:59:29] autotools [19:59:30] etc. [20:00:32] ottomata: ok, i'll start a module and add you as a reviewer when i have something [20:00:46] ok cool [20:01:11] syncs apache [20:03:25] New review: Dzahn; "http://test.wikidata.org" [operations/apache-config] (master) - https://gerrit.wikimedia.org/r/67140 [20:03:53] gracefuls [20:04:31] !log enabling test.wikidata virtual host [20:04:45] Logged the message, Master [20:05:22] oooh, progress [20:06:21] curl test.wikidata.org | grep Sorry [20:06:58] it's no longer directing to www.wikidata.org [20:07:14] i think the config stuff is needed next [20:07:16] :) so now we can install a wiki [20:07:19] it found no wiki [20:07:21] yea, addwiki [20:07:28] that's expected behaviour [20:07:32] yep [20:07:55] csteipp: done^ [20:08:31] i'll do the addwiki in a little bit [20:08:40] cool!thx [20:08:44] ok [20:09:40] if ( in_array( $wgDBname, array( 'wikidatawiki', 'testwikidatawiki' ) ) ) { [20:09:49] oh, i see, Reedy needs to do more stuff like that above [20:10:00] woudlnt have thought of that one [20:12:04] wikidatawiki -> wikidata [20:12:16] New patchset: coren; "Tool Labs: deploy ruby1.9.3 per user request" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/67143 [20:15:23] James_F, I keep getting "Error loading data from server: Unsuccessful request: Error contacting the Parsoid server." from VE's [Review and save] on mediawiki.org [20:16:21] spagewmf: Hmm. [20:16:24] api.php response is {"servedby":"mw1116","error":{"code":"parsoidserver","info":"Error contacting the Parsoid server"}} [20:16:31] Yeah. [20:16:50] !log reedy synchronized php-1.22wmf5/extensions/SecurePoll/resources/SecurePoll.js [20:16:57] Logged the message, Master [20:17:00] spagewmf: It's working for me. :-( [20:17:32] New patchset: Aude; "Fix wikidata singlenode manifest" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/67144 [20:18:28] spagewmf: https://icinga.wikimedia.org/cgi-bin/icinga/status.cgi?hostgroup=parsoid_eqiad&style=detail&nostatusheader isn't good. [20:18:40] New review: Aude; "i still get errors like:" [operations/puppet] (production) C: -1; - https://gerrit.wikimedia.org/r/67144 [20:19:53] !log attempting parsoid restart on wtp1004 [20:19:56] James_F: ^ [20:20:03] Logged the message, Master [20:20:04] mutante: Thanks. [20:20:16] New review: coren; "Trivial package addition" [operations/puppet] (production) C: 2; - https://gerrit.wikimedia.org/r/67143 [20:20:16] Change merged: coren; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/67143 [20:20:24] mutante: But shouldn't the LB route around it [20:20:25] ? [20:21:03] { 'host': 'wtp1004.eqiad.wmnet', 'weight': 15, 'enabled': True } [20:21:12] <- it's enabled in pybal [20:22:25] hmm, is this related to the port change yesterday? [20:22:44] mutante: Not sure; RoanKattouw's en route back to the office right now, will throw to him. [20:22:54] mutante: But the service appears to be working for me. [20:22:57] i don't see it listening on 80 [20:23:02] but we changed monitoring to port 80 [20:23:07] yesterday [20:23:10] ok [20:23:56] James_F mutante FYI I reloaded the page and retried, same error, same served-by mw1116. Maybe the MW servers have a sticky connection to a parsoidserver [20:24:17] spagewmf: They really shouldn't; it's load-balanced for a reason. [20:25:10] PROBLEM - Puppet freshness on erzurumi is CRITICAL: No successful Puppet run in the last 10 hours [20:25:10] PROBLEM - Puppet freshness on lvs1005 is CRITICAL: No successful Puppet run in the last 10 hours [20:25:10] PROBLEM - Puppet freshness on lvs1004 is CRITICAL: No successful Puppet run in the last 10 hours [20:25:10] PROBLEM - Puppet freshness on lvs1006 is CRITICAL: No successful Puppet run in the last 10 hours [20:25:10] PROBLEM - Puppet freshness on mc15 is CRITICAL: No successful Puppet run in the last 10 hours [20:25:11] PROBLEM - Puppet freshness on pdf1 is CRITICAL: No successful Puppet run in the last 10 hours [20:25:11] PROBLEM - Puppet freshness on pdf2 is CRITICAL: No successful Puppet run in the last 10 hours [20:25:12] PROBLEM - Puppet freshness on ms-be1 is CRITICAL: No successful Puppet run in the last 10 hours [20:25:12] PROBLEM - Puppet freshness on ms-fe3001 is CRITICAL: No successful Puppet run in the last 10 hours [20:25:13] PROBLEM - Puppet freshness on virt1 is CRITICAL: No successful Puppet run in the last 10 hours [20:25:13] PROBLEM - Puppet freshness on virt4 is CRITICAL: No successful Puppet run in the last 10 hours [20:25:14] PROBLEM - Puppet freshness on virt3 is CRITICAL: No successful Puppet run in the last 10 hours [20:25:31] RECOVERY - Disk space on ms-be1012 is OK: DISK OK [20:26:28] !log reedy synchronized php-1.22wmf5/extensions/SecurePoll/ [20:26:41] Logged the message, Master [20:29:19] James_F: OK, no worries. Let me know if there's anything I can do to help troubleshoot. [20:29:31] spagewmf: Thanks. :-( [20:33:01] Reedy/AaronSchulz: hey, what needs to happen for maintenance/eraseArchivedFile.php to find its way to e.g. terbium? [20:33:10] is it a simple deploy, is it a scap? [20:33:17] or does it need a backport? [20:33:19] I have no idea :) [20:33:23] it's not scap [20:34:05] hmm, other stuff in maintenance is in puppet in manifests/misc/maintenance.pp isnt it [20:34:38] usually, from what I read [20:35:04] sounds similar to class misc::maintenance::refreshlinks [20:35:35] that's for the cronjob [20:35:39] I don't need a cronjob [20:35:56] one can backport or wait for wmf6 [20:36:06] ie: tomorrow [20:36:12] okay, tomorrow wfm [20:36:13] thanks [20:36:36] AaronSchulz: sorry for asking about it all the time, it kinda helps me understand your processes better :) [20:36:57] tomorrow? [20:36:57] paravoid: aah, this is the script for actually deleting from "deleted" containers? cool [20:37:08] tomorrow works for me, yes [20:37:15] mutante: yep [20:37:16] well, nothing is happening then :) [20:37:19] :)) [20:37:25] the scary script, AaronSchulz? [20:37:38] is there something that needs to be done soon? [20:37:38] ;) [20:37:55] there is one ticket to delete stuff but that has been open for a while [20:37:58] * AaronSchulz can do any backporting if so [20:42:14] New patchset: Dzahn; "add "/entity/" redirects for wikidata per" [operations/apache-config] (master) - https://gerrit.wikimedia.org/r/65463 [20:43:49] New review: Faidon; "So, abandon?" [operations/puppet/cdh4] (master) - https://gerrit.wikimedia.org/r/65408 [20:45:57] Change abandoned: Ottomata; "Abandoned in favor of:" [operations/puppet/cdh4] (master) - https://gerrit.wikimedia.org/r/65408 [20:46:49] New patchset: MaxSem; "Tweak $wgMFCustomLogos grouping" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/67148 [20:54:06] Change merged: Ryan Lane; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/67108 [20:54:52] Change merged: Ryan Lane; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/67109 [20:55:22] Change merged: Ryan Lane; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/63514 [20:56:40] Change merged: Ryan Lane; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/67110 [20:57:26] ^demon: merged all the way through [20:57:32] <^demon> Thanks! [20:57:35] ^demon: any other reviews needed? [20:57:49] <^demon> Just those 4, thanks. [20:57:53] cool. yw [20:59:51] New review: Aude; "note that i have not tried this, and want a second opinion" [operations/apache-config] (master) - https://gerrit.wikimedia.org/r/65463 [21:02:19] New patchset: Jforrester; "Enable VisualEditor (still as alpha) on all Wikipedias" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/67151 [21:05:37] <^demon> !log killed gitweb, long live gitblit [21:05:45] Logged the message, Master [21:06:12] \o/ [21:06:27] New review: Faidon; "As long as you babysit it to make sure no RAID alert storm comes :-)" [operations/puppet] (production) C: 2; - https://gerrit.wikimedia.org/r/67134 [21:06:41] yay [21:06:54] git.wm.o seems to be taking a really long time to load [21:06:56] <^demon> I'm going to have to do some perf tweaking on gitblit. It's kinda slow after I indexed all the repos for lucene :) [21:07:05] :o [21:07:15] <^demon> It was way faster before I made searches work [21:07:15] mutante: Parsoid itself is still on port 8000, it's just the caches that moved ports [21:07:26] wtp1004 is probably still hosed from when I gave it to Gabriel to play with [21:07:28] * RoanKattouw looks [21:07:30] bleh. can we remove the translate bot from the stats? :) [21:07:43] <^demon> Ryan_Lane: No way I know of. [21:07:46] :( [21:07:59] <^demon> Upstream guy is nice, I'll file a bu [21:08:01] <^demon> g [21:08:20] New review: Aude; "i am not able to verify this on my test wiki" [operations/apache-config] (master) - https://gerrit.wikimedia.org/r/65463 [21:08:28] it shows 51% in the stats, ruining it for everyone else :D [21:08:36] Change merged: Andrew Bogott; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/67134 [21:08:40] <3 https://git.wikimedia.org/metrics/operations%2Fpuppet.git [21:08:43] I <3 this so much [21:08:47] actual stats! [21:09:30] <^demon> I think the rss feeds are pretty sweet too [21:09:36] yeah [21:10:04] woah, thats pretty cool [21:10:08] oh, that's cute Ryan_Lane [21:10:31] Shame the plugin doesn't work though [21:10:48] there's something we can run against our repos for much better stats [21:10:55] something the openstack folks are running [21:11:06] wth was that again? [21:11:11] <^demon> Not a clue. [21:11:20] http://vizgrimoire.bitergia.org/ [21:11:25] <^demon> I need to package up gitblit too, that's on my todo list next. [21:11:58] <^demon> And the rewrite rules for old gitweb urls. [21:12:00] Change merged: jenkins-bot; [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/66167 [21:12:28] New review: MaxSem; "To be deployed only with https://gerrit.wikimedia.org/r/66888" [operations/mediawiki-config] (master) C: -2; - https://gerrit.wikimedia.org/r/66891 [21:12:34] <^demon> I could probably allocate more to the JVM, maybe more threads too. [21:12:35] New patchset: Andrew Bogott; "Revert "pep8 cleanup for icinga files"" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/67154 [21:12:38] <^demon> Prolly would speed things up [21:13:01] Change merged: Andrew Bogott; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/67154 [21:14:02] aha http://activity.openstack.org/dash/ [21:14:06] that's what I was looking for [21:14:14] bleh. is it broken now? [21:14:57] <^demon> Filed https://code.google.com/p/gitblit/issues/detail?id=251 for the stats request. [21:15:05] cool, thanks [21:17:26] Change merged: jenkins-bot; [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/67148 [21:17:33] <^demon> Chad needs food, badly. [21:17:52] we've got a flood of 10.64.32.41 apache2[17099]: [notice] child pid 26472 exit signal Bus error (7) [21:17:52] Change merged: jenkins-bot; [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/65843 [21:19:15] that's mw1171 and it doesnt't answer ssh either [21:19:24] looking at mgmt [21:19:27] bus error is either memory or cpu [21:19:44] and it's flooding apache.log [21:19:54] looks frozen, no mgmt output, no ssh [21:20:01] gotta powercycle [21:20:34] if i could quite console now.. arg [21:20:43] mutante: mw1171 is rt 5231 [21:20:44] bad hd [21:20:50] but you can poweroff it [21:21:38] ah, thanks, ok [21:21:46] PROBLEM - DPKG on mw1171 is CRITICAL: Timeout while attempting connection [21:21:50] !log shutting down mw1171 [21:21:50] yes please, make it shut up [21:21:57] Logged the message, Master [21:22:27] RECOVERY - Parsoid on wtp1004 is OK: HTTP OK: HTTP/1.1 200 OK - 1373 bytes in 0.014 second response time [21:22:29] !log Killed old process on wtp1004 that was hogging the port [21:22:37] Logged the message, Mr. Obvious [21:22:52] mutante, James_F: ---^^ [21:23:16] PROBLEM - Host mw1171 is DOWN: PING CRITICAL - Packet loss = 100% [21:23:49] RoanKattouw: that would be nodejs processes? [21:25:29] Yeah [21:25:38] There was an old stale one that was hanging on to port 8000 [21:26:55] gotcha, thanks [21:27:10] ^demon|away, Ryan_Lane: Is there a way to consolidate users in that stats thing? VE has a lot of SVN/git duplicates [21:27:10] i saw the other message as well about port 80 [21:27:31] SVN? [21:27:37] how's that even possible? [21:29:01] Ryan_Lane: From before the switch-over. [21:29:17] I didn't realize VE even existed before it :D [21:29:22] https://git.wikimedia.org/metrics/mediawiki%2Fextensions%2FVisualEditor.git [21:29:28] Ryan_Lane: We've been going a while. [21:29:33] See inez@wikia-inc.com (git) and inez@users.mediawiki.org (SVN) [21:29:53] gwicke and tparscal are also duplicated there [21:30:03] I dunno, you'd need to ask chad [21:30:06] And presumably more people would be too, if only we could see them. :-) [21:30:10] I'd imagine he'll need to file another upstream [21:30:20] Or just munge our histories? [21:30:27] yeah, these stats aren't terrible in-depth [21:30:36] James_F: that would be evil [21:30:43] Ryan_Lane: Indeed. :-) [21:30:47] you don't rebase published history :) [21:30:54] * James_F votes for what he expects Chad to suggest, though. :-D [21:31:23] New patchset: Dzahn; "change test.wikidata docroot to testwikidata or language will be wikidata_test :p" [operations/apache-config] (master) - https://gerrit.wikimedia.org/r/67160 [21:31:40] I included Chad in my initial ping for that reason [21:32:57] :) [21:33:11] New patchset: Dzahn; "move complete docroot for test.wikidata from wikidata_test to testwikidata" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/67161 [21:34:22] New patchset: Dzahn; "move complete docroot for test.wikidata from wikidata_test to testwikidata" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/67161 [21:34:45] Change merged: Dzahn; [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/67161 [21:36:50] sees new images/mobile/*.png [21:37:03] gitweb and gitblit appear down [21:38:43] !log dzahn synchronized docroot/testwikidata [21:38:50] Logged the message, Master [21:38:55] there is no /r/ on manganese [21:39:02] Oh, no, just gitweb. [21:39:20] initial load of gitblit is pretty slow [21:39:36] Clonebomb? [21:39:39] gitweb has been replaced [21:40:00] http://identi.ca/notice/101205228 [21:40:18] Change merged: Dzahn; [operations/apache-config] (master) - https://gerrit.wikimedia.org/r/67160 [21:40:22] LeslieCarr, you around? [21:40:32] * LeslieCarr ducks [21:40:35] wikitech-l would be my first source for that news [21:40:40] LeslieCarr, good idea [21:40:52] LeslieCarr, quick question, i promise. [21:41:06] hehe [21:41:43] LeslieCarr, it doesn't seem customary to me, but do you run across the following types of rules for egress filtering? filter wikipedia1 l34-protocol tcp server-ip 208.80.152.0 0.0.3.255 [21:41:53] marktraceur: subscribe to @wikimediatech, done ;) [21:42:03] greg-g: Unhelpful to the many people on wikitech-l [21:42:18] marktraceur: /me jokes hence the ;) [21:42:22] LeslieCarr, it seems like the subnetting is backwards, and would result in only permitting outbound to 0.0.0.0 - 0.0.3.255 [21:42:23] *nod* [21:43:00] hehehe [21:43:01] looks like a wildcard rather than a netmask [21:43:13] there's some wild card style languages [21:43:15] trying to remember who all uses that [21:43:19] anti-netmask! [21:43:20] cisco [21:43:39] http://www.subnetonline.com/pages/subnet-calculators/ipv4-wildcard-calculator.php [21:44:48] Meh, no redirect for https://gerrit.wikimedia.org/r/gitweb URLs to Gitblit possible? mediawiki.org alone lists 441 pages with Gitweb URLs... [21:44:50] dr0ptp4kt: http://en.wikipedia.org/wiki/Wildcard_mask [21:45:14] andre__: how's the flooding ?:o [21:46:25] mutante: today it's slowly going down again and it feels more like a normal day again, though still no metro running etc. and enough army/police/etc on the streets [21:46:27] mutante and LeslieCarr, thx much! [21:47:32] andre__: when i heard about i though that was just in time to do the boat tour [21:47:35] dr0ptp4kt: yw [21:48:16] mutante: still wondering where one my fav clubs (boat) went. Or when it comes back to where it was. :P [21:49:02] ouch, you're favorite club swam away? [21:49:15] i hope cross club is fine [21:50:01] * mutante gracefuls Apaches again [21:50:15] mutante: I guess they moved it to a safe place. Though I have hard times to imagine a safe place. Crossclub is fine this time, but floodings were about two meters lower than 2002 [21:50:41] Reedy: should be all done, it is "testwikidata" now [21:54:30] about to scap.... [21:59:59] !log maxsem Started syncing Wikimedia installation... : Weekly mobile deployment [22:00:13] Logged the message, Master [22:08:59] !log maxsem Finished syncing Wikimedia installation... : Weekly mobile deployment [22:09:07] Logged the message, Master [22:20:55] !log upgrading salt in labs and production [22:21:03] Logged the message, Master [22:25:30] PROBLEM - Parsoid on wtp1001 is CRITICAL: Connection refused [22:25:30] PROBLEM - Parsoid on wtp1023 is CRITICAL: Connection refused [22:25:30] PROBLEM - Parsoid on wtp1002 is CRITICAL: Connection refused [22:25:30] PROBLEM - Parsoid on wtp1016 is CRITICAL: Connection refused [22:25:31] PROBLEM - Parsoid on kuo is CRITICAL: Connection refused [22:25:31] PROBLEM - Parsoid on lardner is CRITICAL: Connection refused [22:25:31] PROBLEM - Parsoid on wtp1 is CRITICAL: Connection refused [22:25:31] PROBLEM - Parsoid on wtp1015 is CRITICAL: Connection refused [22:25:31] PROBLEM - Parsoid on tola is CRITICAL: Connection refused [22:25:32] PROBLEM - LVS HTTP IPv4 on parsoidcache.svc.pmtpa.wmnet is CRITICAL: HTTP CRITICAL: HTTP/1.1 503 Service Unavailable - 676 bytes in 0.054 second response time [22:26:00] well, shit [22:26:09] we really need to make that not happen [22:26:16] heh [22:26:19] what's up? [22:26:22] parsoid [22:26:30] RoanKattouw: you doing anything ? [22:26:30] I got that [22:26:32] it's processes are connected to salt somehow [22:26:40] oh it is ? how is that ? [22:26:42] likely because of git-deploy [22:26:51] salt restarting a daemon doesn't link it, though [22:26:55] I'm not doing anything [22:26:56] PROBLEM - Parsoid on wtp1014 is CRITICAL: Connection refused [22:26:57] PROBLEM - Parsoid Varnish on celsus is CRITICAL: HTTP CRITICAL: HTTP/1.1 503 Service Unavailable - 674 bytes in 0.062 second response time [22:26:57] PROBLEM - Parsoid on wtp1019 is CRITICAL: Connection refused [22:26:57] PROBLEM - Parsoid on wtp1006 is CRITICAL: Connection refused [22:26:57] PROBLEM - Parsoid on wtp1018 is CRITICAL: Connection refused [22:26:57] PROBLEM - Parsoid on wtp1012 is CRITICAL: Connection refused [22:26:57] PROBLEM - Parsoid on wtp1017 is CRITICAL: Connection refused [22:26:58] it's something wrong with the init script [22:26:58] PROBLEM - Parsoid on wtp1024 is CRITICAL: Connection refused [22:26:58] PROBLEM - Parsoid on wtp1005 is CRITICAL: Connection refused [22:26:59] PROBLEM - Parsoid on mexia is CRITICAL: Connection refused [22:26:59] PROBLEM - Parsoid on wtp1009 is CRITICAL: Connection refused [22:27:00] PROBLEM - Parsoid on wtp1021 is CRITICAL: Connection refused [22:27:04] PROBLEM - Parsoid on wtp1020 is CRITICAL: Connection refused [22:27:04] PROBLEM - Parsoid on wtp1011 is CRITICAL: Connection refused [22:27:04] PROBLEM - LVS HTTP IPv4 on parsoid.svc.pmtpa.wmnet is CRITICAL: Connection refused [22:27:05] AaronSchulz: guess where 40% of master DB query time goes [22:27:15] we need to restart parsoid on all of those boxes [22:27:20] On it [22:27:26] PROBLEM - Parsoid on wtp1010 is CRITICAL: Connection refused [22:27:26] PROBLEM - Parsoid on wtp1003 is CRITICAL: Connection refused [22:27:26] PROBLEM - Parsoid Varnish on constable is CRITICAL: HTTP CRITICAL: HTTP/1.1 503 Service Unavailable - 676 bytes in 0.058 second response time [22:27:34] also starting [22:27:36] PROBLEM - Parsoid on wtp1013 is CRITICAL: Connection refused [22:27:36] RECOVERY - Parsoid on wtp1016 is OK: HTTP OK: HTTP/1.1 200 OK - 1373 bytes in 0.009 second response time [22:27:36] PROBLEM - Parsoid on wtp1022 is CRITICAL: Connection refused [22:27:36] PROBLEM - Parsoid on wtp1007 is CRITICAL: Connection refused [22:27:38] I did it on all of the wtp [22:27:46] RECOVERY - Parsoid on wtp1014 is OK: HTTP OK: HTTP/1.1 200 OK - 1373 bytes in 0.009 second response time [22:27:50] kk [22:27:55] TimStarling: I haven't checked in a few days...last time ishmael looked fine [22:27:56] RECOVERY - Parsoid on wtp1019 is OK: HTTP OK: HTTP/1.1 200 OK - 1373 bytes in 0.010 second response time [22:27:56] RECOVERY - Parsoid on wtp1006 is OK: HTTP OK: HTTP/1.1 200 OK - 1373 bytes in 0.005 second response time [22:27:56] RECOVERY - Parsoid on wtp1017 is OK: HTTP OK: HTTP/1.1 200 OK - 1373 bytes in 0.003 second response time [22:27:56] RECOVERY - Parsoid on wtp1012 is OK: HTTP OK: HTTP/1.1 200 OK - 1373 bytes in 0.007 second response time [22:27:56] RECOVERY - Parsoid on wtp1018 is OK: HTTP OK: HTTP/1.1 200 OK - 1373 bytes in 0.009 second response time [22:27:57] RECOVERY - Parsoid on wtp1024 is OK: HTTP OK: HTTP/1.1 200 OK - 1373 bytes in 0.021 second response time [22:27:58] RECOVERY - Parsoid on wtp1005 is OK: HTTP OK: HTTP/1.1 200 OK - 1373 bytes in 0.008 second response time [22:27:58] RECOVERY - Parsoid on wtp1009 is OK: HTTP OK: HTTP/1.1 200 OK - 1373 bytes in 0.007 second response time [22:27:58] RECOVERY - Parsoid on wtp1021 is OK: HTTP OK: HTTP/1.1 200 OK - 1373 bytes in 0.007 second response time [22:28:06] RECOVERY - Parsoid on wtp1020 is OK: HTTP OK: HTTP/1.1 200 OK - 1373 bytes in 0.002 second response time [22:28:06] RECOVERY - Parsoid on wtp1011 is OK: HTTP OK: HTTP/1.1 200 OK - 1373 bytes in 0.005 second response time [22:28:07] RECOVERY - Parsoid on wtp1 is OK: HTTP OK: HTTP/1.1 200 OK - 1373 bytes in 0.057 second response time [22:28:07] RECOVERY - Parsoid on wtp1010 is OK: HTTP OK: HTTP/1.1 200 OK - 1373 bytes in 0.008 second response time [22:28:07] RECOVERY - Parsoid on wtp1003 is OK: HTTP OK: HTTP/1.1 200 OK - 1373 bytes in 0.004 second response time [22:28:07] RECOVERY - Parsoid on wtp1015 is OK: HTTP OK: HTTP/1.1 200 OK - 1373 bytes in 0.010 second response time [22:28:07] RECOVERY - Parsoid on wtp1001 is OK: HTTP OK: HTTP/1.1 200 OK - 1373 bytes in 0.006 second response time [22:28:08] RECOVERY - Parsoid on wtp1002 is OK: HTTP OK: HTTP/1.1 200 OK - 1373 bytes in 0.010 second response time [22:28:19] TimStarling: where? [22:28:29] we need to take a look at the init script [22:28:37] RECOVERY - Parsoid on wtp1013 is OK: HTTP OK: HTTP/1.1 200 OK - 1373 bytes in 0.003 second response time [22:28:37] RECOVERY - Parsoid on wtp1022 is OK: HTTP OK: HTTP/1.1 200 OK - 1373 bytes in 0.002 second response time [22:28:38] RECOVERY - Parsoid on wtp1023 is OK: HTTP OK: HTTP/1.1 200 OK - 1373 bytes in 0.009 second response time [22:28:38] RECOVERY - Parsoid on wtp1007 is OK: HTTP OK: HTTP/1.1 200 OK - 1373 bytes in 0.006 second response time [22:28:53] Hey they're coming back [22:28:56] yeah [22:29:02] I restarted their daemons with salt [22:29:05] What init script? Parsoid's? [22:29:06] RECOVERY - LVS HTTP IPv4 on parsoidcache.svc.pmtpa.wmnet is OK: HTTP OK: HTTP/1.1 200 OK - 1357 bytes in 0.066 second response time [22:29:08] "# SIGHUP isn't supported by the Parsoid server yet, but this should be easy to implement" [22:29:08] Or salt's? [22:29:10] RoanKattouw: yeah [22:29:11] <--- [22:29:13] parsoid's [22:29:19] mutante: you did that? [22:29:22] Yeah Parsoid doesn't do SIGHUP [22:29:30] It would be nice if it could [22:29:33] Ryan_Lane: no, i just see that comment in the init script [22:29:37] ah. ok [22:29:38] But right now we just restart it [22:29:41] AaronSchulz / paravoid: MessageCache::getMsgFromNamespace() [22:29:46] RECOVERY - Parsoid Varnish on celsus is OK: HTTP OK: HTTP/1.1 200 OK - 1357 bytes in 0.078 second response time [22:29:47] it's likely something to do with how it forks [22:29:54] The Parsoid init script is horrible and I take all the blame [22:29:56] RECOVERY - Parsoid on mexia is OK: HTTP OK: HTTP/1.1 200 OK - 1373 bytes in 0.059 second response time [22:29:58] heh [22:30:02] But I don't understand how it could suddenly break like this [22:30:05] hm. weird [22:30:06] RECOVERY - Parsoid on lardner is OK: HTTP OK: HTTP/1.1 200 OK - 1373 bytes in 0.062 second response time [22:30:06] TimStarling: not seeing that for db1056 [22:30:07] RECOVERY - Parsoid Varnish on constable is OK: HTTP OK: HTTP/1.1 200 OK - 1358 bytes in 0.058 second response time [22:30:18] I didn't do anythong on anything other than the wtp* boxes [22:30:40] maybe upstart is automatically fixing them? [22:31:06] RECOVERY - LVS HTTP IPv4 on parsoid.svc.pmtpa.wmnet is OK: HTTP OK: HTTP/1.1 200 OK - 1373 bytes in 0.055 second response time [22:31:08] AaronSchulz: on ishmael? I assume it only measures slow queries [22:31:20] RoanKattouw: git deploy restarts the process [22:31:30] RoanKattouw: and likely the pid is somehow getting linked to salt [22:31:31] I did statistics on the wall clock time of all query-m profiling sections [22:31:39] salt restart = parsoid dies [22:31:54] I just upgraded salt everywhere [22:31:56] Ooooh I see [22:32:02] so, it restarted it everywhere [22:32:06] Yeah it probably doesn't fork very well [22:32:10] indeed [22:32:14] Parsoid is not natively a daemon [22:32:24] upstart can handle that [22:32:28] So I'm doing all sorts of weird shit in the init script to try and launch it as a daemon [22:32:30] so can start-stop-daemon [22:32:32] Right [22:32:32] 42% was from a revision query characteristic of Revision::READ_LATEST [22:32:36] That's what I'm using [22:32:39] * Ryan_Lane nods [22:32:44] But somehow it seems it must not be working [22:32:53] At this point I think I should let the professionals handle this [22:32:55] we are using supervisord [22:32:56] MessageCache is one of the few things that uses it, and pulling that query out of SHOW FULL PROCESSLIST does indeed show namespace 8 [22:32:58] RoanKattouw: :D [22:33:01] I evidently should not be in the business of writing init scripts [22:33:25] RoanKattouw: so , something i was checking out - constable and celsus aren't referenced in role/cache.pp:988 [22:33:27] but both require it [22:33:30] since there's no else [22:33:35] so it's killing puppet [22:33:51] Ugh [22:33:59] TimStarling: though ishmael has a lot of wonky numbers in the details section [22:34:06] Right I guess I need to set those up properly then [22:34:09] Will work on that [22:34:09] ...that should probably be cleaned up sometime [22:34:21] average time is only 2.4ms [22:34:42] per query [22:35:06] RECOVERY - Parsoid on tola is OK: HTTP OK: HTTP/1.1 200 OK - 1373 bytes in 0.057 second response time [22:35:07] RECOVERY - Parsoid on kuo is OK: HTTP OK: HTTP/1.1 200 OK - 1373 bytes in 0.059 second response time [22:35:22] * AaronSchulz switches to graphite [22:35:39] but that's much slower than the WikiPage::loadPageData query [22:38:28] I can see them in tcpdump also [22:38:47] I think some stats on message name will be interesting, one minute [22:42:24] yeah, so according to tcpdump 90% of those queries are for one message: Guidedtour-help-url [22:43:45] New patchset: Kaldari; "Adding Disambiguator extension to extension list" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/67174 [22:44:14] is that expected, lwelling? [22:45:44] New review: JanZerebecki; "Sorry, was too hasty. This needs the PT flag to work as /wiki is an Alias ( https://httpd.apache.org..." [operations/apache-config] (master) C: -1; - https://gerrit.wikimedia.org/r/65463 [22:46:19] New review: Catrope; "Blocking this pending a Parsoid config change" [operations/mediawiki-config] (master) C: -2; - https://gerrit.wikimedia.org/r/67151 [22:50:17] New review: Reedy; "Need to add it to mediawiki/tools/release in make-wmf-branch too" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/67174 [22:51:09] !log maxsem synchronized wmf-config [22:51:16] Logged the message, Master [22:54:20] PROBLEM - NTP on ssl3003 is CRITICAL: NTP CRITICAL: No response from NTP server [22:54:20] PROBLEM - NTP on ssl3002 is CRITICAL: NTP CRITICAL: No response from NTP server [22:54:23] TimStarling: to bad there is no way to get these breakdowns in graphite [22:54:31] or if there it is, it is not obvious [22:58:23] binasher: does report.py have a json mode or something? [22:59:09] AaronSchulz: nope, want it to? [22:59:28] * binasher reads chan log [22:59:47] heya TimStarling, heads up that we're going to need your help on the SecurePoll stuff for this weekend's election.... [23:00:09] New patchset: Dzahn; "add "/entity/" redirects for wikidata per" [operations/apache-config] (master) - https://gerrit.wikimedia.org/r/65463 [23:00:28] AaronSchulz: I can give you my spreadsheet [23:00:35] TimStarling: Reedy has been point from engineering's side thus far, but we need to get this moved forward as soon as possible, so I hope Reedy (and Jamesofur ) can hand off what knowledge/next steps need to happen now. [23:01:22] greg-g: ok [23:01:30] \o I'm writing up some quick config changes that I'll post on meta [23:01:39] and writing up a jump config for there as well [23:02:00] TimStarling: I'm assuming I'm right that we can't have local auth and remote auth on the same wiki? [23:03:14] * AaronSchulz opened report.py with "view source" [23:03:18] * AaronSchulz will not do that again in FF [23:03:57] lwelling: ping [23:04:15] New review: Dzahn; "so we're doing the real external redirect and always to https, so that the /entity/ links work. if t..." [operations/apache-config] (master); V: 2 C: 2; - https://gerrit.wikimedia.org/r/65463 [23:04:16] Change merged: Dzahn; [operations/apache-config] (master) - https://gerrit.wikimedia.org/r/65463 [23:04:22] * AaronSchulz wants non-crap profiling [23:04:24] Jamesofur: whether to use local or remote is specified in the vote configuration, so no, you couldn't use both at the same time [23:04:35] AaronSchulz: do you want my spreadsheet? [23:04:36] * Jamesofur nods [23:04:47] have you got gnumeric? [23:04:54] AaronSchulz: It's overrated [23:04:57] New review: Dzahn; "http://wikidata.org" [operations/apache-config] (master) - https://gerrit.wikimedia.org/r/65463 [23:04:58] TimStarling: AaronSchulz: re ishmael, https://ishmael.wikimedia.org = just slow queries; http://ishmael.wikimedia.org/sample/ = pulls from a different data source that summarizes all queries during 30 second sample windows several times an hour [23:05:43] AaronSchulz: were you looking at the slow or sampled data via ishmael? [23:05:56] AaronSchulz: or excel or OOo? [23:06:20] binasher: sampled [23:06:25] * AaronSchulz had that bookmarked [23:06:44] TimStarling: I have libre :) [23:06:44] AaronSchulz: still got a link to the query in question? [23:07:22] New review: Kaldari; "Yep, that's at https://gerrit.wikimedia.org/r/#/c/67177/" [operations/mediawiki-config] (master) C: 2; - https://gerrit.wikimedia.org/r/67174 [23:07:36] binasher: I think tim was talking about Revision::fetchFromConds or the like [23:08:23] New patchset: Spage; "Use new login and create acct forms everywhere" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/67182 [23:08:24] AaronSchulz: see if you can open this: http://tstarling.com/stuff/master-wall-clock-2013-06-06.gnumeric [23:08:24] dbtree contains links to both ishmael for both slow and sample for every db, so no need to bookmark or remember what db is currently serving what role [23:08:37] Change merged: jenkins-bot; [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/67174 [23:08:37] Change merged: jenkins-bot; [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/66987 [23:08:46] TimStarling: so right, Jamesofur made the suggestion, due to that limitation, that we use foundationwiki for the votes. [23:09:51] What's wrong with Meta-Wiki? [23:09:52] binasher: yeah, well I like to check enwiki fairly often :) [23:10:02] greg-g: you could theoretically have one vote that jumps to another vote on the same wiki [23:10:18] so it would do a remote login to itself [23:10:21] Susan: some local users on meta are qualified to vote [23:10:35] And? [23:10:48] not sure if that's ever been tested, but it may possibly work [23:10:52] TimStarling: what do you think would be easier for you guys? [23:11:18] Susan: The same vote can't authenticate users on the local wiki and users on remote wikis [23:11:19] Susan: it's a xor: either the local users on Meta get to vote and not users of other projects, or the other way around [23:11:34] really it needs a new auth class for this configuration [23:11:51] yeah, but given the timing my guess is it's easier to work around it for now [23:12:00] greg-g: That seems like a pretty flawed design. [23:12:07] ;) [23:12:14] !log enabling wikidata "entity" URLs like http://www.wikidata.org/entity/Q1 [23:12:18] Susan: well, agree :) SecurePoll has some issues :) [23:12:22] Logged the message, Master [23:12:52] Is there a bug about that? [23:12:58] Wouldn't this have come up previously? [23:13:08] * AaronSchulz installs gnumeric [23:13:16] no, it was hosted by SPI, so everyone was non-local [23:13:23] obviously it has not come up previously [23:13:40] SecurePoll doesn't have any CentralAuth support [23:13:50] no SPI this year? [23:13:56] right, we're hosting it [23:13:56] just because it's never needed it, not because it couldn't be added [23:14:04] limited time, and all that. [23:14:12] Limited time? [23:14:29] to develop SecurePoll. [23:14:46] this idea of needing both local and remote users is just another way of saying there's a lack of support for fetching voter credentials from a CentralAuth repository [23:14:55] Right... the Board election just snuck up on us, I guess. [23:15:13] there it is [23:15:19] MaxSem/Mobile, are you finished with your deploy? Kaldari and I have Lightning Deploys [23:15:20] Susan: ha ha [23:15:21] anyway, like I said, you can use some rarely-used wiki, and then use the trick of having it redirect to itself [23:15:27] I was grabbing my popcorn, expecting a snarky comment [23:15:35] then both local and remote users will be allowed to vote [23:15:37] Susan: well, sure, there's that ;) But seriously, yes, I wasn't aware of the limitiations and assumed some things incorrectly. So we're scrambling a little now. [23:15:47] paravoid: We don't have time for popcorn. There's an election coming up! [23:15:56] * preilly reminds paravoid that it's 2:15 AM Thursday, June 6, 2013 (EEST) [23:16:03] we do have time for an /ignore though! [23:16:38] AaronSchulz: gnumeric has about the same feature set as LibreOffice calc, but it's about 10 times faster [23:16:50] paravoid: <3 [23:16:51] TimStarling: I defer to your judgement here on where to host it and how. [23:17:03] (obviously) [23:17:24] you can actually open large spreadsheets in it without having to draw on your internal reserves of patience via hours of meditation [23:17:49] ha ha ha [23:17:53] :) [23:18:25] I'm sad that we're not going to use SPI for that [23:18:27] :-) [23:18:41] sorry, I guess I'm not being very productive, I should listen to preilly and go to bed [23:19:32] I picked it after creating a spreadsheet with hundreds of thousands of cells, and then trying to open that spreadsheet in every open source spreadsheet app [23:19:37] gnumeric was the only one that didn't choke [23:20:13] Change merged: jenkins-bot; [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/67182 [23:20:29] spagewmf, we have [23:20:36] yeah...so it would be nice to have a web dashboard for this kind of stuff [23:21:06] (sorry, my kvirc doesn't have / as a delimiter so I wasn't pinged) [23:21:15] paravoid: thanks ;-) [23:21:18] lwelling: ping (2) [23:22:39] A spreadsheet with hundreds of thousands of cells sounds like an extreme edge case. [23:23:09] greg-g: wikimediafoundation.org is not ideal, from a security perspective [23:23:23] login.wikimedia.org, heh. [23:23:24] TimStarling: we were wondering about that... :/ [23:23:36] because translators will be able to inject javascript into it [23:23:42] Susan: no no, vote.wikimedia [23:23:53] TimStarling: seriously? [23:23:58] yes, I think the easiest solution would be to create a wiki for it [23:24:09] Reedy can do that for you in like 5 minutes [23:24:09] oh, I was mostly joking.... [23:24:12] The easiest and yet the most insane. [23:24:18] Not the most. But still. [23:24:25] to have 801 wikis, instead of 800.... [23:24:25] then the second easiest would be to try that self-redirect trick [23:24:27] no big deal [23:24:31] wikis are cheap [23:24:34] that would be a few extra minutes and might not work [23:24:38] * greg-g nods [23:24:42] New patchset: Ryan Lane; "Set new salt master fingerprint for labs master" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/67185 [23:24:57] well, every other board vote has had a new wiki created for it [23:25:01] Reedy: you still around? Able to do that (create vote.wikimedia.org for the obvious purpose)? [23:25:04] aude: You've heard the quote about wikis and babies? [23:25:07] so it's not like it's a ridiculous amount of overhead [23:25:11] lol [23:25:15] Yes, it's easy to create one. It's the upkeep that kills you. [23:26:10] we can lock it when it's no longer needed [23:26:16] right [23:26:35] maybe use some more distinctive name for it [23:26:49] in case some less hackish vote thing is needed in the future [23:26:54] TimStarling: we can't reuse it for later votes? [23:27:26] well, yes, but would you want to do that? [23:27:33] maybe you would... [23:27:53] for security, again, it makes sense to have a separate wiki [23:27:57] unless I'm missing something, I don't see why not? the pages would change, but the domain could stay the same. [23:28:04] configured with no user/site JS, like login.wikimedia.org [23:28:05] Change merged: Ryan Lane; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/67185 [23:28:23] !log mlitn synchronized php-1.22wmf5/extensions/ArticleFeedback/ 'Update ArticleFeedbackv5 to master' [23:28:31] Logged the message, Master [23:29:57] This is like an anti-home wiki, right? [23:30:15] refugee.wikimedia.org [23:30:28] hell.wikimedia.org [23:30:51] oneoff. [23:31:07] Does it want to be a fishbowl or something else? [23:31:42] fishbowl should be fine [23:32:23] (at the risk of sounding like an idiot, there won't be any dns propagation issues with the vote planned to start in 48 hours right?) [23:33:11] dns entry definitely needs creating [23:33:15] fishbowl has CentralAuth disabled, that's pretty much all we need for it [23:33:39] I started scap for E2/E3 lightning deploy [23:33:45] you wouldn't expect any DNS cache to last more than 2 hours [23:33:50] ok [23:34:34] New patchset: Andrew Bogott; "Revert "Revert "pep8 cleanup for icinga files""" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/67187 [23:35:00] Change merged: Andrew Bogott; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/67187 [23:35:36] mutante: Fancy graceful-ing the apaches again? ;) [23:35:43] New patchset: Ryan Lane; "New salt master fingerprint for production" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/67188 [23:35:52] Nobody noticed that test.wikipedia.org is broken? [23:35:52] Reedy: he just left a little bit ago [23:36:35] I'm doing the DNS [23:37:17] Susan: spagewmf is scapping.. and i think that extension was "just" deployed by kaldari [23:37:18] Change merged: Ryan Lane; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/67188 [23:37:32] Reedy: Which extension? [23:37:42] !log spage Started syncing Wikimedia installation... : [23:37:42] Disambigusomethingorother [23:37:45] kaldari added disambiguation, yes. [23:37:47] Logged the message, Master [23:37:52] At least you can rate the page using aft [23:37:58] kaldari: You seem to have broken test.wikipedia.org. [23:38:03] I hope it's only deployed on testwiki! [23:38:27] TimStarling: is it trying to load a message that does not exist? [23:38:58] https://bugzilla.wikimedia.org/show_bug.cgi?id=49220 [23:39:06] AaronSchulz: the message exists in GettingStarted.i18n.php but not in the namespace [23:39:12] Susan: things are scapping out, should be fixed then (says kaldari in -dev) [23:39:15] which should be fine, it should have a negative cache in memcached [23:39:21] er, -tech [23:39:39] !log added vote.wikimedia.org to DNS [23:39:47] Logged the message, Master [23:40:10] Reedy++ [23:40:12] New patchset: Reedy; "Add VirtualHost for votewiki" [operations/apache-config] (master) - https://gerrit.wikimedia.org/r/67190 [23:40:27] PROBLEM - Auth DNS on ns0.wikimedia.org is CRITICAL: CRITICAL - Plugin timed out while executing system call [23:40:46] did someone push a dns change? [23:40:58] Ryan_Lane: yeah. [23:41:19] yeah, what happened to ns0? [23:41:40] when pushing dns changes, you need to check to ensure all of them restarted properly [23:41:51] there's been a bug in pdns for ages where it occasionally hangs on restart [23:42:00] I've restarted it [23:42:17] RECOVERY - Auth DNS on ns0.wikimedia.org is OK: DNS OK: 0.031 seconds response time. www.wikipedia.org returns 208.80.154.225 [23:42:21] thanks [23:42:32] yw [23:43:35] TimStarling: Reedy Jamesofur So, I need to head out shortly. Go forth and create a voting infrastructure! ;) [23:43:40] !log spage Finished syncing Wikimedia installation... : [23:43:48] Logged the message, Master [23:44:04] whew, fixed, thanks for the note logmsgbot [23:44:05] Ryan_Lane: you did /etc/pdns restart? [23:44:21] /etc/init.d/pdns restart, yeah [23:44:21] I was just going to update https://wikitech.wikimedia.org/wiki/DNS [23:44:27] cool. thanks [23:44:42] TimStarling: what was the message key? [23:46:38] Guidedtour-help-url [23:46:46] so the memcached key should be enwiki:messages:individual:Guidedtour-help-url [23:47:34] New patchset: Reedy; "Add votewiki docroot" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/67191 [23:48:07] * greg-g heads out [23:48:53] AaronSchulz: I think that memcached key should be on mc1009 [23:49:42] Marc Deslauriers is spamming my inbox [23:51:23] New patchset: Reedy; "Add votewiki docroot" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/67191 [23:51:31] Change merged: Reedy; [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/67191 [23:53:51] New patchset: Reedy; "Initial minimal config for votewiki" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/67193 [23:54:23] New patchset: Reedy; "Initial minimal config for votewiki" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/67193 [23:54:39] Change merged: Reedy; [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/67193 [23:55:17] !log reedy synchronized docroot [23:55:25] Logged the message, Master [23:55:54] !log reedy synchronized database lists files: [23:56:01] Logged the message, Master [23:56:47] !log reedy synchronized wmf-config/ [23:56:55] Logged the message, Master [23:57:26] !log reedy rebuilt wikiversions.cdb and synchronized wikiversions files: [23:57:34] Logged the message, Master [23:58:01] Reedy: i'm literally out on the street, laptop on my knees, via phone, but i can heh [23:58:31] it's good if the network is slow, then the agent doesn't die :p [23:59:43] Tim would be more than capable to do it. IIRC I can graceful the apaches, but still need ops to approve and merge the vhost