[00:01:47] binasher: "enable site", heh [00:07:55] RECOVERY - Puppet freshness on mw107 is OK: puppet ran at Fri Jun 28 00:07:50 UTC 2013 [00:07:55] RECOVERY - Puppet freshness on srv259 is OK: puppet ran at Fri Jun 28 00:07:52 UTC 2013 [00:08:03] RECOVERY - Puppet freshness on srv282 is OK: puppet ran at Fri Jun 28 00:07:53 UTC 2013 [00:08:04] RECOVERY - Puppet freshness on srv270 is OK: puppet ran at Fri Jun 28 00:07:53 UTC 2013 [00:08:04] RECOVERY - Puppet freshness on mw1031 is OK: puppet ran at Fri Jun 28 00:07:55 UTC 2013 [00:08:04] RECOVERY - Puppet freshness on srv240 is OK: puppet ran at Fri Jun 28 00:07:57 UTC 2013 [00:08:04] RECOVERY - Puppet freshness on mw114 is OK: puppet ran at Fri Jun 28 00:07:58 UTC 2013 [00:08:04] RECOVERY - Puppet freshness on mw115 is OK: puppet ran at Fri Jun 28 00:07:58 UTC 2013 [00:08:05] RECOVERY - Puppet freshness on tmh1 is OK: puppet ran at Fri Jun 28 00:08:00 UTC 2013 [00:08:05] RECOVERY - Puppet freshness on srv280 is OK: puppet ran at Fri Jun 28 00:08:00 UTC 2013 [00:08:05] RECOVERY - Puppet freshness on mw1045 is OK: puppet ran at Fri Jun 28 00:08:01 UTC 2013 [00:08:13] RECOVERY - Puppet freshness on mw1215 is OK: puppet ran at Fri Jun 28 00:08:02 UTC 2013 [00:08:13] RECOVERY - Puppet freshness on mw1220 is OK: puppet ran at Fri Jun 28 00:08:03 UTC 2013 [00:08:14] RECOVERY - Puppet freshness on mw65 is OK: puppet ran at Fri Jun 28 00:08:04 UTC 2013 [00:08:14] RECOVERY - Puppet freshness on mw1059 is OK: puppet ran at Fri Jun 28 00:08:04 UTC 2013 [00:08:14] RECOVERY - Puppet freshness on mw1112 is OK: puppet ran at Fri Jun 28 00:08:05 UTC 2013 [00:08:14] RECOVERY - Puppet freshness on terbium is OK: puppet ran at Fri Jun 28 00:08:07 UTC 2013 [00:08:14] RECOVERY - Puppet freshness on mw1086 is OK: puppet ran at Fri Jun 28 00:08:07 UTC 2013 [00:08:15] RECOVERY - Puppet freshness on mw1132 is OK: puppet ran at Fri Jun 28 00:08:08 UTC 2013 [00:08:15] RECOVERY - Puppet freshness on mw52 is OK: puppet ran at Fri Jun 28 00:08:09 UTC 2013 [00:08:16] RECOVERY - Puppet freshness on mw1141 is OK: puppet ran at Fri Jun 28 00:08:10 UTC 2013 [00:08:16] RECOVERY - Puppet freshness on search1004 is OK: puppet ran at Fri Jun 28 00:08:10 UTC 2013 [00:08:23] RECOVERY - Puppet freshness on mw1090 is OK: puppet ran at Fri Jun 28 00:08:12 UTC 2013 [00:08:24] RECOVERY - Puppet freshness on linne is OK: puppet ran at Fri Jun 28 00:08:13 UTC 2013 [00:08:24] RECOVERY - Puppet freshness on sq59 is OK: puppet ran at Fri Jun 28 00:08:14 UTC 2013 [00:08:24] RECOVERY - Puppet freshness on mw1200 is OK: puppet ran at Fri Jun 28 00:08:14 UTC 2013 [00:08:24] RECOVERY - Puppet freshness on db36 is OK: puppet ran at Fri Jun 28 00:08:15 UTC 2013 [00:08:24] RECOVERY - Puppet freshness on analytics1017 is OK: puppet ran at Fri Jun 28 00:08:16 UTC 2013 [00:08:24] RECOVERY - Puppet freshness on cp1054 is OK: puppet ran at Fri Jun 28 00:08:18 UTC 2013 [00:08:24] RECOVERY - Puppet freshness on cp1053 is OK: puppet ran at Fri Jun 28 00:08:19 UTC 2013 [00:08:25] RECOVERY - Puppet freshness on cp1035 is OK: puppet ran at Fri Jun 28 00:08:20 UTC 2013 [00:08:26] RECOVERY - Puppet freshness on db1045 is OK: puppet ran at Fri Jun 28 00:08:20 UTC 2013 [00:08:26] RECOVERY - Puppet freshness on es1008 is OK: puppet ran at Fri Jun 28 00:08:21 UTC 2013 [00:08:27] RECOVERY - Puppet freshness on wtp1020 is OK: puppet ran at Fri Jun 28 00:08:22 UTC 2013 [00:08:35] RECOVERY - Puppet freshness on wtp1008 is OK: puppet ran at Fri Jun 28 00:08:22 UTC 2013 [00:08:35] RECOVERY - Puppet freshness on cp1039 is OK: puppet ran at Fri Jun 28 00:08:23 UTC 2013 [00:08:35] RECOVERY - Puppet freshness on db1057 is OK: puppet ran at Fri Jun 28 00:08:24 UTC 2013 [00:08:35] RECOVERY - Puppet freshness on ssl1002 is OK: puppet ran at Fri Jun 28 00:08:25 UTC 2013 [00:08:35] RECOVERY - Puppet freshness on ms-be1004 is OK: puppet ran at Fri Jun 28 00:08:26 UTC 2013 [00:08:36] RECOVERY - Puppet freshness on sq37 is OK: puppet ran at Fri Jun 28 00:08:27 UTC 2013 [00:08:36] RECOVERY - Puppet freshness on chromium is OK: puppet ran at Fri Jun 28 00:08:28 UTC 2013 [00:08:36] RECOVERY - Puppet freshness on cp1068 is OK: puppet ran at Fri Jun 28 00:08:28 UTC 2013 [00:08:37] RECOVERY - Puppet freshness on sq77 is OK: puppet ran at Fri Jun 28 00:08:30 UTC 2013 [00:08:38] RECOVERY - Puppet freshness on harmon is OK: puppet ran at Fri Jun 28 00:08:30 UTC 2013 [00:08:38] RECOVERY - Puppet freshness on ms-be1003 is OK: puppet ran at Fri Jun 28 00:08:30 UTC 2013 [00:08:39] RECOVERY - Puppet freshness on db1056 is OK: puppet ran at Fri Jun 28 00:08:31 UTC 2013 [00:08:44] RECOVERY - Puppet freshness on db44 is OK: puppet ran at Fri Jun 28 00:08:34 UTC 2013 [00:08:44] RECOVERY - Puppet freshness on srv271 is OK: puppet ran at Fri Jun 28 00:08:35 UTC 2013 [00:08:44] RECOVERY - Puppet freshness on cp1001 is OK: puppet ran at Fri Jun 28 00:08:35 UTC 2013 [00:08:44] RECOVERY - Puppet freshness on virt1005 is OK: puppet ran at Fri Jun 28 00:08:35 UTC 2013 [00:08:44] RECOVERY - Puppet freshness on srv268 is OK: puppet ran at Fri Jun 28 00:08:36 UTC 2013 [00:08:44] RECOVERY - Puppet freshness on mw1135 is OK: puppet ran at Fri Jun 28 00:08:40 UTC 2013 [00:08:45] RECOVERY - Puppet freshness on es2 is OK: puppet ran at Fri Jun 28 00:08:41 UTC 2013 [00:08:46] RECOVERY - Puppet freshness on sq49 is OK: puppet ran at Fri Jun 28 00:08:41 UTC 2013 [00:08:55] RECOVERY - Puppet freshness on ms1004 is OK: puppet ran at Fri Jun 28 00:08:42 UTC 2013 [00:08:55] RECOVERY - Puppet freshness on cp1021 is OK: puppet ran at Fri Jun 28 00:08:43 UTC 2013 [00:08:55] RECOVERY - Puppet freshness on mc13 is OK: puppet ran at Fri Jun 28 00:08:46 UTC 2013 [00:08:55] RECOVERY - Puppet freshness on mw91 is OK: puppet ran at Fri Jun 28 00:08:48 UTC 2013 [00:08:55] RECOVERY - Puppet freshness on emery is OK: puppet ran at Fri Jun 28 00:08:49 UTC 2013 [00:08:55] RECOVERY - Puppet freshness on amssq50 is OK: puppet ran at Fri Jun 28 00:08:49 UTC 2013 [00:08:56] RECOVERY - Puppet freshness on ms-be1010 is OK: puppet ran at Fri Jun 28 00:08:50 UTC 2013 [00:08:57] RECOVERY - Puppet freshness on mw92 is OK: puppet ran at Fri Jun 28 00:08:50 UTC 2013 [00:08:57] RECOVERY - Puppet freshness on mw1110 is OK: puppet ran at Fri Jun 28 00:08:51 UTC 2013 [00:09:06] RECOVERY - Puppet freshness on mc1006 is OK: puppet ran at Fri Jun 28 00:08:54 UTC 2013 [00:09:07] RECOVERY - Puppet freshness on wtp1006 is OK: puppet ran at Fri Jun 28 00:08:57 UTC 2013 [00:09:15] RECOVERY - Puppet freshness on analytics1020 is OK: puppet ran at Fri Jun 28 00:09:06 UTC 2013 [00:09:15] RECOVERY - Puppet freshness on mc1002 is OK: puppet ran at Fri Jun 28 00:09:07 UTC 2013 [00:09:15] RECOVERY - Puppet freshness on mw1082 is OK: puppet ran at Fri Jun 28 00:09:11 UTC 2013 [00:09:24] RECOVERY - Puppet freshness on mw59 is OK: puppet ran at Fri Jun 28 00:09:15 UTC 2013 [00:09:24] RECOVERY - Puppet freshness on mw1145 is OK: puppet ran at Fri Jun 28 00:09:16 UTC 2013 [00:09:24] RECOVERY - Puppet freshness on wtp1016 is OK: puppet ran at Fri Jun 28 00:09:16 UTC 2013 [00:09:24] RECOVERY - Puppet freshness on mw15 is OK: puppet ran at Fri Jun 28 00:09:19 UTC 2013 [00:09:25] RECOVERY - Puppet freshness on arsenic is OK: puppet ran at Fri Jun 28 00:09:20 UTC 2013 [00:09:25] RECOVERY - Puppet freshness on cp3021 is OK: puppet ran at Fri Jun 28 00:09:21 UTC 2013 [00:09:34] RECOVERY - Puppet freshness on analytics1018 is OK: puppet ran at Fri Jun 28 00:09:23 UTC 2013 [00:09:34] RECOVERY - Puppet freshness on mw1 is OK: puppet ran at Fri Jun 28 00:09:25 UTC 2013 [00:09:34] RECOVERY - Puppet freshness on mw113 is OK: puppet ran at Fri Jun 28 00:09:26 UTC 2013 [00:09:35] RECOVERY - Puppet freshness on mw1026 is OK: puppet ran at Fri Jun 28 00:09:27 UTC 2013 [00:09:35] RECOVERY - Puppet freshness on search35 is OK: puppet ran at Fri Jun 28 00:09:30 UTC 2013 [00:09:35] RECOVERY - Puppet freshness on srv298 is OK: puppet ran at Fri Jun 28 00:09:31 UTC 2013 [00:09:45] RECOVERY - Puppet freshness on analytics1021 is OK: puppet ran at Fri Jun 28 00:09:32 UTC 2013 [00:09:45] RECOVERY - Puppet freshness on mw1174 is OK: puppet ran at Fri Jun 28 00:09:34 UTC 2013 [00:09:45] RECOVERY - Puppet freshness on mw1121 is OK: puppet ran at Fri Jun 28 00:09:35 UTC 2013 [00:09:45] RECOVERY - Puppet freshness on search23 is OK: puppet ran at Fri Jun 28 00:09:36 UTC 2013 [00:09:45] RECOVERY - Puppet freshness on mw1158 is OK: puppet ran at Fri Jun 28 00:09:37 UTC 2013 [00:09:46] RECOVERY - Puppet freshness on virt8 is OK: puppet ran at Fri Jun 28 00:09:40 UTC 2013 [00:09:46] RECOVERY - Puppet freshness on mw1108 is OK: puppet ran at Fri Jun 28 00:09:41 UTC 2013 [00:09:53] RECOVERY - Puppet freshness on srv293 is OK: puppet ran at Fri Jun 28 00:09:42 UTC 2013 [00:09:54] RECOVERY - Puppet freshness on mw1073 is OK: puppet ran at Fri Jun 28 00:09:45 UTC 2013 [00:09:54] RECOVERY - Puppet freshness on srv251 is OK: puppet ran at Fri Jun 28 00:09:45 UTC 2013 [00:09:54] RECOVERY - Puppet freshness on mw1008 is OK: puppet ran at Fri Jun 28 00:09:46 UTC 2013 [00:09:54] RECOVERY - Puppet freshness on mw1060 is OK: puppet ran at Fri Jun 28 00:09:47 UTC 2013 [00:09:54] RECOVERY - Puppet freshness on mw25 is OK: puppet ran at Fri Jun 28 00:09:47 UTC 2013 [00:09:54] RECOVERY - Puppet freshness on cp3019 is OK: puppet ran at Fri Jun 28 00:09:50 UTC 2013 [00:09:55] RECOVERY - Puppet freshness on mw1148 is OK: puppet ran at Fri Jun 28 00:09:51 UTC 2013 [00:09:55] RECOVERY - Puppet freshness on mw1199 is OK: puppet ran at Fri Jun 28 00:09:51 UTC 2013 [00:10:05] RECOVERY - Puppet freshness on mw1120 is OK: puppet ran at Fri Jun 28 00:09:52 UTC 2013 [00:10:05] RECOVERY - Puppet freshness on mw1009 is OK: puppet ran at Fri Jun 28 00:09:53 UTC 2013 [00:10:05] RECOVERY - Puppet freshness on lvs4 is OK: puppet ran at Fri Jun 28 00:09:54 UTC 2013 [00:10:05] RECOVERY - Puppet freshness on sq66 is OK: puppet ran at Fri Jun 28 00:09:54 UTC 2013 [00:10:05] RECOVERY - Puppet freshness on sq75 is OK: puppet ran at Fri Jun 28 00:09:55 UTC 2013 [00:10:06] RECOVERY - Puppet freshness on sq36 is OK: puppet ran at Fri Jun 28 00:09:56 UTC 2013 [00:10:06] RECOVERY - Puppet freshness on cp1009 is OK: puppet ran at Fri Jun 28 00:09:57 UTC 2013 [00:10:07] RECOVERY - Puppet freshness on nitrogen is OK: puppet ran at Fri Jun 28 00:09:59 UTC 2013 [00:10:07] RECOVERY - Puppet freshness on db1046 is OK: puppet ran at Fri Jun 28 00:10:00 UTC 2013 [00:10:08] RECOVERY - Puppet freshness on cp1008 is OK: puppet ran at Fri Jun 28 00:10:01 UTC 2013 [00:10:08] RECOVERY - Puppet freshness on cp1064 is OK: puppet ran at Fri Jun 28 00:10:02 UTC 2013 [00:10:13] RECOVERY - Puppet freshness on db1018 is OK: puppet ran at Fri Jun 28 00:10:02 UTC 2013 [00:10:14] RECOVERY - Puppet freshness on mw7 is OK: puppet ran at Fri Jun 28 00:10:05 UTC 2013 [00:10:14] RECOVERY - Puppet freshness on ms-fe1004 is OK: puppet ran at Fri Jun 28 00:10:06 UTC 2013 [00:10:14] RECOVERY - Puppet freshness on snapshot3 is OK: puppet ran at Fri Jun 28 00:10:06 UTC 2013 [00:10:14] RECOVERY - Puppet freshness on db1049 is OK: puppet ran at Fri Jun 28 00:10:07 UTC 2013 [00:10:14] RECOVERY - Puppet freshness on holmium is OK: puppet ran at Fri Jun 28 00:10:09 UTC 2013 [00:10:15] RECOVERY - Puppet freshness on srv235 is OK: puppet ran at Fri Jun 28 00:10:10 UTC 2013 [00:10:15] RECOVERY - Puppet freshness on srv257 is OK: puppet ran at Fri Jun 28 00:10:10 UTC 2013 [00:10:15] RECOVERY - Puppet freshness on srv274 is OK: puppet ran at Fri Jun 28 00:10:11 UTC 2013 [00:10:16] RECOVERY - Puppet freshness on lvs3 is OK: puppet ran at Fri Jun 28 00:10:12 UTC 2013 [00:10:25] RECOVERY - Puppet freshness on db1015 is OK: puppet ran at Fri Jun 28 00:10:13 UTC 2013 [00:10:26] RECOVERY - Puppet freshness on mw56 is OK: puppet ran at Fri Jun 28 00:10:13 UTC 2013 [00:10:26] RECOVERY - Puppet freshness on db64 is OK: puppet ran at Fri Jun 28 00:10:14 UTC 2013 [00:10:26] RECOVERY - Puppet freshness on ms-be10 is OK: puppet ran at Fri Jun 28 00:10:15 UTC 2013 [00:10:26] RECOVERY - Puppet freshness on sq56 is OK: puppet ran at Fri Jun 28 00:10:15 UTC 2013 [00:10:26] RECOVERY - Puppet freshness on amssq60 is OK: puppet ran at Fri Jun 28 00:10:16 UTC 2013 [00:10:26] RECOVERY - Puppet freshness on mw1102 is OK: puppet ran at Fri Jun 28 00:10:16 UTC 2013 [00:10:26] RECOVERY - Puppet freshness on mw12 is OK: puppet ran at Fri Jun 28 00:10:17 UTC 2013 [00:10:27] RECOVERY - Puppet freshness on amssq41 is OK: puppet ran at Fri Jun 28 00:10:21 UTC 2013 [00:10:27] RECOVERY - Puppet freshness on ersch is OK: puppet ran at Fri Jun 28 00:10:21 UTC 2013 [00:10:28] RECOVERY - Puppet freshness on mw1019 is OK: puppet ran at Fri Jun 28 00:10:22 UTC 2013 [00:10:33] RECOVERY - Puppet freshness on db68 is OK: puppet ran at Fri Jun 28 00:10:22 UTC 2013 [00:10:34] RECOVERY - Puppet freshness on mw48 is OK: puppet ran at Fri Jun 28 00:10:25 UTC 2013 [00:10:34] RECOVERY - Puppet freshness on mw1070 is OK: puppet ran at Fri Jun 28 00:10:26 UTC 2013 [00:10:34] RECOVERY - Puppet freshness on ms-be1005 is OK: puppet ran at Fri Jun 28 00:10:27 UTC 2013 [00:10:34] RECOVERY - Puppet freshness on amssq36 is OK: puppet ran at Fri Jun 28 00:10:27 UTC 2013 [00:10:34] RECOVERY - Puppet freshness on mw1119 is OK: puppet ran at Fri Jun 28 00:10:28 UTC 2013 [00:10:34] RECOVERY - Puppet freshness on mw3 is OK: puppet ran at Fri Jun 28 00:10:29 UTC 2013 [00:10:35] RECOVERY - Puppet freshness on db1028 is OK: puppet ran at Fri Jun 28 00:10:30 UTC 2013 [00:10:35] RECOVERY - Puppet freshness on wtp1014 is OK: puppet ran at Fri Jun 28 00:10:30 UTC 2013 [00:10:36] RECOVERY - Puppet freshness on ms-be7 is OK: puppet ran at Fri Jun 28 00:10:31 UTC 2013 [00:10:36] RECOVERY - Puppet freshness on db38 is OK: puppet ran at Fri Jun 28 00:10:31 UTC 2013 [00:10:37] RECOVERY - Puppet freshness on mc5 is OK: puppet ran at Fri Jun 28 00:10:32 UTC 2013 [00:10:45] RECOVERY - Puppet freshness on mw1085 is OK: puppet ran at Fri Jun 28 00:10:33 UTC 2013 [00:10:45] RECOVERY - Puppet freshness on mw1170 is OK: puppet ran at Fri Jun 28 00:10:34 UTC 2013 [00:10:45] RECOVERY - Puppet freshness on virt0 is OK: puppet ran at Fri Jun 28 00:10:38 UTC 2013 [00:10:45] RECOVERY - Puppet freshness on amssq32 is OK: puppet ran at Fri Jun 28 00:10:41 UTC 2013 [00:10:54] RECOVERY - Puppet freshness on mw1058 is OK: puppet ran at Fri Jun 28 00:10:42 UTC 2013 [00:10:55] RECOVERY - Puppet freshness on mw1157 is OK: puppet ran at Fri Jun 28 00:10:43 UTC 2013 [00:10:55] RECOVERY - Puppet freshness on mw1061 is OK: puppet ran at Fri Jun 28 00:10:45 UTC 2013 [00:10:55] RECOVERY - Puppet freshness on ssl1004 is OK: puppet ran at Fri Jun 28 00:10:46 UTC 2013 [00:10:55] RECOVERY - Puppet freshness on analytics1024 is OK: puppet ran at Fri Jun 28 00:10:47 UTC 2013 [00:10:55] RECOVERY - Puppet freshness on mw1184 is OK: puppet ran at Fri Jun 28 00:10:47 UTC 2013 [00:10:56] RECOVERY - Puppet freshness on mw1172 is OK: puppet ran at Fri Jun 28 00:10:48 UTC 2013 [00:10:56] RECOVERY - Puppet freshness on search1013 is OK: puppet ran at Fri Jun 28 00:10:48 UTC 2013 [00:10:57] RECOVERY - Puppet freshness on search20 is OK: puppet ran at Fri Jun 28 00:10:48 UTC 2013 [00:10:57] RECOVERY - Puppet freshness on analytics1006 is OK: puppet ran at Fri Jun 28 00:10:49 UTC 2013 [00:10:58] RECOVERY - Puppet freshness on db1003 is OK: puppet ran at Fri Jun 28 00:10:49 UTC 2013 [00:10:58] RECOVERY - Puppet freshness on hooper is OK: puppet ran at Fri Jun 28 00:10:49 UTC 2013 [00:10:59] RECOVERY - Puppet freshness on mw120 is OK: puppet ran at Fri Jun 28 00:10:50 UTC 2013 [00:10:59] RECOVERY - Puppet freshness on srv247 is OK: puppet ran at Fri Jun 28 00:10:50 UTC 2013 [00:11:00] RECOVERY - Puppet freshness on srv252 is OK: puppet ran at Fri Jun 28 00:10:50 UTC 2013 [00:11:00] RECOVERY - Puppet freshness on srv239 is OK: puppet ran at Fri Jun 28 00:10:51 UTC 2013 [00:11:01] RECOVERY - Puppet freshness on amssq58 is OK: puppet ran at Fri Jun 28 00:10:51 UTC 2013 [00:11:06] RECOVERY - Puppet freshness on search1009 is OK: puppet ran at Fri Jun 28 00:10:52 UTC 2013 [00:11:06] RECOVERY - Puppet freshness on mw78 is OK: puppet ran at Fri Jun 28 00:10:55 UTC 2013 [00:11:06] RECOVERY - Puppet freshness on db37 is OK: puppet ran at Fri Jun 28 00:10:56 UTC 2013 [00:11:07] RECOVERY - Puppet freshness on ms-fe4 is OK: puppet ran at Fri Jun 28 00:11:01 UTC 2013 [00:11:07] RECOVERY - Puppet freshness on db1041 is OK: puppet ran at Fri Jun 28 00:11:02 UTC 2013 [00:11:16] RECOVERY - Puppet freshness on es7 is OK: puppet ran at Fri Jun 28 00:11:02 UTC 2013 [00:11:16] RECOVERY - Puppet freshness on mw1177 is OK: puppet ran at Fri Jun 28 00:11:05 UTC 2013 [00:11:16] RECOVERY - Puppet freshness on mw1191 is OK: puppet ran at Fri Jun 28 00:11:06 UTC 2013 [00:11:16] RECOVERY - Puppet freshness on mw67 is OK: puppet ran at Fri Jun 28 00:11:07 UTC 2013 [00:11:17] RECOVERY - Puppet freshness on tmh1002 is OK: puppet ran at Fri Jun 28 00:11:07 UTC 2013 [00:11:17] RECOVERY - Puppet freshness on gallium is OK: puppet ran at Fri Jun 28 00:11:11 UTC 2013 [00:11:17] RECOVERY - Puppet freshness on es1007 is OK: puppet ran at Fri Jun 28 00:11:12 UTC 2013 [00:11:23] RECOVERY - Puppet freshness on mw1039 is OK: puppet ran at Fri Jun 28 00:11:13 UTC 2013 [00:11:24] RECOVERY - Puppet freshness on virt10 is OK: puppet ran at Fri Jun 28 00:11:16 UTC 2013 [00:11:24] RECOVERY - Puppet freshness on wtp1005 is OK: puppet ran at Fri Jun 28 00:11:17 UTC 2013 [00:11:24] RECOVERY - Puppet freshness on sq57 is OK: puppet ran at Fri Jun 28 00:11:20 UTC 2013 [00:11:24] RECOVERY - Puppet freshness on silver is OK: puppet ran at Fri Jun 28 00:11:21 UTC 2013 [00:11:24] RECOVERY - Puppet freshness on sq53 is OK: puppet ran at Fri Jun 28 00:11:21 UTC 2013 [00:11:24] RECOVERY - Puppet freshness on db1007 is OK: puppet ran at Fri Jun 28 00:11:22 UTC 2013 [00:11:35] RECOVERY - Puppet freshness on es3 is OK: puppet ran at Fri Jun 28 00:11:23 UTC 2013 [00:11:36] RECOVERY - Puppet freshness on sq44 is OK: puppet ran at Fri Jun 28 00:11:23 UTC 2013 [00:11:36] RECOVERY - Puppet freshness on db50 is OK: puppet ran at Fri Jun 28 00:11:23 UTC 2013 [00:11:36] RECOVERY - Puppet freshness on db1043 is OK: puppet ran at Fri Jun 28 00:11:24 UTC 2013 [00:11:36] RECOVERY - Puppet freshness on search26 is OK: puppet ran at Fri Jun 28 00:11:24 UTC 2013 [00:11:37] RECOVERY - Puppet freshness on lvs2 is OK: puppet ran at Fri Jun 28 00:11:26 UTC 2013 [00:11:38] RECOVERY - Puppet freshness on ssl3001 is OK: puppet ran at Fri Jun 28 00:11:26 UTC 2013 [00:11:38] RECOVERY - Puppet freshness on wtp1012 is OK: puppet ran at Fri Jun 28 00:11:27 UTC 2013 [00:11:39] RECOVERY - Puppet freshness on mw1203 is OK: puppet ran at Fri Jun 28 00:11:28 UTC 2013 [00:11:39] RECOVERY - Puppet freshness on nfs2 is OK: puppet ran at Fri Jun 28 00:11:30 UTC 2013 [00:11:43] RECOVERY - Puppet freshness on analytics1016 is OK: puppet ran at Fri Jun 28 00:11:32 UTC 2013 [00:11:43] RECOVERY - Puppet freshness on mc1010 is OK: puppet ran at Fri Jun 28 00:11:33 UTC 2013 [00:11:44] RECOVERY - Puppet freshness on snapshot1001 is OK: puppet ran at Fri Jun 28 00:11:38 UTC 2013 [00:11:44] RECOVERY - Puppet freshness on cp1019 is OK: puppet ran at Fri Jun 28 00:11:38 UTC 2013 [00:11:44] RECOVERY - Puppet freshness on db1039 is OK: puppet ran at Fri Jun 28 00:11:39 UTC 2013 [00:11:44] RECOVERY - Puppet freshness on mw17 is OK: puppet ran at Fri Jun 28 00:11:40 UTC 2013 [00:11:44] RECOVERY - Puppet freshness on mw80 is OK: puppet ran at Fri Jun 28 00:11:41 UTC 2013 [00:11:45] RECOVERY - Puppet freshness on mw1149 is OK: puppet ran at Fri Jun 28 00:11:42 UTC 2013 [00:11:53] RECOVERY - Puppet freshness on mw1195 is OK: puppet ran at Fri Jun 28 00:11:42 UTC 2013 [00:11:54] RECOVERY - Puppet freshness on mw1035 is OK: puppet ran at Fri Jun 28 00:11:44 UTC 2013 [00:11:54] RECOVERY - Puppet freshness on mw69 is OK: puppet ran at Fri Jun 28 00:11:45 UTC 2013 [00:11:54] RECOVERY - Puppet freshness on mw1036 is OK: puppet ran at Fri Jun 28 00:11:46 UTC 2013 [00:11:54] RECOVERY - Puppet freshness on es1006 is OK: puppet ran at Fri Jun 28 00:11:47 UTC 2013 [00:11:54] RECOVERY - Puppet freshness on db56 is OK: puppet ran at Fri Jun 28 00:11:48 UTC 2013 [00:11:54] RECOVERY - Puppet freshness on searchidx2 is OK: puppet ran at Fri Jun 28 00:11:48 UTC 2013 [00:11:55] RECOVERY - Puppet freshness on mw1013 is OK: puppet ran at Fri Jun 28 00:11:51 UTC 2013 [00:11:55] RECOVERY - Puppet freshness on cp1044 is OK: puppet ran at Fri Jun 28 00:11:51 UTC 2013 [00:12:04] RECOVERY - Puppet freshness on mw1167 is OK: puppet ran at Fri Jun 28 00:11:52 UTC 2013 [00:12:05] RECOVERY - Puppet freshness on pdf1 is OK: puppet ran at Fri Jun 28 00:11:53 UTC 2013 [00:12:05] RECOVERY - Puppet freshness on srv254 is OK: puppet ran at Fri Jun 28 00:11:54 UTC 2013 [00:12:05] RECOVERY - Puppet freshness on mw84 is OK: puppet ran at Fri Jun 28 00:11:55 UTC 2013 [00:12:05] RECOVERY - Puppet freshness on mw71 is OK: puppet ran at Fri Jun 28 00:11:55 UTC 2013 [00:12:05] RECOVERY - Puppet freshness on mw1192 is OK: puppet ran at Fri Jun 28 00:11:56 UTC 2013 [00:12:05] RECOVERY - Puppet freshness on search1002 is OK: puppet ran at Fri Jun 28 00:11:56 UTC 2013 [00:12:05] RECOVERY - Puppet freshness on cp1011 is OK: puppet ran at Fri Jun 28 00:11:57 UTC 2013 [00:12:06] RECOVERY - Puppet freshness on mw18 is OK: puppet ran at Fri Jun 28 00:11:58 UTC 2013 [00:12:07] RECOVERY - Puppet freshness on mw1076 is OK: puppet ran at Fri Jun 28 00:11:58 UTC 2013 [00:12:07] RECOVERY - Puppet freshness on mw1014 is OK: puppet ran at Fri Jun 28 00:12:00 UTC 2013 [00:12:08] RECOVERY - Puppet freshness on mw1051 is OK: puppet ran at Fri Jun 28 00:12:02 UTC 2013 [00:12:13] RECOVERY - Puppet freshness on mw1216 is OK: puppet ran at Fri Jun 28 00:12:03 UTC 2013 [00:12:14] RECOVERY - Puppet freshness on mw1168 is OK: puppet ran at Fri Jun 28 00:12:03 UTC 2013 [00:12:14] RECOVERY - Puppet freshness on mw1133 is OK: puppet ran at Fri Jun 28 00:12:07 UTC 2013 [00:12:14] RECOVERY - Puppet freshness on ssl3003 is OK: puppet ran at Fri Jun 28 00:12:07 UTC 2013 [00:12:14] RECOVERY - Puppet freshness on mw1109 is OK: puppet ran at Fri Jun 28 00:12:08 UTC 2013 [00:12:14] RECOVERY - Puppet freshness on mw1202 is OK: puppet ran at Fri Jun 28 00:12:09 UTC 2013 [00:12:14] RECOVERY - Puppet freshness on mw1098 is OK: puppet ran at Fri Jun 28 00:12:10 UTC 2013 [00:12:15] RECOVERY - Puppet freshness on cp1004 is OK: puppet ran at Fri Jun 28 00:12:10 UTC 2013 [00:12:15] RECOVERY - Puppet freshness on rdb1001 is OK: puppet ran at Fri Jun 28 00:12:11 UTC 2013 [00:12:16] RECOVERY - Puppet freshness on mc1004 is OK: puppet ran at Fri Jun 28 00:12:12 UTC 2013 [00:12:23] RECOVERY - Puppet freshness on lvs1003 is OK: puppet ran at Fri Jun 28 00:12:12 UTC 2013 [00:12:23] RECOVERY - Puppet freshness on cp3022 is OK: puppet ran at Fri Jun 28 00:12:13 UTC 2013 [00:12:23] RECOVERY - Puppet freshness on db74 is OK: puppet ran at Fri Jun 28 00:12:14 UTC 2013 [00:12:23] RECOVERY - Puppet freshness on sq69 is OK: puppet ran at Fri Jun 28 00:12:17 UTC 2013 [00:12:23] RECOVERY - Puppet freshness on srv243 is OK: puppet ran at Fri Jun 28 00:12:17 UTC 2013 [00:12:24] RECOVERY - Puppet freshness on iron is OK: puppet ran at Fri Jun 28 00:12:18 UTC 2013 [00:12:24] RECOVERY - Puppet freshness on ms-be1008 is OK: puppet ran at Fri Jun 28 00:12:19 UTC 2013 [00:12:25] RECOVERY - Puppet freshness on formey is OK: puppet ran at Fri Jun 28 00:12:20 UTC 2013 [00:12:25] RECOVERY - Puppet freshness on sq70 is OK: puppet ran at Fri Jun 28 00:12:21 UTC 2013 [00:12:25] RECOVERY - Puppet freshness on virt1007 is OK: puppet ran at Fri Jun 28 00:12:22 UTC 2013 [00:12:33] RECOVERY - Puppet freshness on db72 is OK: puppet ran at Fri Jun 28 00:12:23 UTC 2013 [00:12:34] RECOVERY - Puppet freshness on db1030 is OK: puppet ran at Fri Jun 28 00:12:23 UTC 2013 [00:12:34] RECOVERY - Puppet freshness on ssl3002 is OK: puppet ran at Fri Jun 28 00:12:24 UTC 2013 [00:12:34] RECOVERY - Puppet freshness on db1016 is OK: puppet ran at Fri Jun 28 00:12:25 UTC 2013 [00:12:34] RECOVERY - Puppet freshness on oxygen is OK: puppet ran at Fri Jun 28 00:12:26 UTC 2013 [00:12:34] RECOVERY - Puppet freshness on amslvs3 is OK: puppet ran at Fri Jun 28 00:12:27 UTC 2013 [00:12:34] RECOVERY - Puppet freshness on virt7 is OK: puppet ran at Fri Jun 28 00:12:28 UTC 2013 [00:12:35] RECOVERY - Puppet freshness on search24 is OK: puppet ran at Fri Jun 28 00:12:28 UTC 2013 [00:12:35] RECOVERY - Puppet freshness on analytics1026 is OK: puppet ran at Fri Jun 28 00:12:29 UTC 2013 [00:12:36] RECOVERY - Puppet freshness on db49 is OK: puppet ran at Fri Jun 28 00:12:30 UTC 2013 [00:12:37] RECOVERY - Puppet freshness on mw72 is OK: puppet ran at Fri Jun 28 00:12:31 UTC 2013 [00:12:37] RECOVERY - Puppet freshness on mw88 is OK: puppet ran at Fri Jun 28 00:12:31 UTC 2013 [00:12:43] RECOVERY - Puppet freshness on db1053 is OK: puppet ran at Fri Jun 28 00:12:32 UTC 2013 [00:12:44] RECOVERY - Puppet freshness on db1059 is OK: puppet ran at Fri Jun 28 00:12:33 UTC 2013 [00:12:44] RECOVERY - Puppet freshness on search32 is OK: puppet ran at Fri Jun 28 00:12:33 UTC 2013 [00:12:44] RECOVERY - Puppet freshness on ms1002 is OK: puppet ran at Fri Jun 28 00:12:33 UTC 2013 [00:12:44] RECOVERY - Puppet freshness on mw1151 is OK: puppet ran at Fri Jun 28 00:12:36 UTC 2013 [00:12:44] RECOVERY - Puppet freshness on amssq62 is OK: puppet ran at Fri Jun 28 00:12:36 UTC 2013 [00:12:44] RECOVERY - Puppet freshness on mw1079 is OK: puppet ran at Fri Jun 28 00:12:37 UTC 2013 [00:12:45] RECOVERY - Puppet freshness on amssq46 is OK: puppet ran at Fri Jun 28 00:12:38 UTC 2013 [00:12:46] RECOVERY - Puppet freshness on mw4 is OK: puppet ran at Fri Jun 28 00:12:39 UTC 2013 [00:12:46] RECOVERY - Puppet freshness on mw32 is OK: puppet ran at Fri Jun 28 00:12:39 UTC 2013 [00:12:46] RECOVERY - Puppet freshness on mw86 is OK: puppet ran at Fri Jun 28 00:12:40 UTC 2013 [00:12:47] RECOVERY - Puppet freshness on hooft is OK: puppet ran at Fri Jun 28 00:12:40 UTC 2013 [00:12:47] RECOVERY - Puppet freshness on srv263 is OK: puppet ran at Fri Jun 28 00:12:42 UTC 2013 [00:12:53] RECOVERY - Puppet freshness on cp1055 is OK: puppet ran at Fri Jun 28 00:12:42 UTC 2013 [00:12:54] RECOVERY - Puppet freshness on es8 is OK: puppet ran at Fri Jun 28 00:12:48 UTC 2013 [00:12:54] RECOVERY - Puppet freshness on cp1041 is OK: puppet ran at Fri Jun 28 00:12:48 UTC 2013 [00:12:54] RECOVERY - Puppet freshness on mw123 is OK: puppet ran at Fri Jun 28 00:12:49 UTC 2013 [00:12:54] RECOVERY - Puppet freshness on cp1027 is OK: puppet ran at Fri Jun 28 00:12:49 UTC 2013 [00:13:03] RECOVERY - Puppet freshness on mw1183 is OK: puppet ran at Fri Jun 28 00:12:52 UTC 2013 [00:13:04] RECOVERY - Puppet freshness on virt5 is OK: puppet ran at Fri Jun 28 00:12:53 UTC 2013 [00:13:04] RECOVERY - Puppet freshness on wtp1022 is OK: puppet ran at Fri Jun 28 00:12:54 UTC 2013 [00:13:04] RECOVERY - Puppet freshness on wtp1023 is OK: puppet ran at Fri Jun 28 00:12:54 UTC 2013 [00:13:04] RECOVERY - Puppet freshness on mw1034 is OK: puppet ran at Fri Jun 28 00:12:55 UTC 2013 [00:13:04] RECOVERY - Puppet freshness on mw85 is OK: puppet ran at Fri Jun 28 00:12:56 UTC 2013 [00:13:05] RECOVERY - Puppet freshness on mw1050 is OK: puppet ran at Fri Jun 28 00:12:56 UTC 2013 [00:13:05] RECOVERY - Puppet freshness on virt6 is OK: puppet ran at Fri Jun 28 00:12:56 UTC 2013 [00:13:06] RECOVERY - Puppet freshness on mw1156 is OK: puppet ran at Fri Jun 28 00:12:57 UTC 2013 [00:13:06] RECOVERY - Puppet freshness on dysprosium is OK: puppet ran at Fri Jun 28 00:12:59 UTC 2013 [00:13:07] RECOVERY - Puppet freshness on ssl3 is OK: puppet ran at Fri Jun 28 00:13:00 UTC 2013 [00:13:07] RECOVERY - Puppet freshness on snapshot1004 is OK: puppet ran at Fri Jun 28 00:13:02 UTC 2013 [00:13:13] RECOVERY - Puppet freshness on cp1033 is OK: puppet ran at Fri Jun 28 00:13:02 UTC 2013 [00:13:13] RECOVERY - Puppet freshness on mw102 is OK: puppet ran at Fri Jun 28 00:13:03 UTC 2013 [00:13:14] RECOVERY - Puppet freshness on srv279 is OK: puppet ran at Fri Jun 28 00:13:04 UTC 2013 [00:13:14] RECOVERY - Puppet freshness on search1016 is OK: puppet ran at Fri Jun 28 00:13:05 UTC 2013 [00:13:14] RECOVERY - Puppet freshness on sq80 is OK: puppet ran at Fri Jun 28 00:13:06 UTC 2013 [00:13:14] RECOVERY - Puppet freshness on mw1212 is OK: puppet ran at Fri Jun 28 00:13:06 UTC 2013 [00:13:14] RECOVERY - Puppet freshness on db1008 is OK: puppet ran at Fri Jun 28 00:13:06 UTC 2013 [00:13:14] RECOVERY - Puppet freshness on rdb1004 is OK: puppet ran at Fri Jun 28 00:13:07 UTC 2013 [00:13:15] RECOVERY - Puppet freshness on cp1052 is OK: puppet ran at Fri Jun 28 00:13:07 UTC 2013 [00:13:16] RECOVERY - Puppet freshness on db78 is OK: puppet ran at Fri Jun 28 00:13:07 UTC 2013 [00:13:16] RECOVERY - Puppet freshness on cp1042 is OK: puppet ran at Fri Jun 28 00:13:08 UTC 2013 [00:13:16] RECOVERY - Puppet freshness on hydrogen is OK: puppet ran at Fri Jun 28 00:13:08 UTC 2013 [00:13:17] RECOVERY - Puppet freshness on db35 is OK: puppet ran at Fri Jun 28 00:13:09 UTC 2013 [00:13:17] RECOVERY - Puppet freshness on db1006 is OK: puppet ran at Fri Jun 28 00:13:09 UTC 2013 [00:13:23] RECOVERY - Puppet freshness on srv288 is OK: puppet ran at Fri Jun 28 00:13:13 UTC 2013 [00:13:23] RECOVERY - Puppet freshness on srv241 is OK: puppet ran at Fri Jun 28 00:13:14 UTC 2013 [00:13:24] RECOVERY - Puppet freshness on mw27 is OK: puppet ran at Fri Jun 28 00:13:15 UTC 2013 [00:13:24] RECOVERY - Puppet freshness on search1017 is OK: puppet ran at Fri Jun 28 00:13:16 UTC 2013 [00:13:24] RECOVERY - Puppet freshness on mw1163 is OK: puppet ran at Fri Jun 28 00:13:16 UTC 2013 [00:13:24] RECOVERY - Puppet freshness on ms-be1001 is OK: puppet ran at Fri Jun 28 00:13:17 UTC 2013 [00:13:24] RECOVERY - Puppet freshness on mw6 is OK: puppet ran at Fri Jun 28 00:13:17 UTC 2013 [00:13:24] RECOVERY - Puppet freshness on db62 is OK: puppet ran at Fri Jun 28 00:13:17 UTC 2013 [00:13:25] RECOVERY - Puppet freshness on ms-be1011 is OK: puppet ran at Fri Jun 28 00:13:18 UTC 2013 [00:13:33] RECOVERY - Puppet freshness on ms1001 is OK: puppet ran at Fri Jun 28 00:13:23 UTC 2013 [00:13:33] RECOVERY - Puppet freshness on pc3 is OK: puppet ran at Fri Jun 28 00:13:23 UTC 2013 [00:13:34] RECOVERY - Puppet freshness on es1010 is OK: puppet ran at Fri Jun 28 00:13:26 UTC 2013 [00:13:34] RECOVERY - Puppet freshness on db1047 is OK: puppet ran at Fri Jun 28 00:13:27 UTC 2013 [00:13:34] RECOVERY - Puppet freshness on mw1077 is OK: puppet ran at Fri Jun 28 00:13:29 UTC 2013 [00:13:34] RECOVERY - Puppet freshness on cp1059 is OK: puppet ran at Fri Jun 28 00:13:29 UTC 2013 [00:13:34] RECOVERY - Puppet freshness on mw44 is OK: puppet ran at Fri Jun 28 00:13:30 UTC 2013 [00:13:43] RECOVERY - Puppet freshness on mw1023 is OK: puppet ran at Fri Jun 28 00:13:33 UTC 2013 [00:13:44] RECOVERY - Puppet freshness on mw1002 is OK: puppet ran at Fri Jun 28 00:13:34 UTC 2013 [00:13:44] RECOVERY - Puppet freshness on locke is OK: puppet ran at Fri Jun 28 00:13:34 UTC 2013 [00:13:44] RECOVERY - Puppet freshness on mw1029 is OK: puppet ran at Fri Jun 28 00:13:35 UTC 2013 [00:13:44] RECOVERY - Puppet freshness on mw1097 is OK: puppet ran at Fri Jun 28 00:13:38 UTC 2013 [00:13:44] RECOVERY - Puppet freshness on mw1096 is OK: puppet ran at Fri Jun 28 00:13:38 UTC 2013 [00:13:44] RECOVERY - Puppet freshness on sq60 is OK: puppet ran at Fri Jun 28 00:13:39 UTC 2013 [00:13:45] RECOVERY - Puppet freshness on pc1001 is OK: puppet ran at Fri Jun 28 00:13:39 UTC 2013 [00:13:45] RECOVERY - Puppet freshness on search1020 is OK: puppet ran at Fri Jun 28 00:13:40 UTC 2013 [00:13:46] RECOVERY - Puppet freshness on mc9 is OK: puppet ran at Fri Jun 28 00:13:40 UTC 2013 [00:13:46] RECOVERY - Puppet freshness on amssq33 is OK: puppet ran at Fri Jun 28 00:13:41 UTC 2013 [00:13:47] RECOVERY - Puppet freshness on amssq38 is OK: puppet ran at Fri Jun 28 00:13:42 UTC 2013 [00:13:53] RECOVERY - Puppet freshness on amssq49 is OK: puppet ran at Fri Jun 28 00:13:42 UTC 2013 [00:13:53] RECOVERY - Puppet freshness on amssq42 is OK: puppet ran at Fri Jun 28 00:13:43 UTC 2013 [00:13:53] RECOVERY - Puppet freshness on ms-be6 is OK: puppet ran at Fri Jun 28 00:13:43 UTC 2013 [00:13:54] RECOVERY - Puppet freshness on carbon is OK: puppet ran at Fri Jun 28 00:13:44 UTC 2013 [00:13:54] RECOVERY - Puppet freshness on wtp1019 is OK: puppet ran at Fri Jun 28 00:13:45 UTC 2013 [00:13:54] RECOVERY - Puppet freshness on cp1014 is OK: puppet ran at Fri Jun 28 00:13:44 UTC 2013 [00:13:54] RECOVERY - Puppet freshness on db47 is OK: puppet ran at Fri Jun 28 00:13:47 UTC 2013 [00:13:55] RECOVERY - Puppet freshness on wtp1017 is OK: puppet ran at Fri Jun 28 00:13:49 UTC 2013 [00:13:55] RECOVERY - Puppet freshness on cp1031 is OK: puppet ran at Fri Jun 28 00:13:51 UTC 2013 [00:13:56] RECOVERY - Puppet freshness on ms-fe1 is OK: puppet ran at Fri Jun 28 00:13:51 UTC 2013 [00:14:03] RECOVERY - Puppet freshness on cp1022 is OK: puppet ran at Fri Jun 28 00:13:52 UTC 2013 [00:14:04] RECOVERY - Puppet freshness on cp1029 is OK: puppet ran at Fri Jun 28 00:13:53 UTC 2013 [00:14:04] RECOVERY - Puppet freshness on db71 is OK: puppet ran at Fri Jun 28 00:13:54 UTC 2013 [00:14:04] RECOVERY - Puppet freshness on db1017 is OK: puppet ran at Fri Jun 28 00:13:55 UTC 2013 [00:14:04] RECOVERY - Puppet freshness on cp1070 is OK: puppet ran at Fri Jun 28 00:13:55 UTC 2013 [00:14:04] RECOVERY - Puppet freshness on mw1067 is OK: puppet ran at Fri Jun 28 00:13:56 UTC 2013 [00:14:05] RECOVERY - Puppet freshness on mw1209 is OK: puppet ran at Fri Jun 28 00:13:57 UTC 2013 [00:14:05] RECOVERY - Puppet freshness on mw89 is OK: puppet ran at Fri Jun 28 00:13:58 UTC 2013 [00:14:06] RECOVERY - Puppet freshness on mw1005 is OK: puppet ran at Fri Jun 28 00:13:59 UTC 2013 [00:14:06] RECOVERY - Puppet freshness on mw1028 is OK: puppet ran at Fri Jun 28 00:14:00 UTC 2013 [00:14:07] RECOVERY - Puppet freshness on srv281 is OK: puppet ran at Fri Jun 28 00:14:00 UTC 2013 [00:14:13] RECOVERY - Puppet freshness on cp1040 is OK: puppet ran at Fri Jun 28 00:14:03 UTC 2013 [00:14:14] RECOVERY - Puppet freshness on wtp1009 is OK: puppet ran at Fri Jun 28 00:14:04 UTC 2013 [00:14:14] RECOVERY - Puppet freshness on mw1016 is OK: puppet ran at Fri Jun 28 00:14:04 UTC 2013 [00:14:14] RECOVERY - Puppet freshness on sq61 is OK: puppet ran at Fri Jun 28 00:14:05 UTC 2013 [00:14:14] RECOVERY - Puppet freshness on mw96 is OK: puppet ran at Fri Jun 28 00:14:07 UTC 2013 [00:14:14] RECOVERY - Puppet freshness on mw21 is OK: puppet ran at Fri Jun 28 00:14:08 UTC 2013 [00:14:14] RECOVERY - Puppet freshness on yvon is OK: puppet ran at Fri Jun 28 00:14:09 UTC 2013 [00:14:15] RECOVERY - Puppet freshness on cp1067 is OK: puppet ran at Fri Jun 28 00:14:09 UTC 2013 [00:14:15] RECOVERY - Puppet freshness on search15 is OK: puppet ran at Fri Jun 28 00:14:10 UTC 2013 [00:14:16] RECOVERY - Puppet freshness on es1001 is OK: puppet ran at Fri Jun 28 00:14:10 UTC 2013 [00:14:23] RECOVERY - Puppet freshness on mw1152 is OK: puppet ran at Fri Jun 28 00:14:13 UTC 2013 [00:14:24] RECOVERY - Puppet freshness on cp1032 is OK: puppet ran at Fri Jun 28 00:14:13 UTC 2013 [00:14:24] RECOVERY - Puppet freshness on amssq45 is OK: puppet ran at Fri Jun 28 00:14:14 UTC 2013 [00:14:24] RECOVERY - Puppet freshness on amssq61 is OK: puppet ran at Fri Jun 28 00:14:14 UTC 2013 [00:14:24] RECOVERY - Puppet freshness on nescio is OK: puppet ran at Fri Jun 28 00:14:15 UTC 2013 [00:14:24] RECOVERY - Puppet freshness on amssq39 is OK: puppet ran at Fri Jun 28 00:14:15 UTC 2013 [00:14:24] RECOVERY - Puppet freshness on tridge is OK: puppet ran at Fri Jun 28 00:14:16 UTC 2013 [00:14:25] RECOVERY - Puppet freshness on kaulen is OK: puppet ran at Fri Jun 28 00:14:18 UTC 2013 [00:14:25] RECOVERY - Puppet freshness on zhen is OK: puppet ran at Fri Jun 28 00:14:19 UTC 2013 [00:14:33] RECOVERY - Puppet freshness on xenon is OK: puppet ran at Fri Jun 28 00:14:24 UTC 2013 [00:14:34] RECOVERY - Puppet freshness on srv236 is OK: puppet ran at Fri Jun 28 00:14:25 UTC 2013 [00:14:34] RECOVERY - Puppet freshness on ssl2 is OK: puppet ran at Fri Jun 28 00:14:28 UTC 2013 [00:14:34] RECOVERY - Puppet freshness on db33 is OK: puppet ran at Fri Jun 28 00:14:29 UTC 2013 [00:14:34] RECOVERY - Puppet freshness on srv248 is OK: puppet ran at Fri Jun 28 00:14:30 UTC 2013 [00:14:34] RECOVERY - Puppet freshness on mw1193 is OK: puppet ran at Fri Jun 28 00:14:31 UTC 2013 [00:14:34] RECOVERY - Puppet freshness on mw22 is OK: puppet ran at Fri Jun 28 00:14:32 UTC 2013 [00:14:43] RECOVERY - Puppet freshness on srv265 is OK: puppet ran at Fri Jun 28 00:14:32 UTC 2013 [00:14:44] RECOVERY - Puppet freshness on search34 is OK: puppet ran at Fri Jun 28 00:14:33 UTC 2013 [00:14:44] RECOVERY - Puppet freshness on cp1045 is OK: puppet ran at Fri Jun 28 00:14:34 UTC 2013 [00:14:44] RECOVERY - Puppet freshness on cp1043 is OK: puppet ran at Fri Jun 28 00:14:35 UTC 2013 [00:14:44] RECOVERY - Puppet freshness on cp1047 is OK: puppet ran at Fri Jun 28 00:14:35 UTC 2013 [00:14:44] RECOVERY - Puppet freshness on snapshot2 is OK: puppet ran at Fri Jun 28 00:14:36 UTC 2013 [00:14:44] RECOVERY - Puppet freshness on srv256 is OK: puppet ran at Fri Jun 28 00:14:37 UTC 2013 [00:14:45] RECOVERY - Puppet freshness on cp3020 is OK: puppet ran at Fri Jun 28 00:14:38 UTC 2013 [00:14:45] RECOVERY - Puppet freshness on mw31 is OK: puppet ran at Fri Jun 28 00:14:38 UTC 2013 [00:14:46] RECOVERY - Puppet freshness on cp3008 is OK: puppet ran at Fri Jun 28 00:14:39 UTC 2013 [00:14:46] RECOVERY - Puppet freshness on mw1071 is OK: puppet ran at Fri Jun 28 00:14:40 UTC 2013 [00:14:47] RECOVERY - Puppet freshness on mc1014 is OK: puppet ran at Fri Jun 28 00:14:41 UTC 2013 [00:14:53] RECOVERY - Puppet freshness on mw101 is OK: puppet ran at Fri Jun 28 00:14:43 UTC 2013 [00:14:54] RECOVERY - Puppet freshness on snapshot1003 is OK: puppet ran at Fri Jun 28 00:14:44 UTC 2013 [00:14:54] RECOVERY - Puppet freshness on analytics1025 is OK: puppet ran at Fri Jun 28 00:14:45 UTC 2013 [00:14:54] RECOVERY - Puppet freshness on mw1064 is OK: puppet ran at Fri Jun 28 00:14:45 UTC 2013 [00:14:54] RECOVERY - Puppet freshness on mw108 is OK: puppet ran at Fri Jun 28 00:14:46 UTC 2013 [00:14:54] RECOVERY - Puppet freshness on mw103 is OK: puppet ran at Fri Jun 28 00:14:47 UTC 2013 [00:14:55] RECOVERY - Puppet freshness on mw24 is OK: puppet ran at Fri Jun 28 00:14:48 UTC 2013 [00:14:55] RECOVERY - Puppet freshness on mw54 is OK: puppet ran at Fri Jun 28 00:14:48 UTC 2013 [00:14:56] RECOVERY - Puppet freshness on virt9 is OK: puppet ran at Fri Jun 28 00:14:49 UTC 2013 [00:14:56] RECOVERY - Puppet freshness on db1037 is OK: puppet ran at Fri Jun 28 00:14:50 UTC 2013 [00:14:57] RECOVERY - Puppet freshness on cp3004 is OK: puppet ran at Fri Jun 28 00:14:51 UTC 2013 [00:15:04] RECOVERY - Puppet freshness on db1035 is OK: puppet ran at Fri Jun 28 00:14:53 UTC 2013 [00:15:05] RECOVERY - Puppet freshness on mc1015 is OK: puppet ran at Fri Jun 28 00:14:54 UTC 2013 [00:15:05] RECOVERY - Puppet freshness on maerlant is OK: puppet ran at Fri Jun 28 00:14:55 UTC 2013 [00:15:05] RECOVERY - Puppet freshness on mw94 is OK: puppet ran at Fri Jun 28 00:14:55 UTC 2013 [00:15:05] RECOVERY - Puppet freshness on cp3003 is OK: puppet ran at Fri Jun 28 00:14:56 UTC 2013 [00:15:05] RECOVERY - Puppet freshness on db52 is OK: puppet ran at Fri Jun 28 00:14:58 UTC 2013 [00:15:06] RECOVERY - Puppet freshness on mw1055 is OK: puppet ran at Fri Jun 28 00:14:58 UTC 2013 [00:15:06] RECOVERY - Puppet freshness on strontium is OK: puppet ran at Fri Jun 28 00:15:01 UTC 2013 [00:15:14] RECOVERY - Puppet freshness on sq67 is OK: puppet ran at Fri Jun 28 00:15:05 UTC 2013 [00:15:14] RECOVERY - Puppet freshness on mw1160 is OK: puppet ran at Fri Jun 28 00:15:06 UTC 2013 [00:15:15] RECOVERY - Puppet freshness on mw1006 is OK: puppet ran at Fri Jun 28 00:15:08 UTC 2013 [00:15:15] RECOVERY - Puppet freshness on mw1012 is OK: puppet ran at Fri Jun 28 00:15:09 UTC 2013 [00:15:15] RECOVERY - Puppet freshness on mw73 is OK: puppet ran at Fri Jun 28 00:15:10 UTC 2013 [00:15:15] RECOVERY - Puppet freshness on mw1207 is OK: puppet ran at Fri Jun 28 00:15:11 UTC 2013 [00:15:35] RECOVERY - Puppet freshness on search28 is OK: puppet ran at Fri Jun 28 00:15:14 UTC 2013 [00:15:35] RECOVERY - Puppet freshness on cp3005 is OK: puppet ran at Fri Jun 28 00:15:15 UTC 2013 [00:15:35] RECOVERY - Puppet freshness on mw1187 is OK: puppet ran at Fri Jun 28 00:15:15 UTC 2013 [00:15:35] RECOVERY - Puppet freshness on mw105 is OK: puppet ran at Fri Jun 28 00:15:16 UTC 2013 [00:15:35] RECOVERY - Puppet freshness on mw1113 is OK: puppet ran at Fri Jun 28 00:15:17 UTC 2013 [00:15:35] RECOVERY - Puppet freshness on search1010 is OK: puppet ran at Fri Jun 28 00:15:18 UTC 2013 [00:15:35] RECOVERY - Puppet freshness on srv295 is OK: puppet ran at Fri Jun 28 00:15:19 UTC 2013 [00:15:35] RECOVERY - Puppet freshness on sq86 is OK: puppet ran at Fri Jun 28 00:15:20 UTC 2013 [00:15:35] RECOVERY - Puppet freshness on sq71 is OK: puppet ran at Fri Jun 28 00:15:20 UTC 2013 [00:15:35] RECOVERY - Puppet freshness on sq83 is OK: puppet ran at Fri Jun 28 00:15:20 UTC 2013 [00:15:35] RECOVERY - Puppet freshness on mc1003 is OK: puppet ran at Fri Jun 28 00:15:21 UTC 2013 [00:15:35] RECOVERY - Puppet freshness on pc2 is OK: puppet ran at Fri Jun 28 00:15:21 UTC 2013 [00:15:35] RECOVERY - Puppet freshness on tmh2 is OK: puppet ran at Fri Jun 28 00:15:22 UTC 2013 [00:15:35] RECOVERY - Puppet freshness on mw1176 is OK: puppet ran at Fri Jun 28 00:15:22 UTC 2013 [00:15:35] RECOVERY - Puppet freshness on cp1007 is OK: puppet ran at Fri Jun 28 00:15:23 UTC 2013 [00:15:35] RECOVERY - Puppet freshness on search13 is OK: puppet ran at Fri Jun 28 00:15:23 UTC 2013 [00:15:35] RECOVERY - Puppet freshness on mw16 is OK: puppet ran at Fri Jun 28 00:15:24 UTC 2013 [00:15:35] RECOVERY - Puppet freshness on srv287 is OK: puppet ran at Fri Jun 28 00:15:25 UTC 2013 [00:15:35] RECOVERY - Puppet freshness on analytics1012 is OK: puppet ran at Fri Jun 28 00:15:28 UTC 2013 [00:15:35] RECOVERY - Puppet freshness on srv300 is OK: puppet ran at Fri Jun 28 00:15:30 UTC 2013 [00:15:35] RECOVERY - Puppet freshness on cp1061 is OK: puppet ran at Fri Jun 28 00:15:31 UTC 2013 [00:15:36] RECOVERY - Puppet freshness on cp1056 is OK: puppet ran at Fri Jun 28 00:15:32 UTC 2013 [00:15:44] RECOVERY - Puppet freshness on mw1100 is OK: puppet ran at Fri Jun 28 00:15:33 UTC 2013 [00:15:45] RECOVERY - Puppet freshness on mw1217 is OK: puppet ran at Fri Jun 28 00:15:33 UTC 2013 [00:15:45] RECOVERY - Puppet freshness on mw29 is OK: puppet ran at Fri Jun 28 00:15:34 UTC 2013 [00:15:45] RECOVERY - Puppet freshness on mw1117 is OK: puppet ran at Fri Jun 28 00:15:35 UTC 2013 [00:15:45] RECOVERY - Puppet freshness on mw1164 is OK: puppet ran at Fri Jun 28 00:15:36 UTC 2013 [00:15:45] RECOVERY - Puppet freshness on mw51 is OK: puppet ran at Fri Jun 28 00:15:38 UTC 2013 [00:15:46] RECOVERY - Puppet freshness on mw1088 is OK: puppet ran at Fri Jun 28 00:15:39 UTC 2013 [00:15:46] RECOVERY - Puppet freshness on analytics1005 is OK: puppet ran at Fri Jun 28 00:15:40 UTC 2013 [00:15:47] RECOVERY - Puppet freshness on mw1099 is OK: puppet ran at Fri Jun 28 00:15:40 UTC 2013 [00:15:47] RECOVERY - Puppet freshness on cp3007 is OK: puppet ran at Fri Jun 28 00:15:41 UTC 2013 [00:15:48] RECOVERY - Puppet freshness on analytics1013 is OK: puppet ran at Fri Jun 28 00:15:42 UTC 2013 [00:15:48] RECOVERY - Puppet freshness on mw1018 is OK: puppet ran at Fri Jun 28 00:15:43 UTC 2013 [00:15:55] RECOVERY - Puppet freshness on db1054 is OK: puppet ran at Fri Jun 28 00:15:44 UTC 2013 [00:15:55] RECOVERY - Puppet freshness on labsdb1003 is OK: puppet ran at Fri Jun 28 00:15:45 UTC 2013 [00:15:56] RECOVERY - Puppet freshness on cerium is OK: puppet ran at Fri Jun 28 00:15:45 UTC 2013 [00:15:56] RECOVERY - Puppet freshness on sq65 is OK: puppet ran at Fri Jun 28 00:15:46 UTC 2013 [00:15:56] RECOVERY - Puppet freshness on williams is OK: puppet ran at Fri Jun 28 00:15:47 UTC 2013 [00:15:56] RECOVERY - Puppet freshness on mc3 is OK: puppet ran at Fri Jun 28 00:15:48 UTC 2013 [00:15:56] RECOVERY - Puppet freshness on search1018 is OK: puppet ran at Fri Jun 28 00:15:49 UTC 2013 [00:15:57] RECOVERY - Puppet freshness on search17 is OK: puppet ran at Fri Jun 28 00:15:50 UTC 2013 [00:15:57] RECOVERY - Puppet freshness on search1008 is OK: puppet ran at Fri Jun 28 00:15:50 UTC 2013 [00:15:58] RECOVERY - Puppet freshness on db1023 is OK: puppet ran at Fri Jun 28 00:15:51 UTC 2013 [00:15:58] RECOVERY - Puppet freshness on db1019 is OK: puppet ran at Fri Jun 28 00:15:51 UTC 2013 [00:15:59] RECOVERY - Puppet freshness on cp1012 is OK: puppet ran at Fri Jun 28 00:15:52 UTC 2013 [00:16:03] RECOVERY - Puppet freshness on db40 is OK: puppet ran at Fri Jun 28 00:15:55 UTC 2013 [00:16:04] RECOVERY - Puppet freshness on amssq55 is OK: puppet ran at Fri Jun 28 00:15:56 UTC 2013 [00:16:04] RECOVERY - Puppet freshness on mw97 is OK: puppet ran at Fri Jun 28 00:16:01 UTC 2013 [00:16:04] RECOVERY - Puppet freshness on amssq52 is OK: puppet ran at Fri Jun 28 00:16:01 UTC 2013 [00:16:13] RECOVERY - Puppet freshness on ms-be8 is OK: puppet ran at Fri Jun 28 00:16:03 UTC 2013 [00:16:13] RECOVERY - Puppet freshness on labsdb1002 is OK: puppet ran at Fri Jun 28 00:16:04 UTC 2013 [00:16:14] RECOVERY - Puppet freshness on mw122 is OK: puppet ran at Fri Jun 28 00:16:05 UTC 2013 [00:16:14] RECOVERY - Puppet freshness on mw20 is OK: puppet ran at Fri Jun 28 00:16:06 UTC 2013 [00:16:14] RECOVERY - Puppet freshness on srv297 is OK: puppet ran at Fri Jun 28 00:16:07 UTC 2013 [00:16:14] RECOVERY - Puppet freshness on es1005 is OK: puppet ran at Fri Jun 28 00:16:07 UTC 2013 [00:16:14] RECOVERY - Puppet freshness on mw1123 is OK: puppet ran at Fri Jun 28 00:16:08 UTC 2013 [00:16:15] RECOVERY - Puppet freshness on mw46 is OK: puppet ran at Fri Jun 28 00:16:09 UTC 2013 [00:16:15] RECOVERY - Puppet freshness on mw82 is OK: puppet ran at Fri Jun 28 00:16:10 UTC 2013 [00:16:16] RECOVERY - Puppet freshness on mw1101 is OK: puppet ran at Fri Jun 28 00:16:10 UTC 2013 [00:16:16] RECOVERY - Puppet freshness on mw1095 is OK: puppet ran at Fri Jun 28 00:16:11 UTC 2013 [00:16:16] RECOVERY - Puppet freshness on mw1144 is OK: puppet ran at Fri Jun 28 00:16:12 UTC 2013 [00:16:24] RECOVERY - Puppet freshness on srv238 is OK: puppet ran at Fri Jun 28 00:16:12 UTC 2013 [00:16:25] RECOVERY - Puppet freshness on mw34 is OK: puppet ran at Fri Jun 28 00:16:13 UTC 2013 [00:16:25] RECOVERY - Puppet freshness on mw1052 is OK: puppet ran at Fri Jun 28 00:16:13 UTC 2013 [00:16:25] RECOVERY - Puppet freshness on mw1017 is OK: puppet ran at Fri Jun 28 00:16:14 UTC 2013 [00:16:25] RECOVERY - Puppet freshness on mw1103 is OK: puppet ran at Fri Jun 28 00:16:15 UTC 2013 [00:16:25] RECOVERY - Puppet freshness on dobson is OK: puppet ran at Fri Jun 28 00:16:16 UTC 2013 [00:16:25] RECOVERY - Puppet freshness on es1003 is OK: puppet ran at Fri Jun 28 00:16:17 UTC 2013 [00:16:26] RECOVERY - Puppet freshness on gurvin is OK: puppet ran at Fri Jun 28 00:16:17 UTC 2013 [00:16:27] RECOVERY - Puppet freshness on amssq57 is OK: puppet ran at Fri Jun 28 00:16:19 UTC 2013 [00:16:27] RECOVERY - Puppet freshness on sq72 is OK: puppet ran at Fri Jun 28 00:16:20 UTC 2013 [00:16:28] RECOVERY - Puppet freshness on tin is OK: puppet ran at Fri Jun 28 00:16:21 UTC 2013 [00:16:28] RECOVERY - Puppet freshness on search1019 is OK: puppet ran at Fri Jun 28 00:16:22 UTC 2013 [00:16:36] RECOVERY - Puppet freshness on amssq34 is OK: puppet ran at Fri Jun 28 00:16:24 UTC 2013 [00:16:36] RECOVERY - Puppet freshness on mw1126 is OK: puppet ran at Fri Jun 28 00:16:25 UTC 2013 [00:16:36] RECOVERY - Puppet freshness on mw47 is OK: puppet ran at Fri Jun 28 00:16:27 UTC 2013 [00:16:36] RECOVERY - Puppet freshness on srv286 is OK: puppet ran at Fri Jun 28 00:16:27 UTC 2013 [00:16:36] RECOVERY - Puppet freshness on mw125 is OK: puppet ran at Fri Jun 28 00:16:30 UTC 2013 [00:16:36] RECOVERY - Puppet freshness on mw49 is OK: puppet ran at Fri Jun 28 00:16:32 UTC 2013 [00:16:37] RECOVERY - Puppet freshness on db1038 is OK: puppet ran at Fri Jun 28 00:16:32 UTC 2013 [00:16:43] RECOVERY - Puppet freshness on mw1127 is OK: puppet ran at Fri Jun 28 00:16:36 UTC 2013 [00:16:44] RECOVERY - Puppet freshness on srv237 is OK: puppet ran at Fri Jun 28 00:16:37 UTC 2013 [00:16:45] RECOVERY - Puppet freshness on mw1214 is OK: puppet ran at Fri Jun 28 00:16:37 UTC 2013 [00:16:45] RECOVERY - Puppet freshness on mw1114 is OK: puppet ran at Fri Jun 28 00:16:41 UTC 2013 [00:16:46] RECOVERY - Puppet freshness on srv250 is OK: puppet ran at Fri Jun 28 00:16:42 UTC 2013 [00:16:53] RECOVERY - Puppet freshness on mw1015 is OK: puppet ran at Fri Jun 28 00:16:42 UTC 2013 [00:16:53] RECOVERY - Puppet freshness on mw66 is OK: puppet ran at Fri Jun 28 00:16:44 UTC 2013 [00:16:54] RECOVERY - Puppet freshness on analytics1010 is OK: puppet ran at Fri Jun 28 00:16:45 UTC 2013 [00:16:54] RECOVERY - Puppet freshness on mw62 is OK: puppet ran at Fri Jun 28 00:16:45 UTC 2013 [00:16:54] RECOVERY - Puppet freshness on srv284 is OK: puppet ran at Fri Jun 28 00:16:47 UTC 2013 [00:16:54] RECOVERY - Puppet freshness on mw1042 is OK: puppet ran at Fri Jun 28 00:16:47 UTC 2013 [00:16:54] RECOVERY - Puppet freshness on cp1013 is OK: puppet ran at Fri Jun 28 00:16:48 UTC 2013 [00:16:54] RECOVERY - Puppet freshness on mw1065 is OK: puppet ran at Fri Jun 28 00:16:49 UTC 2013 [00:16:55] RECOVERY - Puppet freshness on es4 is OK: puppet ran at Fri Jun 28 00:16:49 UTC 2013 [00:16:56] RECOVERY - Puppet freshness on db53 is OK: puppet ran at Fri Jun 28 00:16:50 UTC 2013 [00:16:56] RECOVERY - Puppet freshness on es10 is OK: puppet ran at Fri Jun 28 00:16:51 UTC 2013 [00:16:57] RECOVERY - Puppet freshness on ms-be1 is OK: puppet ran at Fri Jun 28 00:16:52 UTC 2013 [00:17:04] RECOVERY - Puppet freshness on cp1069 is OK: puppet ran at Fri Jun 28 00:16:52 UTC 2013 [00:17:04] RECOVERY - Puppet freshness on pc1002 is OK: puppet ran at Fri Jun 28 00:16:53 UTC 2013 [00:17:04] RECOVERY - Puppet freshness on cp1028 is OK: puppet ran at Fri Jun 28 00:16:54 UTC 2013 [00:17:04] RECOVERY - Puppet freshness on db1052 is OK: puppet ran at Fri Jun 28 00:16:54 UTC 2013 [00:17:04] RECOVERY - Puppet freshness on amssq54 is OK: puppet ran at Fri Jun 28 00:16:55 UTC 2013 [00:17:05] RECOVERY - Puppet freshness on ms5 is OK: puppet ran at Fri Jun 28 00:16:56 UTC 2013 [00:17:05] RECOVERY - Puppet freshness on analytics1027 is OK: puppet ran at Fri Jun 28 00:16:57 UTC 2013 [00:17:06] RECOVERY - Puppet freshness on search22 is OK: puppet ran at Fri Jun 28 00:16:57 UTC 2013 [00:17:06] RECOVERY - Puppet freshness on cp3010 is OK: puppet ran at Fri Jun 28 00:16:59 UTC 2013 [00:17:07] RECOVERY - Puppet freshness on mw14 is OK: puppet ran at Fri Jun 28 00:17:00 UTC 2013 [00:17:07] RECOVERY - Puppet freshness on search21 is OK: puppet ran at Fri Jun 28 00:17:01 UTC 2013 [00:17:08] RECOVERY - Puppet freshness on sq45 is OK: puppet ran at Fri Jun 28 00:17:01 UTC 2013 [00:17:13] RECOVERY - Puppet freshness on lvs1 is OK: puppet ran at Fri Jun 28 00:17:02 UTC 2013 [00:17:13] RECOVERY - Puppet freshness on srv275 is OK: puppet ran at Fri Jun 28 00:17:03 UTC 2013 [00:17:14] RECOVERY - Puppet freshness on mw81 is OK: puppet ran at Fri Jun 28 00:17:05 UTC 2013 [00:17:14] RECOVERY - Puppet freshness on cp1003 is OK: puppet ran at Fri Jun 28 00:17:06 UTC 2013 [00:17:14] RECOVERY - Puppet freshness on mw41 is OK: puppet ran at Fri Jun 28 00:17:07 UTC 2013 [00:17:14] RECOVERY - Puppet freshness on mw100 is OK: puppet ran at Fri Jun 28 00:17:08 UTC 2013 [00:17:14] RECOVERY - Puppet freshness on mw1175 is OK: puppet ran at Fri Jun 28 00:17:11 UTC 2013 [00:17:15] RECOVERY - Puppet freshness on mw1196 is OK: puppet ran at Fri Jun 28 00:17:12 UTC 2013 [00:17:23] RECOVERY - Puppet freshness on mw90 is OK: puppet ran at Fri Jun 28 00:17:12 UTC 2013 [00:17:24] RECOVERY - Puppet freshness on mw1162 is OK: puppet ran at Fri Jun 28 00:17:16 UTC 2013 [00:17:24] RECOVERY - Puppet freshness on search1014 is OK: puppet ran at Fri Jun 28 00:17:16 UTC 2013 [00:17:24] RECOVERY - Puppet freshness on mw1182 is OK: puppet ran at Fri Jun 28 00:17:17 UTC 2013 [00:17:24] RECOVERY - Puppet freshness on sq79 is OK: puppet ran at Fri Jun 28 00:17:18 UTC 2013 [00:17:24] RECOVERY - Puppet freshness on cp1051 is OK: puppet ran at Fri Jun 28 00:17:20 UTC 2013 [00:17:25] RECOVERY - Puppet freshness on es1002 is OK: puppet ran at Fri Jun 28 00:17:21 UTC 2013 [00:17:35] RECOVERY - Puppet freshness on mw5 is OK: puppet ran at Fri Jun 28 00:17:22 UTC 2013 [00:17:35] RECOVERY - Puppet freshness on cp1006 is OK: puppet ran at Fri Jun 28 00:17:25 UTC 2013 [00:17:35] RECOVERY - Puppet freshness on pc1003 is OK: puppet ran at Fri Jun 28 00:17:26 UTC 2013 [00:17:35] RECOVERY - Puppet freshness on db1048 is OK: puppet ran at Fri Jun 28 00:17:27 UTC 2013 [00:17:35] RECOVERY - Puppet freshness on cp1050 is OK: puppet ran at Fri Jun 28 00:17:28 UTC 2013 [00:17:36] RECOVERY - Puppet freshness on mc1005 is OK: puppet ran at Fri Jun 28 00:17:31 UTC 2013 [00:17:43] RECOVERY - Puppet freshness on cp1046 is OK: puppet ran at Fri Jun 28 00:17:32 UTC 2013 [00:17:44] RECOVERY - Puppet freshness on ms-be4 is OK: puppet ran at Fri Jun 28 00:17:33 UTC 2013 [00:17:44] RECOVERY - Puppet freshness on mw9 is OK: puppet ran at Fri Jun 28 00:17:34 UTC 2013 [00:17:44] RECOVERY - Puppet freshness on wtp1024 is OK: puppet ran at Fri Jun 28 00:17:35 UTC 2013 [00:17:44] RECOVERY - Puppet freshness on zirconium is OK: puppet ran at Fri Jun 28 00:17:36 UTC 2013 [00:17:44] RECOVERY - Puppet freshness on search14 is OK: puppet ran at Fri Jun 28 00:17:37 UTC 2013 [00:17:44] RECOVERY - Puppet freshness on wtp1018 is OK: puppet ran at Fri Jun 28 00:17:37 UTC 2013 [00:17:45] RECOVERY - Puppet freshness on sq51 is OK: puppet ran at Fri Jun 28 00:17:38 UTC 2013 [00:17:45] RECOVERY - Puppet freshness on cp1063 is OK: puppet ran at Fri Jun 28 00:17:38 UTC 2013 [00:17:45] RECOVERY - Puppet freshness on ms-fe3 is OK: puppet ran at Fri Jun 28 00:17:39 UTC 2013 [00:17:46] RECOVERY - Puppet freshness on mw1190 is OK: puppet ran at Fri Jun 28 00:17:39 UTC 2013 [00:17:46] RECOVERY - Puppet freshness on db1036 is OK: puppet ran at Fri Jun 28 00:17:40 UTC 2013 [00:17:47] RECOVERY - Puppet freshness on search36 is OK: puppet ran at Fri Jun 28 00:17:40 UTC 2013 [00:17:48] RECOVERY - Puppet freshness on mw38 is OK: puppet ran at Fri Jun 28 00:17:42 UTC 2013 [00:17:53] RECOVERY - Puppet freshness on mw83 is OK: puppet ran at Fri Jun 28 00:17:42 UTC 2013 [00:17:53] RECOVERY - Puppet freshness on vanadium is OK: puppet ran at Fri Jun 28 00:17:42 UTC 2013 [00:17:54] RECOVERY - Puppet freshness on db45 is OK: puppet ran at Fri Jun 28 00:17:43 UTC 2013 [00:17:54] RECOVERY - Puppet freshness on mw1044 is OK: puppet ran at Fri Jun 28 00:17:43 UTC 2013 [00:17:54] RECOVERY - Puppet freshness on ms-be2 is OK: puppet ran at Fri Jun 28 00:17:45 UTC 2013 [00:17:54] RECOVERY - Puppet freshness on amssq44 is OK: puppet ran at Fri Jun 28 00:17:45 UTC 2013 [00:17:54] RECOVERY - Puppet freshness on mw119 is OK: puppet ran at Fri Jun 28 00:17:46 UTC 2013 [00:17:55] RECOVERY - Puppet freshness on mw118 is OK: puppet ran at Fri Jun 28 00:17:47 UTC 2013 [00:17:55] RECOVERY - Puppet freshness on ms-be1012 is OK: puppet ran at Fri Jun 28 00:17:47 UTC 2013 [00:17:56] RECOVERY - Puppet freshness on wtp1011 is OK: puppet ran at Fri Jun 28 00:17:49 UTC 2013 [00:17:56] RECOVERY - Puppet freshness on mc1009 is OK: puppet ran at Fri Jun 28 00:17:50 UTC 2013 [00:17:56] RECOVERY - Puppet freshness on wtp1003 is OK: puppet ran at Fri Jun 28 00:17:52 UTC 2013 [00:18:03] RECOVERY - Puppet freshness on srv290 is OK: puppet ran at Fri Jun 28 00:17:55 UTC 2013 [00:18:04] RECOVERY - Puppet freshness on srv301 is OK: puppet ran at Fri Jun 28 00:17:56 UTC 2013 [00:18:04] RECOVERY - Puppet freshness on db43 is OK: puppet ran at Fri Jun 28 00:17:57 UTC 2013 [00:18:04] RECOVERY - Puppet freshness on analytics1015 is OK: puppet ran at Fri Jun 28 00:17:57 UTC 2013 [00:18:04] RECOVERY - Puppet freshness on srv244 is OK: puppet ran at Fri Jun 28 00:18:00 UTC 2013 [00:18:04] RECOVERY - Puppet freshness on srv246 is OK: puppet ran at Fri Jun 28 00:18:01 UTC 2013 [00:18:05] RECOVERY - Puppet freshness on srv245 is OK: puppet ran at Fri Jun 28 00:18:02 UTC 2013 [00:18:13] RECOVERY - Puppet freshness on mw1161 is OK: puppet ran at Fri Jun 28 00:18:03 UTC 2013 [00:18:13] RECOVERY - Puppet freshness on mw30 is OK: puppet ran at Fri Jun 28 00:18:06 UTC 2013 [00:18:14] RECOVERY - Puppet freshness on srv289 is OK: puppet ran at Fri Jun 28 00:18:06 UTC 2013 [00:18:14] RECOVERY - Puppet freshness on analytics1023 is OK: puppet ran at Fri Jun 28 00:18:07 UTC 2013 [00:18:14] RECOVERY - Puppet freshness on mw1111 is OK: puppet ran at Fri Jun 28 00:18:08 UTC 2013 [00:18:14] RECOVERY - Puppet freshness on mw1124 is OK: puppet ran at Fri Jun 28 00:18:10 UTC 2013 [00:18:14] RECOVERY - Puppet freshness on mw1048 is OK: puppet ran at Fri Jun 28 00:18:11 UTC 2013 [00:18:15] RECOVERY - Puppet freshness on mw1125 is OK: puppet ran at Fri Jun 28 00:18:12 UTC 2013 [00:18:23] RECOVERY - Puppet freshness on mw1147 is OK: puppet ran at Fri Jun 28 00:18:12 UTC 2013 [00:18:24] RECOVERY - Puppet freshness on srv253 is OK: puppet ran at Fri Jun 28 00:18:13 UTC 2013 [00:18:24] RECOVERY - Puppet freshness on mw19 is OK: puppet ran at Fri Jun 28 00:18:17 UTC 2013 [00:18:24] RECOVERY - Puppet freshness on fenari is OK: puppet ran at Fri Jun 28 00:18:17 UTC 2013 [00:18:24] RECOVERY - Puppet freshness on mw77 is OK: puppet ran at Fri Jun 28 00:18:18 UTC 2013 [00:18:24] RECOVERY - Puppet freshness on mw11 is OK: puppet ran at Fri Jun 28 00:18:18 UTC 2013 [00:18:24] RECOVERY - Puppet freshness on srv277 is OK: puppet ran at Fri Jun 28 00:18:20 UTC 2013 [00:18:24] RECOVERY - Puppet freshness on sq85 is OK: puppet ran at Fri Jun 28 00:18:21 UTC 2013 [00:18:25] RECOVERY - Puppet freshness on mw1038 is OK: puppet ran at Fri Jun 28 00:18:21 UTC 2013 [00:18:33] RECOVERY - Puppet freshness on neon is OK: puppet ran at Fri Jun 28 00:18:22 UTC 2013 [00:18:34] RECOVERY - Puppet freshness on sq74 is OK: puppet ran at Fri Jun 28 00:18:22 UTC 2013 [00:18:34] RECOVERY - Puppet freshness on db1029 is OK: puppet ran at Fri Jun 28 00:18:23 UTC 2013 [00:18:34] RECOVERY - Puppet freshness on sq43 is OK: puppet ran at Fri Jun 28 00:18:23 UTC 2013 [00:18:34] RECOVERY - Puppet freshness on snapshot1 is OK: puppet ran at Fri Jun 28 00:18:23 UTC 2013 [00:18:34] RECOVERY - Puppet freshness on ms-be3 is OK: puppet ran at Fri Jun 28 00:18:25 UTC 2013 [00:18:34] RECOVERY - Puppet freshness on ms-be9 is OK: puppet ran at Fri Jun 28 00:18:25 UTC 2013 [00:18:35] RECOVERY - Puppet freshness on sq42 is OK: puppet ran at Fri Jun 28 00:18:26 UTC 2013 [00:18:35] RECOVERY - Puppet freshness on mw76 is OK: puppet ran at Fri Jun 28 00:18:26 UTC 2013 [00:18:36] RECOVERY - Puppet freshness on solr2 is OK: puppet ran at Fri Jun 28 00:18:29 UTC 2013 [00:18:36] RECOVERY - Puppet freshness on stafford is OK: puppet ran at Fri Jun 28 00:18:30 UTC 2013 [00:18:37] RECOVERY - Puppet freshness on es1 is OK: puppet ran at Fri Jun 28 00:18:31 UTC 2013 [00:18:37] RECOVERY - Puppet freshness on sq48 is OK: puppet ran at Fri Jun 28 00:18:32 UTC 2013 [00:18:38] RECOVERY - Puppet freshness on mw1037 is OK: puppet ran at Fri Jun 28 00:18:32 UTC 2013 [00:18:43] RECOVERY - Puppet freshness on mw1197 is OK: puppet ran at Fri Jun 28 00:18:32 UTC 2013 [00:18:44] RECOVERY - Puppet freshness on mw58 is OK: puppet ran at Fri Jun 28 00:18:33 UTC 2013 [00:18:44] RECOVERY - Puppet freshness on srv294 is OK: puppet ran at Fri Jun 28 00:18:35 UTC 2013 [00:18:44] RECOVERY - Puppet freshness on srv276 is OK: puppet ran at Fri Jun 28 00:18:36 UTC 2013 [00:18:44] RECOVERY - Puppet freshness on mw1004 is OK: puppet ran at Fri Jun 28 00:18:36 UTC 2013 [00:18:44] RECOVERY - Puppet freshness on mw1140 is OK: puppet ran at Fri Jun 28 00:18:38 UTC 2013 [00:18:44] RECOVERY - Puppet freshness on search1024 is OK: puppet ran at Fri Jun 28 00:18:39 UTC 2013 [00:18:45] RECOVERY - Puppet freshness on mw1087 is OK: puppet ran at Fri Jun 28 00:18:41 UTC 2013 [00:18:45] RECOVERY - Puppet freshness on sq73 is OK: puppet ran at Fri Jun 28 00:18:42 UTC 2013 [00:18:53] RECOVERY - Puppet freshness on mw121 is OK: puppet ran at Fri Jun 28 00:18:43 UTC 2013 [00:18:54] RECOVERY - Puppet freshness on mw1171 is OK: puppet ran at Fri Jun 28 00:18:43 UTC 2013 [00:18:54] RECOVERY - Puppet freshness on solr3 is OK: puppet ran at Fri Jun 28 00:18:44 UTC 2013 [00:18:54] RECOVERY - Puppet freshness on mw1210 is OK: puppet ran at Fri Jun 28 00:18:46 UTC 2013 [00:18:54] RECOVERY - Puppet freshness on mw1007 is OK: puppet ran at Fri Jun 28 00:18:47 UTC 2013 [00:18:54] RECOVERY - Puppet freshness on mw1043 is OK: puppet ran at Fri Jun 28 00:18:48 UTC 2013 [00:18:54] RECOVERY - Puppet freshness on stat1 is OK: puppet ran at Fri Jun 28 00:18:49 UTC 2013 [00:18:54] RECOVERY - Puppet freshness on search19 is OK: puppet ran at Fri Jun 28 00:18:49 UTC 2013 [00:19:03] RECOVERY - Puppet freshness on searchidx1001 is OK: puppet ran at Fri Jun 28 00:18:52 UTC 2013 [00:19:04] RECOVERY - Puppet freshness on mc1001 is OK: puppet ran at Fri Jun 28 00:18:53 UTC 2013 [00:19:04] RECOVERY - Puppet freshness on cp1036 is OK: puppet ran at Fri Jun 28 00:18:54 UTC 2013 [00:19:04] RECOVERY - Puppet freshness on mw1186 is OK: puppet ran at Fri Jun 28 00:18:54 UTC 2013 [00:19:04] RECOVERY - Puppet freshness on mw1041 is OK: puppet ran at Fri Jun 28 00:18:58 UTC 2013 [00:19:04] RECOVERY - Puppet freshness on mw1159 is OK: puppet ran at Fri Jun 28 00:18:59 UTC 2013 [00:19:05] RECOVERY - Puppet freshness on cp1049 is OK: puppet ran at Fri Jun 28 00:19:02 UTC 2013 [00:19:13] RECOVERY - Puppet freshness on pdf3 is OK: puppet ran at Fri Jun 28 00:19:03 UTC 2013 [00:19:13] RECOVERY - Puppet freshness on manganese is OK: puppet ran at Fri Jun 28 00:19:03 UTC 2013 [00:19:13] RECOVERY - Puppet freshness on solr1003 is OK: puppet ran at Fri Jun 28 00:19:04 UTC 2013 [00:19:13] RECOVERY - Puppet freshness on es6 is OK: puppet ran at Fri Jun 28 00:19:05 UTC 2013 [00:19:13] RECOVERY - Puppet freshness on db58 is OK: puppet ran at Fri Jun 28 00:19:05 UTC 2013 [00:19:14] RECOVERY - Puppet freshness on cp1002 is OK: puppet ran at Fri Jun 28 00:19:06 UTC 2013 [00:19:14] RECOVERY - Puppet freshness on cp1005 is OK: puppet ran at Fri Jun 28 00:19:06 UTC 2013 [00:19:15] RECOVERY - Puppet freshness on mc7 is OK: puppet ran at Fri Jun 28 00:19:08 UTC 2013 [00:19:15] RECOVERY - Puppet freshness on db1020 is OK: puppet ran at Fri Jun 28 00:19:08 UTC 2013 [00:19:15] RECOVERY - Puppet freshness on praseodymium is OK: puppet ran at Fri Jun 28 00:19:09 UTC 2013 [00:19:16] RECOVERY - Puppet freshness on pdf2 is OK: puppet ran at Fri Jun 28 00:19:10 UTC 2013 [00:19:16] RECOVERY - Puppet freshness on db34 is OK: puppet ran at Fri Jun 28 00:19:10 UTC 2013 [00:19:17] RECOVERY - Puppet freshness on sq76 is OK: puppet ran at Fri Jun 28 00:19:11 UTC 2013 [00:19:18] RECOVERY - Puppet freshness on pc1 is OK: puppet ran at Fri Jun 28 00:19:11 UTC 2013 [00:19:18] RECOVERY - Puppet freshness on helium is OK: puppet ran at Fri Jun 28 00:19:12 UTC 2013 [00:19:23] RECOVERY - Puppet freshness on sq58 is OK: puppet ran at Fri Jun 28 00:19:12 UTC 2013 [00:19:23] RECOVERY - Puppet freshness on rdb1003 is OK: puppet ran at Fri Jun 28 00:19:13 UTC 2013 [00:19:23] RECOVERY - Puppet freshness on db1044 is OK: puppet ran at Fri Jun 28 00:19:13 UTC 2013 [00:19:23] RECOVERY - Puppet freshness on sq54 is OK: puppet ran at Fri Jun 28 00:19:13 UTC 2013 [00:19:23] RECOVERY - Puppet freshness on potassium is OK: puppet ran at Fri Jun 28 00:19:14 UTC 2013 [00:19:24] RECOVERY - Puppet freshness on ekrem is OK: puppet ran at Fri Jun 28 00:19:14 UTC 2013 [00:19:24] RECOVERY - Puppet freshness on cp1024 is OK: puppet ran at Fri Jun 28 00:19:15 UTC 2013 [00:19:24] RECOVERY - Puppet freshness on db1022 is OK: puppet ran at Fri Jun 28 00:19:16 UTC 2013 [00:19:25] RECOVERY - Puppet freshness on rdb1002 is OK: puppet ran at Fri Jun 28 00:19:16 UTC 2013 [00:19:26] RECOVERY - Puppet freshness on db1031 is OK: puppet ran at Fri Jun 28 00:19:16 UTC 2013 [00:19:26] RECOVERY - Puppet freshness on ms10 is OK: puppet ran at Fri Jun 28 00:19:17 UTC 2013 [00:19:27] RECOVERY - Puppet freshness on cp1020 is OK: puppet ran at Fri Jun 28 00:19:17 UTC 2013 [00:19:27] RECOVERY - Puppet freshness on ms-be1006 is OK: puppet ran at Fri Jun 28 00:19:19 UTC 2013 [00:19:28] RECOVERY - Puppet freshness on search18 is OK: puppet ran at Fri Jun 28 00:19:19 UTC 2013 [00:19:28] RECOVERY - Puppet freshness on cp3012 is OK: puppet ran at Fri Jun 28 00:19:21 UTC 2013 [00:19:34] RECOVERY - Puppet freshness on cp3009 is OK: puppet ran at Fri Jun 28 00:19:22 UTC 2013 [00:19:35] RECOVERY - Puppet freshness on wtp1004 is OK: puppet ran at Fri Jun 28 00:19:24 UTC 2013 [00:19:35] RECOVERY - Puppet freshness on mc10 is OK: puppet ran at Fri Jun 28 00:19:30 UTC 2013 [00:19:43] RECOVERY - Puppet freshness on wtp1015 is OK: puppet ran at Fri Jun 28 00:19:32 UTC 2013 [00:19:44] RECOVERY - Puppet freshness on db39 is OK: puppet ran at Fri Jun 28 00:19:34 UTC 2013 [00:19:44] RECOVERY - Puppet freshness on ms-fe1001 is OK: puppet ran at Fri Jun 28 00:19:38 UTC 2013 [00:19:44] RECOVERY - Puppet freshness on srv292 is OK: puppet ran at Fri Jun 28 00:19:39 UTC 2013 [00:19:44] RECOVERY - Puppet freshness on mw1115 is OK: puppet ran at Fri Jun 28 00:19:40 UTC 2013 [00:19:44] RECOVERY - Puppet freshness on db1001 is OK: puppet ran at Fri Jun 28 00:19:42 UTC 2013 [00:19:53] RECOVERY - Puppet freshness on mw1146 is OK: puppet ran at Fri Jun 28 00:19:43 UTC 2013 [00:19:54] RECOVERY - Puppet freshness on mw1089 is OK: puppet ran at Fri Jun 28 00:19:44 UTC 2013 [00:19:54] RECOVERY - Puppet freshness on mw1063 is OK: puppet ran at Fri Jun 28 00:19:45 UTC 2013 [00:19:54] RECOVERY - Puppet freshness on mw1032 is OK: puppet ran at Fri Jun 28 00:19:47 UTC 2013 [00:19:54] RECOVERY - Puppet freshness on mc1007 is OK: puppet ran at Fri Jun 28 00:19:47 UTC 2013 [00:19:54] RECOVERY - Puppet freshness on mw1106 is OK: puppet ran at Fri Jun 28 00:19:48 UTC 2013 [00:19:54] RECOVERY - Puppet freshness on mw1057 is OK: puppet ran at Fri Jun 28 00:19:49 UTC 2013 [00:19:55] RECOVERY - Puppet freshness on mw124 is OK: puppet ran at Fri Jun 28 00:19:50 UTC 2013 [00:19:55] RECOVERY - Puppet freshness on analytics1014 is OK: puppet ran at Fri Jun 28 00:19:51 UTC 2013 [00:19:56] RECOVERY - Puppet freshness on mw1205 is OK: puppet ran at Fri Jun 28 00:19:52 UTC 2013 [00:20:04] RECOVERY - Puppet freshness on calcium is OK: puppet ran at Fri Jun 28 00:19:52 UTC 2013 [00:20:04] RECOVERY - Puppet freshness on sq63 is OK: puppet ran at Fri Jun 28 00:19:53 UTC 2013 [00:20:04] RECOVERY - Puppet freshness on mc6 is OK: puppet ran at Fri Jun 28 00:19:53 UTC 2013 [00:20:04] RECOVERY - Puppet freshness on mw79 is OK: puppet ran at Fri Jun 28 00:19:54 UTC 2013 [00:20:04] RECOVERY - Puppet freshness on mw98 is OK: puppet ran at Fri Jun 28 00:19:54 UTC 2013 [00:20:04] RECOVERY - Puppet freshness on wtp1007 is OK: puppet ran at Fri Jun 28 00:19:55 UTC 2013 [00:20:04] RECOVERY - Puppet freshness on search33 is OK: puppet ran at Fri Jun 28 00:19:56 UTC 2013 [00:20:05] RECOVERY - Puppet freshness on db48 is OK: puppet ran at Fri Jun 28 00:19:58 UTC 2013 [00:20:06] RECOVERY - Puppet freshness on db1040 is OK: puppet ran at Fri Jun 28 00:19:59 UTC 2013 [00:20:06] RECOVERY - Puppet freshness on cp1030 is OK: puppet ran at Fri Jun 28 00:19:59 UTC 2013 [00:20:07] RECOVERY - Puppet freshness on srv273 is OK: puppet ran at Fri Jun 28 00:20:01 UTC 2013 [00:20:07] RECOVERY - Puppet freshness on mw1033 is OK: puppet ran at Fri Jun 28 00:20:02 UTC 2013 [00:20:13] RECOVERY - Puppet freshness on srv255 is OK: puppet ran at Fri Jun 28 00:20:03 UTC 2013 [00:20:14] RECOVERY - Puppet freshness on srv193 is OK: puppet ran at Fri Jun 28 00:20:04 UTC 2013 [00:20:14] RECOVERY - Puppet freshness on mw1189 is OK: puppet ran at Fri Jun 28 00:20:05 UTC 2013 [00:20:14] RECOVERY - Puppet freshness on mw57 is OK: puppet ran at Fri Jun 28 00:20:07 UTC 2013 [00:20:14] RECOVERY - Puppet freshness on mw43 is OK: puppet ran at Fri Jun 28 00:20:08 UTC 2013 [00:20:14] RECOVERY - Puppet freshness on mw1046 is OK: puppet ran at Fri Jun 28 00:20:09 UTC 2013 [00:20:14] RECOVERY - Puppet freshness on mw1201 is OK: puppet ran at Fri Jun 28 00:20:09 UTC 2013 [00:20:15] RECOVERY - Puppet freshness on mw106 is OK: puppet ran at Fri Jun 28 00:20:10 UTC 2013 [00:20:15] RECOVERY - Puppet freshness on db1002 is OK: puppet ran at Fri Jun 28 00:20:11 UTC 2013 [00:20:23] RECOVERY - Puppet freshness on lvs5 is OK: puppet ran at Fri Jun 28 00:20:14 UTC 2013 [00:20:23] RECOVERY - Puppet freshness on srv258 is OK: puppet ran at Fri Jun 28 00:20:14 UTC 2013 [00:20:24] RECOVERY - Puppet freshness on cp1037 is OK: puppet ran at Fri Jun 28 00:20:18 UTC 2013 [00:20:24] RECOVERY - Puppet freshness on mw1150 is OK: puppet ran at Fri Jun 28 00:20:19 UTC 2013 [00:20:24] RECOVERY - Puppet freshness on mw35 is OK: puppet ran at Fri Jun 28 00:20:19 UTC 2013 [00:20:24] RECOVERY - Puppet freshness on mw1003 is OK: puppet ran at Fri Jun 28 00:20:20 UTC 2013 [00:20:24] RECOVERY - Puppet freshness on cp1010 is OK: puppet ran at Fri Jun 28 00:20:21 UTC 2013 [00:20:33] RECOVERY - Puppet freshness on amslvs1 is OK: puppet ran at Fri Jun 28 00:20:23 UTC 2013 [00:20:34] RECOVERY - Puppet freshness on mw1069 is OK: puppet ran at Fri Jun 28 00:20:24 UTC 2013 [00:20:34] RECOVERY - Puppet freshness on mw1068 is OK: puppet ran at Fri Jun 28 00:20:25 UTC 2013 [00:20:34] RECOVERY - Puppet freshness on mw1153 is OK: puppet ran at Fri Jun 28 00:20:26 UTC 2013 [00:20:34] RECOVERY - Puppet freshness on db63 is OK: puppet ran at Fri Jun 28 00:20:28 UTC 2013 [00:20:34] RECOVERY - Puppet freshness on virt11 is OK: puppet ran at Fri Jun 28 00:20:29 UTC 2013 [00:20:35] RECOVERY - Puppet freshness on srv267 is OK: puppet ran at Fri Jun 28 00:20:30 UTC 2013 [00:20:35] RECOVERY - Puppet freshness on srv262 is OK: puppet ran at Fri Jun 28 00:20:30 UTC 2013 [00:20:36] RECOVERY - Puppet freshness on cp1026 is OK: puppet ran at Fri Jun 28 00:20:31 UTC 2013 [00:20:43] RECOVERY - Puppet freshness on db1034 is OK: puppet ran at Fri Jun 28 00:20:33 UTC 2013 [00:20:44] RECOVERY - Puppet freshness on srv260 is OK: puppet ran at Fri Jun 28 00:20:33 UTC 2013 [00:20:44] RECOVERY - Puppet freshness on solr1002 is OK: puppet ran at Fri Jun 28 00:20:34 UTC 2013 [00:20:44] RECOVERY - Puppet freshness on db54 is OK: puppet ran at Fri Jun 28 00:20:34 UTC 2013 [00:20:44] RECOVERY - Puppet freshness on fluorine is OK: puppet ran at Fri Jun 28 00:20:34 UTC 2013 [00:20:44] RECOVERY - Puppet freshness on srv269 is OK: puppet ran at Fri Jun 28 00:20:35 UTC 2013 [00:20:45] RECOVERY - Puppet freshness on search16 is OK: puppet ran at Fri Jun 28 00:20:36 UTC 2013 [00:20:45] RECOVERY - Puppet freshness on search1022 is OK: puppet ran at Fri Jun 28 00:20:38 UTC 2013 [00:20:46] RECOVERY - Puppet freshness on search25 is OK: puppet ran at Fri Jun 28 00:20:39 UTC 2013 [00:20:46] RECOVERY - Puppet freshness on mw1091 is OK: puppet ran at Fri Jun 28 00:20:39 UTC 2013 [00:20:47] RECOVERY - Puppet freshness on mw68 is OK: puppet ran at Fri Jun 28 00:20:40 UTC 2013 [00:20:53] RECOVERY - Puppet freshness on mw1166 is OK: puppet ran at Fri Jun 28 00:20:42 UTC 2013 [00:20:54] RECOVERY - Puppet freshness on db1033 is OK: puppet ran at Fri Jun 28 00:20:43 UTC 2013 [00:20:54] RECOVERY - Puppet freshness on mw1122 is OK: puppet ran at Fri Jun 28 00:20:44 UTC 2013 [00:20:54] RECOVERY - Puppet freshness on sockpuppet is OK: puppet ran at Fri Jun 28 00:20:46 UTC 2013 [00:20:54] RECOVERY - Puppet freshness on mw2 is OK: puppet ran at Fri Jun 28 00:20:50 UTC 2013 [00:20:54] RECOVERY - Puppet freshness on db1042 is OK: puppet ran at Fri Jun 28 00:20:51 UTC 2013 [00:21:07] RECOVERY - Puppet freshness on mw1024 is OK: puppet ran at Fri Jun 28 00:20:53 UTC 2013 [00:21:07] RECOVERY - Puppet freshness on analytics1004 is OK: puppet ran at Fri Jun 28 00:20:54 UTC 2013 [00:21:07] RECOVERY - Puppet freshness on search1001 is OK: puppet ran at Fri Jun 28 00:20:54 UTC 2013 [00:21:07] RECOVERY - Puppet freshness on mw1010 is OK: puppet ran at Fri Jun 28 00:20:55 UTC 2013 [00:21:07] RECOVERY - Puppet freshness on mw37 is OK: puppet ran at Fri Jun 28 00:20:57 UTC 2013 [00:21:07] RECOVERY - Puppet freshness on mw1092 is OK: puppet ran at Fri Jun 28 00:20:59 UTC 2013 [00:21:07] RECOVERY - Puppet freshness on mw1142 is OK: puppet ran at Fri Jun 28 00:21:00 UTC 2013 [00:21:07] RECOVERY - Puppet freshness on mw1130 is OK: puppet ran at Fri Jun 28 00:21:01 UTC 2013 [00:21:14] RECOVERY - Puppet freshness on mw1118 is OK: puppet ran at Fri Jun 28 00:21:03 UTC 2013 [00:21:14] RECOVERY - Puppet freshness on ssl1001 is OK: puppet ran at Fri Jun 28 00:21:04 UTC 2013 [00:21:14] RECOVERY - Puppet freshness on amssq48 is OK: puppet ran at Fri Jun 28 00:21:05 UTC 2013 [00:21:14] RECOVERY - Puppet freshness on mw104 is OK: puppet ran at Fri Jun 28 00:21:08 UTC 2013 [00:21:14] RECOVERY - Puppet freshness on analytics1003 is OK: puppet ran at Fri Jun 28 00:21:09 UTC 2013 [00:21:15] RECOVERY - Puppet freshness on mw112 is OK: puppet ran at Fri Jun 28 00:21:09 UTC 2013 [00:21:15] RECOVERY - Puppet freshness on virt1000 is OK: puppet ran at Fri Jun 28 00:21:10 UTC 2013 [00:21:16] RECOVERY - Puppet freshness on mw23 is OK: puppet ran at Fri Jun 28 00:21:11 UTC 2013 [00:21:16] RECOVERY - Puppet freshness on amslvs4 is OK: puppet ran at Fri Jun 28 00:21:12 UTC 2013 [00:21:18] New patchset: Dzahn; "add virtual language subdomain redirects for wikidata" [operations/apache-config] (master) - https://gerrit.wikimedia.org/r/65443 [00:21:23] RECOVERY - Puppet freshness on mw116 is OK: puppet ran at Fri Jun 28 00:21:14 UTC 2013 [00:21:24] RECOVERY - Puppet freshness on search30 is OK: puppet ran at Fri Jun 28 00:21:14 UTC 2013 [00:21:24] RECOVERY - Puppet freshness on mw28 is OK: puppet ran at Fri Jun 28 00:21:15 UTC 2013 [00:21:24] RECOVERY - Puppet freshness on mw1204 is OK: puppet ran at Fri Jun 28 00:21:16 UTC 2013 [00:21:24] RECOVERY - Puppet freshness on search1012 is OK: puppet ran at Fri Jun 28 00:21:17 UTC 2013 [00:21:24] RECOVERY - Puppet freshness on mw10 is OK: puppet ran at Fri Jun 28 00:21:17 UTC 2013 [00:21:24] RECOVERY - Puppet freshness on mw1025 is OK: puppet ran at Fri Jun 28 00:21:19 UTC 2013 [00:21:25] RECOVERY - Puppet freshness on mw60 is OK: puppet ran at Fri Jun 28 00:21:20 UTC 2013 [00:21:25] RECOVERY - Puppet freshness on mw1143 is OK: puppet ran at Fri Jun 28 00:21:20 UTC 2013 [00:21:26] RECOVERY - Puppet freshness on mw1213 is OK: puppet ran at Fri Jun 28 00:21:21 UTC 2013 [00:21:26] RECOVERY - Puppet freshness on mw1129 is OK: puppet ran at Fri Jun 28 00:21:22 UTC 2013 [00:21:27] mutante: combo breaker! [00:21:35] RECOVERY - Puppet freshness on mw1211 is OK: puppet ran at Fri Jun 28 00:21:22 UTC 2013 [00:21:35] RECOVERY - Puppet freshness on sq52 is OK: puppet ran at Fri Jun 28 00:21:23 UTC 2013 [00:21:35] RECOVERY - Puppet freshness on mw1027 is OK: puppet ran at Fri Jun 28 00:21:24 UTC 2013 [00:21:35] RECOVERY - Puppet freshness on labstore1001 is OK: puppet ran at Fri Jun 28 00:21:25 UTC 2013 [00:21:35] RECOVERY - Puppet freshness on srv249 is OK: puppet ran at Fri Jun 28 00:21:26 UTC 2013 [00:21:35] RECOVERY - Puppet freshness on mc1012 is OK: puppet ran at Fri Jun 28 00:21:26 UTC 2013 [00:21:36] RECOVERY - Puppet freshness on dataset1001 is OK: puppet ran at Fri Jun 28 00:21:27 UTC 2013 [00:21:36] RECOVERY - Puppet freshness on labstore3 is OK: puppet ran at Fri Jun 28 00:21:28 UTC 2013 [00:21:37] RECOVERY - Puppet freshness on db1014 is OK: puppet ran at Fri Jun 28 00:21:29 UTC 2013 [00:21:37] RECOVERY - Puppet freshness on antimony is OK: puppet ran at Fri Jun 28 00:21:30 UTC 2013 [00:21:43] RECOVERY - Puppet freshness on mw1054 is OK: puppet ran at Fri Jun 28 00:21:35 UTC 2013 [00:21:43] RECOVERY - Puppet freshness on mw1066 is OK: puppet ran at Fri Jun 28 00:21:37 UTC 2013 [00:21:43] RECOVERY - Puppet freshness on mw1011 is OK: puppet ran at Fri Jun 28 00:21:39 UTC 2013 [00:21:43] RECOVERY - Puppet freshness on search29 is OK: puppet ran at Fri Jun 28 00:21:40 UTC 2013 [00:21:44] RECOVERY - Puppet freshness on mw1107 is OK: puppet ran at Fri Jun 28 00:21:41 UTC 2013 [00:21:44] RECOVERY - Puppet freshness on cp3011 is OK: puppet ran at Fri Jun 28 00:21:41 UTC 2013 [00:22:04] RECOVERY - Puppet freshness on db1004 is OK: puppet ran at Fri Jun 28 00:22:01 UTC 2013 [00:22:13] RECOVERY - Puppet freshness on amssq53 is OK: puppet ran at Fri Jun 28 00:22:06 UTC 2013 [00:22:13] RECOVERY - Puppet freshness on titanium is OK: puppet ran at Fri Jun 28 00:22:09 UTC 2013 [00:22:23] RECOVERY - Puppet freshness on gadolinium is OK: puppet ran at Fri Jun 28 00:22:13 UTC 2013 [00:22:43] RECOVERY - Puppet freshness on mc1 is OK: puppet ran at Fri Jun 28 00:22:37 UTC 2013 [00:22:44] RECOVERY - Puppet freshness on mw1188 is OK: puppet ran at Fri Jun 28 00:22:38 UTC 2013 [00:22:52] ori-l: :)) [00:22:54] RECOVERY - Puppet freshness on mw1056 is OK: puppet ran at Fri Jun 28 00:22:49 UTC 2013 [00:23:34] RECOVERY - Puppet freshness on db1051 is OK: puppet ran at Fri Jun 28 00:23:27 UTC 2013 [00:23:34] RECOVERY - Puppet freshness on sq41 is OK: puppet ran at Fri Jun 28 00:23:28 UTC 2013 [00:36:34] RECOVERY - Puppet freshness on mw8 is OK: puppet ran at Fri Jun 28 00:36:23 UTC 2013 [00:46:50] New patchset: Ryan Lane; "Add support for adding salt grains via puppet" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/70954 [00:49:14] New patchset: Ryan Lane; "Add support for adding salt grains via puppet" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/70954 [00:52:54] New patchset: Ryan Lane; "Add support for adding salt grains via puppet" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/70954 [00:56:12] !log updated Parsoid to 68a086a [00:56:21] Logged the message, Master [00:57:51] Ryan_Lane: list(set(grain_ret + value)) ? [00:58:23] combine two lists, make them a set (to make them unique), make it a list [00:58:31] is there a better way of doing that? [00:59:12] setval only takes lists, not iterables? [00:59:31] I only want it to take a list [00:59:46] hm [00:59:50] let me try it with iterables [01:00:27] doing it this way ensures I'm setting what I want [01:01:24] TypeError: can't serialize set(['test', 'test2']) [01:01:54] RECOVERY - NTP on ssl3003 is OK: NTP OK: Offset -0.001705884933 secs [01:03:01] hrm, maybe this: [01:03:25] value = caller.function('grains.get', grain, default=[]) [01:03:36] if sys.argv[2] not in value: [01:03:44] value.append(sys.argv[2]) [01:03:54] RECOVERY - NTP on ssl3002 is OK: NTP OK: Offset -0.001613974571 secs [01:03:59] caller.function('grains.setval', grain, value) [01:06:24] oh. wow. just found a nasty bug [01:06:47] seems you can set a bad value, then it can't handle the grain anymore [01:07:19] I'd imagine that's really only doable when injecting values via the api [01:09:23] PROBLEM - Puppet freshness on lvs1004 is CRITICAL: No successful Puppet run in the last 10 hours [01:09:23] PROBLEM - Puppet freshness on lvs1005 is CRITICAL: No successful Puppet run in the last 10 hours [01:09:24] PROBLEM - Puppet freshness on erzurumi is CRITICAL: No successful Puppet run in the last 10 hours [01:09:24] PROBLEM - Puppet freshness on mc15 is CRITICAL: No successful Puppet run in the last 10 hours [01:09:24] PROBLEM - Puppet freshness on lvs1006 is CRITICAL: No successful Puppet run in the last 10 hours [01:09:24] PROBLEM - Puppet freshness on sodium is CRITICAL: No successful Puppet run in the last 10 hours [01:09:24] PROBLEM - Puppet freshness on ms-fe3001 is CRITICAL: No successful Puppet run in the last 10 hours [01:09:25] PROBLEM - Puppet freshness on spence is CRITICAL: No successful Puppet run in the last 10 hours [01:09:25] PROBLEM - Puppet freshness on virt1 is CRITICAL: No successful Puppet run in the last 10 hours [01:09:26] PROBLEM - Puppet freshness on virt4 is CRITICAL: No successful Puppet run in the last 10 hours [01:09:26] PROBLEM - Puppet freshness on virt3 is CRITICAL: No successful Puppet run in the last 10 hours [01:12:30] PROBLEM - Puppet freshness on celsus is CRITICAL: No successful Puppet run in the last 10 hours [01:19:16] heh [01:19:27] seems it returns an empty string, even if you set a default [01:21:03] hm. I really need to add a remove option to the script too [01:21:08] I guess I should use argparse... [01:21:46] bleh. I'll add that later [01:23:05] New patchset: Ryan Lane; "Have a target class for deployment to set grain" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/70957 [01:29:45] New patchset: Ryan Lane; "Add support for adding salt grains via puppet" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/70954 [01:29:47] ori-l: ^^ [01:31:33] New patchset: Ryan Lane; "Add support for adding salt grains via puppet" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/70954 [01:31:36] I really need to remember to run pep8 locally [01:32:51] PROBLEM - LVS HTTPS IPv6 on wikipedia-lb.eqiad.wikimedia.org_ipv6 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:33:01] PROBLEM - LVS HTTPS IPv6 on mediawiki-lb.eqiad.wikimedia.org_ipv6 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:33:20] PROBLEM - LVS HTTPS IPv6 on wiktionary-lb.eqiad.wikimedia.org_ipv6 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:33:40] RECOVERY - LVS HTTPS IPv6 on wikipedia-lb.eqiad.wikimedia.org_ipv6 is OK: HTTP OK: HTTP/1.1 200 OK - 61881 bytes in 0.019 second response time [01:33:50] RECOVERY - LVS HTTPS IPv6 on mediawiki-lb.eqiad.wikimedia.org_ipv6 is OK: HTTP OK: HTTP/1.1 200 OK - 61881 bytes in 0.014 second response time [01:34:12] RECOVERY - LVS HTTPS IPv6 on wiktionary-lb.eqiad.wikimedia.org_ipv6 is OK: HTTP OK: HTTP/1.1 200 OK - 61881 bytes in 0.023 second response time [01:52:05] Ryan_Lane: i have two comments for PS1 that i forgot to submit, will do so now, but they're of the take-it-or-leave it variety; lgtm otherwise [01:53:05] I was being lazy by not using argparse ;) [01:53:08] I generally do [02:07:05] !log LocalisationUpdate completed (1.22wmf8) at Fri Jun 28 02:07:05 UTC 2013 [02:07:15] Logged the message, Master [02:10:26] !log LocalisationUpdate completed (1.22wmf9) at Fri Jun 28 02:10:26 UTC 2013 [02:10:35] Logged the message, Master [02:18:38] !log LocalisationUpdate ResourceLoader cache refresh completed at Fri Jun 28 02:18:38 UTC 2013 [02:18:48] Logged the message, Master [02:23:47] New patchset: Ryan Lane; "Add support for adding salt grains via puppet" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/70954 [02:23:54] ori-l: ^^ with argparse, and a purge option [03:06:46] PROBLEM - Puppet freshness on manutius is CRITICAL: No successful Puppet run in the last 10 hours [03:08:29] PROBLEM - Puppet freshness on celsus is CRITICAL: No successful Puppet run in the last 10 hours [03:12:04] Change merged: Ryan Lane; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/70954 [03:19:00] New patchset: Ryan Lane; "Have a target class for deployment to set grain" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/70957 [03:21:28] New review: MZMcBride; "(1 comment)" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/69982 [03:23:27] New patchset: Ryan Lane; "Add system role and deployment target for parsoid" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/70978 [03:23:29] RoanKattouw: ^^ I have a treat for you :D [03:24:02] targeting of git-deploy destinations via grains [03:24:42] Change merged: Ryan Lane; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/70957 [03:25:54] Yay [03:26:12] Ryan_Lane: BTW, I've been meaning to ask: the decommissioned boxes are still git-deploy targets somehow [03:26:29] well, they still exist in redis [03:26:58] Wait, redis is involved too? ... [03:27:08] do they just show as targets that aren't getting deployed to? [03:27:20] or are they succeeding? [03:28:02] the deployment system reports back to redis on tin [03:28:16] which is why you can get deployment info at any time [03:29:36] Syntax error at '}'; expected '} [03:29:38] * Ryan_Lane sighs [03:30:29] They show as hanging targets [03:30:38] that makes sense [03:30:41] Like, stated hundreds of minutes ago [03:30:49] I currently don't purge redis at all for removed nodes [03:30:53] I checked that the regexes were updated [03:31:06] The only matches for the decommissioned hosts in puppet are in the lists of decommissioned hosts [03:31:25] I guess I should change how I'm writing to redis so that it is based on the tag, rather than just looking at the nodes [03:32:01] basically, the destination writes to redis saying "here's my info" [03:32:06] it overwrites the old info [03:32:26] if a host doesn't return, it shows old data, which is somewhat good [03:32:40] but if a host is removed from deployment, it needs to also be removed from redis [03:33:55] at minimum, I should add an option to the reporting script to purge hosts [03:34:30] New patchset: Ryan Lane; "Add system role and deployment target for parsoid" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/70978 [03:36:10] Change merged: Ryan Lane; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/70978 [03:41:00] salt -G 'deployment_target:parsoid' test.ping [03:41:01] \o/ [03:41:04] that's working [03:41:38] Whee [03:42:04] now to make targets for the rest and get rid of the regexes :) [03:53:21] New patchset: Ryan Lane; "Add an eventlogging deployment target" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/70980 [04:00:34] Change merged: Ryan Lane; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/70980 [04:21:47] !log catrope Started syncing Wikimedia installation... : Updating VisualEditor to master [04:21:56] Logged the message, Master [04:22:44] RECOVERY - Puppet freshness on mw1081 is OK: puppet ran at Fri Jun 28 04:22:39 UTC 2013 [04:27:53] PROBLEM - LVS HTTP IPv4 on rendering.svc.eqiad.wmnet is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 MediaWiki exception - 612 bytes in 0.181 second response time [04:30:15] !log catrope Finished syncing Wikimedia installation... : Updating VisualEditor to master [04:30:25] Logged the message, Master [04:30:54] PROBLEM - LVS HTTP IPv4 on appservers.svc.eqiad.wmnet is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 MediaWiki exception - 612 bytes in 0.167 second response time [04:31:16] RoanKattuow: I think you broke the wiki [04:31:25] Jasper_Deng: Howso? [04:31:26] PROBLEM - LVS HTTP IPv4 on appservers.svc.pmtpa.wmnet is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 MediaWiki exception - 612 bytes in 0.203 second response time [04:31:28] PROBLEM - Apache HTTP on mw1041 is CRITICAL: Connection timed out [04:31:29] see -tech [04:31:36] PROBLEM - LVS HTTP IPv4 on rendering.svc.pmtpa.wmnet is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 MediaWiki exception - 612 bytes in 0.236 second response time [04:31:37] https://en.wikipedia.org/w/index.php?title=Main_Page&action=edit returns an error. [04:31:56] Crap [04:32:02] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:32:31] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.132 second response time [04:33:37] RoanKattouw: Ryan is sitting next to me [04:33:38] Site is broken, I'm on it [04:33:46] MW is throwing exceptions [04:34:03] GeoData [04:35:07] coordinates [04:35:28] Yeah, wtf [04:35:35] It didn't change recently [04:35:42] RoanKattouw: mustve been undeployed changes that got in your scap [04:35:51] rollback? [04:35:55] WC is clean [04:36:02] Rollback to what? [04:36:19] lets see whats in the history [04:36:23] What was synced? [04:36:41] Oh, [04:36:43] Localizaiton [04:36:52] The exception is about a magic word [04:37:09] Rollback isn't feasible [04:37:15] I have to find it and fix it [04:37:51] * Jasper_Deng guesses that means RoanKattouw can't simply revert the change [04:38:15] It's not my change that caused it [04:38:22] It's something older [04:38:27] The cache rebuild caused it to fail [04:39:19] Aha! [04:39:21] Wikibase is at fault [04:39:27] Wikibase caused a fatal error during the i18n rebuild [04:39:42] The Germans strike again. [04:39:50] hahaha [04:39:55] OK [04:39:59] I know how to fix this [04:40:12] :/ [04:41:13] RoanKattouw: What, its on special login as well [04:42:06] action=history [04:42:20] PROBLEM - LVS HTTP IPv6 on wikiquote-lb.esams.wikimedia.org_ipv6 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 1050 bytes in 0.645 second response time [04:42:30] PROBLEM - LVS HTTP IPv6 on wiktionary-lb.esams.wikimedia.org_ipv6 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 1050 bytes in 0.621 second response time [04:42:30] PROBLEM - LVS HTTP IPv4 on wikipedia-lb.esams.wikimedia.org is CRITICAL: HTTP CRITICAL: HTTP/1.0 500 Internal Server Error - 863 bytes in 0.635 second response time [04:42:30] PROBLEM - LVS HTTPS IPv6 on foundation-lb.esams.wikimedia.org_ipv6 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 1050 bytes in 0.897 second response time [04:42:30] PROBLEM - LVS HTTPS IPv6 on wikinews-lb.esams.wikimedia.org_ipv6 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 1050 bytes in 0.899 second response time [04:42:30] PROBLEM - LVS HTTPS IPv4 on wikipedia-lb.esams.wikimedia.org is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 1050 bytes in 0.890 second response time [04:42:31] PROBLEM - LVS HTTPS IPv6 on wikiversity-lb.esams.wikimedia.org_ipv6 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 1050 bytes in 0.892 second response time [04:42:31] PROBLEM - LVS HTTPS IPv6 on wikiquote-lb.esams.wikimedia.org_ipv6 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 1050 bytes in 0.895 second response time [04:42:32] PROBLEM - LVS HTTPS IPv6 on wikibooks-lb.esams.wikimedia.org_ipv6 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 1050 bytes in 0.921 second response time [04:42:32] PROBLEM - LVS HTTPS IPv6 on wikipedia-lb.esams.wikimedia.org_ipv6 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 1050 bytes in 0.919 second response time [04:42:33] PROBLEM - LVS HTTPS IPv6 on wikisource-lb.esams.wikimedia.org_ipv6 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 1050 bytes in 0.939 second response time [04:42:33] PROBLEM - LVS HTTP IPv4 on wikipedia-lb.pmtpa.wikimedia.org is CRITICAL: HTTP CRITICAL: HTTP/1.0 500 Internal Server Error - 737 bytes in 0.254 second response time [04:42:34] PROBLEM - LVS HTTPS IPv4 on wikipedia-lb.pmtpa.wikimedia.org is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 857 bytes in 0.357 second response time [04:42:34] PROBLEM - LVS HTTP IPv6 on wikinews-lb.pmtpa.wikimedia.org_ipv6 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 857 bytes in 0.444 second response time [04:42:35] PROBLEM - LVS HTTP IPv6 on wikipedia-lb.pmtpa.wikimedia.org_ipv6 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 857 bytes in 0.463 second response time [04:42:35] PROBLEM - LVS HTTP IPv6 on wikiquote-lb.pmtpa.wikimedia.org_ipv6 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 857 bytes in 0.479 second response time [04:42:36] PROBLEM - LVS HTTP IPv6 on wikiversity-lb.pmtpa.wikimedia.org_ipv6 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 857 bytes in 0.481 second response time [04:42:36] PROBLEM - LVS HTTP IPv6 on wiktionary-lb.pmtpa.wikimedia.org_ipv6 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 857 bytes in 0.511 second response time [04:42:36] PROBLEM - LVS HTTP IPv6 on wikipedia-lb.esams.wikimedia.org_ipv6 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 1050 bytes in 0.649 second response time [04:42:37] PROBLEM - LVS HTTPS IPv6 on wikinews-lb.pmtpa.wikimedia.org_ipv6 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 857 bytes in 0.356 second response time [04:42:37] PROBLEM - LVS HTTPS IPv6 on wikiversity-lb.pmtpa.wikimedia.org_ipv6 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 857 bytes in 0.372 second response time [04:42:38] PROBLEM - LVS HTTPS IPv6 on wikiquote-lb.pmtpa.wikimedia.org_ipv6 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 857 bytes in 0.390 second response time [04:42:39] PROBLEM - LVS HTTP IPv6 on wikibooks-lb.esams.wikimedia.org_ipv6 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 1050 bytes in 0.601 second response time [04:42:39] PROBLEM - LVS HTTP IPv6 on wikisource-lb.esams.wikimedia.org_ipv6 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 1050 bytes in 0.604 second response time [04:42:40] PROBLEM - LVS HTTP IPv6 on mediawiki-lb.esams.wikimedia.org_ipv6 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 1050 bytes in 0.647 second response time [04:42:40] PROBLEM - LVS HTTP IPv6 on wikiversity-lb.esams.wikimedia.org_ipv6 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 1050 bytes in 0.651 second response time [04:42:41] PROBLEM - LVS HTTP IPv6 on wikisource-lb.pmtpa.wikimedia.org_ipv6 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 857 bytes in 5.713 second response time [04:42:41] PROBLEM - LVS HTTPS IPv6 on mediawiki-lb.esams.wikimedia.org_ipv6 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 1050 bytes in 0.929 second response time [04:42:42] PROBLEM - LVS HTTPS IPv6 on wiktionary-lb.esams.wikimedia.org_ipv6 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 1050 bytes in 0.911 second response time [04:42:42] PROBLEM - LVS HTTP IPv6 on mediawiki-lb.pmtpa.wikimedia.org_ipv6 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 857 bytes in 0.510 second response time [04:42:43] PROBLEM - LVS HTTP IPv6 on foundation-lb.esams.wikimedia.org_ipv6 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 1050 bytes in 0.646 second response time [04:43:01] and there's LVS depooling everything [04:43:10] PROBLEM - LVS HTTP IPv6 on wikinews-lb.esams.wikimedia.org_ipv6 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 1050 bytes in 0.626 second response time [04:43:20] PROBLEM - LVS HTTPS IPv6 on wikipedia-lb.pmtpa.wikimedia.org_ipv6 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 857 bytes in 0.352 second response time [04:43:23] PROBLEM - LVS HTTPS IPv6 on mediawiki-lb.pmtpa.wikimedia.org_ipv6 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 857 bytes in 0.365 second response time [04:43:23] PROBLEM - LVS HTTPS IPv6 on wiktionary-lb.pmtpa.wikimedia.org_ipv6 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 857 bytes in 0.380 second response time [04:43:23] PROBLEM - LVS HTTPS IPv6 on wikisource-lb.pmtpa.wikimedia.org_ipv6 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 857 bytes in 0.340 second response time [04:43:23] PROBLEM - LVS HTTPS IPv6 on foundation-lb.pmtpa.wikimedia.org_ipv6 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 857 bytes in 0.368 second response time [04:43:23] PROBLEM - LVS HTTPS IPv6 on wikibooks-lb.pmtpa.wikimedia.org_ipv6 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 857 bytes in 0.373 second response time [04:43:24] PROBLEM - LVS HTTP IPv6 on wikibooks-lb.pmtpa.wikimedia.org_ipv6 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 857 bytes in 0.565 second response time [04:43:24] PROBLEM - LVS HTTP IPv4 on mobile-lb.eqiad.wikimedia.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:43:25] PROBLEM - LVS HTTP IPv6 on mobile-lb.eqiad.wikimedia.org_ipv6 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:43:34] PROBLEM - LVS HTTPS IPv4 on mobile-lb.eqiad.wikimedia.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:43:36] PROBLEM - LVS HTTP IPv6 on foundation-lb.pmtpa.wikimedia.org_ipv6 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 857 bytes in 0.535 second response time [04:44:16] action=view too, for logged in users [04:44:20] RoanKattouw: so, yeah, if you have a fix, let's do it [04:44:22] we're 100% down now [04:44:53] I'm already deploying it [04:44:53] Damn it, it got cached [04:45:03] Reedy added WikibaseDataWhatever to wmf9 but not wmf8 [04:45:06] !log catrope synchronized wmf-config/ExtensionMessages-1.22wmf8.php [04:45:07] RECOVERY - LVS HTTP IPv4 on mobile-lb.eqiad.wikimedia.org is OK: HTTP OK: HTTP/1.1 200 OK - 22434 bytes in 0.002 second response time [04:45:08] or rather the cache was purged [04:45:15] of enwiki main page [04:45:16] Logged the message, Master [04:45:22] which is now showing 500 for anons [04:45:50] i'm not getting the 500 [04:46:04] huh, now i am [04:46:26] !log Running scap in an attempt to fix things [04:46:34] must depend which squid i hit or something [04:46:35] Logged the message, Mr. Obvious [04:46:38] RECOVERY - LVS HTTP IPv6 on wikinews-lb.esams.wikimedia.org_ipv6 is OK: HTTP OK: HTTP/1.1 200 OK - 62966 bytes in 0.351 second response time [04:46:39] RECOVERY - LVS HTTPS IPv6 on wikisource-lb.pmtpa.wikimedia.org_ipv6 is OK: HTTP OK: HTTP/1.1 200 OK - 62775 bytes in 0.229 second response time [04:46:39] RECOVERY - LVS HTTPS IPv6 on mediawiki-lb.pmtpa.wikimedia.org_ipv6 is OK: HTTP OK: HTTP/1.1 200 OK - 62775 bytes in 0.263 second response time [04:46:39] RECOVERY - LVS HTTPS IPv6 on wikibooks-lb.pmtpa.wikimedia.org_ipv6 is OK: HTTP OK: HTTP/1.1 200 OK - 62775 bytes in 0.245 second response time [04:46:39] RECOVERY - LVS HTTPS IPv6 on wikipedia-lb.pmtpa.wikimedia.org_ipv6 is OK: HTTP OK: HTTP/1.1 200 OK - 62775 bytes in 0.249 second response time [04:46:47] !log catrope Started syncing Wikimedia installation... : Trying to fix the site [04:46:55] Logged the message, Master [04:47:01] RECOVERY - LVS HTTPS IPv6 on foundation-lb.pmtpa.wikimedia.org_ipv6 is OK: HTTP OK: HTTP/1.1 200 OK - 62775 bytes in 0.243 second response time [04:47:01] RECOVERY - LVS HTTPS IPv6 on wiktionary-lb.pmtpa.wikimedia.org_ipv6 is OK: HTTP OK: HTTP/1.1 200 OK - 62775 bytes in 0.245 second response time [04:47:02] RECOVERY - LVS HTTP IPv6 on wikibooks-lb.pmtpa.wikimedia.org_ipv6 is OK: HTTP OK: HTTP/1.1 200 OK - 62775 bytes in 0.142 second response time [04:47:02] RECOVERY - LVS HTTPS IPv6 on foundation-lb.esams.wikimedia.org_ipv6 is OK: HTTP OK: HTTP/1.1 200 OK - 62966 bytes in 1.193 second response time [04:47:02] RECOVERY - LVS HTTP IPv6 on foundation-lb.pmtpa.wikimedia.org_ipv6 is OK: HTTP OK: HTTP/1.1 200 OK - 62775 bytes in 0.142 second response time [04:47:02] RECOVERY - LVS HTTP IPv6 on wikiquote-lb.esams.wikimedia.org_ipv6 is OK: HTTP OK: HTTP/1.1 200 OK - 62966 bytes in 0.437 second response time [04:47:02] RECOVERY - LVS HTTP IPv4 on wikipedia-lb.esams.wikimedia.org is OK: HTTP OK: HTTP/1.0 200 OK - 62508 bytes in 0.358 second response time [04:47:02] RECOVERY - LVS HTTP IPv6 on wiktionary-lb.esams.wikimedia.org_ipv6 is OK: HTTP OK: HTTP/1.1 200 OK - 62966 bytes in 0.432 second response time [04:47:03] RECOVERY - LVS HTTPS IPv6 on wikinews-lb.esams.wikimedia.org_ipv6 is OK: HTTP OK: HTTP/1.1 200 OK - 62966 bytes in 0.757 second response time [04:47:12] RECOVERY - LVS HTTPS IPv6 on wikibooks-lb.esams.wikimedia.org_ipv6 is OK: HTTP OK: HTTP/1.1 200 OK - 62966 bytes in 0.730 second response time [04:47:12] RECOVERY - LVS HTTPS IPv4 on wikipedia-lb.esams.wikimedia.org is OK: HTTP OK: HTTP/1.1 200 OK - 62966 bytes in 0.741 second response time [04:47:12] RECOVERY - LVS HTTP IPv6 on wikinews-lb.pmtpa.wikimedia.org_ipv6 is OK: HTTP OK: HTTP/1.1 200 OK - 62775 bytes in 0.150 second response time [04:47:12] RECOVERY - LVS HTTPS IPv4 on wikipedia-lb.pmtpa.wikimedia.org is OK: HTTP OK: HTTP/1.1 200 OK - 62775 bytes in 0.228 second response time [04:47:12] RECOVERY - LVS HTTP IPv6 on wikipedia-lb.pmtpa.wikimedia.org_ipv6 is OK: HTTP OK: HTTP/1.1 200 OK - 62775 bytes in 0.140 second response time [04:47:45] RECOVERY - LVS HTTP IPv6 on wikiversity-lb.pmtpa.wikimedia.org_ipv6 is OK: HTTP OK: HTTP/1.1 200 OK - 62775 bytes in 0.145 second response time [04:47:45] RECOVERY - LVS HTTP IPv6 on wikiquote-lb.pmtpa.wikimedia.org_ipv6 is OK: HTTP OK: HTTP/1.1 200 OK - 62775 bytes in 0.136 second response time [04:47:45] RECOVERY - LVS HTTP IPv6 on wikisource-lb.pmtpa.wikimedia.org_ipv6 is OK: HTTP OK: HTTP/1.1 200 OK - 62775 bytes in 0.139 second response time [04:47:45] RECOVERY - LVS HTTP IPv6 on wiktionary-lb.pmtpa.wikimedia.org_ipv6 is OK: HTTP OK: HTTP/1.1 200 OK - 62775 bytes in 0.157 second response time [04:47:45] RECOVERY - LVS HTTP IPv6 on wikipedia-lb.esams.wikimedia.org_ipv6 is OK: HTTP OK: HTTP/1.1 200 OK - 62966 bytes in 0.440 second response time [04:47:47] RECOVERY - LVS HTTPS IPv6 on wikiversity-lb.pmtpa.wikimedia.org_ipv6 is OK: HTTP OK: HTTP/1.1 200 OK - 62775 bytes in 0.229 second response time [04:47:47] RECOVERY - LVS HTTPS IPv6 on wikinews-lb.pmtpa.wikimedia.org_ipv6 is OK: HTTP OK: HTTP/1.1 200 OK - 62775 bytes in 0.246 second response time [04:47:47] RECOVERY - LVS HTTPS IPv6 on wikiquote-lb.pmtpa.wikimedia.org_ipv6 is OK: HTTP OK: HTTP/1.1 200 OK - 62775 bytes in 0.236 second response time [04:47:47] RECOVERY - LVS HTTP IPv6 on mediawiki-lb.esams.wikimedia.org_ipv6 is OK: HTTP OK: HTTP/1.1 200 OK - 62967 bytes in 0.356 second response time [04:47:47] RECOVERY - LVS HTTP IPv6 on wikiversity-lb.esams.wikimedia.org_ipv6 is OK: HTTP OK: HTTP/1.1 200 OK - 62966 bytes in 0.446 second response time [04:47:48] RECOVERY - LVS HTTP IPv6 on wikibooks-lb.esams.wikimedia.org_ipv6 is OK: HTTP OK: HTTP/1.1 200 OK - 62966 bytes in 0.450 second response time [04:47:48] RECOVERY - LVS HTTP IPv6 on wikisource-lb.esams.wikimedia.org_ipv6 is OK: HTTP OK: HTTP/1.1 200 OK - 62966 bytes in 0.453 second response time [04:47:49] RECOVERY - LVS HTTP IPv6 on mediawiki-lb.pmtpa.wikimedia.org_ipv6 is OK: HTTP OK: HTTP/1.1 200 OK - 62775 bytes in 0.160 second response time [04:47:49] RECOVERY - LVS HTTP IPv6 on foundation-lb.esams.wikimedia.org_ipv6 is OK: HTTP OK: HTTP/1.1 200 OK - 62966 bytes in 0.391 second response time [04:47:50] RECOVERY - LVS HTTPS IPv6 on mediawiki-lb.esams.wikimedia.org_ipv6 is OK: HTTP OK: HTTP/1.1 200 OK - 62966 bytes in 0.712 second response time [04:47:50] RECOVERY - LVS HTTP IPv4 on wikipedia-lb.pmtpa.wikimedia.org is OK: HTTP OK: HTTP/1.0 200 OK - 62384 bytes in 0.141 second response time [04:47:51] RECOVERY - LVS HTTPS IPv6 on wiktionary-lb.esams.wikimedia.org_ipv6 is OK: HTTP OK: HTTP/1.1 200 OK - 62966 bytes in 0.727 second response time [04:47:51] RECOVERY - LVS HTTPS IPv6 on wikipedia-lb.esams.wikimedia.org_ipv6 is OK: HTTP OK: HTTP/1.1 200 OK - 62967 bytes in 0.708 second response time [04:47:55] RECOVERY - LVS HTTPS IPv6 on wikisource-lb.esams.wikimedia.org_ipv6 is OK: HTTP OK: HTTP/1.1 200 OK - 62966 bytes in 0.712 second response time [04:47:55] RECOVERY - LVS HTTPS IPv6 on wikiquote-lb.esams.wikimedia.org_ipv6 is OK: HTTP OK: HTTP/1.1 200 OK - 62968 bytes in 0.699 second response time [04:47:55] RECOVERY - LVS HTTPS IPv6 on wikiversity-lb.esams.wikimedia.org_ipv6 is OK: HTTP OK: HTTP/1.1 200 OK - 62966 bytes in 0.751 second response time [04:48:14] how serious is the problem? Fatal! [04:48:59] Cause is a i18n problem in the wikidata i18n sync [04:49:05] PROBLEM - LVS HTTP IPv4 on mobile-lb.eqiad.wikimedia.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:49:08] Roan's reverting [04:49:18] No, I didn't revert, I fixed it [04:49:19] RoanKattouw: ? [04:49:34] LeslieCarr: Reedy deployed a WikiData thing and set things up so that the next deploy after that would explore horribly [04:49:38] germans are just waking up in a few mins i guess [04:49:47] explode* :) [04:49:48] ok [04:49:49] it's not fixed [04:49:56] The sync is still ongoing. [04:49:57] 2013-06-28 04:49:44 mw1050 dewiki: [cdca2cd6] /wiki/Ino_%28Mythologie%29 Exception from line 315 of /usr/local/apache/common-local/php-1.22wmf8/includes/MagicWord.php: Error: invalid magic word 'coordinates' [04:50:00] Or in the process of being fixed [04:50:02] Yeah [04:50:03] we're getting a lot of these [04:50:03] We're getting lots of those [04:50:07] what sync? [04:50:09] ok, i'm going to let you revert this [04:50:11] PROBLEM - LVS HTTPS IPv6 on mobile-lb.eqiad.wikimedia.org_ipv6 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:50:12] scap [04:50:14] See SAL [04:50:17] TimStarling: At :46 in here. [04:50:20] !log catrope Finished syncing Wikimedia installation... : Trying to fix the site [04:50:29] Logged the message, Master [04:50:34] Because of something someone else did earlier today, the l10n build for wmf8 broke horribly [04:50:44] Doesn't seem to be fixed now that the sync is finished. [04:50:45] Crap, the magic word thing is still broken [04:50:58] revert to wmf7? [04:51:22] mwalker: We've been wmf8 for the entire cluster for 12 hours without incident. [04:51:31] -rw-rw-r-- 1 reedy wikidev 0 Jun 28 04:46 ExtensionMessages-1.22wmf8.php [04:51:33] WTF [04:51:46] Updating ExtensionMessages-1.22wmf8.php...Unable to read /a/common/php-1.22wmf8/extensions/WikibaseDataModel/WikibaseDataModel.php [04:51:48] But the file is readable [04:51:59] <20 minutes ago> Krinkle: RoanKattouw: Ryan is sitting next to me [04:51:59] <19 minutes ago> RoanKattouw: MW is throwing exceptions [04:51:59] <18 minutes ago> Krinkle: GeoData [04:52:00] OK I'm removing that extension from extension-list [04:52:00] <18 minutes ago> Krinkle: coordinates [04:52:47] I'm just pushing out the one from wmf9 [04:52:47] ExtensionMessages-wmf8 now built [04:52:47] !log tstarling synchronized wmf-config/ExtensionMessages-1.22wmf9.php [04:52:56] Logged the message, Master [04:52:59] !log catrope synchronized wmf-config/ExtensionMessages-1.22wmf8.php [04:53:08] I guess I synced the wrong one though [04:53:09] !log About 20 minutes ago the site went down for logged in users (and anon users hitting an uncached page) throwing HTTP 500 [04:53:17] Logged the message, Master [04:53:52] !log Cause from GeoData "coordinates" magic word. Deployed through broken i18n stuff from WikibaseDataModel deployment earlier that caused fatal during the scap just now [04:54:01] Logged the message, Master [04:54:20] !log catrope Removing that extension from extension-list [04:54:20] !log sync-dir php-1.22wmf8/cache/l10n [04:54:21] PROBLEM - search indices - check lucene status page on search1019 is CRITICAL: HTTP CRITICAL: HTTP/1.1 200 OK - pattern found - 59141 bytes in 0.015 second response time [04:54:22] PROBLEM - Parsoid on wtp1008 is CRITICAL: Connection refused [04:54:29] Logged the message, Master [04:54:37] Logged the message, Mr. Obvious [04:54:52] =D [04:55:21] RECOVERY - LVS HTTP IPv4 on appservers.svc.pmtpa.wmnet is OK: HTTP OK: HTTP/1.1 200 OK - 62491 bytes in 0.324 second response time [04:55:26] RoanKattouw: Removed GeoData or WikibaseDataModel ? [04:55:27] I'm attempting to just disable geodata [04:55:42] Krinkle: WDM [04:55:53] I'm rebuilding the l10n cache for wmf8 and syncing it [04:56:13] What I'm doing is not super healthy because I accidentally ran both at the same time [04:56:27] But it's gone past English so if this fixes anything it'll take care of a lot [04:57:06] !log tstarling synchronized wmf-config/CommonSettings.php 'disable GeoData' [04:57:15] Logged the message, Master [04:57:21] RECOVERY - LVS HTTP IPv4 on rendering.svc.pmtpa.wmnet is OK: HTTP OK: HTTP/1.1 200 OK - 62491 bytes in 0.309 second response time [04:57:30] And now it's a different magic word that's throwing errors [04:57:38] I think this is because the l10n cache borked itself [04:57:53] ok, lets go back to wmf7 then [04:58:56] Hmm, hang on [04:59:00] I think my sync may be starting to hit eqiad [04:59:09] !log reverting to 1.22wmf7 [04:59:13] Look at http://ganglia.wikimedia.org/latest/graph.php?r=hour&z=xlarge&title=MediaWiki+errors&vl=errors+%2F+sec&n=&hreg[]=vanadium.eqiad.wmnet&mreg[]=fatal|exception>ype=stack&glegend=show&aggregate=1&embed=1 , it's dropping [04:59:14] my sync was running already when you said hang on [04:59:17] Logged the message, Master [04:59:18] Oh, aha [05:00:59] why does it take so long to sync two files? [05:01:06] RoanKattouw: Increasing again [05:01:21] RECOVERY - LVS HTTP IPv4 on mobile-lb.eqiad.wikimedia.org is OK: HTTP OK: HTTP/1.1 200 OK - 22456 bytes in 0.006 second response time [05:01:29] TimStarling: Lemme kill my l10n sync [05:01:31] RECOVERY - LVS HTTPS IPv4 on mobile-lb.eqiad.wikimedia.org is OK: HTTP OK: HTTP/1.1 200 OK - 22501 bytes in 5.349 second response time [05:01:37] RECOVERY - LVS HTTP IPv6 on mobile-lb.eqiad.wikimedia.org_ipv6 is OK: HTTP OK: HTTP/1.1 200 OK - 22460 bytes in 0.587 second response time [05:01:44] Probably eating a lot of bandwidth [05:01:50] RECOVERY - LVS HTTPS IPv6 on mobile-lb.eqiad.wikimedia.org_ipv6 is OK: HTTP OK: HTTP/1.1 200 OK - 22502 bytes in 0.017 second response time [05:02:59] ok, that worked [05:03:01] RECOVERY - LVS HTTP IPv4 on appservers.svc.eqiad.wmnet is OK: HTTP OK: HTTP/1.1 200 OK - 62545 bytes in 0.208 second response time [05:03:11] RECOVERY - LVS HTTP IPv4 on rendering.svc.eqiad.wmnet is OK: HTTP OK: HTTP/1.1 200 OK - 62545 bytes in 0.196 second response time [05:03:14] \o/ [05:03:23] I ended up killing my sync command and running dsh directly with unlimited fanout [05:03:27] Ah yes [05:03:29] That sounds better [05:03:57] * Jamesofur cheers from the rafters [05:04:46] !log tstarling rebuilt wikiversions.cdb and synchronized wikiversions files: [05:04:55] Logged the message, Master [05:04:56] it was fast that time [05:05:05] RECOVERY - Apache HTTP on mw1041 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 747 bytes in 0.053 second response time [05:05:12] maybe the rsyncd on tin was just overloaded from the other sync commands [05:05:34] That's possible [05:06:03] !log tstarling synchronized wmf-config/CommonSettings.php 're-enabling GeoData' [05:06:12] Logged the message, Master [05:06:16] !log Note, anons were eventually getting HTTP 500 too (even for enwiki Main Page) as action=view threw as well squid cache got purged somehow and apaches unable populate it back. And then LVS depooled stuff for apache serving 500 on action=view for main page. [05:06:26] Logged the message, Master [05:06:28] phew [05:06:30] OK [05:06:37] http://ganglia.wikimedia.org/latest/graph.php?r=hour&z=xlarge&title=MediaWiki+errors&vl=errors+%2F+sec&n=&hreg[]=vanadium.eqiad.wmnet&mreg[]=fatal|exception>ype=stack&glegend=show&aggregate=1&embed=1 looks quiet again [05:06:50] Freaking software [05:06:58] :P [05:11:18] PROBLEM - NTP on ssl3003 is CRITICAL: NTP CRITICAL: No response from NTP server [05:11:27] PROBLEM - Puppet freshness on celsus is CRITICAL: No successful Puppet run in the last 10 hours [05:13:57] PROBLEM - NTP on ssl3002 is CRITICAL: NTP CRITICAL: No response from NTP server [05:14:30] RoanKattouw: what commands did you run to rebuild the l10n cache? [05:14:38] mw-update-l10n [05:14:41] i.e. between about 4:51 and 4:55 [05:14:51] But it generated a 0-byte ExtensionMessages-1.22wmf8.php file [05:15:20] I also accidentally got myself in the situation where one of my shells was still rebuilding php-1.22wmf8/cache/l10n and the other was already syncing it out [05:15:24] doesn't that normally take quite a while to run? [05:15:50] From resulting rsync warnings I concluded it had already done English though, which is what I cared about most at the time [05:15:53] Yes, it takes a while [05:16:00] I realized the 0-byte thing quite late [05:16:19] Had I seen that, I should have just copied the wmf9 file, removed WikiDataThingy, then sync that out [05:16:31] Also, this is arguably a bug in scap [05:16:47] scap rebuilds the l10n stuff, doesn't check if it failed, and then spams your terminal immediately after so you can't see if it failed [05:17:06] So the wmf8 l10n cache was probably already messed up because of my scap [05:17:24] I think it does check [05:17:48] Hmm, then maybe the script didn't believe it had failed, or something [05:17:49] it runs mw-update-l10n under set -e, so the scap is aborted if mw-update-l10n returns a non-zero exit status [05:18:02] Right [05:18:15] so you manually ran mw-update-l10n && sync-dir instead [05:18:30] Yes [05:18:31] well [05:18:33] ish [05:18:41] I stupidly run both more or less in parallel, see my bit about rsync warnings [05:18:51] *ran [05:19:01] I got distracted and forgot I had something running in another window [05:19:08] PROBLEM - search indices - check lucene status page on search1007 is CRITICAL: HTTP CRITICAL: HTTP/1.1 200 OK - pattern found - 327 bytes in 0.002 second response time [05:19:50] wouldn't the ( do stuff ) || echo "it failed" in mw-update-l10n mean that it always succeeds? [05:19:57] PROBLEM - search indices - check lucene status page on search1008 is CRITICAL: HTTP CRITICAL: HTTP/1.1 200 OK - pattern found - 327 bytes in 0.002 second response time [05:20:04] $ ( false || echo "yay" ) ; echo $? [05:20:04] yay [05:20:04] 0 [05:20:13] ori-l: I'm not sure that the php process even returned a 0 exit status [05:20:57] wouldn't matter if it did; mw-update-l10n would exit w/0 anyway [05:21:48] The PHP script that builds the ExtensionMessages file printed an error [05:21:50] don't do anything else for now, I'm still trying to analyse things [05:21:56] Yeah I'm not touching anything [05:22:07] Stuff isn't broken right now so no reason for me to [05:22:29] [05:23:20] so it looks like rebuildLocalisationCache.php was running between 4:52 and 4:57 [05:23:56] mw-update-l10n probably runs it [05:23:56] based on cdb modification times [05:24:07] yes [05:24:21] greg-g: Heya. Fun times. [05:25:45] the zero-byte ExtensionMessages-1.22wmf8.php was generated at 4:51 [05:26:28] I got a confusing error message when generating it [05:26:37] Like, it claimed it couldn't read the WikiDataDataBase i18n file [05:26:40] greg-g: https://wikitech.wikimedia.org/wiki/Incident_documentation/20130628-Site is apparently completely wrong, FWIW. [05:26:54] *WikiBaseDatamodel [05:26:57] I'm going to run mergeMessagesFileList.php to see what happens [05:27:08] But when I copypasted the path into less it worked [05:27:11] well, I assume it will succeed, you did something to fix it, right? [05:27:21] Also tried less as apache and as mwdeploy [05:27:32] Well, I removed WikiBaseDatamodel [05:27:43] Even though I'd added it to wmf8 and it looked like it should be worked [05:27:45] *working [05:28:06] you removed it from CommonSettings.php? [05:28:07] *WikibaseDataModel [05:28:14] From extension-list [05:28:23] Which excludes it for i18n purposes only [05:28:29] I suppose I should have disabled it there as well, d'oh [05:29:21] Well, wait a minute [05:29:25] It's only enabled for wmf9 anyway [05:29:30] I added it to wmf8 just an hour ago [05:29:33] So it couldn't have been enabled [05:31:52] so what happened at ~4:22 to statr this off? [05:31:57] I ran scap [05:32:09] 04:22 logmsgbot: catrope Started syncing Wikimedia installation... : Updating VisualEditor to master [05:32:12] and WikibaseDataModel was in extension-list? [05:32:15] Yes [05:32:18] But not in wmf8 [05:32:40] It was added at... [05:32:57] Date: Thu Jun 27 18:32:58 2013 +0100 [05:33:01] f475384d8fc5720576727792a116b6539003581a [05:33:17] So 17:32 UTC (commit time) [05:33:28] to extension-list? [05:33:50] to wmf8 [05:33:53] Checking extension-list [05:34:02] then you checked out the extension directory at /a/common/php-1.22wmf8/extensions/WikibaseDataModel at 04:42? [05:34:04] commit 88e8605ae80396c9b64345af6046be0cca04ff75 [05:34:32] https://git.wikimedia.org/commitdiff/mediawiki%2Fcore.git/f475384d8fc5720576727792a116b6539003581a [05:34:38] CommitDate: Thu Jun 27 18:26:45 2013 +0000 [05:34:47] Yes, I checked out WikibaseDataModel around that time [05:35:37] I tried to regenerate ExtensionMessages after that, but I got a vague error like "Could not read WikibaseDataModel.i18n.php" [05:35:50] (error message had a full path rather than just the file name) [05:36:01] I pasted the path into less and I could read it fine, and so could apache and mwdeploy [05:36:18] I also tried sync-common [05:36:47] I didn't understand how it couldn't read the file, so I just removed it from extension-list and regenerated ExtensionMessages-wmf8 again. That's the first time I got a >0-byte file [05:36:52] Which I immediately synced out [05:37:08] Then realized the l10n cache was probably borked, so I started rebuilding that [05:37:15] That was still in progress when you rolled back to wmf7 [05:38:03] apparently that rollback fixed it [05:38:06] Yeah [05:38:13] wmf8 must still be in a messed-up state somehow [05:38:25] with wmf7 still in a correct state [05:39:31] Some time after you're done with your analysis, I suggest we move a small wiki like test2 back to wmf8, and try to get it back up [05:39:45] test2 will currently be on wmf9, right? [05:40:04] Right [05:40:07] testwiki then? [05:40:13] It's nice and isolated over on srv193 [05:40:36] Probably better given APC cache size [05:41:41] I would like to reproduce the mw-update-l10n bug [05:41:49] OK [05:41:50] that would be easiest with a wiki switched to wmf8 [05:41:53] Yeah [05:51:38] did you run mw-update-l10n twice? [05:51:51] 04:46 logmsgbot: catrope synchronized wmf-config/ExtensionMessages-1.22wmf8.php [05:51:57] you must have run it before that, right? [05:55:15] scap / mw-update-l10n doesn't emit that message [05:57:51] I ran it a couple times [05:58:04] It errored an gave me a 0-byte file all times except the last IIRC [05:59:06] ... last time I removed WBDm from the extension list first [06:00:41] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:01:31] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 5.599 second response time [06:03:22] TimStarling: Given that it's 23:02 here and I've been in the office continuously for 14 hours, I'm gonna go home. I should be back on IRC in about 30 minutes. If you have more questions for me, ping me and I'll respond when I get home, or if they're more pressing text me (phone number in private channel) [06:09:56] PROBLEM - Puppet freshness on celsus is CRITICAL: No successful Puppet run in the last 10 hours [06:27:03] !log tstarling rebuilt wikiversions.cdb and synchronized wikiversions files: testwiki to wmf8 [06:27:13] Logged the message, Master [06:31:06] RECOVERY - Disk space on ms-be1001 is OK: DISK OK [06:31:17] RECOVERY - Disk space on ms-be1002 is OK: DISK OK [06:31:40] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:32:27] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.800 second response time [06:33:40] here's a fun one [06:34:03] http://fvue.nl/wiki/Bash:_Error_handling#Caveat_1:_.60Exit_on_error.27_ignoring_subshell_exit_status [06:34:07] "The `-e' setting does not exit if an error occurs within a subshell" [06:34:42] $ ( set -e; false; echo "success" ) || echo "failure" [06:34:42] success [06:34:43] right [06:34:59] so the 'set -e' inside the parens is voodoo [06:37:10] RECOVERY - Puppet freshness on celsus is OK: puppet ran at Fri Jun 28 06:37:04 UTC 2013 [06:37:57] PROBLEM - Puppet freshness on celsus is CRITICAL: No successful Puppet run in the last 10 hours [06:42:06] it seems to be more complicated than that: http://paste.tstarling.com/p/eceWPb.html [06:44:21] it depends on whether bash was invoked in posix mode; whether the commands are separated by semicolons; whether the command is part of a series of commands joined by || or &&; whether it's part of a while / unless loop [06:44:54] is the phase of the moon in there anywhere? [06:44:56] Holy crap that's scary [06:45:14] And to think I was wondering why our error handling didn't work 100% right [06:46:01] the tl;dr is that it's a bad idea to rely on 'set -e's to propagate failures up to parent shells [06:46:30] the parent shell should explicitly check the exit status of any command it cares about [06:47:21] do something || { echo "it didn't work"; exit 1 } [06:48:28] or use some more sensible language [06:49:01] though in the case of mw-update-l10n even that wouldn't have helped, because of the whole ( ... ) || echo "failure" [06:49:18] yes, it always returns 0 [06:50:17] i haven't checked whether the php scripts return correct status codes, it'd be funny if we're clobbering errors on every level [06:50:55] mergeMessageFileList.php returns 1 on error, I checked that [06:51:42] Oh good [06:52:00] Does that include the case where it gives the weird "Could not read $filename" message and writes an empty file? [06:52:19] cause I had never seen it fail like that before, but I've seen it fail in other ways before [06:53:02] I'm not sure how it wrote an empty file [06:53:17] it didn't do that when I tested it [06:53:45] do you still have the output of that scap command? [06:54:25] No, the relevant bits flooded out of my scrollback buffer in screen [06:54:42] It was a one-line error message, it just said it couldn't read the file and that was it [06:54:57] Did you add WikibaseThingy back to extension-list in an attempt to provoke the failure? [06:55:38] I made a copy of extension-list and added $IP/extensions/Blah/BlahBlah.php to it [06:56:23] http://paste.tstarling.com/p/OoHbsu.html [06:57:52] Right [06:58:10] So that I have seen happen before [06:58:23] But my file was actually /there/ and it failed [06:58:53] well, I'll try that now [07:00:07] if ( !( include_once $fileName ) ) { [07:00:11] fwrite( STDERR, "Unable to read $fileName\n" ); [07:00:11] exit( 1 ); [07:00:12] } [07:00:31] So I suppose for some reason the "PHP Warning" stuff must have been suppressed when I ran it [07:00:38] Maybe due to the way mw-update-l10n invokes it [07:01:08] Hmm, it invokes it with --quiet [07:01:53] if ( defined( 'WIKIBASE_DATAMODEL_VERSION' ) ) { [07:01:54] // Do not initialize more then once. [07:01:54] return; [07:02:04] oooh [07:02:44] Hmm [07:02:52] Can't see how that's related though [07:04:07] Oh, I see [07:04:09] It's include [07:04:24] According to the documentation, it passes along the return value from the included script [07:05:00] And so if WIKIBASE_DATAMODEL_VERSION was already defined somehow (don't see how if it uses include_once, but still), then that would case include_once to return false and mergeMessageFileList.php to exit(1) [07:05:18] s/false/null probably, actually [07:06:00] it's defined by Wikibase [07:06:03] DataModel/DataModel.php [07:06:04] 30:define( 'WIKIBASE_DATAMODEL_VERSION', '0.4 alpha' ); [07:06:16] mind if i file a bug for the error-handling in the shell scripts? [07:06:31] if you like [07:06:40] thanks [07:07:34] the sequence: http://paste.tstarling.com/p/YGuKrr.html [07:08:20] Hah, yeah [07:08:33] So, hurrah [07:08:40] 1) we found the cause, it looks like [07:08:47] 2) include_once's return value is not reliable [07:08:56] Using === false rather than !include_once would be better [07:09:04] but that still isn't infallible [07:09:05] it still doesn't explain how an empty ExtensionMessages was created [07:09:32] I'll finish this post mortem later, I have to go have dinner [07:09:44] I have a timeline and some causes here [07:09:51] but we are still missing one cause [07:10:00] which is how ExtensionMessages was overwritten [07:10:29] Yeah, it's invoked with --output which shouldn't do taht [07:10:37] Thanks for the analysis, Tim [07:10:59] And for bringing the site back up while I wasn't getting us anywhere :) [07:11:07] PROBLEM - Puppet freshness on celsus is CRITICAL: No successful Puppet run in the last 10 hours [07:11:12] I'm going to sleep now, will probably be back on IRC in 12-13 hours [07:11:22] it's been quite the day [07:11:47] good night roan [07:23:52] as noted on #mediawiki, the 0 byte file is caused by the mw-update-l10n shell script too; it's created by 'mktemp' on line 42, not the PHP script on line 43 [07:26:42] the temp file is copied over to $MW_COMMON_SOURCE/wmf-config/ExtensionMessages-"$mwVerNum".php in line 46 regardless of whether or not the PHP script populated it [07:29:47] that's pretty badass, it means that any failure in mergeMessageFileList.php will cause scap to sync an empty l10n file [07:32:35] presumably the 'set -e' was expected to prevent that too [08:00:49] * aude screams!!!! [08:02:20] RECOVERY - NTP on ssl3003 is OK: NTP OK: Offset 0.002748847008 secs [08:06:36] you folks had fun earlier I see [08:12:03] PROBLEM - Puppet freshness on celsus is CRITICAL: No successful Puppet run in the last 10 hours [08:14:20] New review: Micha? ?azowik; "(1 comment)" [operations/apache-config] (master) C: -1; - https://gerrit.wikimedia.org/r/65443 [08:18:24] PROBLEM - NTP on ssl3003 is CRITICAL: NTP CRITICAL: No response from NTP server [08:32:24] RECOVERY - NTP on ssl3003 is OK: NTP OK: Offset -0.001160621643 secs [08:33:44] RECOVERY - NTP on ssl3002 is OK: NTP OK: Offset 0.002249717712 secs [08:34:44] New review: Faidon; "LGTM. As a side note, maybe you or otto are interested pushing these upstream? :)" [operations/puppet] (production) C: 2; - https://gerrit.wikimedia.org/r/70858 [08:35:11] Change merged: Faidon; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/70858 [08:39:10] Change merged: Faidon; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/70846 [08:44:46] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:45:36] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.135 second response time [08:58:26] New patchset: Faidon; "Revert "Puppetizing /etc/init.d/rsync and /etc/default/rsync"" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/70991 [09:02:01] New patchset: Faidon; "Revert "Puppetizing /etc/init.d/rsync and /etc/default/rsync"" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/70991 [09:03:02] Change merged: Faidon; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/70991 [09:12:39] PROBLEM - Puppet freshness on celsus is CRITICAL: No successful Puppet run in the last 10 hours [09:14:00] !g 70915 [09:14:01] https://gerrit.wikimedia.org/r/#q,70915,n,z [09:19:12] * qchris tries to find a lurking op whom I could ask to review https://gerrit.wikimedia.org/r/#/c/70870 [09:19:53] New review: Hashar; "recheck" [operations/puppet/jmxtrans] (master) - https://gerrit.wikimedia.org/r/70915 [09:22:30] qchris: bake up a batch of stroopwafels to drag them back into the channel [09:22:44] :-D [09:23:06] Baking stroopwafels ... that would take me like forever. Hehe. [09:23:32] !log jenkins: made the .erb linting jobs to be voting {{gerrit|70994}} [09:23:42] Logged the message, Master [09:23:59] But I can offer pancakes :-) Come and visit me. I'll make you pan cakes. [09:24:21] New review: Hashar; "recheck" [operations/puppet/jmxtrans] (master) - https://gerrit.wikimedia.org/r/70915 [09:24:49] New review: Hashar; "You now have erb and puppet linting on this repository. Sorry for the spam," [operations/puppet/jmxtrans] (master) - https://gerrit.wikimedia.org/r/70915 [09:25:50] qchris: I am not sure why you asked Coren to review a change made to Zuul configuration https://gerrit.wikimedia.org/r/#/c/70990/ :) [09:25:57] qchris: I guess you have picked up the wrong change [09:26:02] hashar: By mistake :-( [09:26:14] hashar: Yes. I already wrote an email saying Sorry to him. [09:26:19] hehe :-] [09:26:29] just wanted to make sure you did not miss adding him on the proper change! [09:26:44] Could any kind op have a look at https://gerrit.wikimedia.org/r/#/c/70870 please? [09:26:56] I'm looking at it now [09:27:02] hashar: It's too early in the morning *gg* [09:27:10] apergos: Great! Thanks. [09:28:11] Change merged: ArielGlenn; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/70870 [09:28:42] New patchset: Faidon; "Revert "Convert swift's rsyncd from generic::rsyncd to the new rsync module."" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/70995 [09:28:50] New patchset: Faidon; "Revert "Convert swift's rsyncd from generic::rsyncd to the new rsync module."" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/70995 [09:28:53] Thanks apergos. [09:29:31] Change merged: Faidon; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/70995 [09:29:42] :( [09:29:49] paravoid: that is not working is it ? [09:31:12] qchris: change now live, please test [09:32:44] apergos: Yes, works just fine. Thanks. [09:32:48] hashar: !@%$!%@#! [09:33:01] it's a crappy puppetlabs module [09:33:07] :-( [09:33:15] and otto has hacked it to work, now I've reverted it and it's not working again [09:33:29] I guess that is a good example to not reuse third party modules [09:33:37] puppetlabs modules are *so* crappy [09:36:20] heh. i'm glad i'm not the only one who thinks that. [09:37:25] * ori-l sleeps [09:38:28] crap module is crap [09:38:40] like, this one is especially crappy [09:38:47] one of the changes that otto did (and I didn't revert obviously) [09:39:07] is that it unconditionally sets rsync's motd [09:39:08] to this: https://github.com/puppetlabs/puppetlabs-rsync/blob/master/files/motd [09:39:15] I'm shitting you not [09:39:27] ori-l: you should really see this if you haven't gone to bed yet [09:40:46] ahah [09:46:48] New patchset: Hashar; "contint: docroot belong to jenkins-slave now" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/70998 [09:47:19] apergos: paravoid: could one of you merge on sock puppet https://gerrit.wikimedia.org/r/70998 . That change some dir ownership on the contint server :-) [09:48:43] Change merged: ArielGlenn; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/70998 [09:49:22] wonderful thank you [09:49:33] will run puppet on gallium [09:49:37] done [09:49:53] I almost completed the migration of jobs to run on slaves [09:50:05] so whenever RobH has installed lanthanum, the jobs will be able to execute on that second box [09:50:11] and then we can Mooooaaar jobs [09:50:24] great [09:50:34] just need robh to actually reinstall the box hehe [09:56:10] !log gallium : updating some files under /srv/org to belong to jenkins-slave:jenkins-slave instead of jenkins. [09:56:19] Logged the message, Master [10:08:24] PROBLEM - Puppet freshness on celsus is CRITICAL: No successful Puppet run in the last 10 hours [10:22:24] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:24:14] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.127 second response time [10:27:04] PROBLEM - RAID on mc15 is CRITICAL: Timeout while attempting connection [10:27:55] RECOVERY - RAID on mc15 is OK: OK: Active: 2, Working: 2, Failed: 0, Spare: 0 [10:36:44] PROBLEM - Puppet freshness on srv292 is CRITICAL: No successful Puppet run in the last 10 hours [10:38:36] out for lunch [10:38:43] New review: Akosiaris; "(1 comment)" [operations/puppet/jmxtrans] (master) - https://gerrit.wikimedia.org/r/70915 [10:46:28] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:47:19] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.131 second response time [10:52:29] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:53:18] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.128 second response time [11:10:06] PROBLEM - Puppet freshness on erzurumi is CRITICAL: No successful Puppet run in the last 10 hours [11:10:06] PROBLEM - Puppet freshness on ms-fe3001 is CRITICAL: No successful Puppet run in the last 10 hours [11:10:06] PROBLEM - Puppet freshness on lvs1006 is CRITICAL: No successful Puppet run in the last 10 hours [11:10:06] PROBLEM - Puppet freshness on mc15 is CRITICAL: No successful Puppet run in the last 10 hours [11:10:06] PROBLEM - Puppet freshness on lvs1005 is CRITICAL: No successful Puppet run in the last 10 hours [11:10:06] PROBLEM - Puppet freshness on lvs1004 is CRITICAL: No successful Puppet run in the last 10 hours [11:10:06] PROBLEM - Puppet freshness on sodium is CRITICAL: No successful Puppet run in the last 10 hours [11:10:07] PROBLEM - Puppet freshness on virt4 is CRITICAL: No successful Puppet run in the last 10 hours [11:10:08] PROBLEM - Puppet freshness on virt1 is CRITICAL: No successful Puppet run in the last 10 hours [11:10:08] PROBLEM - Puppet freshness on spence is CRITICAL: No successful Puppet run in the last 10 hours [11:10:08] PROBLEM - Puppet freshness on virt3 is CRITICAL: No successful Puppet run in the last 10 hours [11:13:26] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:14:16] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.131 second response time [11:37:57] New review: Faidon; "Puppet is not a deployment system, it's a configuration management one (or at least that's the way w..." [operations/puppet] (production) C: -1; - https://gerrit.wikimedia.org/r/62404 [11:39:22] New patchset: Faidon; "Restore parts of Andrew's changes to rsync" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/71003 [11:41:08] Change merged: Faidon; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/71003 [11:48:24] New review: Akosiaris; "Apart from the inline comments which are minor I really don't like the idea that puppet will checkou..." [operations/puppet] (production) C: -1; - https://gerrit.wikimedia.org/r/62404 [11:52:24] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:53:16] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.131 second response time [11:55:57] New patchset: Aude; "explicitly require WikibaseDataModel in CommonSettings" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/71004 [12:01:24] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:01:58] New review: Aude; "and would prefer it not to be available in wmf8, if possible. in the version of Wikibase we have th..." [operations/mediawiki-config] (master) C: -1; - https://gerrit.wikimedia.org/r/71004 [12:02:15] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.140 second response time [12:03:35] re [12:03:38] New review: Aude; "suppose we could also put this under a separate config variable and have it set to false (by default..." [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/71004 [12:04:01] hashar: any advice about https://gerrit.wikimedia.org/r/#/c/71004/ ? [12:04:09] otherwise i wait for roan and tim [12:05:00] our new extension seems to have caused a bit of trouble last night for localisation update / scap :( [12:05:51] Reedy: ^ [12:06:55] Things like this are even more annoying when they didn't happen at deploy time when I ran scap etc [12:07:00] yeah [12:07:21] well, i wonder if localisation update ever worked correctly for wmf9? [12:08:06] PROBLEM - Puppet freshness on celsus is CRITICAL: No successful Puppet run in the last 10 hours [12:08:39] No idea.. We did still have that missing message. If that kicked out early so Wikibase didn't finish (new message?), while building on an older l10n cache.. [12:08:46] Reedy: ok, test2 does say "Data item" [12:08:57] apparently it worked eventually [12:11:40] Computer ssuck [12:11:48] heh [12:15:23] having WikibaseDataModel and Wikibase/DataModel means that the constant WIKIBASE_DATA_MODEL gets set already before WikibaseDataModel is loaded [12:16:53] not having it there means mergeMessageFileList chokes [12:17:58] can't mergeMessageFileList check if the file exists and if not, skip it? [12:21:55] I am migrating mwext-Wikibase-client-tests Jenkins job to run on slave nodes [12:22:02] ok [12:22:12] testing out right now, that seems to be working :-] [12:22:21] yay! [12:22:36] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:22:37] the idea is to let jjob execute on different servers whenever we add more servers :-] [12:22:45] ah wrong chan [12:23:16] !log jenkins: migrating mwext-Wikibase-client-tests and mwext-Wikibase-repo-tests from master to slaves [12:23:25] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.128 second response time [12:23:26] Logged the message, Master [12:25:57] I guess we're expecting anything in extension-list to exists [12:26:03] In which case, we could just remove it.. [12:26:13] for now i suppose [12:26:18] it's needed for wmf9 [12:26:32] i don't know how it works to make an extension exists there and not in wmf8 [12:28:25] if we do it with a config variable that enables it only for test wikidata and test2 that could work [12:28:33] but then we need to remember to update it on monday [12:29:12] if ( $wgDBname in_array( 'test2wiki', 'testwikidatawiki' ) ) {} [12:29:20] fail [12:29:38] if ( in_array( $wgDBname, array( 'test2wiki', 'testwikidatawiki' ) ) ) {} [12:30:01] that could work [12:30:27] All the other wikis are still rolled back [12:30:29] though i get Unable to read /var/www/common/php-master/extensions/WikibaseDataModel/WikibaseDataModel.php [12:30:45] if i try setting up my wiki farm this way and run the script [12:31:06] the extension is checked out but just not require_once [12:31:24] New review: Faidon; "OK, a mostly superficial review for some common outliers." [operations/puppet] (production) C: -1; - https://gerrit.wikimedia.org/r/70840 [12:33:14] New patchset: Aude; "explicitly require WikibaseDataModel in CommonSettings" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/71004 [12:34:46] New review: Aude; "doing it this way, however, I get "Unable to read /var/www/common/php-master/extensions/WikibaseData..." [operations/mediawiki-config] (master) C: -1; - https://gerrit.wikimedia.org/r/71004 [12:40:50] I'm not sure what putting most of the wikis back to wmf7 did.. Or it just had a populated/useful l10n cache that could be used straight away [12:42:33] just because the cache was intact [12:42:57] if localisation update was run on wmf7, then it probably would fail [12:43:51] nice state of limbo before a long holiday weekend [12:44:01] it's a holiday? [12:44:17] * aude knows that monday is canada day :) [12:44:21] not sure what else [12:45:43] Reedy: monday is off? [12:47:06] It's 4th July week.. And no deployments next week [12:47:26] 4th of july isn't on till middle of next week, no? [12:47:28] * YuviPanda is confused [12:47:32] o really? [12:47:39] no wikidata deployment on monday? [12:48:04] no _what_ deployments next week? [12:48:05] * aude thinks the best solution, though not on a friday, is to update wikibase extensions to our wmf9 branch [12:48:26] even for wmf8 / wmf7 core, which was/is our intention on monday [12:50:24] https://wikitech.wikimedia.org/wiki/Deployments#Upcoming_holidays_.2F_other_events everyone is expected to know when independence day is? [12:53:34] https://www.mediawiki.org/wiki/MediaWiki_1.22/Roadmap [12:53:37] Same day every year? ;) [12:55:30] * aude must have misunderstood when talking with robla etc. then [12:56:22] we could then roll back wikibase etc. on wmf9 to the wmf6 branch? [12:56:34] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:56:58] have no wikibase data model until we update *everything* with the new stuff? [12:57:25] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.130 second response time [12:57:26] or remove it from extensions-list [12:57:55] Considering it only has a -desc line, we won't be missing much [12:58:01] agree [12:58:44] need a test wiki on wmf8 to make sure we get a l10n cache rebuild before moving anything else over [12:58:52] it would be a bit nasty to hack WikibaseDataModel for this use case, although the issue might come up again for other wikibase extensions [12:59:02] so a solution is important [12:59:17] i think testwiki is on wmf8, if that' sgood enough? [12:59:35] PROBLEM - Disk space on ms-be1001 is CRITICAL: DISK CRITICAL - free space: / 5697 MB (3% inode=98%): [13:02:36] Let me look at the cache files. Probably worth running scap again before doing anything further.. [13:02:42] k [13:07:16] PROBLEM - Puppet freshness on manutius is CRITICAL: No successful Puppet run in the last 10 hours [13:10:28] PROBLEM - Puppet freshness on celsus is CRITICAL: No successful Puppet run in the last 10 hours [13:10:32] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:11:22] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.127 second response time [13:14:59] !log jenkins: reducing # of executor on master from 4 to 2 [13:15:08] Logged the message, Master [13:15:57] uhhhh, how com we have 3 MW versions deployed at the same time? [13:16:42] !g Ia9754978098bf9a164ca443d8454fd9b0a622dc9 [13:16:43] https://gerrit.wikimedia.org/r/#q,Ia9754978098bf9a164ca443d8454fd9b0a622dc9,n,z [13:18:26] New patchset: Reedy; "Remove WikibaseDataModel" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/71013 [13:18:27] New patchset: Reedy; "Roll most things back to wmf7" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/71014 [13:19:00] Change merged: jenkins-bot; [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/71013 [13:19:06] Change merged: jenkins-bot; [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/71014 [13:20:15] !log reedy synchronized wmf-config/ [13:20:24] Logged the message, Master [13:21:38] !log reedy Started syncing Wikimedia installation... : Ensure all l10n caches are built and properly built [13:21:47] Logged the message, Master [13:25:04] Scap happy [13:25:13] yay [13:27:37] !log reedy Finished syncing Wikimedia installation... : Ensure all l10n caches are built and properly built [13:27:46] Logged the message, Master [13:28:05] Ok, that didn't take too long [13:29:41] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:30:24] New patchset: Ottomata; "jmxtrans puppetization." [operations/puppet/jmxtrans] (master) - https://gerrit.wikimedia.org/r/70915 [13:31:42] Change merged: Ottomata; [operations/puppet/jmxtrans] (master) - https://gerrit.wikimedia.org/r/70915 [13:32:24] <^demon> manybubbles: I updated the tasks list on wiki, mostly stuff for myself. [13:33:32] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.129 second response time [13:33:32] New patchset: Ottomata; "Adding jmxtrans git submodule" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/71016 [13:34:03] Change merged: Ottomata; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/71016 [13:35:51] yay, manybubbles, if you get that thar jmxtrans.deb going [13:36:00] the jmxtrans puppet module is avail for you! [13:36:26] ottomata: Almost there. I've learned a lot in the past day or so. It is almost ready for review. [13:36:31] ^demon: cool! [13:38:03] !log reedy synchronized php-1.22wmf9/extensions/Wikibase [13:38:12] Logged the message, Master [13:39:27] !log reedy rebuilt wikiversions.cdb and synchronized wikiversions files: Everything back from wmf7 to wmf8 [13:39:37] Logged the message, Master [13:41:35] !log reedy Started syncing Wikimedia installation... : Scaptastic [13:41:44] Logged the message, Master [13:43:38] !log reedy Finished syncing Wikimedia installation... : Scaptastic [13:43:47] Logged the message, Master [13:44:19] aude: Looks like we're still alive [13:44:29] :) [13:45:10] New patchset: Reedy; "All wikis on 1.22wmf7 back to 1.22wmf8" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/71017 [13:45:35] Change merged: jenkins-bot; [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/71017 [13:46:00] Now running l10nupdate [13:50:17] Warning: require() [function.require]: Unable to allocate memory for pool. [13:50:18] Eugh [13:50:28] Guess it's a good job wmf7 was removed again [13:51:49] !log LocalisationUpdate completed (1.22wmf8) at Fri Jun 28 13:51:49 UTC 2013 [13:51:59] Logged the message, Master [13:56:55] !log LocalisationUpdate completed (1.22wmf9) at Fri Jun 28 13:56:55 UTC 2013 [13:57:04] Logged the message, Master [13:57:40] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:58:28] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.127 second response time [13:59:47] New review: Akosiaris; "Comments addressed inline. As for the exported resources issue I can understand why but there is no ..." [operations/puppet] (production) - https://gerrit.wikimedia.org/r/70840 [14:02:46] New patchset: Akosiaris; "Introducing bacula module" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/70840 [14:05:42] !log LocalisationUpdate ResourceLoader cache refresh completed at Fri Jun 28 14:05:42 UTC 2013 [14:05:51] Logged the message, Master [14:09:28] PROBLEM - Puppet freshness on celsus is CRITICAL: No successful Puppet run in the last 10 hours [14:23:48] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:24:40] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.133 second response time [14:29:18] PROBLEM - Puppet freshness on mw1040 is CRITICAL: No successful Puppet run in the last 10 hours [14:30:18] PROBLEM - Puppet freshness on amssq32 is CRITICAL: No successful Puppet run in the last 10 hours [14:30:19] PROBLEM - Puppet freshness on analytics1001 is CRITICAL: No successful Puppet run in the last 10 hours [14:30:19] PROBLEM - Puppet freshness on cp1015 is CRITICAL: No successful Puppet run in the last 10 hours [14:30:19] PROBLEM - Puppet freshness on cp3006 is CRITICAL: No successful Puppet run in the last 10 hours [14:30:19] PROBLEM - Puppet freshness on db1011 is CRITICAL: No successful Puppet run in the last 10 hours [14:32:18] PROBLEM - Puppet freshness on analytics1017 is CRITICAL: No successful Puppet run in the last 10 hours [14:32:18] PROBLEM - Puppet freshness on analytics1021 is CRITICAL: No successful Puppet run in the last 10 hours [14:32:18] PROBLEM - Puppet freshness on analytics1020 is CRITICAL: No successful Puppet run in the last 10 hours [14:32:18] PROBLEM - Puppet freshness on cp1008 is CRITICAL: No successful Puppet run in the last 10 hours [14:32:18] PROBLEM - Puppet freshness on cp1064 is CRITICAL: No successful Puppet run in the last 10 hours [14:33:18] PROBLEM - Puppet freshness on amssq36 is CRITICAL: No successful Puppet run in the last 10 hours [14:33:18] PROBLEM - Puppet freshness on amssq41 is CRITICAL: No successful Puppet run in the last 10 hours [14:33:18] PROBLEM - Puppet freshness on amssq58 is CRITICAL: No successful Puppet run in the last 10 hours [14:33:18] PROBLEM - Puppet freshness on amssq60 is CRITICAL: No successful Puppet run in the last 10 hours [14:33:18] PROBLEM - Puppet freshness on analytics1024 is CRITICAL: No successful Puppet run in the last 10 hours [14:34:18] PROBLEM - Puppet freshness on analytics1016 is CRITICAL: No successful Puppet run in the last 10 hours [14:34:18] PROBLEM - Puppet freshness on chromium is CRITICAL: No successful Puppet run in the last 10 hours [14:34:18] PROBLEM - Puppet freshness on cp1044 is CRITICAL: No successful Puppet run in the last 10 hours [14:34:18] PROBLEM - Puppet freshness on db1003 is CRITICAL: No successful Puppet run in the last 10 hours [14:34:18] PROBLEM - Puppet freshness on analytics1026 is CRITICAL: No successful Puppet run in the last 10 hours [14:35:18] PROBLEM - Puppet freshness on amslvs3 is CRITICAL: No successful Puppet run in the last 10 hours [14:35:18] PROBLEM - Puppet freshness on amssq46 is CRITICAL: No successful Puppet run in the last 10 hours [14:35:18] PROBLEM - Puppet freshness on amssq62 is CRITICAL: No successful Puppet run in the last 10 hours [14:35:18] PROBLEM - Puppet freshness on analytics1013 is CRITICAL: No successful Puppet run in the last 10 hours [14:35:19] PROBLEM - Puppet freshness on cp1027 is CRITICAL: No successful Puppet run in the last 10 hours [14:38:18] PROBLEM - Puppet freshness on amssq39 is CRITICAL: No successful Puppet run in the last 10 hours [14:38:18] PROBLEM - Puppet freshness on amssq42 is CRITICAL: No successful Puppet run in the last 10 hours [14:38:18] PROBLEM - Puppet freshness on amssq49 is CRITICAL: No successful Puppet run in the last 10 hours [14:38:18] PROBLEM - Puppet freshness on amssq54 is CRITICAL: No successful Puppet run in the last 10 hours [14:38:18] PROBLEM - Puppet freshness on amssq61 is CRITICAL: No successful Puppet run in the last 10 hours [14:38:18] PROBLEM - Puppet freshness on analytics1005 is CRITICAL: No successful Puppet run in the last 10 hours [14:46:11] New review: QChris; "(1 comment)" [operations/debs/buck] (master) - https://gerrit.wikimedia.org/r/70673 [15:06:04] New review: Hashar; "Ideally we should use the Facebook branch (which is the real upstream) and ship in the Debian packag..." [operations/debs/buck] (master) - https://gerrit.wikimedia.org/r/70673 [15:09:28] paravoid, can you catch me up with the state of the rsync module? I thought that since the modules expressly specified uid and gid that the defaults were moot… [15:09:37] you're right [15:09:48] I realized this a bit after I reverted [15:09:55] then I was dealing with all the other crap of the rsync module [15:10:00] after I kinda broke everything :) [15:10:32] so, your commit actually did break things in the sense that the rsync module used a different pidfile path [15:11:02] hey opsies, ganglia q [15:11:03] so the init script though it wasn't running because the new filename for pidfile didn't exist, but it was actually running [15:11:13] and you couldn't stop it either [15:11:14] any idea why this ganglia value would be static? [15:11:29] http://bit.ly/13cbQMq [15:11:41] paravoid: OK, I don't follow, lemme look at the code [15:11:47] that metric is not being pushed to ganglia anymore at all [15:11:53] Is it that the module-specified pid file got picked up in some places and not others? [15:11:56] ganglia is reporting the last value it got consistently [15:12:05] andrewbogott: debian uses /var/run/rsync.pid for the pidfile [15:12:10] generic::rsyncd didn't change this [15:12:47] the rsync module used /var/run/rsyncd.pid (not the "d") [15:13:08] the swift systems had an rsyncd running, say on pid 1191, with /var/run/rsync.pid having the value "1191" [15:13:10] Oh, I'm confusing lockfile with pidfile [15:13:14] Now I see what you're saying. [15:13:22] the module went and deployed all that [15:13:38] then /etc/init.d/rsync status said "not running" as it was checking /var/run/rsyncd.pid [15:14:03] /etc/init.d/rsync stop would do nothing, and puppet was trying (because of ensure => running) to start rsync [15:14:14] which failed because of the new daemon trying to bind in the same port [15:14:18] a mess [15:14:32] which took a while to understand because of this super crappy puppetlabs module [15:14:46] with me reverting ottomata1's commit from a year back and reintroducing most of it later [15:17:02] I'm almost caught up but don't quite understand why reverting otto's change wasn't right. Reverting moved the module's pidfile back to the same place as the debian standard, right? [15:17:12] rsync.pid? [15:17:28] no [15:17:37] the original module itself had it changed [15:17:47] (that's what I thought too) [15:17:52] (did ottomata1 do something wrong a year back? or is this just the puppetlabs modules I imported being dumb?) [15:17:55] otto's change made it a manifest variable [15:18:01] ah, I see. [15:18:05] but kept the original path [15:18:14] i.e. defaulted the variable to what puppetlabs did [15:18:26] probably to not break already deployed things [15:18:42] already at the time that is [15:18:47] does that makes sense? [15:18:53] it took me a while to realize too :) [15:19:14] Yeah, I think so. Still reading code... [15:19:21] ottomata: the module was crap [15:19:35] your changes weren't very clean, but mostly okay [15:19:50] I can't blame you considering the original module itself... [15:19:51] New review: Aklapper; "Finally having a way to directly deploy custom patches on top of the upstream Bugzilla code might be..." [operations/puppet] (production) - https://gerrit.wikimedia.org/r/62404 [15:19:55] e.g. https://github.com/puppetlabs/puppetlabs-rsync/blob/master/files/motd [15:19:56] ratso, sorry bout that then [15:20:06] what the fuck are they thinking? [15:20:13] sounds like the module was made on redhat and they just assumed it would work on debian as well [15:20:42] well, it's crappy in other ways too though [15:20:47] the puppet module feature that i liked was the ability to manage rsync uhhh 'modules' independently [15:21:03] like the fact that you can't give it arbitrary config file options [15:21:14] or alternatively a custom header template of yours [15:21:21] yeah that would be real nice [15:21:22] so you had to resort into patching it for the log file location [15:21:41] yeah, it's weirdly inflexible. [15:21:41] i'd love to turn that module into something better [15:22:03] ottomata: please do :) [15:22:10] and github pull request it maybe :) [15:22:28] but anyway, it will do for now [15:22:30] Might also be worth checking the upstream state of the module today -- it might've been improved already. [15:22:33] I did [15:22:39] yeah, ok, i mean, i would probably just redo most of the whole thing [15:22:39] Oh, still the same? [15:22:40] and drop bits of it [15:22:41] like [15:22:42] yeah [15:22:43] rsync::get [15:22:44] i dunno about that [15:22:46] Reedy, why testwiki's on wmf8?:P [15:22:52] rsync::server::module is great [15:22:58] that's the only thing i really liked [15:23:04] paravoid, so, do you think it's in a state where we should try to re-apply my earlier patch? Or would you prefer that we not rely on that module at all for the time being? [15:23:14] New review: BBlack; "The code look good quality-wise, I didn't see anything obviously silly going on and I'm not gonna bi..." [operations/software/varnish/varnishkafka] (master) C: 1; - https://gerrit.wikimedia.org/r/70928 [15:23:17] andrewbogott: I think the former [15:23:34] PROBLEM - Disk space on ms-be1002 is CRITICAL: DISK CRITICAL - free space: / 5696 MB (3% inode=98%): [15:23:47] ok, I'll rebase and cherry-pick and such. Will you be around for a bit to keep an eye on things when we apply? [15:23:55] I was just fed up at the time and didn't have the patience to deal with more of that :) [15:24:48] andrewbogott: I'll be around, but I can do all that too, no worries [15:24:52] yeah, well… I tested in labs to make sure that it generated the right conf files but didn't actually test the running service :( [15:26:03] ok -- I'll start on the next one in hopes that the swift patch survives its rebirth [15:26:55] ottomata, do you want to take on the task of pushing our latest version of that module upstream? [15:28:00] he volunteered to rewrite the thing :D [15:28:10] psh, i thikn i'm out of the loop on the module, i'd rewrite the thing and get rid of the cruft [15:28:22] make it our own version [15:28:24] * paravoid agrees [15:28:30] "good morning" [15:28:32] i'd keep the basic logic for rysnc::server::Module [15:28:36] well, if you can base it on their version and push the changes to them, I wouldn't mind [15:28:45] well, i'd remove things like rsync::get [15:28:48] that just bothers me [15:28:49] i guess its fine [15:28:51] ok [15:28:59] I don't mind it being our version either [15:29:08] I'm surprised that you're taking this stance actually :) [15:29:16] yep, either way [15:29:17] haha [15:29:24] i know no the usual ottomata stance, eh? [15:29:29] yeah :) [15:29:33] but nawww, that's why i wrote the puppet-cdh4 stuff [15:29:38] because all the other ones suuuuuuucked [15:29:52] heh [15:29:56] rsync::get basically uses puppet as cron [15:29:59] I'm sure others think they're great [15:30:02] runs an rsync command whenever puppet runs [15:30:18] no way man! that's why i've got github contributors on puppet-cdh4! :) [15:30:24] andrewbogott: so, what do you think on the latest front of that discussion [15:30:38] andrewbogott: a lot of people already voluntereed to help you [15:30:54] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:30:55] (remains to be seen if they're going to have the time to deliver that of course, but still :) [15:31:01] Yeah, I think it sounds good -- I just need to write an email. [15:31:36] andrewbogott: yeah, i'd love to help, time is limited though :/ [15:31:41] i think you could be master coordinator [15:31:53] Having record of lots of buy-in will be nice when things start to break :) [15:31:54] and ask specific individuals to help with one module at a time or something [15:31:54] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 5.662 second response time [15:32:32] I enjoy the coding work, but when it involves a service that I'm unfamiliar with… I don't relish refactoring blindly. So I'll need to lean on people with more ops knowledge. [15:32:46] yeah, totally [15:32:54] that is really scary to take on [15:32:55] cause if you change it, then you own it [15:35:47] heya LeslieCarr, you there? are you our resident ganglia expert(ish)? [15:35:54] hey [15:35:56] sorta [15:35:57] :) [15:36:00] which is scary [15:36:00] andrewbogott: yeah that was my idea [15:36:04] haha [15:36:04] yeah [15:36:13] is documentation on ganglia as bad as it looks? [15:36:23] documen-what ? [15:36:25] haha [15:36:26] right [15:36:28] on the internet [15:36:33] andrewbogott: I'm also completely on board with helping out -- I've already tried giving you reviews quickly [15:36:34] :) [15:36:40] i'm just trying to understand how ganglia works more generally [15:36:43] paravoid: Yep, I appreciate it. [15:36:44] ok [15:36:56] any clue why a metrics value would be repeated when it stops receiving the metric? [15:37:19] http://bit.ly/13cbQMq [15:37:24] hrm, my first thought is that it's passively receiving the value and so it hasn't updated ..... [15:37:32] andrewbogott: but some things I'm not very familiar with as well, so that's why I suggested involving the right people for each file [15:37:41] passively receiving the value? [15:37:52] andrewbogott: but I can definitely help on e.g. generic.pp/base.pp and such complicated beasts [15:37:58] so, lemme see if i can figure out how that specific value is getting into ganglia :) [15:38:08] well [15:38:22] it *was* being sent to ganglia by jmxtrans [15:38:41] a daemon that queries JMX for values and then sends them to various monitoring services [15:38:56] but, but the JVM and jmxtrans for that metric have been turned of [15:38:56] andrewbogott: but let's make a plan first; modularizing/rewriting swift was on my plans already, notpeter has work ahead on puppetmaster with a different architecture so he might end up rewriting that anyway etc. [15:39:04] and actually, we had a problem last week [15:39:26] because something happened to the udp2log/kafka producer that this metric is from [15:39:26] andrewbogott: or squid is on its way out so no point in modularizing this [15:39:30] but, we didn't notice [15:39:34] because the ganglia value didnt' drop to 0 [15:39:42] we would have gotten an alert if it had [15:39:42] until 3-11 ? [15:39:45] but it just stayed the same [15:39:45] paravoid: Yeah, I need to sit down and make a list. And compare it to available upstream classes, too. [15:39:47] Maybe :) [15:39:51] well, that's when it was busted [15:39:55] we didn't notice until last week [15:39:57] it looks like since it didn't get new information it just was graphing the last value [15:40:07] checked out the rrd [15:40:07] that is a terrible thing to do ganglia! [15:40:16] andrewbogott: so it'd be nice to save you from some time in that front too, like how you modularized some NFS upload bits that were unused [15:40:47] LeslieCarr: i'm looking a bit at t his page right now: [15:40:47] http://monami.sourceforge.net/tutorial/ar01s06.html [15:40:51] about tn, tmax and dmax [15:41:02] do you know if/how I can see what those values are for this metric? [15:41:22] Yeah, well… my inclination is always to change things as little as possible in a given patch (Which includes leaving in dead code unless I really hate it.) I'll have to resist that impulse and Be Bold [15:41:36] heh [15:41:38] Or enlist other people to be bold so I don't have to :) [15:41:48] maybe different patches [15:41:56] like, does ganglia save this stuff in a db somewhere I can look at? [15:42:06] gerrit is super slow now [15:42:15] hrm, [15:43:20] hm, ok, right now on my test instance (with latest puppet) I see "cat: /var/run/rsyncd.pid: No such file or directory" [15:43:24] oh it has dmax as 86400 seconds [15:43:31] lemme see what's happening [15:43:34] modules/ganglia_new/templates/gmond.conf.erb [15:43:50] though are we using ganglia_new or templates/ganglia/gmond_template.erb [15:43:52] and why do we have both [15:43:53] grrr [15:44:12] haha [15:44:26] andrewbogott! i need to understand ganglia and the like! maybe that's one I could work on refactoring :p [15:44:35] we are not using ganglia_new [15:44:37] gah [15:44:37] that's ganglia_new [15:44:40] yes we do [15:44:43] we are ? [15:44:48] for ganglia::web class ? [15:44:55] oh, no [15:44:56] because it looks like we're not for nickel .... [15:45:09] or ganglia::collector [15:45:17] !log reedy rebuilt wikiversions.cdb and synchronized wikiversions files: testwiki back to 1.22wmf9 [15:45:26] Logged the message, Master [15:45:42] mark started to rewrite our ganglia manifests due to shortcomings of the current system [15:45:49] I don't remember the exact details [15:46:05] but ganglia_new is being used in at least esams [15:46:06] New patchset: Reedy; "testwiki to 1.22wmf9" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/71077 [15:46:29] and the plan is to replace ganglia.pp completely with it, but that hasn't happened yet [15:46:35] Change merged: jenkins-bot; [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/71077 [15:47:14] nope we're not using it in production if we believe site.pp [15:47:26] except for hooft [15:47:27] class { "ganglia_new::monitor::aggregator": sites => ["esams"] } [15:47:31] hooft [15:47:34] as I said, esams :) [15:47:40] and torrus [15:47:43] so not all esams [15:47:46] for one aggregation machine [15:47:50] and for torrus [15:48:05] for /the/ aggregation machine I think [15:48:32] the aggregation machine is nickel [15:48:44] the esams aggregation machine that is [15:48:47] if we're going for it in the proper sense [15:49:14] so analytics cluster has 2 ganglia aggregators [15:49:16] isn't that obvious? :) [15:49:18] an03 and an11 [15:49:20] yep [15:49:28] system_role { "ganglia::monitor::aggregator": description => "central Ganglia aggregator" } [15:49:33] central Ganglia aggregator for esams [15:49:41] so, jmxtrans was set to send metrics to 239.192.1.32 [15:49:53] which I think is the analytics ganglia multicast group…uh, right? [15:50:04] then the aggregators pick that up, and then send that over to nickel? [15:50:14] except that it's not … start looking at line 271 in ganglia.pp [15:50:43] !g I8ab15f899f0333428fd8b2a98c58c07c2fce2962 [15:50:43] https://gerrit.wikimedia.org/r/#q,I8ab15f899f0333428fd8b2a98c58c07c2fce2962,n,z [15:50:53] ottomata: yep, the aggregators pick that up and then nickel queries the aggregators [15:51:08] though DMAX is set to a day so it should ignore things older than a day [15:51:13] to thebest of my knowledge should [15:51:15] and obviously isn't [15:51:18] right but in this case yeah [15:51:23] should I be able to query the aggregators manually maybe to see what the metric value is there? [15:52:36] yeah .. i sorta cheated and looked at the rrd on nickelto see the age of the metric [15:54:08] does it say its old or the values are recent? [15:54:51] ok! I think I'm just about to have a working package. [15:55:09] There were some bumps along the way and I've learned a bunch of fun stuff. [15:55:35] so, who can create a repository on gerrit for me to shove this into? [15:55:41] manybubbles: i can [15:56:12] yay! also, what kind of review process would be good for this? [15:56:31] as well as anyone in https://gerrit.wikimedia.org/r/#/admin/groups/119,members :-] [15:56:48] I think we have a page somewhere on mw.org to request repos [15:57:03] cool [15:57:15] the package is called 'jmxtrans'. [15:57:27] manybubbles: https://www.mediawiki.org/wiki/Git/New_repositories/Requests :) [15:57:43] click the request button :-] [15:57:53] I'm so doing that [15:59:52] you probably want [15:59:53] operations/debs/jmxtrans [16:01:56] so requested. almost. I left a typo and can't find the page to edit it [16:02:16] manybubbles: created :) [16:02:36] hashar: thanks! [16:02:48] ottomata: oh the value is old, since 3-11 (that's how i knew 3-11) [16:03:13] oh ok [16:03:15] so hm. [16:03:19] does it have lots of repeat values? [16:03:20] or just the one? [16:03:50] can't tell ... [16:04:20] hashar: it is empty, mind if I push the upstream stuff to it? [16:04:45] I don't think you can push [16:05:22] hm [16:05:35] i'm going to restart the aggregator and see what happens [16:06:25] manybubbles: operations/debs/* repos inherits rights from operations/debs.git https://gerrit.wikimedia.org/r/#/admin/projects/operations/debs,access [16:06:33] which is basically under ops responsability [16:07:10] so you will need someone from ops to import the upstream repo under the upstream branch [16:07:40] RECOVERY - Puppet freshness on analytics1001 is OK: puppet ran at Fri Jun 28 16:07:34 UTC 2013 [16:07:41] RECOVERY - Puppet freshness on mw111 is OK: puppet ran at Fri Jun 28 16:07:35 UTC 2013 [16:07:41] RECOVERY - Puppet freshness on srv285 is OK: puppet ran at Fri Jun 28 16:07:36 UTC 2013 [16:07:41] RECOVERY - Puppet freshness on search27 is OK: puppet ran at Fri Jun 28 16:07:37 UTC 2013 [16:07:41] RECOVERY - Puppet freshness on mw87 is OK: puppet ran at Fri Jun 28 16:07:39 UTC 2013 [16:07:50] RECOVERY - Puppet freshness on cp3006 is OK: puppet ran at Fri Jun 28 16:07:41 UTC 2013 [16:07:50] RECOVERY - Puppet freshness on search1006 is OK: puppet ran at Fri Jun 28 16:07:43 UTC 2013 [16:07:50] RECOVERY - Puppet freshness on mw1219 is OK: puppet ran at Fri Jun 28 16:07:44 UTC 2013 [16:07:50] RECOVERY - Puppet freshness on mw1093 is OK: puppet ran at Fri Jun 28 16:07:45 UTC 2013 [16:07:50] RECOVERY - Puppet freshness on search1011 is OK: puppet ran at Fri Jun 28 16:07:47 UTC 2013 [16:07:51] RECOVERY - Puppet freshness on mw1072 is OK: puppet ran at Fri Jun 28 16:07:48 UTC 2013 [16:07:59] RECOVERY - Puppet freshness on virt2 is OK: puppet ran at Fri Jun 28 16:07:50 UTC 2013 [16:07:59] RECOVERY - Puppet freshness on mw1178 is OK: puppet ran at Fri Jun 28 16:07:51 UTC 2013 [16:07:59] RECOVERY - Puppet freshness on db31 is OK: puppet ran at Fri Jun 28 16:07:52 UTC 2013 [16:07:59] RECOVERY - Puppet freshness on professor is OK: puppet ran at Fri Jun 28 16:07:53 UTC 2013 [16:07:59] RECOVERY - Puppet freshness on lvs1001 is OK: puppet ran at Fri Jun 28 16:07:54 UTC 2013 [16:08:00] RECOVERY - Puppet freshness on db1011 is OK: puppet ran at Fri Jun 28 16:07:55 UTC 2013 [16:08:00] RECOVERY - Puppet freshness on mw1031 is OK: puppet ran at Fri Jun 28 16:07:56 UTC 2013 [16:08:01] RECOVERY - Puppet freshness on db29 is OK: puppet ran at Fri Jun 28 16:07:57 UTC 2013 [16:08:01] RECOVERY - Puppet freshness on ms-be10 is OK: puppet ran at Fri Jun 28 16:07:57 UTC 2013 [16:08:10] RECOVERY - Puppet freshness on mc8 is OK: puppet ran at Fri Jun 28 16:07:59 UTC 2013 [16:08:11] RECOVERY - Puppet freshness on mc1013 is OK: puppet ran at Fri Jun 28 16:08:00 UTC 2013 [16:08:11] RECOVERY - Puppet freshness on sq82 is OK: puppet ran at Fri Jun 28 16:08:00 UTC 2013 [16:08:11] RECOVERY - Puppet freshness on db59 is OK: puppet ran at Fri Jun 28 16:08:00 UTC 2013 [16:08:11] RECOVERY - Puppet freshness on sq78 is OK: puppet ran at Fri Jun 28 16:08:01 UTC 2013 [16:08:11] RECOVERY - Puppet freshness on cp1015 is OK: puppet ran at Fri Jun 28 16:08:01 UTC 2013 [16:08:11] RECOVERY - Puppet freshness on mc1016 is OK: puppet ran at Fri Jun 28 16:08:03 UTC 2013 [16:08:12] RECOVERY - Puppet freshness on amssq32 is OK: puppet ran at Fri Jun 28 16:08:05 UTC 2013 [16:08:12] RECOVERY - Puppet freshness on wtp1010 is OK: puppet ran at Fri Jun 28 16:08:06 UTC 2013 [16:08:13] RECOVERY - Puppet freshness on ms-be1002 is OK: puppet ran at Fri Jun 28 16:08:07 UTC 2013 [16:08:16] manybubbles: I can't even create the upstream branch :( [16:08:21] RECOVERY - Puppet freshness on db73 is OK: puppet ran at Fri Jun 28 16:08:08 UTC 2013 [16:08:21] RECOVERY - Puppet freshness on amssq43 is OK: puppet ran at Fri Jun 28 16:08:12 UTC 2013 [16:08:21] RECOVERY - Puppet freshness on mw1080 is OK: puppet ran at Fri Jun 28 16:08:13 UTC 2013 [16:08:21] RECOVERY - Puppet freshness on mw114 is OK: puppet ran at Fri Jun 28 16:08:16 UTC 2013 [16:08:22] RECOVERY - Puppet freshness on srv270 is OK: puppet ran at Fri Jun 28 16:08:17 UTC 2013 [16:08:22] RECOVERY - Puppet freshness on mw1134 is OK: puppet ran at Fri Jun 28 16:08:18 UTC 2013 [16:08:23] manybubbles: that needs at least one commit in the repository [16:08:30] manybubbles, i can do that for you [16:08:30] RECOVERY - Puppet freshness on mw115 is OK: puppet ran at Fri Jun 28 16:08:21 UTC 2013 [16:08:30] RECOVERY - Puppet freshness on srv280 is OK: puppet ran at Fri Jun 28 16:08:22 UTC 2013 [16:08:30] RECOVERY - Puppet freshness on srv259 is OK: puppet ran at Fri Jun 28 16:08:24 UTC 2013 [16:08:31] RECOVERY - Puppet freshness on srv282 is OK: puppet ran at Fri Jun 28 16:08:26 UTC 2013 [16:08:31] RECOVERY - Puppet freshness on srv240 is OK: puppet ran at Fri Jun 28 16:08:27 UTC 2013 [16:08:39] RECOVERY - Puppet freshness on mchenry is OK: puppet ran at Fri Jun 28 16:08:28 UTC 2013 [16:08:39] ahh [16:08:39] RECOVERY - Puppet freshness on mw1215 is OK: puppet ran at Fri Jun 28 16:08:31 UTC 2013 [16:08:40] RECOVERY - Puppet freshness on db60 is OK: puppet ran at Fri Jun 28 16:08:32 UTC 2013 [16:08:40] RECOVERY - Puppet freshness on analytics1018 is OK: puppet ran at Fri Jun 28 16:08:34 UTC 2013 [16:08:40] RECOVERY - Puppet freshness on mw1220 is OK: puppet ran at Fri Jun 28 16:08:35 UTC 2013 [16:08:40] RECOVERY - Puppet freshness on mw1045 is OK: puppet ran at Fri Jun 28 16:08:35 UTC 2013 [16:08:40] RECOVERY - Puppet freshness on mw1203 is OK: puppet ran at Fri Jun 28 16:08:36 UTC 2013 [16:08:41] RECOVERY - Puppet freshness on terbium is OK: puppet ran at Fri Jun 28 16:08:38 UTC 2013 [16:08:44] I have created the upstream branch [16:08:46] hashar, ottomata: thanks! [16:08:49] RECOVERY - Puppet freshness on mw1090 is OK: puppet ran at Fri Jun 28 16:08:38 UTC 2013 [16:08:49] RECOVERY - Puppet freshness on linne is OK: puppet ran at Fri Jun 28 16:08:40 UTC 2013 [16:08:49] RECOVERY - Puppet freshness on chromium is OK: puppet ran at Fri Jun 28 16:08:40 UTC 2013 [16:08:49] RECOVERY - Puppet freshness on lvs2 is OK: puppet ran at Fri Jun 28 16:08:41 UTC 2013 [16:08:49] RECOVERY - Puppet freshness on sq37 is OK: puppet ran at Fri Jun 28 16:08:41 UTC 2013 [16:08:50] RECOVERY - Puppet freshness on db1045 is OK: puppet ran at Fri Jun 28 16:08:42 UTC 2013 [16:08:50] RECOVERY - Puppet freshness on cp1035 is OK: puppet ran at Fri Jun 28 16:08:42 UTC 2013 [16:08:51] RECOVERY - Puppet freshness on mc1006 is OK: puppet ran at Fri Jun 28 16:08:43 UTC 2013 [16:08:51] RECOVERY - Puppet freshness on search1004 is OK: puppet ran at Fri Jun 28 16:08:43 UTC 2013 [16:08:52] RECOVERY - Puppet freshness on cp1053 is OK: puppet ran at Fri Jun 28 16:08:47 UTC 2013 [16:08:52] RECOVERY - Puppet freshness on ms-be1004 is OK: puppet ran at Fri Jun 28 16:08:47 UTC 2013 [16:08:53] points to the meta config though [16:08:58] so would need a force push I guess [16:08:59] RECOVERY - Puppet freshness on cp1068 is OK: puppet ran at Fri Jun 28 16:08:49 UTC 2013 [16:08:59] RECOVERY - Puppet freshness on mw1040 is OK: puppet ran at Fri Jun 28 16:08:50 UTC 2013 [16:08:59] RECOVERY - Puppet freshness on mw1141 is OK: puppet ran at Fri Jun 28 16:08:52 UTC 2013 [16:08:59] RECOVERY - Puppet freshness on stat1001 is OK: puppet ran at Fri Jun 28 16:08:53 UTC 2013 [16:08:59] RECOVERY - Puppet freshness on ssl3001 is OK: puppet ran at Fri Jun 28 16:08:57 UTC 2013 [16:09:18] RECOVERY - Puppet freshness on db44 is OK: puppet ran at Fri Jun 28 16:08:58 UTC 2013 [16:09:18] RECOVERY - Puppet freshness on db56 is OK: puppet ran at Fri Jun 28 16:08:59 UTC 2013 [16:09:18] RECOVERY - Puppet freshness on pdf2 is OK: puppet ran at Fri Jun 28 16:09:02 UTC 2013 [16:09:18] RECOVERY - Puppet freshness on es1008 is OK: puppet ran at Fri Jun 28 16:09:03 UTC 2013 [16:09:18] RECOVERY - Puppet freshness on analytics1017 is OK: puppet ran at Fri Jun 28 16:09:04 UTC 2013 [16:09:58] PROBLEM - Puppet freshness on celsus is CRITICAL: No successful Puppet run in the last 10 hours [16:10:59] hashar: it won't let me force a push. "no common ancestry" [16:11:14] hashar: i did `git push gerrit -f upstream:refs/for/upstream` [16:13:10] ahh, yeah hashar, when I create repos like this, I clone from the upstream github repo, then add gerrit remote, then push the first commit to gerrit directly [16:13:24] manybubbles: do you have changes to make? [16:13:30] :/ [16:13:30] to debian/ or whatever? [16:14:03] ottomata: yeah - I blow away the whole debian directory because it was half assed [16:14:55] ottomata: but I have to do that in the upstream branch so the pristine tar ball is built from it or else I can't build from source at all. [16:14:58] it was kind of a mess [16:15:24] hmm, you can, if you use git-buildpackage :) [16:16:11] this is what I do: http://urlpaste.com/PQsX2vnd [16:16:52] damn [16:16:56] I also need a project added :D [16:17:15] I'm pretty sure that one of the steps I take require a pristine tarball without a debian/ directory [16:17:22] but i am too lazy to do some build package stuff again [16:18:46] hmm, manybubbles, what if you set --git-upstream-branch=upstream [16:19:27] ottomata: that is the default. the problem is that upstream, as branched from their tag, contains debian control files that are hateful. [16:19:50] oh hm, ja i see, and you can't just modify them in debian? [16:20:03] instead of needing to blowaway debian/ dir in a branch? [16:20:11] I tried that [16:20:28] but when I did that the pristine-tarball contained them and the diff.gz file didn't properly kill them [16:20:34] it was...upsetting [16:20:48] hm [16:20:57] yeah that's probably fine then, maybe we shuldn't call that branch 'upstream' then [16:21:10] maybe jsut wikimedia [16:21:11] or something [16:21:21] cleaned maybe [16:21:24] sure [16:21:29] I can do that. [16:21:37] ok, so what can I help you do? [16:21:53] so I think git review lets me push branches, but I need to push the stuff from github [16:21:59] you basically just need an upstream and a cleaned/wikimedia branch? [16:22:00] right? [16:22:11] and a debian branch too [16:22:14] ja [16:22:27] if i get the stuff from github/master into gerrit/upstream [16:22:30] you can probably do the rest, right? [16:22:35] I think so [16:22:35] or wait, you want the tag? [16:22:43] I need his tags too [16:22:45] that is imperative [16:22:49] ok, lemme see if I can get everything [16:22:53] that is how we'll pull upstream changes [16:22:54] in to the same branch names in gerrit [16:23:03] so master -> master [16:23:08] v242 -> v242 [16:23:10] oo [16:23:15] i don't know how gerrit works with tags [16:27:27] I suppose I can branch from a changeset - but I'll make sure it is the latest tag [16:27:49] !log created operations/debs/jenkins-debian-glue and force pushed upstream sources from https://github.com/mika/jenkins-debian-glue [16:27:58] Logged the message, Master [16:28:19] what's that? [16:28:20] we'll see [16:28:25] gerrit is being reaaaal slow [16:28:52] hm, interesting [16:29:04] mika is a DD, we've met a number of times [16:32:57] I can't push tags in gerrit :-(( [16:33:30] it worked for me! [16:33:31] i think [16:33:35] debian developer ? [16:33:45] my first thought was "designated driver" but i don't think that's accurate [16:34:18] PROBLEM - Puppet freshness on amssq52 is CRITICAL: No successful Puppet run in the last 10 hours [16:34:18] PROBLEM - Puppet freshness on amssq55 is CRITICAL: No successful Puppet run in the last 10 hours [16:34:18] PROBLEM - Puppet freshness on cerium is CRITICAL: No successful Puppet run in the last 10 hours [16:34:18] PROBLEM - Puppet freshness on cp1012 is CRITICAL: No successful Puppet run in the last 10 hours [16:34:18] PROBLEM - Puppet freshness on cp1020 is CRITICAL: No successful Puppet run in the last 10 hours [16:36:31] PROBLEM - Puppet freshness on amssq44 is CRITICAL: No successful Puppet run in the last 10 hours [16:36:31] PROBLEM - Puppet freshness on analytics1015 is CRITICAL: No successful Puppet run in the last 10 hours [16:36:31] PROBLEM - Puppet freshness on analytics1023 is CRITICAL: No successful Puppet run in the last 10 hours [16:36:31] PROBLEM - Puppet freshness on cp1006 is CRITICAL: No successful Puppet run in the last 10 hours [16:36:31] PROBLEM - Puppet freshness on cp1050 is CRITICAL: No successful Puppet run in the last 10 hours [16:37:31] PROBLEM - Puppet freshness on amssq53 is CRITICAL: No successful Puppet run in the last 10 hours [16:37:32] PROBLEM - Puppet freshness on cp1049 is CRITICAL: No successful Puppet run in the last 10 hours [16:37:32] PROBLEM - Puppet freshness on cp1036 is CRITICAL: No successful Puppet run in the last 10 hours [16:37:32] PROBLEM - Puppet freshness on db1020 is CRITICAL: No successful Puppet run in the last 10 hours [16:37:32] PROBLEM - Puppet freshness on db43 is CRITICAL: No successful Puppet run in the last 10 hours [16:37:32] PROBLEM - Puppet freshness on db1004 is CRITICAL: No successful Puppet run in the last 10 hours [16:37:55] ok, manybubbles [16:38:05] reclone operations/debs/jmxtrans [16:38:14] !log restarting snmptrapd and smptt on neon [16:38:14] you should have some branches [16:38:14] and also the v242 tag now [16:38:18] Logged the message, Mistress of the network gear. [16:38:26] RECOVERY - Puppet freshness on ms-be1010 is OK: puppet ran at Fri Jun 28 16:38:10 UTC 2013 [16:38:26] RECOVERY - Puppet freshness on harmon is OK: puppet ran at Fri Jun 28 16:38:13 UTC 2013 [16:38:26] RECOVERY - Puppet freshness on cp3021 is OK: puppet ran at Fri Jun 28 16:38:14 UTC 2013 [16:38:26] RECOVERY - Puppet freshness on cp3019 is OK: puppet ran at Fri Jun 28 16:38:14 UTC 2013 [16:38:26] RECOVERY - Puppet freshness on mw65 is OK: puppet ran at Fri Jun 28 16:38:15 UTC 2013 [16:38:26] RECOVERY - Puppet freshness on wtp1016 is OK: puppet ran at Fri Jun 28 16:38:19 UTC 2013 [16:38:29] i created the wikimedia branch (sorry, liked it better than 'cleaned' :/) [16:38:31] RECOVERY - Puppet freshness on mw91 is OK: puppet ran at Fri Jun 28 16:38:20 UTC 2013 [16:38:31] RECOVERY - Puppet freshness on mw59 is OK: puppet ran at Fri Jun 28 16:38:21 UTC 2013 [16:38:31] PROBLEM - Puppet freshness on cp1002 is CRITICAL: No successful Puppet run in the last 10 hours [16:38:31] PROBLEM - Puppet freshness on cp1005 is CRITICAL: No successful Puppet run in the last 10 hours [16:38:31] PROBLEM - Puppet freshness on analytics1014 is CRITICAL: No successful Puppet run in the last 10 hours [16:38:36] it does not have a debian/ dir [16:38:39] you shoudl be abel to branch from that [16:38:41] and create away [16:38:51] and then push your addition of debian/ for review [16:38:51] RECOVERY - Puppet freshness on mw1139 is OK: puppet ran at Fri Jun 28 16:38:43 UTC 2013 [16:38:51] RECOVERY - Puppet freshness on ms1004 is OK: puppet ran at Fri Jun 28 16:38:45 UTC 2013 [16:38:52] RECOVERY - Puppet freshness on emery is OK: puppet ran at Fri Jun 28 16:38:46 UTC 2013 [16:38:52] RECOVERY - Puppet freshness on search23 is OK: puppet ran at Fri Jun 28 16:38:47 UTC 2013 [16:38:52] RECOVERY - Puppet freshness on nitrogen is OK: puppet ran at Fri Jun 28 16:38:50 UTC 2013 [16:39:01] RECOVERY - Puppet freshness on mc13 is OK: puppet ran at Fri Jun 28 16:38:50 UTC 2013 [16:39:01] cool [16:39:01] RECOVERY - Puppet freshness on virt1005 is OK: puppet ran at Fri Jun 28 16:38:51 UTC 2013 [16:39:01] RECOVERY - Puppet freshness on virt8 is OK: puppet ran at Fri Jun 28 16:38:51 UTC 2013 [16:39:01] RECOVERY - Puppet freshness on cp1008 is OK: puppet ran at Fri Jun 28 16:38:52 UTC 2013 [16:39:01] RECOVERY - Puppet freshness on db1046 is OK: puppet ran at Fri Jun 28 16:38:52 UTC 2013 [16:39:02] RECOVERY - Puppet freshness on hydrogen is OK: puppet ran at Fri Jun 28 16:38:53 UTC 2013 [16:39:02] RECOVERY - Puppet freshness on sq36 is OK: puppet ran at Fri Jun 28 16:38:54 UTC 2013 [16:39:03] RECOVERY - Puppet freshness on cp1064 is OK: puppet ran at Fri Jun 28 16:38:55 UTC 2013 [16:39:03] RECOVERY - Puppet freshness on db1018 is OK: puppet ran at Fri Jun 28 16:38:55 UTC 2013 [16:39:04] RECOVERY - Puppet freshness on srv243 is OK: puppet ran at Fri Jun 28 16:38:56 UTC 2013 [16:39:04] RECOVERY - Puppet freshness on srv251 is OK: puppet ran at Fri Jun 28 16:38:57 UTC 2013 [16:39:05] RECOVERY - Puppet freshness on mw1073 is OK: puppet ran at Fri Jun 28 16:38:58 UTC 2013 [16:39:05] RECOVERY - Puppet freshness on mw1026 is OK: puppet ran at Fri Jun 28 16:38:59 UTC 2013 [16:39:11] RECOVERY - Puppet freshness on mw92 is OK: puppet ran at Fri Jun 28 16:39:00 UTC 2013 [16:39:11] RECOVERY - Puppet freshness on mw1120 is OK: puppet ran at Fri Jun 28 16:39:01 UTC 2013 [16:39:11] RECOVERY - Puppet freshness on mw25 is OK: puppet ran at Fri Jun 28 16:39:01 UTC 2013 [16:39:11] RECOVERY - Puppet freshness on mw7 is OK: puppet ran at Fri Jun 28 16:39:05 UTC 2013 [16:39:11] RECOVERY - Puppet freshness on db1015 is OK: puppet ran at Fri Jun 28 16:39:06 UTC 2013 [16:39:12] RECOVERY - Puppet freshness on mw1008 is OK: puppet ran at Fri Jun 28 16:39:07 UTC 2013 [16:39:21] RECOVERY - Puppet freshness on cp1009 is OK: puppet ran at Fri Jun 28 16:39:17 UTC 2013 [16:39:21] RECOVERY - Puppet freshness on srv293 is OK: puppet ran at Fri Jun 28 16:39:18 UTC 2013 [16:39:21] RECOVERY - Puppet freshness on mc1002 is OK: puppet ran at Fri Jun 28 16:39:19 UTC 2013 [16:39:32] RECOVERY - Puppet freshness on sq66 is OK: puppet ran at Fri Jun 28 16:39:21 UTC 2013 [16:39:32] PROBLEM - Puppet freshness on amslvs1 is CRITICAL: No successful Puppet run in the last 10 hours [16:39:32] PROBLEM - Puppet freshness on amssq47 is CRITICAL: No successful Puppet run in the last 10 hours [16:39:32] PROBLEM - Puppet freshness on amslvs4 is CRITICAL: No successful Puppet run in the last 10 hours [16:39:32] PROBLEM - Puppet freshness on analytics1004 is CRITICAL: No successful Puppet run in the last 10 hours [16:39:32] manybubbles: i remember it being a little different to push to non-master remote branch for review, if it doesn't exist yet, let me know if you have problems with that [16:39:36] i've done it before though [16:39:48] but yeah, just to keep things in sync [16:39:49] ottomata: I was just looking that up. [16:39:55] master is the upstream branch [16:39:57] not 'upstream' [16:40:15] so you'll need to set that [16:40:41] RECOVERY - Puppet freshness on tmh1002 is OK: puppet ran at Fri Jun 28 16:40:30 UTC 2013 [16:40:41] RECOVERY - Puppet freshness on gallium is OK: puppet ran at Fri Jun 28 16:40:31 UTC 2013 [16:40:41] RECOVERY - Puppet freshness on ssl1004 is OK: puppet ran at Fri Jun 28 16:40:32 UTC 2013 [16:40:41] RECOVERY - Puppet freshness on analytics1006 is OK: puppet ran at Fri Jun 28 16:40:33 UTC 2013 [16:40:41] RECOVERY - Puppet freshness on cp1019 is OK: puppet ran at Fri Jun 28 16:40:35 UTC 2013 [16:40:41] RECOVERY - Puppet freshness on es7 is OK: puppet ran at Fri Jun 28 16:40:36 UTC 2013 [16:40:42] RECOVERY - Puppet freshness on db38 is OK: puppet ran at Fri Jun 28 16:40:37 UTC 2013 [16:40:42] RECOVERY - Puppet freshness on mw96 is OK: puppet ran at Fri Jun 28 16:40:40 UTC 2013 [16:40:51] RECOVERY - Puppet freshness on ms-fe4 is OK: puppet ran at Fri Jun 28 16:40:41 UTC 2013 [16:40:51] RECOVERY - Puppet freshness on db37 is OK: puppet ran at Fri Jun 28 16:40:42 UTC 2013 [16:40:51] RECOVERY - Puppet freshness on db1028 is OK: puppet ran at Fri Jun 28 16:40:43 UTC 2013 [16:40:51] RECOVERY - Puppet freshness on mw120 is OK: puppet ran at Fri Jun 28 16:40:46 UTC 2013 [16:40:51] RECOVERY - Puppet freshness on virt10 is OK: puppet ran at Fri Jun 28 16:40:46 UTC 2013 [16:40:51] RECOVERY - Puppet freshness on mw1172 is OK: puppet ran at Fri Jun 28 16:40:48 UTC 2013 [16:41:01] RECOVERY - Puppet freshness on silver is OK: puppet ran at Fri Jun 28 16:40:50 UTC 2013 [16:41:01] RECOVERY - Puppet freshness on mw78 is OK: puppet ran at Fri Jun 28 16:40:51 UTC 2013 [16:41:01] RECOVERY - Puppet freshness on db1043 is OK: puppet ran at Fri Jun 28 16:40:52 UTC 2013 [16:41:01] RECOVERY - Puppet freshness on wtp1005 is OK: puppet ran at Fri Jun 28 16:40:53 UTC 2013 [16:41:01] RECOVERY - Puppet freshness on wtp1012 is OK: puppet ran at Fri Jun 28 16:40:54 UTC 2013 [16:41:01] RECOVERY - Puppet freshness on sq61 is OK: puppet ran at Fri Jun 28 16:40:55 UTC 2013 [16:41:01] RECOVERY - Puppet freshness on sq53 is OK: puppet ran at Fri Jun 28 16:40:56 UTC 2013 [16:41:02] RECOVERY - Puppet freshness on search1009 is OK: puppet ran at Fri Jun 28 16:40:56 UTC 2013 [16:41:02] RECOVERY - Puppet freshness on es1007 is OK: puppet ran at Fri Jun 28 16:40:56 UTC 2013 [16:41:03] RECOVERY - Puppet freshness on hooper is OK: puppet ran at Fri Jun 28 16:40:57 UTC 2013 [16:41:03] RECOVERY - Puppet freshness on db50 is OK: puppet ran at Fri Jun 28 16:40:58 UTC 2013 [16:41:04] RECOVERY - Puppet freshness on db1007 is OK: puppet ran at Fri Jun 28 16:40:58 UTC 2013 [16:41:04] RECOVERY - Puppet freshness on db1003 is OK: puppet ran at Fri Jun 28 16:40:59 UTC 2013 [16:41:05] RECOVERY - Puppet freshness on db1039 is OK: puppet ran at Fri Jun 28 16:40:59 UTC 2013 [16:41:05] RECOVERY - Puppet freshness on es3 is OK: puppet ran at Fri Jun 28 16:41:00 UTC 2013 [16:41:11] RECOVERY - Puppet freshness on search26 is OK: puppet ran at Fri Jun 28 16:41:01 UTC 2013 [16:41:12] RECOVERY - Puppet freshness on cp1054 is OK: puppet ran at Fri Jun 28 16:41:02 UTC 2013 [16:41:12] RECOVERY - Puppet freshness on mc1014 is OK: puppet ran at Fri Jun 28 16:41:02 UTC 2013 [16:41:12] RECOVERY - Puppet freshness on amssq58 is OK: puppet ran at Fri Jun 28 16:41:03 UTC 2013 [16:41:21] RECOVERY - Puppet freshness on nfs1 is OK: puppet ran at Fri Jun 28 16:41:11 UTC 2013 [16:41:21] RECOVERY - Puppet freshness on db1041 is OK: puppet ran at Fri Jun 28 16:41:12 UTC 2013 [16:41:21] RECOVERY - Puppet freshness on nfs2 is OK: puppet ran at Fri Jun 28 16:41:16 UTC 2013 [16:41:21] RECOVERY - Puppet freshness on db33 is OK: puppet ran at Fri Jun 28 16:41:17 UTC 2013 [16:41:21] RECOVERY - Puppet freshness on mc1010 is OK: puppet ran at Fri Jun 28 16:41:18 UTC 2013 [16:41:21] RECOVERY - Puppet freshness on mw1149 is OK: puppet ran at Fri Jun 28 16:41:19 UTC 2013 [16:41:21] RECOVERY - Puppet freshness on mw1167 is OK: puppet ran at Fri Jun 28 16:41:20 UTC 2013 [16:41:32] RECOVERY - Puppet freshness on mw1055 is OK: puppet ran at Fri Jun 28 16:41:21 UTC 2013 [16:41:32] RECOVERY - Puppet freshness on mw69 is OK: puppet ran at Fri Jun 28 16:41:22 UTC 2013 [16:41:32] RECOVERY - Puppet freshness on mw1195 is OK: puppet ran at Fri Jun 28 16:41:23 UTC 2013 [16:41:32] RECOVERY - Puppet freshness on mw1036 is OK: puppet ran at Fri Jun 28 16:41:26 UTC 2013 [16:41:32] RECOVERY - Puppet freshness on mw63 is OK: puppet ran at Fri Jun 28 16:41:27 UTC 2013 [16:41:32] RECOVERY - Puppet freshness on mw84 is OK: puppet ran at Fri Jun 28 16:41:28 UTC 2013 [16:41:32] RECOVERY - Puppet freshness on srv254 is OK: puppet ran at Fri Jun 28 16:41:29 UTC 2013 [16:41:41] RECOVERY - Puppet freshness on mw1035 is OK: puppet ran at Fri Jun 28 16:41:32 UTC 2013 [16:41:41] RECOVERY - Puppet freshness on cp1044 is OK: puppet ran at Fri Jun 28 16:41:33 UTC 2013 [16:41:41] RECOVERY - Puppet freshness on mw52 is OK: puppet ran at Fri Jun 28 16:41:35 UTC 2013 [16:41:42] RECOVERY - Puppet freshness on mw80 is OK: puppet ran at Fri Jun 28 16:41:36 UTC 2013 [16:41:42] RECOVERY - Puppet freshness on sq69 is OK: puppet ran at Fri Jun 28 16:41:37 UTC 2013 [16:41:42] RECOVERY - Puppet freshness on analytics1026 is OK: puppet ran at Fri Jun 28 16:41:38 UTC 2013 [16:41:42] RECOVERY - Puppet freshness on searchidx2 is OK: puppet ran at Fri Jun 28 16:41:39 UTC 2013 [16:41:42] RECOVERY - Puppet freshness on snapshot1001 is OK: puppet ran at Fri Jun 28 16:41:40 UTC 2013 [16:41:52] RECOVERY - Puppet freshness on mw1013 is OK: puppet ran at Fri Jun 28 16:41:40 UTC 2013 [16:41:52] RECOVERY - Puppet freshness on amslvs3 is OK: puppet ran at Fri Jun 28 16:41:42 UTC 2013 [16:41:52] RECOVERY - Puppet freshness on mw71 is OK: puppet ran at Fri Jun 28 16:41:43 UTC 2013 [16:41:52] RECOVERY - Puppet freshness on mw1168 is OK: puppet ran at Fri Jun 28 16:41:44 UTC 2013 [16:41:52] RECOVERY - Puppet freshness on mw17 is OK: puppet ran at Fri Jun 28 16:41:45 UTC 2013 [16:41:52] RECOVERY - Puppet freshness on mw1051 is OK: puppet ran at Fri Jun 28 16:41:46 UTC 2013 [16:41:52] RECOVERY - Puppet freshness on search24 is OK: puppet ran at Fri Jun 28 16:41:47 UTC 2013 [16:41:53] RECOVERY - Puppet freshness on cp3022 is OK: puppet ran at Fri Jun 28 16:41:48 UTC 2013 [16:41:53] RECOVERY - Puppet freshness on mw1192 is OK: puppet ran at Fri Jun 28 16:41:49 UTC 2013 [16:42:02] RECOVERY - Puppet freshness on mw86 is OK: puppet ran at Fri Jun 28 16:41:51 UTC 2013 [16:42:02] RECOVERY - Puppet freshness on search1002 is OK: puppet ran at Fri Jun 28 16:41:53 UTC 2013 [16:42:02] RECOVERY - Puppet freshness on mw88 is OK: puppet ran at Fri Jun 28 16:41:54 UTC 2013 [16:42:02] RECOVERY - Puppet freshness on mw1202 is OK: puppet ran at Fri Jun 28 16:41:56 UTC 2013 [16:42:02] RECOVERY - Puppet freshness on mw1079 is OK: puppet ran at Fri Jun 28 16:41:57 UTC 2013 [16:42:02] RECOVERY - Puppet freshness on ssl3003 is OK: puppet ran at Fri Jun 28 16:41:58 UTC 2013 [16:42:03] RECOVERY - Puppet freshness on mw1076 is OK: puppet ran at Fri Jun 28 16:41:59 UTC 2013 [16:42:11] RECOVERY - Puppet freshness on mw1216 is OK: puppet ran at Fri Jun 28 16:42:01 UTC 2013 [16:42:11] RECOVERY - Puppet freshness on rdb1001 is OK: puppet ran at Fri Jun 28 16:42:03 UTC 2013 [16:42:11] RECOVERY - Puppet freshness on db49 is OK: puppet ran at Fri Jun 28 16:42:03 UTC 2013 [16:42:11] RECOVERY - Puppet freshness on cp1011 is OK: puppet ran at Fri Jun 28 16:42:06 UTC 2013 [16:42:11] RECOVERY - Puppet freshness on lvs1003 is OK: puppet ran at Fri Jun 28 16:42:07 UTC 2013 [16:42:11] RECOVERY - Puppet freshness on cp1004 is OK: puppet ran at Fri Jun 28 16:42:08 UTC 2013 [16:42:11] RECOVERY - Puppet freshness on ms-be1008 is OK: puppet ran at Fri Jun 28 16:42:09 UTC 2013 [16:42:12] RECOVERY - Puppet freshness on formey is OK: puppet ran at Fri Jun 28 16:42:10 UTC 2013 [16:42:21] RECOVERY - Puppet freshness on virt1007 is OK: puppet ran at Fri Jun 28 16:42:11 UTC 2013 [16:42:21] RECOVERY - Puppet freshness on analytics1020 is OK: puppet ran at Fri Jun 28 16:42:12 UTC 2013 [16:42:21] RECOVERY - Puppet freshness on db72 is OK: puppet ran at Fri Jun 28 16:42:13 UTC 2013 [16:42:21] RECOVERY - Puppet freshness on mc1004 is OK: puppet ran at Fri Jun 28 16:42:14 UTC 2013 [16:42:21] RECOVERY - Puppet freshness on iron is OK: puppet ran at Fri Jun 28 16:42:16 UTC 2013 [16:42:22] RECOVERY - Puppet freshness on db1016 is OK: puppet ran at Fri Jun 28 16:42:17 UTC 2013 [16:42:22] RECOVERY - Puppet freshness on sq70 is OK: puppet ran at Fri Jun 28 16:42:18 UTC 2013 [16:42:23] RECOVERY - Puppet freshness on db1030 is OK: puppet ran at Fri Jun 28 16:42:19 UTC 2013 [16:42:23] RECOVERY - Puppet freshness on db74 is OK: puppet ran at Fri Jun 28 16:42:20 UTC 2013 [16:42:33] RECOVERY - Puppet freshness on virt7 is OK: puppet ran at Fri Jun 28 16:42:20 UTC 2013 [16:42:33] RECOVERY - Puppet freshness on wtp1008 is OK: puppet ran at Fri Jun 28 16:42:22 UTC 2013 [16:42:33] RECOVERY - Puppet freshness on mw1014 is OK: puppet ran at Fri Jun 28 16:42:23 UTC 2013 [16:42:33] RECOVERY - Puppet freshness on oxygen is OK: puppet ran at Fri Jun 28 16:42:24 UTC 2013 [16:42:33] RECOVERY - Puppet freshness on hooft is OK: puppet ran at Fri Jun 28 16:42:25 UTC 2013 [16:42:33] RECOVERY - Puppet freshness on mw1133 is OK: puppet ran at Fri Jun 28 16:42:26 UTC 2013 [16:42:33] RECOVERY - Puppet freshness on search32 is OK: puppet ran at Fri Jun 28 16:42:28 UTC 2013 [16:42:34] RECOVERY - Puppet freshness on mw85 is OK: puppet ran at Fri Jun 28 16:42:29 UTC 2013 [16:42:41] RECOVERY - Puppet freshness on mw1151 is OK: puppet ran at Fri Jun 28 16:42:30 UTC 2013 [16:42:41] RECOVERY - Puppet freshness on es8 is OK: puppet ran at Fri Jun 28 16:42:33 UTC 2013 [16:42:42] RECOVERY - Puppet freshness on db1053 is OK: puppet ran at Fri Jun 28 16:42:35 UTC 2013 [16:42:42] RECOVERY - Puppet freshness on db1059 is OK: puppet ran at Fri Jun 28 16:42:35 UTC 2013 [16:42:42] RECOVERY - Puppet freshness on ms1002 is OK: puppet ran at Fri Jun 28 16:42:36 UTC 2013 [16:42:42] RECOVERY - Puppet freshness on es1006 is OK: puppet ran at Fri Jun 28 16:42:37 UTC 2013 [16:42:42] RECOVERY - Puppet freshness on lvs4 is OK: puppet ran at Fri Jun 28 16:42:37 UTC 2013 [16:42:42] RECOVERY - Puppet freshness on sq75 is OK: puppet ran at Fri Jun 28 16:42:39 UTC 2013 [16:42:43] RECOVERY - Puppet freshness on amssq46 is OK: puppet ran at Fri Jun 28 16:42:40 UTC 2013 [16:42:51] RECOVERY - Puppet freshness on amssq62 is OK: puppet ran at Fri Jun 28 16:42:40 UTC 2013 [16:42:51] RECOVERY - Puppet freshness on mw72 is OK: puppet ran at Fri Jun 28 16:42:41 UTC 2013 [16:42:51] RECOVERY - Puppet freshness on mw123 is OK: puppet ran at Fri Jun 28 16:42:42 UTC 2013 [16:42:51] RECOVERY - Puppet freshness on mw1098 is OK: puppet ran at Fri Jun 28 16:42:43 UTC 2013 [16:42:51] RECOVERY - Puppet freshness on cp1055 is OK: puppet ran at Fri Jun 28 16:42:44 UTC 2013 [16:42:51] RECOVERY - Puppet freshness on srv263 is OK: puppet ran at Fri Jun 28 16:42:48 UTC 2013 [16:42:51] RECOVERY - Puppet freshness on cp1041 is OK: puppet ran at Fri Jun 28 16:42:49 UTC 2013 [16:43:01] RECOVERY - Puppet freshness on ssl3002 is OK: puppet ran at Fri Jun 28 16:42:50 UTC 2013 [16:43:01] RECOVERY - Puppet freshness on db78 is OK: puppet ran at Fri Jun 28 16:42:51 UTC 2013 [16:43:01] RECOVERY - Puppet freshness on cp1027 is OK: puppet ran at Fri Jun 28 16:42:52 UTC 2013 [16:43:01] RECOVERY - Puppet freshness on mw4 is OK: puppet ran at Fri Jun 28 16:42:53 UTC 2013 [16:43:01] RECOVERY - Puppet freshness on wtp1023 is OK: puppet ran at Fri Jun 28 16:42:56 UTC 2013 [16:43:01] RECOVERY - Puppet freshness on mw1034 is OK: puppet ran at Fri Jun 28 16:42:58 UTC 2013 [16:43:01] RECOVERY - Puppet freshness on mw1183 is OK: puppet ran at Fri Jun 28 16:42:59 UTC 2013 [16:43:02] what's up? [16:43:11] RECOVERY - Puppet freshness on mw18 is OK: puppet ran at Fri Jun 28 16:43:00 UTC 2013 [16:43:11] RECOVERY - Puppet freshness on mw97 is OK: puppet ran at Fri Jun 28 16:43:01 UTC 2013 [16:43:11] RECOVERY - Puppet freshness on mw1050 is OK: puppet ran at Fri Jun 28 16:43:04 UTC 2013 [16:43:11] RECOVERY - Puppet freshness on ssl3 is OK: puppet ran at Fri Jun 28 16:43:05 UTC 2013 [16:43:11] RECOVERY - Puppet freshness on mw1145 is OK: puppet ran at Fri Jun 28 16:43:06 UTC 2013 [16:43:11] RECOVERY - Puppet freshness on dysprosium is OK: puppet ran at Fri Jun 28 16:43:06 UTC 2013 [16:43:11] RECOVERY - Puppet freshness on wtp1022 is OK: puppet ran at Fri Jun 28 16:43:08 UTC 2013 [16:43:12] RECOVERY - Puppet freshness on srv298 is OK: puppet ran at Fri Jun 28 16:43:09 UTC 2013 [16:43:12] RECOVERY - Puppet freshness on srv279 is OK: puppet ran at Fri Jun 28 16:43:10 UTC 2013 [16:43:21] RECOVERY - Puppet freshness on srv241 is OK: puppet ran at Fri Jun 28 16:43:11 UTC 2013 [16:43:21] RECOVERY - Puppet freshness on snapshot1004 is OK: puppet ran at Fri Jun 28 16:43:11 UTC 2013 [16:43:21] RECOVERY - Puppet freshness on srv288 is OK: puppet ran at Fri Jun 28 16:43:13 UTC 2013 [16:43:21] RECOVERY - Puppet freshness on cp1033 is OK: puppet ran at Fri Jun 28 16:43:14 UTC 2013 [16:43:21] RECOVERY - Puppet freshness on db1008 is OK: puppet ran at Fri Jun 28 16:43:14 UTC 2013 [16:43:21] RECOVERY - Puppet freshness on cp1052 is OK: puppet ran at Fri Jun 28 16:43:15 UTC 2013 [16:43:21] RECOVERY - Puppet freshness on sq80 is OK: puppet ran at Fri Jun 28 16:43:15 UTC 2013 [16:43:22] RECOVERY - Puppet freshness on db1038 is OK: puppet ran at Fri Jun 28 16:43:16 UTC 2013 [16:43:22] RECOVERY - Puppet freshness on db1006 is OK: puppet ran at Fri Jun 28 16:43:16 UTC 2013 [16:43:23] PROBLEM - Varnish HTTP text-backend on cp1038 is CRITICAL: Connection refused [16:43:31] RECOVERY - Puppet freshness on search1016 is OK: puppet ran at Fri Jun 28 16:43:20 UTC 2013 [16:43:31] RECOVERY - Puppet freshness on search1017 is OK: puppet ran at Fri Jun 28 16:43:21 UTC 2013 [16:43:31] RECOVERY - Puppet freshness on mw102 is OK: puppet ran at Fri Jun 28 16:43:21 UTC 2013 [16:43:31] RECOVERY - Puppet freshness on mw44 is OK: puppet ran at Fri Jun 28 16:43:23 UTC 2013 [16:43:31] RECOVERY - Puppet freshness on amssq38 is OK: puppet ran at Fri Jun 28 16:43:25 UTC 2013 [16:43:31] RECOVERY - Puppet freshness on db62 is OK: puppet ran at Fri Jun 28 16:43:26 UTC 2013 [16:43:43] RECOVERY - Puppet freshness on db47 is OK: puppet ran at Fri Jun 28 16:43:31 UTC 2013 [16:43:43] RECOVERY - Puppet freshness on pc3 is OK: puppet ran at Fri Jun 28 16:43:31 UTC 2013 [16:43:43] RECOVERY - Puppet freshness on mw27 is OK: puppet ran at Fri Jun 28 16:43:32 UTC 2013 [16:43:43] RECOVERY - Puppet freshness on cp1059 is OK: puppet ran at Fri Jun 28 16:43:34 UTC 2013 [16:43:43] RECOVERY - Puppet freshness on wtp1019 is OK: puppet ran at Fri Jun 28 16:43:36 UTC 2013 [16:43:43] RECOVERY - Puppet freshness on mw1077 is OK: puppet ran at Fri Jun 28 16:43:38 UTC 2013 [16:43:43] RECOVERY - Puppet freshness on virt6 is OK: puppet ran at Fri Jun 28 16:43:39 UTC 2013 [16:43:51] blugh [16:43:51] RECOVERY - Puppet freshness on es1010 is OK: puppet ran at Fri Jun 28 16:43:40 UTC 2013 [16:43:51] RECOVERY - Puppet freshness on mw1163 is OK: puppet ran at Fri Jun 28 16:43:41 UTC 2013 [16:43:51] RECOVERY - Puppet freshness on virt5 is OK: puppet ran at Fri Jun 28 16:43:45 UTC 2013 [16:43:51] RECOVERY - Puppet freshness on mw1002 is OK: puppet ran at Fri Jun 28 16:43:46 UTC 2013 [16:43:51] RECOVERY - Puppet freshness on mw1023 is OK: puppet ran at Fri Jun 28 16:43:47 UTC 2013 [16:43:51] RECOVERY - Puppet freshness on mw1212 is OK: puppet ran at Fri Jun 28 16:43:48 UTC 2013 [16:43:51] RECOVERY - Puppet freshness on mw1097 is OK: puppet ran at Fri Jun 28 16:43:49 UTC 2013 [16:44:01] RECOVERY - Puppet freshness on mw1096 is OK: puppet ran at Fri Jun 28 16:43:50 UTC 2013 [16:44:01] RECOVERY - Puppet freshness on mw1029 is OK: puppet ran at Fri Jun 28 16:43:51 UTC 2013 [16:44:01] RECOVERY - Puppet freshness on ms1001 is OK: puppet ran at Fri Jun 28 16:43:52 UTC 2013 [16:44:01] RECOVERY - Puppet freshness on locke is OK: puppet ran at Fri Jun 28 16:43:53 UTC 2013 [16:44:01] RECOVERY - Puppet freshness on sq57 is OK: puppet ran at Fri Jun 28 16:43:54 UTC 2013 [16:44:01] RECOVERY - Puppet freshness on cp1031 is OK: puppet ran at Fri Jun 28 16:43:55 UTC 2013 [16:44:01] RECOVERY - Puppet freshness on mw89 is OK: puppet ran at Fri Jun 28 16:43:56 UTC 2013 [16:44:02] RECOVERY - Puppet freshness on sq60 is OK: puppet ran at Fri Jun 28 16:43:57 UTC 2013 [16:44:02] RECOVERY - Puppet freshness on db1047 is OK: puppet ran at Fri Jun 28 16:43:58 UTC 2013 [16:44:03] RECOVERY - Puppet freshness on es1001 is OK: puppet ran at Fri Jun 28 16:43:58 UTC 2013 [16:44:03] RECOVERY - Puppet freshness on srv281 is OK: puppet ran at Fri Jun 28 16:43:59 UTC 2013 [16:44:04] RECOVERY - Puppet freshness on sq44 is OK: puppet ran at Fri Jun 28 16:43:59 UTC 2013 [16:44:04] RECOVERY - Puppet freshness on mc9 is OK: puppet ran at Fri Jun 28 16:43:59 UTC 2013 [16:44:05] RECOVERY - Puppet freshness on rdb1004 is OK: puppet ran at Fri Jun 28 16:44:00 UTC 2013 [16:44:11] RECOVERY - Puppet freshness on cp1014 is OK: puppet ran at Fri Jun 28 16:44:00 UTC 2013 [16:44:11] RECOVERY - Puppet freshness on db1017 is OK: puppet ran at Fri Jun 28 16:44:01 UTC 2013 [16:44:11] RECOVERY - Puppet freshness on mw6 is OK: puppet ran at Fri Jun 28 16:44:02 UTC 2013 [16:44:11] RECOVERY - Puppet freshness on mw1016 is OK: puppet ran at Fri Jun 28 16:44:03 UTC 2013 [16:44:11] RECOVERY - Puppet freshness on cp3008 is OK: puppet ran at Fri Jun 28 16:44:04 UTC 2013 [16:44:11] RECOVERY - Puppet freshness on search1020 is OK: puppet ran at Fri Jun 28 16:44:05 UTC 2013 [16:44:11] RECOVERY - Puppet freshness on amssq54 is OK: puppet ran at Fri Jun 28 16:44:06 UTC 2013 [16:44:12] RECOVERY - Puppet freshness on cp1070 is OK: puppet ran at Fri Jun 28 16:44:08 UTC 2013 [16:44:21] RECOVERY - Puppet freshness on kaulen is OK: puppet ran at Fri Jun 28 16:44:11 UTC 2013 [16:44:21] RECOVERY - Puppet freshness on carbon is OK: puppet ran at Fri Jun 28 16:44:15 UTC 2013 [16:44:21] RECOVERY - Puppet freshness on wtp1009 is OK: puppet ran at Fri Jun 28 16:44:16 UTC 2013 [16:44:21] RECOVERY - Puppet freshness on pc1001 is OK: puppet ran at Fri Jun 28 16:44:16 UTC 2013 [16:44:21] RECOVERY - Puppet freshness on mw1152 is OK: puppet ran at Fri Jun 28 16:44:18 UTC 2013 [16:44:21] RECOVERY - Puppet freshness on ms-be1001 is OK: puppet ran at Fri Jun 28 16:44:18 UTC 2013 [16:44:21] RECOVERY - Puppet freshness on wtp1017 is OK: puppet ran at Fri Jun 28 16:44:19 UTC 2013 [16:44:22] RECOVERY - Puppet freshness on ms-be1011 is OK: puppet ran at Fri Jun 28 16:44:19 UTC 2013 [16:44:30] see you later folks [16:44:31] RECOVERY - Puppet freshness on analytics1027 is OK: puppet ran at Fri Jun 28 16:44:21 UTC 2013 [16:44:31] RECOVERY - Puppet freshness on mw1067 is OK: puppet ran at Fri Jun 28 16:44:24 UTC 2013 [16:44:32] RECOVERY - Puppet freshness on mw1209 is OK: puppet ran at Fri Jun 28 16:44:26 UTC 2013 [16:44:32] RECOVERY - Puppet freshness on mw21 is OK: puppet ran at Fri Jun 28 16:44:27 UTC 2013 [16:44:32] RECOVERY - Puppet freshness on srv247 is OK: puppet ran at Fri Jun 28 16:44:29 UTC 2013 [16:44:41] RECOVERY - Puppet freshness on mw1028 is OK: puppet ran at Fri Jun 28 16:44:30 UTC 2013 [16:44:41] RECOVERY - Puppet freshness on mw1005 is OK: puppet ran at Fri Jun 28 16:44:32 UTC 2013 [16:44:41] PROBLEM - Apache HTTP on mw1157 is CRITICAL: Connection timed out [16:44:41] PROBLEM - Apache HTTP on mw1154 is CRITICAL: Connection timed out [16:44:41] PROBLEM - Apache HTTP on mw1156 is CRITICAL: Connection timed out [16:44:41] RECOVERY - Puppet freshness on yvon is OK: puppet ran at Fri Jun 28 16:44:34 UTC 2013 [16:44:41] RECOVERY - Puppet freshness on ms-fe1 is OK: puppet ran at Fri Jun 28 16:44:34 UTC 2013 [16:44:42] RECOVERY - Puppet freshness on cp1022 is OK: puppet ran at Fri Jun 28 16:44:35 UTC 2013 [16:44:42] RECOVERY - Puppet freshness on cp1067 is OK: puppet ran at Fri Jun 28 16:44:36 UTC 2013 [16:44:43] sorry … the recovery spam will be for a bit longer [16:44:43] RECOVERY - Puppet freshness on db1035 is OK: puppet ran at Fri Jun 28 16:44:37 UTC 2013 [16:44:43] RECOVERY - Puppet freshness on cp1029 is OK: puppet ran at Fri Jun 28 16:44:38 UTC 2013 [16:44:44] RECOVERY - Puppet freshness on mw1193 is OK: puppet ran at Fri Jun 28 16:44:39 UTC 2013 [16:44:52] RECOVERY - Puppet freshness on amssq33 is OK: puppet ran at Fri Jun 28 16:44:40 UTC 2013 [16:44:52] RECOVERY - Puppet freshness on amssq61 is OK: puppet ran at Fri Jun 28 16:44:40 UTC 2013 [16:44:52] RECOVERY - Puppet freshness on cp1040 is OK: puppet ran at Fri Jun 28 16:44:41 UTC 2013 [16:44:52] RECOVERY - Puppet freshness on amssq49 is OK: puppet ran at Fri Jun 28 16:44:41 UTC 2013 [16:44:52] RECOVERY - Puppet freshness on amssq45 is OK: puppet ran at Fri Jun 28 16:44:42 UTC 2013 [16:44:52] RECOVERY - Puppet freshness on amssq42 is OK: puppet ran at Fri Jun 28 16:44:42 UTC 2013 [16:44:52] RECOVERY - Puppet freshness on ms-be6 is OK: puppet ran at Fri Jun 28 16:44:43 UTC 2013 [16:44:52] LeslieCarr: what happened? [16:44:53] PROBLEM - Apache HTTP on mw1158 is CRITICAL: Connection timed out [16:44:53] PROBLEM - Apache HTTP on mw1160 is CRITICAL: Connection timed out [16:44:54] PROBLEM - Apache HTTP on mw1153 is CRITICAL: Connection timed out [16:44:54] RECOVERY - Puppet freshness on zhen is OK: puppet ran at Fri Jun 28 16:44:45 UTC 2013 [16:44:55] RECOVERY - Puppet freshness on db1037 is OK: puppet ran at Fri Jun 28 16:44:47 UTC 2013 [16:45:01] RECOVERY - Puppet freshness on db71 is OK: puppet ran at Fri Jun 28 16:44:51 UTC 2013 [16:45:01] RECOVERY - Puppet freshness on xenon is OK: puppet ran at Fri Jun 28 16:44:51 UTC 2013 [16:45:01] PROBLEM - Apache HTTP on mw1155 is CRITICAL: Connection timed out [16:45:01] PROBLEM - LVS HTTP IPv4 on rendering.svc.eqiad.wmnet is CRITICAL: Connection timed out [16:45:16] paravoid: the recovery is just because smptrapd froze for some reason. the rendering is something new [16:45:20] RECOVERY - Puppet freshness on tridge is OK: puppet ran at Fri Jun 28 16:44:55 UTC 2013 [16:45:20] RECOVERY - Puppet freshness on srv265 is OK: puppet ran at Fri Jun 28 16:44:57 UTC 2013 [16:45:20] RECOVERY - Puppet freshness on cp1047 is OK: puppet ran at Fri Jun 28 16:44:59 UTC 2013 [16:45:20] PROBLEM - Apache HTTP on mw1159 is CRITICAL: Connection timed out [16:45:20] RECOVERY - Puppet freshness on mw22 is OK: puppet ran at Fri Jun 28 16:45:01 UTC 2013 [16:45:20] RECOVERY - Puppet freshness on mw31 is OK: puppet ran at Fri Jun 28 16:45:02 UTC 2013 [16:45:20] RECOVERY - Puppet freshness on srv236 is OK: puppet ran at Fri Jun 28 16:45:03 UTC 2013 [16:45:21] RECOVERY - Puppet freshness on mw1012 is OK: puppet ran at Fri Jun 28 16:45:04 UTC 2013 [16:45:21] RECOVERY - Puppet freshness on srv277 is OK: puppet ran at Fri Jun 28 16:45:05 UTC 2013 [16:45:22] RECOVERY - Puppet freshness on ssl2 is OK: puppet ran at Fri Jun 28 16:45:06 UTC 2013 [16:45:22] RECOVERY - Puppet freshness on srv248 is OK: puppet ran at Fri Jun 28 16:45:08 UTC 2013 [16:45:23] RECOVERY - Puppet freshness on mw1006 is OK: puppet ran at Fri Jun 28 16:45:10 UTC 2013 [16:45:23] RECOVERY - Puppet freshness on srv256 is OK: puppet ran at Fri Jun 28 16:45:11 UTC 2013 [16:45:24] RECOVERY - Puppet freshness on mw77 is OK: puppet ran at Fri Jun 28 16:45:12 UTC 2013 [16:45:24] RECOVERY - Puppet freshness on cp3020 is OK: puppet ran at Fri Jun 28 16:45:15 UTC 2013 [16:45:25] RECOVERY - Puppet freshness on snapshot2 is OK: puppet ran at Fri Jun 28 16:45:16 UTC 2013 [16:45:25] RECOVERY - Puppet freshness on nescio is OK: puppet ran at Fri Jun 28 16:45:18 UTC 2013 [16:45:30] RECOVERY - Puppet freshness on analytics1016 is OK: puppet ran at Fri Jun 28 16:45:20 UTC 2013 [16:45:30] RECOVERY - Puppet freshness on mw19 is OK: puppet ran at Fri Jun 28 16:45:21 UTC 2013 [16:45:30] RECOVERY - Puppet freshness on search15 is OK: puppet ran at Fri Jun 28 16:45:22 UTC 2013 [16:45:30] RECOVERY - Puppet freshness on analytics1025 is OK: puppet ran at Fri Jun 28 16:45:23 UTC 2013 [16:45:31] RECOVERY - Puppet freshness on mw11 is OK: puppet ran at Fri Jun 28 16:45:24 UTC 2013 [16:45:31] RECOVERY - Puppet freshness on mw24 is OK: puppet ran at Fri Jun 28 16:45:25 UTC 2013 [16:45:31] RECOVERY - Puppet freshness on search34 is OK: puppet ran at Fri Jun 28 16:45:26 UTC 2013 [16:45:32] RECOVERY - Puppet freshness on mw1064 is OK: puppet ran at Fri Jun 28 16:45:27 UTC 2013 [16:45:32] RECOVERY - Puppet freshness on mw54 is OK: puppet ran at Fri Jun 28 16:45:28 UTC 2013 [16:45:33] RECOVERY - Puppet freshness on mw1071 is OK: puppet ran at Fri Jun 28 16:45:29 UTC 2013 [16:45:42] RECOVERY - Puppet freshness on mw1207 is OK: puppet ran at Fri Jun 28 16:45:30 UTC 2013 [16:45:43] RECOVERY - Puppet freshness on mw94 is OK: puppet ran at Fri Jun 28 16:45:31 UTC 2013 [16:45:43] RECOVERY - Puppet freshness on mw1187 is OK: puppet ran at Fri Jun 28 16:45:32 UTC 2013 [16:45:43] RECOVERY - Puppet freshness on amssq39 is OK: puppet ran at Fri Jun 28 16:45:33 UTC 2013 [16:45:43] RECOVERY - Puppet freshness on srv295 is OK: puppet ran at Fri Jun 28 16:45:34 UTC 2013 [16:45:43] RECOVERY - Puppet freshness on cp3005 is OK: puppet ran at Fri Jun 28 16:45:36 UTC 2013 [16:45:43] RECOVERY - Puppet freshness on cp1045 is OK: puppet ran at Fri Jun 28 16:45:38 UTC 2013 [16:45:50] RECOVERY - Puppet freshness on analytics1013 is OK: puppet ran at Fri Jun 28 16:45:40 UTC 2013 [16:45:50] RECOVERY - Puppet freshness on maerlant is OK: puppet ran at Fri Jun 28 16:45:41 UTC 2013 [16:45:50] RECOVERY - Puppet freshness on virt9 is OK: puppet ran at Fri Jun 28 16:45:42 UTC 2013 [16:45:50] RECOVERY - Puppet freshness on cp1032 is OK: puppet ran at Fri Jun 28 16:45:44 UTC 2013 [16:45:50] RECOVERY - Puppet freshness on strontium is OK: puppet ran at Fri Jun 28 16:45:46 UTC 2013 [16:45:50] RECOVERY - Puppet freshness on pc2 is OK: puppet ran at Fri Jun 28 16:45:48 UTC 2013 [16:45:50] RECOVERY - Puppet freshness on analytics1005 is OK: puppet ran at Fri Jun 28 16:45:49 UTC 2013 [16:46:01] RECOVERY - Puppet freshness on mc1015 is OK: puppet ran at Fri Jun 28 16:45:50 UTC 2013 [16:46:01] RECOVERY - Puppet freshness on mw32 is OK: puppet ran at Fri Jun 28 16:45:51 UTC 2013 [16:46:01] RECOVERY - Puppet freshness on mc1003 is OK: puppet ran at Fri Jun 28 16:45:52 UTC 2013 [16:46:01] RECOVERY - Puppet freshness on db52 is OK: puppet ran at Fri Jun 28 16:45:53 UTC 2013 [16:46:01] RECOVERY - Puppet freshness on sq67 is OK: puppet ran at Fri Jun 28 16:45:54 UTC 2013 [16:46:01] RECOVERY - Puppet freshness on snapshot1003 is OK: puppet ran at Fri Jun 28 16:45:56 UTC 2013 [16:46:01] RECOVERY - Puppet freshness on cp3004 is OK: puppet ran at Fri Jun 28 16:45:57 UTC 2013 [16:46:02] RECOVERY - Puppet freshness on search28 is OK: puppet ran at Fri Jun 28 16:45:58 UTC 2013 [16:46:04] pages just arrived now [16:46:05] varnish-related [16:46:10] RECOVERY - Puppet freshness on mw105 is OK: puppet ran at Fri Jun 28 16:45:59 UTC 2013 [16:46:10] RECOVERY - Puppet freshness on mw101 is OK: puppet ran at Fri Jun 28 16:46:02 UTC 2013 [16:46:10] RECOVERY - Puppet freshness on mw29 is OK: puppet ran at Fri Jun 28 16:46:02 UTC 2013 [16:46:10] RECOVERY - Puppet freshness on mw103 is OK: puppet ran at Fri Jun 28 16:46:04 UTC 2013 [16:46:10] RECOVERY - Puppet freshness on mw108 is OK: puppet ran at Fri Jun 28 16:46:04 UTC 2013 [16:46:10] RECOVERY - Puppet freshness on mw73 is OK: puppet ran at Fri Jun 28 16:46:05 UTC 2013 [16:46:10] RECOVERY - Puppet freshness on mw1109 is OK: puppet ran at Fri Jun 28 16:46:06 UTC 2013 [16:46:11] RECOVERY - Puppet freshness on cp3003 is OK: puppet ran at Fri Jun 28 16:46:07 UTC 2013 [16:46:11] RECOVERY - Puppet freshness on srv300 is OK: puppet ran at Fri Jun 28 16:46:08 UTC 2013 [16:46:20] RECOVERY - Puppet freshness on pdf1 is OK: puppet ran at Fri Jun 28 16:46:09 UTC 2013 [16:46:20] RECOVERY - Puppet freshness on search1010 is OK: puppet ran at Fri Jun 28 16:46:10 UTC 2013 [16:46:20] RECOVERY - Puppet freshness on mw16 is OK: puppet ran at Fri Jun 28 16:46:11 UTC 2013 [16:46:20] RECOVERY - Puppet freshness on search13 is OK: puppet ran at Fri Jun 28 16:46:12 UTC 2013 [16:46:20] RECOVERY - Puppet freshness on srv287 is OK: puppet ran at Fri Jun 28 16:46:13 UTC 2013 [16:46:20] RECOVERY - Puppet freshness on mw1037 is OK: puppet ran at Fri Jun 28 16:46:14 UTC 2013 [16:46:20] RECOVERY - Puppet freshness on analytics1012 is OK: puppet ran at Fri Jun 28 16:46:15 UTC 2013 [16:46:21] RECOVERY - Puppet freshness on mw1176 is OK: puppet ran at Fri Jun 28 16:46:17 UTC 2013 [16:46:30] RECOVERY - Puppet freshness on mw1099 is OK: puppet ran at Fri Jun 28 16:46:19 UTC 2013 [16:46:30] RECOVERY - Puppet freshness on mw51 is OK: puppet ran at Fri Jun 28 16:46:20 UTC 2013 [16:46:30] RECOVERY - Puppet freshness on tmh2 is OK: puppet ran at Fri Jun 28 16:46:23 UTC 2013 [16:46:30] RECOVERY - Puppet freshness on amssq52 is OK: puppet ran at Fri Jun 28 16:46:24 UTC 2013 [16:46:30] RECOVERY - Puppet freshness on db1054 is OK: puppet ran at Fri Jun 28 16:46:27 UTC 2013 [16:46:30] RECOVERY - Puppet freshness on sq83 is OK: puppet ran at Fri Jun 28 16:46:27 UTC 2013 [16:46:30] RECOVERY - Puppet freshness on labsdb1003 is OK: puppet ran at Fri Jun 28 16:46:28 UTC 2013 [16:46:31] RECOVERY - Puppet freshness on sq86 is OK: puppet ran at Fri Jun 28 16:46:28 UTC 2013 [16:46:31] RECOVERY - Puppet freshness on sq71 is OK: puppet ran at Fri Jun 28 16:46:29 UTC 2013 [16:46:40] RECOVERY - Apache HTTP on mw1154 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 747 bytes in 2.292 second response time [16:46:40] RECOVERY - Puppet freshness on labsdb1002 is OK: puppet ran at Fri Jun 28 16:46:32 UTC 2013 [16:46:40] RECOVERY - Puppet freshness on cp1061 is OK: puppet ran at Fri Jun 28 16:46:32 UTC 2013 [16:46:40] RECOVERY - Apache HTTP on mw1157 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 747 bytes in 3.858 second response time [16:46:40] RECOVERY - Puppet freshness on es1 is OK: puppet ran at Fri Jun 28 16:46:33 UTC 2013 [16:46:40] RECOVERY - Puppet freshness on mw1217 is OK: puppet ran at Fri Jun 28 16:46:33 UTC 2013 [16:46:40] RECOVERY - Puppet freshness on cp1007 is OK: puppet ran at Fri Jun 28 16:46:34 UTC 2013 [16:46:41] RECOVERY - Puppet freshness on db40 is OK: puppet ran at Fri Jun 28 16:46:34 UTC 2013 [16:46:41] RECOVERY - Apache HTTP on mw1156 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 747 bytes in 5.129 second response time [16:46:42] RECOVERY - Puppet freshness on cp1056 is OK: puppet ran at Fri Jun 28 16:46:35 UTC 2013 [16:46:50] RECOVERY - Puppet freshness on mw1113 is OK: puppet ran at Fri Jun 28 16:46:38 UTC 2013 [16:46:50] RECOVERY - Puppet freshness on mw1160 is OK: puppet ran at Fri Jun 28 16:46:38 UTC 2013 [16:46:50] RECOVERY - Puppet freshness on mw1018 is OK: puppet ran at Fri Jun 28 16:46:39 UTC 2013 [16:46:50] RECOVERY - Puppet freshness on cp3007 is OK: puppet ran at Fri Jun 28 16:46:40 UTC 2013 [16:46:50] RECOVERY - Puppet freshness on mw1156 is OK: puppet ran at Fri Jun 28 16:46:42 UTC 2013 [16:46:50] RECOVERY - Apache HTTP on mw1158 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 747 bytes in 0.040 second response time [16:46:50] RECOVERY - Apache HTTP on mw1153 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 747 bytes in 0.036 second response time [16:46:51] RECOVERY - Apache HTTP on mw1160 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 747 bytes in 0.055 second response time [16:46:51] RECOVERY - Apache HTTP on mw1155 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 747 bytes in 0.045 second response time [16:46:52] RECOVERY - LVS HTTP IPv4 on rendering.svc.eqiad.wmnet is OK: HTTP OK: HTTP/1.1 200 OK - 61964 bytes in 0.210 second response time [16:47:00] RECOVERY - Puppet freshness on mw1117 is OK: puppet ran at Fri Jun 28 16:46:49 UTC 2013 [16:47:00] RECOVERY - Puppet freshness on cerium is OK: puppet ran at Fri Jun 28 16:46:50 UTC 2013 [16:47:00] RECOVERY - Puppet freshness on stafford is OK: puppet ran at Fri Jun 28 16:46:50 UTC 2013 [16:47:00] RECOVERY - Puppet freshness on mw1100 is OK: puppet ran at Fri Jun 28 16:46:52 UTC 2013 [16:47:00] RECOVERY - Puppet freshness on mw1103 is OK: puppet ran at Fri Jun 28 16:46:53 UTC 2013 [16:47:00] RECOVERY - Puppet freshness on mw1088 is OK: puppet ran at Fri Jun 28 16:46:54 UTC 2013 [16:47:00] RECOVERY - Puppet freshness on ms-be8 is OK: puppet ran at Fri Jun 28 16:46:55 UTC 2013 [16:47:01] RECOVERY - Apache HTTP on mw1159 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 747 bytes in 0.046 second response time [16:47:01] RECOVERY - Puppet freshness on mw122 is OK: puppet ran at Fri Jun 28 16:46:56 UTC 2013 [16:47:02] RECOVERY - Puppet freshness on mw20 is OK: puppet ran at Fri Jun 28 16:46:57 UTC 2013 [16:47:02] RECOVERY - Puppet freshness on search1008 is OK: puppet ran at Fri Jun 28 16:46:58 UTC 2013 [16:47:03] RECOVERY - Puppet freshness on mw46 is OK: puppet ran at Fri Jun 28 16:46:58 UTC 2013 [16:47:10] RECOVERY - Puppet freshness on search17 is OK: puppet ran at Fri Jun 28 16:47:00 UTC 2013 [16:47:10] RECOVERY - Puppet freshness on search1019 is OK: puppet ran at Fri Jun 28 16:47:03 UTC 2013 [16:47:10] RECOVERY - Puppet freshness on mw1123 is OK: puppet ran at Fri Jun 28 16:47:04 UTC 2013 [16:47:10] RECOVERY - Puppet freshness on mw34 is OK: puppet ran at Fri Jun 28 16:47:04 UTC 2013 [16:47:10] RECOVERY - Puppet freshness on srv238 is OK: puppet ran at Fri Jun 28 16:47:08 UTC 2013 [16:47:10] RECOVERY - Puppet freshness on mw66 is OK: puppet ran at Fri Jun 28 16:47:09 UTC 2013 [16:47:20] RECOVERY - Puppet freshness on mw1095 is OK: puppet ran at Fri Jun 28 16:47:10 UTC 2013 [16:47:20] RECOVERY - Puppet freshness on search1018 is OK: puppet ran at Fri Jun 28 16:47:11 UTC 2013 [16:47:20] RECOVERY - Puppet freshness on mw1015 is OK: puppet ran at Fri Jun 28 16:47:12 UTC 2013 [16:47:20] RECOVERY - Puppet freshness on srv297 is OK: puppet ran at Fri Jun 28 16:47:13 UTC 2013 [16:47:20] RECOVERY - Puppet freshness on mw1017 is OK: puppet ran at Fri Jun 28 16:47:14 UTC 2013 [16:47:20] RECOVERY - Puppet freshness on mw1101 is OK: puppet ran at Fri Jun 28 16:47:15 UTC 2013 [16:47:20] RECOVERY - Puppet freshness on mw1144 is OK: puppet ran at Fri Jun 28 16:47:16 UTC 2013 [16:47:21] RECOVERY - Puppet freshness on mw1042 is OK: puppet ran at Fri Jun 28 16:47:17 UTC 2013 [16:47:21] RECOVERY - Puppet freshness on sq65 is OK: puppet ran at Fri Jun 28 16:47:18 UTC 2013 [16:47:30] RECOVERY - Puppet freshness on mw1065 is OK: puppet ran at Fri Jun 28 16:47:19 UTC 2013 [16:47:30] RECOVERY - Puppet freshness on es1003 is OK: puppet ran at Fri Jun 28 16:47:20 UTC 2013 [16:47:30] RECOVERY - Puppet freshness on cp1012 is OK: puppet ran at Fri Jun 28 16:47:20 UTC 2013 [16:47:30] RECOVERY - Puppet freshness on db35 is OK: puppet ran at Fri Jun 28 16:47:22 UTC 2013 [16:47:30] RECOVERY - Puppet freshness on es1005 is OK: puppet ran at Fri Jun 28 16:47:23 UTC 2013 [16:47:30] RECOVERY - Puppet freshness on williams is OK: puppet ran at Fri Jun 28 16:47:24 UTC 2013 [16:47:30] RECOVERY - Puppet freshness on amssq55 is OK: puppet ran at Fri Jun 28 16:47:25 UTC 2013 [16:47:31] RECOVERY - Puppet freshness on srv258 is OK: puppet ran at Fri Jun 28 16:47:26 UTC 2013 [16:47:31] RECOVERY - Puppet freshness on srv237 is OK: puppet ran at Fri Jun 28 16:47:28 UTC 2013 [16:47:40] RECOVERY - Puppet freshness on db1019 is OK: puppet ran at Fri Jun 28 16:47:29 UTC 2013 [16:47:40] RECOVERY - Puppet freshness on search22 is OK: puppet ran at Fri Jun 28 16:47:30 UTC 2013 [16:47:40] RECOVERY - Puppet freshness on dobson is OK: puppet ran at Fri Jun 28 16:47:33 UTC 2013 [16:47:40] RECOVERY - Puppet freshness on mw1214 is OK: puppet ran at Fri Jun 28 16:47:34 UTC 2013 [16:47:40] RECOVERY - Puppet freshness on mc3 is OK: puppet ran at Fri Jun 28 16:47:35 UTC 2013 [16:47:40] RECOVERY - Puppet freshness on mw9 is OK: puppet ran at Fri Jun 28 16:47:36 UTC 2013 [16:47:40] RECOVERY - Puppet freshness on sq45 is OK: puppet ran at Fri Jun 28 16:47:37 UTC 2013 [16:47:41] RECOVERY - Puppet freshness on db1023 is OK: puppet ran at Fri Jun 28 16:47:38 UTC 2013 [16:47:41] RECOVERY - Puppet freshness on cp1013 is OK: puppet ran at Fri Jun 28 16:47:37 UTC 2013 [16:47:42] RECOVERY - Puppet freshness on cp1051 is OK: puppet ran at Fri Jun 28 16:47:39 UTC 2013 [16:47:50] RECOVERY - Puppet freshness on db53 is OK: puppet ran at Fri Jun 28 16:47:39 UTC 2013 [16:47:50] RECOVERY - Puppet freshness on sq73 is OK: puppet ran at Fri Jun 28 16:47:42 UTC 2013 [16:47:50] RECOVERY - Puppet freshness on sq72 is OK: puppet ran at Fri Jun 28 16:47:43 UTC 2013 [16:47:50] RECOVERY - Puppet freshness on db1052 is OK: puppet ran at Fri Jun 28 16:47:43 UTC 2013 [16:47:50] RECOVERY - Puppet freshness on pc1002 is OK: puppet ran at Fri Jun 28 16:47:44 UTC 2013 [16:47:50] RECOVERY - Puppet freshness on mw1175 is OK: puppet ran at Fri Jun 28 16:47:44 UTC 2013 [16:47:50] RECOVERY - Puppet freshness on gurvin is OK: puppet ran at Fri Jun 28 16:47:45 UTC 2013 [16:47:51] RECOVERY - Puppet freshness on es1002 is OK: puppet ran at Fri Jun 28 16:47:45 UTC 2013 [16:47:51] RECOVERY - Puppet freshness on es10 is OK: puppet ran at Fri Jun 28 16:47:46 UTC 2013 [16:47:52] RECOVERY - Puppet freshness on es4 is OK: puppet ran at Fri Jun 28 16:47:47 UTC 2013 [16:48:00] RECOVERY - Puppet freshness on cp1003 is OK: puppet ran at Fri Jun 28 16:47:49 UTC 2013 [16:48:00] RECOVERY - Puppet freshness on srv284 is OK: puppet ran at Fri Jun 28 16:47:50 UTC 2013 [16:48:00] RECOVERY - Puppet freshness on amssq57 is OK: puppet ran at Fri Jun 28 16:47:50 UTC 2013 [16:48:00] RECOVERY - Puppet freshness on mw1127 is OK: puppet ran at Fri Jun 28 16:47:51 UTC 2013 [16:48:00] RECOVERY - Puppet freshness on cp1028 is OK: puppet ran at Fri Jun 28 16:47:54 UTC 2013 [16:48:00] RECOVERY - Puppet freshness on mw14 is OK: puppet ran at Fri Jun 28 16:47:55 UTC 2013 [16:48:00] RECOVERY - Puppet freshness on pc1003 is OK: puppet ran at Fri Jun 28 16:47:56 UTC 2013 [16:48:01] RECOVERY - Puppet freshness on mw1182 is OK: puppet ran at Fri Jun 28 16:47:56 UTC 2013 [16:48:01] RECOVERY - Puppet freshness on amssq34 is OK: puppet ran at Fri Jun 28 16:47:57 UTC 2013 [16:48:02] RECOVERY - Puppet freshness on tin is OK: puppet ran at Fri Jun 28 16:47:58 UTC 2013 [16:48:02] RECOVERY - Puppet freshness on ms-be1 is OK: puppet ran at Fri Jun 28 16:47:58 UTC 2013 [16:48:10] RECOVERY - Puppet freshness on cp3010 is OK: puppet ran at Fri Jun 28 16:48:08 UTC 2013 [16:48:20] RECOVERY - Puppet freshness on mw62 is OK: puppet ran at Fri Jun 28 16:48:10 UTC 2013 [16:48:20] RECOVERY - Puppet freshness on mw1126 is OK: puppet ran at Fri Jun 28 16:48:11 UTC 2013 [16:48:20] RECOVERY - Puppet freshness on analytics1010 is OK: puppet ran at Fri Jun 28 16:48:12 UTC 2013 [16:48:20] RECOVERY - Puppet freshness on srv286 is OK: puppet ran at Fri Jun 28 16:48:14 UTC 2013 [16:48:20] RECOVERY - Puppet freshness on srv250 is OK: puppet ran at Fri Jun 28 16:48:15 UTC 2013 [16:48:20] RECOVERY - Puppet freshness on mw125 is OK: puppet ran at Fri Jun 28 16:48:16 UTC 2013 [16:48:20] RECOVERY - Puppet freshness on mw47 is OK: puppet ran at Fri Jun 28 16:48:18 UTC 2013 [16:48:21] RECOVERY - Puppet freshness on mw1114 is OK: puppet ran at Fri Jun 28 16:48:19 UTC 2013 [16:48:30] RECOVERY - Puppet freshness on mw90 is OK: puppet ran at Fri Jun 28 16:48:20 UTC 2013 [16:48:30] RECOVERY - Varnish HTTP text-backend on cp1038 is OK: HTTP OK: HTTP/1.1 200 OK - 3153 bytes in 0.010 second response time [16:48:30] RECOVERY - Puppet freshness on mw1162 is OK: puppet ran at Fri Jun 28 16:48:21 UTC 2013 [16:48:30] RECOVERY - Puppet freshness on mw81 is OK: puppet ran at Fri Jun 28 16:48:22 UTC 2013 [16:48:30] RECOVERY - Puppet freshness on mw100 is OK: puppet ran at Fri Jun 28 16:48:23 UTC 2013 [16:48:30] RECOVERY - Puppet freshness on search21 is OK: puppet ran at Fri Jun 28 16:48:24 UTC 2013 [16:48:30] RECOVERY - Puppet freshness on search14 is OK: puppet ran at Fri Jun 28 16:48:25 UTC 2013 [16:48:31] RECOVERY - Puppet freshness on search1014 is OK: puppet ran at Fri Jun 28 16:48:26 UTC 2013 [16:48:31] RECOVERY - Puppet freshness on mw5 is OK: puppet ran at Fri Jun 28 16:48:27 UTC 2013 [16:48:32] RECOVERY - Puppet freshness on mw41 is OK: puppet ran at Fri Jun 28 16:48:29 UTC 2013 [16:48:40] RECOVERY - Puppet freshness on mw1196 is OK: puppet ran at Fri Jun 28 16:48:30 UTC 2013 [16:48:40] RECOVERY - Puppet freshness on ms5 is OK: puppet ran at Fri Jun 28 16:48:31 UTC 2013 [16:48:40] RECOVERY - Puppet freshness on lvs1 is OK: puppet ran at Fri Jun 28 16:48:32 UTC 2013 [16:48:40] RECOVERY - Puppet freshness on cp1069 is OK: puppet ran at Fri Jun 28 16:48:34 UTC 2013 [16:48:40] RECOVERY - Puppet freshness on mw1044 is OK: puppet ran at Fri Jun 28 16:48:35 UTC 2013 [16:48:40] RECOVERY - Puppet freshness on zirconium is OK: puppet ran at Fri Jun 28 16:48:36 UTC 2013 [16:48:40] RECOVERY - Puppet freshness on sq79 is OK: puppet ran at Fri Jun 28 16:48:37 UTC 2013 [16:48:50] RECOVERY - Puppet freshness on fenari is OK: puppet ran at Fri Jun 28 16:48:39 UTC 2013 [16:48:51] RECOVERY - Puppet freshness on cp1006 is OK: puppet ran at Fri Jun 28 16:48:40 UTC 2013 [16:48:51] RECOVERY - Puppet freshness on db1048 is OK: puppet ran at Fri Jun 28 16:48:41 UTC 2013 [16:48:51] RECOVERY - Puppet freshness on cp1043 is OK: puppet ran at Fri Jun 28 16:48:42 UTC 2013 [16:48:51] RECOVERY - Puppet freshness on cp1050 is OK: puppet ran at Fri Jun 28 16:48:44 UTC 2013 [16:48:51] RECOVERY - Puppet freshness on mc1005 is OK: puppet ran at Fri Jun 28 16:48:45 UTC 2013 [16:48:51] RECOVERY - Puppet freshness on ms-be4 is OK: puppet ran at Fri Jun 28 16:48:46 UTC 2013 [16:48:51] RECOVERY - Puppet freshness on wtp1024 is OK: puppet ran at Fri Jun 28 16:48:47 UTC 2013 [16:48:52] RECOVERY - Puppet freshness on search36 is OK: puppet ran at Fri Jun 28 16:48:48 UTC 2013 [16:49:00] RECOVERY - Puppet freshness on mw118 is OK: puppet ran at Fri Jun 28 16:48:49 UTC 2013 [16:49:00] RECOVERY - Puppet freshness on lvs5 is OK: puppet ran at Fri Jun 28 16:48:51 UTC 2013 [16:49:00] RECOVERY - Puppet freshness on srv301 is OK: puppet ran at Fri Jun 28 16:48:52 UTC 2013 [16:49:00] RECOVERY - Puppet freshness on mw1124 is OK: puppet ran at Fri Jun 28 16:48:56 UTC 2013 [16:49:01] RECOVERY - Puppet freshness on mw1161 is OK: puppet ran at Fri Jun 28 16:48:56 UTC 2013 [16:49:12] RECOVERY - Puppet freshness on mw1125 is OK: puppet ran at Fri Jun 28 16:49:00 UTC 2013 [16:49:12] RECOVERY - Puppet freshness on mw1164 is OK: puppet ran at Fri Jun 28 16:49:01 UTC 2013 [16:49:12] RECOVERY - Puppet freshness on analytics1015 is OK: puppet ran at Fri Jun 28 16:49:01 UTC 2013 [16:49:12] RECOVERY - Puppet freshness on mw1130 is OK: puppet ran at Fri Jun 28 16:49:03 UTC 2013 [16:49:12] RECOVERY - Puppet freshness on cp1046 is OK: puppet ran at Fri Jun 28 16:49:04 UTC 2013 [16:49:12] RECOVERY - Puppet freshness on gadolinium is OK: puppet ran at Fri Jun 28 16:49:05 UTC 2013 [16:49:12] RECOVERY - Puppet freshness on mw119 is OK: puppet ran at Fri Jun 28 16:49:06 UTC 2013 [16:49:12] RECOVERY - Puppet freshness on mw83 is OK: puppet ran at Fri Jun 28 16:49:06 UTC 2013 [16:49:12] RECOVERY - Puppet freshness on sq51 is OK: puppet ran at Fri Jun 28 16:49:07 UTC 2013 [16:49:12] RECOVERY - Puppet freshness on db1036 is OK: puppet ran at Fri Jun 28 16:49:08 UTC 2013 [16:49:12] RECOVERY - Puppet freshness on neon is OK: puppet ran at Fri Jun 28 16:49:08 UTC 2013 [16:49:20] RECOVERY - Puppet freshness on amssq44 is OK: puppet ran at Fri Jun 28 16:49:09 UTC 2013 [16:49:20] RECOVERY - Puppet freshness on mw1190 is OK: puppet ran at Fri Jun 28 16:49:10 UTC 2013 [16:49:20] RECOVERY - Puppet freshness on mc1009 is OK: puppet ran at Fri Jun 28 16:49:11 UTC 2013 [16:49:20] RECOVERY - Puppet freshness on ms-fe3 is OK: puppet ran at Fri Jun 28 16:49:12 UTC 2013 [16:49:20] RECOVERY - Puppet freshness on srv290 is OK: puppet ran at Fri Jun 28 16:49:13 UTC 2013 [16:49:20] RECOVERY - Puppet freshness on cp1063 is OK: puppet ran at Fri Jun 28 16:49:15 UTC 2013 [16:49:20] RECOVERY - Puppet freshness on cp1020 is OK: puppet ran at Fri Jun 28 16:49:17 UTC 2013 [16:49:21] RECOVERY - Puppet freshness on ms-be1012 is OK: puppet ran at Fri Jun 28 16:49:18 UTC 2013 [16:49:30] RECOVERY - Puppet freshness on solr2 is OK: puppet ran at Fri Jun 28 16:49:18 UTC 2013 [16:49:30] RECOVERY - Puppet freshness on vanadium is OK: puppet ran at Fri Jun 28 16:49:21 UTC 2013 [16:49:30] RECOVERY - Puppet freshness on mw1147 is OK: puppet ran at Fri Jun 28 16:49:22 UTC 2013 [16:49:30] RECOVERY - Puppet freshness on wtp1018 is OK: puppet ran at Fri Jun 28 16:49:23 UTC 2013 [16:49:30] RECOVERY - Puppet freshness on db45 is OK: puppet ran at Fri Jun 28 16:49:25 UTC 2013 [16:49:31] RECOVERY - Puppet freshness on db43 is OK: puppet ran at Fri Jun 28 16:49:25 UTC 2013 [16:49:31] RECOVERY - Puppet freshness on wtp1011 is OK: puppet ran at Fri Jun 28 16:49:27 UTC 2013 [16:49:31] RECOVERY - Puppet freshness on wtp1003 is OK: puppet ran at Fri Jun 28 16:49:28 UTC 2013 [16:49:41] RECOVERY - Puppet freshness on ms-be2 is OK: puppet ran at Fri Jun 28 16:49:29 UTC 2013 [16:49:41] RECOVERY - Puppet freshness on mw1052 is OK: puppet ran at Fri Jun 28 16:49:30 UTC 2013 [16:49:41] RECOVERY - Puppet freshness on mw82 is OK: puppet ran at Fri Jun 28 16:49:31 UTC 2013 [16:49:41] RECOVERY - Puppet freshness on db1004 is OK: puppet ran at Fri Jun 28 16:49:32 UTC 2013 [16:49:41] RECOVERY - Puppet freshness on analytics1023 is OK: puppet ran at Fri Jun 28 16:49:33 UTC 2013 [16:49:41] RECOVERY - Puppet freshness on srv289 is OK: puppet ran at Fri Jun 28 16:49:34 UTC 2013 [16:49:41] RECOVERY - Puppet freshness on srv244 is OK: puppet ran at Fri Jun 28 16:49:35 UTC 2013 [16:49:42] RECOVERY - Puppet freshness on srv246 is OK: puppet ran at Fri Jun 28 16:49:36 UTC 2013 [16:49:42] RECOVERY - Puppet freshness on search19 is OK: puppet ran at Fri Jun 28 16:49:37 UTC 2013 [16:49:43] RECOVERY - Puppet freshness on srv253 is OK: puppet ran at Fri Jun 28 16:49:38 UTC 2013 [16:49:51] RECOVERY - Puppet freshness on srv245 is OK: puppet ran at Fri Jun 28 16:49:42 UTC 2013 [16:49:51] RECOVERY - Puppet freshness on srv249 is OK: puppet ran at Fri Jun 28 16:49:43 UTC 2013 [16:49:52] RECOVERY - Puppet freshness on mw76 is OK: puppet ran at Fri Jun 28 16:49:47 UTC 2013 [16:49:52] RECOVERY - Puppet freshness on sq48 is OK: puppet ran at Fri Jun 28 16:49:48 UTC 2013 [16:50:01] RECOVERY - Puppet freshness on db1020 is OK: puppet ran at Fri Jun 28 16:49:51 UTC 2013 [16:50:01] RECOVERY - Puppet freshness on mw30 is OK: puppet ran at Fri Jun 28 16:49:52 UTC 2013 [16:50:01] RECOVERY - Puppet freshness on cp1049 is OK: puppet ran at Fri Jun 28 16:49:53 UTC 2013 [16:50:01] RECOVERY - Puppet freshness on db1029 is OK: puppet ran at Fri Jun 28 16:49:54 UTC 2013 [16:50:01] RECOVERY - Puppet freshness on es6 is OK: puppet ran at Fri Jun 28 16:49:54 UTC 2013 [16:50:01] RECOVERY - Puppet freshness on mw1056 is OK: puppet ran at Fri Jun 28 16:49:55 UTC 2013 [16:50:01] RECOVERY - Puppet freshness on mw1004 is OK: puppet ran at Fri Jun 28 16:49:57 UTC 2013 [16:50:02] RECOVERY - Puppet freshness on mc1001 is OK: puppet ran at Fri Jun 28 16:49:57 UTC 2013 [16:50:02] RECOVERY - Puppet freshness on mw1146 is OK: puppet ran at Fri Jun 28 16:49:58 UTC 2013 [16:50:03] RECOVERY - Puppet freshness on mw1038 is OK: puppet ran at Fri Jun 28 16:49:59 UTC 2013 [16:50:11] RECOVERY - Puppet freshness on mw1048 is OK: puppet ran at Fri Jun 28 16:50:00 UTC 2013 [16:50:11] RECOVERY - Puppet freshness on mw1081 is OK: puppet ran at Fri Jun 28 16:50:01 UTC 2013 [16:50:11] RECOVERY - Puppet freshness on mw1115 is OK: puppet ran at Fri Jun 28 16:50:01 UTC 2013 [16:50:11] RECOVERY - Puppet freshness on mw1089 is OK: puppet ran at Fri Jun 28 16:50:02 UTC 2013 [16:50:11] RECOVERY - Puppet freshness on sq85 is OK: puppet ran at Fri Jun 28 16:50:02 UTC 2013 [16:50:11] RECOVERY - Puppet freshness on mw49 is OK: puppet ran at Fri Jun 28 16:50:04 UTC 2013 [16:50:11] RECOVERY - Puppet freshness on solr3 is OK: puppet ran at Fri Jun 28 16:50:08 UTC 2013 [16:50:20] RECOVERY - Puppet freshness on srv276 is OK: puppet ran at Fri Jun 28 16:50:13 UTC 2013 [16:50:20] RECOVERY - Puppet freshness on db58 is OK: puppet ran at Fri Jun 28 16:50:13 UTC 2013 [16:50:20] RECOVERY - Puppet freshness on mw8 is OK: puppet ran at Fri Jun 28 16:50:14 UTC 2013 [16:50:20] RECOVERY - Puppet freshness on sq42 is OK: puppet ran at Fri Jun 28 16:50:15 UTC 2013 [16:50:20] RECOVERY - Puppet freshness on mw1057 is OK: puppet ran at Fri Jun 28 16:50:16 UTC 2013 [16:50:20] RECOVERY - Puppet freshness on mw121 is OK: puppet ran at Fri Jun 28 16:50:17 UTC 2013 [16:50:20] RECOVERY - Puppet freshness on search1024 is OK: puppet ran at Fri Jun 28 16:50:18 UTC 2013 [16:50:32] RECOVERY - Puppet freshness on search18 is OK: puppet ran at Fri Jun 28 16:50:19 UTC 2013 [16:50:32] RECOVERY - Puppet freshness on mw1159 is OK: puppet ran at Fri Jun 28 16:50:20 UTC 2013 [16:50:32] RECOVERY - Puppet freshness on srv294 is OK: puppet ran at Fri Jun 28 16:50:21 UTC 2013 [16:50:32] RECOVERY - Puppet freshness on mw1186 is OK: puppet ran at Fri Jun 28 16:50:22 UTC 2013 [16:50:32] RECOVERY - Puppet freshness on mw1041 is OK: puppet ran at Fri Jun 28 16:50:23 UTC 2013 [16:50:32] RECOVERY - Puppet freshness on srv275 is OK: puppet ran at Fri Jun 28 16:50:24 UTC 2013 [16:50:32] RECOVERY - Puppet freshness on mw1188 is OK: puppet ran at Fri Jun 28 16:50:27 UTC 2013 [16:50:33] RECOVERY - Puppet freshness on manganese is OK: puppet ran at Fri Jun 28 16:50:28 UTC 2013 [16:50:40] RECOVERY - Puppet freshness on mw1087 is OK: puppet ran at Fri Jun 28 16:50:31 UTC 2013 [16:50:40] RECOVERY - Puppet freshness on mw1197 is OK: puppet ran at Fri Jun 28 16:50:32 UTC 2013 [16:50:41] RECOVERY - Puppet freshness on mw1063 is OK: puppet ran at Fri Jun 28 16:50:33 UTC 2013 [16:50:41] RECOVERY - Puppet freshness on mw58 is OK: puppet ran at Fri Jun 28 16:50:34 UTC 2013 [16:50:41] RECOVERY - Puppet freshness on ms-be9 is OK: puppet ran at Fri Jun 28 16:50:37 UTC 2013 [16:50:41] RECOVERY - Puppet freshness on mw1043 is OK: puppet ran at Fri Jun 28 16:50:39 UTC 2013 [16:50:50] RECOVERY - Puppet freshness on mw1106 is OK: puppet ran at Fri Jun 28 16:50:40 UTC 2013 [16:50:50] RECOVERY - Puppet freshness on amssq53 is OK: puppet ran at Fri Jun 28 16:50:41 UTC 2013 [16:50:50] RECOVERY - Puppet freshness on ekrem is OK: puppet ran at Fri Jun 28 16:50:42 UTC 2013 [16:50:50] RECOVERY - Puppet freshness on mw1140 is OK: puppet ran at Fri Jun 28 16:50:43 UTC 2013 [16:50:50] RECOVERY - Puppet freshness on mw1210 is OK: puppet ran at Fri Jun 28 16:50:44 UTC 2013 [16:50:50] RECOVERY - Puppet freshness on pc1 is OK: puppet ran at Fri Jun 28 16:50:47 UTC 2013 [16:50:51] RECOVERY - Puppet freshness on praseodymium is OK: puppet ran at Fri Jun 28 16:50:48 UTC 2013 [16:51:00] RECOVERY - Puppet freshness on cp1002 is OK: puppet ran at Fri Jun 28 16:50:50 UTC 2013 [16:51:00] RECOVERY - Puppet freshness on cp3012 is OK: puppet ran at Fri Jun 28 16:50:52 UTC 2013 [16:51:00] RECOVERY - Puppet freshness on srv292 is OK: puppet ran at Fri Jun 28 16:50:53 UTC 2013 [16:51:00] RECOVERY - Puppet freshness on cp1036 is OK: puppet ran at Fri Jun 28 16:50:54 UTC 2013 [16:51:00] RECOVERY - Puppet freshness on cp3009 is OK: puppet ran at Fri Jun 28 16:50:55 UTC 2013 [16:51:01] RECOVERY - Puppet freshness on db39 is OK: puppet ran at Fri Jun 28 16:50:57 UTC 2013 [16:51:01] RECOVERY - Puppet freshness on mc1007 is OK: puppet ran at Fri Jun 28 16:50:58 UTC 2013 [16:51:01] RECOVERY - Puppet freshness on mw1171 is OK: puppet ran at Fri Jun 28 16:50:59 UTC 2013 [16:51:10] RECOVERY - Puppet freshness on db1031 is OK: puppet ran at Fri Jun 28 16:51:00 UTC 2013 [16:51:10] RECOVERY - Puppet freshness on mw1111 is OK: puppet ran at Fri Jun 28 16:51:01 UTC 2013 [16:51:10] RECOVERY - Puppet freshness on titanium is OK: puppet ran at Fri Jun 28 16:51:02 UTC 2013 [16:51:10] RECOVERY - Puppet freshness on mw1007 is OK: puppet ran at Fri Jun 28 16:51:04 UTC 2013 [16:51:10] RECOVERY - Puppet freshness on cp1024 is OK: puppet ran at Fri Jun 28 16:51:04 UTC 2013 [16:51:10] RECOVERY - Puppet freshness on sq76 is OK: puppet ran at Fri Jun 28 16:51:05 UTC 2013 [16:51:10] RECOVERY - Puppet freshness on srv273 is OK: puppet ran at Fri Jun 28 16:51:06 UTC 2013 [16:51:11] RECOVERY - Puppet freshness on cp1005 is OK: puppet ran at Fri Jun 28 16:51:08 UTC 2013 [16:51:11] RECOVERY - Puppet freshness on searchidx1001 is OK: puppet ran at Fri Jun 28 16:51:09 UTC 2013 [16:51:20] RECOVERY - Puppet freshness on ms10 is OK: puppet ran at Fri Jun 28 16:51:10 UTC 2013 [16:51:20] RECOVERY - Puppet freshness on db34 is OK: puppet ran at Fri Jun 28 16:51:12 UTC 2013 [16:51:20] RECOVERY - Puppet freshness on ms-be1006 is OK: puppet ran at Fri Jun 28 16:51:13 UTC 2013 [16:51:20] RECOVERY - Puppet freshness on sq58 is OK: puppet ran at Fri Jun 28 16:51:14 UTC 2013 [16:51:20] RECOVERY - Puppet freshness on helium is OK: puppet ran at Fri Jun 28 16:51:15 UTC 2013 [16:51:20] RECOVERY - Puppet freshness on rdb1002 is OK: puppet ran at Fri Jun 28 16:51:16 UTC 2013 [16:51:20] RECOVERY - Puppet freshness on db1044 is OK: puppet ran at Fri Jun 28 16:51:18 UTC 2013 [16:51:21] RECOVERY - Puppet freshness on sq54 is OK: puppet ran at Fri Jun 28 16:51:19 UTC 2013 [16:51:30] RECOVERY - Puppet freshness on rdb1003 is OK: puppet ran at Fri Jun 28 16:51:19 UTC 2013 [16:51:30] RECOVERY - Puppet freshness on mw43 is OK: puppet ran at Fri Jun 28 16:51:20 UTC 2013 [16:51:30] RECOVERY - Puppet freshness on db1022 is OK: puppet ran at Fri Jun 28 16:51:22 UTC 2013 [16:51:30] RECOVERY - Puppet freshness on cp1010 is OK: puppet ran at Fri Jun 28 16:51:23 UTC 2013 [16:51:30] RECOVERY - Puppet freshness on mc10 is OK: puppet ran at Fri Jun 28 16:51:24 UTC 2013 [16:51:30] RECOVERY - Puppet freshness on db63 is OK: puppet ran at Fri Jun 28 16:51:26 UTC 2013 [16:51:30] RECOVERY - Puppet freshness on mw38 is OK: puppet ran at Fri Jun 28 16:51:27 UTC 2013 [16:51:33] New patchset: Ottomata; "Creating role::analytics::packages class, installing python-lxml package" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/71078 [16:51:40] RECOVERY - Puppet freshness on db1001 is OK: puppet ran at Fri Jun 28 16:51:30 UTC 2013 [16:51:40] RECOVERY - Puppet freshness on wtp1015 is OK: puppet ran at Fri Jun 28 16:51:34 UTC 2013 [16:51:40] RECOVERY - Puppet freshness on ms-fe1001 is OK: puppet ran at Fri Jun 28 16:51:35 UTC 2013 [16:51:40] RECOVERY - Puppet freshness on wtp1007 is OK: puppet ran at Fri Jun 28 16:51:36 UTC 2013 [16:51:50] RECOVERY - Puppet freshness on mw124 is OK: puppet ran at Fri Jun 28 16:51:40 UTC 2013 [16:51:50] RECOVERY - Puppet freshness on mw57 is OK: puppet ran at Fri Jun 28 16:51:41 UTC 2013 [16:51:50] RECOVERY - Puppet freshness on srv255 is OK: puppet ran at Fri Jun 28 16:51:41 UTC 2013 [16:51:50] RECOVERY - Puppet freshness on analytics1014 is OK: puppet ran at Fri Jun 28 16:51:43 UTC 2013 [16:51:50] RECOVERY - Puppet freshness on mw1032 is OK: puppet ran at Fri Jun 28 16:51:44 UTC 2013 [16:51:50] RECOVERY - Puppet freshness on stat1 is OK: puppet ran at Fri Jun 28 16:51:45 UTC 2013 [16:51:50] RECOVERY - Puppet freshness on mw98 is OK: puppet ran at Fri Jun 28 16:51:46 UTC 2013 [16:51:51] RECOVERY - Puppet freshness on search33 is OK: puppet ran at Fri Jun 28 16:51:47 UTC 2013 [16:51:51] RECOVERY - Puppet freshness on mw106 is OK: puppet ran at Fri Jun 28 16:51:48 UTC 2013 [16:52:00] RECOVERY - Puppet freshness on mw1189 is OK: puppet ran at Fri Jun 28 16:51:49 UTC 2013 [16:52:00] RECOVERY - Puppet freshness on mc1 is OK: puppet ran at Fri Jun 28 16:51:51 UTC 2013 [16:52:00] RECOVERY - Puppet freshness on mw79 is OK: puppet ran at Fri Jun 28 16:51:53 UTC 2013 [16:52:00] RECOVERY - Puppet freshness on mw35 is OK: puppet ran at Fri Jun 28 16:51:54 UTC 2013 [16:52:00] RECOVERY - Puppet freshness on mw1201 is OK: puppet ran at Fri Jun 28 16:51:55 UTC 2013 [16:52:00] RECOVERY - Puppet freshness on mw1205 is OK: puppet ran at Fri Jun 28 16:51:56 UTC 2013 [16:52:10] RECOVERY - Puppet freshness on mw1003 is OK: puppet ran at Fri Jun 28 16:51:59 UTC 2013 [16:52:10] RECOVERY - Puppet freshness on mw1068 is OK: puppet ran at Fri Jun 28 16:52:00 UTC 2013 [16:52:10] RECOVERY - Puppet freshness on mw1122 is OK: puppet ran at Fri Jun 28 16:52:01 UTC 2013 [16:52:10] RECOVERY - Puppet freshness on cp1030 is OK: puppet ran at Fri Jun 28 16:52:04 UTC 2013 [16:52:10] RECOVERY - Puppet freshness on sq63 is OK: puppet ran at Fri Jun 28 16:52:05 UTC 2013 [16:52:10] RECOVERY - Puppet freshness on srv193 is OK: puppet ran at Fri Jun 28 16:52:06 UTC 2013 [16:52:10] RECOVERY - Puppet freshness on mw1024 is OK: puppet ran at Fri Jun 28 16:52:07 UTC 2013 [16:52:11] RECOVERY - Puppet freshness on calcium is OK: puppet ran at Fri Jun 28 16:52:07 UTC 2013 [16:52:11] RECOVERY - Puppet freshness on mw1069 is OK: puppet ran at Fri Jun 28 16:52:09 UTC 2013 [16:52:20] RECOVERY - Puppet freshness on mc6 is OK: puppet ran at Fri Jun 28 16:52:10 UTC 2013 [16:52:20] RECOVERY - Puppet freshness on db1002 is OK: puppet ran at Fri Jun 28 16:52:11 UTC 2013 [16:52:20] RECOVERY - Puppet freshness on db48 is OK: puppet ran at Fri Jun 28 16:52:12 UTC 2013 [16:52:20] RECOVERY - Puppet freshness on db1040 is OK: puppet ran at Fri Jun 28 16:52:13 UTC 2013 [16:52:20] RECOVERY - Puppet freshness on mw1046 is OK: puppet ran at Fri Jun 28 16:52:14 UTC 2013 [16:52:20] RECOVERY - Puppet freshness on solr1002 is OK: puppet ran at Fri Jun 28 16:52:15 UTC 2013 [16:52:20] RECOVERY - Puppet freshness on sq74 is OK: puppet ran at Fri Jun 28 16:52:16 UTC 2013 [16:52:21] RECOVERY - Puppet freshness on virt11 is OK: puppet ran at Fri Jun 28 16:52:17 UTC 2013 [16:52:30] RECOVERY - Puppet freshness on amslvs1 is OK: puppet ran at Fri Jun 28 16:52:19 UTC 2013 [16:52:30] RECOVERY - Puppet freshness on virt1000 is OK: puppet ran at Fri Jun 28 16:52:20 UTC 2013 [16:52:30] RECOVERY - Puppet freshness on ms-be11 is OK: puppet ran at Fri Jun 28 16:52:22 UTC 2013 [16:52:30] RECOVERY - Puppet freshness on ssl1001 is OK: puppet ran at Fri Jun 28 16:52:24 UTC 2013 [16:52:30] RECOVERY - Puppet freshness on analytics1004 is OK: puppet ran at Fri Jun 28 16:52:26 UTC 2013 [16:52:30] RECOVERY - Puppet freshness on srv260 is OK: puppet ran at Fri Jun 28 16:52:27 UTC 2013 [16:52:30] RECOVERY - Puppet freshness on search25 is OK: puppet ran at Fri Jun 28 16:52:28 UTC 2013 [16:52:40] RECOVERY - Puppet freshness on srv262 is OK: puppet ran at Fri Jun 28 16:52:30 UTC 2013 [16:52:40] RECOVERY - Puppet freshness on search16 is OK: puppet ran at Fri Jun 28 16:52:31 UTC 2013 [16:52:40] RECOVERY - Puppet freshness on search1001 is OK: puppet ran at Fri Jun 28 16:52:32 UTC 2013 [16:52:40] RECOVERY - Puppet freshness on db1033 is OK: puppet ran at Fri Jun 28 16:52:33 UTC 2013 [16:52:40] RECOVERY - Puppet freshness on solr1003 is OK: puppet ran at Fri Jun 28 16:52:34 UTC 2013 [16:52:40] RECOVERY - Puppet freshness on db1051 is OK: puppet ran at Fri Jun 28 16:52:35 UTC 2013 [16:52:40] RECOVERY - Puppet freshness on cp1037 is OK: puppet ran at Fri Jun 28 16:52:35 UTC 2013 [16:52:41] RECOVERY - Puppet freshness on db54 is OK: puppet ran at Fri Jun 28 16:52:36 UTC 2013 [16:52:41] RECOVERY - Puppet freshness on db66 is OK: puppet ran at Fri Jun 28 16:52:36 UTC 2013 [16:52:42] RECOVERY - Puppet freshness on cp1026 is OK: puppet ran at Fri Jun 28 16:52:37 UTC 2013 [16:52:50] RECOVERY - Puppet freshness on amssq48 is OK: puppet ran at Fri Jun 28 16:52:39 UTC 2013 [16:52:50] RECOVERY - Puppet freshness on mw37 is OK: puppet ran at Fri Jun 28 16:52:40 UTC 2013 [16:52:50] RECOVERY - Puppet freshness on mw1118 is OK: puppet ran at Fri Jun 28 16:52:40 UTC 2013 [16:52:50] RECOVERY - Puppet freshness on mw104 is OK: puppet ran at Fri Jun 28 16:52:42 UTC 2013 [16:52:50] RECOVERY - Puppet freshness on mw1166 is OK: puppet ran at Fri Jun 28 16:52:44 UTC 2013 [16:52:50] RECOVERY - Puppet freshness on mw68 is OK: puppet ran at Fri Jun 28 16:52:45 UTC 2013 [16:52:50] RECOVERY - Puppet freshness on amssq47 is OK: puppet ran at Fri Jun 28 16:52:45 UTC 2013 [16:52:51] RECOVERY - Puppet freshness on mw2 is OK: puppet ran at Fri Jun 28 16:52:46 UTC 2013 [16:52:51] RECOVERY - Puppet freshness on mw1142 is OK: puppet ran at Fri Jun 28 16:52:46 UTC 2013 [16:52:52] RECOVERY - Puppet freshness on mw1033 is OK: puppet ran at Fri Jun 28 16:52:47 UTC 2013 [16:52:56] Good morning, icinga-wm [16:53:00] RECOVERY - Puppet freshness on sockpuppet is OK: puppet ran at Fri Jun 28 16:52:56 UTC 2013 [16:53:00] RECOVERY - Puppet freshness on mw1010 is OK: puppet ran at Fri Jun 28 16:52:58 UTC 2013 [16:53:00] RECOVERY - Puppet freshness on srv269 is OK: puppet ran at Fri Jun 28 16:52:59 UTC 2013 [16:53:10] RECOVERY - Puppet freshness on sq43 is OK: puppet ran at Fri Jun 28 16:53:00 UTC 2013 [16:53:10] RECOVERY - Puppet freshness on mw1150 is OK: puppet ran at Fri Jun 28 16:53:02 UTC 2013 [16:53:10] RECOVERY - Puppet freshness on mw1025 is OK: puppet ran at Fri Jun 28 16:53:06 UTC 2013 [16:53:10] RECOVERY - Puppet freshness on srv267 is OK: puppet ran at Fri Jun 28 16:53:07 UTC 2013 [16:53:10] RECOVERY - Puppet freshness on mw1092 is OK: puppet ran at Fri Jun 28 16:53:08 UTC 2013 [16:53:10] RECOVERY - Puppet freshness on search1022 is OK: puppet ran at Fri Jun 28 16:53:09 UTC 2013 [16:53:19] New patchset: Ottomata; "Creating role::analytics::packages class, installing python-lxml package" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/71078 [16:53:20] RECOVERY - Puppet freshness on fluorine is OK: puppet ran at Fri Jun 28 16:53:09 UTC 2013 [16:53:20] RECOVERY - Puppet freshness on analytics1003 is OK: puppet ran at Fri Jun 28 16:53:11 UTC 2013 [16:53:20] RECOVERY - Puppet freshness on search29 is OK: puppet ran at Fri Jun 28 16:53:12 UTC 2013 [16:53:20] RECOVERY - Puppet freshness on mw116 is OK: puppet ran at Fri Jun 28 16:53:15 UTC 2013 [16:53:20] RECOVERY - Puppet freshness on mw10 is OK: puppet ran at Fri Jun 28 16:53:15 UTC 2013 [16:53:20] RECOVERY - Puppet freshness on mw28 is OK: puppet ran at Fri Jun 28 16:53:16 UTC 2013 [16:53:20] RECOVERY - Puppet freshness on ms-be3 is OK: puppet ran at Fri Jun 28 16:53:17 UTC 2013 [16:53:21] RECOVERY - Puppet freshness on mw60 is OK: puppet ran at Fri Jun 28 16:53:18 UTC 2013 [16:53:30] RECOVERY - Puppet freshness on snapshot1 is OK: puppet ran at Fri Jun 28 16:53:19 UTC 2013 [16:53:30] RECOVERY - Puppet freshness on mw23 is OK: puppet ran at Fri Jun 28 16:53:20 UTC 2013 [16:53:30] RECOVERY - Puppet freshness on mw1213 is OK: puppet ran at Fri Jun 28 16:53:21 UTC 2013 [16:53:30] RECOVERY - Puppet freshness on mw1054 is OK: puppet ran at Fri Jun 28 16:53:23 UTC 2013 [16:53:30] RECOVERY - Puppet freshness on sq52 is OK: puppet ran at Fri Jun 28 16:53:24 UTC 2013 [16:53:30] RECOVERY - Puppet freshness on mw1107 is OK: puppet ran at Fri Jun 28 16:53:25 UTC 2013 [16:53:30] RECOVERY - Puppet freshness on search1007 is OK: puppet ran at Fri Jun 28 16:53:27 UTC 2013 [16:53:40] RECOVERY - Puppet freshness on search1012 is OK: puppet ran at Fri Jun 28 16:53:30 UTC 2013 [16:53:40] RECOVERY - Puppet freshness on sq41 is OK: puppet ran at Fri Jun 28 16:53:31 UTC 2013 [16:53:40] RECOVERY - Puppet freshness on db1042 is OK: puppet ran at Fri Jun 28 16:53:32 UTC 2013 [16:53:40] RECOVERY - Puppet freshness on mw1129 is OK: puppet ran at Fri Jun 28 16:53:33 UTC 2013 [16:53:40] RECOVERY - Puppet freshness on mw1204 is OK: puppet ran at Fri Jun 28 16:53:34 UTC 2013 [16:53:40] RECOVERY - Puppet freshness on mw1066 is OK: puppet ran at Fri Jun 28 16:53:35 UTC 2013 [16:53:40] RECOVERY - Puppet freshness on mw1211 is OK: puppet ran at Fri Jun 28 16:53:36 UTC 2013 [16:53:41] RECOVERY - Puppet freshness on mw1027 is OK: puppet ran at Fri Jun 28 16:53:37 UTC 2013 [16:53:41] RECOVERY - Puppet freshness on mw1011 is OK: puppet ran at Fri Jun 28 16:53:39 UTC 2013 [16:53:50] RECOVERY - Puppet freshness on mc7 is OK: puppet ran at Fri Jun 28 16:53:40 UTC 2013 [16:53:50] RECOVERY - Puppet freshness on cp3011 is OK: puppet ran at Fri Jun 28 16:53:41 UTC 2013 [16:53:50] RECOVERY - Puppet freshness on amslvs4 is OK: puppet ran at Fri Jun 28 16:53:42 UTC 2013 [16:53:51] RECOVERY - Puppet freshness on dataset1001 is OK: puppet ran at Fri Jun 28 16:53:44 UTC 2013 [16:53:51] RECOVERY - Puppet freshness on labstore1 is OK: puppet ran at Fri Jun 28 16:53:45 UTC 2013 [16:53:51] RECOVERY - Puppet freshness on labstore1001 is OK: puppet ran at Fri Jun 28 16:53:46 UTC 2013 [16:53:51] RECOVERY - Puppet freshness on solr1 is OK: puppet ran at Fri Jun 28 16:53:47 UTC 2013 [16:54:00] RECOVERY - Puppet freshness on antimony is OK: puppet ran at Fri Jun 28 16:53:50 UTC 2013 [16:54:00] RECOVERY - Puppet freshness on mc1012 is OK: puppet ran at Fri Jun 28 16:53:52 UTC 2013 [16:54:00] RECOVERY - Puppet freshness on ssl1 is OK: puppet ran at Fri Jun 28 16:53:53 UTC 2013 [16:54:00] RECOVERY - Puppet freshness on db1014 is OK: puppet ran at Fri Jun 28 16:53:55 UTC 2013 [16:54:00] RECOVERY - Puppet freshness on srv242 is OK: puppet ran at Fri Jun 28 16:53:56 UTC 2013 [16:54:00] RECOVERY - Puppet freshness on labstore3 is OK: puppet ran at Fri Jun 28 16:53:57 UTC 2013 [16:54:00] RECOVERY - Puppet freshness on wtp1004 is OK: puppet ran at Fri Jun 28 16:53:58 UTC 2013 [16:54:09] Change merged: Ottomata; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/71078 [16:54:11] RECOVERY - Puppet freshness on ms-be12 is OK: puppet ran at Fri Jun 28 16:54:01 UTC 2013 [16:54:11] RECOVERY - Puppet freshness on mw42 is OK: puppet ran at Fri Jun 28 16:54:02 UTC 2013 [16:54:11] RECOVERY - Puppet freshness on potassium is OK: puppet ran at Fri Jun 28 16:54:03 UTC 2013 [16:54:11] RECOVERY - Puppet freshness on mw75 is OK: puppet ran at Fri Jun 28 16:54:04 UTC 2013 [16:54:11] RECOVERY - Puppet freshness on cp1058 is OK: puppet ran at Fri Jun 28 16:54:05 UTC 2013 [16:54:11] RECOVERY - Puppet freshness on mw1208 is OK: puppet ran at Fri Jun 28 16:54:06 UTC 2013 [16:54:11] RECOVERY - Puppet freshness on mw1206 is OK: puppet ran at Fri Jun 28 16:54:07 UTC 2013 [16:54:20] RECOVERY - Puppet freshness on mw1104 is OK: puppet ran at Fri Jun 28 16:54:09 UTC 2013 [16:54:20] RECOVERY - Puppet freshness on ms-fe1002 is OK: puppet ran at Fri Jun 28 16:54:12 UTC 2013 [16:54:20] RECOVERY - Puppet freshness on mw1154 is OK: puppet ran at Fri Jun 28 16:54:13 UTC 2013 [16:54:20] RECOVERY - Puppet freshness on mw1155 is OK: puppet ran at Fri Jun 28 16:54:14 UTC 2013 [16:54:20] RECOVERY - Puppet freshness on sq84 is OK: puppet ran at Fri Jun 28 16:54:15 UTC 2013 [16:54:20] RECOVERY - Puppet freshness on es1004 is OK: puppet ran at Fri Jun 28 16:54:19 UTC 2013 [16:54:31] RECOVERY - Puppet freshness on amssq56 is OK: puppet ran at Fri Jun 28 16:54:21 UTC 2013 [16:54:31] RECOVERY - Puppet freshness on cp1066 is OK: puppet ran at Fri Jun 28 16:54:24 UTC 2013 [16:54:31] RECOVERY - Puppet freshness on mw1131 is OK: puppet ran at Fri Jun 28 16:54:27 UTC 2013 [16:54:41] RECOVERY - Puppet freshness on db9 is OK: puppet ran at Fri Jun 28 16:54:29 UTC 2013 [16:54:41] RECOVERY - Puppet freshness on capella is OK: puppet ran at Fri Jun 28 16:54:33 UTC 2013 [16:54:49] paravoid: wtf [16:55:20] RECOVERY - Puppet freshness on pdf3 is OK: puppet ran at Fri Jun 28 16:55:11 UTC 2013 [16:55:29] !log cause of brief outage: operator error caused restart of varnish on all upload caches eqiad, cascaded into a swift overload, cascaded into imagescaler outage & page [16:55:33] AzaToth: ? [16:55:38] Logged the message, Master [16:55:40] RECOVERY - Puppet freshness on mw1143 is OK: puppet ran at Fri Jun 28 16:55:32 UTC 2013 [16:55:50] RECOVERY - Puppet freshness on mw1091 is OK: puppet ran at Fri Jun 28 16:55:39 UTC 2013 [16:56:10] RECOVERY - Puppet freshness on wtp1001 is OK: puppet ran at Fri Jun 28 16:56:00 UTC 2013 [17:00:40] ottomata: looks like that branch needs to exist in gerrit before I can push to it. [17:01:00] ok [17:01:02] can do [17:01:43] thanks [17:01:44] ok, you might want to delete your local debian branch, if you already created it [17:01:46] and then fetch it [17:01:53] i just created it in gerrit branched from wikimedia branch [17:05:11] New patchset: Manybubbles; "Rebuild build.xml and debian/ for git-buildpackage" [operations/debs/jmxtrans] (debian) - https://gerrit.wikimedia.org/r/71079 [17:07:03] ok! I've added ottomata and faidon to the reviewers list because you guys both know about this stuff. [17:07:11] anyone else interested in reviewin jmxtrans? [17:08:00] I'm impressed [17:08:02] that looks very decent [17:11:09] PROBLEM - Puppet freshness on celsus is CRITICAL: No successful Puppet run in the last 10 hours [17:19:40] !log installing some pkg upgrades on fenari [17:19:49] Logged the message, Master [17:24:08] oo, LeslieCarr, check_ganglia is nice! [17:24:11] —list is great [17:25:17] New review: Manybubbles; "You can build the deb by going to a machine with misc::package-builder and then doing this (substitu..." [operations/debs/jmxtrans] (debian) - https://gerrit.wikimedia.org/r/71079 [17:25:22] paravoid, can you log into labs instance puppet-cleanup-rsync2 and look t what's happening with the rsync service? Seems like there may still be disagreement about the pid file but I can't quite sort it out [17:32:34] !log upgrading binutils and libldap on bast1001 [17:32:43] Logged the message, Master [17:56:26] <^demon> !log restarting gerrit, bringing back missing documentation lost during upgrade [17:56:35] Logged the message, Master [17:58:25] ^demon: not sure if you're aware, gerrit has been really slow today [17:58:41] <^demon> Merging has been a tad slow since the upgrade yesterday. [17:58:53] viewing changesets too [17:59:27] it's down atm, presumably still restarting? [17:59:54] <^demon> Back now [18:01:09] thanks [18:09:47] PROBLEM - Puppet freshness on celsus is CRITICAL: No successful Puppet run in the last 10 hours [18:13:29] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:14:17] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.131 second response time [18:15:54] oh, LeslieCarr, that time range doesn't take into account Sean, though, right? [18:16:06] multiple ops folks [18:16:10] are preferred [18:16:17] and sean needs to be here for a while [18:16:27] ok [18:16:29] * greg-g nods [18:16:31] so it does not, on purpose [18:16:57] Does Tim count as .5+ of an op? ;) [18:16:58] are we talking about scheduled times for deployments or for times we're available for emergencies? [18:17:52] well, mostly scheduled [18:18:33] scheduled [18:18:43] i mean for actual emergencies, that's different because we'll all freak out [18:18:52] :-) [18:19:00] but for non emergency, avoidable, high risk situations [18:19:03] such as code pushes [18:19:03] * greg-g gets a song in his head now [18:19:13] le freak, c'est chic [18:19:13] is this particular outage a good example of where ops can help? :) [18:19:31] not sure, honestly, other than the backup idea Tim had [18:19:31] I mean, this was something that 95% of ops wouldn't know how to handle [18:19:35] * greg-g nods [18:19:36] being mediawiki deployment related [18:19:57] of course it's good to not page everyone at their evening or night [18:20:09] however, it did page all of us, so whether or not we could help is irrelevant [18:20:11] exactly [18:20:18] but let's say that this is the problem we're trying to solve, not panicking people at random hours [18:20:42] right, totally want to fix that. [18:20:50] for various values of "fix" [18:21:27] two ideas from today's deploy update meeting I have with engmgt: [18:21:28] 1) [18:21:47] 1) more noticeable "hey, I'm getting ready to deploy" communication (where? how?) [18:21:55] ^demon: I'm restarting out instances of solr so I can scrape them with jmxtrans. you shouldn't notice if all goes well. [18:22:10] 2) more noticeable "hey, we're delayed by X time, we've rescheduled/plan on going at Ytime" [18:22:27] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:22:48] both of those suffer from the "how to notify the *right* people without annoying the rest of them" [18:22:54] ...problem [18:23:30] one technical solution for 1) is something like https://github.com/etsy/PushBot#readme [18:23:33] well [18:23:39] but I'm jumping the gun [18:23:47] one great solution would be 'i'm delayed by 4 hours, perhaps i shall wait until tomorrow' [18:24:17] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.126 second response time [18:24:22] generally, yes, but there are extenuating circumstances that didn't allow that with VE this time [18:24:28] like what ? [18:24:37] was the site broken ? [18:26:28] "tomorrow" being Friday, a rollout that needed to happen before Monday (with >1.5 days between rollout and Monday). Reason why Monday is a limiting factor is that there has been massive communication with the community re turning it on by default for logged in users on English (big. effing. deal.) And E3 wanted a to do some baseline stats pre-Monday, so things with VE needed to be done by today for them to do that (so they have >1.5 days of [18:27:43] so, what you are saying is that your team was unprepared and rolled out too late [18:27:53] my team? [18:28:01] anywho... [18:28:02] sorry, the ve team [18:28:05] LeslieCarr: Several things were broken in VisualEditor, yes. And still are. (Broken in both the not-working and the not-helpful senses.) [18:28:18] rolling out on friday daytime is better than paging everyone at night [18:28:20] LeslieCarr: Patches are welcome. :-) [18:28:31] LeslieCarr: "Day"time? For whom? [18:29:15] for either european or american operations people to be online [18:29:22] this is why we have deployment windows [18:29:41] so, again, I'm not saying everything went perfectly (obviously not) and the timing was great (obviously not). More, there were reasons why the decision was made, and like the law, reasonable minds will disagree. We can have better communication and better cut-off times, but we had none of that until the conversation started this morning. So let's work postively towards a good solution for everyone. [18:29:56] Indeed. [18:30:22] s/can have/should have had yesterday/ # ;-) [18:30:27] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:31:09] I guess the thing that struck me is that there was an implicit assumption that it wouldn't turn into a big deal [18:31:15] and that's just not a saf assumption to make [18:31:17] *safe [18:31:17] one postive thing that's come out of it is that explicit time window of "ops is here and things are ok to schedule during these hours" [18:31:36] any deployment except "the site is broken and we must fix it now" should be dealt with, with that in mind [18:31:41] * greg-g nods [18:32:14] hi apergos [18:32:16] not just "ops is here" but also "the right dev people are here and will be around if something weird breaks" [18:32:19] hey aude [18:32:33] i am trying to clone operations/dumps and then git log shows me the newest change is from february [18:32:39] i know there is newer stuff [18:32:44] you want ariel branch [18:32:44] it's also not in git.wikimedia.org [18:32:48] apergos: that was already sort of know, at least it should have been. Part of the "don't deploy and run" rule (at least how I think about it) [18:32:48] ok [18:32:55] hey, I don't seem to have a login on the new Parsoid varnishes cp1045 and cp1058. Could somebody either give me access to those machines as I had on titanium and cerium, or do some debugging for me? [18:32:56] sorry, it's on the wikitech pages [18:33:10] thanks [18:33:16] greg-g: yeah, I think that's what has us most unhappy [18:33:17] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.134 second response time [18:33:24] apergos: totally. [18:33:25] much better! [18:33:35] if we can work on that, that will go a long way to making everyone's life better [18:33:38] apergos: In this case, the "right dev people" are Tim and Reedy, AFAICT. [18:33:52] that's worth knowing and planning for then [18:33:52] apergos: Which is a reasonable clock-round coverage, but not great, of course. [18:33:58] not great, I agree [18:34:05] well, and whoever wrote/merged the code, but in this situation, it wasn't even that code that broke it [18:34:08] * greg-g looks at aude :P [18:34:11] * James_F nods. [18:34:27] yeah, there are going to be times like that [18:34:30] btw, for aude: https://wikitech.wikimedia.org/wiki/Incident_documentation/20130628-Site [18:34:37] * aude sighs [18:35:02] apergos: We need more people trained up in the intricacies of Platform's deployment system - and ideally in North America and South Asia or similar timezones. [18:35:29] sounds like a new team goal ;-) [18:35:29] greg-g: https://gerrit.wikimedia.org/r/#/c/71056/ (it doesn't solve everything and need tim or someone's opinion) [18:35:31] and mostly engineering (plus some Ops) need to make the whole deploy everything better/more robust [18:35:34] who would be the right person to ask about access to the Parsoid varnishes? [18:35:52] we've made it a pretty high priroity for July+ [18:36:02] gwicke: want me to do some debugging for you? im on the box [18:36:09] that would be nice (more robust deploy) but I think we should assume that the change you think will be small and harmless might not be [18:36:22] seems delicate to introduce the new extension and run l10n for wmf9 while it's not there for wmf8 [18:36:33] because (much like how most caar accidents happen close to home when you think you're out of the danger zone) [18:36:34] mutante: could you do a varnishlog dump? [18:36:38] and seriously, if you all have things you want us to address that will help with this, deployment wise, please do let me know/report a bug (and cc me) [18:36:38] that's when impact will be the worst [18:36:40] i think it should just be ignored if the extension does not exist in some branch [18:36:57] mutante: 30 seconds or so should be enough [18:37:02] aude: I'm curious what you wil be doing with the dumps stuff. anything cool? [18:37:11] apergos: someone wants the entity per page table [18:37:15] apergos: oh, agreed, but if we start thinking that way too much, we'll move back to 6month deploy cycle :) [18:37:24] i can go ahead and make a patch if it helps [18:37:29] how mig is that compared to everything else? [18:37:38] *big [18:37:39] yes, and sure, having a global team would be great, having a good deployment system with tests would be super awesome. but today, we can make sure to not push code unless there are ops online, preferably during the planned deployment windows, and no "but this was a special case" unless the site is actively broken [18:37:51] ie compared to uh [18:37:55] LeslieCarr: we agree. [18:38:01] cool [18:38:03] :) [18:38:05] glad we're all on the same page :) [18:38:13] which one is that again? :P [18:38:17] terms and items per site [18:38:21] <^demon> manybubbles: Cool beans [18:38:22] wikitech or mediawiki... [18:38:43] ^demon: you can even see the metrics now: http://ganglia.wmflabs.org/latest/?r=hour&cs=&ce=&c=solr&h=solr1&tab=m&vn=&mc=2&z=medium&metric_group=NOGROUPS_|_memory [18:39:04] and sure you can totally patch it in and if you get it in by tomorrow I can run it (the current wikidata is still completing so it's no big deal to just do the last table manually, if it's a reasonable size) [18:39:05] apergos: there is one row per entity [18:39:06] ^demon: I'll see if I can get my machine built with puppet which'll be pretty cool too [18:39:15] vs terms which has lots of rows per entityt [18:39:16] entity [18:39:19] anywho, opsen: please do add things related to this outage that can be improved to that incident page, even things as nebulus as "communication" [18:39:25] and items per site has lots of rows [18:39:31] all right, hack it in [18:39:31] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:39:34] ok [18:39:35] add me as a reviewer [18:39:41] ok [18:39:47] i'm coming up with a description now [18:39:52] you can pretty much exactly copy the items per site code [18:39:53] I give ya'll (opsen) permission to add things to my list (this might hurt me later...) [18:40:09] <^demon> manybubbles: jvm metrics :D [18:40:20] greg-g: cool, I am sure we will be taking you up on it [18:40:21] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.128 second response time [18:40:24] mailing varnish logs from those new parsoid boxes to gwicke [18:40:29] apergos: :) [18:41:07] ^demon: I also have some solr metrics running on solr0. neat stuff like cache hit rates and stuff [18:41:35] aude: btw, thanks much for that merge request. [18:41:54] apergos: debugging cache invalidation after template updates [18:42:04] New patchset: Aude; "Add wikidata wb_entity_per_page table to dumps" [operations/dumps] (ariel) - https://gerrit.wikimedia.org/r/71087 [18:42:08] eh, aude [18:42:11] heh [18:42:17] I wondered why that was coming to me [18:42:24] yeah, sorry ;) [18:42:31] https://gerrit.wikimedia.org/r/#/c/71087/ [18:42:33] we had some weird cache issues crop up earlier today but can't imagine it has anything to do with you [18:42:40] apergos: ^ [18:42:43] yep [18:42:46] ^demon: Hm.. I'm trying to disable Verified/Submit for non-bots on a repo but it looks like it didn't work. I applied the same settings as in mw/ext/VisualEditor (where it does work, and I don't see Verified anymore in my UI), but where I applied it just now (mw/ext/MobileFrontend) I can still see them. [18:43:06] oh no the dreaded "working" [18:43:24] needs to go into the lists of jobs [18:43:25] <^demon> Krinkle: What did you do? They're granted by default to project owners at the All-Projects level. [18:43:28] check where the uh [18:43:52] wbitemspersitetable I think that. check other places in the code and do accordingly [18:44:05] (also please test it :-P ) [18:44:22] https://git.wikimedia.org/tree/mediawiki%2Fextensions%2FVisualEditor.git/refs%2Fmeta%2Fconfig [18:44:28] aude: [18:44:30] https://git.wikimedia.org/tree/mediawiki%2Fextensions%2FMobileFrontend.git/refs%2Fmeta%2Fconfig [18:44:55] <^demon> Krinkle: Maybe check exclusive? [18:44:58] apergos: ok [18:45:03] <^demon> I think ALLOW overrules DENY [18:45:25] exclusiveGroupPermissions = create push pushMerge label-Verified submit [18:45:26] Aha [18:45:32] I forgot to add that line as well (or something similar) [18:45:35] <^demon> Gerrit ACL has always been a black box to me tho :p [18:45:40] <^demon> I just guess and hope for the best. [18:45:46] hahaha [18:45:48] hi, who is responsible for en.wikipedia.beta.wmflabs.org ? [18:45:50] should bash that [18:45:50] Then I've been doing it all wrong :P [18:45:59] oh well, harmless fortunately [18:46:03] uh a fw of us work on it, depeneds which bit [18:46:08] se4598 [18:46:17] hashar will likely know more, what's up though? [18:46:35] se4598 I know a lot about en.wikipedia.beta.wmflabs.org, what is your question? [18:47:09] greg-g: thank you [18:47:21] chrismcmahon: php fatals, -> #wikimedia-tech ? [18:48:18] ^demon: Thanks, that did it. [18:48:23] ah nope, that's in a part of the innards I haven't messed with [18:48:27] New patchset: Ottomata; "Adding AsyncProducerEvents to kafka ganglia view" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/71089 [18:48:33] <^demon> Krinkle: yw [18:48:42] Change merged: Ottomata; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/71089 [18:52:06] apergos: hmmm jenkins is complaining about all sorts of syntax issues [18:53:33] the non-voting? ignore [18:53:37] I'll clean up at some point [18:53:51] line too long is going to get ignored for rather a lot longer [18:54:00] ok [18:55:31] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:57:23] LeslieCarr: /dev/md0 610800 554542 56258 91% / [18:57:23] from neon, just cause you were on there earlier, (that's inode count) [18:57:48] ugh, again [18:57:50] before it fills up it woul dbe nice to know if the accumulation is from several days or [18:57:57] from just the restart time (for example) [18:58:00] eep [18:58:03] would relly like to nail that down [18:58:07] *to [18:58:15] would like eyes that work too [18:58:21] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.152 second response time [18:58:22] can't have everything :-/ [18:58:28] i wonder if it's from the ganglios thing [18:58:34] since we need ganglia running on it [18:58:37] snmptt [18:59:09] /var/spool/snmptt/ [18:59:15] so maybe stopping snmptt for a bit, shoot anythig it leaves around, then we can examine the directory and check timestamps (with that many files it's going to take along time to give a sorted list) [18:59:40] deleting them all and restarting snmptt fixed it last time [18:59:49] I admit to not wanting to take point cause 10pm and so that's 13 hours which I gotta stop doing [19:01:12] "ls" doesnt even finish, i'll do some loop with find again [19:01:21] rm, too many arguments, bla bla [19:01:39] mutante: xargs -n [19:04:32] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:05:12] yeah I have a command I run [19:05:23] I would just have liked to do the diagnostics so we have some notion of the cause [19:05:41] apergos: starting Jun 26 [19:05:56] ori-l: thanks, just wanted to see the dates first [19:05:59] ls -U, head to get the first so many, rm those... doing them in batches of 1k each [19:06:10] whatever method works for you though [19:06:19] find . -exec ls -hals [19:06:21] yep [19:06:31] june 26, so not today. that's not awesome [19:06:39] so they are all June 26,27 and 28 [19:06:39] ok this needs cron [19:06:43] eep [19:06:44] nod [19:06:45] how many are june 26? [19:06:48] why does snmptt do those bad things ? [19:06:49] percentage [19:07:05] cause like most software it's buggy. or maybe that's all software.... [19:07:21] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.127 second response time [19:07:46] basically trying to find out if we have spike of today (in which daily cron isn't enough) or we have same behavior for 3 days (in which case daily is fine) [19:07:49] !log deleting /var/spool/snmptt/ files on neon running out of inodes [19:07:58] Logged the message, Master [19:08:47] apergos: let's do a cron with find -mtime 1 day [19:08:55] so deletes anything older than 1 day [19:08:59] and run it daily [19:09:15] * apergos has another idea too. gangia graph the inode count [19:09:19] *ganglia [19:09:38] heh, sorry, i have already started deleting before calculating percentages [19:09:43] that's fine [19:09:51] if ganglia gets it we'll have the record [19:11:28] greg-g: added some comments https://wikitech.wikimedia.org/wiki/Incident_documentation/20130628-Site#Lessons_Learned [19:11:29] apergos: check_disk nagios plugin: [19:11:43] K, --icritical=PERCENT% Exit with CRITICAL status if less than PERCENT of inode space is free [19:11:43] feedback appreciated on anything we can do better from our side [19:12:32] that's another good thing (rather than relying on my luck, which, ok I was lucky the last two times) [19:12:45] aude: I was about to comment on your patch. I think it'd be better to exit with an error if there's a missing file rather than just print an error to stderr and carry on [19:13:07] hmmm, then the extension has to be present in *all* the branches [19:13:15] even if it's only needed for, say a new branch [19:13:26] how else can we handle that situation? [19:13:30] that's already the requirement, afaik [19:13:45] for it to be inall the branches? [19:13:56] yep [19:13:58] problem is we are moving code that is in wikibase out [19:14:02] into stand alone [19:14:18] so it already gets loaded in wmf8 and then to load it again? [19:14:30] it doesn't have to be *loaded* in all branches; just present [19:14:45] hmmmm [19:14:56] we do some magic to autoload it [19:15:11] so we check if the extension is loaded with the constant and then skip loading it again [19:15:16] ori-l: while waiting now i looked at benchmarks for different methods:) seems rsync -delete is fastest http://linuxnote.net/jianingy/en/linux/a-fast-way-to-remove-huge-number-of-files.html [19:15:33] mutante: that was on hackernews recently, i thought it was cool :) [19:15:42] it's looking much better over there [19:15:50] /dev/md0 610800 388175 222625 64% / [19:16:06] switched to find . -type f -delete [19:16:28] that prolly walks through the directory in order , much better [19:16:44] trying to match -name was probably a waste [19:16:47] yep [19:17:00] ori-l: maybe we can allow whoever runs the script the option to ignore the error? [19:17:04] it's done [19:17:08] idk [19:17:57] the files are being created as root:root, I think that's probably wrong [19:17:59] aude: a '--force' option might be useful for independent reasons, but there should be a standard way to run the command and it should abort if something funny is happening. [19:18:15] ok, so i could make it force? [19:18:41] it's almost always better to abort scap and give the deployer a chance to look around and fix things then for the code to assume everything will be fine [19:18:49] having it load twice in wmf8 would be scary [19:19:01] since it might use the *wrong* version, untested of the data model [19:19:07] ori-l: makes sense [19:19:15] i haven't seen your autoloader magic, but i can't imagine it'd be too hard to work around this [19:19:35] open to suggestions [19:19:52] !log attempting clean stop and restart of snmptt service on neon [19:20:01] Logged the message, Master [19:20:37] root 32032 0.0 0.1 42888 8880 ? Ss 19:20 0:00 /usr/bin/perl /usr/sbin/snmptt --daemon [19:20:40] snmptt 32033 10.3 0.1 51320 10004 ? Ss 19:20 0:00 /usr/bin/perl /usr/sbin/snmptt --daemon [19:20:43] apergos: [19:20:45] aude: also we should probably file a new bug for mergeMessageFileList issues; 50347 is about the shell script which has independent issues [19:20:52] I see it [19:20:55] agree [19:21:04] there are multiple issues and should be multiple bugs [19:22:25] aude: and last but not least, tim requested that a scary comment header be added to 'extension-list', but that's not currently possible as every line is interpreted as a file name. if you're changing that code you may want to skip over lines starting with '#'. [19:23:49] ori-l: ok [19:27:08] New patchset: Andrew Bogott; "Don't specify pid file in rsync.conf." [operations/puppet] (production) - https://gerrit.wikimedia.org/r/71105 [19:27:08] New patchset: Andrew Bogott; "Convert the search rsyncd to the new rsync module." [operations/puppet] (production) - https://gerrit.wikimedia.org/r/71106 [19:27:08] New patchset: Andrew Bogott; "Convert swift's rsyncd to the new rsync module." [operations/puppet] (production) - https://gerrit.wikimedia.org/r/71107 [19:27:23] RECOVERY - Puppet freshness on spence is OK: puppet ran at Fri Jun 28 19:27:14 UTC 2013 [19:27:38] New review: Andrew Bogott; "can this be right?" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/71105 [19:30:50] New review: Dzahn; "(1 comment)" [operations/apache-config] (master) - https://gerrit.wikimedia.org/r/65443 [19:31:33] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:31:49] !change 65443 | aude [19:31:49] aude: https://gerrit.wikimedia.org/r/#q,65443,n,z [19:31:59] hi mutante [19:32:29] hey aude, we got another http or https question and the either/or .. hmm [19:32:32] andrewbogott: i don't know what the rsync init.d script does, but that would make sense [19:32:42] aude: for the language subdomain thing [19:32:57] i betcha if pid file is there rsync —daemon, tries to manage one itself? [19:33:00] mutante: for either http or https, it's not so great to choose one [19:33:11] ottomata, I think that must be right. [19:33:16] not sure what/if anythign we can do to work around that [19:33:23] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.125 second response time [19:33:27] since people will get logged out [19:33:38] then edit as an ip and get tracked, when they don't want to [19:34:39] aude: later if we can do it in varnish, it will be possible, but in Apache i dont think we can, it's like https://bugzilla.wikimedia.org/show_bug.cgi?id=47276 [19:35:07] see the latest comment there, it also applies here [19:35:35] aude: going to afk more or less, if there's an updated patch in my review queue tomorrow I'lll look at it at some point and probably dump the table on this run [19:35:39] have a good rest of the day [19:35:46] New review: Aude; "redirecting to https or http from the other is a problem. it means people will get "logged out", may..." [operations/apache-config] (master) C: -1; - https://gerrit.wikimedia.org/r/65443 [19:36:07] mutante: then we might have to wait [19:36:22] apergos: ok [19:36:44] mutante: it's enough of a blocker, imho [19:38:18] aude: yea, unfortunately this issue (making a difference between protocols in redirects.conf) keeps coming back in different contexts [19:38:42] varnish would be it for all of them [19:38:53] mutante: ok [19:39:05] i suppose it's not too far off in the future to have varnish [19:40:28] can ask the bugmeister to link different tickets related to this and raise importance [19:40:43] ok, thanks [19:41:03] RECOVERY - Puppet freshness on ms-fe3001 is OK: puppet ran at Fri Jun 28 19:40:55 UTC 2013 [19:41:21] yw, be back after lunch [20:00:12] PROBLEM - check_mysql on db1025 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 565593 [20:00:31] whew, that's some lag. [20:01:47] just under a week [20:01:57] ha. that's a lie, probably related to just restarting mysql as the check happened [20:05:08] RECOVERY - check_mysql on db1025 is OK: Uptime: 373 Threads: 1 Questions: 673 Slow queries: 0 Opens: 25 Flush tables: 2 Open tables: 51 Queries per second avg: 1.804 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 0 [20:05:10] heya Opsen: today there will be a deploy by the E3 team to turn on some stats collecting code (EventLogging, I assume, right ori-l ?) to get baseline stats before VE is turned on by default for ENWP logged in users on Monday (last chance we have of collecting the data) [20:05:25] time tbd: but before 4:30 [20:05:42] greg-g: Thanks for the notification! [20:12:15] PROBLEM - Puppet freshness on celsus is CRITICAL: No successful Puppet run in the last 10 hours [20:23:27] New patchset: Yuvipanda; "Add Python & PHP redis client libraries to exec environs" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/71113 [20:35:14] New review: coren; "Package installs." [operations/puppet] (production) C: 2; - https://gerrit.wikimedia.org/r/71113 [20:35:15] Change merged: coren; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/71113 [20:48:06] New review: QChris; "(1 comment)" [operations/debs/buck] (master) - https://gerrit.wikimedia.org/r/70673 [20:54:51] !log updated Parsoid to 454f87a [20:55:00] Logged the message, Master [21:09:37] New patchset: Legoktm; "gerrit-wm: pywikibot/* events to #pywikipediabot" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/70780 [21:10:25] New review: Ottomata; "Nik, you should put the custom --git-* flags in debian/gbp.conf." [operations/debs/jmxtrans] (debian) - https://gerrit.wikimedia.org/r/71079 [21:10:28] PROBLEM - Puppet freshness on erzurumi is CRITICAL: No successful Puppet run in the last 10 hours [21:10:28] PROBLEM - Puppet freshness on lvs1005 is CRITICAL: No successful Puppet run in the last 10 hours [21:10:28] PROBLEM - Puppet freshness on lvs1006 is CRITICAL: No successful Puppet run in the last 10 hours [21:10:28] PROBLEM - Puppet freshness on lvs1004 is CRITICAL: No successful Puppet run in the last 10 hours [21:10:28] PROBLEM - Puppet freshness on mc15 is CRITICAL: No successful Puppet run in the last 10 hours [21:10:28] PROBLEM - Puppet freshness on sodium is CRITICAL: No successful Puppet run in the last 10 hours [21:10:28] PROBLEM - Puppet freshness on virt1 is CRITICAL: No successful Puppet run in the last 10 hours [21:10:29] PROBLEM - Puppet freshness on virt3 is CRITICAL: No successful Puppet run in the last 10 hours [21:10:29] PROBLEM - Puppet freshness on virt4 is CRITICAL: No successful Puppet run in the last 10 hours [21:10:35] * legoktm looks for someone to merge that... [21:12:06] PROBLEM - Puppet freshness on celsus is CRITICAL: No successful Puppet run in the last 10 hours [21:17:44] Sorry for the flood, all: [21:17:58] New patchset: Andrew Bogott; "Convert swift's rsyncd to the new rsync module." [operations/puppet] (production) - https://gerrit.wikimedia.org/r/71107 [21:17:58] New patchset: Andrew Bogott; "Convert the search rsyncd to the new rsync module." [operations/puppet] (production) - https://gerrit.wikimedia.org/r/71106 [21:17:59] New patchset: Andrew Bogott; "Don't specify pid file in rsync.conf." [operations/puppet] (production) - https://gerrit.wikimedia.org/r/71105 [21:17:59] New patchset: Andrew Bogott; "Remove misc::deployment::scap_proxy as it appears unused." [operations/puppet] (production) - https://gerrit.wikimedia.org/r/71122 [21:17:59] New patchset: Andrew Bogott; "Convert scap's rsyncd to the new rsync module." [operations/puppet] (production) - https://gerrit.wikimedia.org/r/71123 [21:17:59] New patchset: Andrew Bogott; "Convert the nfs rsyncd to the new rsync module." [operations/puppet] (production) - https://gerrit.wikimedia.org/r/71124 [21:18:00] New patchset: Andrew Bogott; "Convert the udp2log rsyncd to use the rsyncd module." [operations/puppet] (production) - https://gerrit.wikimedia.org/r/71125 [21:18:00] New patchset: Andrew Bogott; "Remove abandoned rsync configs" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/71126 [21:18:00] New patchset: Andrew Bogott; "Remove the now unused generic::rsyncd" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/71127 [21:32:46] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:37:37] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.124 second response time [21:44:50] New review: Faidon; "This is pretty decent, if this is your first package I'm amazed! A few inline comments, many of them..." [operations/debs/jmxtrans] (debian) C: -1; - https://gerrit.wikimedia.org/r/71079 [22:01:38] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:02:40] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.127 second response time [22:06:38] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:07:43] New review: Manybubbles; "This is my first Debian package, yeah. I've built a few rpms though so the concepts all make sense...." [operations/debs/jmxtrans] (debian) - https://gerrit.wikimedia.org/r/71079 [22:07:59] PROBLEM - Puppet freshness on celsus is CRITICAL: No successful Puppet run in the last 10 hours [22:08:09] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.121 second response time [22:41:56] PROBLEM - NTP on ssl3003 is CRITICAL: NTP CRITICAL: No response from NTP server [22:51:36] PROBLEM - NTP on ssl3002 is CRITICAL: NTP CRITICAL: No response from NTP server [23:07:16] PROBLEM - Puppet freshness on manutius is CRITICAL: No successful Puppet run in the last 10 hours [23:08:43] PROBLEM - Puppet freshness on celsus is CRITICAL: No successful Puppet run in the last 10 hours [23:19:21] Reedy: Got a second for a 3 character change merge? [23:19:23] https://gerrit.wikimedia.org/r/#/c/71146/ [23:19:45] Scary! [23:19:54] * James_F grins. [23:27:43] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:28:33] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.128 second response time [23:32:44] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:33:33] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.131 second response time [23:41:47] New patchset: Dzahn; "cron job to delete snmptt spool files to prevent neon from running out of inodes and monitoring breakage" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/71149 [23:56:46] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:57:34] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.125 second response time [23:57:55] New patchset: Dzahn; "cron job to delete snmptt spool files to prevent neon from running out of inodes and monitoring breakage" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/71149 [23:58:36] notpeter: ori-l: actually... /lost+found/ _should_ always be empty. so rsync -a --delete /lost+found/ /var/spool/snmptt/ [23:58:52] it's fast