[02:06:35] !log LocalisationUpdate completed (1.22wmf16) at Mon Sep 16 02:06:34 UTC 2013 [02:06:39] Logged the message, Master [02:11:46] !log LocalisationUpdate completed (1.22wmf17) at Mon Sep 16 02:11:46 UTC 2013 [02:11:49] Logged the message, Master [02:23:57] !log LocalisationUpdate ResourceLoader cache refresh completed at Mon Sep 16 02:23:57 UTC 2013 [02:24:00] Logged the message, Master [07:18:00] what happened to icinga [07:18:20] seems awfully quiet [07:18:29] you put it on ignore? [07:19:00] i didn't. or if i did, i didn't mean too.. [07:19:18] ah icinga.wikimedia.org Warning: Status data OUTDATED! Last status data update was 174729 seconds ago! [07:19:23] that seems wwrong [07:20:13] 48 hours [07:20:48] you and logmsgbot are the only things on the server admin log :P [07:21:00] did you disable icinga checks when you were investigating db1049? [07:21:13] nope. neon log says it's pegged at max xoncurrent connections [07:21:26] concurrent* [07:21:53] lovely [07:22:03] * springle tries a restart [07:25:10] heh. funny how the error was obvious from the lack of errors. [07:25:24] that didn't go so well. neon seems unhappy [07:25:38] OSError: [Errno 30] Read-only file system: '/etc/icinga/tmpI4UDdK' [07:40:13] !log rebooted neon. icinga pegged at max concurrent connections for 48h. root ext3 fs read-only as of 2013-09-14 06:45 detected aborted journal [07:40:17] Logged the message, Master [07:46:20] icinga needs a dead man's switch [07:47:01] we need to resurrect nagios to check that icinga is alive [07:47:21] SAL has more reboots of icinga than of all other servers combined, lately [07:48:20] hope i restarted everything that wasn't automatic [07:51:16] RECOVERY - Puppet freshness on ssl1006 is OK: puppet ran at Mon Sep 16 07:51:05 UTC 2013 [07:51:16] RECOVERY - Puppet freshness on cp3006 is OK: puppet ran at Mon Sep 16 07:51:10 UTC 2013 [07:51:56] RECOVERY - Puppet freshness on search1006 is OK: puppet ran at Mon Sep 16 07:51:51 UTC 2013 [07:52:06] RECOVERY - Puppet freshness on mw1072 is OK: puppet ran at Mon Sep 16 07:52:01 UTC 2013 [07:52:06] RECOVERY - Puppet freshness on mw1219 is OK: puppet ran at Mon Sep 16 07:52:01 UTC 2013 [07:52:06] RECOVERY - Puppet freshness on search1011 is OK: puppet ran at Mon Sep 16 07:52:01 UTC 2013 [07:52:16] RECOVERY - Puppet freshness on magnesium is OK: puppet ran at Mon Sep 16 07:52:06 UTC 2013 [07:52:16] RECOVERY - Puppet freshness on db1011 is OK: puppet ran at Mon Sep 16 07:52:06 UTC 2013 [07:52:16] RECOVERY - Puppet freshness on cp1015 is OK: puppet ran at Mon Sep 16 07:52:06 UTC 2013 [07:52:16] RECOVERY - Puppet freshness on mc1013 is OK: puppet ran at Mon Sep 16 07:52:06 UTC 2013 [07:52:16] RECOVERY - Puppet freshness on ms-be10 is OK: puppet ran at Mon Sep 16 07:52:06 UTC 2013 [07:53:23] RECOVERY - Puppet freshness on analytics1018 is OK: puppet ran at Mon Sep 16 07:53:16 UTC 2013 [07:53:23] RECOVERY - Puppet freshness on mw1086 is OK: puppet ran at Mon Sep 16 07:53:16 UTC 2013 [07:53:23] RECOVERY - Puppet freshness on mw59 is OK: puppet ran at Mon Sep 16 07:53:21 UTC 2013 [07:53:23] RECOVERY - Puppet freshness on mw1121 is OK: puppet ran at Mon Sep 16 07:53:21 UTC 2013 [07:53:23] RECOVERY - Puppet freshness on mw1141 is OK: puppet ran at Mon Sep 16 07:53:21 UTC 2013 [07:53:32] RECOVERY - Puppet freshness on mw1135 is OK: puppet ran at Mon Sep 16 07:53:26 UTC 2013 [07:53:32] RECOVERY - Puppet freshness on pdf3 is OK: puppet ran at Mon Sep 16 07:53:26 UTC 2013 [07:53:33] RECOVERY - Puppet freshness on mw1158 is OK: puppet ran at Mon Sep 16 07:53:26 UTC 2013 [07:53:33] RECOVERY - Puppet freshness on db1045 is OK: puppet ran at Mon Sep 16 07:53:26 UTC 2013 [07:53:33] RECOVERY - Puppet freshness on lvs1003 is OK: puppet ran at Mon Sep 16 07:53:31 UTC 2013 [07:53:33] RECOVERY - Puppet freshness on mw1082 is OK: puppet ran at Mon Sep 16 07:53:31 UTC 2013 [07:53:33] RECOVERY - Puppet freshness on db1057 is OK: puppet ran at Mon Sep 16 07:53:31 UTC 2013 [07:53:34] RECOVERY - Puppet freshness on ssl1002 is OK: puppet ran at Mon Sep 16 07:53:31 UTC 2013 [07:53:34] RECOVERY - Puppet freshness on cp1053 is OK: puppet ran at Mon Sep 16 07:53:31 UTC 2013 [07:53:34] RECOVERY - Puppet freshness on amssq50 is OK: puppet ran at Mon Sep 16 07:53:31 UTC 2013 [07:53:35] RECOVERY - Puppet freshness on analytics1017 is OK: puppet ran at Mon Sep 16 07:53:31 UTC 2013 [07:53:42] RECOVERY - Puppet freshness on cp1039 is OK: puppet ran at Mon Sep 16 07:53:36 UTC 2013 [07:53:42] RECOVERY - Puppet freshness on sq77 is OK: puppet ran at Mon Sep 16 07:53:36 UTC 2013 [07:53:42] RECOVERY - Puppet freshness on ms1004 is OK: puppet ran at Mon Sep 16 07:53:36 UTC 2013 [07:53:42] RECOVERY - Puppet freshness on mw1110 is OK: puppet ran at Mon Sep 16 07:53:36 UTC 2013 [07:53:43] RECOVERY - Puppet freshness on sq49 is OK: puppet ran at Mon Sep 16 07:53:41 UTC 2013 [07:53:43] RECOVERY - Puppet freshness on es2 is OK: puppet ran at Mon Sep 16 07:53:41 UTC 2013 [07:53:43] RECOVERY - Puppet freshness on analytics1020 is OK: puppet ran at Mon Sep 16 07:53:41 UTC 2013 [07:53:52] RECOVERY - Puppet freshness on cp1001 is OK: puppet ran at Mon Sep 16 07:53:46 UTC 2013 [07:53:52] RECOVERY - Puppet freshness on mw1 is OK: puppet ran at Mon Sep 16 07:53:46 UTC 2013 [07:53:52] RECOVERY - Puppet freshness on srv271 is OK: puppet ran at Mon Sep 16 07:53:46 UTC 2013 [07:53:52] RECOVERY - Puppet freshness on cp3019 is OK: puppet ran at Mon Sep 16 07:53:46 UTC 2013 [07:53:52] RECOVERY - Puppet freshness on cp3021 is OK: puppet ran at Mon Sep 16 07:53:46 UTC 2013 [07:54:02] RECOVERY - Puppet freshness on mw92 is OK: puppet ran at Mon Sep 16 07:53:51 UTC 2013 [07:54:02] RECOVERY - Puppet freshness on mw91 is OK: puppet ran at Mon Sep 16 07:53:51 UTC 2013 [07:54:02] RECOVERY - Puppet freshness on search35 is OK: puppet ran at Mon Sep 16 07:53:51 UTC 2013 [07:54:02] RECOVERY - Puppet freshness on emery is OK: puppet ran at Mon Sep 16 07:53:51 UTC 2013 [07:54:02] RECOVERY - Puppet freshness on virt1005 is OK: puppet ran at Mon Sep 16 07:53:51 UTC 2013 [07:54:03] RECOVERY - Puppet freshness on harmon is OK: puppet ran at Mon Sep 16 07:53:56 UTC 2013 [07:54:03] RECOVERY - Puppet freshness on virt8 is OK: puppet ran at Mon Sep 16 07:53:56 UTC 2013 [07:54:04] RECOVERY - Puppet freshness on ms-be1010 is OK: puppet ran at Mon Sep 16 07:53:56 UTC 2013 [07:54:12] RECOVERY - Puppet freshness on mw1174 is OK: puppet ran at Mon Sep 16 07:54:01 UTC 2013 [07:54:12] RECOVERY - Puppet freshness on wtp1016 is OK: puppet ran at Mon Sep 16 07:54:01 UTC 2013 [07:54:12] RECOVERY - Puppet freshness on mw7 is OK: puppet ran at Mon Sep 16 07:54:01 UTC 2013 [07:54:12] RECOVERY - Puppet freshness on srv293 is OK: puppet ran at Mon Sep 16 07:54:06 UTC 2013 [07:54:12] RECOVERY - Puppet freshness on srv251 is OK: puppet ran at Mon Sep 16 07:54:06 UTC 2013 [07:54:13] RECOVERY - Puppet freshness on srv243 is OK: puppet ran at Mon Sep 16 07:54:06 UTC 2013 [07:54:13] RECOVERY - Puppet freshness on mw3 is OK: puppet ran at Mon Sep 16 07:54:07 UTC 2013 [07:54:14] RECOVERY - Puppet freshness on search23 is OK: puppet ran at Mon Sep 16 07:54:07 UTC 2013 [07:54:14] RECOVERY - Puppet freshness on mw1026 is OK: puppet ran at Mon Sep 16 07:54:07 UTC 2013 [07:54:15] RECOVERY - Puppet freshness on mw25 is OK: puppet ran at Mon Sep 16 07:54:07 UTC 2013 [07:54:22] RECOVERY - Puppet freshness on analytics1021 is OK: puppet ran at Mon Sep 16 07:54:12 UTC 2013 [07:54:22] RECOVERY - Puppet freshness on mc1002 is OK: puppet ran at Mon Sep 16 07:54:12 UTC 2013 [07:54:22] RECOVERY - Puppet freshness on mw1199 is OK: puppet ran at Mon Sep 16 07:54:12 UTC 2013 [07:54:22] RECOVERY - Puppet freshness on mw1148 is OK: puppet ran at Mon Sep 16 07:54:17 UTC 2013 [07:54:22] RECOVERY - Puppet freshness on mw1009 is OK: puppet ran at Mon Sep 16 07:54:17 UTC 2013 [07:54:23] RECOVERY - Puppet freshness on mw1120 is OK: puppet ran at Mon Sep 16 07:54:17 UTC 2013 [07:54:23] RECOVERY - Puppet freshness on mw1108 is OK: puppet ran at Mon Sep 16 07:54:17 UTC 2013 [07:54:24] RECOVERY - Puppet freshness on mw1060 is OK: puppet ran at Mon Sep 16 07:54:17 UTC 2013 [07:54:24] RECOVERY - Puppet freshness on mw1008 is OK: puppet ran at Mon Sep 16 07:54:17 UTC 2013 [07:54:32] RECOVERY - Puppet freshness on mw1073 is OK: puppet ran at Mon Sep 16 07:54:27 UTC 2013 [07:54:42] RECOVERY - Puppet freshness on sq66 is OK: puppet ran at Mon Sep 16 07:54:32 UTC 2013 [07:54:42] RECOVERY - Puppet freshness on sq75 is OK: puppet ran at Mon Sep 16 07:54:32 UTC 2013 [07:54:42] RECOVERY - Puppet freshness on cp1008 is OK: puppet ran at Mon Sep 16 07:54:37 UTC 2013 [07:54:52] RECOVERY - Puppet freshness on nitrogen is OK: puppet ran at Mon Sep 16 07:54:42 UTC 2013 [07:54:52] RECOVERY - Puppet freshness on db1046 is OK: puppet ran at Mon Sep 16 07:54:42 UTC 2013 [07:54:52] RECOVERY - Puppet freshness on db1018 is OK: puppet ran at Mon Sep 16 07:54:42 UTC 2013 [07:54:52] RECOVERY - Puppet freshness on cp1009 is OK: puppet ran at Mon Sep 16 07:54:42 UTC 2013 [07:54:52] RECOVERY - Puppet freshness on db64 is OK: puppet ran at Mon Sep 16 07:54:42 UTC 2013 [07:54:52] RECOVERY - Puppet freshness on hydrogen is OK: puppet ran at Mon Sep 16 07:54:47 UTC 2013 [07:54:52] RECOVERY - Puppet freshness on db1015 is OK: puppet ran at Mon Sep 16 07:54:47 UTC 2013 [07:55:02] RECOVERY - Puppet freshness on lvs2 is OK: puppet ran at Mon Sep 16 07:54:52 UTC 2013 [07:55:02] RECOVERY - Puppet freshness on ms-fe1004 is OK: puppet ran at Mon Sep 16 07:54:53 UTC 2013 [07:55:02] RECOVERY - Puppet freshness on snapshot3 is OK: puppet ran at Mon Sep 16 07:54:53 UTC 2013 [07:55:02] RECOVERY - Puppet freshness on db1049 is OK: puppet ran at Mon Sep 16 07:54:58 UTC 2013 [07:55:03] RECOVERY - Puppet freshness on srv274 is OK: puppet ran at Mon Sep 16 07:54:58 UTC 2013 [07:55:12] RECOVERY - Puppet freshness on srv235 is OK: puppet ran at Mon Sep 16 07:55:03 UTC 2013 [07:55:12] RECOVERY - Puppet freshness on mw12 is OK: puppet ran at Mon Sep 16 07:55:03 UTC 2013 [07:55:13] RECOVERY - Puppet freshness on cp1064 is OK: puppet ran at Mon Sep 16 07:55:03 UTC 2013 [07:55:13] RECOVERY - Puppet freshness on srv257 is OK: puppet ran at Mon Sep 16 07:55:03 UTC 2013 [07:55:13] RECOVERY - Puppet freshness on mw56 is OK: puppet ran at Mon Sep 16 07:55:03 UTC 2013 [07:55:13] RECOVERY - Puppet freshness on holmium is OK: puppet ran at Mon Sep 16 07:55:03 UTC 2013 [07:55:13] RECOVERY - Puppet freshness on mw1170 is OK: puppet ran at Mon Sep 16 07:55:08 UTC 2013 [07:55:13] RECOVERY - Puppet freshness on mw1085 is OK: puppet ran at Mon Sep 16 07:55:08 UTC 2013 [07:56:03] RECOVERY - Puppet freshness on mw1102 is OK: puppet ran at Mon Sep 16 07:55:53 UTC 2013 [07:56:05] RECOVERY - Puppet freshness on mw1070 is OK: puppet ran at Mon Sep 16 07:55:53 UTC 2013 [07:56:05] RECOVERY - Puppet freshness on mw1061 is OK: puppet ran at Mon Sep 16 07:55:53 UTC 2013 [07:56:08] RECOVERY - Puppet freshness on mw1119 is OK: puppet ran at Mon Sep 16 07:55:53 UTC 2013 [07:56:08] RECOVERY - Puppet freshness on virt0 is OK: puppet ran at Mon Sep 16 07:55:53 UTC 2013 [07:56:08] RECOVERY - Puppet freshness on mw1019 is OK: puppet ran at Mon Sep 16 07:55:53 UTC 2013 [07:56:08] RECOVERY - Puppet freshness on ersch is OK: puppet ran at Mon Sep 16 07:55:53 UTC 2013 [07:56:08] RECOVERY - Puppet freshness on ssl1004 is OK: puppet ran at Mon Sep 16 07:55:53 UTC 2013 [07:56:08] RECOVERY - Puppet freshness on sq56 is OK: puppet ran at Mon Sep 16 07:55:53 UTC 2013 [07:56:09] RECOVERY - Puppet freshness on amssq41 is OK: puppet ran at Mon Sep 16 07:55:58 UTC 2013 [07:56:09] RECOVERY - Puppet freshness on wtp1014 is OK: puppet ran at Mon Sep 16 07:55:58 UTC 2013 [07:56:10] RECOVERY - Puppet freshness on amssq60 is OK: puppet ran at Mon Sep 16 07:55:58 UTC 2013 [07:56:10] RECOVERY - Puppet freshness on mw1058 is OK: puppet ran at Mon Sep 16 07:55:58 UTC 2013 [07:56:11] RECOVERY - Puppet freshness on analytics1024 is OK: puppet ran at Mon Sep 16 07:55:58 UTC 2013 [07:56:11] RECOVERY - Puppet freshness on db1028 is OK: puppet ran at Mon Sep 16 07:55:58 UTC 2013 [07:56:12] RECOVERY - Puppet freshness on wtp1010 is OK: puppet ran at Mon Sep 16 07:55:58 UTC 2013 [07:56:12] RECOVERY - Puppet freshness on analytics1006 is OK: puppet ran at Mon Sep 16 07:55:58 UTC 2013 [07:56:13] RECOVERY - Puppet freshness on mw120 is OK: puppet ran at Mon Sep 16 07:55:58 UTC 2013 [07:56:13] RECOVERY - Puppet freshness on mw96 is OK: puppet ran at Mon Sep 16 07:55:58 UTC 2013 [07:56:14] RECOVERY - Puppet freshness on srv239 is OK: puppet ran at Mon Sep 16 07:56:03 UTC 2013 [07:56:22] RECOVERY - Puppet freshness on amssq36 is OK: puppet ran at Mon Sep 16 07:56:13 UTC 2013 [07:56:22] RECOVERY - Puppet freshness on srv252 is OK: puppet ran at Mon Sep 16 07:56:14 UTC 2013 [07:56:22] RECOVERY - Puppet freshness on search20 is OK: puppet ran at Mon Sep 16 07:56:14 UTC 2013 [07:56:22] RECOVERY - Puppet freshness on mw78 is OK: puppet ran at Mon Sep 16 07:56:14 UTC 2013 [07:56:22] RECOVERY - Puppet freshness on srv298 is OK: puppet ran at Mon Sep 16 07:56:14 UTC 2013 [07:56:22] RECOVERY - Puppet freshness on gallium is OK: puppet ran at Mon Sep 16 07:56:14 UTC 2013 [07:56:22] RECOVERY - Puppet freshness on mw1184 is OK: puppet ran at Mon Sep 16 07:56:15 UTC 2013 [07:56:23] RECOVERY - Puppet freshness on mw1172 is OK: puppet ran at Mon Sep 16 07:56:15 UTC 2013 [07:56:23] RECOVERY - Puppet freshness on virt12 is OK: puppet ran at Mon Sep 16 07:56:15 UTC 2013 [07:56:24] RECOVERY - Puppet freshness on mw1191 is OK: puppet ran at Mon Sep 16 07:56:15 UTC 2013 [07:56:24] RECOVERY - Puppet freshness on mw1177 is OK: puppet ran at Mon Sep 16 07:56:15 UTC 2013 [07:56:25] RECOVERY - Puppet freshness on mw1039 is OK: puppet ran at Mon Sep 16 07:56:15 UTC 2013 [07:56:25] RECOVERY - Puppet freshness on search1013 is OK: puppet ran at Mon Sep 16 07:56:15 UTC 2013 [07:56:26] RECOVERY - Puppet freshness on srv247 is OK: puppet ran at Mon Sep 16 07:56:20 UTC 2013 [07:56:26] RECOVERY - Puppet freshness on mw67 is OK: puppet ran at Mon Sep 16 07:56:20 UTC 2013 [07:56:27] RECOVERY - Puppet freshness on cp1057 is OK: puppet ran at Mon Sep 16 07:56:20 UTC 2013 [07:56:27] RECOVERY - Puppet freshness on ms-be1005 is OK: puppet ran at Mon Sep 16 07:56:20 UTC 2013 [07:56:28] RECOVERY - Puppet freshness on db38 is OK: puppet ran at Mon Sep 16 07:56:20 UTC 2013 [07:56:32] RECOVERY - Puppet freshness on sq61 is OK: puppet ran at Mon Sep 16 07:56:25 UTC 2013 [07:56:32] RECOVERY - Puppet freshness on ms-be7 is OK: puppet ran at Mon Sep 16 07:56:25 UTC 2013 [07:56:32] RECOVERY - Puppet freshness on search1009 is OK: puppet ran at Mon Sep 16 07:56:25 UTC 2013 [07:56:42] RECOVERY - Puppet freshness on db1041 is OK: puppet ran at Mon Sep 16 07:56:35 UTC 2013 [07:56:42] RECOVERY - Puppet freshness on nfs2 is OK: puppet ran at Mon Sep 16 07:56:35 UTC 2013 [07:56:43] RECOVERY - Puppet freshness on amssq58 is OK: puppet ran at Mon Sep 16 07:56:35 UTC 2013 [07:56:43] RECOVERY - Puppet freshness on tmh1002 is OK: puppet ran at Mon Sep 16 07:56:35 UTC 2013 [07:56:43] RECOVERY - Puppet freshness on cp1019 is OK: puppet ran at Mon Sep 16 07:56:41 UTC 2013 [07:56:43] RECOVERY - Puppet freshness on hooper is OK: puppet ran at Mon Sep 16 07:56:41 UTC 2013 [07:56:52] RECOVERY - Puppet freshness on db1003 is OK: puppet ran at Mon Sep 16 07:56:46 UTC 2013 [07:56:52] RECOVERY - Puppet freshness on db37 is OK: puppet ran at Mon Sep 16 07:56:46 UTC 2013 [07:56:52] RECOVERY - Puppet freshness on db33 is OK: puppet ran at Mon Sep 16 07:56:51 UTC 2013 [07:57:02] RECOVERY - Puppet freshness on virt10 is OK: puppet ran at Mon Sep 16 07:56:56 UTC 2013 [07:57:02] RECOVERY - Puppet freshness on wtp1012 is OK: puppet ran at Mon Sep 16 07:56:56 UTC 2013 [07:57:03] RECOVERY - Puppet freshness on pdf2 is OK: puppet ran at Mon Sep 16 07:57:01 UTC 2013 [07:57:03] RECOVERY - Puppet freshness on db1043 is OK: puppet ran at Mon Sep 16 07:57:01 UTC 2013 [07:57:03] RECOVERY - Puppet freshness on silver is OK: puppet ran at Mon Sep 16 07:57:01 UTC 2013 [07:57:03] RECOVERY - Puppet freshness on mw84 is OK: puppet ran at Mon Sep 16 07:57:01 UTC 2013 [07:57:12] RECOVERY - Puppet freshness on mw63 is OK: puppet ran at Mon Sep 16 07:57:11 UTC 2013 [07:57:23] RECOVERY - NTP on neon is OK: NTP OK: Offset -0.0008730888367 secs [07:58:02] RECOVERY - Puppet freshness on mw71 is OK: puppet ran at Mon Sep 16 07:57:56 UTC 2013 [07:58:12] RECOVERY - Puppet freshness on wtp1005 is OK: puppet ran at Mon Sep 16 07:58:06 UTC 2013 [07:58:12] RECOVERY - Puppet freshness on mw1055 is OK: puppet ran at Mon Sep 16 07:58:06 UTC 2013 [07:58:12] RECOVERY - Puppet freshness on ms-fe4 is OK: puppet ran at Mon Sep 16 07:58:06 UTC 2013 [07:58:12] RECOVERY - Puppet freshness on mw1076 is OK: puppet ran at Mon Sep 16 07:58:06 UTC 2013 [07:58:12] RECOVERY - Puppet freshness on mw69 is OK: puppet ran at Mon Sep 16 07:58:06 UTC 2013 [07:58:13] RECOVERY - Puppet freshness on mw1013 is OK: puppet ran at Mon Sep 16 07:58:06 UTC 2013 [07:58:13] RECOVERY - Puppet freshness on mw1167 is OK: puppet ran at Mon Sep 16 07:58:06 UTC 2013 [07:58:14] RECOVERY - Puppet freshness on es7 is OK: puppet ran at Mon Sep 16 07:58:06 UTC 2013 [07:58:14] RECOVERY - Puppet freshness on mw1035 is OK: puppet ran at Mon Sep 16 07:58:06 UTC 2013 [07:58:15] RECOVERY - Puppet freshness on nfs1 is OK: puppet ran at Mon Sep 16 07:58:06 UTC 2013 [07:58:15] RECOVERY - Puppet freshness on db50 is OK: puppet ran at Mon Sep 16 07:58:06 UTC 2013 [07:58:16] RECOVERY - Puppet freshness on analytics1016 is OK: puppet ran at Mon Sep 16 07:58:06 UTC 2013 [07:58:16] RECOVERY - Puppet freshness on db1007 is OK: puppet ran at Mon Sep 16 07:58:06 UTC 2013 [07:58:17] RECOVERY - Puppet freshness on es3 is OK: puppet ran at Mon Sep 16 07:58:06 UTC 2013 [07:58:17] RECOVERY - Puppet freshness on db1039 is OK: puppet ran at Mon Sep 16 07:58:06 UTC 2013 [07:58:18] RECOVERY - Puppet freshness on sq53 is OK: puppet ran at Mon Sep 16 07:58:06 UTC 2013 [07:58:22] RECOVERY - Puppet freshness on analytics1026 is OK: puppet ran at Mon Sep 16 07:58:12 UTC 2013 [07:58:22] RECOVERY - Puppet freshness on sq69 is OK: puppet ran at Mon Sep 16 07:58:12 UTC 2013 [07:58:22] RECOVERY - Puppet freshness on mw1195 is OK: puppet ran at Mon Sep 16 07:58:12 UTC 2013 [07:58:22] RECOVERY - Puppet freshness on es1006 is OK: puppet ran at Mon Sep 16 07:58:12 UTC 2013 [07:58:22] RECOVERY - Puppet freshness on ssl1008 is OK: puppet ran at Mon Sep 16 07:58:12 UTC 2013 [07:58:22] RECOVERY - Puppet freshness on sq44 is OK: puppet ran at Mon Sep 16 07:58:12 UTC 2013 [07:58:22] RECOVERY - Puppet freshness on mc1010 is OK: puppet ran at Mon Sep 16 07:58:12 UTC 2013 [07:58:23] RECOVERY - Puppet freshness on mw1036 is OK: puppet ran at Mon Sep 16 07:58:12 UTC 2013 [07:58:23] RECOVERY - Puppet freshness on search26 is OK: puppet ran at Mon Sep 16 07:58:12 UTC 2013 [07:58:24] RECOVERY - Puppet freshness on mw18 is OK: puppet ran at Mon Sep 16 07:58:12 UTC 2013 [07:58:25] RECOVERY - Puppet freshness on mw88 is OK: puppet ran at Mon Sep 16 07:58:12 UTC 2013 [07:58:25] RECOVERY - Puppet freshness on mw86 is OK: puppet ran at Mon Sep 16 07:58:12 UTC 2013 [07:58:25] RECOVERY - Puppet freshness on mw1149 is OK: puppet ran at Mon Sep 16 07:58:12 UTC 2013 [07:58:26] RECOVERY - Puppet freshness on srv254 is OK: puppet ran at Mon Sep 16 07:58:12 UTC 2013 [07:58:26] RECOVERY - Puppet freshness on snapshot1001 is OK: puppet ran at Mon Sep 16 07:58:12 UTC 2013 [07:58:27] RECOVERY - Puppet freshness on ytterbium is OK: puppet ran at Mon Sep 16 07:58:12 UTC 2013 [07:58:27] RECOVERY - Puppet freshness on mw80 is OK: puppet ran at Mon Sep 16 07:58:12 UTC 2013 [07:58:52] RECOVERY - Puppet freshness on mw1109 is OK: puppet ran at Mon Sep 16 07:58:47 UTC 2013 [07:58:53] RECOVERY - Puppet freshness on db49 is OK: puppet ran at Mon Sep 16 07:58:47 UTC 2013 [07:58:54] RECOVERY - Puppet freshness on search1002 is OK: puppet ran at Mon Sep 16 07:58:47 UTC 2013 [07:58:54] RECOVERY - Puppet freshness on db74 is OK: puppet ran at Mon Sep 16 07:58:47 UTC 2013 [07:58:54] RECOVERY - Puppet freshness on mc1014 is OK: puppet ran at Mon Sep 16 07:58:47 UTC 2013 [07:58:54] RECOVERY - Puppet freshness on mw1216 is OK: puppet ran at Mon Sep 16 07:58:47 UTC 2013 [07:58:54] RECOVERY - Puppet freshness on cp1004 is OK: puppet ran at Mon Sep 16 07:58:47 UTC 2013 [07:58:54] RECOVERY - Puppet freshness on formey is OK: puppet ran at Mon Sep 16 07:58:47 UTC 2013 [07:58:55] RECOVERY - Puppet freshness on ssl3003 is OK: puppet ran at Mon Sep 16 07:58:47 UTC 2013 [07:58:55] RECOVERY - Puppet freshness on mw1168 is OK: puppet ran at Mon Sep 16 07:58:47 UTC 2013 [07:59:02] RECOVERY - Puppet freshness on mw1051 is OK: puppet ran at Mon Sep 16 07:58:53 UTC 2013 [07:59:02] RECOVERY - Puppet freshness on mw1133 is OK: puppet ran at Mon Sep 16 07:58:53 UTC 2013 [07:59:02] RECOVERY - Puppet freshness on cp3022 is OK: puppet ran at Mon Sep 16 07:58:53 UTC 2013 [07:59:02] RECOVERY - Puppet freshness on mw48 is OK: puppet ran at Mon Sep 16 07:58:53 UTC 2013 [07:59:03] RECOVERY - Puppet freshness on mw1157 is OK: puppet ran at Mon Sep 16 07:58:58 UTC 2013 [07:59:03] RECOVERY - Puppet freshness on iron is OK: puppet ran at Mon Sep 16 07:58:58 UTC 2013 [07:59:03] RECOVERY - Puppet freshness on rdb1001 is OK: puppet ran at Mon Sep 16 07:58:58 UTC 2013 [07:59:12] RECOVERY - Puppet freshness on sq70 is OK: puppet ran at Mon Sep 16 07:59:03 UTC 2013 [07:59:12] RECOVERY - Puppet freshness on mw32 is OK: puppet ran at Mon Sep 16 07:59:03 UTC 2013 [07:59:13] RECOVERY - Puppet freshness on srv263 is OK: puppet ran at Mon Sep 16 07:59:08 UTC 2013 [07:59:13] RECOVERY - Puppet freshness on cp1048 is OK: puppet ran at Mon Sep 16 07:59:08 UTC 2013 [07:59:13] RECOVERY - Puppet freshness on ssl3002 is OK: puppet ran at Mon Sep 16 07:59:08 UTC 2013 [07:59:13] RECOVERY - Puppet freshness on mw1079 is OK: puppet ran at Mon Sep 16 07:59:08 UTC 2013 [08:00:12] RECOVERY - Puppet freshness on mw97 is OK: puppet ran at Mon Sep 16 07:59:58 UTC 2013 [08:00:12] RECOVERY - Puppet freshness on ssl1009 is OK: puppet ran at Mon Sep 16 08:00:02 UTC 2013 [08:00:12] RECOVERY - Puppet freshness on ms1002 is OK: puppet ran at Mon Sep 16 08:00:02 UTC 2013 [08:00:12] RECOVERY - Puppet freshness on hooft is OK: puppet ran at Mon Sep 16 08:00:07 UTC 2013 [08:00:12] RECOVERY - Puppet freshness on mw72 is OK: puppet ran at Mon Sep 16 08:00:07 UTC 2013 [08:00:47] there we are [08:00:50] lol [08:02:03] RECOVERY - Puppet freshness on mw1029 is OK: puppet ran at Mon Sep 16 08:02:01 UTC 2013 [08:02:04] RECOVERY - Puppet freshness on ms-be6 is OK: puppet ran at Mon Sep 16 08:02:01 UTC 2013 [08:02:04] RECOVERY - Puppet freshness on mw1096 is OK: puppet ran at Mon Sep 16 08:02:01 UTC 2013 [08:02:04] RECOVERY - Puppet freshness on kaulen is OK: puppet ran at Mon Sep 16 08:02:01 UTC 2013 [08:02:04] RECOVERY - Puppet freshness on es1001 is OK: puppet ran at Mon Sep 16 08:02:01 UTC 2013 [08:02:32] RECOVERY - Puppet freshness on es1007 is OK: puppet ran at Mon Sep 16 08:02:21 UTC 2013 [08:02:52] RECOVERY - Puppet freshness on srv236 is OK: puppet ran at Mon Sep 16 08:02:47 UTC 2013 [08:02:52] RECOVERY - Puppet freshness on cp1040 is OK: puppet ran at Mon Sep 16 08:02:47 UTC 2013 [08:02:52] RECOVERY - Puppet freshness on srv248 is OK: puppet ran at Mon Sep 16 08:02:47 UTC 2013 [08:02:52] RECOVERY - Puppet freshness on mw73 is OK: puppet ran at Mon Sep 16 08:02:47 UTC 2013 [08:02:52] RECOVERY - Puppet freshness on yvon is OK: puppet ran at Mon Sep 16 08:02:47 UTC 2013 [08:02:52] RECOVERY - Puppet freshness on tridge is OK: puppet ran at Mon Sep 16 08:02:47 UTC 2013 [08:02:52] RECOVERY - Puppet freshness on amssq39 is OK: puppet ran at Mon Sep 16 08:02:47 UTC 2013 [08:02:53] RECOVERY - Puppet freshness on search34 is OK: puppet ran at Mon Sep 16 08:02:47 UTC 2013 [08:03:03] RECOVERY - Puppet freshness on search15 is OK: puppet ran at Mon Sep 16 08:02:52 UTC 2013 [08:03:03] RECOVERY - Puppet freshness on amssq33 is OK: puppet ran at Mon Sep 16 08:02:52 UTC 2013 [08:03:03] RECOVERY - Puppet freshness on mw1193 is OK: puppet ran at Mon Sep 16 08:02:52 UTC 2013 [08:03:03] RECOVERY - Puppet freshness on amssq45 is OK: puppet ran at Mon Sep 16 08:02:52 UTC 2013 [08:03:03] RECOVERY - Puppet freshness on mw31 is OK: puppet ran at Mon Sep 16 08:02:52 UTC 2013 [08:03:04] RECOVERY - Puppet freshness on srv265 is OK: puppet ran at Mon Sep 16 08:02:52 UTC 2013 [08:03:04] RECOVERY - Puppet freshness on eeden is OK: puppet ran at Mon Sep 16 08:02:52 UTC 2013 [08:03:05] RECOVERY - Puppet freshness on mw22 is OK: puppet ran at Mon Sep 16 08:02:52 UTC 2013 [08:03:05] RECOVERY - Puppet freshness on mw1064 is OK: puppet ran at Mon Sep 16 08:02:52 UTC 2013 [08:03:06] RECOVERY - Puppet freshness on mw19 is OK: puppet ran at Mon Sep 16 08:02:52 UTC 2013 [08:03:06] RECOVERY - Puppet freshness on mw11 is OK: puppet ran at Mon Sep 16 08:02:52 UTC 2013 [08:03:07] RECOVERY - Puppet freshness on mw1012 is OK: puppet ran at Mon Sep 16 08:02:52 UTC 2013 [08:03:07] RECOVERY - Puppet freshness on amssq61 is OK: puppet ran at Mon Sep 16 08:02:52 UTC 2013 [08:03:08] RECOVERY - Puppet freshness on mexia is OK: puppet ran at Mon Sep 16 08:02:52 UTC 2013 [08:03:08] RECOVERY - Puppet freshness on cp3005 is OK: puppet ran at Mon Sep 16 08:02:52 UTC 2013 [08:03:09] RECOVERY - Puppet freshness on db1037 is OK: puppet ran at Mon Sep 16 08:02:52 UTC 2013 [08:03:09] RECOVERY - Puppet freshness on cp3004 is OK: puppet ran at Mon Sep 16 08:02:52 UTC 2013 [08:03:13] RECOVERY - Puppet freshness on analytics1025 is OK: puppet ran at Mon Sep 16 08:03:02 UTC 2013 [08:03:13] RECOVERY - Puppet freshness on analytics1005 is OK: puppet ran at Mon Sep 16 08:03:07 UTC 2013 [08:03:22] RECOVERY - Puppet freshness on mw108 is OK: puppet ran at Mon Sep 16 08:03:13 UTC 2013 [08:03:22] RECOVERY - Puppet freshness on mw77 is OK: puppet ran at Mon Sep 16 08:03:19 UTC 2013 [08:03:23] RECOVERY - Puppet freshness on mw94 is OK: puppet ran at Mon Sep 16 08:03:19 UTC 2013 [08:03:23] RECOVERY - Puppet freshness on mw54 is OK: puppet ran at Mon Sep 16 08:03:19 UTC 2013 [08:03:23] RECOVERY - Puppet freshness on snapshot1003 is OK: puppet ran at Mon Sep 16 08:03:19 UTC 2013 [08:03:23] RECOVERY - Puppet freshness on srv295 is OK: puppet ran at Mon Sep 16 08:03:19 UTC 2013 [08:03:33] RECOVERY - Puppet freshness on mw101 is OK: puppet ran at Mon Sep 16 08:03:24 UTC 2013 [08:03:34] RECOVERY - Puppet freshness on virt9 is OK: puppet ran at Mon Sep 16 08:03:25 UTC 2013 [08:03:34] RECOVERY - Puppet freshness on mw24 is OK: puppet ran at Mon Sep 16 08:03:28 UTC 2013 [08:03:34] RECOVERY - Puppet freshness on mw1187 is OK: puppet ran at Mon Sep 16 08:03:28 UTC 2013 [08:03:34] RECOVERY - Puppet freshness on db1035 is OK: puppet ran at Mon Sep 16 08:03:28 UTC 2013 [08:03:42] RECOVERY - Puppet freshness on mw1207 is OK: puppet ran at Mon Sep 16 08:03:33 UTC 2013 [08:04:02] RECOVERY - Puppet freshness on mw1037 is OK: puppet ran at Mon Sep 16 08:03:53 UTC 2013 [08:04:02] RECOVERY - Puppet freshness on mw1160 is OK: puppet ran at Mon Sep 16 08:03:53 UTC 2013 [08:04:02] RECOVERY - Puppet freshness on maerlant is OK: puppet ran at Mon Sep 16 08:03:54 UTC 2013 [08:04:02] RECOVERY - Puppet freshness on sq67 is OK: puppet ran at Mon Sep 16 08:03:54 UTC 2013 [08:04:02] RECOVERY - Puppet freshness on lvs3 is OK: puppet ran at Mon Sep 16 08:03:54 UTC 2013 [08:04:13] RECOVERY - Puppet freshness on cp1047 is OK: puppet ran at Mon Sep 16 08:03:59 UTC 2013 [08:04:14] RECOVERY - Puppet freshness on mw1113 is OK: puppet ran at Mon Sep 16 08:03:59 UTC 2013 [08:04:14] RECOVERY - Puppet freshness on db52 is OK: puppet ran at Mon Sep 16 08:04:04 UTC 2013 [08:04:14] RECOVERY - Puppet freshness on cp3003 is OK: puppet ran at Mon Sep 16 08:04:04 UTC 2013 [08:04:14] RECOVERY - Puppet freshness on lvs4 is OK: puppet ran at Mon Sep 16 08:04:04 UTC 2013 [08:04:14] RECOVERY - Puppet freshness on mw1071 is OK: puppet ran at Mon Sep 16 08:04:04 UTC 2013 [08:04:14] RECOVERY - Puppet freshness on pc2 is OK: puppet ran at Mon Sep 16 08:04:04 UTC 2013 [08:04:15] RECOVERY - Puppet freshness on mw1006 is OK: puppet ran at Mon Sep 16 08:04:04 UTC 2013 [08:04:15] RECOVERY - Puppet freshness on xenon is OK: puppet ran at Mon Sep 16 08:04:04 UTC 2013 [08:04:16] RECOVERY - Puppet freshness on mw29 is OK: puppet ran at Mon Sep 16 08:04:04 UTC 2013 [08:04:16] RECOVERY - Puppet freshness on search28 is OK: puppet ran at Mon Sep 16 08:04:09 UTC 2013 [08:04:52] RECOVERY - Puppet freshness on srv287 is OK: puppet ran at Mon Sep 16 08:04:49 UTC 2013 [08:04:52] RECOVERY - Puppet freshness on tmh2 is OK: puppet ran at Mon Sep 16 08:04:49 UTC 2013 [08:04:52] RECOVERY - Puppet freshness on mw16 is OK: puppet ran at Mon Sep 16 08:04:49 UTC 2013 [08:04:52] RECOVERY - Puppet freshness on mw17 is OK: puppet ran at Mon Sep 16 08:04:49 UTC 2013 [08:05:02] RECOVERY - Puppet freshness on cp1007 is OK: puppet ran at Mon Sep 16 08:04:54 UTC 2013 [08:05:02] RECOVERY - Puppet freshness on amssq52 is OK: puppet ran at Mon Sep 16 08:04:54 UTC 2013 [08:05:02] RECOVERY - Puppet freshness on db40 is OK: puppet ran at Mon Sep 16 08:04:54 UTC 2013 [08:05:02] RECOVERY - Puppet freshness on search13 is OK: puppet ran at Mon Sep 16 08:04:54 UTC 2013 [08:05:02] RECOVERY - Puppet freshness on mc1015 is OK: puppet ran at Mon Sep 16 08:04:54 UTC 2013 [08:05:02] RECOVERY - Puppet freshness on mw1217 is OK: puppet ran at Mon Sep 16 08:04:54 UTC 2013 [08:05:02] RECOVERY - Puppet freshness on sq86 is OK: puppet ran at Mon Sep 16 08:04:54 UTC 2013 [08:05:03] RECOVERY - Puppet freshness on mw1100 is OK: puppet ran at Mon Sep 16 08:04:56 UTC 2013 [08:05:03] RECOVERY - Puppet freshness on mw1088 is OK: puppet ran at Mon Sep 16 08:04:56 UTC 2013 [08:05:04] RECOVERY - Puppet freshness on mw1117 is OK: puppet ran at Mon Sep 16 08:04:56 UTC 2013 [08:05:04] RECOVERY - Puppet freshness on mw1176 is OK: puppet ran at Mon Sep 16 08:04:55 UTC 2013 [08:05:05] RECOVERY - Puppet freshness on mw51 is OK: puppet ran at Mon Sep 16 08:04:56 UTC 2013 [08:05:05] RECOVERY - Puppet freshness on sq71 is OK: puppet ran at Mon Sep 16 08:04:56 UTC 2013 [08:05:06] RECOVERY - Puppet freshness on es1 is OK: puppet ran at Mon Sep 16 08:04:56 UTC 2013 [08:05:06] RECOVERY - Puppet freshness on mw1018 is OK: puppet ran at Mon Sep 16 08:04:56 UTC 2013 [08:05:13] RECOVERY - Puppet freshness on mw1202 is OK: puppet ran at Mon Sep 16 08:05:01 UTC 2013 [08:05:13] RECOVERY - Puppet freshness on labsdb1002 is OK: puppet ran at Mon Sep 16 08:05:02 UTC 2013 [08:05:13] RECOVERY - Puppet freshness on srv300 is OK: puppet ran at Mon Sep 16 08:05:07 UTC 2013 [08:05:25] RECOVERY - Puppet freshness on db1054 is OK: puppet ran at Mon Sep 16 08:05:12 UTC 2013 [08:05:25] RECOVERY - Puppet freshness on cp1056 is OK: puppet ran at Mon Sep 16 08:05:13 UTC 2013 [08:05:25] RECOVERY - Puppet freshness on mw1164 is OK: puppet ran at Mon Sep 16 08:05:13 UTC 2013 [08:05:32] RECOVERY - Puppet freshness on mw1103 is OK: puppet ran at Mon Sep 16 08:05:28 UTC 2013 [08:05:42] RECOVERY - Puppet freshness on lanthanum is OK: puppet ran at Mon Sep 16 08:05:38 UTC 2013 [08:05:42] RECOVERY - Puppet freshness on ms-be8 is OK: puppet ran at Mon Sep 16 08:05:38 UTC 2013 [08:05:42] RECOVERY - Puppet freshness on williams is OK: puppet ran at Mon Sep 16 08:05:38 UTC 2013 [08:05:52] RECOVERY - Puppet freshness on search17 is OK: puppet ran at Mon Sep 16 08:05:45 UTC 2013 [08:05:52] RECOVERY - Puppet freshness on search1018 is OK: puppet ran at Mon Sep 16 08:05:46 UTC 2013 [08:05:52] RECOVERY - Puppet freshness on srv297 is OK: puppet ran at Mon Sep 16 08:05:46 UTC 2013 [08:05:52] RECOVERY - Puppet freshness on mw20 is OK: puppet ran at Mon Sep 16 08:05:46 UTC 2013 [08:05:52] RECOVERY - Puppet freshness on mw34 is OK: puppet ran at Mon Sep 16 08:05:46 UTC 2013 [08:05:52] RECOVERY - Puppet freshness on mw1095 is OK: puppet ran at Mon Sep 16 08:05:46 UTC 2013 [08:05:52] RECOVERY - Puppet freshness on amssq55 is OK: puppet ran at Mon Sep 16 08:05:46 UTC 2013 [08:05:53] RECOVERY - Puppet freshness on mw1015 is OK: puppet ran at Mon Sep 16 08:05:46 UTC 2013 [08:05:53] RECOVERY - Puppet freshness on cp3007 is OK: puppet ran at Mon Sep 16 08:05:46 UTC 2013 [08:05:54] RECOVERY - Puppet freshness on mw122 is OK: puppet ran at Mon Sep 16 08:05:46 UTC 2013 [08:05:54] RECOVERY - Puppet freshness on pdf1 is OK: puppet ran at Mon Sep 16 08:05:46 UTC 2013 [08:05:55] RECOVERY - Puppet freshness on mw1017 is OK: puppet ran at Mon Sep 16 08:05:46 UTC 2013 [08:05:55] RECOVERY - Puppet freshness on search1010 is OK: puppet ran at Mon Sep 16 08:05:46 UTC 2013 [08:05:56] RECOVERY - Puppet freshness on analytics1012 is OK: puppet ran at Mon Sep 16 08:05:46 UTC 2013 [08:05:56] RECOVERY - Puppet freshness on sq80 is OK: puppet ran at Mon Sep 16 08:05:46 UTC 2013 [08:05:57] RECOVERY - Puppet freshness on search24 is OK: puppet ran at Mon Sep 16 08:05:46 UTC 2013 [08:05:57] RECOVERY - Puppet freshness on mw1151 is OK: puppet ran at Mon Sep 16 08:05:46 UTC 2013 [08:05:58] RECOVERY - Puppet freshness on mw1014 is OK: puppet ran at Mon Sep 16 08:05:46 UTC 2013 [08:05:58] RECOVERY - Puppet freshness on db1019 is OK: puppet ran at Mon Sep 16 08:05:51 UTC 2013 [08:05:59] RECOVERY - Puppet freshness on cp1012 is OK: puppet ran at Mon Sep 16 08:05:51 UTC 2013 [08:05:59] RECOVERY - Puppet freshness on es1003 is OK: puppet ran at Mon Sep 16 08:05:51 UTC 2013 [08:06:03] RECOVERY - Puppet freshness on db1023 is OK: puppet ran at Mon Sep 16 08:05:56 UTC 2013 [08:06:04] RECOVERY - Puppet freshness on labsdb1003 is OK: puppet ran at Mon Sep 16 08:05:57 UTC 2013 [08:06:12] RECOVERY - Puppet freshness on mw1192 is OK: puppet ran at Mon Sep 16 08:06:02 UTC 2013 [08:06:43] RECOVERY - Puppet freshness on cp1061 is OK: puppet ran at Mon Sep 16 08:06:37 UTC 2013 [08:06:43] RECOVERY - Puppet freshness on cp1020 is OK: puppet ran at Mon Sep 16 08:06:37 UTC 2013 [08:06:43] RECOVERY - Puppet freshness on ms-be1 is OK: puppet ran at Mon Sep 16 08:06:40 UTC 2013 [08:06:43] RECOVERY - Puppet freshness on analytics1010 is OK: puppet ran at Mon Sep 16 08:06:40 UTC 2013 [08:06:43] RECOVERY - Puppet freshness on search22 is OK: puppet ran at Mon Sep 16 08:06:40 UTC 2013 [08:06:44] RECOVERY - Puppet freshness on search1019 is OK: puppet ran at Mon Sep 16 08:06:40 UTC 2013 [08:06:44] RECOVERY - Puppet freshness on mw66 is OK: puppet ran at Mon Sep 16 08:06:40 UTC 2013 [08:06:45] RECOVERY - Puppet freshness on mw1175 is OK: puppet ran at Mon Sep 16 08:06:40 UTC 2013 [08:06:45] RECOVERY - Puppet freshness on mw14 is OK: puppet ran at Mon Sep 16 08:06:40 UTC 2013 [08:06:46] RECOVERY - Puppet freshness on mw47 is OK: puppet ran at Mon Sep 16 08:06:40 UTC 2013 [08:06:46] RECOVERY - Puppet freshness on mw1214 is OK: puppet ran at Mon Sep 16 08:06:40 UTC 2013 [08:06:47] RECOVERY - Puppet freshness on mw1065 is OK: puppet ran at Mon Sep 16 08:06:40 UTC 2013 [08:06:47] RECOVERY - Puppet freshness on search1008 is OK: puppet ran at Mon Sep 16 08:06:40 UTC 2013 [08:06:48] RECOVERY - Puppet freshness on srv238 is OK: puppet ran at Mon Sep 16 08:06:40 UTC 2013 [08:06:48] RECOVERY - Puppet freshness on srv237 is OK: puppet ran at Mon Sep 16 08:06:40 UTC 2013 [08:06:49] RECOVERY - Puppet freshness on mw125 is OK: puppet ran at Mon Sep 16 08:06:40 UTC 2013 [08:06:49] RECOVERY - Puppet freshness on mw82 is OK: puppet ran at Mon Sep 16 08:06:40 UTC 2013 [08:06:50] RECOVERY - Puppet freshness on srv250 is OK: puppet ran at Mon Sep 16 08:06:40 UTC 2013 [08:06:50] RECOVERY - Puppet freshness on srv286 is OK: puppet ran at Mon Sep 16 08:06:40 UTC 2013 [08:06:51] RECOVERY - Puppet freshness on mw46 is OK: puppet ran at Mon Sep 16 08:06:40 UTC 2013 [08:06:51] RECOVERY - Puppet freshness on es1005 is OK: puppet ran at Mon Sep 16 08:06:40 UTC 2013 [08:06:52] RECOVERY - Puppet freshness on mw1182 is OK: puppet ran at Mon Sep 16 08:06:40 UTC 2013 [08:07:12] RECOVERY - Puppet freshness on mw62 is OK: puppet ran at Mon Sep 16 08:07:00 UTC 2013 [08:07:12] RECOVERY - Puppet freshness on dobson is OK: puppet ran at Mon Sep 16 08:07:02 UTC 2013 [08:07:12] RECOVERY - Puppet freshness on mw1123 is OK: puppet ran at Mon Sep 16 08:07:02 UTC 2013 [08:07:12] RECOVERY - Puppet freshness on mw1101 is OK: puppet ran at Mon Sep 16 08:07:02 UTC 2013 [08:07:12] RECOVERY - Puppet freshness on mw1042 is OK: puppet ran at Mon Sep 16 08:07:02 UTC 2013 [08:07:12] RECOVERY - Puppet freshness on lvs1006 is OK: puppet ran at Mon Sep 16 08:07:02 UTC 2013 [08:07:12] RECOVERY - Puppet freshness on srv258 is OK: puppet ran at Mon Sep 16 08:07:02 UTC 2013 [08:07:13] RECOVERY - Puppet freshness on gurvin is OK: puppet ran at Mon Sep 16 08:07:09 UTC 2013 [08:07:13] RECOVERY - Puppet freshness on srv284 is OK: puppet ran at Mon Sep 16 08:07:09 UTC 2013 [08:07:23] RECOVERY - Puppet freshness on db62 is OK: puppet ran at Mon Sep 16 08:07:14 UTC 2013 [08:07:23] RECOVERY - Puppet freshness on mw1127 is OK: puppet ran at Mon Sep 16 08:07:14 UTC 2013 [08:07:23] RECOVERY - Puppet freshness on sq72 is OK: puppet ran at Mon Sep 16 08:07:14 UTC 2013 [08:07:23] RECOVERY - Puppet freshness on mw49 is OK: puppet ran at Mon Sep 16 08:07:14 UTC 2013 [08:07:23] RECOVERY - Puppet freshness on sq45 is OK: puppet ran at Mon Sep 16 08:07:14 UTC 2013 [08:07:23] RECOVERY - Puppet freshness on amssq34 is OK: puppet ran at Mon Sep 16 08:07:14 UTC 2013 [08:07:23] RECOVERY - Puppet freshness on mw132 is OK: puppet ran at Mon Sep 16 08:07:14 UTC 2013 [08:07:24] RECOVERY - Puppet freshness on amssq57 is OK: puppet ran at Mon Sep 16 08:07:14 UTC 2013 [08:07:24] RECOVERY - Puppet freshness on nescio is OK: puppet ran at Mon Sep 16 08:07:14 UTC 2013 [08:07:25] RECOVERY - Puppet freshness on cp1013 is OK: puppet ran at Mon Sep 16 08:07:19 UTC 2013 [08:07:32] RECOVERY - Puppet freshness on cp1003 is OK: puppet ran at Mon Sep 16 08:07:24 UTC 2013 [08:07:32] RECOVERY - Puppet freshness on es4 is OK: puppet ran at Mon Sep 16 08:07:24 UTC 2013 [08:07:32] RECOVERY - Puppet freshness on es1002 is OK: puppet ran at Mon Sep 16 08:07:24 UTC 2013 [08:07:32] RECOVERY - Puppet freshness on mw9 is OK: puppet ran at Mon Sep 16 08:07:29 UTC 2013 [08:07:32] RECOVERY - Puppet freshness on cp3010 is OK: puppet ran at Mon Sep 16 08:07:29 UTC 2013 [08:07:52] RECOVERY - Puppet freshness on pc1002 is OK: puppet ran at Mon Sep 16 08:07:44 UTC 2013 [08:07:52] RECOVERY - Puppet freshness on cp1069 is OK: puppet ran at Mon Sep 16 08:07:44 UTC 2013 [08:07:52] RECOVERY - Puppet freshness on mw1126 is OK: puppet ran at Mon Sep 16 08:07:44 UTC 2013 [08:07:52] RECOVERY - Puppet freshness on sq79 is OK: puppet ran at Mon Sep 16 08:07:50 UTC 2013 [08:08:03] RECOVERY - Puppet freshness on mw100 is OK: puppet ran at Mon Sep 16 08:07:55 UTC 2013 [08:08:03] RECOVERY - Puppet freshness on cp1006 is OK: puppet ran at Mon Sep 16 08:07:55 UTC 2013 [08:08:12] RECOVERY - Puppet freshness on mw81 is OK: puppet ran at Mon Sep 16 08:08:00 UTC 2013 [08:08:12] RECOVERY - Puppet freshness on mw1196 is OK: puppet ran at Mon Sep 16 08:08:08 UTC 2013 [08:08:13] RECOVERY - Puppet freshness on srv275 is OK: puppet ran at Mon Sep 16 08:08:08 UTC 2013 [08:08:13] RECOVERY - Puppet freshness on mw90 is OK: puppet ran at Mon Sep 16 08:08:08 UTC 2013 [08:08:22] RECOVERY - Puppet freshness on mw41 is OK: puppet ran at Mon Sep 16 08:08:18 UTC 2013 [08:08:22] RECOVERY - Puppet freshness on db53 is OK: puppet ran at Mon Sep 16 08:08:19 UTC 2013 [08:08:22] RECOVERY - Puppet freshness on wtp1024 is OK: puppet ran at Mon Sep 16 08:08:19 UTC 2013 [08:08:22] RECOVERY - Puppet freshness on ms-be4 is OK: puppet ran at Mon Sep 16 08:08:20 UTC 2013 [08:08:23] RECOVERY - Puppet freshness on search14 is OK: puppet ran at Mon Sep 16 08:08:20 UTC 2013 [08:08:23] RECOVERY - Puppet freshness on mc1005 is OK: puppet ran at Mon Sep 16 08:08:20 UTC 2013 [08:08:23] RECOVERY - Puppet freshness on sq73 is OK: puppet ran at Mon Sep 16 08:08:20 UTC 2013 [08:08:24] RECOVERY - Puppet freshness on search21 is OK: puppet ran at Mon Sep 16 08:08:20 UTC 2013 [08:08:24] RECOVERY - Puppet freshness on cp1046 is OK: puppet ran at Mon Sep 16 08:08:20 UTC 2013 [08:08:32] RECOVERY - Puppet freshness on ms5 is OK: puppet ran at Mon Sep 16 08:08:25 UTC 2013 [08:08:32] RECOVERY - Puppet freshness on cp1050 is OK: puppet ran at Mon Sep 16 08:08:25 UTC 2013 [08:08:32] RECOVERY - Puppet freshness on mw1044 is OK: puppet ran at Mon Sep 16 08:08:25 UTC 2013 [08:08:33] RECOVERY - Puppet freshness on search1014 is OK: puppet ran at Mon Sep 16 08:08:30 UTC 2013 [08:08:45] RECOVERY - Puppet freshness on es10 is OK: puppet ran at Mon Sep 16 08:08:35 UTC 2013 [08:08:45] RECOVERY - Puppet freshness on db1052 is OK: puppet ran at Mon Sep 16 08:08:35 UTC 2013 [08:08:45] RECOVERY - Puppet freshness on srv301 is OK: puppet ran at Mon Sep 16 08:08:35 UTC 2013 [08:08:45] RECOVERY - Puppet freshness on gadolinium is OK: puppet ran at Mon Sep 16 08:08:35 UTC 2013 [08:08:45] RECOVERY - Puppet freshness on mw83 is OK: puppet ran at Mon Sep 16 08:08:35 UTC 2013 [08:08:45] RECOVERY - Puppet freshness on mw119 is OK: puppet ran at Mon Sep 16 08:08:35 UTC 2013 [08:08:45] RECOVERY - Puppet freshness on pc1003 is OK: puppet ran at Mon Sep 16 08:08:40 UTC 2013 [08:08:46] RECOVERY - Puppet freshness on mw1161 is OK: puppet ran at Mon Sep 16 08:08:40 UTC 2013 [08:08:46] RECOVERY - Puppet freshness on cp1051 is OK: puppet ran at Mon Sep 16 08:08:40 UTC 2013 [08:08:52] RECOVERY - Puppet freshness on mw118 is OK: puppet ran at Mon Sep 16 08:08:46 UTC 2013 [08:08:52] RECOVERY - Puppet freshness on db1036 is OK: puppet ran at Mon Sep 16 08:08:51 UTC 2013 [08:09:05] RECOVERY - Puppet freshness on sq51 is OK: puppet ran at Mon Sep 16 08:08:56 UTC 2013 [08:09:05] RECOVERY - Puppet freshness on db1048 is OK: puppet ran at Mon Sep 16 08:08:56 UTC 2013 [08:09:05] RECOVERY - Puppet freshness on mw1111 is OK: puppet ran at Mon Sep 16 08:08:56 UTC 2013 [08:09:05] RECOVERY - Puppet freshness on mw1125 is OK: puppet ran at Mon Sep 16 08:08:56 UTC 2013 [08:09:05] RECOVERY - Puppet freshness on mw1130 is OK: puppet ran at Mon Sep 16 08:08:56 UTC 2013 [08:09:05] RECOVERY - Puppet freshness on mw1124 is OK: puppet ran at Mon Sep 16 08:08:56 UTC 2013 [08:09:05] RECOVERY - Puppet freshness on srv277 is OK: puppet ran at Mon Sep 16 08:08:56 UTC 2013 [08:09:06] RECOVERY - Puppet freshness on srv256 is OK: puppet ran at Mon Sep 16 08:08:56 UTC 2013 [08:09:06] RECOVERY - Puppet freshness on ms-fe3001 is OK: puppet ran at Mon Sep 16 08:09:01 UTC 2013 [08:09:07] RECOVERY - Puppet freshness on mw5 is OK: puppet ran at Mon Sep 16 08:09:01 UTC 2013 [08:09:13] RECOVERY - Puppet freshness on rubidium is OK: puppet ran at Mon Sep 16 08:09:08 UTC 2013 [08:09:14] RECOVERY - Puppet freshness on ms-fe3 is OK: puppet ran at Mon Sep 16 08:09:08 UTC 2013 [08:09:14] RECOVERY - Puppet freshness on zirconium is OK: puppet ran at Mon Sep 16 08:09:08 UTC 2013 [08:09:23] RECOVERY - Puppet freshness on db43 is OK: puppet ran at Mon Sep 16 08:09:13 UTC 2013 [08:09:32] RECOVERY - Puppet freshness on mw1162 is OK: puppet ran at Mon Sep 16 08:09:28 UTC 2013 [08:09:33] RECOVERY - Puppet freshness on ms-be2 is OK: puppet ran at Mon Sep 16 08:09:30 UTC 2013 [08:09:33] RECOVERY - Puppet freshness on wtp1003 is OK: puppet ran at Mon Sep 16 08:09:30 UTC 2013 [08:09:33] RECOVERY - Puppet freshness on analytics1023 is OK: puppet ran at Mon Sep 16 08:09:30 UTC 2013 [08:09:33] RECOVERY - Puppet freshness on db1020 is OK: puppet ran at Mon Sep 16 08:09:30 UTC 2013 [08:09:43] RECOVERY - Puppet freshness on wtp1018 is OK: puppet ran at Mon Sep 16 08:09:35 UTC 2013 [08:09:43] RECOVERY - Puppet freshness on vanadium is OK: puppet ran at Mon Sep 16 08:09:35 UTC 2013 [08:09:43] RECOVERY - Puppet freshness on cp1063 is OK: puppet ran at Mon Sep 16 08:09:35 UTC 2013 [08:09:43] RECOVERY - Puppet freshness on wtp1011 is OK: puppet ran at Mon Sep 16 08:09:35 UTC 2013 [08:09:43] RECOVERY - Puppet freshness on fenari is OK: puppet ran at Mon Sep 16 08:09:35 UTC 2013 [08:09:52] RECOVERY - Puppet freshness on sq48 is OK: puppet ran at Mon Sep 16 08:09:46 UTC 2013 [08:09:52] RECOVERY - Puppet freshness on srv249 is OK: puppet ran at Mon Sep 16 08:09:46 UTC 2013 [08:09:52] RECOVERY - Puppet freshness on srv290 is OK: puppet ran at Mon Sep 16 08:09:46 UTC 2013 [08:09:52] RECOVERY - Puppet freshness on analytics1015 is OK: puppet ran at Mon Sep 16 08:09:45 UTC 2013 [08:09:52] RECOVERY - Puppet freshness on search19 is OK: puppet ran at Mon Sep 16 08:09:51 UTC 2013 [08:09:52] RECOVERY - Puppet freshness on mw1190 is OK: puppet ran at Mon Sep 16 08:09:51 UTC 2013 [08:09:52] RECOVERY - Puppet freshness on mw38 is OK: puppet ran at Mon Sep 16 08:09:51 UTC 2013 [08:09:53] RECOVERY - Puppet freshness on srv253 is OK: puppet ran at Mon Sep 16 08:09:51 UTC 2013 [08:09:53] RECOVERY - Puppet freshness on mw1147 is OK: puppet ran at Mon Sep 16 08:09:51 UTC 2013 [08:10:02] RECOVERY - Puppet freshness on srv244 is OK: puppet ran at Mon Sep 16 08:09:56 UTC 2013 [08:10:02] RECOVERY - Puppet freshness on srv289 is OK: puppet ran at Mon Sep 16 08:10:01 UTC 2013 [08:10:02] RECOVERY - Puppet freshness on srv245 is OK: puppet ran at Mon Sep 16 08:10:01 UTC 2013 [08:10:02] RECOVERY - Puppet freshness on mw76 is OK: puppet ran at Mon Sep 16 08:10:01 UTC 2013 [08:10:02] RECOVERY - Puppet freshness on srv246 is OK: puppet ran at Mon Sep 16 08:10:01 UTC 2013 [08:10:02] RECOVERY - Puppet freshness on sq74 is OK: puppet ran at Mon Sep 16 08:10:01 UTC 2013 [08:10:03] RECOVERY - Puppet freshness on db58 is OK: puppet ran at Mon Sep 16 08:10:01 UTC 2013 [08:10:03] RECOVERY - Puppet freshness on sq43 is OK: puppet ran at Mon Sep 16 08:10:01 UTC 2013 [08:10:04] RECOVERY - Puppet freshness on solr3 is OK: puppet ran at Mon Sep 16 08:10:01 UTC 2013 [08:10:04] RECOVERY - Puppet freshness on amssq44 is OK: puppet ran at Mon Sep 16 08:10:01 UTC 2013 [08:10:12] RECOVERY - Puppet freshness on mw1038 is OK: puppet ran at Mon Sep 16 08:10:01 UTC 2013 [08:10:12] RECOVERY - Puppet freshness on mw1081 is OK: puppet ran at Mon Sep 16 08:10:01 UTC 2013 [08:10:13] RECOVERY - Puppet freshness on solr1003 is OK: puppet ran at Mon Sep 16 08:10:01 UTC 2013 [08:10:13] RECOVERY - Puppet freshness on mw1115 is OK: puppet ran at Mon Sep 16 08:10:01 UTC 2013 [08:10:13] RECOVERY - Puppet freshness on mw1146 is OK: puppet ran at Mon Sep 16 08:10:01 UTC 2013 [08:10:13] RECOVERY - Puppet freshness on mw1159 is OK: puppet ran at Mon Sep 16 08:10:09 UTC 2013 [08:10:25] RECOVERY - Puppet freshness on ms-be9 is OK: puppet ran at Mon Sep 16 08:10:20 UTC 2013 [08:10:32] RECOVERY - Puppet freshness on mc1009 is OK: puppet ran at Mon Sep 16 08:10:23 UTC 2013 [08:10:32] RECOVERY - Puppet freshness on mc1001 is OK: puppet ran at Mon Sep 16 08:10:23 UTC 2013 [08:10:32] RECOVERY - Puppet freshness on testsearch1003 is OK: puppet ran at Mon Sep 16 08:10:23 UTC 2013 [08:10:32] RECOVERY - Puppet freshness on mw1048 is OK: puppet ran at Mon Sep 16 08:10:23 UTC 2013 [08:10:32] RECOVERY - Puppet freshness on snapshot1 is OK: puppet ran at Mon Sep 16 08:10:23 UTC 2013 [08:10:32] RECOVERY - Puppet freshness on mw1057 is OK: puppet ran at Mon Sep 16 08:10:24 UTC 2013 [08:10:32] RECOVERY - Puppet freshness on es6 is OK: puppet ran at Mon Sep 16 08:10:24 UTC 2013 [08:10:33] RECOVERY - Puppet freshness on mw1004 is OK: puppet ran at Mon Sep 16 08:10:24 UTC 2013 [08:10:33] RECOVERY - Puppet freshness on cp1049 is OK: puppet ran at Mon Sep 16 08:10:24 UTC 2013 [08:10:34] RECOVERY - Puppet freshness on solr2 is OK: puppet ran at Mon Sep 16 08:10:24 UTC 2013 [08:10:34] RECOVERY - Puppet freshness on wtp1004 is OK: puppet ran at Mon Sep 16 08:10:24 UTC 2013 [08:10:42] RECOVERY - Puppet freshness on ms-be1012 is OK: puppet ran at Mon Sep 16 08:10:34 UTC 2013 [08:10:42] RECOVERY - Puppet freshness on srv294 is OK: puppet ran at Mon Sep 16 08:10:34 UTC 2013 [08:10:42] RECOVERY - Puppet freshness on db1029 is OK: puppet ran at Mon Sep 16 08:10:34 UTC 2013 [08:10:42] RECOVERY - Puppet freshness on srv276 is OK: puppet ran at Mon Sep 16 08:10:34 UTC 2013 [08:10:42] RECOVERY - Puppet freshness on mw8 is OK: puppet ran at Mon Sep 16 08:10:34 UTC 2013 [08:10:42] RECOVERY - Puppet freshness on mw1171 is OK: puppet ran at Mon Sep 16 08:10:40 UTC 2013 [08:10:52] RECOVERY - Puppet freshness on sq85 is OK: puppet ran at Mon Sep 16 08:10:45 UTC 2013 [08:10:53] RECOVERY - Puppet freshness on mw1188 is OK: puppet ran at Mon Sep 16 08:10:49 UTC 2013 [08:10:53] RECOVERY - Puppet freshness on mw30 is OK: puppet ran at Mon Sep 16 08:10:49 UTC 2013 [08:10:53] RECOVERY - Puppet freshness on cp1045 is OK: puppet ran at Mon Sep 16 08:10:49 UTC 2013 [08:11:02] RECOVERY - Puppet freshness on mw1186 is OK: puppet ran at Mon Sep 16 08:10:54 UTC 2013 [08:11:02] RECOVERY - Puppet freshness on mw1087 is OK: puppet ran at Mon Sep 16 08:10:54 UTC 2013 [08:11:02] RECOVERY - Puppet freshness on mw105 is OK: puppet ran at Mon Sep 16 08:10:54 UTC 2013 [08:11:02] RECOVERY - Puppet freshness on titanium is OK: puppet ran at Mon Sep 16 08:10:54 UTC 2013 [08:11:02] RECOVERY - Puppet freshness on search18 is OK: puppet ran at Mon Sep 16 08:10:54 UTC 2013 [08:11:02] RECOVERY - Puppet freshness on mw58 is OK: puppet ran at Mon Sep 16 08:10:54 UTC 2013 [08:11:02] RECOVERY - Puppet freshness on mw103 is OK: puppet ran at Mon Sep 16 08:10:54 UTC 2013 [08:11:03] RECOVERY - Puppet freshness on ekrem is OK: puppet ran at Mon Sep 16 08:10:54 UTC 2013 [08:11:03] RECOVERY - Puppet freshness on mw1063 is OK: puppet ran at Mon Sep 16 08:11:00 UTC 2013 [08:11:04] RECOVERY - Puppet freshness on pc1 is OK: puppet ran at Mon Sep 16 08:11:00 UTC 2013 [08:11:04] RECOVERY - Puppet freshness on mw121 is OK: puppet ran at Mon Sep 16 08:11:00 UTC 2013 [08:11:05] RECOVERY - Puppet freshness on search1024 is OK: puppet ran at Mon Sep 16 08:11:00 UTC 2013 [08:11:05] RECOVERY - Puppet freshness on mw1106 is OK: puppet ran at Mon Sep 16 08:11:00 UTC 2013 [08:11:06] RECOVERY - Puppet freshness on cp1005 is OK: puppet ran at Mon Sep 16 08:11:00 UTC 2013 [08:11:06] RECOVERY - Puppet freshness on mw1197 is OK: puppet ran at Mon Sep 16 08:11:00 UTC 2013 [08:11:07] RECOVERY - Puppet freshness on mw1007 is OK: puppet ran at Mon Sep 16 08:11:00 UTC 2013 [08:11:13] RECOVERY - Puppet freshness on mw1043 is OK: puppet ran at Mon Sep 16 08:11:06 UTC 2013 [08:11:13] RECOVERY - Puppet freshness on praseodymium is OK: puppet ran at Mon Sep 16 08:11:06 UTC 2013 [08:11:34] RECOVERY - Puppet freshness on potassium is OK: puppet ran at Mon Sep 16 08:11:26 UTC 2013 [08:11:35] RECOVERY - Puppet freshness on db39 is OK: puppet ran at Mon Sep 16 08:11:29 UTC 2013 [08:11:42] RECOVERY - Puppet freshness on mw1056 is OK: puppet ran at Mon Sep 16 08:11:34 UTC 2013 [08:11:42] RECOVERY - Puppet freshness on neon is OK: puppet ran at Mon Sep 16 08:11:34 UTC 2013 [08:11:42] RECOVERY - Puppet freshness on db34 is OK: puppet ran at Mon Sep 16 08:11:34 UTC 2013 [08:11:42] RECOVERY - Puppet freshness on srv292 is OK: puppet ran at Mon Sep 16 08:11:34 UTC 2013 [08:11:42] RECOVERY - Puppet freshness on ms-be1006 is OK: puppet ran at Mon Sep 16 08:11:34 UTC 2013 [08:11:42] RECOVERY - Puppet freshness on mw1140 is OK: puppet ran at Mon Sep 16 08:11:34 UTC 2013 [08:11:42] RECOVERY - Puppet freshness on mw124 is OK: puppet ran at Mon Sep 16 08:11:34 UTC 2013 [08:11:43] RECOVERY - Puppet freshness on rdb1002 is OK: puppet ran at Mon Sep 16 08:11:34 UTC 2013 [08:11:43] RECOVERY - Puppet freshness on cp3009 is OK: puppet ran at Mon Sep 16 08:11:34 UTC 2013 [08:11:44] RECOVERY - Puppet freshness on lvs1002 is OK: puppet ran at Mon Sep 16 08:11:34 UTC 2013 [08:11:44] RECOVERY - Puppet freshness on amssq53 is OK: puppet ran at Mon Sep 16 08:11:34 UTC 2013 [08:11:45] RECOVERY - Puppet freshness on cp1002 is OK: puppet ran at Mon Sep 16 08:11:34 UTC 2013 [08:11:45] RECOVERY - Puppet freshness on db1031 is OK: puppet ran at Mon Sep 16 08:11:34 UTC 2013 [08:11:46] RECOVERY - Puppet freshness on helium is OK: puppet ran at Mon Sep 16 08:11:34 UTC 2013 [08:11:46] RECOVERY - Puppet freshness on mc1007 is OK: puppet ran at Mon Sep 16 08:11:34 UTC 2013 [08:11:47] RECOVERY - Puppet freshness on mw1210 is OK: puppet ran at Mon Sep 16 08:11:34 UTC 2013 [08:11:47] RECOVERY - Puppet freshness on ms10 is OK: puppet ran at Mon Sep 16 08:11:34 UTC 2013 [08:11:48] RECOVERY - Puppet freshness on cp3012 is OK: puppet ran at Mon Sep 16 08:11:34 UTC 2013 [08:11:48] RECOVERY - Puppet freshness on srv273 is OK: puppet ran at Mon Sep 16 08:11:34 UTC 2013 [08:11:49] RECOVERY - Puppet freshness on mw1089 is OK: puppet ran at Mon Sep 16 08:11:34 UTC 2013 [08:11:49] RECOVERY - Puppet freshness on stat1 is OK: puppet ran at Mon Sep 16 08:11:39 UTC 2013 [08:11:50] RECOVERY - Puppet freshness on mc1003 is OK: puppet ran at Mon Sep 16 08:11:40 UTC 2013 [08:11:52] RECOVERY - Puppet freshness on srv255 is OK: puppet ran at Mon Sep 16 08:11:41 UTC 2013 [08:11:52] RECOVERY - Puppet freshness on sq83 is OK: puppet ran at Mon Sep 16 08:11:41 UTC 2013 [08:11:52] RECOVERY - Puppet freshness on mw1041 is OK: puppet ran at Mon Sep 16 08:11:46 UTC 2013 [08:11:52] RECOVERY - Puppet freshness on wtp1015 is OK: puppet ran at Mon Sep 16 08:11:46 UTC 2013 [08:11:59] this is what I get for not doing my normal 'what does icinga say about puppet freshness' check first thing... instead I was coding. meh [08:12:02] RECOVERY - Puppet freshness on sq58 is OK: puppet ran at Mon Sep 16 08:11:51 UTC 2013 [08:12:02] RECOVERY - Puppet freshness on sq54 is OK: puppet ran at Mon Sep 16 08:11:51 UTC 2013 [08:12:12] RECOVERY - Puppet freshness on sq65 is OK: puppet ran at Mon Sep 16 08:12:02 UTC 2013 [08:12:22] RECOVERY - Puppet freshness on mw57 is OK: puppet ran at Mon Sep 16 08:12:08 UTC 2013 [08:12:33] RECOVERY - Puppet freshness on db63 is OK: puppet ran at Mon Sep 16 08:12:26 UTC 2013 [08:12:33] RECOVERY - Puppet freshness on analytics1014 is OK: puppet ran at Mon Sep 16 08:12:26 UTC 2013 [08:12:42] RECOVERY - Puppet freshness on lvs1005 is OK: puppet ran at Mon Sep 16 08:12:33 UTC 2013 [08:12:42] RECOVERY - Puppet freshness on mw106 is OK: puppet ran at Mon Sep 16 08:12:33 UTC 2013 [08:12:42] RECOVERY - Puppet freshness on mw43 is OK: puppet ran at Mon Sep 16 08:12:33 UTC 2013 [08:12:42] RECOVERY - Puppet freshness on mw1032 is OK: puppet ran at Mon Sep 16 08:12:33 UTC 2013 [08:12:42] RECOVERY - Puppet freshness on srv193 is OK: puppet ran at Mon Sep 16 08:12:33 UTC 2013 [08:12:43] RECOVERY - Puppet freshness on ms-fe1001 is OK: puppet ran at Mon Sep 16 08:12:33 UTC 2013 [08:12:43] RECOVERY - Puppet freshness on db1044 is OK: puppet ran at Mon Sep 16 08:12:38 UTC 2013 [08:12:44] RECOVERY - Puppet freshness on mw1099 is OK: puppet ran at Mon Sep 16 08:12:39 UTC 2013 [08:12:44] RECOVERY - Puppet freshness on mw79 is OK: puppet ran at Mon Sep 16 08:12:39 UTC 2013 [08:12:45] RECOVERY - Puppet freshness on mw35 is OK: puppet ran at Mon Sep 16 08:12:39 UTC 2013 [08:12:45] RECOVERY - Puppet freshness on cp1010 is OK: puppet ran at Mon Sep 16 08:12:39 UTC 2013 [08:12:46] RECOVERY - Puppet freshness on mw2 is OK: puppet ran at Mon Sep 16 08:12:40 UTC 2013 [08:12:46] RECOVERY - Puppet freshness on mw1033 is OK: puppet ran at Mon Sep 16 08:12:40 UTC 2013 [08:12:47] RECOVERY - Puppet freshness on search33 is OK: puppet ran at Mon Sep 16 08:12:41 UTC 2013 [08:12:52] RECOVERY - Puppet freshness on db1022 is OK: puppet ran at Mon Sep 16 08:12:46 UTC 2013 [08:12:53] RECOVERY - Puppet freshness on stafford is OK: puppet ran at Mon Sep 16 08:12:46 UTC 2013 [08:12:53] RECOVERY - Puppet freshness on mw1024 is OK: puppet ran at Mon Sep 16 08:12:46 UTC 2013 [08:12:53] RECOVERY - Puppet freshness on mw1205 is OK: puppet ran at Mon Sep 16 08:12:46 UTC 2013 [08:12:53] RECOVERY - Puppet freshness on mw1068 is OK: puppet ran at Mon Sep 16 08:12:46 UTC 2013 [08:12:53] RECOVERY - Puppet freshness on mw1173 is OK: puppet ran at Mon Sep 16 08:12:46 UTC 2013 [08:12:53] RECOVERY - Puppet freshness on db1033 is OK: puppet ran at Mon Sep 16 08:12:46 UTC 2013 [08:12:54] RECOVERY - Puppet freshness on mw1069 is OK: puppet ran at Mon Sep 16 08:12:46 UTC 2013 [08:12:54] RECOVERY - Puppet freshness on rdb1003 is OK: puppet ran at Mon Sep 16 08:12:46 UTC 2013 [08:13:03] RECOVERY - Puppet freshness on mw1150 is OK: puppet ran at Mon Sep 16 08:12:52 UTC 2013 [08:13:04] RECOVERY - Puppet freshness on mw1122 is OK: puppet ran at Mon Sep 16 08:12:52 UTC 2013 [08:13:04] RECOVERY - Puppet freshness on mw1003 is OK: puppet ran at Mon Sep 16 08:12:52 UTC 2013 [08:13:04] RECOVERY - Puppet freshness on virt1000 is OK: puppet ran at Mon Sep 16 08:12:57 UTC 2013 [08:13:04] RECOVERY - Puppet freshness on sq63 is OK: puppet ran at Mon Sep 16 08:12:57 UTC 2013 [08:13:04] RECOVERY - Puppet freshness on mw1144 is OK: puppet ran at Mon Sep 16 08:12:57 UTC 2013 [08:13:04] RECOVERY - Puppet freshness on calcium is OK: puppet ran at Mon Sep 16 08:12:57 UTC 2013 [08:13:13] RECOVERY - Puppet freshness on db1040 is OK: puppet ran at Mon Sep 16 08:13:02 UTC 2013 [08:13:13] RECOVERY - Puppet freshness on db1002 is OK: puppet ran at Mon Sep 16 08:13:03 UTC 2013 [08:13:37] RECOVERY - Puppet freshness on db48 is OK: puppet ran at Mon Sep 16 08:13:23 UTC 2013 [08:13:37] RECOVERY - Puppet freshness on analytics1004 is OK: puppet ran at Mon Sep 16 08:13:28 UTC 2013 [08:13:38] RECOVERY - Puppet freshness on ms-be11 is OK: puppet ran at Mon Sep 16 08:13:28 UTC 2013 [08:13:38] RECOVERY - Puppet freshness on virt11 is OK: puppet ran at Mon Sep 16 08:13:28 UTC 2013 [08:13:38] RECOVERY - Puppet freshness on iodine is OK: puppet ran at Mon Sep 16 08:13:28 UTC 2013 [08:13:38] RECOVERY - Puppet freshness on mw98 is OK: puppet ran at Mon Sep 16 08:13:28 UTC 2013 [08:13:38] RECOVERY - Puppet freshness on mw1052 is OK: puppet ran at Mon Sep 16 08:13:28 UTC 2013 [08:13:39] RECOVERY - Puppet freshness on mw1189 is OK: puppet ran at Mon Sep 16 08:13:29 UTC 2013 [08:13:39] RECOVERY - Puppet freshness on cp1037 is OK: puppet ran at Mon Sep 16 08:13:29 UTC 2013 [08:13:40] RECOVERY - Puppet freshness on ssl1001 is OK: puppet ran at Mon Sep 16 08:13:34 UTC 2013 [08:13:40] RECOVERY - Puppet freshness on srv262 is OK: puppet ran at Mon Sep 16 08:13:34 UTC 2013 [08:13:41] RECOVERY - Puppet freshness on srv267 is OK: puppet ran at Mon Sep 16 08:13:34 UTC 2013 [08:13:41] RECOVERY - Puppet freshness on mw1201 is OK: puppet ran at Mon Sep 16 08:13:34 UTC 2013 [08:13:45] RECOVERY - Puppet freshness on mw112 is OK: puppet ran at Mon Sep 16 08:13:39 UTC 2013 [08:13:45] RECOVERY - Puppet freshness on srv269 is OK: puppet ran at Mon Sep 16 08:13:40 UTC 2013 [08:13:45] RECOVERY - Puppet freshness on srv260 is OK: puppet ran at Mon Sep 16 08:13:40 UTC 2013 [08:13:45] RECOVERY - Puppet freshness on mw68 is OK: puppet ran at Mon Sep 16 08:13:40 UTC 2013 [08:13:45] RECOVERY - Puppet freshness on mw1025 is OK: puppet ran at Mon Sep 16 08:13:40 UTC 2013 [08:13:55] RECOVERY - Puppet freshness on search25 is OK: puppet ran at Mon Sep 16 08:13:45 UTC 2013 [08:13:55] RECOVERY - Puppet freshness on search16 is OK: puppet ran at Mon Sep 16 08:13:51 UTC 2013 [08:13:55] RECOVERY - Puppet freshness on mw37 is OK: puppet ran at Mon Sep 16 08:13:51 UTC 2013 [08:14:05] RECOVERY - Puppet freshness on mw1114 is OK: puppet ran at Mon Sep 16 08:13:57 UTC 2013 [08:14:05] RECOVERY - Puppet freshness on mw1118 is OK: puppet ran at Mon Sep 16 08:13:57 UTC 2013 [08:14:05] RECOVERY - Puppet freshness on solr1002 is OK: puppet ran at Mon Sep 16 08:13:57 UTC 2013 [08:14:05] RECOVERY - Puppet freshness on search1022 is OK: puppet ran at Mon Sep 16 08:13:57 UTC 2013 [08:14:05] RECOVERY - Puppet freshness on searchidx1001 is OK: puppet ran at Mon Sep 16 08:13:57 UTC 2013 [08:14:05] RECOVERY - Puppet freshness on mw1091 is OK: puppet ran at Mon Sep 16 08:13:57 UTC 2013 [08:14:06] RECOVERY - Puppet freshness on fluorine is OK: puppet ran at Mon Sep 16 08:13:57 UTC 2013 [08:14:06] RECOVERY - Puppet freshness on mw1142 is OK: puppet ran at Mon Sep 16 08:13:57 UTC 2013 [08:14:07] RECOVERY - Puppet freshness on mw133 is OK: puppet ran at Mon Sep 16 08:13:57 UTC 2013 [08:14:07] RECOVERY - Puppet freshness on tin is OK: puppet ran at Mon Sep 16 08:14:02 UTC 2013 [08:14:08] RECOVERY - Puppet freshness on db54 is OK: puppet ran at Mon Sep 16 08:14:02 UTC 2013 [08:14:08] RECOVERY - Puppet freshness on db1034 is OK: puppet ran at Mon Sep 16 08:14:02 UTC 2013 [08:14:46] RECOVERY - Puppet freshness on mw104 is OK: puppet ran at Mon Sep 16 08:14:37 UTC 2013 [08:14:46] RECOVERY - Puppet freshness on sq52 is OK: puppet ran at Mon Sep 16 08:14:42 UTC 2013 [08:14:46] RECOVERY - Puppet freshness on db1051 is OK: puppet ran at Mon Sep 16 08:14:42 UTC 2013 [08:14:46] RECOVERY - Puppet freshness on db66 is OK: puppet ran at Mon Sep 16 08:14:42 UTC 2013 [08:14:46] RECOVERY - Puppet freshness on search30 is OK: puppet ran at Mon Sep 16 08:14:42 UTC 2013 [08:14:46] RECOVERY - Puppet freshness on analytics1003 is OK: puppet ran at Mon Sep 16 08:14:42 UTC 2013 [08:14:47] RECOVERY - Puppet freshness on search1001 is OK: puppet ran at Mon Sep 16 08:14:42 UTC 2013 [08:14:47] RECOVERY - Puppet freshness on mw1166 is OK: puppet ran at Mon Sep 16 08:14:42 UTC 2013 [08:14:48] RECOVERY - Puppet freshness on sockpuppet is OK: puppet ran at Mon Sep 16 08:14:42 UTC 2013 [08:14:48] RECOVERY - Puppet freshness on mw10 is OK: puppet ran at Mon Sep 16 08:14:42 UTC 2013 [08:14:49] RECOVERY - Puppet freshness on mw23 is OK: puppet ran at Mon Sep 16 08:14:42 UTC 2013 [08:14:49] RECOVERY - Puppet freshness on mw1204 is OK: puppet ran at Mon Sep 16 08:14:42 UTC 2013 [08:14:50] RECOVERY - Puppet freshness on search29 is OK: puppet ran at Mon Sep 16 08:14:42 UTC 2013 [08:14:50] RECOVERY - Puppet freshness on amssq48 is OK: puppet ran at Mon Sep 16 08:14:42 UTC 2013 [08:14:51] RECOVERY - Puppet freshness on mw1092 is OK: puppet ran at Mon Sep 16 08:14:44 UTC 2013 [08:14:51] RECOVERY - Puppet freshness on mw131 is OK: puppet ran at Mon Sep 16 08:14:44 UTC 2013 [08:14:52] RECOVERY - Puppet freshness on mw60 is OK: puppet ran at Mon Sep 16 08:14:44 UTC 2013 [08:14:52] RECOVERY - Puppet freshness on mw116 is OK: puppet ran at Mon Sep 16 08:14:44 UTC 2013 [08:14:53] RECOVERY - Puppet freshness on search1012 is OK: puppet ran at Mon Sep 16 08:14:44 UTC 2013 [08:14:53] RECOVERY - Puppet freshness on mw1010 is OK: puppet ran at Mon Sep 16 08:14:44 UTC 2013 [08:14:54] RECOVERY - Puppet freshness on mw1011 is OK: puppet ran at Mon Sep 16 08:14:44 UTC 2013 [08:14:56] RECOVERY - Puppet freshness on mw1054 is OK: puppet ran at Mon Sep 16 08:14:46 UTC 2013 [08:14:56] RECOVERY - Puppet freshness on mw1066 is OK: puppet ran at Mon Sep 16 08:14:51 UTC 2013 [08:14:56] RECOVERY - Puppet freshness on mw1107 is OK: puppet ran at Mon Sep 16 08:14:51 UTC 2013 [08:14:56] RECOVERY - Puppet freshness on amssq47 is OK: puppet ran at Mon Sep 16 08:14:51 UTC 2013 [08:14:56] RECOVERY - Puppet freshness on db1042 is OK: puppet ran at Mon Sep 16 08:14:51 UTC 2013 [08:14:57] RECOVERY - Puppet freshness on labstore1001 is OK: puppet ran at Mon Sep 16 08:14:51 UTC 2013 [08:15:06] RECOVERY - Puppet freshness on wtp1001 is OK: puppet ran at Mon Sep 16 08:15:01 UTC 2013 [08:15:07] RECOVERY - Puppet freshness on ms-fe1002 is OK: puppet ran at Mon Sep 16 08:15:02 UTC 2013 [08:15:16] RECOVERY - Puppet freshness on cp1058 is OK: puppet ran at Mon Sep 16 08:15:07 UTC 2013 [08:15:27] RECOVERY - Puppet freshness on mw1211 is OK: puppet ran at Mon Sep 16 08:15:17 UTC 2013 [08:16:05] RECOVERY - Puppet freshness on sq76 is OK: puppet ran at Mon Sep 16 08:15:57 UTC 2013 [08:16:05] RECOVERY - Puppet freshness on mw1129 is OK: puppet ran at Mon Sep 16 08:16:00 UTC 2013 [08:16:05] RECOVERY - Puppet freshness on db45 is OK: puppet ran at Mon Sep 16 08:16:00 UTC 2013 [08:16:05] RECOVERY - Puppet freshness on mw75 is OK: puppet ran at Mon Sep 16 08:16:00 UTC 2013 [08:16:05] RECOVERY - Puppet freshness on mw1143 is OK: puppet ran at Mon Sep 16 08:16:00 UTC 2013 [08:16:05] RECOVERY - Puppet freshness on cp1017 is OK: puppet ran at Mon Sep 16 08:16:00 UTC 2013 [08:17:15] RECOVERY - Puppet freshness on db1009 is OK: puppet ran at Mon Sep 16 08:17:08 UTC 2013 [08:17:15] RECOVERY - Puppet freshness on mw1137 is OK: puppet ran at Mon Sep 16 08:17:13 UTC 2013 [08:17:16] RECOVERY - Puppet freshness on search1005 is OK: puppet ran at Mon Sep 16 08:17:13 UTC 2013 [08:17:25] RECOVERY - Puppet freshness on mw1084 is OK: puppet ran at Mon Sep 16 08:17:18 UTC 2013 [08:17:25] RECOVERY - Puppet freshness on srv291 is OK: puppet ran at Mon Sep 16 08:17:18 UTC 2013 [08:17:25] RECOVERY - Puppet freshness on mw61 is OK: puppet ran at Mon Sep 16 08:17:23 UTC 2013 [08:17:25] RECOVERY - Puppet freshness on search31 is OK: puppet ran at Mon Sep 16 08:17:23 UTC 2013 [08:17:25] RECOVERY - Puppet freshness on mw45 is OK: puppet ran at Mon Sep 16 08:17:23 UTC 2013 [08:17:26] RECOVERY - Puppet freshness on mw1165 is OK: puppet ran at Mon Sep 16 08:17:23 UTC 2013 [08:17:35] RECOVERY - Puppet freshness on mw1047 is OK: puppet ran at Mon Sep 16 08:17:29 UTC 2013 [08:17:35] RECOVERY - Puppet freshness on mw1181 is OK: puppet ran at Mon Sep 16 08:17:29 UTC 2013 [08:17:36] RECOVERY - Puppet freshness on mw1179 is OK: puppet ran at Mon Sep 16 08:17:29 UTC 2013 [08:17:36] RECOVERY - Puppet freshness on mw1078 is OK: puppet ran at Mon Sep 16 08:17:29 UTC 2013 [08:17:36] RECOVERY - Puppet freshness on mw74 is OK: puppet ran at Mon Sep 16 08:17:29 UTC 2013 [08:17:36] RECOVERY - Puppet freshness on hume is OK: puppet ran at Mon Sep 16 08:17:29 UTC 2013 [08:17:36] RECOVERY - Puppet freshness on mw1020 is OK: puppet ran at Mon Sep 16 08:17:34 UTC 2013 [08:17:37] RECOVERY - Puppet freshness on mw1075 is OK: puppet ran at Mon Sep 16 08:17:34 UTC 2013 [08:17:37] RECOVERY - Puppet freshness on mw1105 is OK: puppet ran at Mon Sep 16 08:17:34 UTC 2013 [08:17:46] RECOVERY - Puppet freshness on cp1062 is OK: puppet ran at Mon Sep 16 08:17:41 UTC 2013 [08:17:46] RECOVERY - Puppet freshness on db77 is OK: puppet ran at Mon Sep 16 08:17:41 UTC 2013 [08:17:55] RECOVERY - Puppet freshness on tmh1001 is OK: puppet ran at Mon Sep 16 08:17:46 UTC 2013 [08:17:55] RECOVERY - Puppet freshness on amssq37 is OK: puppet ran at Mon Sep 16 08:17:46 UTC 2013 [08:17:55] RECOVERY - Puppet freshness on amssq35 is OK: puppet ran at Mon Sep 16 08:17:46 UTC 2013 [08:17:55] RECOVERY - Puppet freshness on sq68 is OK: puppet ran at Mon Sep 16 08:17:51 UTC 2013 [08:17:55] RECOVERY - Puppet freshness on ms-fe2 is OK: puppet ran at Mon Sep 16 08:17:51 UTC 2013 [08:18:06] RECOVERY - Puppet freshness on ms-be1009 is OK: puppet ran at Mon Sep 16 08:17:57 UTC 2013 [08:18:06] RECOVERY - Puppet freshness on search1015 is OK: puppet ran at Mon Sep 16 08:17:57 UTC 2013 [08:18:06] RECOVERY - Puppet freshness on wtp1007 is OK: puppet ran at Mon Sep 16 08:17:57 UTC 2013 [08:18:06] RECOVERY - Puppet freshness on db1058 is OK: puppet ran at Mon Sep 16 08:17:57 UTC 2013 [08:18:06] RECOVERY - Puppet freshness on db65 is OK: puppet ran at Mon Sep 16 08:17:57 UTC 2013 [08:18:06] RECOVERY - Puppet freshness on wtp1002 is OK: puppet ran at Mon Sep 16 08:18:02 UTC 2013 [08:18:07] RECOVERY - Puppet freshness on cp1038 is OK: puppet ran at Mon Sep 16 08:18:02 UTC 2013 [08:18:07] RECOVERY - Puppet freshness on mw1030 is OK: puppet ran at Mon Sep 16 08:18:02 UTC 2013 [08:18:08] RECOVERY - Puppet freshness on analytics1008 is OK: puppet ran at Mon Sep 16 08:18:02 UTC 2013 [08:18:15] RECOVERY - Puppet freshness on lvs1001 is OK: puppet ran at Mon Sep 16 08:18:07 UTC 2013 [08:18:17] RECOVERY - Puppet freshness on srv272 is OK: puppet ran at Mon Sep 16 08:18:07 UTC 2013 [08:18:17] RECOVERY - Puppet freshness on amssq31 is OK: puppet ran at Mon Sep 16 08:18:07 UTC 2013 [08:18:17] RECOVERY - Puppet freshness on nickel is OK: puppet ran at Mon Sep 16 08:18:12 UTC 2013 [08:18:25] RECOVERY - Puppet freshness on mw95 is OK: puppet ran at Mon Sep 16 08:18:15 UTC 2013 [08:18:25] RECOVERY - Puppet freshness on mw99 is OK: puppet ran at Mon Sep 16 08:18:20 UTC 2013 [08:18:25] RECOVERY - Puppet freshness on mw39 is OK: puppet ran at Mon Sep 16 08:18:20 UTC 2013 [08:18:25] RECOVERY - Puppet freshness on srv264 is OK: puppet ran at Mon Sep 16 08:18:20 UTC 2013 [08:18:35] RECOVERY - Puppet freshness on erbium is OK: puppet ran at Mon Sep 16 08:18:25 UTC 2013 [08:18:35] RECOVERY - Puppet freshness on mw1169 is OK: puppet ran at Mon Sep 16 08:18:27 UTC 2013 [08:18:35] RECOVERY - Puppet freshness on search1023 is OK: puppet ran at Mon Sep 16 08:18:27 UTC 2013 [08:18:35] RECOVERY - Puppet freshness on brewster is OK: puppet ran at Mon Sep 16 08:18:27 UTC 2013 [08:18:35] RECOVERY - Puppet freshness on mw1053 is OK: puppet ran at Mon Sep 16 08:18:27 UTC 2013 [08:18:36] RECOVERY - Puppet freshness on mw1198 is OK: puppet ran at Mon Sep 16 08:18:27 UTC 2013 [08:18:36] RECOVERY - Puppet freshness on search1021 is OK: puppet ran at Mon Sep 16 08:18:27 UTC 2013 [08:18:37] RECOVERY - Puppet freshness on mw1074 is OK: puppet ran at Mon Sep 16 08:18:27 UTC 2013 [08:18:37] RECOVERY - Puppet freshness on ms-be1007 is OK: puppet ran at Mon Sep 16 08:18:27 UTC 2013 [08:18:38] RECOVERY - Puppet freshness on mw1136 is OK: puppet ran at Mon Sep 16 08:18:27 UTC 2013 [08:18:38] RECOVERY - Puppet freshness on db57 is OK: puppet ran at Mon Sep 16 08:18:32 UTC 2013 [08:18:39] RECOVERY - Puppet freshness on sq50 is OK: puppet ran at Mon Sep 16 08:18:32 UTC 2013 [08:18:45] RECOVERY - Puppet freshness on db1024 is OK: puppet ran at Mon Sep 16 08:18:39 UTC 2013 [08:18:55] RECOVERY - Puppet freshness on es5 is OK: puppet ran at Mon Sep 16 08:18:46 UTC 2013 [08:18:56] RECOVERY - Puppet freshness on db51 is OK: puppet ran at Mon Sep 16 08:18:46 UTC 2013 [08:18:56] RECOVERY - Puppet freshness on dataset2 is OK: puppet ran at Mon Sep 16 08:18:46 UTC 2013 [08:18:57] RECOVERY - Puppet freshness on mw1153 is OK: puppet ran at Mon Sep 16 08:18:46 UTC 2013 [08:18:57] RECOVERY - Puppet freshness on mw1094 is OK: puppet ran at Mon Sep 16 08:18:46 UTC 2013 [08:18:57] RECOVERY - Puppet freshness on mw1083 is OK: puppet ran at Mon Sep 16 08:18:46 UTC 2013 [08:18:58] RECOVERY - Puppet freshness on mc1008 is OK: puppet ran at Mon Sep 16 08:18:51 UTC 2013 [08:18:58] RECOVERY - Puppet freshness on bast1001 is OK: puppet ran at Mon Sep 16 08:18:51 UTC 2013 [08:19:05] RECOVERY - Puppet freshness on wtp1013 is OK: puppet ran at Mon Sep 16 08:18:56 UTC 2013 [08:19:05] RECOVERY - Puppet freshness on mw117 is OK: puppet ran at Mon Sep 16 08:18:57 UTC 2013 [08:19:05] RECOVERY - Puppet freshness on mw33 is OK: puppet ran at Mon Sep 16 08:18:57 UTC 2013 [08:19:05] RECOVERY - Puppet freshness on cp1065 is OK: puppet ran at Mon Sep 16 08:19:02 UTC 2013 [08:19:05] RECOVERY - Puppet freshness on stat1002 is OK: puppet ran at Mon Sep 16 08:19:02 UTC 2013 [08:19:06] RECOVERY - Puppet freshness on mw26 is OK: puppet ran at Mon Sep 16 08:19:02 UTC 2013 [08:19:06] RECOVERY - Puppet freshness on mw36 is OK: puppet ran at Mon Sep 16 08:19:02 UTC 2013 [08:19:07] RECOVERY - Puppet freshness on mw64 is OK: puppet ran at Mon Sep 16 08:19:02 UTC 2013 [08:19:16] RECOVERY - Puppet freshness on analytics1019 is OK: puppet ran at Mon Sep 16 08:19:07 UTC 2013 [08:19:17] RECOVERY - Puppet freshness on analytics1011 is OK: puppet ran at Mon Sep 16 08:19:07 UTC 2013 [08:19:17] RECOVERY - Puppet freshness on mw1138 is OK: puppet ran at Mon Sep 16 08:19:07 UTC 2013 [08:19:26] RECOVERY - Puppet freshness on mw28 is OK: puppet ran at Mon Sep 16 08:19:22 UTC 2013 [08:19:35] RECOVERY - Puppet freshness on dataset1001 is OK: puppet ran at Mon Sep 16 08:19:27 UTC 2013 [08:19:45] RECOVERY - Puppet freshness on tarin is OK: puppet ran at Mon Sep 16 08:19:34 UTC 2013 [08:19:45] RECOVERY - Puppet freshness on mw134 is OK: puppet ran at Mon Sep 16 08:19:35 UTC 2013 [08:19:45] RECOVERY - Puppet freshness on mw1213 is OK: puppet ran at Mon Sep 16 08:19:35 UTC 2013 [08:19:45] RECOVERY - Puppet freshness on mw1027 is OK: puppet ran at Mon Sep 16 08:19:40 UTC 2013 [08:19:45] RECOVERY - Puppet freshness on db55 is OK: puppet ran at Mon Sep 16 08:19:40 UTC 2013 [08:19:46] RECOVERY - Puppet freshness on labsdb1001 is OK: puppet ran at Mon Sep 16 08:19:40 UTC 2013 [08:19:46] RECOVERY - Puppet freshness on testsearch1002 is OK: puppet ran at Mon Sep 16 08:19:40 UTC 2013 [08:19:55] RECOVERY - Puppet freshness on wtp1021 is OK: puppet ran at Mon Sep 16 08:19:45 UTC 2013 [08:19:55] RECOVERY - Puppet freshness on snapshot4 is OK: puppet ran at Mon Sep 16 08:19:50 UTC 2013 [08:19:56] RECOVERY - Puppet freshness on ssl1007 is OK: puppet ran at Mon Sep 16 08:19:50 UTC 2013 [08:19:56] RECOVERY - Puppet freshness on cp1043 is OK: puppet ran at Mon Sep 16 08:19:50 UTC 2013 [08:19:56] RECOVERY - Puppet freshness on es1009 is OK: puppet ran at Mon Sep 16 08:19:51 UTC 2013 [08:20:07] RECOVERY - Puppet freshness on mw109 is OK: puppet ran at Mon Sep 16 08:19:56 UTC 2013 [08:20:09] RECOVERY - Puppet freshness on mw107 is OK: puppet ran at Mon Sep 16 08:19:56 UTC 2013 [08:20:09] RECOVERY - Puppet freshness on mw40 is OK: puppet ran at Mon Sep 16 08:19:56 UTC 2013 [08:20:09] RECOVERY - Puppet freshness on mw70 is OK: puppet ran at Mon Sep 16 08:19:56 UTC 2013 [08:20:09] RECOVERY - Puppet freshness on mw53 is OK: puppet ran at Mon Sep 16 08:20:02 UTC 2013 [08:20:09] RECOVERY - Puppet freshness on srv296 is OK: puppet ran at Mon Sep 16 08:20:03 UTC 2013 [08:20:09] RECOVERY - Puppet freshness on mw1218 is OK: puppet ran at Mon Sep 16 08:20:03 UTC 2013 [08:20:15] RECOVERY - Puppet freshness on mw1001 is OK: puppet ran at Mon Sep 16 08:20:08 UTC 2013 [08:20:15] RECOVERY - Puppet freshness on mw1185 is OK: puppet ran at Mon Sep 16 08:20:08 UTC 2013 [08:20:15] RECOVERY - Puppet freshness on mw1062 is OK: puppet ran at Mon Sep 16 08:20:08 UTC 2013 [08:20:15] RECOVERY - Puppet freshness on mw1022 is OK: puppet ran at Mon Sep 16 08:20:09 UTC 2013 [08:20:15] RECOVERY - Puppet freshness on mw1040 is OK: puppet ran at Mon Sep 16 08:20:09 UTC 2013 [08:20:45] RECOVERY - Puppet freshness on db1010 is OK: puppet ran at Mon Sep 16 08:20:39 UTC 2013 [08:20:45] RECOVERY - Puppet freshness on mw1132 is OK: puppet ran at Mon Sep 16 08:20:39 UTC 2013 [08:20:56] RECOVERY - Puppet freshness on search1003 is OK: puppet ran at Mon Sep 16 08:20:44 UTC 2013 [08:20:57] RECOVERY - Puppet freshness on sq64 is OK: puppet ran at Mon Sep 16 08:20:45 UTC 2013 [08:20:57] RECOVERY - Puppet freshness on aluminium is OK: puppet ran at Mon Sep 16 08:20:50 UTC 2013 [08:20:57] RECOVERY - Puppet freshness on amssq59 is OK: puppet ran at Mon Sep 16 08:20:51 UTC 2013 [08:20:57] RECOVERY - Puppet freshness on testsearch1001 is OK: puppet ran at Mon Sep 16 08:20:51 UTC 2013 [08:20:57] RECOVERY - Puppet freshness on sq55 is OK: puppet ran at Mon Sep 16 08:20:51 UTC 2013 [08:20:57] RECOVERY - Puppet freshness on sq81 is OK: puppet ran at Mon Sep 16 08:20:51 UTC 2013 [08:20:58] RECOVERY - Puppet freshness on virt2 is OK: puppet ran at Mon Sep 16 08:20:51 UTC 2013 [08:20:58] RECOVERY - Puppet freshness on mc1011 is OK: puppet ran at Mon Sep 16 08:20:51 UTC 2013 [08:20:59] RECOVERY - Puppet freshness on analytics1001 is OK: puppet ran at Mon Sep 16 08:20:51 UTC 2013 [08:20:59] RECOVERY - Puppet freshness on ms-fe1003 is OK: puppet ran at Mon Sep 16 08:20:51 UTC 2013 [08:21:00] RECOVERY - Puppet freshness on mw111 is OK: puppet ran at Mon Sep 16 08:20:51 UTC 2013 [08:21:00] RECOVERY - Puppet freshness on mw1049 is OK: puppet ran at Mon Sep 16 08:20:51 UTC 2013 [08:21:01] RECOVERY - Puppet freshness on search27 is OK: puppet ran at Mon Sep 16 08:20:51 UTC 2013 [08:21:05] RECOVERY - Puppet freshness on manutius is OK: puppet ran at Mon Sep 16 08:20:56 UTC 2013 [08:21:05] RECOVERY - Puppet freshness on professor is OK: puppet ran at Mon Sep 16 08:20:56 UTC 2013 [08:21:05] RECOVERY - Puppet freshness on mw1116 is OK: puppet ran at Mon Sep 16 08:20:56 UTC 2013 [08:21:06] RECOVERY - Puppet freshness on db31 is OK: puppet ran at Mon Sep 16 08:21:03 UTC 2013 [08:21:06] RECOVERY - Puppet freshness on amssq40 is OK: puppet ran at Mon Sep 16 08:21:03 UTC 2013 [08:21:06] RECOVERY - Puppet freshness on mw87 is OK: puppet ran at Mon Sep 16 08:21:03 UTC 2013 [08:21:18] RECOVERY - Puppet freshness on srv285 is OK: puppet ran at Mon Sep 16 08:21:08 UTC 2013 [08:21:18] RECOVERY - Puppet freshness on lvs1 is OK: puppet ran at Mon Sep 16 08:21:08 UTC 2013 [08:22:28] RECOVERY - Puppet freshness on db46 is OK: puppet ran at Mon Sep 16 08:22:18 UTC 2013 [08:22:28] RECOVERY - Puppet freshness on mw1178 is OK: puppet ran at Mon Sep 16 08:22:18 UTC 2013 [08:22:28] RECOVERY - Puppet freshness on amslvs2 is OK: puppet ran at Mon Sep 16 08:22:18 UTC 2013 [08:32:26] RECOVERY - Puppet freshness on cp3008 is OK: puppet ran at Mon Sep 16 08:32:22 UTC 2013 [11:58:15] PROBLEM - Apache HTTP on mw1174 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:59:05] RECOVERY - Apache HTTP on mw1174 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 808 bytes in 0.051 second response time [12:17:25] PROBLEM - Apache HTTP on mw1178 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:17:55] PROBLEM - Apache HTTP on mw1167 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:18:24] RECOVERY - Apache HTTP on mw1178 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 808 bytes in 7.447 second response time [12:19:05] RECOVERY - Apache HTTP on mw1167 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 808 bytes in 9.267 second response time [12:19:34] PROBLEM - Apache HTTP on mw1166 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:20:04] PROBLEM - Apache HTTP on mw1173 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:20:24] RECOVERY - Apache HTTP on mw1166 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 808 bytes in 7.235 second response time [12:24:05] RECOVERY - Apache HTTP on mw1173 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 808 bytes in 3.078 second response time [12:26:30] hmm [12:28:24] PROBLEM - Apache HTTP on mw1164 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:29:24] RECOVERY - Apache HTTP on mw1164 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 808 bytes in 7.805 second response time [12:31:24] PROBLEM - Apache HTTP on mw1180 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:32:24] PROBLEM - Apache HTTP on mw1181 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:32:34] PROBLEM - Apache HTTP on mw1209 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:33:14] RECOVERY - Apache HTTP on mw1180 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 808 bytes in 5.905 second response time [12:33:25] RECOVERY - Apache HTTP on mw1181 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 808 bytes in 7.749 second response time [12:34:25] PROBLEM - Apache HTTP on mw1163 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:34:34] RECOVERY - Apache HTTP on mw1209 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 808 bytes in 7.355 second response time [12:35:24] RECOVERY - Apache HTTP on mw1163 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 808 bytes in 4.350 second response time [12:38:15] RECOVERY - check_job_queue on hume is OK: JOBQUEUE OK - all job queues below 10,000 [12:38:24] PROBLEM - Apache HTTP on mw1180 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:39:25] RECOVERY - Apache HTTP on mw1180 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 808 bytes in 8.292 second response time [12:40:05] PROBLEM - Apache HTTP on mw1214 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:40:24] PROBLEM - Apache HTTP on mw1181 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:41:28] PROBLEM - check_job_queue on hume is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [12:41:28] PROBLEM - Apache HTTP on mw1164 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:41:59] PROBLEM - Apache HTTP on mw1162 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:42:09] RECOVERY - Apache HTTP on mw1214 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 808 bytes in 9.466 second response time [12:42:28] RECOVERY - Apache HTTP on mw1181 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 808 bytes in 8.768 second response time [12:42:28] RECOVERY - Apache HTTP on mw1164 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 808 bytes in 9.736 second response time [12:42:48] RECOVERY - Apache HTTP on mw1162 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 808 bytes in 2.982 second response time [12:43:28] PROBLEM - Apache HTTP on mw1163 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:44:28] RECOVERY - Apache HTTP on mw1163 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 808 bytes in 7.787 second response time [12:45:28] PROBLEM - Apache HTTP on mw1164 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:46:08] PROBLEM - Apache HTTP on mw1168 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:49:08] PROBLEM - Apache HTTP on mw1183 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:49:28] PROBLEM - Apache HTTP on mw1219 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:50:01] RECOVERY - Apache HTTP on mw1168 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 808 bytes in 6.217 second response time [12:50:28] RECOVERY - Apache HTTP on mw1164 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 808 bytes in 6.990 second response time [12:50:28] RECOVERY - Apache HTTP on mw1219 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 808 bytes in 8.938 second response time [12:50:28] PROBLEM - Apache HTTP on mw1181 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:52:28] RECOVERY - Apache HTTP on mw1181 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 808 bytes in 7.377 second response time [12:53:08] RECOVERY - Apache HTTP on mw1183 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 808 bytes in 7.708 second response time [12:55:08] PROBLEM - Apache HTTP on mw1179 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:56:09] RECOVERY - Apache HTTP on mw1179 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 808 bytes in 6.846 second response time [12:57:38] PROBLEM - Apache HTTP on mw1169 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:01:28] RECOVERY - Apache HTTP on mw1169 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 808 bytes in 6.491 second response time [13:05:09] PROBLEM - Apache HTTP on mw1173 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:05:38] PROBLEM - Apache HTTP on mw1163 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:06:30] RECOVERY - Apache HTTP on mw1163 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 808 bytes in 3.962 second response time [13:07:58] PROBLEM - Apache HTTP on mw1162 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:08:08] RECOVERY - Apache HTTP on mw1173 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 808 bytes in 3.792 second response time [13:08:38] PROBLEM - Apache HTTP on mw1186 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:09:00] PROBLEM - Apache HTTP on mw1215 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:09:58] RECOVERY - Apache HTTP on mw1162 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 808 bytes in 6.736 second response time [13:10:38] RECOVERY - Apache HTTP on mw1186 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 808 bytes in 9.828 second response time [13:11:38] PROBLEM - Apache HTTP on mw1217 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:12:38] PROBLEM - Apache HTTP on mw1174 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:12:59] RECOVERY - Apache HTTP on mw1215 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 808 bytes in 6.865 second response time [13:13:39] RECOVERY - Apache HTTP on mw1174 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 808 bytes in 9.530 second response time [13:14:12] RECOVERY - Apache HTTP on mw1217 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 808 bytes in 8.282 second response time [13:18:58] PROBLEM - Apache HTTP on mw1162 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:20:38] PROBLEM - Apache HTTP on mw1187 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:20:38] PROBLEM - Apache HTTP on mw1166 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:21:28] RECOVERY - Apache HTTP on mw1187 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 808 bytes in 4.971 second response time [13:21:38] RECOVERY - Apache HTTP on mw1166 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 808 bytes in 8.395 second response time [13:22:58] RECOVERY - Apache HTTP on mw1162 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 808 bytes in 9.251 second response time [13:24:08] PROBLEM - Apache HTTP on mw1216 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:25:08] RECOVERY - Apache HTTP on mw1216 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 808 bytes in 9.850 second response time [13:25:28] PROBLEM - Apache HTTP on mw1169 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:27:38] PROBLEM - Apache HTTP on mw1181 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:28:28] RECOVERY - Apache HTTP on mw1169 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 808 bytes in 6.434 second response time [13:28:42] PROBLEM - Apache HTTP on mw1209 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:29:08] PROBLEM - Apache HTTP on mw1177 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:29:38] RECOVERY - Apache HTTP on mw1181 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 808 bytes in 8.143 second response time [13:29:38] RECOVERY - Apache HTTP on mw1209 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 808 bytes in 9.254 second response time [13:30:08] RECOVERY - Apache HTTP on mw1177 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 808 bytes in 9.448 second response time [13:30:08] PROBLEM - Apache HTTP on mw1183 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:31:00] PROBLEM - Apache HTTP on mw1175 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:31:08] RECOVERY - Apache HTTP on mw1183 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 808 bytes in 4.858 second response time [13:31:29] PROBLEM - Apache HTTP on mw1180 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:31:59] RECOVERY - Apache HTTP on mw1175 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 808 bytes in 9.791 second response time [13:32:38] PROBLEM - Apache HTTP on mw1174 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:33:10] PROBLEM - Apache HTTP on mw1167 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:33:28] PROBLEM - Apache HTTP on mw1219 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:34:11] RECOVERY - Apache HTTP on mw1167 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 808 bytes in 6.796 second response time [13:34:11] PROBLEM - Apache HTTP on mw1218 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:34:29] PROBLEM - Apache HTTP on mw1186 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:34:42] RECOVERY - Apache HTTP on mw1174 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 808 bytes in 8.694 second response time [13:35:08] RECOVERY - Apache HTTP on mw1218 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 808 bytes in 9.775 second response time [13:35:08] PROBLEM - Apache HTTP on mw1179 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:35:28] RECOVERY - Apache HTTP on mw1180 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 808 bytes in 8.273 second response time [13:36:18] PROBLEM - Apache HTTP on mw1211 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:36:18] RECOVERY - Apache HTTP on mw1179 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 808 bytes in 8.882 second response time [13:36:29] RECOVERY - Apache HTTP on mw1219 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 808 bytes in 5.674 second response time [13:36:31] RECOVERY - Apache HTTP on mw1186 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 808 bytes in 8.535 second response time [13:37:08] PROBLEM - Apache HTTP on mw1216 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:37:08] PROBLEM - Apache HTTP on mw1210 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:38:08] RECOVERY - Apache HTTP on mw1216 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 808 bytes in 8.836 second response time [13:38:20] PROBLEM - Apache HTTP on mw1220 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:39:08] RECOVERY - Apache HTTP on mw1210 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 808 bytes in 6.691 second response time [13:39:30] PROBLEM - Apache HTTP on mw1180 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:40:02] PROBLEM - Apache HTTP on mw1163 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:40:08] RECOVERY - Apache HTTP on mw1211 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 808 bytes in 7.485 second response time [13:40:08] PROBLEM - Apache HTTP on mw1179 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:40:08] PROBLEM - Apache HTTP on mw1188 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:40:28] RECOVERY - Apache HTTP on mw1180 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 808 bytes in 5.644 second response time [13:40:58] RECOVERY - Apache HTTP on mw1163 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 808 bytes in 8.040 second response time [13:41:08] RECOVERY - Apache HTTP on mw1188 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 808 bytes in 6.707 second response time [13:41:08] PROBLEM - Apache HTTP on mw1216 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:41:38] PROBLEM - Apache HTTP on mw1187 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:42:01] PROBLEM - Apache HTTP on mw1162 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:42:18] RECOVERY - Apache HTTP on mw1179 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 808 bytes in 9.151 second response time [13:42:39] PROBLEM - Apache HTTP on mw1178 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:42:58] RECOVERY - Apache HTTP on mw1162 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 808 bytes in 8.945 second response time [13:43:08] PROBLEM - Apache HTTP on mw1172 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:43:08] PROBLEM - Apache HTTP on mw1183 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:43:08] PROBLEM - Apache HTTP on mw1167 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:43:18] RECOVERY - Apache HTTP on mw1220 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 808 bytes in 8.789 second response time [13:43:39] RECOVERY - Apache HTTP on mw1178 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 808 bytes in 7.784 second response time [13:44:08] RECOVERY - Apache HTTP on mw1167 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 808 bytes in 6.746 second response time [13:44:32] PROBLEM - Apache HTTP on mw1186 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:44:59] PROBLEM - Apache HTTP on mw1185 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:45:08] RECOVERY - Apache HTTP on mw1216 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 808 bytes in 7.347 second response time [13:45:08] RECOVERY - Apache HTTP on mw1172 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 808 bytes in 9.648 second response time [13:45:31] RECOVERY - Apache HTTP on mw1186 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 808 bytes in 9.819 second response time [13:45:31] PROBLEM - Apache HTTP on mw1180 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:46:09] RECOVERY - Apache HTTP on mw1183 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 808 bytes in 8.658 second response time [13:46:28] RECOVERY - Apache HTTP on mw1180 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 808 bytes in 5.757 second response time [13:46:59] RECOVERY - Apache HTTP on mw1185 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 808 bytes in 7.863 second response time [13:48:00] PROBLEM - Apache HTTP on mw1170 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:50:20] PROBLEM - Apache HTTP on mw1184 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:50:28] PROBLEM - Apache HTTP on mw1186 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:50:30] PROBLEM - Apache HTTP on mw1169 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:50:58] RECOVERY - Apache HTTP on mw1170 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 808 bytes in 7.586 second response time [13:51:18] PROBLEM - Apache HTTP on mw1172 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:51:28] RECOVERY - Apache HTTP on mw1169 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 808 bytes in 4.715 second response time [13:52:10] PROBLEM - Apache HTTP on mw1175 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:52:18] PROBLEM - Apache HTTP on mw1216 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:53:18] RECOVERY - Apache HTTP on mw1184 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 808 bytes in 8.314 second response time [13:54:28] RECOVERY - Apache HTTP on mw1186 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 808 bytes in 5.177 second response time [13:55:08] PROBLEM - Apache HTTP on mw1163 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:55:31] PROBLEM - Apache HTTP on mw1219 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:56:28] RECOVERY - Apache HTTP on mw1219 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 808 bytes in 4.520 second response time [13:56:31] PROBLEM - Apache HTTP on mw1180 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:57:11] PROBLEM - Apache HTTP on mw1182 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:57:18] PROBLEM - Apache HTTP on mw1214 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:57:28] RECOVERY - Apache HTTP on mw1180 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 808 bytes in 7.640 second response time [13:57:38] RECOVERY - Apache HTTP on mw1187 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 808 bytes in 8.236 second response time [13:58:09] RECOVERY - Apache HTTP on mw1175 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 808 bytes in 7.396 second response time [13:58:09] RECOVERY - Apache HTTP on mw1182 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 808 bytes in 8.981 second response time [13:58:19] RECOVERY - Apache HTTP on mw1216 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 808 bytes in 7.630 second response time [13:58:19] RECOVERY - Apache HTTP on mw1214 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 808 bytes in 8.147 second response time [13:58:30] PROBLEM - Apache HTTP on mw1186 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:59:10] RECOVERY - Apache HTTP on mw1163 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 808 bytes in 8.748 second response time [13:59:18] RECOVERY - Apache HTTP on mw1172 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 808 bytes in 9.831 second response time [13:59:18] PROBLEM - Apache HTTP on mw1179 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:00:29] RECOVERY - Apache HTTP on mw1186 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 808 bytes in 7.478 second response time [14:02:38] PROBLEM - Apache HTTP on mw1174 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:02:38] PROBLEM - Apache HTTP on mw1166 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:03:11] PROBLEM - Apache HTTP on mw1185 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:03:18] PROBLEM - Apache HTTP on mw1220 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:03:18] PROBLEM - Apache HTTP on mw1210 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:03:29] PROBLEM - Apache HTTP on mw1169 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:04:19] RECOVERY - Apache HTTP on mw1179 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 808 bytes in 9.485 second response time [14:05:09] RECOVERY - Apache HTTP on mw1185 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 808 bytes in 9.368 second response time [14:05:09] PROBLEM - Apache HTTP on mw1215 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:05:09] PROBLEM - Apache HTTP on mw1162 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:05:09] PROBLEM - Apache HTTP on mw1175 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:05:19] RECOVERY - Apache HTTP on mw1220 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 808 bytes in 8.667 second response time [14:05:20] PROBLEM - Apache HTTP on mw1212 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:05:28] RECOVERY - Apache HTTP on mw1169 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 808 bytes in 8.540 second response time [14:05:38] RECOVERY - Apache HTTP on mw1174 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 808 bytes in 8.780 second response time [14:06:08] RECOVERY - Apache HTTP on mw1175 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 808 bytes in 9.782 second response time [14:07:08] RECOVERY - Apache HTTP on mw1215 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 808 bytes in 9.429 second response time [14:07:22] PROBLEM - Apache HTTP on mw1214 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:07:31] PROBLEM - Apache HTTP on mw1186 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:07:40] RECOVERY - Apache HTTP on mw1166 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 808 bytes in 9.086 second response time [14:08:08] PROBLEM - Apache HTTP on mw1182 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:08:18] PROBLEM - Apache HTTP on mw1179 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:08:18] PROBLEM - Apache HTTP on mw1216 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:09:08] RECOVERY - Apache HTTP on mw1162 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 808 bytes in 9.459 second response time [14:09:19] RECOVERY - Apache HTTP on mw1212 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 808 bytes in 8.374 second response time [14:09:19] RECOVERY - Apache HTTP on mw1179 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 808 bytes in 9.319 second response time [14:09:31] RECOVERY - Apache HTTP on mw1186 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 808 bytes in 9.749 second response time [14:09:38] RECOVERY - check_job_queue on hume is OK: JOBQUEUE OK - all job queues below 10,000 [14:09:39] PROBLEM - Apache HTTP on mw1209 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:10:20] RECOVERY - Apache HTTP on mw1216 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 808 bytes in 6.535 second response time [14:11:18] RECOVERY - Apache HTTP on mw1210 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 808 bytes in 6.146 second response time [14:11:40] PROBLEM - Apache HTTP on mw1185 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:12:30] PROBLEM - Apache HTTP on mw1180 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:12:39] RECOVERY - Apache HTTP on mw1185 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 808 bytes in 8.614 second response time [14:12:39] PROBLEM - check_job_queue on hume is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [14:13:28] RECOVERY - Apache HTTP on mw1180 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 808 bytes in 9.115 second response time [14:13:39] RECOVERY - Apache HTTP on mw1209 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 808 bytes in 9.612 second response time [14:13:49] PROBLEM - Apache HTTP on mw1176 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:14:28] PROBLEM - Apache HTTP on mw1210 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:14:29] PROBLEM - Apache HTTP on mw1186 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:14:38] RECOVERY - Apache HTTP on mw1176 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 808 bytes in 6.031 second response time [14:15:12] RECOVERY - Apache HTTP on mw1182 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 808 bytes in 8.745 second response time [14:15:28] RECOVERY - Apache HTTP on mw1214 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 808 bytes in 4.424 second response time [14:15:28] RECOVERY - Apache HTTP on mw1210 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 808 bytes in 4.840 second response time [14:15:29] PROBLEM - Apache HTTP on mw1216 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:15:29] PROBLEM - Apache HTTP on mw1184 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:16:37] (03CR) 10Dereckson: [C: 04-1] "The echo deployment is currently under a scheduled roadmap. See https://www.mediawiki.org/wiki/Echo/Release_Plan_2013" [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/84203 (owner: 10Ladsgroup) [14:16:40] RECOVERY - Apache HTTP on mw1186 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 808 bytes in 9.671 second response time [14:17:29] RECOVERY - Apache HTTP on mw1184 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 808 bytes in 3.256 second response time [14:17:55] (03CR) 10Dereckson: [C: 04-1] "Need to rebase — This change should be rebased against master, to remove the change I476d5519a82328c74b2381d39917974c6e590289 dependency." [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/84206 (owner: 10Ladsgroup) [14:18:50] PROBLEM - Apache HTTP on mw1163 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:19:39] PROBLEM - Apache HTTP on mw1173 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:19:39] PROBLEM - Apache HTTP on mw1219 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:19:48] RECOVERY - Apache HTTP on mw1163 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 808 bytes in 9.078 second response time [14:19:48] PROBLEM - Apache HTTP on mw1166 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:20:29] PROBLEM - Apache HTTP on mw1164 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:20:38] PROBLEM - Apache HTTP on mw1162 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:20:49] RECOVERY - Apache HTTP on mw1166 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 808 bytes in 9.227 second response time [14:21:08] PROBLEM - Apache HTTP on mw1182 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:21:31] RECOVERY - Apache HTTP on mw1219 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 808 bytes in 6.215 second response time [14:21:39] PROBLEM - Apache HTTP on mw1179 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:22:28] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:22:38] RECOVERY - Apache HTTP on mw1162 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 808 bytes in 8.339 second response time [14:22:38] PROBLEM - Apache HTTP on mw1218 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:22:38] PROBLEM - Apache HTTP on mw1220 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:22:38] PROBLEM - Apache HTTP on mw1217 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:23:08] PROBLEM - Apache HTTP on mw1171 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:23:18] (03CR) 10Dereckson: "(1 comment)" [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/84206 (owner: 10Ladsgroup) [14:23:38] RECOVERY - Apache HTTP on mw1218 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 808 bytes in 7.156 second response time [14:23:38] RECOVERY - Apache HTTP on mw1173 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 808 bytes in 8.012 second response time [14:24:08] RECOVERY - Apache HTTP on mw1171 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 808 bytes in 9.863 second response time [14:24:19] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.126 second response time [14:24:40] RECOVERY - Apache HTTP on mw1217 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 808 bytes in 9.773 second response time [14:24:40] RECOVERY - Apache HTTP on mw1179 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 808 bytes in 9.953 second response time [14:24:40] PROBLEM - Apache HTTP on mw1180 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:25:38] PROBLEM - Apache HTTP on mw1219 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:25:48] PROBLEM - Apache HTTP on mw1163 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:26:48] PROBLEM - Apache HTTP on mw1185 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:26:49] PROBLEM - Apache HTTP on mw1174 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:27:08] RECOVERY - Apache HTTP on mw1182 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 808 bytes in 9.989 second response time [14:27:28] RECOVERY - Apache HTTP on mw1164 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 808 bytes in 8.547 second response time [14:27:48] RECOVERY - Apache HTTP on mw1216 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 808 bytes in 8.909 second response time [14:27:48] RECOVERY - Apache HTTP on mw1185 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 808 bytes in 9.774 second response time [14:27:48] RECOVERY - Apache HTTP on mw1163 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 808 bytes in 8.385 second response time [14:27:48] PROBLEM - Apache HTTP on mw1178 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:28:48] RECOVERY - Apache HTTP on mw1178 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 808 bytes in 8.893 second response time [14:29:41] RECOVERY - Apache HTTP on mw1220 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 808 bytes in 9.408 second response time [14:29:41] RECOVERY - Apache HTTP on mw1180 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 808 bytes in 9.561 second response time [14:29:41] RECOVERY - Apache HTTP on mw1174 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 808 bytes in 8.383 second response time [14:30:40] PROBLEM - Apache HTTP on mw1184 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:30:41] PROBLEM - Apache HTTP on mw1218 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:30:49] PROBLEM - Apache HTTP on mw1185 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:31:10] PROBLEM - Apache HTTP on mw1182 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:31:44] PROBLEM - Apache HTTP on mw1173 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:31:44] PROBLEM - Apache HTTP on mw1162 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:31:44] PROBLEM - Apache HTTP on mw1211 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:32:38] RECOVERY - Apache HTTP on mw1219 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 808 bytes in 8.768 second response time [14:32:38] PROBLEM - Apache HTTP on mw1220 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:32:38] PROBLEM - Apache HTTP on mw1179 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:32:48] PROBLEM - Apache HTTP on mw1170 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:32:48] PROBLEM - Apache HTTP on mw1163 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:33:39] RECOVERY - Apache HTTP on mw1173 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 808 bytes in 7.961 second response time [14:33:39] RECOVERY - Apache HTTP on mw1211 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 808 bytes in 8.930 second response time [14:33:39] RECOVERY - Apache HTTP on mw1162 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 808 bytes in 9.400 second response time [14:33:48] PROBLEM - Apache HTTP on mw1166 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:34:08] RECOVERY - Apache HTTP on mw1182 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 808 bytes in 9.927 second response time [14:34:39] RECOVERY - Apache HTTP on mw1170 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 808 bytes in 3.246 second response time [14:34:40] RECOVERY - Apache HTTP on mw1179 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 808 bytes in 9.321 second response time [14:34:48] RECOVERY - Apache HTTP on mw1166 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 808 bytes in 6.401 second response time [14:34:48] RECOVERY - Apache HTTP on mw1163 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 808 bytes in 8.982 second response time [14:35:41] RECOVERY - Apache HTTP on mw1220 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 808 bytes in 9.283 second response time [14:35:41] RECOVERY - Apache HTTP on mw1184 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 808 bytes in 9.379 second response time [14:35:48] PROBLEM - Apache HTTP on mw1210 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:36:08] PROBLEM - Apache HTTP on mw1171 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:36:40] PROBLEM - Apache HTTP on mw1211 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:36:48] RECOVERY - Apache HTTP on mw1185 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 808 bytes in 4.629 second response time [14:37:48] RECOVERY - Apache HTTP on mw1210 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 808 bytes in 8.581 second response time [14:37:48] RECOVERY - Apache HTTP on mw1211 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 808 bytes in 9.950 second response time [14:38:09] RECOVERY - Apache HTTP on mw1171 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 808 bytes in 7.611 second response time [14:38:38] RECOVERY - Apache HTTP on mw1218 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 808 bytes in 3.687 second response time [14:38:48] PROBLEM - Apache HTTP on mw1161 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:39:49] PROBLEM - Apache HTTP on mw1181 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:40:42] PROBLEM - Apache HTTP on mw1168 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:41:01] PROBLEM - Apache HTTP on mw1182 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:41:42] RECOVERY - Apache HTTP on mw1161 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 808 bytes in 8.264 second response time [14:41:52] RECOVERY - Apache HTTP on mw1181 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 808 bytes in 8.846 second response time [14:42:43] RECOVERY - Apache HTTP on mw1168 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 808 bytes in 9.370 second response time [14:45:41] PROBLEM - Apache HTTP on mw1169 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:46:43] PROBLEM - Apache HTTP on mw1168 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:47:42] RECOVERY - Apache HTTP on mw1169 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 808 bytes in 9.978 second response time [14:47:54] PROBLEM - Apache HTTP on mw1170 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:48:53] PROBLEM - Apache HTTP on mw1185 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:49:52] RECOVERY - Apache HTTP on mw1170 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 808 bytes in 5.534 second response time [14:50:42] RECOVERY - Apache HTTP on mw1168 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 808 bytes in 7.163 second response time [14:50:52] RECOVERY - Apache HTTP on mw1182 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 808 bytes in 8.184 second response time [14:51:51] RECOVERY - Apache HTTP on mw1185 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 808 bytes in 9.175 second response time [14:54:01] PROBLEM - Apache HTTP on mw1182 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:55:52] PROBLEM - Apache HTTP on mw1188 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:56:51] PROBLEM - Apache HTTP on mw1218 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:57:52] PROBLEM - Apache HTTP on mw1172 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:58:01] RECOVERY - Apache HTTP on mw1182 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 808 bytes in 9.764 second response time [14:58:53] RECOVERY - Apache HTTP on mw1188 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 808 bytes in 7.874 second response time [14:58:53] RECOVERY - Apache HTTP on mw1218 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 808 bytes in 8.138 second response time [14:59:51] PROBLEM - Apache HTTP on mw1185 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:00:51] RECOVERY - Apache HTTP on mw1172 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 808 bytes in 8.492 second response time [15:01:02] PROBLEM - Apache HTTP on mw1170 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:01:55] RECOVERY - Apache HTTP on mw1185 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 808 bytes in 9.277 second response time [15:03:01] RECOVERY - Apache HTTP on mw1170 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 808 bytes in 7.870 second response time [15:03:51] PROBLEM - Apache HTTP on mw1161 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:04:51] RECOVERY - Apache HTTP on mw1161 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 808 bytes in 6.485 second response time [15:05:55] PROBLEM - Apache HTTP on mw1213 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:06:51] RECOVERY - Apache HTTP on mw1213 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 808 bytes in 7.392 second response time [15:07:53] PROBLEM - Apache HTTP on mw1161 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:08:02] PROBLEM - Apache HTTP on mw1185 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:09:01] PROBLEM - Apache HTTP on mw1209 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:10:02] RECOVERY - Apache HTTP on mw1185 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 808 bytes in 5.475 second response time [15:10:54] RECOVERY - Apache HTTP on mw1161 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 808 bytes in 7.631 second response time [15:12:01] RECOVERY - Apache HTTP on mw1209 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 808 bytes in 8.506 second response time [15:12:02] PROBLEM - Apache HTTP on mw1163 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:12:55] PROBLEM - Apache HTTP on mw1214 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:13:02] RECOVERY - Apache HTTP on mw1163 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 808 bytes in 8.781 second response time [15:13:51] RECOVERY - Apache HTTP on mw1214 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 808 bytes in 7.651 second response time [15:13:51] PROBLEM - Apache HTTP on mw1180 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:13:51] PROBLEM - Apache HTTP on mw1210 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:14:52] RECOVERY - Apache HTTP on mw1210 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 808 bytes in 8.327 second response time [15:15:07] PROBLEM - Apache HTTP on mw1166 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:15:54] RECOVERY - Apache HTTP on mw1180 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 808 bytes in 8.570 second response time [15:22:39] (03PS1) 10ArielGlenn: bump MaxClients from 40 to 50 for appservers [operations/puppet] - 10https://gerrit.wikimedia.org/r/84327 [15:23:01] PROBLEM - Apache HTTP on mw1185 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:24:00] (03CR) 10ArielGlenn: [C: 032] bump MaxClients from 40 to 50 for appservers [operations/puppet] - 10https://gerrit.wikimedia.org/r/84327 (owner: 10ArielGlenn) [15:24:31] apergos: it should be dependent on the machines [15:24:34] on # cpu cores [15:25:01] RECOVERY - Apache HTTP on mw1166 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 808 bytes in 7.922 second response time [15:25:01] RECOVERY - Apache HTTP on mw1185 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 808 bytes in 8.538 second response time [15:25:25] hmm, well it currently doesn't seem to be, scalers get less and the ones that are 'slow' get 5 [15:25:33] everything else gets 40 (soon to be 50) [15:26:52] yeah [15:26:52] RECOVERY - check_job_queue on fenari is OK: JOBQUEUE OK - all job queues below 10,000 [15:30:01] PROBLEM - check_job_queue on fenari is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [15:30:28] hm 12, 16 or 24 it looks like [15:31:03] PROBLEM - Apache HTTP on mw1181 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:31:03] PROBLEM - Apache HTTP on mw1176 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:32:02] RECOVERY - Apache HTTP on mw1176 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 808 bytes in 9.386 second response time [15:33:02] RECOVERY - Apache HTTP on mw1181 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 808 bytes in 7.627 second response time [15:36:12] PROBLEM - Apache HTTP on mw1181 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:37:01] PROBLEM - Apache HTTP on mw1167 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:38:02] RECOVERY - Apache HTTP on mw1167 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 808 bytes in 7.563 second response time [15:38:02] PROBLEM - Apache HTTP on mw1163 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:39:02] RECOVERY - Apache HTTP on mw1163 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 808 bytes in 6.506 second response time [15:39:03] RECOVERY - Apache HTTP on mw1181 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 808 bytes in 6.807 second response time [15:40:01] PROBLEM - Apache HTTP on mw1211 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:41:02] RECOVERY - Apache HTTP on mw1211 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 808 bytes in 8.218 second response time [15:41:02] PROBLEM - Apache HTTP on mw1219 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:42:35] is there anyone around who can walk me through some connection issues? I'm trying to get some logs from vanadium and I'm not allowed to connect. I though I had been given permission for that recently. [15:43:29] I think they're a bit busy right now (whoevers awake) with those apache PROBLEMs [15:43:52] oh, yeah, I can wait [15:44:00] RECOVERY - Apache HTTP on mw1219 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 808 bytes in 6.535 second response time [15:44:00] PROBLEM - Apache HTTP on mw1168 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:46:00] PROBLEM - Apache HTTP on mw1162 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:47:00] RECOVERY - Apache HTTP on mw1168 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 808 bytes in 5.144 second response time [15:47:00] PROBLEM - Apache HTTP on mw1169 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:47:10] PROBLEM - Apache HTTP on mw1174 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:48:00] RECOVERY - Apache HTTP on mw1169 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 808 bytes in 8.130 second response time [15:48:00] PROBLEM - Apache HTTP on mw1175 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:48:00] PROBLEM - Apache HTTP on mw1171 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:48:10] RECOVERY - Apache HTTP on mw1174 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 808 bytes in 6.125 second response time [15:48:29] apergos: we dont have any 24 core, those are hyperthreading lies [15:48:37] just fyi [15:48:42] ah [15:48:46] salt reports the lies I guess [15:48:50] they are 12 core [15:48:56] 6 per cpu is our max apache core config [15:49:00] RECOVERY - Apache HTTP on mw1175 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 808 bytes in 9.990 second response time [15:49:01] PROBLEM - Apache HTTP on mw1188 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:49:03] I'd better see what facter says on those boxes [15:49:13] so 12/24 are same, and 8/16 are same [15:49:23] the 16 is HT enabled 4 core dual cpu boxen [15:49:33] processorcount 24 well that's inconvenient [15:49:36] can't use that so well [15:49:37] I am uncertain if HT helps at all on these. [15:50:08] are you doing a config based on count (thats auto populated)? [15:50:18] or manually setting? [15:50:33] (if you are in middle, i can wait, just asking so I can try to help ;) [15:50:55] also, im here, and you seem to be triaging this,so just lemme know what i can do to help. [15:51:02] RECOVERY - Apache HTTP on mw1162 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 808 bytes in 6.543 second response time [15:51:02] RECOVERY - Apache HTTP on mw1171 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 808 bytes in 8.407 second response time [15:51:15] well I guess we will have the 8 (16) core ones with 40 maxclients and the 12 (24) core ones with 50 [15:51:43] sounds feasible to me [15:52:03] later we should find out if HT being enabled has any actual help or hinderance [15:52:06] and make them match. [15:52:10] PROBLEM - Apache HTTP on mw1170 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:52:14] (they should all be disabled now, but obviously not ;) [15:52:21] (03PS2) 10Ladsgroup: Adding option of 'patrolmarks' to autoconfirmed users in ckbwiki [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/84206 [15:53:03] PROBLEM - Apache HTTP on mw1213 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:53:13] PROBLEM - Apache HTTP on mw1185 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:53:36] yeah salt gives a nice list of 24 core boxes [15:53:41] (03CR) 10Ladsgroup: "I tried to rebase against master but I couldn't. So I don't know what to do now :(" [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/84206 (owner: 10Ladsgroup) [15:53:46] /root/numcores on sockpuppet if you care [15:54:00] RECOVERY - Apache HTTP on mw1213 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 808 bytes in 9.737 second response time [15:54:01] PROBLEM - Apache HTTP on mw1171 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:54:10] RECOVERY - Apache HTTP on mw1185 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 808 bytes in 7.015 second response time [15:54:30] bleh, its a lot of the 'bulk assigned' mw servers it seems... [15:54:37] i guess the bulk script missed some. [15:55:00] RECOVERY - Apache HTTP on mw1171 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 808 bytes in 4.399 second response time [15:56:02] RECOVERY - Apache HTTP on mw1188 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 808 bytes in 8.452 second response time [15:56:20] (03PS1) 10ArielGlenn: MaxClients from 40 to 50 in appserver module too, for 8 core boxes [operations/puppet] - 10https://gerrit.wikimedia.org/r/84335 [15:58:10] RECOVERY - Apache HTTP on mw1170 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 808 bytes in 9.115 second response time [15:58:10] PROBLEM - LVS HTTPS IPv4 on wikivoyage-lb.pmtpa.wikimedia.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:58:10] PROBLEM - Apache HTTP on mw1181 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:59:10] RECOVERY - LVS HTTPS IPv4 on wikivoyage-lb.pmtpa.wikimedia.org is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 757 bytes in 5.875 second response time [15:59:11] RECOVERY - Apache HTTP on mw1181 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 808 bytes in 8.224 second response time [16:00:13] PROBLEM - Apache HTTP on mw1210 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:01:01] PROBLEM - Apache HTTP on mw1217 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:01:10] PROBLEM - Apache HTTP on mw1185 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:01:11] PROBLEM - Apache HTTP on mw1178 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:02:00] RECOVERY - Apache HTTP on mw1217 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 808 bytes in 8.349 second response time [16:02:10] RECOVERY - Apache HTTP on mw1178 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 808 bytes in 4.718 second response time [16:02:10] RECOVERY - Apache HTTP on mw1210 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 808 bytes in 6.257 second response time [16:02:10] (03CR) 10ArielGlenn: [C: 032] MaxClients from 40 to 50 in appserver module too, for 8 core boxes [operations/puppet] - 10https://gerrit.wikimedia.org/r/84335 (owner: 10ArielGlenn) [16:02:10] RECOVERY - Apache HTTP on mw1185 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 808 bytes in 9.368 second response time [16:05:51] (03PS1) 10Diederik: Linkify references to Analytics Mingle projects. [operations/puppet] - 10https://gerrit.wikimedia.org/r/84338 [16:07:12] PROBLEM - Apache HTTP on mw1172 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:07:33] gah that was of course backwards [16:07:38] what math. I cannot even read [16:08:53] heh [16:09:06] apergos: want anyone else to +1 (when you fix it) patchset? [16:09:10] RECOVERY - Apache HTTP on mw1172 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 808 bytes in 4.657 second response time [16:09:14] (03PS1) 10ArielGlenn: 50 maxclients for the 12 core boxes!! not the 8 core [operations/puppet] - 10https://gerrit.wikimedia.org/r/84339 [16:09:17] also nice to distribute blame =] [16:09:25] feel free to stare at it if you like [16:09:40] (03CR) 10RobH: [C: 031] 50 maxclients for the 12 core boxes!! not the 8 core [operations/puppet] - 10https://gerrit.wikimedia.org/r/84339 (owner: 10ArielGlenn) [16:09:46] that one looks much nicer [16:09:59] but quickly because the other one is already making its way around [16:10:07] I don't think they are going to fall over but still [16:11:15] (03CR) 10ArielGlenn: [C: 032] 50 maxclients for the 12 core boxes!! not the 8 core [operations/puppet] - 10https://gerrit.wikimedia.org/r/84339 (owner: 10ArielGlenn) [16:13:25] !log demon synchronized wmf-config/InitialiseSettings.php [16:13:28] Logged the message, Master [16:13:40] (03CR) 10QChris: [C: 04-1] "(2 comments)" [operations/puppet] - 10https://gerrit.wikimedia.org/r/84338 (owner: 10Diederik) [16:14:10] PROBLEM - Apache HTTP on mw1179 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:15:10] PROBLEM - Apache HTTP on mw1167 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:16:10] RECOVERY - Apache HTTP on mw1167 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 808 bytes in 8.495 second response time [16:16:10] PROBLEM - Apache HTTP on mw1214 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:16:10] PROBLEM - Apache HTTP on mw1175 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:16:20] PROBLEM - Apache HTTP on mw1187 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:17:10] RECOVERY - Apache HTTP on mw1179 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 808 bytes in 5.717 second response time [16:17:20] RECOVERY - Apache HTTP on mw1187 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 808 bytes in 9.526 second response time [16:17:26] well maybe 50 is too much for those 8 core... [16:17:48] they'll be coming back as the next puppet run goes around but I'll help it out too a little [16:18:10] PROBLEM - Apache HTTP on mw1162 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:19:11] RECOVERY - Apache HTTP on mw1175 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 808 bytes in 7.464 second response time [16:19:21] PROBLEM - Apache HTTP on mw1166 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:19:51] oh. heh this does not of course restart the apaches... [16:20:04] RECOVERY - Apache HTTP on mw1214 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 808 bytes in 7.565 second response time [16:20:10] RECOVERY - Apache HTTP on mw1162 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 808 bytes in 9.968 second response time [16:20:10] PROBLEM - Apache HTTP on mw1217 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:20:10] PROBLEM - Apache HTTP on mw1168 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:20:10] PROBLEM - Apache HTTP on mw1173 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:21:11] RECOVERY - Apache HTTP on mw1217 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 808 bytes in 9.094 second response time [16:21:22] RECOVERY - Apache HTTP on mw1168 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 808 bytes in 9.828 second response time [16:21:22] PROBLEM - Apache HTTP on mw1179 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:21:22] PROBLEM - Apache HTTP on mw1186 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:21:22] RECOVERY - Apache HTTP on mw1166 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 808 bytes in 8.945 second response time [16:21:22] PROBLEM - Apache HTTP on mw1176 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:22:10] PROBLEM - Apache HTTP on mw1093 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:22:10] RECOVERY - Apache HTTP on mw1186 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 808 bytes in 9.320 second response time [16:22:10] PROBLEM - Apache HTTP on mw1172 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:22:21] PROBLEM - Apache HTTP on mw1181 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:22:25] apergos: i have to use salt for that [16:22:33] cuz the apache restart script has issues for me. [16:22:46] well I don't want to restart them all right now [16:22:51] puppet needs to run first [16:23:11] RECOVERY - Apache HTTP on mw1172 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 808 bytes in 8.089 second response time [16:23:11] RECOVERY - Apache HTTP on mw1179 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 808 bytes in 8.089 second response time [16:23:11] PROBLEM - Apache HTTP on mw1214 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:23:11] PROBLEM - Apache HTTP on mw1212 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:23:11] PROBLEM - Apache HTTP on mw1188 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:23:22] RECOVERY - Apache HTTP on mw1176 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 808 bytes in 6.279 second response time [16:23:22] PROBLEM - Apache HTTP on mw1174 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:23:23] PROBLEM - Apache HTTP on mw1170 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:24:11] RECOVERY - Apache HTTP on mw1093 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 808 bytes in 6.599 second response time [16:24:11] RECOVERY - Apache HTTP on mw1212 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 808 bytes in 6.738 second response time [16:24:11] RECOVERY - Apache HTTP on mw1214 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 808 bytes in 9.691 second response time [16:24:22] RECOVERY - Apache HTTP on mw1170 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 808 bytes in 3.488 second response time [16:24:23] RECOVERY - Apache HTTP on mw1174 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 808 bytes in 7.529 second response time [16:24:23] PROBLEM - Apache HTTP on mw1182 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:25:10] RECOVERY - Apache HTTP on mw1173 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 808 bytes in 5.820 second response time [16:25:21] RECOVERY - Apache HTTP on mw1182 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 808 bytes in 9.169 second response time [16:26:21] RECOVERY - Apache HTTP on mw1181 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 808 bytes in 6.136 second response time [16:26:21] PROBLEM - Apache HTTP on mw1176 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:27:10] RECOVERY - Apache HTTP on mw1188 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 808 bytes in 7.497 second response time [16:28:23] RECOVERY - Apache HTTP on mw1176 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 808 bytes in 6.365 second response time [16:28:56] more okssssss [16:29:26] apergos: woot? [16:29:31] not yet [16:31:19] (03CR) 10Ottomata: "Hmm, looks good, but what's the reason to use logrotate over the built in dailyRollingFile (log4j?) setup?" [operations/puppet] - 10https://gerrit.wikimedia.org/r/82623 (owner: 10Manybubbles) [16:31:30] apergos: how so, i see mostly green on icinga now? [16:31:46] i mean, we may need more capacity [16:31:48] until we have pupet run on all of them and a restart evrywhere [16:31:54] then I can call it good [16:33:25] hrmm, so evidentally we need to re-evaluate the apache settings for the higher capacity systems we now use [16:33:31] and possibly also order more apaches.. [16:33:35] yep [16:33:49] actually, can reuse some new tampa stuff in eqiad shortly [16:33:50] (03CR) 10JanZerebecki: [C: 031] "This should work, just needs to be rebased. Can one of you test it with apache-fast-test and merge it?" [operations/apache-config] - 10https://gerrit.wikimedia.org/r/53403 (owner: 10JanZerebecki) [16:33:56] we'll have them shipped for the apache stuff [16:33:59] new and not old sorryboxes I hope [16:34:05] (the new 6 core per cpu stuff) [16:34:11] RECOVERY - MySQL Slave Running on db1005 is OK: OK replication Slave_IO_Running: Yes Slave_SQL_Running: Yes Last_Error: [16:34:12] moving quad core apaches isnt worthwhile [16:34:30] RECOVERY - MySQL Slave Running on db1007 is OK: OK replication Slave_IO_Running: Yes Slave_SQL_Running: Yes Last_Error: [16:36:10] PROBLEM - MySQL Slave Delay on db1005 is CRITICAL: CRIT replication delay 103515 seconds [16:36:29] jobqueue going up and up https://ganglia.wikimedia.org/latest/graph_all_periods.php?c=Miscellaneous%20pmtpa&h=hume.wikimedia.org&v=823574&m=Global_JobQueue_length&r=hour&z=default&jr=&js=&st=1365625056&z=large [16:36:52] PROBLEM - MySQL Slave Delay on db1007 is CRITICAL: CRIT replication delay 106874 seconds [16:41:20] PROBLEM - Apache HTTP on mw1168 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:42:10] RECOVERY - Apache HTTP on mw1168 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 808 bytes in 9.225 second response time [16:42:23] PROBLEM - Apache HTTP on mw1180 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:44:20] RECOVERY - Apache HTTP on mw1180 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 808 bytes in 8.041 second response time [16:45:20] PROBLEM - Apache HTTP on mw1168 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:45:24] oh, AaronSchulz, this bug was reopened (job queue related) https://bugzilla.wikimedia.org/show_bug.cgi?id=43936 it appears the queue has been steadily growing recently. [16:46:20] PROBLEM - Apache HTTP on mw1173 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:47:21] RECOVERY - Apache HTTP on mw1173 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 808 bytes in 8.653 second response time [16:47:21] PROBLEM - Apache HTTP on mw1186 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:48:20] RECOVERY - Apache HTTP on mw1168 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 808 bytes in 9.263 second response time [16:50:20] RECOVERY - Apache HTTP on mw1186 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 808 bytes in 8.246 second response time [16:51:07] (03CR) 10Manybubbles: ">Hmm, looks good, but what's the reason to use logrotate over the built in dailyRollingFile (log4j?) setup?" [operations/puppet] - 10https://gerrit.wikimedia.org/r/82623 (owner: 10Manybubbles) [16:53:09] cmjohnson1: Don't suppose we keep ram around in EQIAD do we? [16:53:31] Wanting to bump up memory in the tmh100* boxes... Which is going to need a procurement ticket either way I guess [16:53:33] we have some spare. [16:54:10] PROBLEM - Apache HTTP on mw1175 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:54:30] PROBLEM - Apache HTTP on mw1209 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:54:55] yes, put in a procurement ticket but before robh actsf on it I will see what I have.....iirc its 4GB [16:55:17] thanks. need to work out a sensible upgrade amount first [16:55:32] RECOVERY - Apache HTTP on mw1209 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 808 bytes in 7.507 second response time [16:56:10] RECOVERY - Apache HTTP on mw1175 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 808 bytes in 7.223 second response time [16:56:39] its also older systems [16:56:43] those are 610s [16:56:52] so maybe have some spare in tampa for them [16:56:57] (off 610 decoms) [17:00:58] !log demon synchronized php-1.22wmf17/extensions/CirrusSearch/CirrusSearchSearcher.php 'Bug 54167' [17:01:01] Logged the message, Master [17:01:14] RobH, apergos: speaking of which, IIRC the number of jobs run on standard jobrunners was reduced at some point to prevent video scalers from crashing them; now that those jobs were moved to their machines, can't jobrunners be made a bit busier again? (<-- greg-g) [17:01:43] Nemo_bis: good question :) [17:02:12] !log demon synchronized wmf-config/InitialiseSettings.php [17:02:12] (03PS3) 10coren: Tool Labs: package fabric [operations/puppet] - 10https://gerrit.wikimedia.org/r/84287 (owner: 10DrTrigon) [17:02:14] dunno, i know i went to add more jobrunner servers at one point [17:02:19] and was told more servers for that causes issues [17:02:32] which i think makes that question more valid. [17:02:47] (since more servers was bad, adjusting the current servers may be good, but will have to ask) [17:02:54] too bad the dude who told me that doesnt work here now. [17:02:56] =P [17:03:25] :/ [17:03:56] (03CR) 10coren: [C: 032] "Is package addition." [operations/puppet] - 10https://gerrit.wikimedia.org/r/84287 (owner: 10DrTrigon) [17:04:25] ah, right, there was also the DB overload issue Asher identified [17:04:53] does someone have any idea who's supposed to answer such questions nowadays? :| [17:05:04] springle is the db guru now [17:05:21] PROBLEM - Apache HTTP on mw1186 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:05:21] apart from domas in the sky of course [17:05:28] * greg-g looks up [17:05:42] of course there's also the trial and error approach :p [17:05:49] Or the no approach [17:06:21] RECOVERY - Apache HTTP on mw1186 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 808 bytes in 7.925 second response time [17:06:30] (03PS3) 10coren: Tool Labs: package libav-tools [operations/puppet] - 10https://gerrit.wikimedia.org/r/84286 (owner: 10DrTrigon) [17:07:02] !log demon synchronized php-1.22wmf17/extensions/CirrusSearch [17:07:06] Logged the message, Master [17:07:21] (03CR) 10coren: [C: 032] Tool Labs: package libav-tools [operations/puppet] - 10https://gerrit.wikimedia.org/r/84286 (owner: 10DrTrigon) [17:07:39] (03PS3) 10coren: Tool Labs: package poppler-utils [operations/puppet] - 10https://gerrit.wikimedia.org/r/84269 (owner: 10DrTrigon) [17:09:15] (03CR) 10coren: [C: 032] Tool Labs: package poppler-utils [operations/puppet] - 10https://gerrit.wikimedia.org/r/84269 (owner: 10DrTrigon) [17:09:30] (03PS3) 10coren: Tool Labs: package libimage-exiftool-perl [operations/puppet] - 10https://gerrit.wikimedia.org/r/84236 (owner: 10DrTrigon) [17:10:56] !log demon synchronized wmf-config/InitialiseSettings.php [17:11:16] (03CR) 10coren: [C: 032] Tool Labs: package libimage-exiftool-perl [operations/puppet] - 10https://gerrit.wikimedia.org/r/84236 (owner: 10DrTrigon) [17:16:27] (03PS2) 10Akosiaris: 'gmond' is not a Puppet keyword [operations/puppet] - 10https://gerrit.wikimedia.org/r/84191 (owner: 10Ori.livneh) [17:16:59] (03PS4) 10coren: Tool Labs: packages libdmtx0a, libdmtx-dev [operations/puppet] - 10https://gerrit.wikimedia.org/r/84231 (owner: 10DrTrigon) [17:17:50] (03CR) 10Akosiaris: [C: 032] "LGTM" [operations/puppet] - 10https://gerrit.wikimedia.org/r/84191 (owner: 10Ori.livneh) [17:18:25] (03CR) 10coren: [C: 032] "With a tweak: -dev package added to dev_environ instead of exec_environ." [operations/puppet] - 10https://gerrit.wikimedia.org/r/84231 (owner: 10DrTrigon) [17:19:59] Reedy: yeah, until a password reminder takes a day or hhvm solves all our problems [17:24:01] http://ganglia.wikimedia.org/latest/?r=year&cs=&ce=&s=by+name&c=Jobrunners%2520eqiad&tab=m&vn= [17:24:19] CPU use, you could almost triple the running capacity [17:24:23] *CPU wise [17:26:41] yep, main bottleneck is database here AFAIK [17:27:22] PROBLEM - RAID on searchidx1001 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [17:27:36] though the recent anti-recursive-refreshes fixes by Tim may have alleviated the problem [17:27:50] * Nemo_bis just throwing completely random guesses [17:28:20] PROBLEM - Disk space on db1045 is CRITICAL: DISK CRITICAL - free space: / 0 MB (0% inode=80%): [17:29:16] hmmm https://gerrit.wikimedia.org/r/#/c/57030/ [17:29:30] RECOVERY - RAID on searchidx1001 is OK: OK: State is Optimal, checked 4 logical device(s) [17:29:30] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:30:04] raising iprioprocs might be enough to fix https://bugzilla.wikimedia.org/show_bug.cgi?id=43936 [17:30:05] how is memory on those? [17:30:20] RECOVERY - Disk space on db1045 is OK: DISK OK [17:30:55] use looks to be around 10% [17:31:01] with buffer and cache it goes to 75% [17:31:30] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 9.835 second response time [17:36:00] PROBLEM - SSH on searchidx1001 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:37:00] RECOVERY - SSH on searchidx1001 is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1.1 (protocol 2.0) [17:37:30] PROBLEM - RAID on searchidx1001 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [17:38:31] RECOVERY - RAID on searchidx1001 is OK: OK: State is Optimal, checked 4 logical device(s) [17:41:15] you can hardly see the effects of https://git.wikimedia.org/commitdiff/operations%2Fpuppet.git/b871a0362844a705e48b7ec4ced81e1227adca5c on https://ganglia.wikimedia.org/latest/?r=custom&cs=04%2F01%2F2013+00%3A00+&ce=04%2F14%2F2013+00%3A00+&m=cpu_report&s=by+name&c=Jobrunners+eqiad&h=&host_regex=&max_graphs=0&tab=m&vn=&sh=1&z=small&hc=4 [17:53:12] (03PS2) 10Diederik: Linkify references to Analytics Mingle projects. [operations/puppet] - 10https://gerrit.wikimedia.org/r/84338 [17:54:44] (03PS3) 10Akosiaris: Remove redundant operatingsystem check for Ubuntu [operations/puppet] - 10https://gerrit.wikimedia.org/r/84104 (owner: 10Yuvipanda) [17:55:22] (03CR) 10Akosiaris: [C: 032] "LGTM" [operations/puppet] - 10https://gerrit.wikimedia.org/r/84104 (owner: 10Yuvipanda) [17:59:45] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:00:40] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 7.527 second response time [18:02:33] akosiaris: thanks for merging :) [18:03:26] YuviPanda: you are welcome. More to come :-) [18:03:30] :D [18:03:56] akosiaris: there are two classes used for bastions. Should be consolidated to one, hopefully - but I'm afraid to touch it because if it breaks then it locks people out... [18:04:51] YuviPanda: be bold :-D [18:04:56] hehe :D [18:05:36] root/password1234 works fine [18:06:03] password1245 is clearly more secure [18:06:45] duh. I also like password1243 (I have actually seen this btw) [18:07:22] !log Starting import of some old EL data into the 'log' database on db1047. [18:07:26] Logged the message, Master [18:11:36] YuviPanda: kind of confused about this one. https://gerrit.wikimedia.org/r/#/c/84105/1. Won't that add mosh everywhere (not just bastion hosts)? [18:11:45] checking [18:11:57] akosiaris: gah, you're right [18:12:05] (03PS2) 10Faidon Liambotis: Increase socket timeout allowance for Icinga Puppetmaster HTTPS check [operations/puppet] - 10https://gerrit.wikimedia.org/r/83360 (owner: 10Ori.livneh) [18:12:06] akosiaris: hmm, or maybe not [18:12:30] (03CR) 10Faidon Liambotis: [C: 032] "Sure, what the hell." [operations/puppet] - 10https://gerrit.wikimedia.org/r/83360 (owner: 10Ori.livneh) [18:12:57] akosiaris: i guess it should be in ssh::bastion [18:13:04] akosiaris: let me fix that [18:13:07] :-) [18:13:12] (03PS2) 10Yuvipanda: Add mosh to labs bastions [operations/puppet] - 10https://gerrit.wikimedia.org/r/84105 [18:13:45] (03CR) 10Faidon Liambotis: [V: 032] Increase socket timeout allowance for Icinga Puppetmaster HTTPS check [operations/puppet] - 10https://gerrit.wikimedia.org/r/83360 (owner: 10Ori.livneh) [18:14:10] (03PS3) 10Yuvipanda: Add mosh to labs bastions [operations/puppet] - 10https://gerrit.wikimedia.org/r/84105 [18:14:10] akosiaris: updated! [18:14:28] (03CR) 10Dzahn: [C: 032] Add elwikivoyage logo [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/83989 (owner: 10Jalexander) [18:15:22] akosiaris: thanks for catching thtat :) [18:15:41] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:16:05] !log dzahn synchronized ./wmf-config/InitialiseSettings.php [18:16:08] Logged the message, Master [18:16:40] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 8.991 second response time [18:16:45] !log reedy rebuilt wikiversions.cdb and synchronized wikiversions files: wikimedia, special, wikinews and wikivoyage to 1.22wmf16 [18:16:48] Logged the message, Master [18:17:33] !log dzahn synchronized ./wmf-config/InitialiseSettings.php [18:17:40] !log custom logo for el.wikivoyage [18:17:43] Logged the message, Master [18:18:03] (03PS4) 10Akosiaris: Add mosh to labs bastions [operations/puppet] - 10https://gerrit.wikimedia.org/r/84105 (owner: 10Yuvipanda) [18:18:14] (03CR) 10Dzahn: "the custom logo is live" [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/83989 (owner: 10Jalexander) [18:19:09] (03CR) 10Akosiaris: [C: 032] "LGTM" [operations/puppet] - 10https://gerrit.wikimedia.org/r/84105 (owner: 10Yuvipanda) [18:19:15] wohoo, thank you :) [18:20:17] YuviPanda: :-D. [18:20:36] paravoid: i merged the HTTPS timeout increase on sockpuppet [18:21:04] (03PS3) 10Akosiaris: Selectively enable SSL on dynamicproxy via a class param [operations/puppet] - 10https://gerrit.wikimedia.org/r/84215 (owner: 10Yuvipanda) [18:21:14] (03PS3) 10Dzahn: Fold wikiquote rewrites into one. [operations/apache-config] - 10https://gerrit.wikimedia.org/r/53403 (owner: 10JanZerebecki) [18:22:17] (03CR) 10Akosiaris: [C: 032] "LGTM" [operations/puppet] - 10https://gerrit.wikimedia.org/r/84215 (owner: 10Yuvipanda) [18:22:24] ah... small and easy to review changes. What i just like :-) [18:22:50] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:24:10] hehe :D [18:25:30] PROBLEM - Disk space on vanadium is CRITICAL: DISK CRITICAL - free space: / 2574 MB (2% inode=95%): [18:25:51] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 5.535 second response time [18:25:58] vanadium is my fault, fixing [18:27:53] !log reedy rebuilt wikiversions.cdb and synchronized wikiversions files: Everything non 'pedia to 1.22wmf17 [18:27:57] Logged the message, Master [18:28:29] (03PS1) 10Reedy: Everything non 'pedia to 1.22wmf17 [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/84352 [18:28:42] RECOVERY - Disk space on vanadium is OK: DISK OK [18:28:42] (03CR) 10Reedy: [C: 032] Everything non 'pedia to 1.22wmf17 [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/84352 (owner: 10Reedy) [18:28:52] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:28:55] (03Merged) 10jenkins-bot: Everything non 'pedia to 1.22wmf17 [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/84352 (owner: 10Reedy) [18:29:50] PROBLEM - Apache HTTP on mw1175 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:30:50] !log reedy synchronized php-1.22wmf17/extensions/Wikibase [18:30:50] RECOVERY - Apache HTTP on mw1175 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 808 bytes in 6.763 second response time [18:30:53] Logged the message, Master [18:31:41] !log reedy synchronized php-1.22wmf17/extensions/DataValues/ [18:31:44] Logged the message, Master [18:36:28] (03PS1) 10RobH: fixing uldfo dhcpd settings [operations/puppet] - 10https://gerrit.wikimedia.org/r/84354 [18:36:34] (03CR) 10Dzahn: [C: 031] "works for me. tested and see no difference to the separate rules" [operations/apache-config] - 10https://gerrit.wikimedia.org/r/53403 (owner: 10JanZerebecki) [18:37:50] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 2.968 second response time [18:38:06] (03CR) 10Dzahn: [C: 032] "works for me. tested and see no difference to the separate rules" [operations/apache-config] - 10https://gerrit.wikimedia.org/r/53403 (owner: 10JanZerebecki) [18:38:15] !log reedy synchronized php-1.22wmf16/extensions/VisualEditor [18:38:18] Logged the message, Master [18:38:36] That was quick [18:39:49] (03PS1) 10Nikerabbit: ULS eventlogging phase 2 [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/84355 [18:39:59] RobH: a brief detour to the opposite end of the hardware requirement spectrum -- any thoughts re: 5777? [18:40:00] RECOVERY - MySQL Replication Heartbeat on db1007 is OK: OK replication delay 0 seconds [18:40:01] RECOVERY - MySQL Slave Delay on db1007 is OK: OK replication delay 0 seconds [18:40:31] RECOVERY - MySQL Slave Delay on db1005 is OK: OK replication delay 0 seconds [18:40:31] RECOVERY - MySQL Replication Heartbeat on db1005 is OK: OK replication delay -0 seconds [18:40:32] you could probably run statsd on a TI-83 graphing calculator [18:40:34] (03Abandoned) 10RobH: fixing uldfo dhcpd settings [operations/puppet] - 10https://gerrit.wikimedia.org/r/84354 (owner: 10RobH) [18:40:50] RECOVERY - search indices - check lucene status page on search20 is OK: HTTP OK: HTTP/1.1 200 OK - 60075 bytes in 0.112 second response time [18:40:50] <^d> Nemo_bis: Ping :) [18:40:53] but then you wouldn't have anything to play mafia wars on [18:41:04] ori-l: make RobH build it from raspberry pi?:) [18:41:21] you probably could [18:41:31] ori-l: hrmm [18:41:49] yea i can find something i think [18:42:22] ori-l: mark wanted this isolated from things, or he wanted to keep the logging server isolated? [18:42:23] cool [18:42:45] cuz either could be true, ticket isnt clear to me. [18:43:04] (matters, since if it is logging stays isolated, statd can still live with something elsewhere) [18:43:10] ^d: pong [18:43:19] but if its statd stays isolated, then its own server kind of thing [18:43:29] sorry to be daft, but what does it mean for logging to stay isolated vs statsd being isolated? [18:43:31] which is non ideal when possible, so would want to detail why it has to stay isolated [18:43:43] ori-l: So can statD not be on the logging server because its statd [18:43:45] <^d> Nemo_bis: Howdy. I heard you guys are looking to play with Cirrus on it.wiktionary. How do you feel about us turning it on as a secondary backend this afternoon? [18:43:45] I wanted to comment on that ticket today [18:43:49] but since you're discussing it [18:43:50] or becuase nothing else should be on logging server? [18:43:51] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:43:56] ^d: the sooner the better! [18:43:58] let's find a box for graphite and statsd in eqiad [18:44:03] not just statsd [18:44:04] ^d: even primary if you want :) [18:44:08] RobH: ^ [18:44:20] paravoid: That was my question, so the concern is to not place things on the logging box [18:44:21] you're going to need a lot more raspeberry pis [18:44:23] ? [18:44:29] what logging box? [18:44:31] lots of people have access to the logging box [18:44:33] ^d: that way it's easier for them to test without URL parameters; the internal search is of little use now to the point most use google only [18:44:34] fluorine [18:44:39] they don't need access to statsd/graphite [18:44:40] oh [18:44:45] <^d> Nemo_bis: Once it's secondary and the index is built turning it to primary is easy :) [18:44:48] ok, so it is indeed, keep it off LOGGING box [18:44:51] yes [18:44:51] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 7.330 second response time [18:44:53] not statd must be alone. [18:44:56] indeed [18:45:00] thats much easier to understand now, thx [18:45:09] paravoid: that sounds feasible to me, your solution. [18:45:14] ^d: yeah, hence why I don't feel guilty asking you do switch to it quickly ;) but sure, doesn't have to be immediately [18:45:17] thanks, sorry for the confusion [18:45:30] ori-l: no worries, just wanna make sure everyone is happy =] [18:45:39] <^d> Nemo_bis: Ok, so we've got a deployment window at 23:00 UTC today (4pm PST). [18:45:41] statsd proxies to graphite and doesn't do much besides that [18:45:48] graphite is still all in tampa and needs to be moved [18:45:53] (03PS1) 10Akosiaris: Fix typo in backing up /var/vmail [operations/puppet] - 10https://gerrit.wikimedia.org/r/84358 [18:45:55] so i'll expand this ticket to include a server for graphite as well [18:45:58] yep [18:45:59] as they will share. [18:46:06] I have the RT starred to reply this [18:46:06] and i'll get one assigned in a bit [18:46:10] cool [18:46:12] but this is even easier :) [18:46:23] i don't think we should copy the whisper files over just yet [18:46:37] ori-l: yep, sure, let's work on that [18:46:54] but let's plan the hw for moving it eventually [18:47:16] ^d: ok, I assume it's not a problem if we are not around at the time to test it immediately? :) [18:47:28] (03CR) 10Akosiaris: [C: 032] Fix typo in backing up /var/vmail [operations/puppet] - 10https://gerrit.wikimedia.org/r/84358 (owner: 10Akosiaris) [18:47:34] yeah, i figure let's keep professor up just a tad longer so we can select what to port, make sure everyone's ok with the selection, then decom professor [18:47:41] <^d> Nemo_bis: No worries at all. When you guys do play with it and find bugs, we've got an extension component in BZ for it. [18:47:52] ^d: ok great [18:51:00] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:53:51] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 1.460 second response time [18:54:33] (03PS1) 10Lcarr: fixing subnet for public servers [operations/dns] - 10https://gerrit.wikimedia.org/r/84362 [18:55:07] (03CR) 10Lcarr: [C: 032] fixing subnet for public servers [operations/dns] - 10https://gerrit.wikimedia.org/r/84362 (owner: 10Lcarr) [18:56:39] (03PS1) 10Petr Onderka: Simple progress reporting for idumps create [operations/dumps/incremental] (gsoc) - 10https://gerrit.wikimedia.org/r/84363 [18:57:00] PROBLEM - LVS HTTP IPv6 on wikiversity-lb.esams.wikimedia.org_ipv6 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 325 bytes in 7.194 second response time [18:57:00] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:57:52] RECOVERY - LVS HTTP IPv6 on wikiversity-lb.esams.wikimedia.org_ipv6 is OK: HTTP OK: HTTP/1.1 200 OK - 67280 bytes in 0.687 second response time [18:58:39] ^d: community notified and instructed [19:00:29] (03CR) 10Kaldari: [C: 032] Enable XFO: SAMEORIGIN for enwiki [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/82751 (owner: 10CSteipp) [19:00:44] (03Merged) 10jenkins-bot: Enable XFO: SAMEORIGIN for enwiki [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/82751 (owner: 10CSteipp) [19:03:12] <^d> Nemo_bis: Cool. Did you have a link to BZ? This is a link to file bugs directly for Cirrus. https://bugzilla.wikimedia.org/enter_bug.cgi?product=MediaWiki%20extensions&component=CirrusSearch [19:03:22] PROBLEM - LVS HTTPS IPv6 on wikimedia-lb.esams.wikimedia.org_ipv6 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:04:20] RECOVERY - LVS HTTPS IPv6 on wikimedia-lb.esams.wikimedia.org_ipv6 is OK: HTTP OK: HTTP/1.1 200 OK - 93309 bytes in 2.376 second response time [19:05:36] !log reedy synchronized php-1.22wmf17/extensions/VisualEditor [19:05:40] Logged the message, Master [19:10:01] PROBLEM - RAID on searchidx1001 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [19:11:02] (03CR) 10Faidon Liambotis: "That's RT 5777 now." [operations/puppet] - 10https://gerrit.wikimedia.org/r/83603 (owner: 10Ori.livneh) [19:12:39] thanks paravoid [19:12:44] (03CR) 10Faidon Liambotis: "That's RT 5762." [operations/puppet] - 10https://gerrit.wikimedia.org/r/83201 (owner: 10Ori.livneh) [19:13:10] x2 [19:13:37] (03CR) 10Faidon Liambotis: "Let's wait to see if the OpenJDK 7 works to see if this is actually needed now?" [operations/puppet] - 10https://gerrit.wikimedia.org/r/83960 (owner: 10Ottomata) [19:13:51] RECOVERY - RAID on searchidx1001 is OK: OK: State is Optimal, checked 4 logical device(s) [19:14:01] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 8.802 second response time [19:14:46] (03PS1) 10Jforrester: Enable VisualEditor on "phase 2" Wikipedias (logged-in users only) [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/84369 [19:14:47] (03PS1) 10Jforrester: Enable VisualEditor on "phase 2" Wikipedias (all users) [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/84370 [19:14:54] (03CR) 10Ottomata: "Hm, sounds good. The current role::analytics::dclass sets up this symlink, so we'll have to remove it to make sure it works." [operations/puppet] - 10https://gerrit.wikimedia.org/r/83960 (owner: 10Ottomata) [19:17:03] (03CR) 10Faidon Liambotis: "LGTM, although the commit message could be a bit more descriptive :)" [operations/puppet] - 10https://gerrit.wikimedia.org/r/84091 (owner: 10Lcarr) [19:17:19] ottomata: http://search.maven.org/remotecontent?filepath=com/yammer/metrics/metrics-graphite/2.2.0/metrics-graphite-2.2.0.jar and http://search.maven.org/remotecontent?filepath=com/yammer/metrics/metrics-ganglia/2.2.0/metrics-ganglia-2.2.0.jar [19:17:28] danke [19:26:30] (03PS1) 10ArielGlenn: newer eqiad servers really have 16 cores, give them 60 maxclients [operations/puppet] - 10https://gerrit.wikimedia.org/r/84371 [19:28:30] RECOVERY - Disk space on professor is OK: DISK OK [19:29:44] mark: ^^ what do you think? [19:30:00] i cleared the nav timing browser stats from graphite; using ganglia for those anyhow. that freed up a lot of space. [19:30:10] apergos: why not a bunch more than 60? [19:30:18] they're like 2x as powerful right? also more memory [19:30:23] you can start slowly of course [19:30:27] but in the end it should be more than 60 I think [19:30:27] i'm going to disable the writing of that data to graphite so it doesn't just fill up again [19:30:55] tbh I was gonna watch em all for a day or two [19:31:11] but if you want to give me a more agressive starting point, fire away [19:32:39] (03PS1) 10Manybubbles: itwiktionary should use CirrusSearch as an option. [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/84376 [19:34:16] ok well I'm just going to merge this now so we can get this party started, if you or anyone wants to up it sooner than me cause the servers are looking bored, that's fine [19:34:41] (03PS2) 10Ottomata: Setup logrotation for elasticsearch. [operations/puppet] - 10https://gerrit.wikimedia.org/r/82623 (owner: 10Manybubbles) [19:34:46] (03CR) 10Ottomata: [C: 032 V: 032] Setup logrotation for elasticsearch. [operations/puppet] - 10https://gerrit.wikimedia.org/r/82623 (owner: 10Manybubbles) [19:34:59] (03CR) 10ArielGlenn: [C: 032] newer eqiad servers really have 16 cores, give them 60 maxclients [operations/puppet] - 10https://gerrit.wikimedia.org/r/84371 (owner: 10ArielGlenn) [19:35:03] (03PS1) 10Akosiaris: Adding ganglia/graphite backends to kafka [operations/debs/kafka] (debian) - 10https://gerrit.wikimedia.org/r/84399 [19:35:17] (03CR) 10Akosiaris: [C: 032 V: 032] Adding ganglia/graphite backends to kafka [operations/debs/kafka] (debian) - 10https://gerrit.wikimedia.org/r/84399 (owner: 10Akosiaris) [19:37:26] (03PS1) 10Ottomata: Adding kafka-ganglia jar. [operations/debs/kafka] (debian) - 10https://gerrit.wikimedia.org/r/84411 [19:37:59] !log Disabled csw1-esams:e8/1 [19:38:03] Logged the message, Master [19:38:15] (03CR) 10Ottomata: [C: 032 V: 032] Adding kafka-ganglia jar. [operations/debs/kafka] (debian) - 10https://gerrit.wikimedia.org/r/84411 (owner: 10Ottomata) [19:39:00] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:40:02] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 7.681 second response time [19:40:40] come on stafford [19:40:42] poor thing [19:41:31] Reedy: other than the [[How to deploy code]] page, do we have any "training-ish" materials? I ask because someone is being trained up on deploying and they asked me :) [19:41:58] (03PS1) 10Ottomata: Installing kafka-ganglia-1.0.0.jar and using it in CLASSPATH. [operations/debs/kafka] (debian) - 10https://gerrit.wikimedia.org/r/84412 [19:42:12] akosiaris: ^ [19:43:03] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:43:13] (03CR) 10Akosiaris: [C: 032 V: 032] "LGTM" [operations/debs/kafka] (debian) - 10https://gerrit.wikimedia.org/r/84412 (owner: 10Ottomata) [19:43:54] grrrit-wm: Looks like https://wikitech.wikimedia.org/wiki/Heterogeneous_deployment has bits of info How to deploy code doesn't. Then there's my condensed version https://wikitech.wikimedia.org/wiki/Heterogeneous_deployment_v2 [19:46:35] Reedy: ty [19:49:01] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 8.788 second response time [19:57:00] PROBLEM - search indices - check lucene status page on search20 is CRITICAL: HTTP CRITICAL: HTTP/1.1 200 OK - pattern found - 60051 bytes in 0.122 second response time [19:59:49] (03CR) 10Diederik: "the version/vendor of java is immaterial to the problem being solved, please merge." [operations/puppet] - 10https://gerrit.wikimedia.org/r/83960 (owner: 10Ottomata) [20:03:01] (03PS1) 10Dzahn: - fix wrong sum of "good" articles for Wikipedias - rename column articles to "good" in grand total for consistency - make footer table more readable by using heredoc [operations/debs/wikistats] - 10https://gerrit.wikimedia.org/r/84415 [20:05:13] (03CR) 10Faidon Liambotis: "That's incorrect. The symlinks are only needed if the Java version in the system doesn't support multiarch and if the symlinks aren't need" [operations/puppet] - 10https://gerrit.wikimedia.org/r/83960 (owner: 10Ottomata) [20:12:00] !log re-enable allow_attachment_display on bugzilla [20:12:04] Logged the message, Master [20:15:00] (03CR) 10Lcarr: [C: 032] fixing up the ntp ip list [operations/puppet] - 10https://gerrit.wikimedia.org/r/84091 (owner: 10Lcarr) [20:23:25] !log demon synchronized php-1.22wmf17/extensions/CirrusSearch [20:23:29] Logged the message, Master [20:23:35] (03CR) 10Dzahn: [C: 032] - fix wrong sum of "good" articles for Wikipedias - rename column articles to "good" in grand total for consistency - make footer table more [operations/debs/wikistats] - 10https://gerrit.wikimedia.org/r/84415 (owner: 10Dzahn) [20:25:51] PROBLEM - NTP peers on linne is CRITICAL: NTP CRITICAL: Server not synchronized, Offset unknown [20:27:50] RECOVERY - NTP peers on linne is OK: NTP OK: Offset -0.006828 secs [20:28:36] (03CR) 10Ottomata: "I will check jenkins+dclass+java version stuff before merging this." [operations/puppet] - 10https://gerrit.wikimedia.org/r/83960 (owner: 10Ottomata) [20:28:57] (03CR) 10Chad: [C: 031] itwiktionary should use CirrusSearch as an option. [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/84376 (owner: 10Manybubbles) [20:29:32] ^d: who should I add as another reviewer or this? [20:29:51] <^d> I'll +2 before the window, just wanted to let you know I thought it was ok :) [20:30:00] <^d> Don't want it to merge until then tho [20:30:26] cool [20:39:45] PROBLEM - NTP peers on dobson is CRITICAL: NTP CRITICAL: Server not synchronized, Offset unknown [20:41:25] RECOVERY - check_job_queue on fenari is OK: JOBQUEUE OK - all job queues below 10,000 [20:41:45] RECOVERY - NTP peers on dobson is OK: NTP OK: Offset -0.001074 secs [20:44:36] PROBLEM - check_job_queue on fenari is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [20:49:36] ottomata: and of course the 4 new kafka patches break compilation... will look into it [20:49:59] /me sighs and thanks akosiaris ! [20:50:22] woot! [20:50:26] (03CR) 10Jforrester: [C: 04-1] "Please do not merge before 30 September 2013." [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/84370 (owner: 10Jforrester) [20:50:40] (03CR) 10Jforrester: [C: 04-1] "Please do not merge before 24 September 2013." [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/84369 (owner: 10Jforrester) [20:50:52] when are they gonna release 0.8???? we want 0.8!!! we want 0.8 !!! we want 0.8!!! [20:51:06] YuviPanda: I hear you have an ssl proxy somewhere? [20:51:11] can I use that somehow? [20:53:41] Ryan_Lane: i filed https://bugzilla.wikimedia.org/show_bug.cgi?id=54185, feel free to add details/hints [20:54:02] ottomata: hey! [20:54:04] ottomata: on labs, yeah [20:54:06] ottomata: sure! [20:54:09] oops, i meant to ask that in analytics chanel [20:54:13] heh, word goes around fast ;) [20:54:17] drdee just told me you are already using it for metrics! [20:54:18] i didn't know! [20:54:24] drdee asked me to make a change to ssl proxy stuff for metrcis [20:54:34] and I'm all "My proxy doesn't work anymore!" [20:54:39] ottomata: metrics.wmflabs.org is already ssl [20:54:50] i know, i had my own ssl proxy running on the wikimetrics instacne [20:55:15] aha! [20:55:25] hm [20:55:26] ottomata: this proxy does its own ssl termination [20:55:29] yeah, that's good [20:55:31] i like yours better [20:55:31] ottomata: and hence will only pass back http to you [20:55:44] (moving to #analytics) [20:57:06] ^d: faster! [20:59:13] (03PS4) 10Dzahn: Move logs to /var/log/mediawiki [operations/puppet] - 10https://gerrit.wikimedia.org/r/83574 (owner: 10Reedy) [21:02:13] (03PS5) 10Dzahn: Move logs to /var/log/mediawiki [operations/puppet] - 10https://gerrit.wikimedia.org/r/83574 (owner: 10Reedy) [21:04:16] (03PS1) 10Ori.livneh: Make StatsD module backend-agnostic [operations/puppet] - 10https://gerrit.wikimedia.org/r/84425 [21:07:35] (03PS2) 10Ori.livneh: Make StatsD module backend-agnostic; make NavigationTiming instance Ganglia-only [operations/puppet] - 10https://gerrit.wikimedia.org/r/84425 [21:08:16] !log demon synchronized php-1.22wmf17/extensions/CirrusSearch [21:09:51] !log demon synchronized php-1.22wmf17/extensions/CirrusSearch [21:09:59] (03PS1) 10Bsitu: Enable Echo and Thanks on various wikis [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/84427 [21:10:42] (03CR) 10Bsitu: [C: 04-2] Enable Echo and Thanks on various wikis [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/84427 (owner: 10Bsitu) [21:13:57] ^d: Guess you're aware of the fatals? [21:14:08] Catchable fatal error: Argument 2 passed to CirrusSearchTextSanitizer::getSantizedTextFromTitle() must be an instance of ParserOutput, boolean given, called in /usr/local/apache/common-local/php-1.22wmf17/extensions/CirrusSearch/CirrusSearch.body.php on line 72 and defined in /usr/local/apache/common-local/php-1.22wmf17/extensions/CirrusSearch/CirrusSearchTextSanitizer.php on line 35 [21:14:20] <^d> Yeah, fixed now. [21:30:18] (03PS1) 10Jforrester: Update how VisualEditor's experimental config works [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/84433 [21:33:07] (03CR) 10Esanders: [C: 031] Update how VisualEditor's experimental config works [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/84433 (owner: 10Jforrester) [21:37:32] (03CR) 10MarkTraceur: [C: 031] "(1 comment)" [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/84433 (owner: 10Jforrester) [21:50:22] (03PS1) 10Chad: Move itwiktionary to default Cirrus [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/84434 [21:50:40] (03CR) 10QChris: "Blocking change has merged:" [operations/puppet] - 10https://gerrit.wikimedia.org/r/84222 (owner: 10QChris) [21:50:46] greg-g, what's the depl like today? [21:51:09] sunny with a chance of rain [21:55:27] ^d: you still using the LD today? [21:55:40] <^d> Yessir. [21:55:55] <^d> Starts in an hour, right? [21:56:55] ^d: correct [21:57:03] yuri_k: what's up? [21:57:52] greg-g, thinking of pushing something out, any lighting slots? [21:58:05] yuri_k: depends, what is it? :) [21:58:27] greg-g, breaking Zero, as usual, sync diring :) [22:11:07] greg-g, so when can i break it :) [22:17:29] yuri_k: what's the issue being solved? (short version) (and sorry, wa sin a meeting) [22:32:34] greg-g, syncing up some weird changes in mobile ( MaxSem patch ), plus our own code change for dom parsing. [22:35:32] yuri_k: ok, just curious why it can't wait until Wed. :) [22:36:01] greg-g, because wed we will have bigger (and more demaging) patch :) [22:36:12] well then [22:36:14] :P [22:36:23] yuri_k: coordinate with Chad, he [22:36:27] greg-g, which depends on the stability on this one :) [22:36:27] 's also using the LD [22:36:32] * greg-g nods [22:38:05] ^d, are you deploying? [22:38:13] <^d> In 20 minutes, yes. [22:38:50] ^d, oki, will wait for you to finish [22:39:12] btw, if you want to go earlier, i don't think greg-g has anoyone planned, right? [22:40:39] ^d: yuri_k yeah, if ya'll are ready, go ahead and start so you two can finish before 5 :) [22:40:42] <^d> I can go ahead. I was going to let Nik do the deploy but he doesn't seem to be around. [22:41:17] (03CR) 10Chad: [C: 032] itwiktionary should use CirrusSearch as an option. [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/84376 (owner: 10Manybubbles) [22:41:29] (03Merged) 10jenkins-bot: itwiktionary should use CirrusSearch as an option. [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/84376 (owner: 10Manybubbles) [22:41:31] :) [22:42:05] i can hear happy child's laughter from the server room... [22:42:26] That's better than unhappy cild [22:42:28] *child [22:43:23] !log demon synchronized wmf-config/InitialiseSettings.php [22:43:32] Logged the message, Master [22:43:43] Reedy, an unhappy child will sit sadly in the corner, while the happy one might reconnect a few cables... in the best case [22:46:30] ^d, greg-g wants us to deploy at the same time... [22:46:47] <^d> Hehe, I've already pushed my stuff live. [22:46:52] what?! not same time :) [22:47:04] bummer, i'm late to the party [22:47:12] ^d, are you done? [22:47:56] <^d> Well the reindexing is going to take some time, but I'm done pushing any code/config changes yes :) [22:48:03] cool [22:48:36] <^d> !log ES: indexing itwiktionary from terbium [22:48:40] Logged the message, Master [22:49:01] oki, i will push out our stuff shortly [22:49:29] <^d> Whee, > 30 pages/s now. [22:49:31] <^d> Much better. [22:53:51] <^d> Not joining against text and not parsing twice made for nice improvements. [22:54:46] Reedy, are all the fatalmonitor entries ok? [22:57:19] ^d: I am here! [22:57:26] just snuck away from the beginnings of bed time [22:57:33] <^d> Haha [22:57:41] <^d> itwiktionary's almost indexed. [22:57:47] oh! [22:57:48] <^d> We're getting nice index speeds again. [22:57:58] no waiting for me? [22:57:59] :( [22:58:08] * manybubbles sad [22:58:12] deploying... [22:58:23] <^d> You weren't here, and yuri_k also wanted to deploy. [22:58:31] <^d> So we got bumped up [22:58:36] ah! [22:58:38] manybubbles, do you have 1 index per wiki? [22:58:55] MaxSem: two - one for the content articles and one for the rest. [22:59:45] ^d: I suppose any time you pick to deploy has to be inconvenient for someone. I mean, it is quite late for Europe. [23:00:11] !log yurik synchronized php-1.22wmf16/extensions/ZeroRatedMobileAccess/ [23:00:14] Logged the message, Master [23:00:57] ^d: so what kinda speeds were you able to get? [23:01:23] <^d> Indexed a total of 31169 pages at 47/second; Indexed a total of 38730 pages at 47/second [23:01:28] <^d> ^ 2 of the threads that finished [23:01:35] <^d> 2 others almost done [23:01:37] 47/second is really not bad at all [23:02:36] <^d> thread 3: Indexed a total of 33799 pages at 37/second [23:03:00] <^d> With ~14k pages to go on the last thread [23:03:35] !log yurik synchronized php-1.22wmf17/extensions/ZeroRatedMobileAccess/ [23:03:38] Logged the message, Master [23:03:53] ^d: cool! I never checked search on wiktionary before, wow! [23:04:06] greg-g, done deploying... I wonder why the sync-dir is so fast now... did someone break it? :) [23:04:07] http://en.wiktionary.org/w/index.php?search=clobber&title=Special%3ASearch&fulltext=1 [23:04:11] see the first entry [23:04:55] <^d> Same result for Cirrus? [23:05:17] that is enwiktionary [23:05:37] itwiktionary does look better at first blush [23:05:43] we're very did you mean happy [23:05:44] still [23:05:55] but I think that is one of the elasticsearch 0.90.4 fixes [23:06:28] <^d> Compare https://it.wiktionary.org/w/index.php?search=implicitamente&title=Speciale%3ARicerca&fulltext=1 and https://it.wiktionary.org/w/index.php?search=implicitamente&title=Speciale%3ARicerca&fulltext=1&srbackend=CirrusSearch [23:06:32] <^d> way nicer in the latter. [23:08:59] I think so. but I can't read it. [23:09:19] is this a send email to ambasadors thing or just nemo? [23:10:53] We call that a false dichotomy. ;-) [23:11:03] If it's specific to it.wiktionary, post at it.wiktionary. [23:11:09] I'm sure they have a village pump or equivalent. [23:11:21] Elsie: I could post it in English.... [23:11:25] <^d> He said he notified their pump earlier today. [23:11:27] <^d> Lemme find where. [23:11:43] manybubbles: Much to the horror of everyone everywhere, English is the international language. [23:12:02] Elsie: thank god for me [23:12:09] ikr [23:12:11] Elsie: or, rather, I thank God for that [23:12:25] <^d> https://it.wiktionary.org/wiki/Wikizionario:Bar - seems to be the right page :) [23:12:52] <^d> Ah, https://it.wiktionary.org/wiki/Wikizionario:Bar#Nuovo_motore_di_ricerca_interno [23:15:09] ^d: looks like he's given enough information [23:15:17] I'mma go help with bed time [23:18:22] (03PS2) 10MaxSem: Sync device detection with MobileFrontend [operations/puppet] - 10https://gerrit.wikimedia.org/r/83919 [23:21:59] (03PS1) 10Ottomata: Supporting arbitrary Kafka metrics_properties, rather than only the Kafka CSV metrics reporter. [operations/puppet/kafka] - 10https://gerrit.wikimedia.org/r/84460 [23:26:29] (03CR) 10Ottomata: [C: 032 V: 032] Supporting arbitrary Kafka metrics_properties, rather than only the Kafka CSV metrics reporter. [operations/puppet/kafka] - 10https://gerrit.wikimedia.org/r/84460 (owner: 10Ottomata) [23:29:16] PROBLEM - DPKG on db1037 is CRITICAL: Timeout while attempting connection [23:30:15] PROBLEM - Host db1037 is DOWN: CRITICAL - Plugin timed out after 15 seconds [23:30:40] ottomata: can you merge https://gerrit.wikimedia.org/r/#/c/84425/ for me possibly? [23:32:15] RECOVERY - DPKG on db1037 is OK: All packages OK [23:32:15] RECOVERY - Host db1037 is UP: PING OK - Packet loss = 0%, RTA = 0.84 ms [23:34:03] ordered_json!! :) [23:34:19] yep :P [23:34:22] (03CR) 10Cmcmahon: [C: 031] "I would use Flow the moment it is on beta labs" [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/82766 (owner: 10Matthias Mullie) [23:36:23] (03CR) 10Ottomata: [C: 032 V: 032] Make StatsD module backend-agnostic; make NavigationTiming instance Ganglia-only [operations/puppet] - 10https://gerrit.wikimedia.org/r/84425 (owner: 10Ori.livneh) [23:37:27] thanks! [23:45:20] (03PS1) 10Springle: prepare db1037 for s1 [operations/puppet] - 10https://gerrit.wikimedia.org/r/84464 [23:46:35] ori-l, is the graphite instance you are using different than graphite.wikimedia.org? [23:46:36] !log change timeout for watchmouse "wikimedia foundation script"-check from 8 sec to 15 sec [23:46:40] Logged the message, Master [23:47:00] ottomata: no, it's that one. [23:47:01] (03CR) 10Springle: [C: 032 V: 032] prepare db1037 for s1 [operations/puppet] - 10https://gerrit.wikimedia.org/r/84464 (owner: 10Springle) [23:47:16] well, i'm not using graphite for navigation timing as of the patch you just merged [23:47:18] only ganglia [23:47:32] aye ok [23:47:33] ganke [23:47:34] danke