[00:00:38] PROBLEM - HHVM rendering on mw1033 is CRITICAL: HTTP CRITICAL: HTTP/1.1 503 Service Unavailable - 50350 bytes in 0.008 second response time [00:00:58] PROBLEM - DPKG on mw1033 is CRITICAL: DPKG CRITICAL dpkg reports broken packages [00:01:57] PROBLEM - Apache HTTP on mw1033 is CRITICAL: HTTP CRITICAL: HTTP/1.1 503 Service Unavailable - 50350 bytes in 0.013 second response time [00:03:39] RECOVERY - HHVM rendering on mw1033 is OK: HTTP OK: HTTP/1.1 200 OK - 73250 bytes in 5.872 second response time [00:03:53] RECOVERY - DPKG on mw1033 is OK: All packages OK [00:04:57] RECOVERY - Apache HTTP on mw1033 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 440 bytes in 0.056 second response time [00:05:45] RECOVERY - HHVM processes on mw1033 is OK: PROCS OK: 1 process with command name hhvm [00:19:36] (03PS1) 10Yuvipanda: dynamicproxy: Store only one day of nginx logs [puppet] - 10https://gerrit.wikimedia.org/r/178137 [00:21:18] (03CR) 10Yuvipanda: [C: 032] dynamicproxy: Make no user agent 403 more verbose [puppet] - 10https://gerrit.wikimedia.org/r/178135 (owner: 10Yuvipanda) [00:21:31] (03CR) 10Yuvipanda: [C: 032] dynamicproxy: Store only one day of nginx logs [puppet] - 10https://gerrit.wikimedia.org/r/178137 (owner: 10Yuvipanda) [02:10:15] !log l10nupdate Synchronized php-1.25wmf10/cache/l10n: (no message) (duration: 00m 03s) [02:10:20] !log LocalisationUpdate completed (1.25wmf10) at 2014-12-08 02:10:20+00:00 [02:10:24] Logged the message, Master [02:10:26] Logged the message, Master [02:15:50] !log l10nupdate Synchronized php-1.25wmf11/cache/l10n: (no message) (duration: 00m 01s) [02:15:54] !log LocalisationUpdate completed (1.25wmf11) at 2014-12-08 02:15:53+00:00 [02:15:54] Logged the message, Master [02:15:58] Logged the message, Master [03:42:28] !log LocalisationUpdate ResourceLoader cache refresh completed at Mon Dec 8 03:42:28 UTC 2014 (duration 42m 27s) [03:42:30] Logged the message, Master [04:02:44] (03PS1) 10Springle: depool db1049 for upgrade [mediawiki-config] - 10https://gerrit.wikimedia.org/r/178142 [04:03:38] (03CR) 10Springle: [C: 032] depool db1049 for upgrade [mediawiki-config] - 10https://gerrit.wikimedia.org/r/178142 (owner: 10Springle) [04:11:03] RECOVERY - puppet last run on ms-be1009 is OK: OK: Puppet is currently enabled, last run 31 seconds ago with 0 failures [04:11:12] !log puppet fact_values hit auto_inc limit. altered table to restart from 1 to get puppet running (seems safe, but needs checking, maybe also truncate) [04:11:15] Logged the message, Master [04:11:21] RECOVERY - puppet last run on mc1013 is OK: OK: Puppet is currently enabled, last run 25 seconds ago with 0 failures [04:11:38] RECOVERY - puppet last run on mw1248 is OK: OK: Puppet is currently enabled, last run 10 seconds ago with 0 failures [04:11:48] RECOVERY - puppet last run on mw1163 is OK: OK: Puppet is currently enabled, last run 8 seconds ago with 0 failures [04:11:48] RECOVERY - puppet last run on wtp1022 is OK: OK: Puppet is currently enabled, last run 52 seconds ago with 0 failures [04:11:58] RECOVERY - puppet last run on search1015 is OK: OK: Puppet is currently enabled, last run 40 seconds ago with 0 failures [04:11:58] RECOVERY - puppet last run on elastic1011 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [04:12:07] RECOVERY - puppet last run on protactinium is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [04:12:08] RECOVERY - puppet last run on search1024 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [04:12:16] RECOVERY - puppet last run on db1055 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [04:12:24] RECOVERY - puppet last run on ms-be1007 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [04:12:25] RECOVERY - puppet last run on analytics1014 is OK: OK: Puppet is currently enabled, last run 34 seconds ago with 0 failures [04:12:25] RECOVERY - puppet last run on elastic1005 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [04:12:26] RECOVERY - puppet last run on mw1171 is OK: OK: Puppet is currently enabled, last run 25 seconds ago with 0 failures [04:12:27] RECOVERY - puppet last run on db1062 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [04:12:28] RECOVERY - puppet last run on elastic1014 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [04:12:28] RECOVERY - puppet last run on elastic1002 is OK: OK: Puppet is currently enabled, last run 49 seconds ago with 0 failures [04:12:28] RECOVERY - puppet last run on mw1023 is OK: OK: Puppet is currently enabled, last run 2 seconds ago with 0 failures [04:12:28] RECOVERY - puppet last run on mw1097 is OK: OK: Puppet is currently enabled, last run 21 seconds ago with 0 failures [04:12:28] RECOVERY - puppet last run on mc1001 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [04:12:29] RECOVERY - puppet last run on analytics1037 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [04:12:38] RECOVERY - puppet last run on mw1029 is OK: OK: Puppet is currently enabled, last run 28 seconds ago with 0 failures [04:12:38] RECOVERY - puppet last run on db2003 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [04:12:38] RECOVERY - puppet last run on mw1210 is OK: OK: Puppet is currently enabled, last run 51 seconds ago with 0 failures [04:12:38] RECOVERY - puppet last run on db1057 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [04:12:49] RECOVERY - puppet last run on acamar is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [04:12:50] RECOVERY - puppet last run on es1002 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [04:12:50] RECOVERY - puppet last run on ms-be2007 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [04:12:55] RECOVERY - puppet last run on analytics1011 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [04:13:19] RECOVERY - puppet last run on db1001 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [04:13:19] RECOVERY - puppet last run on db1049 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [04:13:19] RECOVERY - puppet last run on wtp1011 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [04:13:19] RECOVERY - puppet last run on wtp1002 is OK: OK: Puppet is currently enabled, last run 3 minutes ago with 0 failures [04:13:20] RECOVERY - puppet last run on labsdb1004 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [04:13:20] RECOVERY - puppet last run on pc1003 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [04:13:20] RECOVERY - puppet last run on mw1212 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [04:13:21] RECOVERY - puppet last run on db1070 is OK: OK: Puppet is currently enabled, last run 58 seconds ago with 0 failures [04:13:32] RECOVERY - puppet last run on mw1087 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [04:13:32] RECOVERY - puppet last run on wtp1015 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [04:13:32] RECOVERY - puppet last run on wtp1003 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [04:13:35] RECOVERY - puppet last run on rcs1002 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [04:13:42] RECOVERY - puppet last run on mw1053 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [04:13:45] RECOVERY - puppet last run on rdb1002 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [04:13:48] RECOVERY - puppet last run on search1023 is OK: OK: Puppet is currently enabled, last run 3 minutes ago with 0 failures [04:13:51] RECOVERY - puppet last run on db1054 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [04:13:51] RECOVERY - puppet last run on lvs2003 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [04:13:59] RECOVERY - puppet last run on analytics1032 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [04:13:59] RECOVERY - puppet last run on lvs1001 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [04:14:10] RECOVERY - puppet last run on rcs1001 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [04:14:18] RECOVERY - puppet last run on mw1116 is OK: OK: Puppet is currently enabled, last run 32 seconds ago with 0 failures [04:14:22] RECOVERY - puppet last run on wtp1013 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [04:14:22] RECOVERY - puppet last run on wtp1007 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [04:14:29] RECOVERY - puppet last run on amslvs3 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [04:14:29] RECOVERY - puppet last run on cp3009 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [04:14:34] RECOVERY - puppet last run on db1033 is OK: OK: Puppet is currently enabled, last run 27 seconds ago with 0 failures [04:14:34] RECOVERY - puppet last run on elastic1023 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [04:14:34] RECOVERY - puppet last run on db1061 is OK: OK: Puppet is currently enabled, last run 34 seconds ago with 0 failures [04:14:35] RECOVERY - puppet last run on mw1074 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [04:14:35] RECOVERY - puppet last run on cp1038 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [04:14:39] RECOVERY - puppet last run on ms-fe3001 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [04:14:51] RECOVERY - puppet last run on iodine is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [04:14:51] RECOVERY - puppet last run on db2012 is OK: OK: Puppet is currently enabled, last run 24 seconds ago with 0 failures [04:14:51] RECOVERY - puppet last run on analytics1004 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [04:14:51] RECOVERY - puppet last run on mw1167 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [04:15:01] RECOVERY - puppet last run on cp3004 is OK: OK: Puppet is currently enabled, last run 24 seconds ago with 0 failures [04:15:07] RECOVERY - puppet last run on mw1185 is OK: OK: Puppet is currently enabled, last run 9 seconds ago with 0 failures [04:15:07] RECOVERY - puppet last run on elastic1017 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [04:15:07] RECOVERY - puppet last run on mc1007 is OK: OK: Puppet is currently enabled, last run 3 minutes ago with 0 failures [04:15:07] RECOVERY - puppet last run on db1011 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [04:15:07] RECOVERY - puppet last run on mw1032 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [04:15:08] RECOVERY - puppet last run on analytics1027 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [04:15:08] RECOVERY - puppet last run on mw1105 is OK: OK: Puppet is currently enabled, last run 12 seconds ago with 0 failures [04:15:09] RECOVERY - puppet last run on ocg1003 is OK: OK: Puppet is currently enabled, last run 3 minutes ago with 0 failures [04:15:20] RECOVERY - puppet last run on calcium is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [04:15:20] RECOVERY - puppet last run on search1011 is OK: OK: Puppet is currently enabled, last run 48 seconds ago with 0 failures [04:15:21] RECOVERY - puppet last run on rdb1003 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [04:15:21] RECOVERY - puppet last run on mw1239 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [04:15:23] RECOVERY - puppet last run on db1064 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [04:15:33] RECOVERY - puppet last run on search1022 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [04:15:33] RECOVERY - puppet last run on nitrogen is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [04:15:33] RECOVERY - puppet last run on db1044 is OK: OK: Puppet is currently enabled, last run 3 minutes ago with 0 failures [04:15:33] RECOVERY - puppet last run on graphite1001 is OK: OK: Puppet is currently enabled, last run 3 minutes ago with 0 failures [04:15:33] RECOVERY - puppet last run on mw1243 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [04:15:34] RECOVERY - puppet last run on virt1000 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [04:15:34] RECOVERY - puppet last run on mw1148 is OK: OK: Puppet is currently enabled, last run 32 seconds ago with 0 failures [04:15:35] RECOVERY - puppet last run on mw1198 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [04:15:35] RECOVERY - puppet last run on mercury is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [04:15:36] RECOVERY - puppet last run on search1013 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [04:15:36] RECOVERY - puppet last run on fluorine is OK: OK: Puppet is currently enabled, last run 1 second ago with 0 failures [04:15:37] RECOVERY - puppet last run on es1010 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [04:15:37] RECOVERY - puppet last run on titanium is OK: OK: Puppet is currently enabled, last run 3 minutes ago with 0 failures [04:15:38] RECOVERY - puppet last run on elastic1003 is OK: OK: Puppet is currently enabled, last run 17 seconds ago with 0 failures [04:15:38] RECOVERY - puppet last run on cp1060 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [04:15:39] RECOVERY - puppet last run on dbstore1002 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [04:15:39] RECOVERY - puppet last run on mw1022 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [04:15:40] RECOVERY - puppet last run on mw1033 is OK: OK: Puppet is currently enabled, last run 28 seconds ago with 0 failures [04:15:45] RECOVERY - puppet last run on copper is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [04:15:46] RECOVERY - puppet last run on db1063 is OK: OK: Puppet is currently enabled, last run 3 minutes ago with 0 failures [04:15:46] RECOVERY - puppet last run on db1030 is OK: OK: Puppet is currently enabled, last run 3 minutes ago with 0 failures [04:15:46] RECOVERY - puppet last run on ms-be1011 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [04:15:46] RECOVERY - puppet last run on analytics1028 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [04:15:47] RECOVERY - puppet last run on db1047 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [04:16:06] RECOVERY - puppet last run on db1006 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [04:16:07] RECOVERY - puppet last run on mw1024 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [04:16:08] RECOVERY - puppet last run on db2030 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [04:16:08] RECOVERY - puppet last run on db2011 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [04:16:17] !log also truncated puppet fact_values, see http://projects.puppetlabs.com/issues/9225 and https://tickets.puppetlabs.com/browse/PUP-1173 [04:16:17] RECOVERY - puppet last run on achernar is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [04:16:17] RECOVERY - puppet last run on db2028 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [04:16:17] RECOVERY - puppet last run on search1004 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [04:16:22] Logged the message, Master [04:16:37] RECOVERY - puppet last run on analytics1003 is OK: OK: Puppet is currently enabled, last run 30 seconds ago with 0 failures [04:16:47] RECOVERY - puppet last run on db1038 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [04:16:48] RECOVERY - puppet last run on amssq62 is OK: OK: Puppet is currently enabled, last run 18 seconds ago with 0 failures [04:16:56] RECOVERY - puppet last run on mw1225 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [04:16:56] RECOVERY - puppet last run on lvs1004 is OK: OK: Puppet is currently enabled, last run 19 seconds ago with 0 failures [04:16:56] RECOVERY - puppet last run on db1072 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [04:16:56] RECOVERY - puppet last run on search1006 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [04:16:56] RECOVERY - puppet last run on wtp1008 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [04:16:57] RECOVERY - puppet last run on wtp1010 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [04:16:57] RECOVERY - puppet last run on mw1209 is OK: OK: Puppet is currently enabled, last run 13 seconds ago with 0 failures [04:17:01] RECOVERY - puppet last run on lvs1003 is OK: OK: Puppet is currently enabled, last run 26 seconds ago with 0 failures [04:17:01] RECOVERY - puppet last run on stat1001 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [04:17:09] !log springle Synchronized wmf-config/db-eqiad.php: depool db1049 (duration: 00m 06s) [04:17:12] Logged the message, Master [04:17:12] RECOVERY - puppet last run on ms-fe2002 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [04:17:12] RECOVERY - puppet last run on mw1186 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [04:17:12] RECOVERY - puppet last run on analytics1018 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [04:17:21] RECOVERY - puppet last run on ms-be1015 is OK: OK: Puppet is currently enabled, last run 27 seconds ago with 0 failures [04:17:38] RECOVERY - puppet last run on search1012 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [04:17:49] RECOVERY - puppet last run on rbf1001 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [04:17:49] RECOVERY - puppet last run on mw1231 is OK: OK: Puppet is currently enabled, last run 52 seconds ago with 0 failures [04:17:50] RECOVERY - puppet last run on cp3012 is OK: OK: Puppet is currently enabled, last run 34 seconds ago with 0 failures [04:17:50] RECOVERY - puppet last run on mw1001 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [04:17:50] RECOVERY - puppet last run on ms-be1002 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [04:17:51] RECOVERY - puppet last run on cp1044 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [04:17:51] RECOVERY - puppet last run on mw1093 is OK: OK: Puppet is currently enabled, last run 18 seconds ago with 0 failures [04:17:51] RECOVERY - puppet last run on mw1108 is OK: OK: Puppet is currently enabled, last run 3 minutes ago with 0 failures [04:18:02] RECOVERY - puppet last run on mw1229 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [04:18:10] RECOVERY - puppet last run on lvs3003 is OK: OK: Puppet is currently enabled, last run 3 minutes ago with 0 failures [04:18:21] RECOVERY - puppet last run on stat1003 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [04:18:21] RECOVERY - puppet last run on cp1052 is OK: OK: Puppet is currently enabled, last run 24 seconds ago with 0 failures [04:18:21] RECOVERY - puppet last run on ms-fe1002 is OK: OK: Puppet is currently enabled, last run 41 seconds ago with 0 failures [04:18:30] RECOVERY - puppet last run on mw1201 is OK: OK: Puppet is currently enabled, last run 3 minutes ago with 0 failures [04:18:31] RECOVERY - puppet last run on virt1008 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [04:18:31] RECOVERY - puppet last run on elastic1029 is OK: OK: Puppet is currently enabled, last run 21 seconds ago with 0 failures [04:18:31] RECOVERY - puppet last run on chromium is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [04:18:31] RECOVERY - puppet last run on mw1219 is OK: OK: Puppet is currently enabled, last run 3 minutes ago with 0 failures [04:18:31] RECOVERY - puppet last run on db1056 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [04:18:40] RECOVERY - puppet last run on db1068 is OK: OK: Puppet is currently enabled, last run 43 seconds ago with 0 failures [04:18:42] RECOVERY - puppet last run on radon is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [04:18:43] RECOVERY - puppet last run on db1035 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [04:18:43] RECOVERY - puppet last run on radium is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [04:18:50] RECOVERY - puppet last run on mw1152 is OK: OK: Puppet is currently enabled, last run 54 seconds ago with 0 failures [04:18:51] RECOVERY - puppet last run on mw1043 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [04:18:52] RECOVERY - puppet last run on db1045 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [04:18:52] RECOVERY - puppet last run on labsdb1005 is OK: OK: Puppet is currently enabled, last run 3 seconds ago with 0 failures [04:18:52] RECOVERY - puppet last run on sca1001 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [04:18:53] RECOVERY - puppet last run on cp1045 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [04:18:53] RECOVERY - puppet last run on mw1091 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [04:18:53] RECOVERY - puppet last run on db1037 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [04:18:53] RECOVERY - puppet last run on wtp1001 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [04:18:54] RECOVERY - puppet last run on analytics1031 is OK: OK: Puppet is currently enabled, last run 27 seconds ago with 0 failures [04:19:01] RECOVERY - puppet last run on elastic1013 is OK: OK: Puppet is currently enabled, last run 6 seconds ago with 0 failures [04:19:04] RECOVERY - puppet last run on analytics1012 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [04:19:11] RECOVERY - puppet last run on db1009 is OK: OK: Puppet is currently enabled, last run 15 seconds ago with 0 failures [04:19:21] RECOVERY - puppet last run on baham is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [04:19:21] RECOVERY - puppet last run on es2006 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [04:19:21] RECOVERY - puppet last run on dysprosium is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [04:19:21] RECOVERY - puppet last run on ms-be1014 is OK: OK: Puppet is currently enabled, last run 26 seconds ago with 0 failures [04:19:22] RECOVERY - puppet last run on db2010 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [04:19:32] RECOVERY - puppet last run on db2017 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [04:19:32] RECOVERY - puppet last run on lvs2002 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [04:19:34] RECOVERY - puppet last run on lvs4004 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [04:19:35] RECOVERY - puppet last run on bast2001 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [04:19:37] RECOVERY - puppet last run on lvs2005 is OK: OK: Puppet is currently enabled, last run 3 seconds ago with 0 failures [04:19:52] RECOVERY - puppet last run on osm-cp1001 is OK: OK: Puppet is currently enabled, last run 3 minutes ago with 0 failures [04:19:53] RECOVERY - puppet last run on labsdb1002 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [04:19:56] RECOVERY - puppet last run on cp3018 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [04:19:56] RECOVERY - puppet last run on mw1071 is OK: OK: Puppet is currently enabled, last run 33 seconds ago with 0 failures [04:20:02] RECOVERY - puppet last run on hydrogen is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [04:20:02] RECOVERY - puppet last run on zinc is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [04:20:02] RECOVERY - puppet last run on elastic1025 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [04:20:10] RECOVERY - puppet last run on mw1121 is OK: OK: Puppet is currently enabled, last run 3 minutes ago with 0 failures [04:20:10] RECOVERY - puppet last run on es1004 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [04:20:20] RECOVERY - puppet last run on cp1037 is OK: OK: Puppet is currently enabled, last run 44 seconds ago with 0 failures [04:20:21] RECOVERY - puppet last run on mw1016 is OK: OK: Puppet is currently enabled, last run 57 seconds ago with 0 failures [04:20:22] RECOVERY - puppet last run on ms-be2009 is OK: OK: Puppet is currently enabled, last run 14 seconds ago with 0 failures [04:20:22] RECOVERY - puppet last run on mc1015 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [04:20:22] RECOVERY - puppet last run on labsdb1007 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [04:20:22] RECOVERY - puppet last run on mw1223 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [04:20:22] RECOVERY - puppet last run on ms-be1010 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [04:21:00] RECOVERY - puppet last run on bast1001 is OK: OK: Puppet is currently enabled, last run 8 seconds ago with 0 failures [04:21:12] RECOVERY - puppet last run on lvs3002 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [04:21:16] RECOVERY - puppet last run on es1005 is OK: OK: Puppet is currently enabled, last run 30 seconds ago with 0 failures [04:21:16] RECOVERY - puppet last run on elastic1016 is OK: OK: Puppet is currently enabled, last run 36 seconds ago with 0 failures [04:21:20] RECOVERY - puppet last run on db1027 is OK: OK: Puppet is currently enabled, last run 3 minutes ago with 0 failures [04:21:31] RECOVERY - puppet last run on lvs4001 is OK: OK: Puppet is currently enabled, last run 7 seconds ago with 0 failures [04:21:31] RECOVERY - puppet last run on amssq38 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [04:21:32] RECOVERY - puppet last run on cp3017 is OK: OK: Puppet is currently enabled, last run 21 seconds ago with 0 failures [04:21:42] RECOVERY - puppet last run on cp1064 is OK: OK: Puppet is currently enabled, last run 19 seconds ago with 0 failures [04:21:42] RECOVERY - puppet last run on analytics1034 is OK: OK: Puppet is currently enabled, last run 14 seconds ago with 0 failures [04:21:42] RECOVERY - puppet last run on db1019 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [04:21:47] RECOVERY - puppet last run on db1058 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [04:21:47] RECOVERY - puppet last run on mw1010 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [04:21:53] RECOVERY - puppet last run on analytics1024 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [04:21:53] RECOVERY - puppet last run on mc1008 is OK: OK: Puppet is currently enabled, last run 0 seconds ago with 0 failures [04:21:53] RECOVERY - puppet last run on ms-be1005 is OK: OK: Puppet is currently enabled, last run 44 seconds ago with 0 failures [04:22:02] RECOVERY - puppet last run on mw1215 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [04:22:02] RECOVERY - puppet last run on elastic1009 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [04:22:05] RECOVERY - puppet last run on mw1122 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [04:22:05] RECOVERY - puppet last run on es1003 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [04:22:05] RECOVERY - puppet last run on mw1107 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [04:22:05] RECOVERY - puppet last run on wtp1014 is OK: OK: Puppet is currently enabled, last run 39 seconds ago with 0 failures [04:22:05] RECOVERY - puppet last run on elastic1010 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [04:22:06] RECOVERY - puppet last run on palladium is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [04:22:06] RECOVERY - puppet last run on cp1053 is OK: OK: Puppet is currently enabled, last run 52 seconds ago with 0 failures [04:22:13] RECOVERY - puppet last run on praseodymium is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [04:22:18] RECOVERY - puppet last run on db1005 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [04:22:31] RECOVERY - puppet last run on ms-be2015 is OK: OK: Puppet is currently enabled, last run 36 seconds ago with 0 failures [04:22:31] RECOVERY - puppet last run on pollux is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [04:22:31] RECOVERY - puppet last run on ms-be2010 is OK: OK: Puppet is currently enabled, last run 30 seconds ago with 0 failures [04:22:32] RECOVERY - puppet last run on mw1203 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [04:22:41] RECOVERY - puppet last run on ms-be3004 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [04:22:55] RECOVERY - puppet last run on amslvs4 is OK: OK: Puppet is currently enabled, last run 6 seconds ago with 0 failures [04:22:59] RECOVERY - puppet last run on db1010 is OK: OK: Puppet is currently enabled, last run 6 seconds ago with 0 failures [04:23:09] RECOVERY - puppet last run on mw1241 is OK: OK: Puppet is currently enabled, last run 16 seconds ago with 0 failures [04:23:09] RECOVERY - puppet last run on virt1009 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [04:23:21] RECOVERY - puppet last run on elastic1026 is OK: OK: Puppet is currently enabled, last run 20 seconds ago with 0 failures [04:23:23] RECOVERY - puppet last run on cp1040 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [04:23:24] RECOVERY - puppet last run on mw1090 is OK: OK: Puppet is currently enabled, last run 36 seconds ago with 0 failures [04:23:24] RECOVERY - puppet last run on search1003 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [04:23:24] RECOVERY - puppet last run on wtp1021 is OK: OK: Puppet is currently enabled, last run 47 seconds ago with 0 failures [04:23:24] RECOVERY - puppet last run on mw1193 is OK: OK: Puppet is currently enabled, last run 14 seconds ago with 0 failures [04:23:24] RECOVERY - puppet last run on mw1086 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [04:23:41] RECOVERY - puppet last run on lvs1006 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [04:23:43] RECOVERY - puppet last run on db1024 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [04:23:43] RECOVERY - puppet last run on analytics1019 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [04:23:45] RECOVERY - puppet last run on analytics1036 is OK: OK: Puppet is currently enabled, last run 32 seconds ago with 0 failures [04:23:49] RECOVERY - puppet last run on labsdb1001 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [04:23:50] RECOVERY - puppet last run on virt1005 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [04:23:50] RECOVERY - puppet last run on lanthanum is OK: OK: Puppet is currently enabled, last run 17 seconds ago with 0 failures [04:24:00] RECOVERY - puppet last run on es1006 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [04:24:05] RECOVERY - puppet last run on ocg1002 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [04:24:06] RECOVERY - puppet last run on cp3007 is OK: OK: Puppet is currently enabled, last run 50 seconds ago with 0 failures [04:24:06] RECOVERY - puppet last run on cp3005 is OK: OK: Puppet is currently enabled, last run 3 minutes ago with 0 failures [04:24:15] RECOVERY - puppet last run on analytics1029 is OK: OK: Puppet is currently enabled, last run 3 minutes ago with 0 failures [04:24:18] RECOVERY - puppet last run on analytics1015 is OK: OK: Puppet is currently enabled, last run 5 seconds ago with 0 failures [04:24:21] RECOVERY - puppet last run on search1008 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [04:24:28] RECOVERY - puppet last run on mw1066 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [04:24:29] RECOVERY - puppet last run on cp4015 is OK: OK: Puppet is currently enabled, last run 15 seconds ago with 0 failures [04:24:29] RECOVERY - puppet last run on eeden is OK: OK: Puppet is currently enabled, last run 21 seconds ago with 0 failures [04:24:39] RECOVERY - puppet last run on search1019 is OK: OK: Puppet is currently enabled, last run 3 minutes ago with 0 failures [04:24:51] RECOVERY - puppet last run on analytics1039 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [04:25:00] RECOVERY - puppet last run on cp1054 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [04:25:01] RECOVERY - puppet last run on mw1143 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [04:25:01] RECOVERY - puppet last run on dbstore1001 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [04:25:01] RECOVERY - puppet last run on mw1220 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [04:25:01] RECOVERY - puppet last run on mw1142 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [04:25:01] RECOVERY - puppet last run on virt1002 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [04:25:02] RECOVERY - puppet last run on mw1204 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [04:25:02] RECOVERY - puppet last run on cp1043 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [04:25:03] RECOVERY - puppet last run on mw1139 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [04:25:03] RECOVERY - puppet last run on elastic1028 is OK: OK: Puppet is currently enabled, last run 56 seconds ago with 0 failures [04:25:04] RECOVERY - puppet last run on search1021 is OK: OK: Puppet is currently enabled, last run 51 seconds ago with 0 failures [04:25:12] RECOVERY - puppet last run on es1009 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [04:25:13] RECOVERY - puppet last run on mw1236 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [04:25:13] RECOVERY - puppet last run on db1007 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [04:25:13] RECOVERY - puppet last run on uranium is OK: OK: Puppet is currently enabled, last run 4 seconds ago with 0 failures [04:25:13] RECOVERY - puppet last run on mw1112 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [04:25:13] RECOVERY - puppet last run on ytterbium is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [04:25:14] RECOVERY - puppet last run on mw1113 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [04:25:14] RECOVERY - puppet last run on mw1064 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [04:25:14] RECOVERY - puppet last run on mc1010 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [04:25:21] RECOVERY - puppet last run on search1009 is OK: OK: Puppet is currently enabled, last run 7 seconds ago with 0 failures [04:25:22] RECOVERY - puppet last run on db1041 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [04:25:31] RECOVERY - puppet last run on es1001 is OK: OK: Puppet is currently enabled, last run 18 seconds ago with 0 failures [04:25:31] RECOVERY - puppet last run on db2035 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [04:25:43] RECOVERY - puppet last run on logstash1003 is OK: OK: Puppet is currently enabled, last run 16 seconds ago with 0 failures [04:25:43] RECOVERY - puppet last run on ocg1001 is OK: OK: Puppet is currently enabled, last run 30 seconds ago with 0 failures [04:25:52] RECOVERY - puppet last run on cp4002 is OK: OK: Puppet is currently enabled, last run 52 seconds ago with 0 failures [04:25:54] RECOVERY - puppet last run on cp4009 is OK: OK: Puppet is currently enabled, last run 24 seconds ago with 0 failures [04:25:55] RECOVERY - puppet last run on amssq33 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [04:26:02] RECOVERY - puppet last run on ms-be1013 is OK: OK: Puppet is currently enabled, last run 39 seconds ago with 0 failures [04:26:02] RECOVERY - puppet last run on ms-fe1003 is OK: OK: Puppet is currently enabled, last run 25 seconds ago with 0 failures [04:26:02] RECOVERY - puppet last run on amslvs2 is OK: OK: Puppet is currently enabled, last run 29 seconds ago with 0 failures [04:26:12] RECOVERY - puppet last run on wtp1024 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [04:26:12] RECOVERY - puppet last run on mw1110 is OK: OK: Puppet is currently enabled, last run 23 seconds ago with 0 failures [04:26:21] RECOVERY - puppet last run on mw1037 is OK: OK: Puppet is currently enabled, last run 5 seconds ago with 0 failures [04:26:26] RECOVERY - puppet last run on pc1001 is OK: OK: Puppet is currently enabled, last run 38 seconds ago with 0 failures [04:26:31] RECOVERY - puppet last run on cp1067 is OK: OK: Puppet is currently enabled, last run 33 seconds ago with 0 failures [04:26:32] RECOVERY - puppet last run on analytics1017 is OK: OK: Puppet is currently enabled, last run 27 seconds ago with 0 failures [04:26:33] RECOVERY - puppet last run on cp1068 is OK: OK: Puppet is currently enabled, last run 55 seconds ago with 0 failures [04:26:45] RECOVERY - puppet last run on mw1253 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [04:26:47] RECOVERY - puppet last run on wtp1017 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [04:26:47] RECOVERY - puppet last run on zirconium is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [04:26:47] RECOVERY - puppet last run on mw1158 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [04:26:47] RECOVERY - puppet last run on mc1004 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [04:26:47] RECOVERY - puppet last run on mw1027 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [04:27:06] RECOVERY - puppet last run on elastic1031 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [04:27:06] RECOVERY - puppet last run on ms-be1001 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [04:27:06] RECOVERY - puppet last run on rdb1004 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [04:27:06] RECOVERY - puppet last run on db1029 is OK: OK: Puppet is currently enabled, last run 24 seconds ago with 0 failures [04:27:06] RECOVERY - puppet last run on cp4013 is OK: OK: Puppet is currently enabled, last run 28 seconds ago with 0 failures [04:27:07] RECOVERY - puppet last run on mw1131 is OK: OK: Puppet is currently enabled, last run 16 seconds ago with 0 failures [04:27:07] RECOVERY - puppet last run on mw1135 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [04:27:23] RECOVERY - puppet last run on mw1207 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [04:27:23] RECOVERY - puppet last run on cp1051 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [04:27:24] RECOVERY - puppet last run on mc1009 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [04:27:24] RECOVERY - puppet last run on cp3011 is OK: OK: Puppet is currently enabled, last run 42 seconds ago with 0 failures [04:27:34] RECOVERY - puppet last run on dbproxy1002 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [04:27:34] RECOVERY - puppet last run on ms-be1004 is OK: OK: Puppet is currently enabled, last run 26 seconds ago with 0 failures [04:27:34] RECOVERY - puppet last run on rbf1002 is OK: OK: Puppet is currently enabled, last run 28 seconds ago with 0 failures [04:27:34] RECOVERY - puppet last run on db1031 is OK: OK: Puppet is currently enabled, last run 5 seconds ago with 0 failures [04:27:34] RECOVERY - puppet last run on cp4020 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [04:27:35] RECOVERY - puppet last run on search1014 is OK: OK: Puppet is currently enabled, last run 5 seconds ago with 0 failures [04:27:41] RECOVERY - puppet last run on vanadium is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [04:27:53] RECOVERY - puppet last run on db1065 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [04:27:56] RECOVERY - puppet last run on mw1104 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [04:27:57] RECOVERY - puppet last run on analytics1041 is OK: OK: Puppet is currently enabled, last run 38 seconds ago with 0 failures [04:28:03] RECOVERY - puppet last run on db1053 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [04:28:03] RECOVERY - puppet last run on analytics1033 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [04:28:03] RECOVERY - puppet last run on wtp1009 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [04:28:03] RECOVERY - puppet last run on wtp1019 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [04:28:11] RECOVERY - puppet last run on labstore1003 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [04:28:11] RECOVERY - puppet last run on analytics1040 is OK: OK: Puppet is currently enabled, last run 22 seconds ago with 0 failures [04:28:22] RECOVERY - puppet last run on mc1011 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [04:28:25] RECOVERY - puppet last run on potassium is OK: OK: Puppet is currently enabled, last run 46 seconds ago with 0 failures [04:28:56] RECOVERY - puppet last run on es1008 is OK: OK: Puppet is currently enabled, last run 54 seconds ago with 0 failures [04:28:56] RECOVERY - puppet last run on db2039 is OK: OK: Puppet is currently enabled, last run 39 seconds ago with 0 failures [04:28:58] RECOVERY - puppet last run on es2008 is OK: OK: Puppet is currently enabled, last run 3 seconds ago with 0 failures [04:28:58] RECOVERY - puppet last run on db2005 is OK: OK: Puppet is currently enabled, last run 12 seconds ago with 0 failures [04:28:58] RECOVERY - puppet last run on ms-be2013 is OK: OK: Puppet is currently enabled, last run 3 minutes ago with 0 failures [04:28:58] RECOVERY - puppet last run on ms-be2002 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [04:28:58] RECOVERY - puppet last run on lvs4002 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [04:28:59] RECOVERY - puppet last run on cp4012 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [04:28:59] RECOVERY - puppet last run on amssq39 is OK: OK: Puppet is currently enabled, last run 3 minutes ago with 0 failures [04:29:00] RECOVERY - puppet last run on amssq31 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [04:29:23] RECOVERY - puppet last run on wtp1020 is OK: OK: Puppet is currently enabled, last run 48 seconds ago with 0 failures [04:29:25] RECOVERY - puppet last run on platinum is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [04:29:25] RECOVERY - puppet last run on analytics1020 is OK: OK: Puppet is currently enabled, last run 23 seconds ago with 0 failures [04:29:25] RECOVERY - puppet last run on mw1155 is OK: OK: Puppet is currently enabled, last run 2 seconds ago with 0 failures [04:29:26] RECOVERY - puppet last run on elastic1021 is OK: OK: Puppet is currently enabled, last run 7 seconds ago with 0 failures [04:29:26] RECOVERY - puppet last run on gold is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [04:29:26] RECOVERY - puppet last run on mc1016 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [04:29:35] RECOVERY - puppet last run on magnesium is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [04:29:36] RECOVERY - puppet last run on db2019 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [04:29:46] RECOVERY - puppet last run on ms1004 is OK: OK: Puppet is currently enabled, last run 46 seconds ago with 0 failures [04:29:46] RECOVERY - puppet last run on analytics1025 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [04:29:46] RECOVERY - puppet last run on ms-be2003 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [04:29:46] RECOVERY - puppet last run on logstash1001 is OK: OK: Puppet is currently enabled, last run 59 seconds ago with 0 failures [04:29:52] RECOVERY - puppet last run on erbium is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [04:29:52] RECOVERY - puppet last run on lvs1002 is OK: OK: Puppet is currently enabled, last run 32 seconds ago with 0 failures [04:30:03] RECOVERY - puppet last run on mw1194 is OK: OK: Puppet is currently enabled, last run 36 seconds ago with 0 failures [04:30:06] RECOVERY - puppet last run on mw1103 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [04:30:07] RECOVERY - puppet last run on helium is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [04:30:07] RECOVERY - puppet last run on sca1002 is OK: OK: Puppet is currently enabled, last run 33 seconds ago with 0 failures [04:30:13] RECOVERY - puppet last run on cp3006 is OK: OK: Puppet is currently enabled, last run 15 seconds ago with 0 failures [04:30:13] RECOVERY - puppet last run on amssq37 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [04:30:22] RECOVERY - puppet last run on netmon1001 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [04:30:22] RECOVERY - puppet last run on db1022 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [04:30:22] RECOVERY - puppet last run on mw1101 is OK: OK: Puppet is currently enabled, last run 29 seconds ago with 0 failures [04:30:22] RECOVERY - puppet last run on db2034 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [04:30:22] RECOVERY - puppet last run on ms-be2004 is OK: OK: Puppet is currently enabled, last run 6 seconds ago with 0 failures [04:30:31] RECOVERY - puppet last run on cp1057 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [04:30:31] RECOVERY - puppet last run on mc1006 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [04:30:31] RECOVERY - puppet last run on lvs3001 is OK: OK: Puppet is currently enabled, last run 51 seconds ago with 0 failures [04:30:31] RECOVERY - puppet last run on mw1015 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [04:30:31] RECOVERY - puppet last run on db1066 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [04:30:32] RECOVERY - puppet last run on db1073 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [04:30:32] RECOVERY - puppet last run on ms-be1006 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [04:30:41] RECOVERY - puppet last run on mw1047 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [04:30:44] RECOVERY - puppet last run on mw1018 is OK: OK: Puppet is currently enabled, last run 55 seconds ago with 0 failures [04:30:44] RECOVERY - puppet last run on db2009 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [04:30:44] RECOVERY - puppet last run on lead is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [04:30:45] RECOVERY - puppet last run on db1046 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [04:30:52] RECOVERY - puppet last run on elastic1018 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [04:30:53] RECOVERY - puppet last run on elastic1027 is OK: OK: Puppet is currently enabled, last run 28 seconds ago with 0 failures [04:30:54] RECOVERY - puppet last run on db1050 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [04:30:58] RECOVERY - puppet last run on elastic1004 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [04:31:13] RECOVERY - puppet last run on xenon is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [04:31:13] RECOVERY - puppet last run on neptunium is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [04:31:13] RECOVERY - puppet last run on wtp1006 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [04:31:13] RECOVERY - puppet last run on elastic1001 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [04:31:14] RECOVERY - puppet last run on ms-be1003 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [04:31:14] RECOVERY - puppet last run on mw1073 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [04:31:23] RECOVERY - puppet last run on elastic1007 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [04:31:23] RECOVERY - puppet last run on mw1021 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [04:31:23] RECOVERY - puppet last run on elastic1012 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [04:31:23] RECOVERY - puppet last run on caesium is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [04:31:23] RECOVERY - puppet last run on mc1003 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [04:31:24] RECOVERY - puppet last run on labsdb1003 is OK: OK: Puppet is currently enabled, last run 38 seconds ago with 0 failures [04:31:24] RECOVERY - puppet last run on analytics1035 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [04:31:25] RECOVERY - puppet last run on db1015 is OK: OK: Puppet is currently enabled, last run 48 seconds ago with 0 failures [04:31:25] RECOVERY - puppet last run on labstore1001 is OK: OK: Puppet is currently enabled, last run 8 seconds ago with 0 failures [04:31:26] RECOVERY - puppet last run on db1040 is OK: OK: Puppet is currently enabled, last run 57 seconds ago with 0 failures [04:31:36] RECOVERY - puppet last run on ms-fe2004 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [04:31:36] RECOVERY - puppet last run on cp1066 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [04:31:42] RECOVERY - puppet last run on db2002 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [04:31:42] RECOVERY - puppet last run on db2018 is OK: OK: Puppet is currently enabled, last run 40 seconds ago with 0 failures [04:31:47] RECOVERY - puppet last run on lvs2004 is OK: OK: Puppet is currently enabled, last run 29 seconds ago with 0 failures [04:31:48] RECOVERY - puppet last run on es2001 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [04:31:54] RECOVERY - puppet last run on cp4007 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [04:31:57] RECOVERY - puppet last run on cp4016 is OK: OK: Puppet is currently enabled, last run 11 seconds ago with 0 failures [04:31:58] RECOVERY - puppet last run on amssq50 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [04:31:58] RECOVERY - puppet last run on lvs1005 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [04:32:08] RECOVERY - puppet last run on db1059 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [04:32:08] RECOVERY - puppet last run on cp3019 is OK: OK: Puppet is currently enabled, last run 59 seconds ago with 0 failures [04:32:09] RECOVERY - puppet last run on mw1199 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [04:32:12] RECOVERY - puppet last run on mw1128 is OK: OK: Puppet is currently enabled, last run 10 seconds ago with 0 failures [04:32:13] RECOVERY - puppet last run on mw1095 is OK: OK: Puppet is currently enabled, last run 9 seconds ago with 0 failures [04:32:13] RECOVERY - puppet last run on wtp1016 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [04:32:17] RECOVERY - puppet last run on mw1157 is OK: OK: Puppet is currently enabled, last run 23 seconds ago with 0 failures [04:32:17] RECOVERY - puppet last run on mw1137 is OK: OK: Puppet is currently enabled, last run 48 seconds ago with 0 failures [04:32:27] RECOVERY - puppet last run on elastic1008 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [04:32:35] RECOVERY - puppet last run on iron is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [04:32:35] RECOVERY - puppet last run on ms-be2006 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [04:32:41] RECOVERY - puppet last run on db1067 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [04:32:42] RECOVERY - puppet last run on ms-fe1004 is OK: OK: Puppet is currently enabled, last run 6 seconds ago with 0 failures [04:32:43] RECOVERY - puppet last run on db1002 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [04:32:47] RECOVERY - puppet last run on dbproxy1001 is OK: OK: Puppet is currently enabled, last run 54 seconds ago with 0 failures [04:32:47] RECOVERY - puppet last run on mc1002 is OK: OK: Puppet is currently enabled, last run 3 minutes ago with 0 failures [04:33:19] RECOVERY - puppet last run on cp3021 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [04:33:21] RECOVERY - puppet last run on mw1102 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [04:33:22] RECOVERY - puppet last run on search1020 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [04:33:38] RECOVERY - puppet last run on cp4010 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [04:33:38] RECOVERY - puppet last run on cp1049 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [04:33:38] RECOVERY - puppet last run on cp3015 is OK: OK: Puppet is currently enabled, last run 18 seconds ago with 0 failures [04:33:38] RECOVERY - puppet last run on ms-fe1001 is OK: OK: Puppet is currently enabled, last run 3 minutes ago with 0 failures [04:33:39] RECOVERY - puppet last run on neon is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [04:33:47] RECOVERY - puppet last run on analytics1030 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [04:33:48] RECOVERY - puppet last run on amslvs1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [04:33:48] RECOVERY - puppet last run on amssq45 is OK: OK: Puppet is currently enabled, last run 4 seconds ago with 0 failures [04:33:48] RECOVERY - puppet last run on db1052 is OK: OK: Puppet is currently enabled, last run 4 seconds ago with 0 failures [04:33:48] RECOVERY - puppet last run on cp1069 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [04:33:57] RECOVERY - puppet last run on mw1078 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [04:34:00] RECOVERY - puppet last run on mw1230 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [04:34:06] RECOVERY - puppet last run on db1051 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [04:34:08] RECOVERY - puppet last run on db1042 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [04:34:08] RECOVERY - puppet last run on lithium is OK: OK: Puppet is currently enabled, last run 55 seconds ago with 0 failures [04:34:08] RECOVERY - puppet last run on mw1017 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [04:34:08] RECOVERY - puppet last run on cp4011 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [04:34:08] RECOVERY - puppet last run on labnet1001 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [04:34:09] RECOVERY - puppet last run on db1018 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [04:34:20] RECOVERY - puppet last run on mw1179 is OK: OK: Puppet is currently enabled, last run 3 minutes ago with 0 failures [04:34:20] RECOVERY - puppet last run on mw1058 is OK: OK: Puppet is currently enabled, last run 3 minutes ago with 0 failures [04:34:20] RECOVERY - puppet last run on es1007 is OK: OK: Puppet is currently enabled, last run 20 seconds ago with 0 failures [04:34:20] RECOVERY - puppet last run on db1028 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [04:34:31] RECOVERY - puppet last run on db1023 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [04:34:31] RECOVERY - puppet last run on mw1020 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [04:34:31] RECOVERY - puppet last run on db1021 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [04:34:31] RECOVERY - puppet last run on mw1075 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [04:34:32] RECOVERY - puppet last run on ms-fe2003 is OK: OK: Puppet is currently enabled, last run 49 seconds ago with 0 failures [04:34:32] RECOVERY - puppet last run on ms-fe2001 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [04:34:32] RECOVERY - puppet last run on cp1065 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [04:34:40] RECOVERY - puppet last run on db2036 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [04:34:41] RECOVERY - puppet last run on db1043 is OK: OK: Puppet is currently enabled, last run 51 seconds ago with 0 failures [04:34:41] RECOVERY - puppet last run on db1016 is OK: OK: Puppet is currently enabled, last run 5 seconds ago with 0 failures [04:34:46] RECOVERY - puppet last run on lvs2001 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [04:34:47] RECOVERY - puppet last run on db2029 is OK: OK: Puppet is currently enabled, last run 4 seconds ago with 0 failures [04:35:02] RECOVERY - puppet last run on cp3008 is OK: OK: Puppet is currently enabled, last run 52 seconds ago with 0 failures [04:35:02] RECOVERY - puppet last run on db1048 is OK: OK: Puppet is currently enabled, last run 19 seconds ago with 0 failures [04:35:02] RECOVERY - puppet last run on db1036 is OK: OK: Puppet is currently enabled, last run 10 seconds ago with 0 failures [04:35:12] RECOVERY - puppet last run on ruthenium is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [04:35:24] RECOVERY - puppet last run on mc1012 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [04:35:34] RECOVERY - puppet last run on mw1182 is OK: OK: Puppet is currently enabled, last run 48 seconds ago with 0 failures [04:35:35] RECOVERY - puppet last run on db1003 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [04:35:35] RECOVERY - puppet last run on virt1006 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [04:35:35] RECOVERY - puppet last run on mw1019 is OK: OK: Puppet is currently enabled, last run 19 seconds ago with 0 failures [04:35:35] RECOVERY - puppet last run on db1071 is OK: OK: Puppet is currently enabled, last run 7 seconds ago with 0 failures [04:35:35] RECOVERY - puppet last run on db2038 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [04:35:36] RECOVERY - puppet last run on db2007 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [04:35:45] RECOVERY - puppet last run on logstash1002 is OK: OK: Puppet is currently enabled, last run 17 seconds ago with 0 failures [04:35:51] RECOVERY - puppet last run on tmh1002 is OK: OK: Puppet is currently enabled, last run 52 seconds ago with 0 failures [04:35:59] RECOVERY - puppet last run on elastic1030 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [04:36:00] RECOVERY - puppet last run on lvs2006 is OK: OK: Puppet is currently enabled, last run 13 seconds ago with 0 failures [04:36:00] RECOVERY - puppet last run on db1004 is OK: OK: Puppet is currently enabled, last run 12 seconds ago with 0 failures [04:36:01] RECOVERY - puppet last run on ms-be2011 is OK: OK: Puppet is currently enabled, last run 46 seconds ago with 0 failures [04:36:01] RECOVERY - puppet last run on db1020 is OK: OK: Puppet is currently enabled, last run 10 seconds ago with 0 failures [04:36:01] RECOVERY - puppet last run on db1034 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [04:36:02] RECOVERY - puppet last run on mc1005 is OK: OK: Puppet is currently enabled, last run 29 seconds ago with 0 failures [04:36:02] RECOVERY - puppet last run on db1026 is OK: OK: Puppet is currently enabled, last run 49 seconds ago with 0 failures [04:36:02] RECOVERY - puppet last run on cp4017 is OK: OK: Puppet is currently enabled, last run 44 seconds ago with 0 failures [04:36:17] RECOVERY - puppet last run on mw1070 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [04:36:23] RECOVERY - puppet last run on nescio is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [04:36:24] RECOVERY - puppet last run on ms-be3001 is OK: OK: Puppet is currently enabled, last run 21 seconds ago with 0 failures [04:36:24] RECOVERY - puppet last run on amssq52 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [04:36:25] RECOVERY - puppet last run on ms-fe3002 is OK: OK: Puppet is currently enabled, last run 27 seconds ago with 0 failures [04:36:26] RECOVERY - puppet last run on polonium is OK: OK: Puppet is currently enabled, last run 47 seconds ago with 0 failures [04:36:26] RECOVERY - puppet last run on analytics1022 is OK: OK: Puppet is currently enabled, last run 46 seconds ago with 0 failures [04:36:27] RECOVERY - puppet last run on analytics1026 is OK: OK: Puppet is currently enabled, last run 36 seconds ago with 0 failures [04:36:27] RECOVERY - puppet last run on analytics1023 is OK: OK: Puppet is currently enabled, last run 32 seconds ago with 0 failures [04:36:27] RECOVERY - puppet last run on virt1001 is OK: OK: Puppet is currently enabled, last run 42 seconds ago with 0 failures [04:36:27] RECOVERY - puppet last run on elastic1022 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [04:36:36] RECOVERY - puppet last run on wtp1004 is OK: OK: Puppet is currently enabled, last run 18 seconds ago with 0 failures [04:36:36] RECOVERY - puppet last run on mw1013 is OK: OK: Puppet is currently enabled, last run 18 seconds ago with 0 failures [04:36:37] RECOVERY - puppet last run on lvs4003 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [04:36:37] RECOVERY - puppet last run on bast4001 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [04:36:37] RECOVERY - puppet last run on amssq58 is OK: OK: Puppet is currently enabled, last run 26 seconds ago with 0 failures [04:36:37] RECOVERY - puppet last run on cp3022 is OK: OK: Puppet is currently enabled, last run 8 seconds ago with 0 failures [04:36:37] RECOVERY - puppet last run on cp1058 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [04:36:49] RECOVERY - puppet last run on mw1246 is OK: OK: Puppet is currently enabled, last run 6 seconds ago with 0 failures [04:36:49] RECOVERY - puppet last run on analytics1010 is OK: OK: Puppet is currently enabled, last run 25 seconds ago with 0 failures [04:36:49] RECOVERY - puppet last run on db1060 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [04:36:49] RECOVERY - puppet last run on mw1184 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [04:36:49] RECOVERY - puppet last run on amssq57 is OK: OK: Puppet is currently enabled, last run 43 seconds ago with 0 failures [04:36:50] RECOVERY - puppet last run on mw1191 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [04:36:50] RECOVERY - puppet last run on virt1004 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [04:36:59] RECOVERY - puppet last run on mw1035 is OK: OK: Puppet is currently enabled, last run 17 seconds ago with 0 failures [04:37:09] RECOVERY - puppet last run on mw1218 is OK: OK: Puppet is currently enabled, last run 9 seconds ago with 0 failures [04:37:10] RECOVERY - puppet last run on mw1169 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [04:37:10] RECOVERY - puppet last run on elastic1019 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [04:37:10] RECOVERY - puppet last run on wtp1005 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [04:37:10] RECOVERY - puppet last run on pc1002 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [04:37:10] RECOVERY - puppet last run on elastic1006 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [04:37:11] RECOVERY - puppet last run on wtp1012 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [04:37:11] RECOVERY - puppet last run on cp4006 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [04:37:12] RECOVERY - puppet last run on labsdb1006 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [04:37:12] RECOVERY - puppet last run on mw1192 is OK: OK: Puppet is currently enabled, last run 3 seconds ago with 0 failures [04:37:13] RECOVERY - puppet last run on mw1161 is OK: OK: Puppet is currently enabled, last run 1 second ago with 0 failures [04:37:26] RECOVERY - puppet last run on mw1085 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [04:37:26] RECOVERY - puppet last run on plutonium is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [04:37:26] RECOVERY - puppet last run on elastic1015 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [04:37:26] RECOVERY - puppet last run on mw1234 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [04:37:26] RECOVERY - puppet last run on mw1096 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [04:37:27] RECOVERY - puppet last run on cp1047 is OK: OK: Puppet is currently enabled, last run 24 seconds ago with 0 failures [04:37:27] RECOVERY - puppet last run on cp1059 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [04:37:28] RECOVERY - puppet last run on mw1005 is OK: OK: Puppet is currently enabled, last run 16 seconds ago with 0 failures [04:37:28] RECOVERY - puppet last run on db1039 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [04:37:29] RECOVERY - puppet last run on cp1039 is OK: OK: Puppet is currently enabled, last run 30 seconds ago with 0 failures [04:37:29] RECOVERY - puppet last run on labmon1001 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [04:37:30] RECOVERY - puppet last run on rdb1001 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [04:37:30] RECOVERY - puppet last run on mw1083 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [04:37:31] RECOVERY - puppet last run on analytics1016 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [04:37:31] RECOVERY - puppet last run on mw1130 is OK: OK: Puppet is currently enabled, last run 2 seconds ago with 0 failures [04:37:32] RECOVERY - puppet last run on analytics1013 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [04:37:38] RECOVERY - puppet last run on mw1214 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [04:37:42] RECOVERY - puppet last run on cp1061 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [04:37:42] RECOVERY - puppet last run on db1069 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [04:37:43] RECOVERY - puppet last run on mc1014 is OK: OK: Puppet is currently enabled, last run 3 minutes ago with 0 failures [04:37:43] RECOVERY - puppet last run on cp1050 is OK: OK: Puppet is currently enabled, last run 25 seconds ago with 0 failures [04:37:44] RECOVERY - puppet last run on mw1178 is OK: OK: Puppet is currently enabled, last run 10 seconds ago with 0 failures [04:37:44] RECOVERY - puppet last run on antimony is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [04:37:44] RECOVERY - puppet last run on db2004 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [04:37:56] RECOVERY - puppet last run on argon is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [04:37:59] RECOVERY - puppet last run on db2037 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [04:37:59] RECOVERY - puppet last run on elastic1024 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [04:37:59] RECOVERY - puppet last run on ms-be2008 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [04:37:59] RECOVERY - puppet last run on cp1062 is OK: OK: Puppet is currently enabled, last run 24 seconds ago with 0 failures [04:37:59] RECOVERY - puppet last run on labcontrol2001 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [04:38:00] RECOVERY - puppet last run on gadolinium is OK: OK: Puppet is currently enabled, last run 20 seconds ago with 0 failures [04:38:00] RECOVERY - puppet last run on mw1257 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [04:38:01] RECOVERY - puppet last run on db2023 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [04:38:01] RECOVERY - puppet last run on thallium is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [04:38:02] RECOVERY - puppet last run on hafnium is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [04:38:02] RECOVERY - puppet last run on virt1007 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [04:38:03] RECOVERY - puppet last run on analytics1038 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [04:38:03] RECOVERY - puppet last run on searchidx1001 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [04:38:04] RECOVERY - puppet last run on ms-be2005 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [04:38:04] RECOVERY - puppet last run on ms-be2001 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [04:38:05] RECOVERY - puppet last run on db2016 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [04:38:05] RECOVERY - puppet last run on install2001 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [04:38:07] RECOVERY - puppet last run on cp4014 is OK: OK: Puppet is currently enabled, last run 56 seconds ago with 0 failures [04:38:20] RECOVERY - puppet last run on mw1233 is OK: OK: Puppet is currently enabled, last run 39 seconds ago with 0 failures [04:38:20] RECOVERY - puppet last run on cp4005 is OK: OK: Puppet is currently enabled, last run 24 seconds ago with 0 failures [04:38:22] RECOVERY - puppet last run on ms-be3002 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [04:38:22] RECOVERY - puppet last run on cp3016 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [04:38:22] RECOVERY - puppet last run on mw1127 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [04:38:23] RECOVERY - puppet last run on cp3003 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [04:38:23] RECOVERY - puppet last run on lvs3004 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [04:38:23] RECOVERY - puppet last run on amssq43 is OK: OK: Puppet is currently enabled, last run 38 seconds ago with 0 failures [04:38:23] RECOVERY - puppet last run on cp3013 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [04:38:24] RECOVERY - puppet last run on mw1115 is OK: OK: Puppet is currently enabled, last run 21 seconds ago with 0 failures [04:38:24] RECOVERY - puppet last run on mw1200 is OK: OK: Puppet is currently enabled, last run 27 seconds ago with 0 failures [04:38:31] RECOVERY - puppet last run on mw1008 is OK: OK: Puppet is currently enabled, last run 31 seconds ago with 0 failures [04:38:36] RECOVERY - puppet last run on search1017 is OK: OK: Puppet is currently enabled, last run 36 seconds ago with 0 failures [04:38:36] RECOVERY - puppet last run on virt1003 is OK: OK: Puppet is currently enabled, last run 3 minutes ago with 0 failures [04:38:36] RECOVERY - puppet last run on cp1048 is OK: OK: Puppet is currently enabled, last run 50 seconds ago with 0 failures [04:38:36] RECOVERY - puppet last run on amssq53 is OK: OK: Puppet is currently enabled, last run 10 seconds ago with 0 failures [04:38:36] RECOVERY - puppet last run on cp3020 is OK: OK: Puppet is currently enabled, last run 40 seconds ago with 0 failures [04:38:37] RECOVERY - puppet last run on amssq59 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [04:38:37] RECOVERY - puppet last run on mw1160 is OK: OK: Puppet is currently enabled, last run 9 seconds ago with 0 failures [04:38:38] RECOVERY - puppet last run on mw1196 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [04:38:38] RECOVERY - puppet last run on mw1189 is OK: OK: Puppet is currently enabled, last run 12 seconds ago with 0 failures [04:38:39] RECOVERY - puppet last run on ms-be1008 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [04:38:39] RECOVERY - puppet last run on wtp1018 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [04:38:40] RECOVERY - puppet last run on rhenium is OK: OK: Puppet is currently enabled, last run 3 minutes ago with 0 failures [04:38:40] RECOVERY - puppet last run on mw1026 is OK: OK: Puppet is currently enabled, last run 19 seconds ago with 0 failures [04:38:48] RECOVERY - puppet last run on mw1232 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [04:38:58] RECOVERY - puppet last run on search1016 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [04:38:58] RECOVERY - puppet last run on gallium is OK: OK: Puppet is currently enabled, last run 56 seconds ago with 0 failures [04:39:02] RECOVERY - puppet last run on mw1006 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [04:39:02] RECOVERY - puppet last run on mw1254 is OK: OK: Puppet is currently enabled, last run 35 seconds ago with 0 failures [04:39:02] RECOVERY - puppet last run on mw1063 is OK: OK: Puppet is currently enabled, last run 27 seconds ago with 0 failures [04:39:02] RECOVERY - puppet last run on snapshot1003 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [04:39:02] RECOVERY - puppet last run on mw1141 is OK: OK: Puppet is currently enabled, last run 41 seconds ago with 0 failures [04:39:03] RECOVERY - puppet last run on mw1216 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [04:39:03] RECOVERY - puppet last run on elastic1020 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [04:39:04] RECOVERY - puppet last run on mw1140 is OK: OK: Puppet is currently enabled, last run 32 seconds ago with 0 failures [04:39:04] RECOVERY - puppet last run on mw1109 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [04:39:05] RECOVERY - puppet last run on mw1009 is OK: OK: Puppet is currently enabled, last run 39 seconds ago with 0 failures [04:39:05] RECOVERY - puppet last run on mw1136 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [04:39:06] RECOVERY - puppet last run on mw1062 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [04:39:06] RECOVERY - puppet last run on mw1134 is OK: OK: Puppet is currently enabled, last run 51 seconds ago with 0 failures [04:39:07] RECOVERY - puppet last run on search1018 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [04:39:07] RECOVERY - puppet last run on mw1099 is OK: OK: Puppet is currently enabled, last run 15 seconds ago with 0 failures [04:39:08] RECOVERY - puppet last run on mw1240 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [04:39:08] RECOVERY - puppet last run on mw1028 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [04:39:09] RECOVERY - puppet last run on mw1138 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [04:39:09] RECOVERY - puppet last run on mw1174 is OK: OK: Puppet is currently enabled, last run 32 seconds ago with 0 failures [04:39:10] RECOVERY - puppet last run on ms-be1012 is OK: OK: Puppet is currently enabled, last run 3 minutes ago with 0 failures [04:39:10] RECOVERY - puppet last run on mw1245 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [04:39:12] RECOVERY - puppet last run on search1001 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [04:39:12] RECOVERY - puppet last run on mw1222 is OK: OK: Puppet is currently enabled, last run 46 seconds ago with 0 failures [04:39:12] RECOVERY - puppet last run on mw1100 is OK: OK: Puppet is currently enabled, last run 28 seconds ago with 0 failures [04:39:12] RECOVERY - puppet last run on mw1132 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [04:39:13] RECOVERY - puppet last run on mw1094 is OK: OK: Puppet is currently enabled, last run 3 minutes ago with 0 failures [04:39:13] RECOVERY - puppet last run on mw1036 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [04:39:14] RECOVERY - puppet last run on mw1092 is OK: OK: Puppet is currently enabled, last run 10 seconds ago with 0 failures [04:39:14] RECOVERY - puppet last run on sodium is OK: OK: Puppet is currently enabled, last run 3 minutes ago with 0 failures [04:39:15] RECOVERY - puppet last run on snapshot1004 is OK: OK: Puppet is currently enabled, last run 59 seconds ago with 0 failures [04:39:17] RECOVERY - puppet last run on mw1150 is OK: OK: Puppet is currently enabled, last run 31 seconds ago with 0 failures [04:39:26] RECOVERY - puppet last run on mw1004 is OK: OK: Puppet is currently enabled, last run 4 seconds ago with 0 failures [04:39:26] RECOVERY - puppet last run on mw1197 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [04:39:26] RECOVERY - puppet last run on mw1065 is OK: OK: Puppet is currently enabled, last run 26 seconds ago with 0 failures [04:39:26] RECOVERY - puppet last run on mw1147 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [04:39:27] RECOVERY - puppet last run on mw1003 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [04:39:27] RECOVERY - puppet last run on ms-be2012 is OK: OK: Puppet is currently enabled, last run 3 minutes ago with 0 failures [04:39:27] RECOVERY - puppet last run on mw1187 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [04:39:32] RECOVERY - puppet last run on cp1070 is OK: OK: Puppet is currently enabled, last run 3 minutes ago with 0 failures [04:39:36] RECOVERY - puppet last run on cp1055 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [04:39:39] RECOVERY - puppet last run on wtp1023 is OK: OK: Puppet is currently enabled, last run 3 minutes ago with 0 failures [04:39:39] RECOVERY - puppet last run on osmium is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [04:39:41] RECOVERY - puppet last run on mw1164 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [04:39:45] RECOVERY - puppet last run on amssq54 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [04:39:45] RECOVERY - puppet last run on mw1228 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [04:39:45] RECOVERY - puppet last run on mw1129 is OK: OK: Puppet is currently enabled, last run 23 seconds ago with 0 failures [04:39:45] RECOVERY - puppet last run on mw1244 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [04:39:45] RECOVERY - puppet last run on mw1247 is OK: OK: Puppet is currently enabled, last run 5 seconds ago with 0 failures [04:39:46] RECOVERY - puppet last run on mw1045 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [04:39:54] RECOVERY - puppet last run on mw1175 is OK: OK: Puppet is currently enabled, last run 48 seconds ago with 0 failures [04:39:54] RECOVERY - puppet last run on oxygen is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [04:39:54] RECOVERY - puppet last run on search1010 is OK: OK: Puppet is currently enabled, last run 3 minutes ago with 0 failures [04:39:54] RECOVERY - puppet last run on mw1114 is OK: OK: Puppet is currently enabled, last run 18 seconds ago with 0 failures [04:39:54] RECOVERY - puppet last run on mw1202 is OK: OK: Puppet is currently enabled, last run 29 seconds ago with 0 failures [04:39:55] RECOVERY - puppet last run on mw1221 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [04:39:55] RECOVERY - puppet last run on mw1072 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [04:39:56] RECOVERY - puppet last run on mw1120 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [04:40:03] RECOVERY - puppet last run on cp1063 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [04:40:07] RECOVERY - puppet last run on mw1119 is OK: OK: Puppet is currently enabled, last run 52 seconds ago with 0 failures [04:40:07] RECOVERY - puppet last run on cp4003 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [04:40:11] RECOVERY - puppet last run on amssq55 is OK: OK: Puppet is currently enabled, last run 43 seconds ago with 0 failures [04:40:11] RECOVERY - puppet last run on amssq44 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [04:40:17] RECOVERY - puppet last run on amssq60 is OK: OK: Puppet is currently enabled, last run 53 seconds ago with 0 failures [04:40:17] RECOVERY - puppet last run on amssq49 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [04:40:17] RECOVERY - puppet last run on amssq46 is OK: OK: Puppet is currently enabled, last run 31 seconds ago with 0 failures [04:40:17] RECOVERY - puppet last run on rubidium is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [04:40:17] RECOVERY - puppet last run on mw1162 is OK: OK: Puppet is currently enabled, last run 44 seconds ago with 0 failures [04:40:18] RECOVERY - puppet last run on mw1176 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [04:40:18] RECOVERY - puppet last run on tin is OK: OK: Puppet is currently enabled, last run 15 seconds ago with 0 failures [04:40:19] RECOVERY - puppet last run on search1002 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [04:40:19] RECOVERY - puppet last run on search1007 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [04:40:22] RECOVERY - puppet last run on mw1238 is OK: OK: Puppet is currently enabled, last run 36 seconds ago with 0 failures [04:40:22] RECOVERY - puppet last run on mw1061 is OK: OK: Puppet is currently enabled, last run 50 seconds ago with 0 failures [04:40:22] RECOVERY - puppet last run on mw1126 is OK: OK: Puppet is currently enabled, last run 46 seconds ago with 0 failures [04:40:22] RECOVERY - puppet last run on mw1077 is OK: OK: Puppet is currently enabled, last run 20 seconds ago with 0 failures [04:40:22] RECOVERY - puppet last run on mw1049 is OK: OK: Puppet is currently enabled, last run 19 seconds ago with 0 failures [04:40:23] RECOVERY - puppet last run on cp4008 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [04:40:23] RECOVERY - puppet last run on mw1208 is OK: OK: Puppet is currently enabled, last run 50 seconds ago with 0 failures [04:40:24] RECOVERY - puppet last run on cp3010 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [04:40:40] RECOVERY - puppet last run on mw1042 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [04:40:41] RECOVERY - puppet last run on tmh1001 is OK: OK: Puppet is currently enabled, last run 30 seconds ago with 0 failures [04:40:41] RECOVERY - puppet last run on mw1145 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [04:40:42] RECOVERY - puppet last run on mw1190 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [04:40:42] RECOVERY - puppet last run on mw1069 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [04:40:42] RECOVERY - puppet last run on mw1030 is OK: OK: Puppet is currently enabled, last run 30 seconds ago with 0 failures [04:40:42] RECOVERY - puppet last run on mw1124 is OK: OK: Puppet is currently enabled, last run 3 minutes ago with 0 failures [04:40:42] RECOVERY - puppet last run on mw1044 is OK: OK: Puppet is currently enabled, last run 53 seconds ago with 0 failures [04:40:43] RECOVERY - puppet last run on mw1153 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [04:40:43] RECOVERY - puppet last run on mw1089 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [04:40:44] RECOVERY - puppet last run on strontium is OK: OK: Puppet is currently enabled, last run 3 minutes ago with 0 failures [04:40:44] RECOVERY - puppet last run on mw1106 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [04:40:45] RECOVERY - puppet last run on cp1056 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [04:40:45] RECOVERY - puppet last run on mw1195 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [04:40:46] RECOVERY - puppet last run on snapshot1002 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [04:40:46] RECOVERY - puppet last run on mw1052 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [04:40:47] RECOVERY - puppet last run on mw1041 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [04:40:47] RECOVERY - puppet last run on mw1034 is OK: OK: Puppet is currently enabled, last run 35 seconds ago with 0 failures [04:40:48] RECOVERY - puppet last run on mw1252 is OK: OK: Puppet is currently enabled, last run 3 minutes ago with 0 failures [04:40:48] RECOVERY - puppet last run on cp4019 is OK: OK: Puppet is currently enabled, last run 50 seconds ago with 0 failures [04:40:49] RECOVERY - puppet last run on amssq32 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [04:40:49] RECOVERY - puppet last run on amssq47 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [04:40:50] RECOVERY - puppet last run on amssq34 is OK: OK: Puppet is currently enabled, last run 51 seconds ago with 0 failures [04:40:50] RECOVERY - puppet last run on mw1235 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [04:40:51] RECOVERY - puppet last run on amssq51 is OK: OK: Puppet is currently enabled, last run 45 seconds ago with 0 failures [04:40:51] RECOVERY - puppet last run on mw1002 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [04:40:52] RECOVERY - puppet last run on stat1002 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [04:40:52] RECOVERY - puppet last run on mw1177 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [04:40:53] RECOVERY - puppet last run on mw1084 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [04:40:53] RECOVERY - puppet last run on mw1046 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [04:40:57] RECOVERY - puppet last run on mw1156 is OK: OK: Puppet is currently enabled, last run 45 seconds ago with 0 failures [04:40:58] RECOVERY - puppet last run on mw1056 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [04:40:58] RECOVERY - puppet last run on mw1149 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [04:40:58] RECOVERY - puppet last run on mw1237 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [04:40:58] RECOVERY - puppet last run on search1005 is OK: OK: Puppet is currently enabled, last run 3 minutes ago with 0 failures [04:40:58] RECOVERY - puppet last run on mw1082 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [04:40:58] RECOVERY - puppet last run on mw1111 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [04:41:08] !log upgrade db1049 trusty [04:41:11] Logged the message, Master [04:42:22] RECOVERY - puppet last run on amssq42 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [04:42:22] RECOVERY - puppet last run on mw1014 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [04:42:23] RECOVERY - puppet last run on mw1146 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [04:42:23] RECOVERY - puppet last run on mw1011 is OK: OK: Puppet is currently enabled, last run 3 minutes ago with 0 failures [04:42:24] RECOVERY - puppet last run on mw1188 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [04:42:27] RECOVERY - puppet last run on mw1172 is OK: OK: Puppet is currently enabled, last run 3 minutes ago with 0 failures [04:42:27] RECOVERY - puppet last run on mw1144 is OK: OK: Puppet is currently enabled, last run 3 minutes ago with 0 failures [04:42:28] RECOVERY - puppet last run on mw1249 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [04:42:28] RECOVERY - puppet last run on mw1054 is OK: OK: Puppet is currently enabled, last run 3 minutes ago with 0 failures [04:42:28] RECOVERY - puppet last run on mw1180 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [04:42:28] RECOVERY - puppet last run on mw1079 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [04:42:28] RECOVERY - puppet last run on mw1181 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [04:42:29] RECOVERY - puppet last run on mw1076 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [04:42:38] RECOVERY - puppet last run on mw1125 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [04:42:40] RECOVERY - puppet last run on mw1170 is OK: OK: Puppet is currently enabled, last run 3 minutes ago with 0 failures [04:42:46] RECOVERY - puppet last run on terbium is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [04:42:47] RECOVERY - puppet last run on mw1159 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [04:42:57] RECOVERY - puppet last run on mw1057 is OK: OK: Puppet is currently enabled, last run 3 minutes ago with 0 failures [04:43:10] RECOVERY - puppet last run on amssq40 is OK: OK: Puppet is currently enabled, last run 3 minutes ago with 0 failures [04:43:56] RECOVERY - puppet last run on mw1059 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [04:44:11] PROBLEM - puppet last run on graphite1001 is CRITICAL: CRITICAL: Puppet has 1 failures [04:44:23] PROBLEM - puppet last run on elastic1005 is CRITICAL: CRITICAL: Puppet has 1 failures [04:47:13] RECOVERY - puppet last run on amssq35 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [04:48:42] RECOVERY - puppet last run on dataset1001 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [04:49:25] (03PS1) 10Springle: upgrade db1049 to trusty and mariadb 10 [puppet] - 10https://gerrit.wikimedia.org/r/178143 [04:50:38] (03CR) 10Springle: [C: 032] upgrade db1049 to trusty and mariadb 10 [puppet] - 10https://gerrit.wikimedia.org/r/178143 (owner: 10Springle) [04:52:32] RECOVERY - puppet last run on db2001 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [04:53:25] RECOVERY - puppet last run on graphite1001 is OK: OK: Puppet is currently enabled, last run 32 seconds ago with 0 failures [04:53:29] RECOVERY - puppet last run on elastic1005 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [05:06:32] PROBLEM - Router interfaces on mr1-esams is CRITICAL: CRITICAL: host 91.198.174.247, interfaces up: 36, down: 1, dormant: 0, excluded: 1, unused: 0BRge-0/0/0: down - Core: msw-oe12-esamsBR [05:06:41] (03PS1) 10Springle: repool db1049 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/178145 [05:07:20] (03CR) 10Springle: [C: 032] repool db1049 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/178145 (owner: 10Springle) [05:08:27] !log springle Synchronized wmf-config/db-eqiad.php: repool db1049, warm up (duration: 00m 06s) [05:08:32] Logged the message, Master [05:41:26] (03CR) 10MZMcBride: "Would string protection work here? That is, can we replace "https://phabricator.wikimedia.org/T1234" with "STRIP_MARKER_T/1234", have the " [puppet] - 10https://gerrit.wikimedia.org/r/177128 (owner: 10Krinkle) [05:50:26] PROBLEM - puppet last run on cp3013 is CRITICAL: CRITICAL: Puppet has 1 failures [06:02:14] RECOVERY - puppet last run on cp3013 is OK: OK: Puppet is currently enabled, last run 40 seconds ago with 0 failures [06:21:27] PROBLEM - graphite.wikimedia.org on tungsten is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 525 bytes in 0.001 second response time [06:30:45] RECOVERY - graphite.wikimedia.org on tungsten is OK: HTTP OK: HTTP/1.1 200 OK - 1607 bytes in 0.034 second response time [06:35:35] PROBLEM - puppet last run on cp3016 is CRITICAL: CRITICAL: Puppet has 1 failures [06:35:45] PROBLEM - puppet last run on analytics1030 is CRITICAL: CRITICAL: Puppet has 1 failures [06:45:54] RECOVERY - puppet last run on cp3016 is OK: OK: Puppet is currently enabled, last run 13 seconds ago with 0 failures [06:45:58] RECOVERY - puppet last run on analytics1030 is OK: OK: Puppet is currently enabled, last run 57 seconds ago with 0 failures [06:50:45] PROBLEM - puppet last run on amssq44 is CRITICAL: CRITICAL: Puppet has 1 failures [06:59:49] PROBLEM - puppet last run on carbon is CRITICAL: CRITICAL: Puppet last ran 2 days ago [06:59:59] PROBLEM - puppetmaster https on virt1000 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 8140: HTTP/1.1 500 Internal Server Error [07:02:49] RECOVERY - puppet last run on carbon is OK: OK: Puppet is currently enabled, last run 55 seconds ago with 0 failures [07:02:49] RECOVERY - puppet last run on amssq44 is OK: OK: Puppet is currently enabled, last run 13 seconds ago with 0 failures [07:15:08] This is good news for Docker at scale https://news.ycombinator.com/item?id=8715044 [07:15:28] can someone restart WDQ ? [07:15:31] https://wikitech.wikimedia.org/wiki/User:Yuvipanda/Restarting_magnus_wdq [07:47:15] PROBLEM - puppet last run on mw1244 is CRITICAL: CRITICAL: Puppet has 1 failures [08:02:23] RECOVERY - puppet last run on mw1244 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [08:10:10] (03CR) 10Hashar: "Sorry should have caught the red line :-(" [puppet] - 10https://gerrit.wikimedia.org/r/177864 (owner: 10Alexandros Kosiaris) [08:34:54] RECOVERY - Disk space on dataset1001 is OK: DISK OK [09:01:05] (03PS1) 10Faidon Liambotis: Kill unused files/misc/update-repository [puppet] - 10https://gerrit.wikimedia.org/r/178153 [09:01:07] (03PS1) 10Faidon Liambotis: autoinstall: add missing subnets to preseed.cfg [puppet] - 10https://gerrit.wikimedia.org/r/178154 [09:01:09] (03PS1) 10Faidon Liambotis: autoinstall: kill i386/i686 variants [puppet] - 10https://gerrit.wikimedia.org/r/178155 [09:01:11] (03PS1) 10Faidon Liambotis: autoinstall: switch to autoinstall/enable=true [puppet] - 10https://gerrit.wikimedia.org/r/178156 [09:01:13] (03PS1) 10Faidon Liambotis: autoinstall: add jessie-installer [puppet] - 10https://gerrit.wikimedia.org/r/178157 [09:01:15] (03PS1) 10Faidon Liambotis: autoinstall: preseed xkb-keymap for jessie [puppet] - 10https://gerrit.wikimedia.org/r/178158 [09:01:17] (03PS1) 10Faidon Liambotis: autoinstall: s/--/---/ for jessie's kernel cmdline [puppet] - 10https://gerrit.wikimedia.org/r/178159 [09:01:19] (03PS1) 10Faidon Liambotis: autoinstall: set get_domain=unassigned on jessie [puppet] - 10https://gerrit.wikimedia.org/r/178160 [09:01:21] (03PS1) 10Faidon Liambotis: autoinstall: switch subnet detection to default gw [puppet] - 10https://gerrit.wikimedia.org/r/178161 [09:01:23] (03PS1) 10Faidon Liambotis: autoinstall: switch based on distroname too [puppet] - 10https://gerrit.wikimedia.org/r/178162 [09:01:25] (03PS1) 10Faidon Liambotis: autoinstall: preseed popularity-contest [puppet] - 10https://gerrit.wikimedia.org/r/178163 [09:01:27] (03PS1) 10Faidon Liambotis: autoinstall: tell d-i to not install Recommends [puppet] - 10https://gerrit.wikimedia.org/r/178164 [09:01:29] (03PS1) 10Faidon Liambotis: reprepro: add jessie-wikimedia to reprepro [puppet] - 10https://gerrit.wikimedia.org/r/178165 [09:01:31] (03PS1) 10Faidon Liambotis: apt: switch components based on $lsbdistid [puppet] - 10https://gerrit.wikimedia.org/r/178166 [09:01:33] (03PS1) 10Faidon Liambotis: autoinstall: apt components based on Debian/Ubuntu [puppet] - 10https://gerrit.wikimedia.org/r/178167 [09:01:35] (03PS1) 10Faidon Liambotis: autoinstall: move rootdelay=90 to Ubuntu-specific [puppet] - 10https://gerrit.wikimedia.org/r/178168 [09:01:53] have fun, jenkins [09:03:28] (03CR) 10Faidon Liambotis: [C: 032] Kill unused files/misc/update-repository [puppet] - 10https://gerrit.wikimedia.org/r/178153 (owner: 10Faidon Liambotis) [09:03:46] (03CR) 10Faidon Liambotis: [C: 032] autoinstall: add missing subnets to preseed.cfg [puppet] - 10https://gerrit.wikimedia.org/r/178154 (owner: 10Faidon Liambotis) [09:04:02] (03CR) 10Faidon Liambotis: [C: 032] autoinstall: kill i386/i686 variants [puppet] - 10https://gerrit.wikimedia.org/r/178155 (owner: 10Faidon Liambotis) [09:04:12] (03CR) 10jenkins-bot: [V: 04-1] apt: switch components based on $lsbdistid [puppet] - 10https://gerrit.wikimedia.org/r/178166 (owner: 10Faidon Liambotis) [09:06:12] (03PS2) 10Faidon Liambotis: autoinstall: move rootdelay=90 to Ubuntu-specific [puppet] - 10https://gerrit.wikimedia.org/r/178168 [09:06:14] (03PS2) 10Faidon Liambotis: autoinstall: apt components based on Debian/Ubuntu [puppet] - 10https://gerrit.wikimedia.org/r/178167 [09:06:16] (03PS2) 10Faidon Liambotis: apt: switch components based on $lsbdistid [puppet] - 10https://gerrit.wikimedia.org/r/178166 [09:07:40] (03CR) 10Faidon Liambotis: [C: 032] "Tested." [puppet] - 10https://gerrit.wikimedia.org/r/178156 (owner: 10Faidon Liambotis) [09:08:46] (03CR) 10Faidon Liambotis: [C: 032] autoinstall: add jessie-installer [puppet] - 10https://gerrit.wikimedia.org/r/178157 (owner: 10Faidon Liambotis) [09:09:03] (03CR) 10Faidon Liambotis: [C: 032] autoinstall: preseed xkb-keymap for jessie [puppet] - 10https://gerrit.wikimedia.org/r/178158 (owner: 10Faidon Liambotis) [09:09:23] (03CR) 10Faidon Liambotis: [C: 032] autoinstall: s/--/---/ for jessie's kernel cmdline [puppet] - 10https://gerrit.wikimedia.org/r/178159 (owner: 10Faidon Liambotis) [09:11:39] RECOVERY - puppetmaster https on virt1000 is OK: HTTP OK: Status line output matched 400 - 335 bytes in 0.040 second response time [09:17:46] (03PS3) 10Faidon Liambotis: autoinstall: move rootdelay=90 to Ubuntu-specific [puppet] - 10https://gerrit.wikimedia.org/r/178168 [09:17:48] (03PS2) 10Faidon Liambotis: autoinstall: switch subnet detection to default gw [puppet] - 10https://gerrit.wikimedia.org/r/178161 [09:17:50] (03PS2) 10Faidon Liambotis: autoinstall: set get_domain=unassigned on jessie [puppet] - 10https://gerrit.wikimedia.org/r/178160 [09:17:52] (03PS2) 10Faidon Liambotis: autoinstall: preseed popularity-contest [puppet] - 10https://gerrit.wikimedia.org/r/178163 [09:17:54] (03PS2) 10Faidon Liambotis: autoinstall: switch based on distroname too [puppet] - 10https://gerrit.wikimedia.org/r/178162 [09:17:56] (03PS2) 10Faidon Liambotis: reprepro: add jessie-wikimedia to reprepro [puppet] - 10https://gerrit.wikimedia.org/r/178165 [09:17:58] (03PS2) 10Faidon Liambotis: autoinstall: tell d-i to not install Recommends [puppet] - 10https://gerrit.wikimedia.org/r/178164 [09:18:00] (03PS3) 10Faidon Liambotis: autoinstall: apt components based on Debian/Ubuntu [puppet] - 10https://gerrit.wikimedia.org/r/178167 [09:18:02] (03PS3) 10Faidon Liambotis: apt: switch components based on $::lsbdistid [puppet] - 10https://gerrit.wikimedia.org/r/178166 [09:18:19] !log graceful'd apache on virt1000 -- resolving a mysterious puppetmaster outage [09:18:26] Logged the message, Master [09:18:40] !log the failure looked like this: "Unexpected error in mod_passenger: Could not connect to the ApplicationPool server: Broken pipe (32)" [09:18:43] Logged the message, Master [09:25:04] greetings [09:25:58] good morning [09:26:18] godog: reviews for ^^^ very much welcome :) [09:26:39] paravoid: heheh sure, thanks for working on that! jessie on the way \o/ [09:30:26] multiple bugs/workarounds as you'll see [09:34:36] indeed, started from https://gerrit.wikimedia.org/r/#/c/178160/ and it is mildly painful already :P [09:35:47] (03PS1) 10Nemo bis: Update cached article count monthly to avoid social unrest [puppet] - 10https://gerrit.wikimedia.org/r/178170 [09:35:57] any experience with Debian mirroring foo btw? [09:36:29] I'm using http.debian.net + squid (webproxy.eqiad.wmnet) for now but why not set up our own mirror a la ubuntu [09:36:58] there's ftpsync, which is the official way but a horrible unreadable non-packaged shell script [09:37:03] then there's a bunch of others [09:37:51] (03PS3) 10Qgil: phabricator: community metrics stats mail [puppet] - 10https://gerrit.wikimedia.org/r/177792 (owner: 10Dzahn) [09:38:02] I've had success with debmirror in the past (over rsync IIRC) and it is pretty straightforward [09:38:16] me too [09:39:31] yeah it was my top contender :) [09:40:54] Is this a different issue from the one solved by reprepro currently? [09:41:00] yes [09:41:02] (Or maybe reprepro is just the config tool?) [09:41:05] no [09:41:10] reprepro is for our own repository [09:41:20] Ah, I see, vs. mirroring. [09:41:23] I'm talking about a mirror of Debian upstream [09:41:25] right [09:41:31] this is apt.wm.org vs. ubuntu.wm.org currently [09:41:34] (03CR) 10Filippo Giunchedi: [C: 032] autoinstall: set get_domain=unassigned on jessie [puppet] - 10https://gerrit.wikimedia.org/r/178160 (owner: 10Faidon Liambotis) [09:50:26] paravoid: have you worked on any of the new HP servers? [09:50:32] no [09:51:04] why? [09:51:14] Eh, can't get mine to pxe-boot. [09:51:18] andrewbogott: i'm waiting for them eagerly :) [09:51:27] what's the issue you're seeing? [09:51:28] I think that Chris and Jeff had this same problem last week, emailed. [09:51:48] "PXE-E61: Media test failure, check cable" [09:51:54] Not much to go on there [09:52:01] for which card? [09:52:02] lemme make sure all three fail the same way [09:52:18] if all fail like this, it's a cable problem :) [09:52:23] or a switch config issue [09:52:25] which server is this? [09:52:28] I believe this box has only one nic, but let me check... [09:52:31] virt1011 [09:53:01] ge-5/0/7 up up virt1011 [09:53:21] virt1010 and 1012 are the same hardware [09:53:30] Although I'm not positive the raid is set up on those [09:53:52] ge-5/0/8 up up virt1012 [09:54:04] (03CR) 10Filippo Giunchedi: "nitpick comment, LGTM" (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/178161 (owner: 10Faidon Liambotis) [09:54:15] yeah, so, not unplugged. [09:55:21] maybe that's for the other nic [09:55:38] I doubt there's any server on the market nowadays that only has one nic [09:55:48] Oh, well, there mgmt [09:55:51] (03CR) 10Filippo Giunchedi: [C: 032] autoinstall: switch based on distroname too [puppet] - 10https://gerrit.wikimedia.org/r/178162 (owner: 10Faidon Liambotis) [09:56:08] mgmt I can tell is plugged in since I can connect [09:56:16] (03CR) 10Filippo Giunchedi: [C: 032] autoinstall: preseed popularity-contest [puppet] - 10https://gerrit.wikimedia.org/r/178163 (owner: 10Faidon Liambotis) [09:56:58] (03CR) 10Filippo Giunchedi: [C: 032] autoinstall: move rootdelay=90 to Ubuntu-specific [puppet] - 10https://gerrit.wikimedia.org/r/178168 (owner: 10Faidon Liambotis) [09:57:34] iLO4_MACAddress=9c:b6:54:ae:62:70 [09:57:35] Port1NIC_MACAddress=f0:92:1c:05:4a:98 [09:57:36] Port2NIC_MACAddress=f0:92:1c:05:4a:9c [09:57:43] So I could try booting from Port2 instead [09:57:49] * andrewbogott tries [09:58:11] maybe it's cabled wrong [09:58:17] port2 instead of port1 [09:58:35] yeah, might not be obvious which is which from outside the box [09:59:03] Do you think it would pxe-boot from port2 if that's the one hooked up? [09:59:24] not sure, sometimes it's disabled in the bios [10:00:51] (03PS1) 10Andrew Bogott: Stab in the dark: Try booting virt1011 from port2 [puppet] - 10https://gerrit.wikimedia.org/r/178172 [10:01:05] you don't need this to try it :) [10:01:15] I don't? [10:01:22] you'd see DHCP trying instead of "cable failure" [10:01:31] DHCP will fail of course [10:01:43] but it'll be a different error [10:01:48] You're optimistic about the quality of error reporting :) [10:02:09] you're also going to see the mac on the dhcp logs [10:02:16] with a "no free leases" message [10:02:38] root@carbon:/var/log# grep f0:92:1c:05:4a: syslog [10:02:38] root@carbon:/var/log# [10:02:39] fwiw [10:04:00] hunting in the bios for setting options [10:05:04] (03CR) 10Filippo Giunchedi: reprepro: add jessie-wikimedia to reprepro (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/178165 (owner: 10Faidon Liambotis) [10:05:47] (03CR) 10Filippo Giunchedi: [C: 032] apt: switch components based on $::lsbdistid [puppet] - 10https://gerrit.wikimedia.org/r/178166 (owner: 10Faidon Liambotis) [10:06:44] godog: there's a mess with arch:all packages right now [10:07:06] I just copied trusty, which is non-empty [10:07:35] but maybe you're right and we should fix this now [10:07:43] I also want to s/universe/backports/ on all distros [10:07:56] and I also want to create a third-party component and have reprepro updates write there instead of main [10:08:02] (and cleanup the existing repositories) [10:08:18] if the fix doesn't snowball into something much bigger then yeah I think we should start fresh [10:08:32] I'm pondering whether I should do all that now or just leave it for later [10:09:14] I wanted to do all this when I did the trusty work too [10:09:17] and left it for later :) [10:09:28] hehe it is also usually the case of "I'll just fix this" and then the house of cards half-collapses :P [10:09:42] the thing is, it's a bit tricky [10:09:45] because of pinning [10:09:58] if the packages disappear form systems, they might get upgraded to a non-wikimedia version [10:11:13] oh mhh, though new uploads of arch: all into jessie-wikimedia shouldn't be affected even if there is no i386? [10:11:23] perhaps I'm misunderstading the arch: all mess [10:11:39] oh i'm talking about different things, sorry :) [10:11:54] I think we should drop i386 from all suites [10:12:09] I'm not sure how easy it's going to be [10:12:16] I should check it out [10:12:50] oh ok, I see what you mean now, perhaps start with jessie without i386 and we're leaving behind the rest over time anyway [10:13:25] I mean, we're not uploading i386 packages anyway [10:16:28] (03CR) 10Filippo Giunchedi: [C: 032] autoinstall: tell d-i to not install Recommends [puppet] - 10https://gerrit.wikimedia.org/r/178164 (owner: 10Faidon Liambotis) [10:41:41] !log upload diamond 3.5-2 to trusty-wikimedia [10:41:45] Logged the message, Master [10:45:58] PROBLEM - puppet last run on amssq58 is CRITICAL: CRITICAL: puppet fail [10:54:09] (03CR) 10Filippo Giunchedi: "see comment, perhaps we can use something already in place for this?" (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/177801 (owner: 10Dzahn) [10:55:35] (03CR) 10Filippo Giunchedi: [C: 031] "another option would be to rename (or alias) the existing 'universe' component with 'backports' as discussed with Faidon. I'm +1 on this f" [puppet] - 10https://gerrit.wikimedia.org/r/178167 (owner: 10Faidon Liambotis) [11:03:32] RECOVERY - puppet last run on amssq58 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [11:34:00] (03PS1) 10Andrew Bogott: On labs, raise the timeout for apt. [puppet] - 10https://gerrit.wikimedia.org/r/178181 [11:34:16] YuviPanda: ^ maybe? [11:35:03] andrewbogott: 5 mins seems enough, yeah. [11:35:09] andrewbogott: let’s try :) [11:35:45] PROBLEM - puppet last run on mw1057 is CRITICAL: CRITICAL: Puppet has 1 failures [11:37:32] (03CR) 10Andrew Bogott: [C: 032] On labs, raise the timeout for apt. [puppet] - 10https://gerrit.wikimedia.org/r/178181 (owner: 10Andrew Bogott) [11:50:49] RECOVERY - puppet last run on mw1057 is OK: OK: Puppet is currently enabled, last run 40 seconds ago with 0 failures [12:17:58] (03PS1) 10Yuvipanda: shinken: Add fixed up init script [puppet] - 10https://gerrit.wikimedia.org/r/178184 [12:30:40] (03CR) 10Yuvipanda: [C: 032] shinken: Add fixed up init script [puppet] - 10https://gerrit.wikimedia.org/r/178184 (owner: 10Yuvipanda) [12:34:40] YuviPanda: are you a dm or dd ? [12:35:57] matanya: nah, just a super newbie packager :) neither [12:37:02] oh, thanks. i'm planning to package some mediawiki-perl packages, packaged the prerequisites, and looking for a review before i go to the heavy lifting [12:41:04] (03PS1) 10Gergő Tisza: Attempt to fix double-encoding in old-bugzilla HTTPS redirects [puppet] - 10https://gerrit.wikimedia.org/r/178186 [13:10:33] !log Zuul: rebasing our fork to bring some upstream changes [13:10:37] Logged the message, Master [13:11:47] !log Restarting zuul and zuul-merger on gallium [13:11:49] Logged the message, Master [14:14:42] (03PS1) 10Christopher Johnson (WMDE): adds Sprint extension to role::phabricator::main [puppet] - 10https://gerrit.wikimedia.org/r/178194 [14:19:26] (03PS1) 10Filippo Giunchedi: diamond: require python-statsd [puppet] - 10https://gerrit.wikimedia.org/r/178195 [14:34:59] (03CR) 10Anomie: robots.txt: Use max lastmod time (031 comment) [mediawiki-config] - 10https://gerrit.wikimedia.org/r/177940 (owner: 10BryanDavis) [14:36:06] !log reboot graphite1001 for kernel upgrade [14:36:12] Logged the message, Master [14:45:49] (03CR) 10Mark Bergsma: [C: 031] "Sure, agree with the concept. Using the broadcast address was just easier back in 2006." [puppet] - 10https://gerrit.wikimedia.org/r/178161 (owner: 10Faidon Liambotis) [15:03:29] !log Broke zuul-cloner by mistake [15:03:36] Logged the message, Master [15:14:23] (03PS2) 10Christopher Johnson (WMDE): adds Sprint extension (0.6.1.0) to role::phabricator::main [puppet] - 10https://gerrit.wikimedia.org/r/178194 [15:26:46] ori: apergos: Please remove https://dumps.wikimedia.org/other/wikidata/20141208.json.gz [15:26:52] incomplete [15:26:56] shouldn't be public [15:27:06] or any other op [15:27:55] Resides on one of the dataset hosts, can be deleted from a snapshot host [15:28:16] resides in /mnt/data/public/... AFAIR [15:28:20] mutante: ^ [15:29:40] (03Restored) 10Hoo man: Allow "hoo" to sudo into datasets [puppet] - 10https://gerrit.wikimedia.org/r/152724 (owner: 10Hoo man) [15:44:27] impatience [15:52:18] marktraceur, ^d, manybubbles: Who wants to SWAT this morning? [15:53:03] anomie: meh. I can totally do it [15:53:18] manybubbles: ok! [15:53:46] Don't sound so excited [15:53:49] :) [15:54:01] <^d> I can do it too. [15:56:37] <^d> I'm already on tin, I'll take it [15:57:41] <^d> James_F: yo yo. time for swat :) [15:57:49] * James_F nods. [15:57:54] I'm here. [15:58:03] <^d> Okie doke, lemme start the gerrit dance. [15:58:06] <^d> *dokie [15:58:18] Whee. [16:00:04] manybubbles, anomie, ^d, marktraceur, James_F: Respected human, time to deploy Morning SWAT (Max 8 patches) (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20141208T1600). Please do the needful. [16:00:17] k [16:00:25] ^d: have fun [16:00:29] <^d> I shall! [16:01:14] ^d: you know, logging into tin is ctrl-shift-t, ctrl-r, tin, enter, cd /srv/med, tab, s, tab, enter. its takes < a second [16:01:19] (03CR) 10Ottomata: "Hm, it is possible that the mediawiki actually requires ssh server to be set up before mediawiki. Maybe in the case of installing new app" [puppet] - 10https://gerrit.wikimedia.org/r/177863 (owner: 10Alexandros Kosiaris) [16:02:01] (03CR) 10Ottomata: "This also would not run into the problem of not being able to override class parameters, as all this does is introduce a dependency. It d" [puppet] - 10https://gerrit.wikimedia.org/r/177863 (owner: 10Alexandros Kosiaris) [16:02:47] <^d> manybubbles: Sssh, don't let everyone know how easy it is ;-) [16:03:13] actually running the sync is a bit more complex. I mean you have to pay attention to directories and stuff [16:03:24] and logstash! [16:04:03] <^d> I miss /home/wikipedia/common sometimes :) [16:05:04] It all should be reduced to clicking a link in Jenkins (or a Jenkins wrapper application) [16:05:12] <^d> blasphemy. [16:05:20] https://github.com/etsy/deployinator [16:05:20] bd808: +1. [16:05:34] bd808: Or, maybe I should say "+2" given who would have the rights. :-) [16:08:35] !log demon Synchronized php-1.25wmf10/extensions/VisualEditor: (no message) (duration: 00m 09s) [16:08:40] Logged the message, Master [16:08:46] !log demon Synchronized php-1.25wmf11/extensions/VisualEditor: (no message) (duration: 00m 06s) [16:08:48] Logged the message, Master [16:08:52] <^d> James_F: happy fun times ^ [16:08:54] <^d> plz verify [16:09:03] * James_F does so. [16:10:20] ^d: WFM on enwiki. Testing MW.org [16:11:32] ^d: Yup, seems to be working. Thanks! [16:11:37] <^d> yw. [16:11:39] aaah, missed swat [16:11:56] we'll be around again later [16:12:09] aude: If you ask ^d very nicely… :-) [16:12:21] it would take quite a few minutes to prepare [16:12:52] <^d> Aw, guess not then. [16:18:15] <^d> Ok, declaring swat closed. Thanks for playing James_F. [16:18:22] <^d> aude: More prizes next time! [16:18:47] :) [16:21:56] !log Jenkins: disconnected / reconnected gallium slave from the web interface. It was locked not being able to run the mediawiki/vagrant postmerge doc job [16:21:58] Logged the message, Master [16:45:41] (03PS1) 10Ori.livneh: hhvm: on mw1081, load tidy.so [puppet] - 10https://gerrit.wikimedia.org/r/178209 [16:48:24] (03PS2) 10Ori.livneh: hhvm: on mw1081, load tidy.so [puppet] - 10https://gerrit.wikimedia.org/r/178209 [16:48:46] (03CR) 10Ori.livneh: [C: 032 V: 032] hhvm: on mw1081, load tidy.so [puppet] - 10https://gerrit.wikimedia.org/r/178209 (owner: 10Ori.livneh) [16:56:45] !log doing controlled restart of kafka broker analytics1021, and then initiating replica election to bring it back into leadership [16:56:51] Logged the message, Master [16:58:04] (03PS1) 10Cmjohnson: Changing virt1011 mac address [puppet] - 10https://gerrit.wikimedia.org/r/178211 [16:58:55] (03CR) 10Cmjohnson: [C: 032] Changing virt1011 mac address [puppet] - 10https://gerrit.wikimedia.org/r/178211 (owner: 10Cmjohnson) [16:59:20] (03PS1) 10Ori.livneh: Enable tidy extension on mw1081 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/178212 [17:01:57] (03CR) 10Ori.livneh: [C: 032] Enable tidy extension on mw1081 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/178212 (owner: 10Ori.livneh) [17:02:05] (03Merged) 10jenkins-bot: Enable tidy extension on mw1081 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/178212 (owner: 10Ori.livneh) [17:02:38] !log ori Synchronized wmf-config/CommonSettings.php: enable tidy extension on mw1081 (duration: 00m 06s) [17:02:40] Logged the message, Master [17:27:52] (03CR) 10Gage: [C: 032] "ok because only affects beta cluster, see comments about future use in prod" [puppet] - 10https://gerrit.wikimedia.org/r/143788 (https://bugzilla.wikimedia.org/60690) (owner: 10BryanDavis) [17:29:21] (03PS13) 10Gage: Allow puppetmaster to send reports to logstash [puppet] - 10https://gerrit.wikimedia.org/r/143788 (https://bugzilla.wikimedia.org/60690) (owner: 10BryanDavis) [17:29:31] jgage: omg. Thanks! [17:29:46] :D [17:30:11] will merge after jenkins is happy with rebase [17:30:33] (03CR) 10Gage: [C: 032] Allow puppetmaster to send reports to logstash [puppet] - 10https://gerrit.wikimedia.org/r/143788 (https://bugzilla.wikimedia.org/60690) (owner: 10BryanDavis) [17:30:54] done [17:31:20] (03PS2) 10Filippo Giunchedi: diamond: require python-statsd [puppet] - 10https://gerrit.wikimedia.org/r/178195 [17:31:21] * bd808 will check that it merges properly in beta [17:31:27] (03CR) 10Filippo Giunchedi: [C: 032 V: 032] diamond: require python-statsd [puppet] - 10https://gerrit.wikimedia.org/r/178195 (owner: 10Filippo Giunchedi) [17:32:08] thanks bd808 [17:40:32] (03PS1) 10Ori.livneh: load tidy.so everywhere (but don't enable it) [puppet] - 10https://gerrit.wikimedia.org/r/178217 [17:40:44] (03PS2) 10Ori.livneh: load tidy.so everywhere (but don't enable it) [puppet] - 10https://gerrit.wikimedia.org/r/178217 [17:40:55] (03CR) 10Ori.livneh: [C: 032 V: 032] load tidy.so everywhere (but don't enable it) [puppet] - 10https://gerrit.wikimedia.org/r/178217 (owner: 10Ori.livneh) [17:41:44] PROBLEM - puppet last run on ms-be1010 is CRITICAL: CRITICAL: puppet fail [17:42:24] godog: Error: Could not retrieve catalog from remote server: Error 400 on SERVER: Duplicate declaration: Package[python-statsd] is already declared; cannot redeclare at /etc/puppet/manifests/swift.pp:39 on node ms-be1010.eqiad.wmnet [17:43:11] I guess I have to use require_package everywhere don't I ori ? [17:43:19] godog: yep [17:44:30] ack, fixing it now [17:45:01] I wonder how that is "fixed" in puppet without our custom function to allow duplicate packages [17:45:29] PROBLEM - puppet last run on ms-be2009 is CRITICAL: CRITICAL: puppet fail [17:46:40] (03CR) 10Ori.livneh: "Alexandros: it includes it because of the ssh override below, which notifies Service['ssh']. One easier solution would be to move that ssh" [puppet] - 10https://gerrit.wikimedia.org/r/177863 (owner: 10Alexandros Kosiaris) [17:47:57] manybubbles: ping [17:48:06] gwicke: pong [17:48:32] hey; I was just searching for a ticket I created an hour ago in phabricator, and got no results [17:48:52] could that be elasticsearch indexing delay? [17:49:02] <^d> It's not elasticsearch delay. [17:49:04] <^d> Phab delay. [17:49:24] <^d> PhabricatorSearchWorker 242,538 [17:49:26] I was searching for the subject line in https://phabricator.wikimedia.org/T77172 [17:49:28] PROBLEM - puppet last run on ms-be1004 is CRITICAL: CRITICAL: puppet fail [17:49:40] ^d: ah, I see [17:49:46] ^d: I told you you might want to do some priority fiddling.... [17:49:53] phab is capable of causing 1hr delay? yikes. [17:50:21] <^d> manybubbles: I couldn't care less about the search jobs at the moment really. It's backing up on e-mail jobs. [17:50:31] oh god [17:50:37] (03PS4) 10Faidon Liambotis: autoinstall: move rootdelay=90 to Ubuntu-specific [puppet] - 10https://gerrit.wikimedia.org/r/178168 [17:50:39] (03PS3) 10Faidon Liambotis: autoinstall: switch subnet detection to default gw [puppet] - 10https://gerrit.wikimedia.org/r/178161 [17:50:41] (03PS3) 10Faidon Liambotis: autoinstall: set get_domain=unassigned on jessie [puppet] - 10https://gerrit.wikimedia.org/r/178160 [17:50:43] (03PS3) 10Faidon Liambotis: autoinstall: preseed popularity-contest [puppet] - 10https://gerrit.wikimedia.org/r/178163 [17:50:47] (03PS3) 10Faidon Liambotis: autoinstall: switch based on distroname too [puppet] - 10https://gerrit.wikimedia.org/r/178162 [17:50:49] (03PS3) 10Faidon Liambotis: reprepro: add jessie-wikimedia to reprepro [puppet] - 10https://gerrit.wikimedia.org/r/178165 [17:50:51] (03PS3) 10Faidon Liambotis: autoinstall: tell d-i to not install Recommends [puppet] - 10https://gerrit.wikimedia.org/r/178164 [17:50:53] (03PS4) 10Faidon Liambotis: autoinstall: apt components based on Debian/Ubuntu [puppet] - 10https://gerrit.wikimedia.org/r/178167 [17:50:55] (03PS4) 10Faidon Liambotis: apt: switch components based on $::lsbdistid [puppet] - 10https://gerrit.wikimedia.org/r/178166 [17:50:59] (03PS1) 10Ori.livneh: Unset $wgTidyInternal, so its default value is used. [mediawiki-config] - 10https://gerrit.wikimedia.org/r/178220 [17:51:01] (03Abandoned) 10Ori.livneh: Unset $wgTidyInternal, so its default value is used. [mediawiki-config] - 10https://gerrit.wikimedia.org/r/176884 (owner: 10Ori.livneh) [17:51:03] (03Abandoned) 10Ori.livneh: Enable internal tidy if extension is loaded and hostname == mw1114 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/176880 (owner: 10Ori.livneh) [17:51:05] jgage: it has its own job queue and if you add lots of stuff fast it'll get backed up [17:51:17] and the job executors aren't like ours - everything gets in a single line [17:51:20] looks like gerrit knows how to delay things too [17:52:01] (03PS1) 10Filippo Giunchedi: python-statsd: use require_package everywhere [puppet] - 10https://gerrit.wikimedia.org/r/178221 [17:52:05] (03CR) 10Ori.livneh: [C: 032] Unset $wgTidyInternal, so its default value is used. [mediawiki-config] - 10https://gerrit.wikimedia.org/r/178220 (owner: 10Ori.livneh) [17:52:22] (03Merged) 10jenkins-bot: Unset $wgTidyInternal, so its default value is used. [mediawiki-config] - 10https://gerrit.wikimedia.org/r/178220 (owner: 10Ori.livneh) [17:52:43] ori: https://gerrit.wikimedia.org/r/#/c/178221/ if you spare 2min [17:52:46] !log rebuilding eswiki's cirrus index to pick up fix for slow prefix searches [17:52:51] Logged the message, Master [17:52:55] (03CR) 10Faidon Liambotis: [C: 032] autoinstall: set get_domain=unassigned on jessie [puppet] - 10https://gerrit.wikimedia.org/r/178160 (owner: 10Faidon Liambotis) [17:53:09] (03CR) 10Faidon Liambotis: [C: 032] autoinstall: switch subnet detection to default gw [puppet] - 10https://gerrit.wikimedia.org/r/178161 (owner: 10Faidon Liambotis) [17:53:28] (03CR) 10Faidon Liambotis: [C: 032] autoinstall: switch based on distroname too [puppet] - 10https://gerrit.wikimedia.org/r/178162 (owner: 10Faidon Liambotis) [17:53:47] (03CR) 10Faidon Liambotis: [C: 032] autoinstall: move rootdelay=90 to Ubuntu-specific [puppet] - 10https://gerrit.wikimedia.org/r/178168 (owner: 10Faidon Liambotis) [17:54:00] (03CR) 10Faidon Liambotis: [C: 032] autoinstall: preseed popularity-contest [puppet] - 10https://gerrit.wikimedia.org/r/178163 (owner: 10Faidon Liambotis) [17:54:15] (03CR) 10Faidon Liambotis: [C: 032] autoinstall: tell d-i to not install Recommends [puppet] - 10https://gerrit.wikimedia.org/r/178164 (owner: 10Faidon Liambotis) [17:54:40] (these aren't self-merges, fwiw :) [17:54:54] !log ori Synchronized wmf-config/CommonSettings.php: Unset \$wgTidyInternal (duration: 00m 07s) [17:54:57] Logged the message, Master [17:56:10] PROBLEM - puppet last run on ms-fe3002 is CRITICAL: CRITICAL: puppet fail [17:58:08] paravoid: happy graphs is happy: http://ganglia.wikimedia.org/latest/graph.php?r=year&z=xlarge&c=Memcached+eqiad&m=cpu_report&s=by+name&mc=2&g=network_report [17:58:15] well done hoo|away, aude [17:58:17] \o/ [17:58:22] \o/ \o/ [17:58:40] is this the sitelistcache? [17:58:42] yeah [17:58:46] or just hoo's work from last week? [17:58:56] no, i think it's the sitelistcache too [17:59:07] I thought sitelistcache was going in with wmf11, did someone backport it? [17:59:22] oh hm [17:59:36] RECOVERY - puppet last run on ms-be1010 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [18:00:08] paravoid: yes: https://gerrit.wikimedia.org/r/#/c/177693/ [18:00:39] RECOVERY - puppet last run on ms-be2009 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [18:00:39] beat me to it ;) [18:00:52] <^d> manybubbles, gwicke: We upped the number of runners for phabricator jobs. Hopefully the backlog will start going down a bit. [18:01:00] <^d> Will be keeping a close eye on it [18:01:14] ^d: thanks! [18:02:35] ori: that's not SiteListFileCache [18:02:46] that would be https://gerrit.wikimedia.org/r/#/c/174874/ [18:03:35] oh, that hasn't been cherry-picked it seems [18:04:11] nope [18:04:33] but I guess it won't make a difference now :) [18:04:36] !log removed restbase/ from graphite for T77172 on tungsten [18:04:38] Logged the message, Master [18:04:47] RECOVERY - puppet last run on ms-be1004 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [18:06:40] ori: http://ganglia.wikimedia.org/latest/graph.php?r=month&z=xlarge&c=Application+servers+eqiad&m=cpu_report&s=by+name&mc=2&g=cpu_report [18:06:44] is that last drop hhvm? [18:06:52] ori is it related/ [18:07:03] because it looks awfully corellated with http://ganglia.wikimedia.org/latest/graph.php?r=month&z=xlarge&c=Application+servers+eqiad&m=cpu_report&s=by+name&mc=2&g=network_report [18:07:48] oh it might be your stat_cache fix, did you deploy this across the cluster? [18:07:52] dunno, in the middle of something but can look then [18:07:55] no, not yet [18:07:58] that's just mw1081 still [18:08:18] it was the weekend, you know [18:08:31] :P [18:10:05] no, that goes back further [18:10:18] but it looks like going further than dec 3rd/4th, when hoo deployed his patches [18:10:26] so it might not be related after all [18:10:40] PROBLEM - puppet last run on ms1004 is CRITICAL: CRITICAL: Puppet has 1 failures [18:11:08] RECOVERY - puppet last run on ms-fe3002 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [18:12:39] godog: the Package[packages::python_statsd] thing shouldn't be required [18:13:10] hhvm cluster at 10% load [18:13:11] so funny [18:13:54] PROBLEM - puppet last run on gallium is CRITICAL: CRITICAL: puppet fail [18:14:58] ori: it is even wrong :( it was based on I8ce1727d4ebf1 [18:15:02] ori: it was just that we missed a place where we still pulled sites, plu some other optimizations [18:15:30] PROBLEM - puppet last run on ms-be2011 is CRITICAL: CRITICAL: puppet fail [18:15:31] we have another change coming to further optimize and then also to use the file cache [18:17:09] ganglia is quite slow these days [18:17:14] is it no longer on a ramdisk? [18:17:17] i get: There was an error collecting ganglia data (127.0.0.1:8654): fsockopen error: Connection timed out [18:17:18] quite often [18:18:22] I think not anymore [18:18:28] alex was experimenting with rrdcached [18:18:48] and yeah my perception matches yours, it feels slower [18:19:33] is it on SSD or spinning disk? might be worth an expensive SSD for this particular case [18:19:37] SSD [18:20:02] oh, huh. well, i guess ram is still faster. [18:20:03] (03PS2) 10Filippo Giunchedi: python-statsd: use require_package everywhere [puppet] - 10https://gerrit.wikimedia.org/r/178221 [18:20:33] the idea (alex's) was to switch from the ugly tmpfs solution with a combination of SSD + rrdcached [18:20:54] the machine doesn't seem to be saturated in I/O though [18:21:04] so that worked I guess, something else may be the culprit [18:21:12] this is newer gmetad too mind you [18:21:14] (the new box is trusty) [18:21:27] gmetad could be the culrpit, that is consistent with the fsockopen error above [18:21:46] let's wait for alex I guess [18:21:53] yeah [18:22:30] ori: I've updated ^ the original reasoning was https://gerrit.wikimedia.org/r/#/c/176289/ [18:24:43] godog: i don't think the Package['foo'] -> Class['::packages::foo'] change is needed. are you sure about that? [18:25:08] (03PS1) 10Ori.livneh: HHVM: enable stat cache everywhere [puppet] - 10https://gerrit.wikimedia.org/r/178226 [18:25:52] RECOVERY - puppet last run on ms1004 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [18:27:35] (03PS1) 10Yuvipanda: shinken: Add basic monitoring for cvn project [puppet] - 10https://gerrit.wikimedia.org/r/178229 [18:27:47] PROBLEM - puppet last run on ms-be2010 is CRITICAL: CRITICAL: puppet fail [18:27:50] ori: no I'm not, let's try without first, amending [18:27:56] godog: thanks [18:27:57] RECOVERY - puppet last run on ms-be2011 is OK: OK: Puppet is currently enabled, last run 0 seconds ago with 0 failures [18:29:01] (03CR) 10Yuvipanda: [C: 032] shinken: Add basic monitoring for cvn project [puppet] - 10https://gerrit.wikimedia.org/r/178229 (owner: 10Yuvipanda) [18:29:11] YuviPanda: might want to recheck that commit [18:29:22] RECOVERY - puppet last run on gallium is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [18:29:24] PROBLEM - puppet last run on ms-be1006 is CRITICAL: CRITICAL: puppet fail [18:29:35] JohnFLewis: dammit [18:29:45] :p [18:30:08] JohnFLewis-bot :) [18:30:11] (03PS1) 10Yuvipanda: shinken: Fix (stupid) copypasta typo in previous commit [puppet] - 10https://gerrit.wikimedia.org/r/178232 [18:30:19] greg-g: eta for hhvm on jenkins? [18:30:31] PROBLEM - puppet last run on ms-be2006 is CRITICAL: CRITICAL: puppet fail [18:30:52] (03CR) 10John F. Lewis: "Alias as well?" [puppet] - 10https://gerrit.wikimedia.org/r/178232 (owner: 10Yuvipanda) [18:31:03] (03PS1) 10Yuvipanda: shinken: Add guest to integration group [puppet] - 10https://gerrit.wikimedia.org/r/178233 [18:31:13] PROBLEM - puppet last run on ms-be1003 is CRITICAL: CRITICAL: puppet fail [18:31:43] (03PS2) 10Yuvipanda: shinken: Fix (stupid) copypasta typo in previous commit [puppet] - 10https://gerrit.wikimedia.org/r/178232 [18:31:45] (03PS2) 10Yuvipanda: shinken: Add guest to integration group [puppet] - 10https://gerrit.wikimedia.org/r/178233 [18:31:58] JohnFLewis: ^ +1/CR? :) [18:32:10] (03PS3) 10Filippo Giunchedi: python-statsd: use require_package everywhere [puppet] - 10https://gerrit.wikimedia.org/r/178221 [18:32:22] ori: updated ^ [18:33:00] (03CR) 10John F. Lewis: [C: 031] shinken: Fix (stupid) copypasta typo in previous commit [puppet] - 10https://gerrit.wikimedia.org/r/178232 (owner: 10Yuvipanda) [18:33:26] (03CR) 10Yuvipanda: [C: 032] shinken: Fix (stupid) copypasta typo in previous commit [puppet] - 10https://gerrit.wikimedia.org/r/178232 (owner: 10Yuvipanda) [18:34:01] YuviPanda: this is where I should go 'wait, you forgot this as well' ;) [18:34:34] JohnFLewis: hehe :) [18:34:44] (03CR) 10Yuvipanda: [C: 032] shinken: Add guest to integration group [puppet] - 10https://gerrit.wikimedia.org/r/178233 (owner: 10Yuvipanda) [18:36:24] PROBLEM - puppet last run on ms-be2007 is CRITICAL: CRITICAL: puppet fail [18:40:48] godog: looking [18:41:06] (03CR) 10Ori.livneh: [C: 031] python-statsd: use require_package everywhere [puppet] - 10https://gerrit.wikimedia.org/r/178221 (owner: 10Filippo Giunchedi) [18:41:19] PROBLEM - puppet last run on ms-be1002 is CRITICAL: CRITICAL: puppet fail [18:41:23] (03PS1) 10Gage: berkelium & curium: add ipv6, for ipsec testing [dns] - 10https://gerrit.wikimedia.org/r/178237 [18:41:46] ori: \o/ merging, thanks [18:41:51] (03PS4) 10Filippo Giunchedi: python-statsd: use require_package everywhere [puppet] - 10https://gerrit.wikimedia.org/r/178221 [18:41:59] (03CR) 10Filippo Giunchedi: [C: 032 V: 032] python-statsd: use require_package everywhere [puppet] - 10https://gerrit.wikimedia.org/r/178221 (owner: 10Filippo Giunchedi) [18:44:09] PROBLEM - puppet last run on lanthanum is CRITICAL: CRITICAL: puppet fail [18:44:18] RECOVERY - puppet last run on ms-be1006 is OK: OK: Puppet is currently enabled, last run 34 seconds ago with 0 failures [18:44:33] PROBLEM - puppet last run on ms-be1010 is CRITICAL: CRITICAL: puppet fail [18:45:38] RECOVERY - puppet last run on ms-be2006 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [18:46:27] RECOVERY - puppet last run on ms-be1003 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [18:46:35] !log ori Started scap: I5a7e258d2: Optimize how user options are delivered to the client [18:46:41] Logged the message, Master [18:48:37] still find amusing that logmsgbot is "talking" to morebots, /r/botsrights [18:50:32] godog: what did you say is the max size for files ? 5GB ? [18:51:03] matanya: correct, 5 [18:51:10] very bad [18:51:31] the smallest one is 4.7 [18:51:35] godog: I find that offensive. [18:51:38] RECOVERY - puppet last run on ms-be2007 is OK: OK: Puppet is currently enabled, last run 23 seconds ago with 0 failures [18:51:48] PROBLEM - puppet last run on ms-fe2001 is CRITICAL: CRITICAL: puppet fail [18:52:00] the rest are 11-26 GB [18:52:12] i'll need to scale them down [18:52:47] logmsgbot: <3 what do you find offensive? [18:53:25] matanya: mmhh do you know how we did for previous wikimanias? surely there were files larger than 5gb [18:53:51] most of prev wikimania aren't on commons [18:54:00] and also quite low quality [18:54:11] i guess Reedy might know [18:54:14] (03PS1) 10MaxSem: Revert "Disable gadgets caching" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/178240 [18:54:21] godog: youtube or send to ashburn [18:54:52] aude: in terms of file size upload, not the transfer of the files [18:54:59] or internet archive [18:55:06] I already have them [18:55:08] ah [18:55:14] split it into chunks [18:55:14] the problem is they are huge [18:55:18] aude: did they get uploaded to commons after do you remember? [18:55:42] godog: i think some were eventually [18:55:48] RECOVERY - puppet last run on ms-be1002 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [18:56:16] godog: https://commons.wikimedia.org/wiki/Category:Wikimania_2013_videos [18:56:32] https://commons.wikimedia.org/wiki/Category:Wikimania_videos [18:57:27] very low quality relative to what i have now [18:58:02] highest there is around 3.5 Mbps, i start at 16.7 Mbps and go up to 35 Mbps [18:58:10] https://commons.wikimedia.org/wiki/Category:Wikimania_2009_presentation_videos [18:59:00] RECOVERY - puppet last run on lanthanum is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [18:59:10] aude: note that 99% are below 100 MB [18:59:23] yeah [18:59:29] RECOVERY - puppet last run on ms-be1010 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [18:59:36] like seeing James_F here https://commons.wikimedia.org/wiki/File:Wikimania_2013_-_VisualEditor_-_The_present_and_future_of_editing_our_wikis.webm is not fun like seeing him in 4k [19:00:02] 5gb, it's like downloading wikidata dump twice [19:00:08] almost [19:00:10] or even in reality, which is a different story :) [19:00:40] RECOVERY - puppet last run on ms-be2010 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [19:02:32] (03PS2) 10Ori.livneh: HHVM: enable stat cache everywhere [puppet] - 10https://gerrit.wikimedia.org/r/178226 [19:02:41] matanya: yeah it is a bit overkill IMO to have full resolution, how big would it be 420p to commons? [19:03:28] godog: about half the size, maybe less [19:03:45] it is a combination of very long videos in very high quality [19:06:21] MaxSem: did disabling the gadgets caching make it worse? [19:06:31] yup [19:06:38] :/ [19:06:57] and no visible improvement of memcached servers load [19:09:32] MaxSem: are you deploying right now? [19:09:49] nope, just nominated it for SWAT [19:10:20] * greg-g nods [19:12:18] PROBLEM - puppet last run on ms-fe1001 is CRITICAL: CRITICAL: puppet fail [19:12:42] (03CR) 10Faidon Liambotis: [C: 031] "Sounds good in principle and from a high-level overview of what this does, although I'm clearly not fully informed about the full impact." [puppet] - 10https://gerrit.wikimedia.org/r/178226 (owner: 10Ori.livneh) [19:13:19] paravoid: thanks. i'll update the commit message to be more informative before i merge it; i just wanted it up so i can cherry-pick it on labs. [19:13:20] !log ori Finished scap: I5a7e258d2: Optimize how user options are delivered to the client (duration: 26m 45s) [19:13:22] Logged the message, Master [19:19:13] !log deployed patches for T77028 and T76686 [19:19:19] Logged the message, Master [19:21:14] (03PS1) 10Ori.livneh: Add StashEdit log group on fluorine [mediawiki-config] - 10https://gerrit.wikimedia.org/r/178249 [19:21:27] (03CR) 10Ori.livneh: [C: 032] Add StashEdit log group on fluorine [mediawiki-config] - 10https://gerrit.wikimedia.org/r/178249 (owner: 10Ori.livneh) [19:22:26] (03CR) 10Ori.livneh: [V: 032] Add StashEdit log group on fluorine [mediawiki-config] - 10https://gerrit.wikimedia.org/r/178249 (owner: 10Ori.livneh) [19:23:27] matanya: swift won't take files >= 5GB, right godog? [19:24:13] Reedy: default maximum size it is that yeah, you can do chunks though http://docs.openstack.org/developer/swift/overview_large_objects.html [19:24:27] Does MW support that? [19:24:33] /the swift library we use [19:27:16] RECOVERY - puppet last run on ms-fe1001 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [19:27:35] Probably worth opening a task in phab investigating/adding support [19:28:05] matanya: 4K? I thought that video was only captured in 1K. [19:28:56] Hey, would anyone object to enabling CORS on betalabs? I'm trying to start work on a gadget that uploads files to betacommons from betaenwiki [19:29:03] (cross-posting from -qa) [19:29:03] bblack, do you happen to know if the dallas IPs are serving traffic yet? [19:29:20] not for the public, no [19:29:30] Reedy: in team meeting, bbl [19:30:28] bblack: okay, thx. do you know when they're going in? [19:30:52] I'm sure it will be quite a while yet. we don't yet have appserver or cache machines there [19:31:06] (03PS3) 10Ori.livneh: HHVM: enable stat cache everywhere [puppet] - 10https://gerrit.wikimedia.org/r/178226 [19:31:31] don't ask me what quite a while is. A couple of months, all things considered, before we might have the potential to offer full service, and then there's still the question of when/if we actually fail over to it? [19:31:50] (03CR) 10Ori.livneh: [C: 032 V: 032] HHVM: enable stat cache everywhere [puppet] - 10https://gerrit.wikimedia.org/r/178226 (owner: 10Ori.livneh) [19:32:03] Actually, weich [19:32:06] weird even [19:32:35] It looks like wmgUseCORS is enabled for public wikis and labs has *.beta.wmflabs.org marked as OK [19:32:43] But I got an error when I tried to do an upload [19:34:36] James_F: they were, just kidding [19:34:45] matanya: Aha. :-) [19:35:19] Reedy: did you scale down in prev years ? [19:38:36] https://www.mediawiki.org/wiki/Talk:Flow looks fine [19:38:40] wrong channel [19:40:56] !log krinkle Synchronized php-1.25wmf11/resources/src/startup.js: touch (duration: 00m 06s) [19:40:58] Logged the message, Master [19:42:12] PROBLEM - puppet last run on mw1241 is CRITICAL: CRITICAL: Puppet has 1 failures [19:47:59] RECOVERY - puppet last run on mw1241 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [19:50:40] PROBLEM - puppet last run on ms-fe1001 is CRITICAL: CRITICAL: puppet fail [19:52:29] PROBLEM - puppet last run on mw1189 is CRITICAL: CRITICAL: Puppet has 1 failures [19:53:04] matanya: Nope [19:53:15] They were provided ready to upload I think [19:53:27] hmm, ok [19:53:35] i'll split and scale down [20:07:22] RECOVERY - puppet last run on mw1189 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [20:17:30] (03PS1) 10Cmjohnson: Adding new MACs for virt1010 and virt1012 [puppet] - 10https://gerrit.wikimedia.org/r/178259 [20:18:25] (03CR) 10Cmjohnson: [C: 032] Adding new MACs for virt1010 and virt1012 [puppet] - 10https://gerrit.wikimedia.org/r/178259 (owner: 10Cmjohnson) [20:22:48] (03PS1) 10RobH: setting graphite2001 mgmt entries [dns] - 10https://gerrit.wikimedia.org/r/178262 [20:23:57] (03CR) 10RobH: [C: 032] setting graphite2001 mgmt entries [dns] - 10https://gerrit.wikimedia.org/r/178262 (owner: 10RobH) [20:46:22] I'm testing stuff on terbium, should load on external storage or terbium itself grow to much, feel free to kill that. Unlikely, though. [21:00:04] gwicke, cscott, arlolra, subbu: Dear anthropoid, the time has come. Please deploy Parsoid/OCG (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20141208T2100). [21:07:21] PROBLEM - Host labstore2001 is DOWN: PING CRITICAL - Packet loss = 100% [21:07:32] bblack: mutante: Can you confirm my conclusion on https://phabricator.wikimedia.org/T38126#831441 and close the ticket in that case? [21:17:10] manybubbles: what's the plan regarding old search (lsearch)? [21:17:23] are we keeping it for the moment? are we decommissioning the service & the servers? [21:17:55] paravoid: at this point, leave it on until after the new year and then validate that no one is using and it then drop it [21:20:36] manybubbles: private wikis moved to cirrus too, right ? [21:20:55] matanya: yeah - cirrus had no problem with private wikis [21:21:17] are you planning to remove "this wiki is using a new search" message ? [21:22:58] manybubbles: ^ [21:23:11] matanya: you know, we should probably do that :) [21:23:23] filing [21:23:32] cc me please [21:24:19] cced [21:24:39] thanks [21:29:03] RECOVERY - Host labstore2001 is UP: PING OK - Packet loss = 0%, RTA = 43.75 ms [21:33:46] Krinkle: I need a little to digest it and I'm in a meeting, but I'll get back to you [21:37:49] (03PS2) 10Gage: berkelium & curium: add ipv6, for ipsec testing [dns] - 10https://gerrit.wikimedia.org/r/178237 [21:38:37] (03CR) 10Gage: [C: 032] berkelium & curium: add ipv6, for ipsec testing [dns] - 10https://gerrit.wikimedia.org/r/178237 (owner: 10Gage) [21:38:57] Does anyone know if Wikipedia was sending P3P headers till recently? [21:39:40] (03CR) 10Brian Wolff: "Note in cover message: comma count is less a performance issue and instead probably wont work for that script at all." [puppet] - 10https://gerrit.wikimedia.org/r/178170 (owner: 10Nemo bis) [21:40:08] If we recently stopped sending P3P header, that might be causing donors using IE to keep getting fundraising banners [21:40:54] ejegg, we recently stopped serving JS to IE users [21:42:23] IE < 8 [21:42:28] Hmm, without any JS they'd see no banners at all. [21:42:39] And we're hearing this from IE11 users [21:42:43] yup [21:44:34] IE's default cookie policy has been to deny 3rd party cookies when no P3P header is present. The thank-you page on foundationwiki sources a banner hiding page from multiple domains (as an img tag) to get the cookies set, so they're treated as 3rd party by IE [21:45:04] (03PS1) 10EBernhardson: Enable flow on catalan wiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/178278 [21:47:47] (03CR) 10Jalexander: "Leaving a comment here that for now I want to revert this. The script running caused breakage with the recent arbcom election and while we" [puppet] - 10https://gerrit.wikimedia.org/r/74592 (https://bugzilla.wikimedia.org/51574) (owner: 10Reedy) [21:50:40] (03PS1) 10Ori.livneh: Revert of Iab860b8a5: Make puppet cronjob to run SecurePoll/cli/purgePrivateVoteData.php [puppet] - 10https://gerrit.wikimedia.org/r/178280 [21:51:36] (03CR) 10Jalexander: [C: 031] Revert of Iab860b8a5: Make puppet cronjob to run SecurePoll/cli/purgePrivateVoteData.php [puppet] - 10https://gerrit.wikimedia.org/r/178280 (owner: 10Ori.livneh) [21:51:54] (03CR) 10Ori.livneh: [C: 032] Revert of Iab860b8a5: Make puppet cronjob to run SecurePoll/cli/purgePrivateVoteData.php [puppet] - 10https://gerrit.wikimedia.org/r/178280 (owner: 10Ori.livneh) [21:59:54] ottomata: did you say that analytics doesn't use Ciscos anymore? [22:07:04] hah [22:07:08] we do! [22:07:10] but we don't want to! [22:07:22] paravoid: https://rt.wikimedia.org/Ticket/Display.html?id=8984 [22:07:34] ok [22:07:42] can you do me a quick favor? [22:08:02] can you do a quick check of install-server's partman/ and see if all these analytics profiles are used/needed? [22:08:20] checking... [22:10:35] paravoid, yes. they are all needed, until we actually shed the ciscos [22:10:44] ok [22:10:47] then we don't need analytics-cisco anymore [22:10:51] I'm trying to cleanup the partman mess a bit (again) [22:11:15] analytics101[1-9]|analytics102[0-2]) echo partman/raid1-30G.cfg ;; \ [22:11:18] what are these? [22:11:28] this should probably renamed to analytics-something.cfg [22:11:43] those are the same as analytics-flex, except without the flex bays. [22:11:58] which is for what purpose? [22:12:02] analytics-flex are the new dell nodes we got this summer [22:12:08] datanodes? [22:12:26] analytics1011-analytics1022 are datanodes and kafka brokers. [22:12:43] they are not set up the same way, but it was too complicated to set them up the way they neede to be in partman [22:13:07] so there are post install steps needed, depending on node purpose [22:13:27] e.g. [22:13:27] https://wikitech.wikimedia.org/wiki/Analytics/Cluster/Hadoop/Administration#New_Worker_Installation [22:13:48] and [22:13:48] https://wikitech.wikimedia.org/wiki/Analytics/Cluster/Kafka/Administration#Partitioning [22:14:50] ok [22:14:51] thanks [22:14:55] yup :) [22:15:40] 3 lucid, 437 precise, 366 trusty [22:15:43] interesting. [22:15:49] yes [22:15:54] ? [22:15:55] oh [22:15:57] wait, all servers? [22:16:00] interesting! [22:16:03] did you get that via salt? [22:17:08] no, servermon [22:19:10] of those precises, 77 mw, 73 cp, 57 db, 24 search, 19 ms-be, 16 mc, 14 lvs, 10 virt, 10 analytics [22:19:37] not too bad [22:21:07] 10 analytics? I can think of 7 that are still precise (and actually not broken, cough, ciscos) [22:21:28] ah [22:21:30] no 8, [22:21:35] haha, hmm, what are the other 2 [22:24:22] ahhh i haven't upgraded journalnodes yet [22:24:24] right right right [22:25:07] (03CR) 10QChris: "> Would string protection work here?" [puppet] - 10https://gerrit.wikimedia.org/r/177128 (owner: 10Krinkle) [22:29:35] Could someone please run: "sudo -u datasets /usr/local/bin/dumpwikidatajson.sh" on snapshoot1003 [22:29:47] ori is on it [22:33:29] (03PS1) 10Legoktm: multiversion: cdb/cdb was renamed to wikimedia/cdb [mediawiki-config] - 10https://gerrit.wikimedia.org/r/178340 [22:36:58] (03CR) 10Krinkle: [C: 031] multiversion: cdb/cdb was renamed to wikimedia/cdb [mediawiki-config] - 10https://gerrit.wikimedia.org/r/178340 (owner: 10Legoktm) [22:39:33] (03PS1) 10Gage: node definitions: cp3001, cp3002, berkelium, curium [puppet] - 10https://gerrit.wikimedia.org/r/178345 [22:41:05] (03PS1) 10Faidon Liambotis: partman: set confirm_nooverwrite everywhere [puppet] - 10https://gerrit.wikimedia.org/r/178346 [22:41:07] (03PS1) 10Faidon Liambotis: partman: partial profile cleanups [puppet] - 10https://gerrit.wikimedia.org/r/178347 [22:41:09] (03PS1) 10Faidon Liambotis: Add temporary host d-i-test [puppet] - 10https://gerrit.wikimedia.org/r/178348 [22:41:39] (03CR) 10Faidon Liambotis: [C: 032 V: 032] partman: set confirm_nooverwrite everywhere [puppet] - 10https://gerrit.wikimedia.org/r/178346 (owner: 10Faidon Liambotis) [22:42:02] (03CR) 10Faidon Liambotis: [C: 032 V: 032] partman: partial profile cleanups [puppet] - 10https://gerrit.wikimedia.org/r/178347 (owner: 10Faidon Liambotis) [22:42:20] (03CR) 10Faidon Liambotis: [C: 032 V: 032] Add temporary host d-i-test [puppet] - 10https://gerrit.wikimedia.org/r/178348 (owner: 10Faidon Liambotis) [22:42:29] paravoid: 00:42, you need sleep sometime, we need to create ori's icinga alert for you too ? [22:44:36] (03PS1) 10Ori.livneh: Set $wgAjaxEditStash to false while API cluster is on Zend [mediawiki-config] - 10https://gerrit.wikimedia.org/r/178350 [22:46:06] RECOVERY - puppet last run on ms-fe1001 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [22:49:02] (03CR) 10Gage: [C: 032] node definitions: cp3001, cp3002, berkelium, curium [puppet] - 10https://gerrit.wikimedia.org/r/178345 (owner: 10Gage) [22:50:12] dpkg-shlibs is cool [22:50:17] well, shlibdeps [22:50:36] (03PS1) 10Faidon Liambotis: Add libmaxminddb to m::statistics::packages::utilities [puppet] - 10https://gerrit.wikimedia.org/r/178353 [22:51:04] (03CR) 10Faidon Liambotis: [C: 032] Add libmaxminddb to m::statistics::packages::utilities [puppet] - 10https://gerrit.wikimedia.org/r/178353 (owner: 10Faidon Liambotis) [22:51:17] ottomata: misc::statistics could really use a cleanup... :)) [22:52:42] oh i know paravoid, i know. [22:52:58] :) [23:02:23] (03PS1) 10Ottomata: Send limn-public-data cron stderr to log file [puppet] - 10https://gerrit.wikimedia.org/r/178358 [23:04:31] (03CR) 10Ottomata: [C: 032] Send limn-public-data cron stderr to log file [puppet] - 10https://gerrit.wikimedia.org/r/178358 (owner: 10Ottomata) [23:09:46] PROBLEM - puppet last run on ms-fe1001 is CRITICAL: CRITICAL: puppet fail [23:12:27] (03PS1) 10Plucas: Use generic snappy version and fix CLASSPATH [debs/kafka] - 10https://gerrit.wikimedia.org/r/178361 [23:16:48] (03PS1) 10Yuvipanda: shinken: Fix terrible, terrible typo [puppet] - 10https://gerrit.wikimedia.org/r/178364 [23:17:53] (03CR) 10Yuvipanda: [C: 032] shinken: Fix terrible, terrible typo [puppet] - 10https://gerrit.wikimedia.org/r/178364 (owner: 10Yuvipanda) [23:18:23] WMF Error: [23:18:24] Request: GET http://en.wikipedia.org/wiki/MediaWiki:Gadget-popups.js, from 10.64.0.105 via cp1053 cp1053 ([10.64.32.105]:3128), Varnish XID 2405699790 [23:18:24] Forwarded for: 191.191.52.172, 10.64.0.105, 10.64.0.105 [23:18:24] Error: 503, Service Unavailable at Mon, 08 Dec 2014 23:17:54 GMT [23:20:24] legoktm: ^ [23:20:40] someone reported that in -tech too [23:20:53] weird... [23:21:29] * legoktm will look in a few [23:22:03] blergh [23:22:27] FWIW the talk page works fine [23:49:03] (03PS1) 10Faidon Liambotis: autoinstall: comment out/empty partman_early_command [puppet] - 10https://gerrit.wikimedia.org/r/178368 [23:49:05] (03PS1) 10Faidon Liambotis: autoinstall: setup network-console within d-i [puppet] - 10https://gerrit.wikimedia.org/r/178369 [23:49:26] (03CR) 10Faidon Liambotis: [C: 032] autoinstall: comment out/empty partman_early_command [puppet] - 10https://gerrit.wikimedia.org/r/178368 (owner: 10Faidon Liambotis) [23:49:52] (03PS1) 10Gage: berkelium & curium: add ipv6, for ipsec testing [dns] - 10https://gerrit.wikimedia.org/r/178370 [23:50:10] (03CR) 10Faidon Liambotis: [C: 032] autoinstall: setup network-console within d-i [puppet] - 10https://gerrit.wikimedia.org/r/178369 (owner: 10Faidon Liambotis) [23:50:58] (03CR) 10Gage: [C: 032] berkelium & curium: add ipv6, for ipsec testing [dns] - 10https://gerrit.wikimedia.org/r/178370 (owner: 10Gage) [23:58:01] Error: Could not retrieve catalog from remote server: Error 400 on SERVER: Cannot reassign variable gid at /etc/puppet/manifests/realm.pp:68 on node d-i-test.eqiad.wmnet [23:58:05] huh