[00:11:11] PROBLEM - puppet last run on curium is CRITICAL: CRITICAL: Puppet has 1 failures [00:11:30] PROBLEM - puppet last run on mw1048 is CRITICAL: CRITICAL: Puppet has 1 failures [00:11:40] PROBLEM - puppet last run on ms-be1001 is CRITICAL: CRITICAL: Puppet has 1 failures [00:12:17] PROBLEM - puppet last run on analytics1033 is CRITICAL: CRITICAL: Puppet has 1 failures [00:12:22] PROBLEM - puppet last run on cp3006 is CRITICAL: CRITICAL: Puppet has 1 failures [00:12:23] PROBLEM - puppet last run on ms-be1004 is CRITICAL: CRITICAL: Puppet has 1 failures [00:12:43] PROBLEM - puppet last run on netmon1001 is CRITICAL: CRITICAL: Puppet has 1 failures [00:12:54] PROBLEM - puppet last run on cerium is CRITICAL: CRITICAL: Puppet has 1 failures [00:13:15] PROBLEM - puppet last run on neptunium is CRITICAL: CRITICAL: Puppet has 1 failures [00:13:24] PROBLEM - puppet last run on mc1016 is CRITICAL: CRITICAL: Puppet has 1 failures [00:13:24] PROBLEM - puppet last run on es1001 is CRITICAL: CRITICAL: Puppet has 1 failures [00:13:24] PROBLEM - puppet last run on mw1063 is CRITICAL: CRITICAL: Puppet has 1 failures [00:13:25] PROBLEM - puppet last run on cp1049 is CRITICAL: CRITICAL: Puppet has 1 failures [00:13:33] PROBLEM - puppet last run on caesium is CRITICAL: CRITICAL: Puppet has 1 failures [00:13:37] PROBLEM - puppet last run on searchidx1001 is CRITICAL: CRITICAL: Puppet has 1 failures [00:14:04] PROBLEM - puppet last run on magnesium is CRITICAL: CRITICAL: Puppet has 1 failures [00:14:04] PROBLEM - puppet last run on dbproxy1002 is CRITICAL: CRITICAL: Puppet has 1 failures [00:14:15] PROBLEM - puppet last run on mw1197 is CRITICAL: CRITICAL: Puppet has 1 failures [00:14:15] PROBLEM - puppet last run on db1053 is CRITICAL: CRITICAL: Puppet has 1 failures [00:14:16] PROBLEM - puppet last run on logstash1001 is CRITICAL: CRITICAL: Puppet has 1 failures [00:14:24] PROBLEM - puppet last run on ms-be2003 is CRITICAL: CRITICAL: Puppet has 1 failures [00:14:36] PROBLEM - puppet last run on mw1059 is CRITICAL: CRITICAL: Puppet has 1 failures [00:14:44] PROBLEM - puppet last run on mw1041 is CRITICAL: CRITICAL: Puppet has 1 failures [00:14:53] PROBLEM - puppet last run on cp3008 is CRITICAL: CRITICAL: Puppet has 1 failures [00:14:53] PROBLEM - puppet last run on mw1045 is CRITICAL: CRITICAL: Puppet has 1 failures [00:15:03] PROBLEM - puppet last run on lvs4002 is CRITICAL: CRITICAL: Puppet has 1 failures [00:15:10] PROBLEM - puppet last run on mw1145 is CRITICAL: CRITICAL: Puppet has 1 failures [00:15:13] PROBLEM - puppet last run on mw1012 is CRITICAL: CRITICAL: Puppet has 1 failures [00:15:17] PROBLEM - puppet last run on mw1140 is CRITICAL: CRITICAL: Puppet has 1 failures [00:15:28] PROBLEM - puppet last run on mw1106 is CRITICAL: CRITICAL: Puppet has 1 failures [00:15:41] PROBLEM - puppet last run on cp3020 is CRITICAL: CRITICAL: Puppet has 1 failures [00:16:21] PROBLEM - puppet last run on mw1141 is CRITICAL: CRITICAL: Puppet has 1 failures [00:16:21] PROBLEM - puppet last run on mw1006 is CRITICAL: CRITICAL: Puppet has 1 failures [00:19:43] PROBLEM - puppet last run on virt1001 is CRITICAL: CRITICAL: Puppet has 1 failures [00:22:31] RECOVERY - puppet last run on mw1048 is OK: OK: Puppet is currently enabled, last run 35 seconds ago with 0 failures [00:22:45] RECOVERY - puppet last run on ms-be1001 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [00:23:03] RECOVERY - puppet last run on ms-be2003 is OK: OK: Puppet is currently enabled, last run 17 seconds ago with 0 failures [00:23:39] RECOVERY - puppet last run on analytics1033 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [00:23:39] RECOVERY - puppet last run on cp3006 is OK: OK: Puppet is currently enabled, last run 30 seconds ago with 0 failures [00:23:40] RECOVERY - puppet last run on lvs4002 is OK: OK: Puppet is currently enabled, last run 48 seconds ago with 0 failures [00:23:40] RECOVERY - puppet last run on mw1012 is OK: OK: Puppet is currently enabled, last run 54 seconds ago with 0 failures [00:23:40] RECOVERY - puppet last run on mw1140 is OK: OK: Puppet is currently enabled, last run 1 second ago with 0 failures [00:24:00] RECOVERY - puppet last run on netmon1001 is OK: OK: Puppet is currently enabled, last run 8 seconds ago with 0 failures [00:24:02] RECOVERY - puppet last run on mw1106 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [00:24:02] RECOVERY - puppet last run on cerium is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [00:24:41] RECOVERY - puppet last run on neptunium is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [00:24:46] RECOVERY - puppet last run on mc1016 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [00:24:48] RECOVERY - puppet last run on es1001 is OK: OK: Puppet is currently enabled, last run 59 seconds ago with 0 failures [00:24:48] RECOVERY - puppet last run on mw1063 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [00:24:48] RECOVERY - puppet last run on cp1049 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [00:24:48] RECOVERY - puppet last run on caesium is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [00:24:48] RECOVERY - puppet last run on searchidx1001 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [00:24:49] RECOVERY - puppet last run on mw1141 is OK: OK: Puppet is currently enabled, last run 57 seconds ago with 0 failures [00:24:49] RECOVERY - puppet last run on mw1006 is OK: OK: Puppet is currently enabled, last run 53 seconds ago with 0 failures [00:25:01] RECOVERY - puppet last run on curium is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [00:25:32] RECOVERY - puppet last run on magnesium is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [00:25:32] RECOVERY - puppet last run on dbproxy1002 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [00:25:32] RECOVERY - puppet last run on mw1197 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [00:25:32] RECOVERY - puppet last run on db1053 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [00:25:45] RECOVERY - puppet last run on logstash1001 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [00:26:00] RECOVERY - puppet last run on mw1059 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [00:26:11] RECOVERY - puppet last run on mw1041 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [00:26:12] RECOVERY - puppet last run on cp3008 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [00:26:20] RECOVERY - puppet last run on mw1045 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [00:26:21] RECOVERY - puppet last run on mw1145 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [00:26:21] RECOVERY - puppet last run on ms-be1004 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [00:26:52] RECOVERY - puppet last run on cp3020 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [00:30:41] RECOVERY - puppet last run on virt1001 is OK: OK: Puppet is currently enabled, last run 31 seconds ago with 0 failures [01:03:06] PROBLEM - HHVM busy threads on mw1230 is CRITICAL: CRITICAL: 11.11% of data above the critical threshold [115.2] [01:04:17] PROBLEM - HHVM busy threads on mw1223 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [115.2] [01:04:20] PROBLEM - HHVM queue size on mw1235 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [80.0] [01:04:21] PROBLEM - HHVM busy threads on mw1222 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [115.2] [01:04:42] PROBLEM - HHVM busy threads on mw1234 is CRITICAL: CRITICAL: 11.11% of data above the critical threshold [115.2] [01:04:45] PROBLEM - HHVM busy threads on mw1233 is CRITICAL: CRITICAL: 11.11% of data above the critical threshold [115.2] [01:04:45] PROBLEM - HHVM busy threads on mw1235 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [115.2] [01:05:43] PROBLEM - HHVM busy threads on mw1229 is CRITICAL: CRITICAL: 11.11% of data above the critical threshold [115.2] [01:06:05] PROBLEM - HHVM busy threads on mw1231 is CRITICAL: CRITICAL: 10.00% of data above the critical threshold [115.2] [01:06:54] PROBLEM - HHVM queue size on mw1222 is CRITICAL: CRITICAL: 11.11% of data above the critical threshold [80.0] [01:07:43] RECOVERY - HHVM busy threads on mw1235 is OK: OK: Less than 1.00% above the threshold [76.8] [01:07:44] PROBLEM - HHVM busy threads on mw1228 is CRITICAL: CRITICAL: 11.11% of data above the critical threshold [115.2] [01:08:55] RECOVERY - HHVM busy threads on mw1231 is OK: OK: Less than 1.00% above the threshold [76.8] [01:09:27] RECOVERY - HHVM queue size on mw1222 is OK: OK: Less than 1.00% above the threshold [10.0] [01:09:35] RECOVERY - HHVM busy threads on mw1223 is OK: OK: Less than 1.00% above the threshold [76.8] [01:09:35] RECOVERY - HHVM queue size on mw1235 is OK: OK: Less than 1.00% above the threshold [10.0] [01:09:45] RECOVERY - HHVM busy threads on mw1222 is OK: OK: Less than 1.00% above the threshold [76.8] [01:10:08] RECOVERY - HHVM busy threads on mw1234 is OK: OK: Less than 1.00% above the threshold [76.8] [01:10:09] RECOVERY - HHVM busy threads on mw1233 is OK: OK: Less than 1.00% above the threshold [76.8] [01:10:16] RECOVERY - HHVM busy threads on mw1228 is OK: OK: Less than 1.00% above the threshold [76.8] [01:11:06] RECOVERY - HHVM busy threads on mw1229 is OK: OK: Less than 1.00% above the threshold [76.8] [01:11:16] RECOVERY - HHVM busy threads on mw1230 is OK: OK: Less than 1.00% above the threshold [76.8] [01:23:16] PROBLEM - puppet last run on strontium is CRITICAL: CRITICAL: Puppet has 1 failures [01:36:44] RECOVERY - puppet last run on strontium is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [01:57:09] PROBLEM - puppet last run on virt1003 is CRITICAL: CRITICAL: Puppet has 1 failures [01:57:32] PROBLEM - puppet last run on mc1005 is CRITICAL: CRITICAL: Puppet has 1 failures [01:57:32] PROBLEM - puppet last run on amssq56 is CRITICAL: CRITICAL: Puppet has 1 failures [01:57:41] PROBLEM - puppet last run on analytics1022 is CRITICAL: CRITICAL: Puppet has 1 failures [01:57:49] PROBLEM - puppet last run on virt1001 is CRITICAL: CRITICAL: Puppet has 1 failures [01:57:49] PROBLEM - puppet last run on mw1151 is CRITICAL: CRITICAL: Puppet has 1 failures [01:57:49] PROBLEM - puppet last run on db1036 is CRITICAL: CRITICAL: Puppet has 1 failures [01:57:52] PROBLEM - puppet last run on mw1079 is CRITICAL: CRITICAL: Puppet has 1 failures [01:58:01] PROBLEM - puppet last run on lvs2006 is CRITICAL: CRITICAL: Puppet has 1 failures [01:58:13] PROBLEM - puppet last run on mw1098 is CRITICAL: CRITICAL: Puppet has 1 failures [01:58:13] PROBLEM - puppet last run on rdb1001 is CRITICAL: CRITICAL: Puppet has 1 failures [01:58:14] PROBLEM - puppet last run on bast4001 is CRITICAL: CRITICAL: Puppet has 1 failures [01:58:22] PROBLEM - puppet last run on mw1084 is CRITICAL: CRITICAL: Puppet has 1 failures [01:58:41] PROBLEM - puppet last run on mc1014 is CRITICAL: CRITICAL: Puppet has 1 failures [01:58:41] PROBLEM - puppet last run on gadolinium is CRITICAL: CRITICAL: Puppet has 1 failures [01:58:42] PROBLEM - puppet last run on cp1063 is CRITICAL: CRITICAL: Puppet has 1 failures [01:58:42] PROBLEM - puppet last run on db2016 is CRITICAL: CRITICAL: Puppet has 1 failures [01:58:42] PROBLEM - puppet last run on cp4001 is CRITICAL: CRITICAL: Puppet has 1 failures [01:58:51] PROBLEM - puppet last run on mw1133 is CRITICAL: CRITICAL: Puppet has 1 failures [01:58:52] PROBLEM - puppet last run on mw1227 is CRITICAL: CRITICAL: Puppet has 1 failures [01:58:52] PROBLEM - puppet last run on amssq34 is CRITICAL: CRITICAL: Puppet has 1 failures [01:58:52] PROBLEM - puppet last run on mw1014 is CRITICAL: CRITICAL: Puppet has 1 failures [01:59:02] PROBLEM - puppet last run on mw1111 is CRITICAL: CRITICAL: Puppet has 1 failures [01:59:03] PROBLEM - puppet last run on mw1190 is CRITICAL: CRITICAL: Puppet has 1 failures [01:59:13] PROBLEM - puppet last run on mw1081 is CRITICAL: CRITICAL: Puppet has 1 failures [01:59:14] PROBLEM - puppet last run on ssl1009 is CRITICAL: CRITICAL: Puppet has 1 failures [01:59:14] PROBLEM - puppet last run on elastic1015 is CRITICAL: CRITICAL: Puppet has 1 failures [01:59:23] PROBLEM - puppet last run on ms-be2012 is CRITICAL: CRITICAL: Puppet has 1 failures [01:59:33] PROBLEM - puppet last run on mw1238 is CRITICAL: CRITICAL: Puppet has 1 failures [01:59:33] PROBLEM - puppet last run on search1017 is CRITICAL: CRITICAL: Puppet has 1 failures [01:59:33] PROBLEM - puppet last run on cp4018 is CRITICAL: CRITICAL: Puppet has 1 failures [01:59:33] PROBLEM - puppet last run on mw1004 is CRITICAL: CRITICAL: Puppet has 1 failures [01:59:34] PROBLEM - puppet last run on lvs3004 is CRITICAL: CRITICAL: Puppet has 1 failures [01:59:34] PROBLEM - puppet last run on tungsten is CRITICAL: CRITICAL: Puppet has 1 failures [01:59:34] PROBLEM - puppet last run on amssq41 is CRITICAL: CRITICAL: Puppet has 1 failures [01:59:35] PROBLEM - puppet last run on ms-fe3002 is CRITICAL: CRITICAL: Puppet has 1 failures [01:59:35] PROBLEM - puppet last run on amssq36 is CRITICAL: CRITICAL: Puppet has 1 failures [01:59:41] PROBLEM - puppet last run on mw1034 is CRITICAL: CRITICAL: Puppet has 1 failures [01:59:43] PROBLEM - puppet last run on mw1146 is CRITICAL: CRITICAL: Puppet has 1 failures [01:59:44] PROBLEM - puppet last run on mw1183 is CRITICAL: CRITICAL: Puppet has 1 failures [01:59:52] PROBLEM - puppet last run on mw1159 is CRITICAL: CRITICAL: Puppet has 1 failures [02:00:01] PROBLEM - puppet last run on snapshot1002 is CRITICAL: CRITICAL: Puppet has 1 failures [02:00:01] PROBLEM - puppet last run on mw1181 is CRITICAL: CRITICAL: Puppet has 1 failures [02:00:02] PROBLEM - puppet last run on snapshot1004 is CRITICAL: CRITICAL: Puppet has 1 failures [02:00:02] PROBLEM - puppet last run on mw1208 is CRITICAL: CRITICAL: Puppet has 1 failures [02:00:11] PROBLEM - puppet last run on plutonium is CRITICAL: CRITICAL: Puppet has 1 failures [02:00:12] PROBLEM - puppet last run on mw1180 is CRITICAL: CRITICAL: Puppet has 1 failures [02:00:12] PROBLEM - puppet last run on wtp1004 is CRITICAL: CRITICAL: Puppet has 1 failures [02:00:12] PROBLEM - puppet last run on labsdb1006 is CRITICAL: CRITICAL: Puppet has 1 failures [02:00:12] PROBLEM - puppet last run on virt1007 is CRITICAL: CRITICAL: Puppet has 1 failures [02:00:12] PROBLEM - puppet last run on mw1051 is CRITICAL: CRITICAL: Puppet has 1 failures [02:00:31] PROBLEM - puppet last run on db2001 is CRITICAL: CRITICAL: Puppet has 1 failures [02:00:33] PROBLEM - puppet last run on ms-be2008 is CRITICAL: CRITICAL: Puppet has 1 failures [02:00:53] PROBLEM - puppet last run on argon is CRITICAL: CRITICAL: Puppet has 1 failures [02:01:03] PROBLEM - puppet last run on cp1062 is CRITICAL: CRITICAL: Puppet has 1 failures [02:01:04] PROBLEM - puppet last run on db2037 is CRITICAL: CRITICAL: Puppet has 1 failures [02:01:12] PROBLEM - puppet last run on db2023 is CRITICAL: CRITICAL: Puppet has 1 failures [02:01:23] PROBLEM - puppet last run on mw1258 is CRITICAL: CRITICAL: Puppet has 1 failures [02:01:36] PROBLEM - puppet last run on ssl3002 is CRITICAL: CRITICAL: Puppet has 1 failures [02:08:47] RECOVERY - puppet last run on mw1208 is OK: OK: Puppet is currently enabled, last run 55 seconds ago with 0 failures [02:09:26] RECOVERY - puppet last run on db1036 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [02:10:03] !log l10nupdate Synchronized php-1.25wmf9/cache/l10n: (no message) (duration: 00m 01s) [02:10:06] RECOVERY - puppet last run on db2016 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [02:10:06] RECOVERY - puppet last run on cp4001 is OK: OK: Puppet is currently enabled, last run 53 seconds ago with 0 failures [02:10:06] RECOVERY - puppet last run on mw1227 is OK: OK: Puppet is currently enabled, last run 3 seconds ago with 0 failures [02:10:06] !log LocalisationUpdate completed (1.25wmf9) at 2014-11-28 02:10:06+00:00 [02:10:06] RECOVERY - puppet last run on mw1014 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [02:10:06] RECOVERY - puppet last run on amssq34 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [02:10:12] Logged the message, Master [02:10:15] Logged the message, Master [02:10:22] RECOVERY - puppet last run on mw1081 is OK: OK: Puppet is currently enabled, last run 5 seconds ago with 0 failures [02:10:22] RECOVERY - puppet last run on elastic1015 is OK: OK: Puppet is currently enabled, last run 9 seconds ago with 0 failures [02:10:29] RECOVERY - puppet last run on ms-be2012 is OK: OK: Puppet is currently enabled, last run 11 seconds ago with 0 failures [02:10:48] RECOVERY - puppet last run on search1017 is OK: OK: Puppet is currently enabled, last run 27 seconds ago with 0 failures [02:10:48] RECOVERY - puppet last run on mw1238 is OK: OK: Puppet is currently enabled, last run 21 seconds ago with 0 failures [02:10:49] RECOVERY - puppet last run on lvs3004 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [02:10:49] RECOVERY - puppet last run on tungsten is OK: OK: Puppet is currently enabled, last run 40 seconds ago with 0 failures [02:10:49] RECOVERY - puppet last run on ms-fe3002 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [02:10:49] RECOVERY - puppet last run on mw1034 is OK: OK: Puppet is currently enabled, last run 10 seconds ago with 0 failures [02:10:50] RECOVERY - puppet last run on mw1146 is OK: OK: Puppet is currently enabled, last run 25 seconds ago with 0 failures [02:11:08] RECOVERY - puppet last run on mw1159 is OK: OK: Puppet is currently enabled, last run 30 seconds ago with 0 failures [02:11:21] RECOVERY - puppet last run on snapshot1002 is OK: OK: Puppet is currently enabled, last run 48 seconds ago with 0 failures [02:11:21] RECOVERY - puppet last run on mw1181 is OK: OK: Puppet is currently enabled, last run 27 seconds ago with 0 failures [02:11:21] RECOVERY - puppet last run on snapshot1004 is OK: OK: Puppet is currently enabled, last run 23 seconds ago with 0 failures [02:11:21] RECOVERY - puppet last run on virt1003 is OK: OK: Puppet is currently enabled, last run 37 seconds ago with 0 failures [02:11:22] RECOVERY - puppet last run on plutonium is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [02:11:22] RECOVERY - puppet last run on mw1180 is OK: OK: Puppet is currently enabled, last run 36 seconds ago with 0 failures [02:11:22] RECOVERY - puppet last run on mw1051 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [02:11:23] RECOVERY - puppet last run on wtp1004 is OK: OK: Puppet is currently enabled, last run 45 seconds ago with 0 failures [02:11:23] RECOVERY - puppet last run on labsdb1006 is OK: OK: Puppet is currently enabled, last run 52 seconds ago with 0 failures [02:11:24] RECOVERY - puppet last run on virt1007 is OK: OK: Puppet is currently enabled, last run 55 seconds ago with 0 failures [02:11:39] RECOVERY - puppet last run on db2001 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [02:11:39] RECOVERY - puppet last run on ms-be2008 is OK: OK: Puppet is currently enabled, last run 31 seconds ago with 0 failures [02:11:39] RECOVERY - puppet last run on mc1005 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [02:11:50] RECOVERY - puppet last run on amssq56 is OK: OK: Puppet is currently enabled, last run 15 seconds ago with 0 failures [02:11:51] RECOVERY - puppet last run on analytics1022 is OK: OK: Puppet is currently enabled, last run 56 seconds ago with 0 failures [02:11:51] RECOVERY - puppet last run on virt1001 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [02:11:51] RECOVERY - puppet last run on mw1151 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [02:11:51] RECOVERY - puppet last run on mw1079 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [02:11:58] RECOVERY - puppet last run on argon is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [02:12:09] RECOVERY - puppet last run on cp1062 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [02:12:19] RECOVERY - puppet last run on lvs2006 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [02:12:20] RECOVERY - puppet last run on db2037 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [02:12:21] RECOVERY - puppet last run on mw1098 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [02:12:28] RECOVERY - puppet last run on rdb1001 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [02:12:32] RECOVERY - puppet last run on db2023 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [02:12:32] RECOVERY - puppet last run on bast4001 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [02:12:32] RECOVERY - puppet last run on mw1258 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [02:12:40] RECOVERY - puppet last run on mw1084 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [02:12:40] RECOVERY - puppet last run on mc1014 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [02:12:40] RECOVERY - puppet last run on gadolinium is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [02:12:53] RECOVERY - puppet last run on cp1063 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [02:12:53] RECOVERY - puppet last run on mw1133 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [02:12:53] RECOVERY - puppet last run on ssl3002 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [02:12:53] RECOVERY - puppet last run on mw1111 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [02:13:08] RECOVERY - puppet last run on mw1190 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [02:13:09] RECOVERY - puppet last run on ssl1009 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [02:13:28] RECOVERY - puppet last run on mw1004 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [02:13:32] RECOVERY - puppet last run on cp4018 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [02:13:34] RECOVERY - puppet last run on amssq41 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [02:13:34] RECOVERY - puppet last run on amssq36 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [02:13:40] RECOVERY - puppet last run on mw1183 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [02:17:38] PROBLEM - Host db2033 is DOWN: CRITICAL - Plugin timed out after 15 seconds [02:22:25] !log l10nupdate Synchronized php-1.25wmf10/cache/l10n: (no message) (duration: 00m 04s) [02:22:28] !log LocalisationUpdate completed (1.25wmf10) at 2014-11-28 02:22:28+00:00 [02:22:29] Logged the message, Master [02:22:33] Logged the message, Master [03:13:34] PROBLEM - HHVM busy threads on mw1233 is CRITICAL: CRITICAL: 11.11% of data above the critical threshold [115.2] [03:19:13] RECOVERY - HHVM busy threads on mw1233 is OK: OK: Less than 1.00% above the threshold [76.8] [03:39:53] !log LocalisationUpdate ResourceLoader cache refresh completed at Fri Nov 28 03:39:52 UTC 2014 (duration 39m 51s) [03:39:59] Logged the message, Master [04:14:15] PROBLEM - puppet last run on ms-be2001 is CRITICAL: CRITICAL: puppet fail [04:30:43] RECOVERY - puppet last run on ms-be2001 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [05:38:09] springle: If you're around and have a minute, Ellie poked me about not being able to access the Wikimania Scholarships app. It's apparently not connecting to the database -- https://phabricator.wikimedia.org/T76174 [05:38:47] I don't think I have any access to zirconium any more to debug deeper than what ended up in logstash. [05:41:35] bd808: hmm ok [05:47:43] springle: It's not a drop everything and do this now, but it would be swell if we at least found out what's broken. [05:47:47] And thanks [05:51:14] bd808: do you recall why we allowed access to scholarships from tin? it was during the setup process, right? and zirconium only for prod? [05:51:44] * springle trawls through rt [05:52:14] springle: That sounds vaguely right. We would have used tin/terbium to load the initial database I think. [05:52:47] I also sort of recall that the initial grants were done incorrectly and we had to have them fixed. [06:33:15] PROBLEM - puppet last run on mw1118 is CRITICAL: CRITICAL: Puppet has 1 failures [06:33:36] Thanks for the quick fix springle :) [06:33:53] PROBLEM - puppet last run on lvs2004 is CRITICAL: CRITICAL: Puppet has 1 failures [06:36:13] PROBLEM - puppet last run on cp3003 is CRITICAL: CRITICAL: Puppet has 1 failures [06:42:14] PROBLEM - puppet last run on db1047 is CRITICAL: CRITICAL: Puppet has 1 failures [06:45:25] bd808: np [06:46:44] RECOVERY - puppet last run on lvs2004 is OK: OK: Puppet is currently enabled, last run 55 seconds ago with 0 failures [06:48:54] RECOVERY - puppet last run on cp3003 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [06:49:18] RECOVERY - puppet last run on mw1118 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [06:54:37] RECOVERY - puppet last run on db1047 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:09:15] <_joe_> morning [07:10:18] <_joe_> springle: yt? db2033 is down since two days [07:10:28] <_joe_> we may want to ack that [07:10:59] _joe_: it's being reassigned to frack. i guess not renamed fast enough [07:11:50] <_joe_> yeah I'll ack that [07:12:38] <_joe_> springle: or do we need to clean the facts on puppet? [07:13:01] I think we need to ask Jeff Green [07:13:02] ACKNOWLEDGEMENT - Host db2033 is DOWN: CRITICAL - Plugin timed out after 15 seconds Giuseppe Lavagetto being moved to frack [07:16:10] <_joe_> well, next week then [07:16:48] <_joe_> US people are in front of the malls, with their double-edged axes ready for black friday [07:17:18] <_joe_> (that's how I imagine it anyways) [07:17:26] :D [07:23:30] Hoi I am so effing annoyed ... I cannot rely on Labs for the work that I do [07:23:46] I cannot do phabricator because it is broken for me [07:23:59] The only thing I could do is blog http://ultimategerardm.blogspot.nl/2014/11/wikimedia-labs-my-stake-is-rare-bruised.html [07:24:13] Autolist does NOT work [07:37:11] <_joe_> GerardM-: how's phabricator broken? [07:37:44] I cannot use it [07:37:52] <_joe_> why is so? [07:38:05] GerardM-: I don't see anything wrong with https://tools.wmflabs.org/autolist/? [07:38:05] <_joe_> is it a techincal problem? [07:38:09] because my user configuration is broken [07:39:28] <_joe_> GerardM-: can you be more specific? so that if I'm not able to fix it (I'm no phab expert) I can at least let the experts know on monday [07:39:38] <_joe_> or open a ticket on your behalf :) [07:40:23] Andre klapper has looked at it Quim Gill yesterday [07:40:34] yes please open a ticket on my behalf [07:41:01] <_joe_> yeah but I'd need the specifics [07:41:18] <_joe_> but if andre looked into it, I'm sure he'll be around shortly [07:41:26] I have to go to work [07:41:39] sorry [07:41:39] <_joe_> ok, have a good day [07:41:51] <_joe_> I'll be around later, andre as well probably [07:52:50] PROBLEM - puppet last run on search1018 is CRITICAL: CRITICAL: Puppet has 1 failures [07:58:03] soooo many transient puppet failures [08:06:54] RECOVERY - puppet last run on search1018 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [08:07:42] <_joe_> YuviPanda: if you were reading puppet's code, like I am doing now, you'd be surprised of how little there are [08:07:57] hehe :) [08:15:59] (03PS14) 10Yuvipanda: shinken: Setup IRC notification for shinken [puppet] - 10https://gerrit.wikimedia.org/r/173080 [08:17:16] (03PS15) 10Yuvipanda: shinken: Setup IRC notification for shinken [puppet] - 10https://gerrit.wikimedia.org/r/173080 [08:19:28] (03PS16) 10Yuvipanda: shinken: Setup IRC notification for shinken [puppet] - 10https://gerrit.wikimedia.org/r/173080 [08:24:45] PROBLEM - puppet last run on amslvs4 is CRITICAL: CRITICAL: puppet fail [08:41:31] RECOVERY - puppet last run on amslvs4 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [09:24:37] PROBLEM - puppet last run on mw1086 is CRITICAL: CRITICAL: Puppet has 1 failures [09:24:48] PROBLEM - puppet last run on elastic1003 is CRITICAL: CRITICAL: Puppet has 1 failures [09:25:07] PROBLEM - puppet last run on db1035 is CRITICAL: CRITICAL: Puppet has 1 failures [09:25:07] PROBLEM - puppet last run on radium is CRITICAL: CRITICAL: Puppet has 1 failures [09:25:27] PROBLEM - puppet last run on virt1008 is CRITICAL: CRITICAL: Puppet has 1 failures [09:25:47] PROBLEM - puppet last run on amssq37 is CRITICAL: CRITICAL: Puppet has 1 failures [09:25:47] PROBLEM - puppet last run on terbium is CRITICAL: CRITICAL: Puppet has 1 failures [09:25:51] PROBLEM - puppet last run on chromium is CRITICAL: CRITICAL: Puppet has 1 failures [09:25:58] PROBLEM - puppet last run on mw1215 is CRITICAL: CRITICAL: Puppet has 1 failures [09:26:18] PROBLEM - puppet last run on cp4010 is CRITICAL: CRITICAL: Puppet has 1 failures [09:26:31] PROBLEM - puppet last run on palladium is CRITICAL: CRITICAL: Puppet has 1 failures [09:26:47] PROBLEM - puppet last run on db1056 is CRITICAL: CRITICAL: Puppet has 1 failures [09:27:10] PROBLEM - puppet last run on mw1064 is CRITICAL: CRITICAL: Puppet has 1 failures [09:27:10] PROBLEM - puppet last run on mw1113 is CRITICAL: CRITICAL: Puppet has 1 failures [09:27:10] PROBLEM - puppet last run on db1037 is CRITICAL: CRITICAL: Puppet has 1 failures [09:27:10] PROBLEM - puppet last run on osm-cp1001 is CRITICAL: CRITICAL: Puppet has 1 failures [09:27:31] PROBLEM - puppet last run on mw1110 is CRITICAL: CRITICAL: Puppet has 1 failures [09:27:48] PROBLEM - puppet last run on mw1253 is CRITICAL: CRITICAL: Puppet has 1 failures [09:27:53] PROBLEM - puppet last run on db2010 is CRITICAL: CRITICAL: Puppet has 1 failures [09:27:59] PROBLEM - puppet last run on cp1054 is CRITICAL: CRITICAL: Puppet has 1 failures [09:35:12] RECOVERY - puppet last run on mw1064 is OK: OK: Puppet is currently enabled, last run 28 seconds ago with 0 failures [09:35:18] RECOVERY - puppet last run on mw1113 is OK: OK: Puppet is currently enabled, last run 7 seconds ago with 0 failures [09:35:19] RECOVERY - puppet last run on osm-cp1001 is OK: OK: Puppet is currently enabled, last run 41 seconds ago with 0 failures [09:35:38] RECOVERY - puppet last run on mw1086 is OK: OK: Puppet is currently enabled, last run 21 seconds ago with 0 failures [09:35:50] RECOVERY - puppet last run on mw1253 is OK: OK: Puppet is currently enabled, last run 41 seconds ago with 0 failures [09:35:50] RECOVERY - puppet last run on elastic1003 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [09:36:01] RECOVERY - puppet last run on cp1054 is OK: OK: Puppet is currently enabled, last run 3 seconds ago with 0 failures [09:36:18] RECOVERY - puppet last run on db1035 is OK: OK: Puppet is currently enabled, last run 26 seconds ago with 0 failures [09:36:19] RECOVERY - puppet last run on radium is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [09:36:31] RECOVERY - puppet last run on virt1008 is OK: OK: Puppet is currently enabled, last run 31 seconds ago with 0 failures [09:36:51] RECOVERY - puppet last run on terbium is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [09:36:58] RECOVERY - puppet last run on chromium is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [09:36:58] RECOVERY - puppet last run on mw1215 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [09:37:18] RECOVERY - puppet last run on cp4010 is OK: OK: Puppet is currently enabled, last run 1 second ago with 0 failures [09:37:37] RECOVERY - puppet last run on palladium is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [09:37:42] RECOVERY - puppet last run on db1056 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [09:38:16] RECOVERY - puppet last run on db1037 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [09:38:25] RECOVERY - puppet last run on mw1110 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [09:38:54] RECOVERY - puppet last run on db2010 is OK: OK: Puppet is currently enabled, last run 6 seconds ago with 0 failures [09:39:34] RECOVERY - puppet last run on amssq37 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [09:42:06] YuviPanda, possibly GerardM's problem with autolist, is related to the 3 example queries linked in https://lists.wikimedia.org/pipermail/wikimedia-l/2014-November/075465.html ? (I was trying them out a few hours ago, and they're not giving any output.) [09:48:31] _joe_, GerardM's problem with phab, (IIUC, from scrollback in -devtools) is that he has an old ex-staff email account associated with https://wikitech.wikimedia.org/wiki/User:GerardM that has been accidentally attached to https://phabricator.wikimedia.org/p/GerardM/ and so he can't access the confirmation email, nor change settings. Qgil has promised to find someone to delete the existing account, so that GerardM can recreate it [09:48:31] (qgil doesn't have rights to delete accounts), but explained that it would probably take until Monday evening (EU time). Hence, frustration. [09:48:44] (just to explain the scrollback. :) [09:49:28] <_joe_> quiddity: thanks, I think I don't have that right either [09:49:59] <_joe_> and that needs fixing [09:56:59] quiddity: hmm, so http://wdq.wmflabs.org/ isn't loading for me. is it for you? [09:58:36] YuviPanda, nope, not loading. [09:58:43] yeah, wdq.wmflabs.org is down, and that's why autolist isn't loading. [09:59:07] and it runs in its own project, so I'm not sure what exactly I can do. [09:59:25] * YuviPanda looks to see who are the maintainers [10:01:07] * YuviPanda suspects magnus [10:05:43] yup magnus [10:06:04] hmm, the instance seems down [10:06:09] and I have no idea which project it is in [10:06:11] * YuviPanda looks [10:06:29] wikidata-wdq-mm [10:12:05] quiddity: thedj hmm instance is up, and wd_server is 95% MEM, and I'm not sure what to do. [10:12:16] I've no idea about what this service does or how it is set up [10:12:53] it queries a copy of the wikidata db. [10:13:24] so probably someone build a query that is too expensive. (which is why we don't have a production wikidata query yet :) [10:20:27] heh [10:20:39] thedj: I suppose I should email magnus [10:22:00] emailed [10:35:47] (03PS1) 10Yuvipanda: Make ircecho run as ircecho user [debs/ircecho] - 10https://gerrit.wikimedia.org/r/176333 [10:55:12] (03PS1) 10Giuseppe Lavagetto: hiera: role-based backend, role() helper function (WiP) [puppet] - 10https://gerrit.wikimedia.org/r/176334 [10:55:30] <_joe_> paravoid: ^^ still WiP, but does what we said yesterday [10:57:52] <_joe_> hiera looks up in all roles declared with role('foo::bar') using a custom hierarchy, so that any variable will be searched e.g. in role/foo/bar.yaml and in role/eqiad/foo/bar.yaml, and multiple role to lookup into are supported as well; if a class parameter is defined with different values, we raise an exception [10:58:59] <_joe_> it's funny how in puppet scoped variables are immutable, except if they are an hash, then you can append keys to it indefinitely [11:14:41] (03PS3) 10KartikMistry: Add ContentTranslation database in wikishared [mediawiki-config] - 10https://gerrit.wikimedia.org/r/175979 [11:20:35] !log Updated gerrit plugin its-phabricator to 9edf90a182e43bfeea7ebbcb20d4a52b6213600d [11:20:43] !log Updated gerrit plugin its-phabricator-from-bugzilla to 97c5f02d3ca6259488a763515251c5cc57a11a51 [11:20:43] Logged the message, Master [11:20:45] Logged the message, Master [11:26:59] (03PS5) 10Giuseppe Lavagetto: reimage: add a few configs, beautify output [puppet] - 10https://gerrit.wikimedia.org/r/175965 [11:28:39] (03PS1) 10QChris: Upgrade phabricator plugins to force UTF-8 for conduit [gerrit/plugins] - 10https://gerrit.wikimedia.org/r/176336 [11:32:56] PROBLEM - puppet last run on amssq40 is CRITICAL: CRITICAL: puppet fail [11:33:08] PROBLEM - puppet last run on cp3010 is CRITICAL: CRITICAL: puppet fail [11:33:43] (03CR) 10Giuseppe Lavagetto: [C: 032] reimage: add a few configs, beautify output [puppet] - 10https://gerrit.wikimedia.org/r/175965 (owner: 10Giuseppe Lavagetto) [11:49:12] RECOVERY - puppet last run on cp3010 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [11:51:56] RECOVERY - puppet last run on amssq40 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [12:06:15] (03PS4) 10KartikMistry: Add ContentTranslation database in wikishared [mediawiki-config] - 10https://gerrit.wikimedia.org/r/175979 [13:01:35] PROBLEM - puppet last run on cp1066 is CRITICAL: CRITICAL: Puppet has 20 failures [13:01:47] PROBLEM - puppet last run on mw1131 is CRITICAL: CRITICAL: Puppet has 34 failures [13:01:47] PROBLEM - puppet last run on palladium is CRITICAL: CRITICAL: Puppet has 39 failures [13:01:47] PROBLEM - puppet last run on mw1015 is CRITICAL: CRITICAL: puppet fail [13:01:47] PROBLEM - puppet last run on elastic1029 is CRITICAL: CRITICAL: Puppet has 26 failures [13:01:56] PROBLEM - puppet last run on mw1037 is CRITICAL: CRITICAL: Puppet has 32 failures [13:01:56] PROBLEM - puppet last run on cp4020 is CRITICAL: CRITICAL: Puppet has 24 failures [13:01:56] PROBLEM - puppet last run on baham is CRITICAL: CRITICAL: Puppet has 2 failures [13:01:57] PROBLEM - puppet last run on cp4016 is CRITICAL: CRITICAL: puppet fail [13:01:57] PROBLEM - puppet last run on mw1199 is CRITICAL: CRITICAL: Puppet has 61 failures [13:02:08] PROBLEM - puppet last run on mw1085 is CRITICAL: CRITICAL: puppet fail [13:02:08] PROBLEM - puppet last run on es1004 is CRITICAL: CRITICAL: Puppet has 28 failures [13:02:16] PROBLEM - puppet last run on mw1020 is CRITICAL: CRITICAL: puppet fail [13:02:16] PROBLEM - puppet last run on cp1064 is CRITICAL: CRITICAL: puppet fail [13:02:16] PROBLEM - puppet last run on elastic1009 is CRITICAL: CRITICAL: puppet fail [13:02:16] PROBLEM - puppet last run on ms-be3004 is CRITICAL: CRITICAL: puppet fail [13:02:16] PROBLEM - puppet last run on amssq45 is CRITICAL: CRITICAL: puppet fail [13:02:17] PROBLEM - puppet last run on amssq39 is CRITICAL: CRITICAL: Puppet has 4 failures [13:02:17] PROBLEM - puppet last run on amssq52 is CRITICAL: CRITICAL: puppet fail [13:02:18] PROBLEM - puppet last run on amssq33 is CRITICAL: CRITICAL: Puppet has 7 failures [13:02:18] PROBLEM - puppet last run on cp3019 is CRITICAL: CRITICAL: Puppet has 17 failures [13:02:19] PROBLEM - puppet last run on eeden is CRITICAL: CRITICAL: Puppet has 4 failures [13:02:19] PROBLEM - puppet last run on mw1113 is CRITICAL: CRITICAL: Puppet has 34 failures [13:02:20] PROBLEM - puppet last run on ms-fe3001 is CRITICAL: CRITICAL: Puppet has 2 failures [13:02:20] PROBLEM - puppet last run on dysprosium is CRITICAL: CRITICAL: Puppet has 23 failures [13:02:21] PROBLEM - puppet last run on mc1015 is CRITICAL: CRITICAL: Puppet has 19 failures [13:02:25] PROBLEM - puppet last run on mw1142 is CRITICAL: CRITICAL: Puppet has 2 failures [13:02:25] PROBLEM - puppet last run on lanthanum is CRITICAL: CRITICAL: puppet fail [13:02:25] PROBLEM - puppet last run on search1003 is CRITICAL: CRITICAL: Puppet has 9 failures [13:02:26] PROBLEM - puppet last run on mw1047 is CRITICAL: CRITICAL: Puppet has 63 failures [13:02:26] PROBLEM - puppet last run on labsdb1002 is CRITICAL: CRITICAL: Puppet has 24 failures [13:02:35] PROBLEM - puppet last run on mw1110 is CRITICAL: CRITICAL: Puppet has 32 failures [13:02:35] PROBLEM - puppet last run on mw1179 is CRITICAL: CRITICAL: puppet fail [13:02:35] PROBLEM - puppet last run on mw1073 is CRITICAL: CRITICAL: Puppet has 76 failures [13:02:35] PROBLEM - puppet last run on ms-be1005 is CRITICAL: CRITICAL: puppet fail [13:02:35] PROBLEM - puppet last run on mw1075 is CRITICAL: CRITICAL: Puppet has 66 failures [13:02:46] PROBLEM - puppet last run on elastic1010 is CRITICAL: CRITICAL: puppet fail [13:02:46] PROBLEM - puppet last run on tmh1001 is CRITICAL: CRITICAL: Puppet has 55 failures [13:02:46] PROBLEM - puppet last run on radon is CRITICAL: CRITICAL: Puppet has 29 failures [13:02:55] PROBLEM - puppet last run on db1009 is CRITICAL: CRITICAL: Puppet has 22 failures [13:02:56] PROBLEM - puppet last run on mw1093 is CRITICAL: CRITICAL: Puppet has 6 failures [13:02:56] PROBLEM - puppet last run on ssl1003 is CRITICAL: CRITICAL: Puppet has 22 failures [13:02:57] PROBLEM - puppet last run on elastic1013 is CRITICAL: CRITICAL: puppet fail [13:03:06] PROBLEM - puppet last run on mw1186 is CRITICAL: CRITICAL: Puppet has 3 failures [13:03:06] PROBLEM - puppet last run on zinc is CRITICAL: CRITICAL: Puppet has 26 failures [13:03:06] PROBLEM - puppet last run on mw1095 is CRITICAL: CRITICAL: Puppet has 60 failures [13:03:07] PROBLEM - puppet last run on db1068 is CRITICAL: CRITICAL: Puppet has 21 failures [13:03:07] PROBLEM - puppet last run on mw1128 is CRITICAL: CRITICAL: Puppet has 69 failures [13:03:07] PROBLEM - puppet last run on db2010 is CRITICAL: CRITICAL: Puppet has 13 failures [13:03:07] PROBLEM - puppet last run on virt1009 is CRITICAL: CRITICAL: puppet fail [13:03:08] PROBLEM - puppet last run on db1027 is CRITICAL: CRITICAL: Puppet has 24 failures [13:03:08] PROBLEM - puppet last run on search1008 is CRITICAL: CRITICAL: Puppet has 54 failures [13:03:09] PROBLEM - puppet last run on es1003 is CRITICAL: CRITICAL: puppet fail [13:03:09] PROBLEM - puppet last run on mw1155 is CRITICAL: CRITICAL: Puppet has 38 failures [13:03:10] PROBLEM - puppet last run on lvs2005 is CRITICAL: CRITICAL: Puppet has 27 failures [13:03:10] PROBLEM - puppet last run on cp4009 is CRITICAL: CRITICAL: Puppet has 34 failures [13:03:11] PROBLEM - puppet last run on cp4011 is CRITICAL: CRITICAL: Puppet has 29 failures [13:03:12] PROBLEM - puppet last run on mw1157 is CRITICAL: CRITICAL: puppet fail [13:03:12] PROBLEM - puppet last run on mw1103 is CRITICAL: CRITICAL: Puppet has 66 failures [13:03:12] PROBLEM - puppet last run on pollux is CRITICAL: CRITICAL: puppet fail [13:03:13] PROBLEM - puppet last run on mw1070 is CRITICAL: CRITICAL: puppet fail [13:03:19] PROBLEM - puppet last run on lvs3002 is CRITICAL: CRITICAL: puppet fail [13:03:19] PROBLEM - puppet last run on cp3021 is CRITICAL: CRITICAL: Puppet has 26 failures [13:03:19] PROBLEM - puppet last run on cp3005 is CRITICAL: CRITICAL: Puppet has 18 failures [13:03:19] PROBLEM - puppet last run on mw1058 is CRITICAL: CRITICAL: Puppet has 68 failures [13:03:27] PROBLEM - puppet last run on labsdb1005 is CRITICAL: CRITICAL: Puppet has 19 failures [13:03:28] PROBLEM - puppet last run on elastic1016 is CRITICAL: CRITICAL: Puppet has 23 failures [13:03:28] PROBLEM - puppet last run on analytics1012 is CRITICAL: CRITICAL: Puppet has 22 failures [13:03:36] PROBLEM - puppet last run on mw1094 is CRITICAL: CRITICAL: puppet fail [13:03:37] PROBLEM - puppet last run on mw1137 is CRITICAL: CRITICAL: Puppet has 69 failures [13:03:37] PROBLEM - puppet last run on hydrogen is CRITICAL: CRITICAL: Puppet has 17 failures [13:03:37] PROBLEM - puppet last run on mw1102 is CRITICAL: CRITICAL: puppet fail [13:03:37] PROBLEM - puppet last run on search1009 is CRITICAL: CRITICAL: Puppet has 53 failures [13:03:37] PROBLEM - puppetmaster backend https on palladium is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 8141: HTTP/1.1 500 Internal Server Error [13:03:38] PROBLEM - puppet last run on search1019 is CRITICAL: CRITICAL: puppet fail [13:03:38] PROBLEM - puppet last run on labsdb1007 is CRITICAL: CRITICAL: Puppet has 22 failures [13:03:39] PROBLEM - puppet last run on praseodymium is CRITICAL: CRITICAL: Puppet has 26 failures [13:03:39] PROBLEM - puppet last run on neon is CRITICAL: CRITICAL: puppet fail [13:03:51] PROBLEM - puppet last run on es1005 is CRITICAL: CRITICAL: Puppet has 27 failures [13:03:55] PROBLEM - puppet last run on cp4015 is CRITICAL: CRITICAL: puppet fail [13:03:56] PROBLEM - puppet last run on db2017 is CRITICAL: CRITICAL: Puppet has 3 failures [13:03:56] PROBLEM - puppet last run on ms-be2009 is CRITICAL: CRITICAL: Puppet has 26 failures [13:03:56] PROBLEM - puppet last run on es2006 is CRITICAL: CRITICAL: Puppet has 19 failures [13:03:56] PROBLEM - puppet last run on ms-be2015 is CRITICAL: CRITICAL: Puppet has 27 failures [13:03:57] PROBLEM - puppet last run on lvs4001 is CRITICAL: CRITICAL: puppet fail [13:03:57] PROBLEM - puppet last run on bast2001 is CRITICAL: CRITICAL: Puppet has 18 failures [13:03:57] PROBLEM - puppet last run on cp4002 is CRITICAL: CRITICAL: Puppet has 15 failures [13:03:57] PROBLEM - puppet last run on amssq37 is CRITICAL: CRITICAL: Puppet has 33 failures [13:03:58] PROBLEM - puppet last run on mw1191 is CRITICAL: CRITICAL: puppet fail [13:03:58] PROBLEM - puppet last run on mw1017 is CRITICAL: CRITICAL: puppet fail [13:03:59] PROBLEM - puppet last run on db1049 is CRITICAL: CRITICAL: puppet fail [13:03:59] PROBLEM - puppet last run on cp3018 is CRITICAL: CRITICAL: Puppet has 5 failures [13:04:00] PROBLEM - puppet last run on cp3007 is CRITICAL: CRITICAL: puppet fail [13:04:00] PROBLEM - puppet last run on amssq31 is CRITICAL: CRITICAL: Puppet has 17 failures [13:04:01] PROBLEM - puppet last run on mw1194 is CRITICAL: CRITICAL: Puppet has 60 failures [13:04:01] PROBLEM - puppet last run on mw1196 is CRITICAL: CRITICAL: puppet fail [13:04:02] PROBLEM - puppet last run on virt1005 is CRITICAL: CRITICAL: Puppet has 22 failures [13:04:07] PROBLEM - puppet last run on amslvs4 is CRITICAL: CRITICAL: puppet fail [13:04:07] PROBLEM - puppet last run on amssq50 is CRITICAL: CRITICAL: Puppet has 30 failures [13:04:08] PROBLEM - puppet last run on db1019 is CRITICAL: CRITICAL: Puppet has 24 failures [13:04:08] PROBLEM - puppet last run on mw1136 is CRITICAL: CRITICAL: puppet fail [13:04:08] PROBLEM - puppet last run on db1058 is CRITICAL: CRITICAL: puppet fail [13:04:08] PROBLEM - puppet last run on mw1101 is CRITICAL: CRITICAL: Puppet has 66 failures [13:04:18] PROBLEM - puppet last run on ms-be1014 is CRITICAL: CRITICAL: Puppet has 29 failures [13:04:18] PROBLEM - puppet last run on search1014 is CRITICAL: CRITICAL: puppet fail [13:04:18] PROBLEM - puppet last run on analytics1021 is CRITICAL: CRITICAL: Puppet has 17 failures [13:04:18] PROBLEM - puppet last run on lvs2002 is CRITICAL: CRITICAL: Puppet has 25 failures [13:04:19] PROBLEM - puppet last run on cp4012 is CRITICAL: CRITICAL: Puppet has 42 failures [13:04:19] PROBLEM - puppet last run on cp4010 is CRITICAL: CRITICAL: Puppet has 31 failures [13:04:32] PROBLEM - puppet last run on db1005 is CRITICAL: CRITICAL: Puppet has 18 failures [13:04:41] PROBLEM - puppet last run on mw1019 is CRITICAL: CRITICAL: Puppet has 75 failures [13:04:41] PROBLEM - puppet last run on cp3017 is CRITICAL: CRITICAL: Puppet has 28 failures [13:04:51] PROBLEM - puppet last run on mw1138 is CRITICAL: CRITICAL: puppet fail [13:05:02] PROBLEM - puppet last run on mw1230 is CRITICAL: CRITICAL: Puppet has 74 failures [13:05:02] PROBLEM - puppet last run on wtp1014 is CRITICAL: CRITICAL: Puppet has 29 failures [13:05:02] PROBLEM - puppet last run on mw1062 is CRITICAL: CRITICAL: puppet fail [13:05:02] PROBLEM - puppet last run on mw1214 is CRITICAL: CRITICAL: puppet fail [13:05:13] PROBLEM - puppet last run on mw1216 is CRITICAL: CRITICAL: puppet fail [13:05:13] PROBLEM - puppet last run on mw1078 is CRITICAL: CRITICAL: Puppet has 64 failures [13:05:13] PROBLEM - puppet last run on virt1002 is CRITICAL: CRITICAL: puppet fail [13:05:13] PROBLEM - puppet last run on erbium is CRITICAL: CRITICAL: puppet fail [13:05:14] PROBLEM - puppet last run on ytterbium is CRITICAL: CRITICAL: puppet fail [13:05:14] PROBLEM - puppet last run on analytics1015 is CRITICAL: CRITICAL: puppet fail [13:05:26] PROBLEM - puppet last run on db1010 is CRITICAL: CRITICAL: puppet fail [13:05:28] PROBLEM - puppet last run on pc1001 is CRITICAL: CRITICAL: puppet fail [13:05:28] PROBLEM - puppet last run on wtp1024 is CRITICAL: CRITICAL: puppet fail [13:05:28] PROBLEM - puppet last run on wtp1021 is CRITICAL: CRITICAL: puppet fail [13:05:41] PROBLEM - puppet last run on cp1043 is CRITICAL: CRITICAL: puppet fail [13:05:41] PROBLEM - puppet last run on analytics1034 is CRITICAL: CRITICAL: Puppet has 26 failures [13:05:41] PROBLEM - puppet last run on tmh1002 is CRITICAL: CRITICAL: puppet fail [13:05:41] PROBLEM - puppet last run on db1007 is CRITICAL: CRITICAL: puppet fail [13:05:51] PROBLEM - puppet last run on ocg1002 is CRITICAL: CRITICAL: Puppet has 33 failures [13:05:51] PROBLEM - puppet last run on elastic1026 is CRITICAL: CRITICAL: puppet fail [13:05:52] PROBLEM - puppet last run on es1009 is CRITICAL: CRITICAL: puppet fail [13:05:52] PROBLEM - puppet last run on lvs1006 is CRITICAL: CRITICAL: Puppet has 19 failures [13:05:52] PROBLEM - puppet last run on analytics1019 is CRITICAL: CRITICAL: puppet fail [13:05:52] PROBLEM - puppet last run on mw1182 is CRITICAL: CRITICAL: Puppet has 80 failures [13:05:52] PROBLEM - puppet last run on zirconium is CRITICAL: CRITICAL: puppet fail [13:06:03] PROBLEM - puppet last run on mw1257 is CRITICAL: CRITICAL: puppet fail [13:06:03] PROBLEM - puppet last run on labsdb1001 is CRITICAL: CRITICAL: puppet fail [13:06:03] PROBLEM - puppet last run on elastic1028 is CRITICAL: CRITICAL: puppet fail [13:06:03] PROBLEM - puppet last run on ssl3003 is CRITICAL: CRITICAL: puppet fail [13:06:03] PROBLEM - puppet last run on cp3022 is CRITICAL: CRITICAL: puppet fail [13:06:04] PROBLEM - puppet last run on amssq58 is CRITICAL: CRITICAL: puppet fail [13:06:21] PROBLEM - puppet last run on analytics1036 is CRITICAL: CRITICAL: puppet fail [13:06:21] PROBLEM - puppet last run on mw1072 is CRITICAL: CRITICAL: puppet fail [13:06:21] PROBLEM - puppet last run on search1021 is CRITICAL: CRITICAL: Puppet has 55 failures [13:06:21] PROBLEM - puppet last run on elastic1031 is CRITICAL: CRITICAL: Puppet has 19 failures [13:06:21] PROBLEM - puppet last run on mw1036 is CRITICAL: CRITICAL: puppet fail [13:06:22] PROBLEM - puppet last run on db1041 is CRITICAL: CRITICAL: puppet fail [13:06:22] PROBLEM - puppet last run on mw1218 is CRITICAL: CRITICAL: puppet fail [13:06:23] PROBLEM - puppet last run on db2035 is CRITICAL: CRITICAL: puppet fail [13:06:35] PROBLEM - puppet last run on ms-be2010 is CRITICAL: CRITICAL: Puppet has 33 failures [13:06:36] PROBLEM - puppet last run on ms-be2013 is CRITICAL: CRITICAL: puppet fail [13:06:36] PROBLEM - puppet last run on amssq44 is CRITICAL: CRITICAL: puppet fail [13:06:36] PROBLEM - puppet last run on cp3006 is CRITICAL: CRITICAL: puppet fail [13:06:42] PROBLEM - puppet last run on mw1252 is CRITICAL: CRITICAL: Puppet has 68 failures [13:06:42] PROBLEM - puppet last run on mw1232 is CRITICAL: CRITICAL: puppet fail [13:06:43] PROBLEM - puppet last run on ms-be1013 is CRITICAL: CRITICAL: puppet fail [13:06:43] PROBLEM - puppet last run on mw1233 is CRITICAL: CRITICAL: puppet fail [13:07:11] PROBLEM - puppet last run on analytics1039 is CRITICAL: CRITICAL: Puppet has 27 failures [13:07:15] PROBLEM - puppet last run on mc1011 is CRITICAL: CRITICAL: puppet fail [13:07:15] PROBLEM - puppet last run on caesium is CRITICAL: CRITICAL: puppet fail [13:07:15] PROBLEM - puppet last run on cp1051 is CRITICAL: CRITICAL: Puppet has 26 failures [13:07:16] PROBLEM - puppet last run on mw1256 is CRITICAL: CRITICAL: Puppet has 71 failures [13:07:16] PROBLEM - puppet last run on cp4006 is CRITICAL: CRITICAL: puppet fail [13:07:16] PROBLEM - puppet last run on cp4007 is CRITICAL: CRITICAL: puppet fail [13:07:16] PROBLEM - puppet last run on mw1132 is CRITICAL: CRITICAL: puppet fail [13:07:17] PROBLEM - puppet last run on vanadium is CRITICAL: CRITICAL: puppet fail [13:07:17] PROBLEM - puppet last run on mw1192 is CRITICAL: CRITICAL: puppet fail [13:07:27] PROBLEM - puppet last run on mw1115 is CRITICAL: CRITICAL: Puppet has 61 failures [13:07:27] PROBLEM - puppet last run on analytics1029 is CRITICAL: CRITICAL: Puppet has 20 failures [13:07:27] PROBLEM - puppet last run on magnesium is CRITICAL: CRITICAL: puppet fail [13:07:27] PROBLEM - puppet last run on mw1234 is CRITICAL: CRITICAL: Puppet has 72 failures [13:07:28] PROBLEM - puppet last run on mw1080 is CRITICAL: CRITICAL: puppet fail [13:07:28] PROBLEM - puppet last run on mc1004 is CRITICAL: CRITICAL: Puppet has 19 failures [13:07:28] PROBLEM - puppet last run on mw1147 is CRITICAL: CRITICAL: puppet fail [13:07:29] PROBLEM - puppet last run on mw1005 is CRITICAL: CRITICAL: puppet fail [13:07:29] PROBLEM - puppet last run on mw1124 is CRITICAL: CRITICAL: puppet fail [13:07:30] PROBLEM - puppet last run on mc1010 is CRITICAL: CRITICAL: puppet fail [13:07:37] PROBLEM - puppet last run on db1053 is CRITICAL: CRITICAL: puppet fail [13:07:42] PROBLEM - puppet last run on mw1048 is CRITICAL: CRITICAL: Puppet has 53 failures [13:07:44] PROBLEM - puppet last run on cp3020 is CRITICAL: CRITICAL: puppet fail [13:07:46] PROBLEM - puppet last run on amssq59 is CRITICAL: CRITICAL: puppet fail [13:07:46] PROBLEM - puppet last run on cp3013 is CRITICAL: CRITICAL: Puppet has 24 failures [13:07:46] PROBLEM - puppet last run on cp3015 is CRITICAL: CRITICAL: puppet fail [13:07:47] PROBLEM - puppet last run on mw1083 is CRITICAL: CRITICAL: Puppet has 82 failures [13:07:48] PROBLEM - puppet last run on wtp1009 is CRITICAL: CRITICAL: puppet fail [13:07:49] PROBLEM - puppet last run on mw1240 is CRITICAL: CRITICAL: Puppet has 76 failures [13:07:49] PROBLEM - puppet last run on cp1039 is CRITICAL: CRITICAL: puppet fail [13:07:51] PROBLEM - puppet last run on curium is CRITICAL: CRITICAL: puppet fail [13:07:51] PROBLEM - puppet last run on ocg1001 is CRITICAL: CRITICAL: Puppet has 29 failures [13:07:51] PROBLEM - puppet last run on logstash1001 is CRITICAL: CRITICAL: puppet fail [13:07:51] PROBLEM - puppet last run on uranium is CRITICAL: CRITICAL: Puppet has 26 failures [13:07:51] PROBLEM - puppet last run on dbproxy1002 is CRITICAL: CRITICAL: puppet fail [13:07:52] PROBLEM - puppet last run on platinum is CRITICAL: CRITICAL: puppet fail [13:07:52] PROBLEM - puppet last run on dbstore1001 is CRITICAL: CRITICAL: Puppet has 19 failures [13:07:53] PROBLEM - puppet last run on mw1040 is CRITICAL: CRITICAL: Puppet has 57 failures [13:07:53] PROBLEM - puppet last run on mw1169 is CRITICAL: CRITICAL: Puppet has 67 failures [13:07:54] PROBLEM - puppet last run on mw1067 is CRITICAL: CRITICAL: Puppet has 71 failures [13:07:54] PROBLEM - puppet last run on mw1134 is CRITICAL: CRITICAL: Puppet has 58 failures [13:07:55] PROBLEM - puppet last run on ms-be1001 is CRITICAL: CRITICAL: puppet fail [13:07:55] PROBLEM - puppet last run on mw1221 is CRITICAL: CRITICAL: Puppet has 60 failures [13:07:56] PROBLEM - puppet last run on cp1069 is CRITICAL: CRITICAL: Puppet has 26 failures [13:07:56] PROBLEM - puppet last run on analytics1033 is CRITICAL: CRITICAL: Puppet has 27 failures [13:07:57] PROBLEM - puppet last run on db1029 is CRITICAL: CRITICAL: puppet fail [13:07:57] PROBLEM - puppet last run on cp1059 is CRITICAL: CRITICAL: puppet fail [13:07:58] PROBLEM - puppet last run on mw1096 is CRITICAL: CRITICAL: Puppet has 62 failures [13:07:58] PROBLEM - puppet last run on cp1065 is CRITICAL: CRITICAL: Puppet has 26 failures [13:07:59] PROBLEM - puppet last run on mw1059 is CRITICAL: CRITICAL: puppet fail [13:08:01] PROBLEM - puppet last run on mw1012 is CRITICAL: CRITICAL: puppet fail [13:08:03] PROBLEM - puppet last run on mw1031 is CRITICAL: CRITICAL: Puppet has 70 failures [13:08:12] PROBLEM - puppet last run on mw1038 is CRITICAL: CRITICAL: puppet fail [13:08:12] PROBLEM - puppet last run on mw1246 is CRITICAL: CRITICAL: Puppet has 72 failures [13:08:12] PROBLEM - puppet last run on mw1041 is CRITICAL: CRITICAL: puppet fail [13:08:12] PROBLEM - puppet last run on wtp1017 is CRITICAL: CRITICAL: puppet fail [13:08:12] PROBLEM - puppet last run on search1020 is CRITICAL: CRITICAL: Puppet has 54 failures [13:08:13] PROBLEM - puppet last run on mw1200 is CRITICAL: CRITICAL: Puppet has 68 failures [13:08:13] PROBLEM - puppet last run on ms-be1004 is CRITICAL: CRITICAL: puppet fail [13:08:14] PROBLEM - puppet last run on mw1035 is CRITICAL: CRITICAL: Puppet has 70 failures [13:08:14] PROBLEM - puppet last run on wtp1019 is CRITICAL: CRITICAL: Puppet has 27 failures [13:08:15] PROBLEM - puppet last run on mw1140 is CRITICAL: CRITICAL: puppet fail [13:08:15] PROBLEM - puppet last run on mc1008 is CRITICAL: CRITICAL: Puppet has 22 failures [13:08:21] PROBLEM - puppet last run on db1024 is CRITICAL: CRITICAL: Puppet has 20 failures [13:08:23] PROBLEM - puppet last run on mw1145 is CRITICAL: CRITICAL: puppet fail [13:08:23] PROBLEM - puppet last run on db1065 is CRITICAL: CRITICAL: puppet fail [13:08:23] PROBLEM - puppet last run on es1008 is CRITICAL: CRITICAL: puppet fail [13:08:23] PROBLEM - puppet last run on analytics1024 is CRITICAL: CRITICAL: Puppet has 26 failures [13:08:24] PROBLEM - puppet last run on mw1184 is CRITICAL: CRITICAL: Puppet has 67 failures [13:08:24] PROBLEM - puppet last run on logstash1003 is CRITICAL: CRITICAL: Puppet has 33 failures [13:08:33] PROBLEM - puppet last run on elastic1001 is CRITICAL: CRITICAL: Puppet has 25 failures [13:08:33] PROBLEM - puppet last run on ms-be1006 is CRITICAL: CRITICAL: puppet fail [13:08:33] PROBLEM - puppet last run on rdb1004 is CRITICAL: CRITICAL: Puppet has 29 failures [13:08:33] PROBLEM - puppet last run on mw1161 is CRITICAL: CRITICAL: Puppet has 66 failures [13:08:33] PROBLEM - puppet last run on netmon1001 is CRITICAL: CRITICAL: Puppet has 28 failures [13:08:33] PROBLEM - puppet last run on ms-be2002 is CRITICAL: CRITICAL: Puppet has 30 failures [13:08:33] PROBLEM - puppet last run on cp4013 is CRITICAL: CRITICAL: Puppet has 32 failures [13:08:34] PROBLEM - puppet last run on cp4017 is CRITICAL: CRITICAL: Puppet has 27 failures [13:08:34] PROBLEM - puppet last run on db2019 is CRITICAL: CRITICAL: puppet fail [13:08:35] PROBLEM - puppet last run on mw1130 is CRITICAL: CRITICAL: Puppet has 74 failures [13:08:35] PROBLEM - puppet last run on mw1007 is CRITICAL: CRITICAL: puppet fail [13:08:36] PROBLEM - puppet last run on cp1070 is CRITICAL: CRITICAL: puppet fail [13:08:36] PROBLEM - puppet last run on mw1106 is CRITICAL: CRITICAL: puppet fail [13:08:37] PROBLEM - puppet last run on ssl1004 is CRITICAL: CRITICAL: Puppet has 26 failures [13:08:37] PROBLEM - puppet last run on amssq57 is CRITICAL: CRITICAL: Puppet has 28 failures [13:08:38] PROBLEM - puppet last run on amssq54 is CRITICAL: CRITICAL: puppet fail [13:08:41] PROBLEM - puppet last run on mw1187 is CRITICAL: CRITICAL: puppet fail [13:08:41] PROBLEM - puppet last run on es1006 is CRITICAL: CRITICAL: Puppet has 25 failures [13:08:41] PROBLEM - puppet last run on analytics1041 is CRITICAL: CRITICAL: puppet fail [13:08:42] PROBLEM - puppet last run on cp1047 is CRITICAL: CRITICAL: Puppet has 37 failures [13:08:42] PROBLEM - puppet last run on cp1057 is CRITICAL: CRITICAL: Puppet has 30 failures [13:08:42] PROBLEM - puppet last run on mc1009 is CRITICAL: CRITICAL: Puppet has 20 failures [13:08:52] PROBLEM - puppet last run on mw1250 is CRITICAL: CRITICAL: puppet fail [13:08:52] PROBLEM - puppet last run on ssl1002 is CRITICAL: CRITICAL: puppet fail [13:08:52] PROBLEM - puppet last run on gold is CRITICAL: CRITICAL: puppet fail [13:08:52] PROBLEM - puppet last run on mw1127 is CRITICAL: CRITICAL: Puppet has 70 failures [13:08:52] PROBLEM - puppet last run on cerium is CRITICAL: CRITICAL: puppet fail [13:09:05] PROBLEM - puppet last run on mw1245 is CRITICAL: CRITICAL: Puppet has 76 failures [13:09:05] PROBLEM - puppet last run on mw1028 is CRITICAL: CRITICAL: Puppet has 65 failures [13:09:06] PROBLEM - puppet last run on analytics1017 is CRITICAL: CRITICAL: puppet fail [13:09:07] PROBLEM - puppet last run on ms-be2004 is CRITICAL: CRITICAL: puppet fail [13:09:10] PROBLEM - puppet last run on db2039 is CRITICAL: CRITICAL: Puppet has 16 failures [13:09:13] PROBLEM - puppet last run on mw1178 is CRITICAL: CRITICAL: Puppet has 61 failures [13:09:14] PROBLEM - puppet last run on wtp1020 is CRITICAL: CRITICAL: Puppet has 26 failures [13:09:14] PROBLEM - puppet last run on db2009 is CRITICAL: CRITICAL: puppet fail [13:09:14] PROBLEM - puppet last run on es2008 is CRITICAL: CRITICAL: puppet fail [13:09:14] PROBLEM - puppet last run on ms-be2003 is CRITICAL: CRITICAL: Puppet has 30 failures [13:09:14] PROBLEM - puppet last run on ms-be2006 is CRITICAL: CRITICAL: puppet fail [13:09:15] PROBLEM - puppet last run on carbon is CRITICAL: CRITICAL: puppet fail [13:09:15] PROBLEM - puppet last run on es1001 is CRITICAL: CRITICAL: Puppet has 21 failures [13:09:16] PROBLEM - puppet last run on amslvs2 is CRITICAL: CRITICAL: Puppet has 16 failures [13:09:16] PROBLEM - puppet last run on mc1016 is CRITICAL: CRITICAL: Puppet has 13 failures [13:09:17] PROBLEM - puppet last run on cp3008 is CRITICAL: CRITICAL: Puppet has 30 failures [13:09:17] PROBLEM - puppet last run on amssq49 is CRITICAL: CRITICAL: puppet fail [13:09:18] PROBLEM - puppet last run on amssq43 is CRITICAL: CRITICAL: Puppet has 24 failures [13:09:18] PROBLEM - puppet last run on ms-fe1003 is CRITICAL: CRITICAL: Puppet has 34 failures [13:09:19] PROBLEM - puppet last run on mw1082 is CRITICAL: CRITICAL: puppet fail [13:09:19] PROBLEM - puppet last run on mw1244 is CRITICAL: CRITICAL: Puppet has 80 failures [13:09:20] PROBLEM - puppet last run on mw1013 is CRITICAL: CRITICAL: Puppet has 60 failures [13:09:24] PROBLEM - puppet last run on cp1049 is CRITICAL: CRITICAL: Puppet has 27 failures [13:09:25] PROBLEM - puppet last run on mw1141 is CRITICAL: CRITICAL: Puppet has 66 failures [13:09:25] PROBLEM - puppet last run on mw1063 is CRITICAL: CRITICAL: puppet fail [13:09:25] PROBLEM - puppet last run on potassium is CRITICAL: CRITICAL: Puppet has 19 failures [13:09:35] PROBLEM - puppet last run on analytics1035 is CRITICAL: CRITICAL: puppet fail [13:09:35] PROBLEM - puppet last run on db2034 is CRITICAL: CRITICAL: puppet fail [13:09:35] PROBLEM - puppet last run on ms-fe2004 is CRITICAL: CRITICAL: Puppet has 30 failures [13:09:35] PROBLEM - puppet last run on mc1006 is CRITICAL: CRITICAL: Puppet has 20 failures [13:09:35] PROBLEM - puppet last run on mc1002 is CRITICAL: CRITICAL: puppet fail [13:09:36] PROBLEM - puppet last run on mw1099 is CRITICAL: CRITICAL: puppet fail [13:09:36] PROBLEM - puppet last run on elastic1021 is CRITICAL: CRITICAL: puppet fail [13:09:37] PROBLEM - puppet last run on lead is CRITICAL: CRITICAL: puppet fail [13:09:37] PROBLEM - puppet last run on mw1117 is CRITICAL: CRITICAL: puppet fail [13:09:38] PROBLEM - puppet last run on mw1109 is CRITICAL: CRITICAL: Puppet has 67 failures [13:09:38] PROBLEM - puppet last run on neptunium is CRITICAL: CRITICAL: Puppet has 28 failures [13:09:39] PROBLEM - puppet last run on analytics1040 is CRITICAL: CRITICAL: puppet fail [13:09:39] PROBLEM - puppet last run on lvs4002 is CRITICAL: CRITICAL: Puppet has 22 failures [13:09:44] PROBLEM - puppet last run on db2005 is CRITICAL: CRITICAL: puppet fail [13:09:44] PROBLEM - puppet last run on mw1006 is CRITICAL: CRITICAL: puppet fail [13:09:44] PROBLEM - puppet last run on mw1089 is CRITICAL: CRITICAL: Puppet has 70 failures [13:09:45] PROBLEM - puppet last run on mw1205 is CRITICAL: CRITICAL: puppet fail [13:09:45] PROBLEM - puppet last run on mw1222 is CRITICAL: CRITICAL: puppet fail [13:09:55] PROBLEM - puppet last run on ms-be1003 is CRITICAL: CRITICAL: Puppet has 27 failures [13:09:55] PROBLEM - puppet last run on xenon is CRITICAL: CRITICAL: puppet fail [13:09:55] PROBLEM - puppet last run on mw1173 is CRITICAL: CRITICAL: Puppet has 64 failures [13:09:55] PROBLEM - puppet last run on mw1008 is CRITICAL: CRITICAL: puppet fail [13:09:55] PROBLEM - puppet last run on mw1069 is CRITICAL: CRITICAL: Puppet has 60 failures [13:09:56] PROBLEM - puppet last run on search1016 is CRITICAL: CRITICAL: Puppet has 39 failures [13:09:56] PROBLEM - puppet last run on wtp1006 is CRITICAL: CRITICAL: puppet fail [13:09:57] PROBLEM - puppet last run on mw1068 is CRITICAL: CRITICAL: puppet fail [13:10:05] PROBLEM - puppet last run on mw1224 is CRITICAL: CRITICAL: puppet fail [13:10:05] PROBLEM - puppet last run on mc1003 is CRITICAL: CRITICAL: Puppet has 15 failures [13:10:05] PROBLEM - puppet last run on mw1197 is CRITICAL: CRITICAL: Puppet has 72 failures [13:10:05] PROBLEM - puppet last run on elastic1004 is CRITICAL: CRITICAL: Puppet has 27 failures [13:10:13] PROBLEM - puppet last run on db1073 is CRITICAL: CRITICAL: Puppet has 17 failures [13:10:18] PROBLEM - puppet last run on lvs3001 is CRITICAL: CRITICAL: puppet fail [13:10:18] PROBLEM - puppet last run on amssq35 is CRITICAL: CRITICAL: puppet fail [13:10:18] PROBLEM - puppet last run on amssq32 is CRITICAL: CRITICAL: Puppet has 27 failures [13:10:19] PROBLEM - puppet last run on cp3016 is CRITICAL: CRITICAL: puppet fail [13:10:19] PROBLEM - puppet last run on mw1150 is CRITICAL: CRITICAL: Puppet has 67 failures [13:10:23] PROBLEM - puppet last run on elastic1007 is CRITICAL: CRITICAL: Puppet has 19 failures [13:10:23] PROBLEM - puppet last run on labstore1003 is CRITICAL: CRITICAL: Puppet has 13 failures [13:10:24] PROBLEM - puppet last run on search1010 is CRITICAL: CRITICAL: puppet fail [13:10:24] PROBLEM - puppet last run on mw1254 is CRITICAL: CRITICAL: Puppet has 77 failures [13:10:33] PROBLEM - puppet last run on mw1228 is CRITICAL: CRITICAL: Puppet has 71 failures [13:10:36] PROBLEM - puppet last run on mw1003 is CRITICAL: CRITICAL: Puppet has 55 failures [13:10:36] PROBLEM - puppet last run on virt1006 is CRITICAL: CRITICAL: puppet fail [13:10:36] PROBLEM - puppet last run on analytics1020 is CRITICAL: CRITICAL: Puppet has 23 failures [13:10:36] PROBLEM - puppet last run on db1031 is CRITICAL: CRITICAL: Puppet has 20 failures [13:10:36] PROBLEM - puppet last run on mw1120 is CRITICAL: CRITICAL: puppet fail [13:10:36] PROBLEM - puppet last run on lvs1002 is CRITICAL: CRITICAL: Puppet has 26 failures [13:10:37] PROBLEM - puppet last run on cp1055 is CRITICAL: CRITICAL: Puppet has 26 failures [13:10:37] PROBLEM - puppet last run on elastic1012 is CRITICAL: CRITICAL: puppet fail [13:10:38] PROBLEM - puppet last run on mw1174 is CRITICAL: CRITICAL: puppet fail [13:10:47] PROBLEM - puppet last run on mw1009 is CRITICAL: CRITICAL: puppet fail [13:10:47] PROBLEM - puppet last run on mw1046 is CRITICAL: CRITICAL: Puppet has 79 failures [13:10:47] PROBLEM - puppet last run on mw1166 is CRITICAL: CRITICAL: puppet fail [13:10:47] PROBLEM - puppet last run on db1050 is CRITICAL: CRITICAL: Puppet has 22 failures [13:10:47] PROBLEM - puppet last run on mw1217 is CRITICAL: CRITICAL: Puppet has 66 failures [13:10:48] PROBLEM - puppet last run on cp1061 is CRITICAL: CRITICAL: puppet fail [13:10:48] PROBLEM - puppet last run on mw1026 is CRITICAL: CRITICAL: Puppet has 75 failures [13:10:49] PROBLEM - puppet last run on rbf1002 is CRITICAL: CRITICAL: Puppet has 21 failures [13:10:49] PROBLEM - puppet last run on mw1153 is CRITICAL: CRITICAL: puppet fail [13:10:50] PROBLEM - puppet last run on mw1242 is CRITICAL: CRITICAL: Puppet has 77 failures [13:10:50] PROBLEM - puppet last run on snapshot1003 is CRITICAL: CRITICAL: Puppet has 59 failures [13:10:51] PROBLEM - puppet last run on mw1100 is CRITICAL: CRITICAL: Puppet has 66 failures [13:10:51] PROBLEM - puppet last run on db1066 is CRITICAL: CRITICAL: puppet fail [13:10:57] PROBLEM - puppet last run on mw1045 is CRITICAL: CRITICAL: Puppet has 75 failures [13:10:58] PROBLEM - puppet last run on labsdb1003 is CRITICAL: CRITICAL: Puppet has 20 failures [13:11:00] wat [13:11:03] PROBLEM - puppet last run on ms-fe1001 is CRITICAL: CRITICAL: Puppet has 26 failures [13:11:03] PROBLEM - puppet last run on mw1092 is CRITICAL: CRITICAL: puppet fail [13:11:03] PROBLEM - puppet last run on db1046 is CRITICAL: CRITICAL: puppet fail [13:11:13] PROBLEM - puppet last run on analytics1010 is CRITICAL: CRITICAL: puppet fail [13:11:15] PROBLEM - puppet last run on mw1088 is CRITICAL: CRITICAL: puppet fail [13:11:15] PROBLEM - puppet last run on db1015 is CRITICAL: CRITICAL: puppet fail [13:11:15] PROBLEM - puppet last run on wtp1016 is CRITICAL: CRITICAL: Puppet has 24 failures [13:11:16] PROBLEM - puppet last run on mw1118 is CRITICAL: CRITICAL: puppet fail [13:11:17] PROBLEM - puppet last run on mw1160 is CRITICAL: CRITICAL: Puppet has 76 failures [13:11:17] PROBLEM - puppet last run on mw1061 is CRITICAL: CRITICAL: Puppet has 62 failures [13:11:17] PROBLEM - puppet last run on db1002 is CRITICAL: CRITICAL: puppet fail [13:11:17] PROBLEM - puppet last run on mw1054 is CRITICAL: CRITICAL: puppet fail [13:11:17] PROBLEM - puppet last run on cp3003 is CRITICAL: CRITICAL: Puppet has 27 failures [13:11:18] PROBLEM - puppet last run on amssq61 is CRITICAL: CRITICAL: Puppet has 22 failures [13:11:33] PROBLEM - puppet last run on elastic1018 is CRITICAL: CRITICAL: Puppet has 21 failures [13:11:33] PROBLEM - puppet last run on mw1123 is CRITICAL: CRITICAL: Puppet has 63 failures [13:11:33] PROBLEM - puppet last run on mw1226 is CRITICAL: CRITICAL: puppet fail [13:11:33] PROBLEM - puppet last run on mw1189 is CRITICAL: CRITICAL: puppet fail [13:11:33] PROBLEM - puppet last run on analytics1030 is CRITICAL: CRITICAL: puppet fail [13:11:43] PROBLEM - puppet last run on mw1170 is CRITICAL: CRITICAL: puppet fail [13:11:44] PROBLEM - puppet last run on sca1002 is CRITICAL: CRITICAL: Puppet has 27 failures [13:11:44] PROBLEM - puppet last run on db1067 is CRITICAL: CRITICAL: puppet fail [13:11:45] PROBLEM - puppet last run on elastic1008 is CRITICAL: CRITICAL: Puppet has 31 failures [13:11:45] PROBLEM - puppet last run on es2001 is CRITICAL: CRITICAL: Puppet has 16 failures [13:11:45] PROBLEM - puppet last run on db2002 is CRITICAL: CRITICAL: Puppet has 22 failures [13:11:45] PROBLEM - puppet last run on db1018 is CRITICAL: CRITICAL: puppet fail [13:11:45] PROBLEM - puppet last run on analytics1025 is CRITICAL: CRITICAL: Puppet has 21 failures [13:11:46] PROBLEM - puppet last run on db2018 is CRITICAL: CRITICAL: Puppet has 22 failures [13:11:46] PROBLEM - puppet last run on cp4003 is CRITICAL: CRITICAL: puppet fail [13:11:54] PROBLEM - puppet last run on db1022 is CRITICAL: CRITICAL: puppet fail [13:11:54] PROBLEM - puppet last run on mw1208 is CRITICAL: CRITICAL: puppet fail [13:11:54] PROBLEM - puppet last run on amssq53 is CRITICAL: CRITICAL: Puppet has 31 failures [13:12:03] PROBLEM - puppet last run on lvs1005 is CRITICAL: CRITICAL: Puppet has 25 failures [13:12:03] PROBLEM - puppet last run on mw1149 is CRITICAL: CRITICAL: puppet fail [13:12:06] PROBLEM - puppet last run on mw1164 is CRITICAL: CRITICAL: Puppet has 65 failures [13:12:06] PROBLEM - puppet last run on lvs2001 is CRITICAL: CRITICAL: Puppet has 23 failures [13:12:06] PROBLEM - puppet last run on iron is CRITICAL: CRITICAL: Puppet has 25 failures [13:12:06] PROBLEM - puppet last run on gallium is CRITICAL: CRITICAL: Puppet has 39 failures [13:12:13] PROBLEM - puppet last run on elastic1022 is CRITICAL: CRITICAL: Puppet has 24 failures [13:12:14] PROBLEM - puppet last run on mw1176 is CRITICAL: CRITICAL: Puppet has 67 failures [13:12:14] PROBLEM - puppet last run on mw1052 is CRITICAL: CRITICAL: Puppet has 75 failures [13:12:14] PROBLEM - puppet last run on mw1042 is CRITICAL: CRITICAL: Puppet has 63 failures [13:12:14] PROBLEM - puppet last run on elastic1027 is CRITICAL: CRITICAL: puppet fail [13:12:14] PROBLEM - puppet last run on lvs2004 is CRITICAL: CRITICAL: Puppet has 24 failures [13:12:23] PROBLEM - puppet last run on db1042 is CRITICAL: CRITICAL: puppet fail [13:12:25] hmm [13:12:26] PROBLEM - puppet last run on mw1060 is CRITICAL: CRITICAL: Puppet has 73 failures [13:12:26] PROBLEM - puppet last run on search1001 is CRITICAL: CRITICAL: puppet fail [13:12:27] PROBLEM - puppet last run on ms-fe2001 is CRITICAL: CRITICAL: Puppet has 30 failures [13:12:27] puppetmaster failure [13:12:28] PROBLEM - puppet last run on ms-fe2003 is CRITICAL: CRITICAL: Puppet has 26 failures [13:12:29] PROBLEM - puppet last run on mw1065 is CRITICAL: CRITICAL: puppet fail [13:12:33] PROBLEM - puppet last run on cp1056 is CRITICAL: CRITICAL: puppet fail [13:12:43] PROBLEM - puppet last run on db1051 is CRITICAL: CRITICAL: Puppet has 28 failures [13:12:45] PROBLEM - puppet last run on search1007 is CRITICAL: CRITICAL: puppet fail [13:12:45] PROBLEM - puppet last run on mw1237 is CRITICAL: CRITICAL: puppet fail [13:12:45] PROBLEM - puppet last run on mw1126 is CRITICAL: CRITICAL: puppet fail [13:12:45] PROBLEM - puppet last run on antimony is CRITICAL: CRITICAL: Puppet has 30 failures [13:12:46] PROBLEM - puppet last run on mw1119 is CRITICAL: CRITICAL: puppet fail [13:12:46] PROBLEM - puppet last run on mw1177 is CRITICAL: CRITICAL: Puppet has 70 failures [13:12:46] PROBLEM - puppet last run on sodium is CRITICAL: CRITICAL: puppet fail [13:12:47] PROBLEM - puppet last run on ms-fe1004 is CRITICAL: CRITICAL: Puppet has 22 failures [13:12:53] PROBLEM - puppet last run on db1059 is CRITICAL: CRITICAL: Puppet has 24 failures [13:12:53] PROBLEM - puppet last run on mw1172 is CRITICAL: CRITICAL: puppet fail [13:12:53] PROBLEM - puppet last run on virt1001 is CRITICAL: CRITICAL: puppet fail [13:12:54] PROBLEM - puppet last run on wtp1005 is CRITICAL: CRITICAL: Puppet has 22 failures [13:12:54] PROBLEM - puppet last run on amslvs1 is CRITICAL: CRITICAL: puppet fail [13:12:54] PROBLEM - puppet last run on amssq48 is CRITICAL: CRITICAL: puppet fail [13:12:54] PROBLEM - puppet last run on ms-be3001 is CRITICAL: CRITICAL: puppet fail [13:12:55] PROBLEM - puppet last run on amssq47 is CRITICAL: CRITICAL: puppet fail [13:12:55] PROBLEM - puppet last run on amssq55 is CRITICAL: CRITICAL: Puppet has 28 failures [13:12:56] PROBLEM - puppet last run on cp3010 is CRITICAL: CRITICAL: puppet fail [13:12:56] PROBLEM - puppet last run on amssq60 is CRITICAL: CRITICAL: puppet fail [13:12:57] PROBLEM - puppet last run on cp3014 is CRITICAL: CRITICAL: Puppet has 24 failures [13:13:03] PROBLEM - puppet last run on db1040 is CRITICAL: CRITICAL: Puppet has 30 failures [13:13:03] PROBLEM - puppet last run on mw1114 is CRITICAL: CRITICAL: puppet fail [13:13:03] PROBLEM - puppet last run on labnet1001 is CRITICAL: CRITICAL: puppet fail [13:13:04] PROBLEM - puppet last run on mw1039 is CRITICAL: CRITICAL: puppet fail [13:13:15] PROBLEM - puppet last run on search1018 is CRITICAL: CRITICAL: Puppet has 53 failures [13:13:18] PROBLEM - puppet last run on polonium is CRITICAL: CRITICAL: Puppet has 24 failures [13:13:19] PROBLEM - puppet last run on tin is CRITICAL: CRITICAL: puppet fail [13:13:22] PROBLEM - puppet last run on searchidx1001 is CRITICAL: CRITICAL: Puppet has 59 failures [13:13:34] PROBLEM - puppet last run on mw1211 is CRITICAL: CRITICAL: puppet fail [13:13:35] PROBLEM - puppet last run on elastic1030 is CRITICAL: CRITICAL: puppet fail [13:13:35] PROBLEM - puppet last run on db1034 is CRITICAL: CRITICAL: Puppet has 16 failures [13:13:35] PROBLEM - puppet last run on db1028 is CRITICAL: CRITICAL: Puppet has 22 failures [13:13:35] PROBLEM - puppet last run on mc1014 is CRITICAL: CRITICAL: Puppet has 22 failures [13:13:35] PROBLEM - puppet last run on mw1195 is CRITICAL: CRITICAL: Puppet has 79 failures [13:13:36] PROBLEM - puppet last run on mw1251 is CRITICAL: CRITICAL: puppet fail [13:13:37] PROBLEM - puppet last run on db1026 is CRITICAL: CRITICAL: puppet fail [13:13:37] PROBLEM - puppet last run on gadolinium is CRITICAL: CRITICAL: puppet fail [13:13:37] PROBLEM - puppet last run on mw1025 is CRITICAL: CRITICAL: Puppet has 79 failures [13:13:38] PROBLEM - puppet last run on lithium is CRITICAL: CRITICAL: puppet fail [13:13:38] PROBLEM - puppet last run on mw1235 is CRITICAL: CRITICAL: Puppet has 66 failures [13:13:39] PROBLEM - puppet last run on snapshot1001 is CRITICAL: CRITICAL: puppet fail [13:13:39] PROBLEM - puppet last run on mw1162 is CRITICAL: CRITICAL: Puppet has 73 failures [13:13:45] PROBLEM - puppet last run on wtp1012 is CRITICAL: CRITICAL: Puppet has 26 failures [13:13:46] PROBLEM - puppet last run on mw1168 is CRITICAL: CRITICAL: puppet fail [13:13:46] PROBLEM - puppet last run on mw1206 is CRITICAL: CRITICAL: puppet fail [13:13:46] PROBLEM - puppet last run on db1016 is CRITICAL: CRITICAL: Puppet has 17 failures [13:13:46] PROBLEM - puppet last run on dataset1001 is CRITICAL: CRITICAL: Puppet has 28 failures [13:13:46] PROBLEM - puppet last run on virt1004 is CRITICAL: CRITICAL: puppet fail [13:13:47] PROBLEM - puppet last run on mw1227 is CRITICAL: CRITICAL: Puppet has 78 failures [13:13:47] PROBLEM - puppet last run on mw1190 is CRITICAL: CRITICAL: puppet fail [13:13:58] PROBLEM - puppet last run on mw1002 is CRITICAL: CRITICAL: Puppet has 61 failures [13:13:58] PROBLEM - puppet last run on logstash1002 is CRITICAL: CRITICAL: Puppet has 37 failures [13:13:58] PROBLEM - puppet last run on labstore1001 is CRITICAL: CRITICAL: Puppet has 16 failures [13:13:58] PROBLEM - puppet last run on analytics1016 is CRITICAL: CRITICAL: puppet fail [13:14:00] PROBLEM - puppet last run on lvs2006 is CRITICAL: CRITICAL: puppet fail [13:14:05] PROBLEM - puppet last run on cp4008 is CRITICAL: CRITICAL: Puppet has 33 failures [13:14:07] PROBLEM - puppet last run on cp4014 is CRITICAL: CRITICAL: Puppet has 27 failures [13:14:08] PROBLEM - puppet last run on cp4004 is CRITICAL: CRITICAL: Puppet has 25 failures [13:14:14] PROBLEM - puppet last run on mc1012 is CRITICAL: CRITICAL: puppet fail [13:14:17] PROBLEM - puppet last run on ssl1005 is CRITICAL: CRITICAL: Puppet has 21 failures [13:14:18] PROBLEM - puppet last run on db2038 is CRITICAL: CRITICAL: Puppet has 21 failures [13:14:18] PROBLEM - puppet last run on db2036 is CRITICAL: CRITICAL: Puppet has 17 failures [13:14:18] PROBLEM - puppet last run on install2001 is CRITICAL: CRITICAL: Puppet has 23 failures [13:14:18] PROBLEM - puppet last run on ssl1009 is CRITICAL: CRITICAL: puppet fail [13:14:18] PROBLEM - puppet last run on mw1011 is CRITICAL: CRITICAL: Puppet has 54 failures [13:14:19] PROBLEM - puppet last run on ruthenium is CRITICAL: CRITICAL: Puppet has 22 failures [13:14:19] PROBLEM - puppet last run on amssq56 is CRITICAL: CRITICAL: puppet fail [13:14:20] PROBLEM - puppet last run on ms-be3002 is CRITICAL: CRITICAL: Puppet has 22 failures [13:14:20] PROBLEM - puppet last run on cp1058 is CRITICAL: CRITICAL: puppet fail [13:14:21] PROBLEM - puppet last run on mw1165 is CRITICAL: CRITICAL: puppet fail [13:14:21] PROBLEM - puppet last run on dbproxy1001 is CRITICAL: CRITICAL: Puppet has 20 failures [13:14:22] PROBLEM - puppet last run on db1004 is CRITICAL: CRITICAL: puppet fail [13:14:22] PROBLEM - puppet last run on mw1144 is CRITICAL: CRITICAL: Puppet has 74 failures [13:14:23] PROBLEM - puppet last run on db1043 is CRITICAL: CRITICAL: puppet fail [13:14:30] PROBLEM - puppet last run on db1003 is CRITICAL: CRITICAL: Puppet has 22 failures [13:14:30] PROBLEM - puppet last run on elastic1024 is CRITICAL: CRITICAL: Puppet has 21 failures [13:14:30] PROBLEM - puppet last run on analytics1038 is CRITICAL: CRITICAL: puppet fail [13:14:30] PROBLEM - puppet last run on db2037 is CRITICAL: CRITICAL: puppet fail [13:14:30] PROBLEM - puppet last run on mw1247 is CRITICAL: CRITICAL: puppet fail [13:14:30] PROBLEM - puppet last run on db1023 is CRITICAL: CRITICAL: Puppet has 27 failures [13:14:31] PROBLEM - puppet last run on mw1238 is CRITICAL: CRITICAL: Puppet has 75 failures [13:14:31] PROBLEM - puppet last run on db1048 is CRITICAL: CRITICAL: puppet fail [13:14:32] PROBLEM - puppet last run on mw1034 is CRITICAL: CRITICAL: puppet fail [13:14:32] PROBLEM - puppet last run on db2004 is CRITICAL: CRITICAL: puppet fail [13:14:33] PROBLEM - puppet last run on db2023 is CRITICAL: CRITICAL: puppet fail [13:14:33] PROBLEM - puppet last run on labcontrol2001 is CRITICAL: CRITICAL: Puppet has 34 failures [13:14:34] PROBLEM - puppet last run on db2016 is CRITICAL: CRITICAL: Puppet has 17 failures [13:14:34] PROBLEM - puppet last run on lvs4003 is CRITICAL: CRITICAL: Puppet has 24 failures [13:14:35] PROBLEM - puppet last run on mw1050 is CRITICAL: CRITICAL: puppet fail [13:14:35] PROBLEM - puppet last run on db2007 is CRITICAL: CRITICAL: Puppet has 24 failures [13:14:36] PROBLEM - puppet last run on wtp1018 is CRITICAL: CRITICAL: puppet fail [13:14:36] PROBLEM - puppet last run on cp4019 is CRITICAL: CRITICAL: puppet fail [13:14:37] PROBLEM - puppet last run on db2029 is CRITICAL: CRITICAL: puppet fail [13:14:37] PROBLEM - puppet last run on cp4001 is CRITICAL: CRITICAL: puppet fail [13:14:38] PROBLEM - puppet last run on elastic1006 is CRITICAL: CRITICAL: puppet fail [13:14:38] PROBLEM - puppet last run on bast4001 is CRITICAL: CRITICAL: puppet fail [13:14:39] PROBLEM - puppet last run on mw1180 is CRITICAL: CRITICAL: puppet fail [13:14:49] PROBLEM - puppet last run on amssq34 is CRITICAL: CRITICAL: Puppet has 25 failures [13:14:49] PROBLEM - puppet last run on mw1213 is CRITICAL: CRITICAL: Puppet has 75 failures [13:14:49] PROBLEM - puppet last run on amssq51 is CRITICAL: CRITICAL: Puppet has 31 failures [13:14:50] PROBLEM - puppet last run on mw1055 is CRITICAL: CRITICAL: puppet fail [13:14:50] PROBLEM - puppet last run on amssq42 is CRITICAL: CRITICAL: puppet fail [13:14:50] PROBLEM - puppet last run on virt1003 is CRITICAL: CRITICAL: puppet fail [13:14:50] PROBLEM - puppet last run on mw1159 is CRITICAL: CRITICAL: puppet fail [13:14:51] PROBLEM - puppet last run on db1021 is CRITICAL: CRITICAL: Puppet has 23 failures [13:14:51] PROBLEM - puppet last run on mw1171 is CRITICAL: CRITICAL: puppet fail [13:14:52] PROBLEM - puppet last run on snapshot1002 is CRITICAL: CRITICAL: puppet fail [13:14:52] PROBLEM - puppet last run on labsdb1006 is CRITICAL: CRITICAL: Puppet has 26 failures [13:15:03] PROBLEM - puppet last run on amssq46 is CRITICAL: CRITICAL: Puppet has 26 failures [13:15:03] PROBLEM - puppet last run on mw1076 is CRITICAL: CRITICAL: Puppet has 64 failures [13:15:03] PROBLEM - puppet last run on elastic1019 is CRITICAL: CRITICAL: puppet fail [13:15:03] PROBLEM - puppet last run on mw1129 is CRITICAL: CRITICAL: Puppet has 73 failures [13:15:10] PROBLEM - puppet last run on osmium is CRITICAL: CRITICAL: puppet fail [13:15:10] PROBLEM - puppet last run on mw1249 is CRITICAL: CRITICAL: Puppet has 76 failures [13:15:10] PROBLEM - puppet last run on mw1175 is CRITICAL: CRITICAL: Puppet has 73 failures [13:15:10] PROBLEM - puppet last run on mw1051 is CRITICAL: CRITICAL: Puppet has 64 failures [13:15:10] PROBLEM - puppet last run on cp1048 is CRITICAL: CRITICAL: puppet fail [13:15:11] PROBLEM - puppet last run on cp1050 is CRITICAL: CRITICAL: Puppet has 22 failures [13:15:11] PROBLEM - puppet last run on mw1044 is CRITICAL: CRITICAL: puppet fail [13:15:12] PROBLEM - puppet last run on db1060 is CRITICAL: CRITICAL: Puppet has 24 failures [13:15:12] PROBLEM - puppet last run on db1039 is CRITICAL: CRITICAL: puppet fail [13:15:13] PROBLEM - puppet last run on ms-be2011 is CRITICAL: CRITICAL: Puppet has 27 failures [13:15:25] PROBLEM - puppet last run on cp4018 is CRITICAL: CRITICAL: Puppet has 27 failures [13:15:25] PROBLEM - puppet last run on cp4005 is CRITICAL: CRITICAL: Puppet has 24 failures [13:15:25] PROBLEM - puppet last run on virt1007 is CRITICAL: CRITICAL: puppet fail [13:15:26] PROBLEM - puppet last run on lvs3004 is CRITICAL: CRITICAL: Puppet has 26 failures [13:15:26] PROBLEM - puppet last run on pc1002 is CRITICAL: CRITICAL: Puppet has 36 failures [13:15:27] PROBLEM - puppet last run on mc1005 is CRITICAL: CRITICAL: Puppet has 22 failures [13:15:27] PROBLEM - puppet last run on berkelium is CRITICAL: CRITICAL: puppet fail [13:15:27] PROBLEM - puppet last run on mw1023 is CRITICAL: CRITICAL: puppet fail [13:15:32] PROBLEM - puppet last run on db1052 is CRITICAL: CRITICAL: Puppet has 25 failures [13:15:32] PROBLEM - puppet last run on es1007 is CRITICAL: CRITICAL: Puppet has 27 failures [13:15:41] PROBLEM - puppet last run on mw1151 is CRITICAL: CRITICAL: Puppet has 67 failures [13:15:41] PROBLEM - puppet last run on plutonium is CRITICAL: CRITICAL: Puppet has 27 failures [13:15:44] PROBLEM - puppet last run on search1002 is CRITICAL: CRITICAL: Puppet has 60 failures [13:15:44] PROBLEM - puppet last run on ms-be1012 is CRITICAL: CRITICAL: puppet fail [13:15:44] PROBLEM - puppet last run on search1005 is CRITICAL: CRITICAL: puppet fail [13:15:52] PROBLEM - puppet last run on mw1079 is CRITICAL: CRITICAL: Puppet has 67 failures [13:15:57] PROBLEM - puppet last run on amssq62 is CRITICAL: CRITICAL: puppet fail [13:15:58] PROBLEM - puppet last run on amssq40 is CRITICAL: CRITICAL: Puppet has 21 failures [13:15:58] PROBLEM - puppet last run on db1036 is CRITICAL: CRITICAL: Puppet has 16 failures [13:15:58] PROBLEM - puppet last run on amssq36 is CRITICAL: CRITICAL: Puppet has 26 failures [13:15:58] PROBLEM - puppet last run on ms-fe3002 is CRITICAL: CRITICAL: Puppet has 27 failures [13:15:58] PROBLEM - puppet last run on mw1057 is CRITICAL: CRITICAL: puppet fail [13:15:59] PROBLEM - puppet last run on cp1046 is CRITICAL: CRITICAL: Puppet has 28 failures [13:15:59] PROBLEM - puppet last run on argon is CRITICAL: CRITICAL: puppet fail [13:16:00] PROBLEM - puppet last run on mw1248 is CRITICAL: CRITICAL: puppet fail [13:16:00] PROBLEM - puppet last run on labmon1001 is CRITICAL: CRITICAL: puppet fail [13:16:01] PROBLEM - puppet last run on ms-be1008 is CRITICAL: CRITICAL: puppet fail [13:16:01] PROBLEM - puppet last run on db1062 is CRITICAL: CRITICAL: puppet fail [13:16:02] PROBLEM - puppet last run on mw1030 is CRITICAL: CRITICAL: Puppet has 83 failures [13:16:02] PROBLEM - puppet last run on mw1098 is CRITICAL: CRITICAL: Puppet has 66 failures [13:16:11] PROBLEM - puppet last run on mw1084 is CRITICAL: CRITICAL: Puppet has 70 failures [13:16:12] PROBLEM - puppet last run on rdb1001 is CRITICAL: CRITICAL: Puppet has 20 failures [13:16:14] PROBLEM - puppet last run on analytics1037 is CRITICAL: CRITICAL: puppet fail [13:16:14] PROBLEM - puppet last run on analytics1022 is CRITICAL: CRITICAL: Puppet has 26 failures [13:16:14] PROBLEM - puppet last run on stat1002 is CRITICAL: CRITICAL: Puppet has 56 failures [13:16:14] PROBLEM - puppet last run on oxygen is CRITICAL: CRITICAL: puppet fail [13:16:14] PROBLEM - puppet last run on mw1049 is CRITICAL: CRITICAL: Puppet has 70 failures [13:16:15] PROBLEM - puppet last run on analytics1026 is CRITICAL: CRITICAL: Puppet has 28 failures [13:16:22] PROBLEM - puppet last run on mw1212 is CRITICAL: CRITICAL: puppet fail [13:16:35] PROBLEM - puppet last run on mw1133 is CRITICAL: CRITICAL: Puppet has 65 failures [13:16:35] PROBLEM - puppet last run on mc1007 is CRITICAL: CRITICAL: puppet fail [13:16:36] PROBLEM - puppet last run on mw1202 is CRITICAL: CRITICAL: Puppet has 65 failures [13:16:36] PROBLEM - puppet last run on mw1111 is CRITICAL: CRITICAL: Puppet has 84 failures [13:16:36] PROBLEM - puppet last run on wtp1023 is CRITICAL: CRITICAL: puppet fail [13:16:36] PROBLEM - puppet last run on mw1116 is CRITICAL: CRITICAL: puppet fail [13:16:43] PROBLEM - puppet last run on mw1029 is CRITICAL: CRITICAL: puppet fail [13:16:51] PROBLEM - puppet last run on mc1001 is CRITICAL: CRITICAL: puppet fail [13:16:51] PROBLEM - puppet last run on db1071 is CRITICAL: CRITICAL: puppet fail [13:16:51] PROBLEM - puppet last run on mw1014 is CRITICAL: CRITICAL: Puppet has 57 failures [13:16:51] PROBLEM - puppet last run on search1023 is CRITICAL: CRITICAL: puppet fail [13:16:52] PROBLEM - puppet last run on mw1125 is CRITICAL: CRITICAL: Puppet has 68 failures [13:17:00] PROBLEM - puppet last run on mw1148 is CRITICAL: CRITICAL: puppet fail [13:17:02] PROBLEM - puppet last run on ms-be2001 is CRITICAL: CRITICAL: puppet fail [13:17:02] PROBLEM - puppet last run on mw1081 is CRITICAL: CRITICAL: puppet fail [13:17:02] PROBLEM - puppet last run on db1054 is CRITICAL: CRITICAL: puppet fail [13:17:03] PROBLEM - puppet last run on hafnium is CRITICAL: CRITICAL: puppet fail [13:17:03] PROBLEM - puppet last run on mw1243 is CRITICAL: CRITICAL: puppet fail [13:17:03] PROBLEM - puppet last run on mw1033 is CRITICAL: CRITICAL: puppet fail [13:17:03] PROBLEM - puppet last run on search1017 is CRITICAL: CRITICAL: Puppet has 50 failures [13:17:14] PROBLEM - puppet last run on mw1183 is CRITICAL: CRITICAL: puppet fail [13:17:16] PROBLEM - puppet last run on mw1004 is CRITICAL: CRITICAL: puppet fail [13:17:21] PROBLEM - puppet last run on analytics1013 is CRITICAL: CRITICAL: Puppet has 19 failures [13:17:31] PROBLEM - puppet last run on mw1146 is CRITICAL: CRITICAL: Puppet has 79 failures [13:17:32] PROBLEM - puppet last run on analytics1014 is CRITICAL: CRITICAL: puppet fail [13:17:32] PROBLEM - puppet last run on db1001 is CRITICAL: CRITICAL: puppet fail [13:17:32] PROBLEM - puppet last run on analytics1011 is CRITICAL: CRITICAL: puppet fail [13:17:32] PROBLEM - puppet last run on wtp1011 is CRITICAL: CRITICAL: puppet fail [13:17:32] PROBLEM - puppet last run on elastic1015 is CRITICAL: CRITICAL: puppet fail [13:17:32] PROBLEM - puppet last run on elastic1014 is CRITICAL: CRITICAL: puppet fail [13:17:33] PROBLEM - puppet last run on db2003 is CRITICAL: CRITICAL: puppet fail [13:17:41] PROBLEM - puppet last run on mw1105 is CRITICAL: CRITICAL: puppet fail [13:17:44] PROBLEM - puppet last run on mw1024 is CRITICAL: CRITICAL: puppet fail [13:17:44] PROBLEM - puppet last run on mw1181 is CRITICAL: CRITICAL: Puppet has 72 failures [13:17:44] PROBLEM - puppet last run on rubidium is CRITICAL: CRITICAL: Puppet has 26 failures [13:17:50] PROBLEM - puppet last run on mw1001 is CRITICAL: CRITICAL: puppet fail [13:18:00] PROBLEM - puppet last run on labsdb1004 is CRITICAL: CRITICAL: puppet fail [13:18:02] PROBLEM - puppet last run on db1057 is CRITICAL: CRITICAL: puppet fail [13:18:03] PROBLEM - puppet last run on db1055 is CRITICAL: CRITICAL: puppet fail [13:18:03] PROBLEM - puppet last run on wtp1004 is CRITICAL: CRITICAL: Puppet has 27 failures [13:18:03] PROBLEM - puppet last run on protactinium is CRITICAL: CRITICAL: Puppet has 17 failures [13:18:16] PROBLEM - puppet last run on ssl1007 is CRITICAL: CRITICAL: Puppet has 18 failures [13:18:17] PROBLEM - puppet last run on wtp1013 is CRITICAL: CRITICAL: Puppet has 19 failures [13:18:17] PROBLEM - puppet last run on wtp1015 is CRITICAL: CRITICAL: puppet fail [13:18:17] PROBLEM - puppet last run on elastic1017 is CRITICAL: CRITICAL: puppet fail [13:18:17] PROBLEM - puppet last run on ms-be2007 is CRITICAL: CRITICAL: puppet fail [13:18:18] PROBLEM - puppet last run on db1063 is CRITICAL: CRITICAL: Puppet has 21 failures [13:18:18] PROBLEM - puppet last run on mc1013 is CRITICAL: CRITICAL: puppet fail [13:18:18] PROBLEM - puppet last run on mw1225 is CRITICAL: CRITICAL: puppet fail [13:18:19] PROBLEM - puppet last run on mw1210 is CRITICAL: CRITICAL: Puppet has 63 failures [13:18:19] PROBLEM - puppet last run on mw1167 is CRITICAL: CRITICAL: puppet fail [13:18:20] PROBLEM - puppet last run on cp1060 is CRITICAL: CRITICAL: puppet fail [13:18:21] PROBLEM - puppet last run on wtp1022 is CRITICAL: CRITICAL: puppet fail [13:18:21] PROBLEM - puppet last run on thallium is CRITICAL: CRITICAL: Puppet has 15 failures [13:18:23] PROBLEM - puppet last run on ssl1008 is CRITICAL: CRITICAL: Puppet has 24 failures [13:18:23] PROBLEM - puppet last run on search1024 is CRITICAL: CRITICAL: Puppet has 66 failures [13:18:23] PROBLEM - puppet last run on db1038 is CRITICAL: CRITICAL: puppet fail [13:18:23] PROBLEM - puppet last run on ocg1003 is CRITICAL: CRITICAL: puppet fail [13:18:32] PROBLEM - puppet last run on rhenium is CRITICAL: CRITICAL: Puppet has 24 failures [13:18:32] RECOVERY - puppet last run on elastic1009 is OK: OK: Puppet is currently enabled, last run 5 seconds ago with 0 failures [13:18:32] PROBLEM - puppet last run on elastic1023 is CRITICAL: CRITICAL: puppet fail [13:18:32] PROBLEM - puppet last run on db1020 is CRITICAL: CRITICAL: Puppet has 29 failures [13:18:32] PROBLEM - puppet last run on rcs1001 is CRITICAL: CRITICAL: puppet fail [13:18:33] PROBLEM - puppet last run on elastic1002 is CRITICAL: CRITICAL: Puppet has 24 failures [13:18:33] PROBLEM - puppet last run on rdb1003 is CRITICAL: CRITICAL: Puppet has 24 failures [13:18:34] PROBLEM - puppet last run on amslvs3 is CRITICAL: CRITICAL: puppet fail [13:18:40] PROBLEM - puppet last run on search1011 is CRITICAL: CRITICAL: puppet fail [13:18:40] PROBLEM - puppet last run on mw1121 is CRITICAL: CRITICAL: puppet fail [13:18:41] PROBLEM - puppet last run on cp1062 is CRITICAL: CRITICAL: Puppet has 25 failures [13:18:41] PROBLEM - puppet last run on cp1038 is CRITICAL: CRITICAL: Puppet has 30 failures [13:18:41] PROBLEM - puppet last run on rdb1002 is CRITICAL: CRITICAL: Puppet has 24 failures [13:18:52] PROBLEM - puppet last run on mw1043 is CRITICAL: CRITICAL: puppet fail [13:18:59] PROBLEM - puppet last run on mw1239 is CRITICAL: CRITICAL: puppet fail [13:18:59] PROBLEM - puppet last run on search1022 is CRITICAL: CRITICAL: puppet fail [13:18:59] PROBLEM - puppet last run on mw1056 is CRITICAL: CRITICAL: Puppet has 68 failures [13:18:59] PROBLEM - puppet last run on search1013 is CRITICAL: CRITICAL: Puppet has 51 failures [13:18:59] PROBLEM - puppet last run on elastic1005 is CRITICAL: CRITICAL: puppet fail [13:19:00] PROBLEM - puppet last run on db1064 is CRITICAL: CRITICAL: Puppet has 17 failures [13:19:00] PROBLEM - puppet last run on mw1016 is CRITICAL: CRITICAL: puppet fail [13:19:01] RECOVERY - puppet last run on ms-be1005 is OK: OK: Puppet is currently enabled, last run 9 seconds ago with 0 failures [13:19:01] PROBLEM - puppet last run on mw1010 is CRITICAL: CRITICAL: puppet fail [13:19:02] PROBLEM - puppet last run on db1030 is CRITICAL: CRITICAL: Puppet has 23 failures [13:19:02] PROBLEM - puppet last run on db1033 is CRITICAL: CRITICAL: puppet fail [13:19:03] PROBLEM - puppet last run on mw1032 is CRITICAL: CRITICAL: Puppet has 76 failures [13:19:03] PROBLEM - puppet last run on mw1258 is CRITICAL: CRITICAL: Puppet has 75 failures [13:19:12] PROBLEM - puppet last run on mw1198 is CRITICAL: CRITICAL: Puppet has 74 failures [13:19:13] PROBLEM - puppet last run on db1070 is CRITICAL: CRITICAL: Puppet has 24 failures [13:19:13] RECOVERY - puppet last run on mc1008 is OK: OK: Puppet is currently enabled, last run 3 seconds ago with 0 failures [13:19:14] PROBLEM - puppet last run on mw1077 is CRITICAL: CRITICAL: puppet fail [13:19:14] RECOVERY - puppet last run on analytics1034 is OK: OK: Puppet is currently enabled, last run 2 seconds ago with 0 failures [13:19:20] PROBLEM - puppet last run on cp1063 is CRITICAL: CRITICAL: Puppet has 30 failures [13:19:20] PROBLEM - puppet last run on rbf1001 is CRITICAL: CRITICAL: puppet fail [13:19:20] PROBLEM - puppet last run on db1069 is CRITICAL: CRITICAL: Puppet has 20 failures [13:19:20] PROBLEM - puppet last run on db1006 is CRITICAL: CRITICAL: puppet fail [13:19:20] RECOVERY - puppet last run on elastic1013 is OK: OK: Puppet is currently enabled, last run 11 seconds ago with 0 failures [13:19:21] PROBLEM - puppet last run on wtp1003 is CRITICAL: CRITICAL: puppet fail [13:19:21] PROBLEM - puppet last run on rcs1002 is CRITICAL: CRITICAL: Puppet has 21 failures [13:19:22] PROBLEM - puppet last run on mw1223 is CRITICAL: CRITICAL: puppet fail [13:19:22] PROBLEM - puppet last run on pc1003 is CRITICAL: CRITICAL: Puppet has 28 failures [13:19:23] RECOVERY - puppet last run on mw1184 is OK: OK: Puppet is currently enabled, last run 27 seconds ago with 0 failures [13:19:30] PROBLEM - puppet last run on mw1241 is CRITICAL: CRITICAL: puppet fail [13:19:30] PROBLEM - puppet last run on ms-be1009 is CRITICAL: CRITICAL: Puppet has 34 failures [13:19:30] PROBLEM - puppet last run on ms-be2008 is CRITICAL: CRITICAL: Puppet has 30 failures [13:19:30] PROBLEM - puppet last run on mw1163 is CRITICAL: CRITICAL: Puppet has 71 failures [13:19:30] PROBLEM - puppet last run on bast1001 is CRITICAL: CRITICAL: puppet fail [13:19:30] !log restarted apache on palladium, things are recovering [13:19:31] RECOVERY - puppet last run on mw1182 is OK: OK: Puppet is currently enabled, last run 34 seconds ago with 0 failures [13:19:31] PROBLEM - puppet last run on mw1139 is CRITICAL: CRITICAL: Puppet has 83 failures [13:19:32] PROBLEM - puppet last run on lvs2003 is CRITICAL: CRITICAL: Puppet has 15 failures [13:19:32] RECOVERY - puppet last run on mw1157 is OK: OK: Puppet is currently enabled, last run 2 seconds ago with 0 failures [13:19:33] RECOVERY - puppet last run on mw1070 is OK: OK: Puppet is currently enabled, last run 28 seconds ago with 0 failures [13:19:33] PROBLEM - puppet last run on db1044 is CRITICAL: CRITICAL: puppet fail [13:19:34] PROBLEM - puppet last run on analytics1027 is CRITICAL: CRITICAL: puppet fail [13:19:34] PROBLEM - puppet last run on cp1067 is CRITICAL: CRITICAL: puppet fail [13:19:34] Logged the message, Master [13:19:41] PROBLEM - puppet last run on db2030 is CRITICAL: CRITICAL: puppet fail [13:19:43] PROBLEM - puppet last run on search1006 is CRITICAL: CRITICAL: puppet fail [13:19:43] PROBLEM - puppet last run on db2001 is CRITICAL: CRITICAL: Puppet has 29 failures [13:19:43] RECOVERY - puppet last run on lvs3002 is OK: OK: Puppet is currently enabled, last run 34 seconds ago with 0 failures [13:19:44] PROBLEM - puppet last run on cp3004 is CRITICAL: CRITICAL: Puppet has 32 failures [13:19:44] PROBLEM - puppet last run on mw1156 is CRITICAL: CRITICAL: Puppet has 72 failures [13:19:44] PROBLEM - puppet last run on mw1074 is CRITICAL: CRITICAL: Puppet has 66 failures [13:19:44] PROBLEM - puppet last run on es1002 is CRITICAL: CRITICAL: Puppet has 20 failures [13:19:45] PROBLEM - puppet last run on titanium is CRITICAL: CRITICAL: Puppet has 18 failures [13:19:52] PROBLEM - puppet last run on wtp1007 is CRITICAL: CRITICAL: puppet fail [13:19:52] PROBLEM - puppet last run on mw1188 is CRITICAL: CRITICAL: Puppet has 70 failures [13:19:52] PROBLEM - puppet last run on terbium is CRITICAL: CRITICAL: puppet fail [13:19:52] PROBLEM - puppet last run on nitrogen is CRITICAL: CRITICAL: Puppet has 21 failures [13:20:00] PROBLEM - puppet last run on mw1107 is CRITICAL: CRITICAL: puppet fail [13:20:00] RECOVERY - puppet last run on mw1127 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [13:20:00] RECOVERY - puppet last run on mw1036 is OK: OK: Puppet is currently enabled, last run 23 seconds ago with 0 failures [13:20:00] RECOVERY - puppetmaster backend https on palladium is OK: HTTP OK: Status line output matched 400 - 335 bytes in 0.088 second response time [13:20:00] PROBLEM - puppet last run on mw1122 is CRITICAL: CRITICAL: Puppet has 67 failures [13:20:01] PROBLEM - puppet last run on tungsten is CRITICAL: CRITICAL: Puppet has 26 failures [13:20:11] RECOVERY - puppet last run on db1041 is OK: OK: Puppet is currently enabled, last run 52 seconds ago with 0 failures [13:20:11] PROBLEM - puppet last run on mw1091 is CRITICAL: CRITICAL: puppet fail [13:20:12] PROBLEM - puppet last run on ms-fe2002 is CRITICAL: CRITICAL: Puppet has 29 failures [13:20:12] PROBLEM - puppet last run on db2012 is CRITICAL: CRITICAL: Puppet has 22 failures [13:20:12] PROBLEM - puppet last run on db2028 is CRITICAL: CRITICAL: Puppet has 19 failures [13:20:14] PROBLEM - puppet last run on mw1215 is CRITICAL: CRITICAL: Puppet has 64 failures [13:20:18] PROBLEM - puppet last run on acamar is CRITICAL: CRITICAL: Puppet has 26 failures [13:20:18] RECOVERY - puppet last run on lvs4001 is OK: OK: Puppet is currently enabled, last run 49 seconds ago with 0 failures [13:20:18] PROBLEM - puppet last run on mw1201 is CRITICAL: CRITICAL: Puppet has 60 failures [13:20:24] PROBLEM - puppet last run on stat1003 is CRITICAL: CRITICAL: Puppet has 54 failures [13:20:31] PROBLEM - puppet last run on mw1220 is CRITICAL: CRITICAL: puppet fail [13:20:32] PROBLEM - puppet last run on analytics1023 is CRITICAL: CRITICAL: Puppet has 19 failures [13:20:32] PROBLEM - puppet last run on mw1203 is CRITICAL: CRITICAL: puppet fail [13:20:32] PROBLEM - puppet last run on ms-be1002 is CRITICAL: CRITICAL: puppet fail [13:20:32] PROBLEM - puppet last run on mw1053 is CRITICAL: CRITICAL: Puppet has 75 failures [13:20:32] PROBLEM - puppet last run on search1015 is CRITICAL: CRITICAL: Puppet has 47 failures [13:20:33] PROBLEM - puppet last run on snapshot1004 is CRITICAL: CRITICAL: Puppet has 57 failures [13:20:33] RECOVERY - puppet last run on mw1252 is OK: OK: Puppet is currently enabled, last run 22 seconds ago with 0 failures [13:20:34] PROBLEM - puppet last run on lvs3003 is CRITICAL: CRITICAL: puppet fail [13:20:34] PROBLEM - puppet last run on hooft is CRITICAL: CRITICAL: Puppet has 57 failures [13:20:35] PROBLEM - puppet last run on lvs1001 is CRITICAL: CRITICAL: Puppet has 18 failures [13:20:35] PROBLEM - puppet last run on db1061 is CRITICAL: CRITICAL: Puppet has 24 failures [13:20:36] RECOVERY - puppet last run on mw1196 is OK: OK: Puppet is currently enabled, last run 1 second ago with 0 failures [13:20:36] RECOVERY - puppet last run on mw1013 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [13:20:37] PROBLEM - puppet last run on mw1185 is CRITICAL: CRITICAL: Puppet has 67 failures [13:20:40] PROBLEM - puppet last run on wtp1002 is CRITICAL: CRITICAL: Puppet has 28 failures [13:20:43] PROBLEM - puppet last run on fluorine is CRITICAL: CRITICAL: Puppet has 61 failures [13:20:43] PROBLEM - puppet last run on ms-be2012 is CRITICAL: CRITICAL: Puppet has 26 failures [13:20:43] PROBLEM - puppet last run on ssl3002 is CRITICAL: CRITICAL: Puppet has 17 failures [13:20:43] PROBLEM - puppet last run on mw1087 is CRITICAL: CRITICAL: Puppet has 65 failures [13:20:44] PROBLEM - puppet last run on iodine is CRITICAL: CRITICAL: Puppet has 31 failures [13:20:44] PROBLEM - puppet last run on analytics1028 is CRITICAL: CRITICAL: Puppet has 22 failures [13:20:44] PROBLEM - puppet last run on mw1152 is CRITICAL: CRITICAL: Puppet has 65 failures [13:20:45] RECOVERY - puppet last run on analytics1039 is OK: OK: Puppet is currently enabled, last run 41 seconds ago with 0 failures [13:20:45] RECOVERY - puppet last run on cp1051 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [13:20:46] PROBLEM - puppet last run on mercury is CRITICAL: CRITICAL: Puppet has 21 failures [13:20:51] PROBLEM - puppet last run on mw1231 is CRITICAL: CRITICAL: puppet fail [13:20:52] RECOVERY - puppet last run on mw1015 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [13:20:52] PROBLEM - puppet last run on ms-be1007 is CRITICAL: CRITICAL: Puppet has 34 failures [13:20:52] RECOVERY - puppet last run on search1014 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [13:20:52] PROBLEM - puppet last run on mw1112 is CRITICAL: CRITICAL: puppet fail [13:20:52] PROBLEM - puppet last run on mw1022 is CRITICAL: CRITICAL: Puppet has 80 failures [13:21:00] PROBLEM - puppet last run on ms-be2005 is CRITICAL: CRITICAL: Puppet has 34 failures [13:21:09] PROBLEM - puppet last run on search1004 is CRITICAL: CRITICAL: Puppet has 42 failures [13:21:12] PROBLEM - puppet last run on mw1027 is CRITICAL: CRITICAL: puppet fail [13:21:12] PROBLEM - puppet last run on mw1108 is CRITICAL: CRITICAL: Puppet has 62 failures [13:21:12] PROBLEM - puppet last run on mw1097 is CRITICAL: CRITICAL: Puppet has 60 failures [13:21:12] PROBLEM - puppet last run on ssl1006 is CRITICAL: CRITICAL: Puppet has 29 failures [13:21:22] PROBLEM - puppet last run on cp1044 is CRITICAL: CRITICAL: Puppet has 26 failures [13:21:22] PROBLEM - puppet last run on elastic1011 is CRITICAL: CRITICAL: Puppet has 24 failures [13:21:23] PROBLEM - puppet last run on mw1104 is CRITICAL: CRITICAL: Puppet has 65 failures [13:21:23] RECOVERY - puppet last run on mw1234 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [13:21:23] PROBLEM - puppet last run on mw1236 is CRITICAL: CRITICAL: Puppet has 85 failures [13:21:23] RECOVERY - puppet last run on mw1138 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [13:21:23] PROBLEM - puppet last run on analytics1032 is CRITICAL: CRITICAL: Puppet has 18 failures [13:21:24] PROBLEM - puppet last run on mw1143 is CRITICAL: CRITICAL: puppet fail [13:21:32] RECOVERY - puppet last run on mc1004 is OK: OK: Puppet is currently enabled, last run 33 seconds ago with 0 failures [13:21:33] RECOVERY - puppet last run on mw1147 is OK: OK: Puppet is currently enabled, last run 20 seconds ago with 0 failures [13:21:33] PROBLEM - puppet last run on mw1064 is CRITICAL: CRITICAL: puppet fail [13:21:33] RECOVERY - puppet last run on mc1010 is OK: OK: Puppet is currently enabled, last run 40 seconds ago with 0 failures [13:21:33] RECOVERY - puppet last run on mw1124 is OK: OK: Puppet is currently enabled, last run 15 seconds ago with 0 failures [13:21:33] PROBLEM - puppet last run on elastic1020 is CRITICAL: CRITICAL: puppet fail [13:21:33] RECOVERY - puppet last run on mw1020 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [13:21:34] RECOVERY - puppet last run on cp1064 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [13:21:35] PROBLEM - puppet last run on cp1037 is CRITICAL: CRITICAL: puppet fail [13:21:35] PROBLEM - puppet last run on db1037 is CRITICAL: CRITICAL: Puppet has 29 failures [13:21:35] PROBLEM - puppet last run on mw1066 is CRITICAL: CRITICAL: puppet fail [13:21:36] PROBLEM - puppet last run on osm-cp1001 is CRITICAL: CRITICAL: Puppet has 15 failures [13:21:36] PROBLEM - puppet last run on cp3012 is CRITICAL: CRITICAL: Puppet has 28 failures [13:21:37] PROBLEM - puppet last run on ssl3001 is CRITICAL: CRITICAL: puppet fail [13:21:37] RECOVERY - puppet last run on ocg1001 is OK: OK: Puppet is currently enabled, last run 3 seconds ago with 0 failures [13:21:38] PROBLEM - puppet last run on analytics1004 is CRITICAL: CRITICAL: Puppet has 33 failures [13:21:38] PROBLEM - puppet last run on amssq41 is CRITICAL: CRITICAL: Puppet has 28 failures [13:21:39] PROBLEM - puppet last run on mw1204 is CRITICAL: CRITICAL: puppet fail [13:21:39] PROBLEM - puppet last run on cp3009 is CRITICAL: CRITICAL: Puppet has 32 failures [13:21:40] PROBLEM - puppet last run on amssq38 is CRITICAL: CRITICAL: Puppet has 23 failures [13:21:40] RECOVERY - puppet last run on amssq52 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [13:21:41] PROBLEM - puppet last run on mw1229 is CRITICAL: CRITICAL: Puppet has 74 failures [13:21:41] RECOVERY - puppet last run on uranium is OK: OK: Puppet is currently enabled, last run 34 seconds ago with 0 failures [13:21:42] RECOVERY - puppet last run on mw1062 is OK: OK: Puppet is currently enabled, last run 20 seconds ago with 0 failures [13:21:42] PROBLEM - puppet last run on calcium is CRITICAL: CRITICAL: puppet fail [13:21:45] RECOVERY - puppet last run on mw1040 is OK: OK: Puppet is currently enabled, last run 33 seconds ago with 0 failures [13:21:48] RECOVERY - puppet last run on mw1169 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [13:21:48] PROBLEM - puppet last run on virt1000 is CRITICAL: CRITICAL: Puppet has 60 failures [13:21:52] RECOVERY - puppet last run on mw1214 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [13:21:52] RECOVERY - puppet last run on mw1216 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [13:21:52] RECOVERY - puppet last run on ms-be1001 is OK: OK: Puppet is currently enabled, last run 44 seconds ago with 0 failures [13:21:52] PROBLEM - puppet last run on db1072 is CRITICAL: CRITICAL: puppet fail [13:21:52] RECOVERY - puppet last run on mw1221 is OK: OK: Puppet is currently enabled, last run 29 seconds ago with 0 failures [13:21:53] PROBLEM - puppet last run on elastic1025 is CRITICAL: CRITICAL: puppet fail [13:21:53] RECOVERY - puppet last run on virt1002 is OK: OK: Puppet is currently enabled, last run 46 seconds ago with 0 failures [13:21:54] PROBLEM - puppet last run on cp1052 is CRITICAL: CRITICAL: Puppet has 27 failures [13:21:54] PROBLEM - puppet last run on mw1255 is CRITICAL: CRITICAL: Puppet has 83 failures [13:21:55] RECOVERY - puppet last run on cp1069 is OK: OK: Puppet is currently enabled, last run 55 seconds ago with 0 failures [13:21:55] PROBLEM - puppet last run on analytics1018 is CRITICAL: CRITICAL: Puppet has 24 failures [13:21:56] PROBLEM - puppet last run on mw1021 is CRITICAL: CRITICAL: Puppet has 73 failures [13:21:56] RECOVERY - puppet last run on ytterbium is OK: OK: Puppet is currently enabled, last run 41 seconds ago with 0 failures [13:21:57] RECOVERY - puppet last run on cp1059 is OK: OK: Puppet is currently enabled, last run 28 seconds ago with 0 failures [13:21:57] RECOVERY - puppet last run on mw1179 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [13:21:58] PROBLEM - puppet last run on ms-fe1002 is CRITICAL: CRITICAL: puppet fail [13:21:58] RECOVERY - puppet last run on mw1096 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [13:21:59] RECOVERY - puppet last run on cp1065 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [13:21:59] RECOVERY - puppet last run on dbstore1001 is OK: OK: Puppet is currently enabled, last run 51 seconds ago with 0 failures [13:22:00] PROBLEM - puppet last run on lvs1004 is CRITICAL: CRITICAL: puppet fail [13:22:00] RECOVERY - puppet last run on analytics1015 is OK: OK: Puppet is currently enabled, last run 33 seconds ago with 0 failures [13:22:02] PROBLEM - puppet last run on ssl1001 is CRITICAL: CRITICAL: Puppet has 20 failures [13:22:02] PROBLEM - puppet last run on mw1207 is CRITICAL: CRITICAL: puppet fail [13:22:02] PROBLEM - puppet last run on mw1158 is CRITICAL: CRITICAL: puppet fail [13:22:02] PROBLEM - puppet last run on dbstore1002 is CRITICAL: CRITICAL: Puppet has 17 failures [13:22:03] RECOVERY - puppet last run on mw1031 is OK: OK: Puppet is currently enabled, last run 12 seconds ago with 0 failures [13:22:03] PROBLEM - puppet last run on ms-be1010 is CRITICAL: CRITICAL: puppet fail [13:22:04] PROBLEM - puppet last run on mw1219 is CRITICAL: CRITICAL: Puppet has 65 failures [13:22:04] RECOVERY - puppet last run on mw1038 is OK: OK: Puppet is currently enabled, last run 21 seconds ago with 0 failures [13:22:05] PROBLEM - puppet last run on es1010 is CRITICAL: CRITICAL: Puppet has 26 failures [13:22:05] RECOVERY - puppet last run on mw1246 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [13:22:06] PROBLEM - puppet last run on mw1086 is CRITICAL: CRITICAL: puppet fail [13:22:12] PROBLEM - puppet last run on mw1090 is CRITICAL: CRITICAL: puppet fail [13:22:20] RECOVERY - puppet last run on search1020 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [13:22:21] RECOVERY - puppet last run on pc1001 is OK: OK: Puppet is currently enabled, last run 32 seconds ago with 0 failures [13:22:22] RECOVERY - puppet last run on wtp1024 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [13:22:22] PROBLEM - puppet last run on cp1053 is CRITICAL: CRITICAL: Puppet has 34 failures [13:22:29] RECOVERY - puppet last run on wtp1017 is OK: OK: Puppet is currently enabled, last run 28 seconds ago with 0 failures [13:22:29] RECOVERY - puppet last run on db1010 is OK: OK: Puppet is currently enabled, last run 57 seconds ago with 0 failures [13:22:29] PROBLEM - puppet last run on wtp1008 is CRITICAL: CRITICAL: puppet fail [13:22:29] PROBLEM - puppet last run on wtp1001 is CRITICAL: CRITICAL: Puppet has 22 failures [13:22:29] RECOVERY - puppet last run on wtp1021 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [13:22:30] PROBLEM - puppet last run on strontium is CRITICAL: CRITICAL: Puppet has 29 failures [13:22:30] PROBLEM - puppet last run on db1011 is CRITICAL: CRITICAL: Puppet has 20 failures [13:22:31] PROBLEM - puppet last run on db1047 is CRITICAL: CRITICAL: Puppet has 22 failures [13:22:31] RECOVERY - puppet last run on mw1035 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [13:22:32] PROBLEM - puppet last run on cp1068 is CRITICAL: CRITICAL: puppet fail [13:22:32] RECOVERY - puppet last run on wtp1019 is OK: OK: Puppet is currently enabled, last run 39 seconds ago with 0 failures [13:22:33] RECOVERY - puppet last run on cp1043 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [13:22:33] PROBLEM - puppet last run on mw1071 is CRITICAL: CRITICAL: puppet fail [13:22:34] PROBLEM - puppet last run on mw1253 is CRITICAL: CRITICAL: Puppet has 79 failures [13:22:34] RECOVERY - puppet last run on db1024 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [13:22:35] RECOVERY - puppet last run on db1065 is OK: OK: Puppet is currently enabled, last run 37 seconds ago with 0 failures [13:22:35] PROBLEM - puppet last run on cp1054 is CRITICAL: CRITICAL: puppet fail [13:22:36] RECOVERY - puppet last run on db1007 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [13:22:36] PROBLEM - puppet last run on elastic1003 is CRITICAL: CRITICAL: Puppet has 22 failures [13:22:40] RECOVERY - puppet last run on logstash1003 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [13:22:40] RECOVERY - puppet last run on mw1095 is OK: OK: Puppet is currently enabled, last run 3 minutes ago with 0 failures [13:22:41] RECOVERY - puppet last run on elastic1026 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [13:22:41] RECOVERY - puppet last run on es1009 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [13:22:41] PROBLEM - puppet last run on analytics1031 is CRITICAL: CRITICAL: Puppet has 24 failures [13:22:41] PROBLEM - puppet last run on db1035 is CRITICAL: CRITICAL: Puppet has 22 failures [13:22:41] RECOVERY - puppet last run on analytics1019 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [13:22:42] PROBLEM - puppet last run on mw1135 is CRITICAL: CRITICAL: puppet fail [13:22:42] PROBLEM - puppet last run on virt1008 is CRITICAL: CRITICAL: Puppet has 28 failures [13:22:43] RECOVERY - puppet last run on mw1161 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [13:22:43] RECOVERY - puppet last run on rdb1004 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [13:22:44] RECOVERY - puppet last run on zirconium is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [13:22:44] RECOVERY - puppet last run on ms-be2002 is OK: OK: Puppet is currently enabled, last run 57 seconds ago with 0 failures [13:22:50] PROBLEM - puppet last run on mw1193 is CRITICAL: CRITICAL: Puppet has 63 failures [13:22:51] RECOVERY - puppet last run on ssl3003 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [13:22:51] RECOVERY - puppet last run on mw1257 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [13:22:52] RECOVERY - puppet last run on amssq58 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [13:22:52] RECOVERY - puppet last run on amssq57 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [13:22:52] PROBLEM - puppet last run on radium is CRITICAL: CRITICAL: puppet fail [13:22:52] RECOVERY - puppet last run on labsdb1001 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [13:22:52] RECOVERY - puppet last run on cp3022 is OK: OK: Puppet is currently enabled, last run 32 seconds ago with 0 failures [13:22:53] RECOVERY - puppet last run on mw1130 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [13:22:53] RECOVERY - puppet last run on mw1007 is OK: OK: Puppet is currently enabled, last run 15 seconds ago with 0 failures [13:22:54] PROBLEM - puppet last run on search1012 is CRITICAL: CRITICAL: Puppet has 62 failures [13:22:54] RECOVERY - puppet last run on cp1070 is OK: OK: Puppet is currently enabled, last run 42 seconds ago with 0 failures [13:22:55] RECOVERY - puppet last run on mw1106 is OK: OK: Puppet is currently enabled, last run 16 seconds ago with 0 failures [13:22:55] RECOVERY - puppet last run on elastic1028 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [13:23:06] RECOVERY - puppet last run on analytics1036 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [13:23:07] RECOVERY - puppet last run on es1006 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [13:23:07] RECOVERY - puppet last run on mc1009 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [13:23:07] RECOVERY - puppet last run on mw1072 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [13:23:07] PROBLEM - puppet last run on chromium is CRITICAL: CRITICAL: Puppet has 20 failures [13:23:07] PROBLEM - puppet last run on cp1040 is CRITICAL: CRITICAL: Puppet has 23 failures [13:23:07] RECOVERY - puppet last run on elastic1031 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [13:23:08] RECOVERY - puppet last run on gold is OK: OK: Puppet is currently enabled, last run 10 seconds ago with 0 failures [13:23:08] PROBLEM - puppet last run on wtp1010 is CRITICAL: CRITICAL: puppet fail [13:23:10] RECOVERY - puppet last run on cerium is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [13:23:21] RECOVERY - puppet last run on mw1245 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [13:23:21] PROBLEM - puppet last run on db1045 is CRITICAL: CRITICAL: Puppet has 24 failures [13:23:21] RECOVERY - puppet last run on mw1028 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [13:23:21] RECOVERY - puppet last run on analytics1017 is OK: OK: Puppet is currently enabled, last run 23 seconds ago with 0 failures [13:23:21] PROBLEM - puppet last run on ms-be1011 is CRITICAL: CRITICAL: Puppet has 26 failures [13:23:22] RECOVERY - puppet last run on mw1218 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [13:23:22] RECOVERY - puppet last run on mw1178 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [13:23:23] PROBLEM - puppet last run on achernar is CRITICAL: CRITICAL: Puppet has 27 failures [13:23:23] PROBLEM - puppet last run on mw1209 is CRITICAL: CRITICAL: Puppet has 71 failures [13:23:24] PROBLEM - puppet last run on sca1001 is CRITICAL: CRITICAL: Puppet has 24 failures [13:23:24] RECOVERY - puppet last run on db2035 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [13:23:25] RECOVERY - puppet last run on wtp1020 is OK: OK: Puppet is currently enabled, last run 6 seconds ago with 0 failures [13:23:25] PROBLEM - puppet last run on ms-be1015 is CRITICAL: CRITICAL: puppet fail [13:23:26] RECOVERY - puppet last run on ms-be2013 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [13:23:26] RECOVERY - puppet last run on ms-be2003 is OK: OK: Puppet is currently enabled, last run 19 seconds ago with 0 failures [13:23:27] RECOVERY - puppet last run on carbon is OK: OK: Puppet is currently enabled, last run 36 seconds ago with 0 failures [13:23:27] RECOVERY - puppet last run on es1001 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [13:23:28] RECOVERY - puppet last run on mc1016 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [13:23:31] RECOVERY - puppet last run on ms-fe1003 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [13:23:31] RECOVERY - puppet last run on mw1232 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [13:23:31] PROBLEM - puppet last run on mw1154 is CRITICAL: CRITICAL: Puppet has 71 failures [13:23:31] RECOVERY - puppet last run on ms-be1013 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [13:23:31] RECOVERY - puppet last run on amslvs2 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [13:23:32] RECOVERY - puppet last run on amssq44 is OK: OK: Puppet is currently enabled, last run 38 seconds ago with 0 failures [13:23:32] RECOVERY - puppet last run on amssq43 is OK: OK: Puppet is currently enabled, last run 21 seconds ago with 0 failures [13:23:33] RECOVERY - puppet last run on cp3006 is OK: OK: Puppet is currently enabled, last run 50 seconds ago with 0 failures [13:23:33] RECOVERY - puppet last run on mw1233 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [13:23:34] RECOVERY - puppet last run on mw1244 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [13:23:34] RECOVERY - puppet last run on cp1049 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [13:23:40] RECOVERY - puppet last run on mw1141 is OK: OK: Puppet is currently enabled, last run 43 seconds ago with 0 failures [13:23:41] RECOVERY - puppet last run on amslvs4 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [13:23:41] RECOVERY - puppet last run on potassium is OK: OK: Puppet is currently enabled, last run 13 seconds ago with 0 failures [13:23:41] RECOVERY - puppet last run on mw1063 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [13:23:41] RECOVERY - puppet last run on mc1011 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [13:23:41] PROBLEM - puppet last run on cp1045 is CRITICAL: CRITICAL: Puppet has 24 failures [13:23:41] RECOVERY - puppet last run on caesium is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [13:23:50] RECOVERY - puppet last run on mw1256 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [13:23:50] RECOVERY - puppet last run on db2034 is OK: OK: Puppet is currently enabled, last run 31 seconds ago with 0 failures [13:23:50] RECOVERY - puppet last run on palladium is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [13:23:51] RECOVERY - puppet last run on mc1006 is OK: OK: Puppet is currently enabled, last run 45 seconds ago with 0 failures [13:23:51] PROBLEM - puppet last run on copper is CRITICAL: CRITICAL: Puppet has 21 failures [13:23:51] RECOVERY - puppet last run on elastic1021 is OK: OK: Puppet is currently enabled, last run 2 seconds ago with 0 failures [13:23:51] RECOVERY - puppet last run on mw1109 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [13:23:52] RECOVERY - puppet last run on analytics1040 is OK: OK: Puppet is currently enabled, last run 39 seconds ago with 0 failures [13:23:52] RECOVERY - puppet last run on neptunium is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [13:23:53] PROBLEM - puppet last run on lvs4004 is CRITICAL: CRITICAL: Puppet has 25 failures [13:23:53] RECOVERY - puppet last run on lvs4002 is OK: OK: Puppet is currently enabled, last run 37 seconds ago with 0 failures [13:23:54] RECOVERY - puppet last run on cp4006 is OK: OK: Puppet is currently enabled, last run 42 seconds ago with 0 failures [13:23:59] RECOVERY - puppet last run on mw1006 is OK: OK: Puppet is currently enabled, last run 57 seconds ago with 0 failures [13:24:06] RECOVERY - puppet last run on db2005 is OK: OK: Puppet is currently enabled, last run 13 seconds ago with 0 failures [13:24:07] RECOVERY - puppet last run on cp4007 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [13:24:07] RECOVERY - puppet last run on mw1089 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [13:24:07] PROBLEM - puppet last run on db1056 is CRITICAL: CRITICAL: Puppet has 16 failures [13:24:07] RECOVERY - puppet last run on mw1132 is OK: OK: Puppet is currently enabled, last run 3 minutes ago with 0 failures [13:24:07] PROBLEM - puppet last run on lvs1003 is CRITICAL: CRITICAL: Puppet has 20 failures [13:24:07] RECOVERY - puppet last run on vanadium is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [13:24:08] RECOVERY - puppet last run on mw1192 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [13:24:08] RECOVERY - puppet last run on ms-be1003 is OK: OK: Puppet is currently enabled, last run 16 seconds ago with 0 failures [13:24:22] RECOVERY - puppet last run on mw1115 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [13:24:22] RECOVERY - puppet last run on xenon is OK: OK: Puppet is currently enabled, last run 33 seconds ago with 0 failures [13:24:22] RECOVERY - puppet last run on magnesium is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [13:24:23] RECOVERY - puppet last run on wtp1006 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [13:24:23] RECOVERY - puppet last run on search1016 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [13:24:23] RECOVERY - puppet last run on mw1080 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [13:24:29] RECOVERY - puppet last run on mw1224 is OK: OK: Puppet is currently enabled, last run 15 seconds ago with 0 failures [13:24:30] RECOVERY - puppet last run on mc1003 is OK: OK: Puppet is currently enabled, last run 3 seconds ago with 0 failures [13:24:30] RECOVERY - puppet last run on mw1197 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [13:24:30] RECOVERY - puppet last run on mw1005 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [13:24:30] RECOVERY - puppet last run on elastic1004 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [13:24:30] RECOVERY - puppet last run on db1053 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [13:24:31] RECOVERY - puppet last run on db1073 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [13:24:31] RECOVERY - puppet last run on mw1048 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [13:24:32] RECOVERY - puppet last run on curium is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [13:24:32] PROBLEM - puppet last run on cp3011 is CRITICAL: CRITICAL: Puppet has 28 failures [13:24:33] RECOVERY - puppet last run on wtp1009 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [13:24:39] RECOVERY - puppet last run on cp3013 is OK: OK: Puppet is currently enabled, last run 3 minutes ago with 0 failures [13:24:40] RECOVERY - puppet last run on amssq59 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [13:24:40] RECOVERY - puppet last run on cp3020 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [13:24:40] RECOVERY - puppet last run on mw1240 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [13:24:41] RECOVERY - puppet last run on logstash1001 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [13:24:41] RECOVERY - puppet last run on elastic1007 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [13:24:41] RECOVERY - puppet last run on cp3015 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [13:24:41] RECOVERY - puppet last run on labstore1003 is OK: OK: Puppet is currently enabled, last run 50 seconds ago with 0 failures [13:24:42] RECOVERY - puppet last run on dbproxy1002 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [13:24:42] RECOVERY - puppet last run on platinum is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [13:24:49] RECOVERY - puppet last run on mw1067 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [13:24:58] RECOVERY - puppet last run on mw1134 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [13:25:00] RECOVERY - puppet last run on mw1254 is OK: OK: Puppet is currently enabled, last run 47 seconds ago with 0 failures [13:25:02] RECOVERY - puppet last run on mw1228 is OK: OK: Puppet is currently enabled, last run 21 seconds ago with 0 failures [13:25:02] RECOVERY - puppet last run on mw1003 is OK: OK: Puppet is currently enabled, last run 34 seconds ago with 0 failures [13:25:02] RECOVERY - puppet last run on analytics1033 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [13:25:02] RECOVERY - puppet last run on cp1039 is OK: OK: Puppet is currently enabled, last run 58 seconds ago with 0 failures [13:25:02] RECOVERY - puppet last run on db1029 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [13:25:03] RECOVERY - puppet last run on analytics1020 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [13:25:03] RECOVERY - puppet last run on db1031 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [13:25:04] RECOVERY - puppet last run on searchidx1001 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [13:25:04] RECOVERY - puppet last run on lvs1002 is OK: OK: Puppet is currently enabled, last run 36 seconds ago with 0 failures [13:25:05] RECOVERY - puppet last run on mw1059 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [13:25:05] RECOVERY - puppet last run on cp1055 is OK: OK: Puppet is currently enabled, last run 58 seconds ago with 0 failures [13:25:06] RECOVERY - puppet last run on elastic1012 is OK: OK: Puppet is currently enabled, last run 42 seconds ago with 0 failures [13:25:06] RECOVERY - puppet last run on mw1174 is OK: OK: Puppet is currently enabled, last run 54 seconds ago with 0 failures [13:25:07] RECOVERY - puppet last run on mw1012 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [13:25:07] RECOVERY - puppet last run on mw1009 is OK: OK: Puppet is currently enabled, last run 34 seconds ago with 0 failures [13:25:08] RECOVERY - puppet last run on mw1046 is OK: OK: Puppet is currently enabled, last run 10 seconds ago with 0 failures [13:25:08] RECOVERY - puppet last run on mw1217 is OK: OK: Puppet is currently enabled, last run 23 seconds ago with 0 failures [13:25:09] RECOVERY - puppet last run on rbf1002 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [13:25:09] RECOVERY - puppet last run on mw1026 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [13:25:10] RECOVERY - puppet last run on mw1242 is OK: OK: Puppet is currently enabled, last run 26 seconds ago with 0 failures [13:25:10] RECOVERY - puppet last run on mw1041 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [13:25:11] RECOVERY - puppet last run on snapshot1003 is OK: OK: Puppet is currently enabled, last run 58 seconds ago with 0 failures [13:25:11] RECOVERY - puppet last run on mw1100 is OK: OK: Puppet is currently enabled, last run 6 seconds ago with 0 failures [13:25:12] RECOVERY - puppet last run on mw1200 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [13:25:12] RECOVERY - puppet last run on db1066 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [13:25:13] RECOVERY - puppet last run on ms-be1004 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [13:25:23] RECOVERY - puppet last run on mw1045 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [13:25:23] RECOVERY - puppet last run on cp1068 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [13:25:23] RECOVERY - puppet last run on mw1140 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [13:25:23] RECOVERY - puppet last run on ms-fe1001 is OK: OK: Puppet is currently enabled, last run 56 seconds ago with 0 failures [13:25:23] RECOVERY - puppet last run on mw1145 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [13:25:23] RECOVERY - puppet last run on es1008 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [13:25:32] RECOVERY - puppet last run on mw1088 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [13:25:36] RECOVERY - puppet last run on elastic1001 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [13:25:36] RECOVERY - puppet last run on ms-be1006 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [13:25:36] RECOVERY - puppet last run on netmon1001 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [13:25:36] RECOVERY - puppet last run on mw1160 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [13:25:43] RECOVERY - puppet last run on db2019 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [13:25:49] RECOVERY - puppet last run on amssq61 is OK: OK: Puppet is currently enabled, last run 57 seconds ago with 0 failures [13:25:50] RECOVERY - puppet last run on amssq54 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [13:25:51] RECOVERY - puppet last run on mw1187 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [13:25:51] RECOVERY - puppet last run on elastic1018 is OK: OK: Puppet is currently enabled, last run 9 seconds ago with 0 failures [13:25:51] RECOVERY - puppet last run on mw1226 is OK: OK: Puppet is currently enabled, last run 58 seconds ago with 0 failures [13:25:51] RECOVERY - puppet last run on analytics1041 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [13:25:51] RECOVERY - puppet last run on cp1047 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [13:25:52] RECOVERY - puppet last run on ssl1002 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [13:25:52] RECOVERY - puppet last run on mw1250 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [13:26:03] RECOVERY - puppet last run on db1067 is OK: OK: Puppet is currently enabled, last run 9 seconds ago with 0 failures [13:26:05] RECOVERY - puppet last run on sca1002 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [13:26:05] RECOVERY - puppet last run on elastic1008 is OK: OK: Puppet is currently enabled, last run 30 seconds ago with 0 failures [13:26:05] RECOVERY - puppet last run on ms-be2004 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [13:26:05] RECOVERY - puppet last run on db1018 is OK: OK: Puppet is currently enabled, last run 24 seconds ago with 0 failures [13:26:05] RECOVERY - puppet last run on analytics1025 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [13:26:06] RECOVERY - puppet last run on db2039 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [13:26:06] RECOVERY - puppet last run on db2002 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [13:26:07] RECOVERY - puppet last run on es2001 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [13:26:07] RECOVERY - puppet last run on db2018 is OK: OK: Puppet is currently enabled, last run 5 seconds ago with 0 failures [13:26:08] RECOVERY - puppet last run on db2009 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [13:26:12] RECOVERY - puppet last run on es2008 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [13:26:12] RECOVERY - puppet last run on ms-be2006 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [13:26:12] RECOVERY - puppet last run on db1022 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [13:26:27] RECOVERY - puppet last run on mw1082 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [13:26:29] RECOVERY - puppet last run on lvs1005 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [13:26:30] RECOVERY - puppet last run on amssq53 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [13:26:31] RECOVERY - puppet last run on cp3008 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [13:26:32] RECOVERY - puppet last run on amssq49 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [13:26:41] RECOVERY - puppet last run on mw1164 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [13:26:42] RECOVERY - puppet last run on iron is OK: OK: Puppet is currently enabled, last run 18 seconds ago with 0 failures [13:26:42] RECOVERY - puppet last run on lvs2001 is OK: OK: Puppet is currently enabled, last run 11 seconds ago with 0 failures [13:26:42] RECOVERY - puppet last run on gallium is OK: OK: Puppet is currently enabled, last run 10 seconds ago with 0 failures [13:26:42] RECOVERY - puppet last run on elastic1022 is OK: OK: Puppet is currently enabled, last run 10 seconds ago with 0 failures [13:26:42] RECOVERY - puppet last run on analytics1035 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [13:26:43] RECOVERY - puppet last run on elastic1027 is OK: OK: Puppet is currently enabled, last run 46 seconds ago with 0 failures [13:26:43] RECOVERY - puppet last run on mw1117 is OK: OK: Puppet is currently enabled, last run 23 seconds ago with 0 failures [13:26:44] RECOVERY - puppet last run on ms-fe2004 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [13:26:44] RECOVERY - puppet last run on mw1176 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [13:26:45] RECOVERY - puppet last run on mc1002 is OK: OK: Puppet is currently enabled, last run 54 seconds ago with 0 failures [13:26:45] RECOVERY - puppet last run on mw1099 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [13:26:46] RECOVERY - puppet last run on mw1042 is OK: OK: Puppet is currently enabled, last run 9 seconds ago with 0 failures [13:26:46] RECOVERY - puppet last run on mw1052 is OK: OK: Puppet is currently enabled, last run 0 seconds ago with 0 failures [13:26:47] RECOVERY - puppet last run on lead is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [13:26:47] RECOVERY - puppet last run on db1042 is OK: OK: Puppet is currently enabled, last run 30 seconds ago with 0 failures [13:26:48] RECOVERY - puppet last run on mw1060 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [13:26:48] RECOVERY - puppet last run on search1001 is OK: OK: Puppet is currently enabled, last run 39 seconds ago with 0 failures [13:26:51] RECOVERY - puppet last run on ms-fe2003 is OK: OK: Puppet is currently enabled, last run 6 seconds ago with 0 failures [13:26:51] RECOVERY - puppet last run on ms-fe2001 is OK: OK: Puppet is currently enabled, last run 38 seconds ago with 0 failures [13:26:51] RECOVERY - puppet last run on mw1065 is OK: OK: Puppet is currently enabled, last run 23 seconds ago with 0 failures [13:26:52] RECOVERY - puppet last run on mw1205 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [13:26:52] RECOVERY - puppet last run on cp1056 is OK: OK: Puppet is currently enabled, last run 46 seconds ago with 0 failures [13:26:52] RECOVERY - puppet last run on mw1222 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [13:26:52] RECOVERY - puppet last run on db1051 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [13:26:53] RECOVERY - puppet last run on pc1002 is OK: OK: Puppet is currently enabled, last run 11 seconds ago with 0 failures [13:26:53] RECOVERY - puppet last run on search1007 is OK: OK: Puppet is currently enabled, last run 51 seconds ago with 0 failures [13:27:05] RECOVERY - puppet last run on mw1126 is OK: OK: Puppet is currently enabled, last run 23 seconds ago with 0 failures [13:27:07] RECOVERY - puppet last run on mw1173 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [13:27:08] RECOVERY - puppet last run on mw1119 is OK: OK: Puppet is currently enabled, last run 46 seconds ago with 0 failures [13:27:09] RECOVERY - puppet last run on db1052 is OK: OK: Puppet is currently enabled, last run 7 seconds ago with 0 failures [13:27:09] RECOVERY - puppet last run on mw1177 is OK: OK: Puppet is currently enabled, last run 20 seconds ago with 0 failures [13:27:09] RECOVERY - puppet last run on mw1069 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [13:27:09] RECOVERY - puppet last run on mw1008 is OK: OK: Puppet is currently enabled, last run 56 seconds ago with 0 failures [13:27:10] RECOVERY - puppet last run on ms-fe1004 is OK: OK: Puppet is currently enabled, last run 44 seconds ago with 0 failures [13:27:10] RECOVERY - puppet last run on mw1068 is OK: OK: Puppet is currently enabled, last run 56 seconds ago with 0 failures [13:27:10] RECOVERY - puppet last run on db1059 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [13:27:12] RECOVERY - puppet last run on lvs3001 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [13:27:12] RECOVERY - puppet last run on amssq32 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [13:27:12] RECOVERY - puppet last run on mw1150 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [13:27:13] RECOVERY - puppet last run on amslvs1 is OK: OK: Puppet is currently enabled, last run 48 seconds ago with 0 failures [13:27:13] RECOVERY - puppet last run on amssq35 is OK: OK: Puppet is currently enabled, last run 26 seconds ago with 0 failures [13:27:13] RECOVERY - puppet last run on cp3014 is OK: OK: Puppet is currently enabled, last run 8 seconds ago with 0 failures [13:27:21] RECOVERY - puppet last run on cp3016 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [13:27:23] RECOVERY - puppet last run on db1040 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [13:27:23] RECOVERY - puppet last run on search1010 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [13:27:23] RECOVERY - puppet last run on mw1114 is OK: OK: Puppet is currently enabled, last run 28 seconds ago with 0 failures [13:27:23] RECOVERY - puppet last run on labnet1001 is OK: OK: Puppet is currently enabled, last run 20 seconds ago with 0 failures [13:27:31] RECOVERY - puppet last run on mw1039 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [13:27:31] RECOVERY - puppet last run on virt1006 is OK: OK: Puppet is currently enabled, last run 49 seconds ago with 0 failures [13:27:32] RECOVERY - puppet last run on search1018 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [13:27:32] RECOVERY - puppet last run on mw1120 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [13:27:32] RECOVERY - puppet last run on mw1211 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [13:27:32] RECOVERY - puppet last run on elastic1030 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [13:27:32] RECOVERY - puppet last run on mw1166 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [13:27:33] RECOVERY - puppet last run on db1034 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [13:27:41] RECOVERY - puppet last run on db1050 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [13:27:41] RECOVERY - puppet last run on cp1061 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [13:27:41] RECOVERY - puppet last run on mw1153 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [13:27:41] RECOVERY - puppet last run on db1028 is OK: OK: Puppet is currently enabled, last run 57 seconds ago with 0 failures [13:27:43] RECOVERY - puppet last run on mw1195 is OK: OK: Puppet is currently enabled, last run 6 seconds ago with 0 failures [13:27:44] RECOVERY - puppet last run on mw1251 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [13:27:54] RECOVERY - puppet last run on mw1025 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [13:27:58] RECOVERY - puppet last run on lithium is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [13:28:00] RECOVERY - puppet last run on labsdb1003 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [13:28:01] RECOVERY - puppet last run on mw1235 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [13:28:02] RECOVERY - puppet last run on snapshot1001 is OK: OK: Puppet is currently enabled, last run 24 seconds ago with 0 failures [13:28:02] RECOVERY - puppet last run on mw1092 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [13:28:02] RECOVERY - puppet last run on wtp1012 is OK: OK: Puppet is currently enabled, last run 57 seconds ago with 0 failures [13:28:02] RECOVERY - puppet last run on db1046 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [13:28:14] RECOVERY - puppet last run on db1016 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [13:28:14] RECOVERY - puppet last run on dataset1001 is OK: OK: Puppet is currently enabled, last run 38 seconds ago with 0 failures [13:28:14] RECOVERY - puppet last run on analytics1010 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [13:28:14] RECOVERY - puppet last run on db1015 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [13:28:14] RECOVERY - puppet last run on wtp1016 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [13:28:14] RECOVERY - puppet last run on mw1118 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [13:28:15] RECOVERY - puppet last run on analytics1016 is OK: OK: Puppet is currently enabled, last run 47 seconds ago with 0 failures [13:28:15] RECOVERY - puppet last run on mw1002 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [13:28:16] RECOVERY - puppet last run on logstash1002 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [13:28:16] RECOVERY - puppet last run on labstore1001 is OK: OK: Puppet is currently enabled, last run 47 seconds ago with 0 failures [13:28:17] RECOVERY - puppet last run on lvs2006 is OK: OK: Puppet is currently enabled, last run 7 seconds ago with 0 failures [13:28:17] RECOVERY - puppet last run on cp4008 is OK: OK: Puppet is currently enabled, last run 44 seconds ago with 0 failures [13:28:18] RECOVERY - puppet last run on cp4014 is OK: OK: Puppet is currently enabled, last run 34 seconds ago with 0 failures [13:28:18] RECOVERY - puppet last run on cp4004 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [13:28:19] RECOVERY - puppet last run on db1002 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [13:28:19] RECOVERY - puppet last run on mc1012 is OK: OK: Puppet is currently enabled, last run 58 seconds ago with 0 failures [13:28:20] RECOVERY - puppet last run on mw1054 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [13:28:23] RECOVERY - puppet last run on mw1061 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [13:28:23] RECOVERY - puppet last run on db2038 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [13:28:24] RECOVERY - puppet last run on db2036 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [13:28:24] RECOVERY - puppet last run on ruthenium is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [13:28:24] RECOVERY - puppet last run on mw1011 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [13:28:25] RECOVERY - puppet last run on install2001 is OK: OK: Puppet is currently enabled, last run 52 seconds ago with 0 failures [13:28:25] RECOVERY - puppet last run on ms-be3002 is OK: OK: Puppet is currently enabled, last run 56 seconds ago with 0 failures [13:28:25] RECOVERY - puppet last run on cp3003 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [13:28:25] RECOVERY - puppet last run on cp1058 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [13:28:26] RECOVERY - puppet last run on mw1123 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [13:28:26] RECOVERY - puppet last run on dbproxy1001 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [13:28:27] RECOVERY - puppet last run on mw1144 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [13:28:27] RECOVERY - puppet last run on db1043 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [13:28:28] RECOVERY - puppet last run on mw1189 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [13:28:35] RECOVERY - puppet last run on analytics1030 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [13:28:43] RECOVERY - puppet last run on analytics1038 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [13:28:43] RECOVERY - puppet last run on db1003 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [13:28:43] RECOVERY - puppet last run on elastic1024 is OK: OK: Puppet is currently enabled, last run 53 seconds ago with 0 failures [13:28:43] RECOVERY - puppet last run on analytics1013 is OK: OK: Puppet is currently enabled, last run 2 seconds ago with 0 failures [13:28:43] RECOVERY - puppet last run on mw1170 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [13:28:44] RECOVERY - puppet last run on mw1247 is OK: OK: Puppet is currently enabled, last run 1 second ago with 0 failures [13:28:44] RECOVERY - puppet last run on db1023 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [13:28:45] RECOVERY - puppet last run on mw1238 is OK: OK: Puppet is currently enabled, last run 8 seconds ago with 0 failures [13:28:45] RECOVERY - puppet last run on db1048 is OK: OK: Puppet is currently enabled, last run 21 seconds ago with 0 failures [13:28:46] RECOVERY - puppet last run on labcontrol2001 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [13:28:46] RECOVERY - puppet last run on db2016 is OK: OK: Puppet is currently enabled, last run 5 seconds ago with 0 failures [13:28:47] RECOVERY - puppet last run on lvs4003 is OK: OK: Puppet is currently enabled, last run 5 seconds ago with 0 failures [13:28:53] RECOVERY - puppet last run on db2007 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [13:28:54] RECOVERY - puppet last run on db2029 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [13:28:54] RECOVERY - puppet last run on cp4003 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [13:28:54] RECOVERY - puppet last run on cp4019 is OK: OK: Puppet is currently enabled, last run 37 seconds ago with 0 failures [13:28:54] RECOVERY - puppet last run on elastic1006 is OK: OK: Puppet is currently enabled, last run 19 seconds ago with 0 failures [13:28:54] RECOVERY - puppet last run on cp4001 is OK: OK: Puppet is currently enabled, last run 26 seconds ago with 0 failures [13:28:54] RECOVERY - puppet last run on bast4001 is OK: OK: Puppet is currently enabled, last run 15 seconds ago with 0 failures [13:28:55] RECOVERY - puppet last run on mw1208 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [13:28:55] RECOVERY - puppet last run on mw1213 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [13:28:56] RECOVERY - puppet last run on mw1055 is OK: OK: Puppet is currently enabled, last run 46 seconds ago with 0 failures [13:29:03] RECOVERY - puppet last run on mw1149 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [13:29:04] RECOVERY - puppet last run on db1021 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [13:29:04] RECOVERY - puppet last run on snapshot1002 is OK: OK: Puppet is currently enabled, last run 33 seconds ago with 0 failures [13:29:15] RECOVERY - puppet last run on amssq46 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [13:29:24] RECOVERY - puppet last run on mw1129 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [13:29:24] RECOVERY - puppet last run on mw1249 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [13:29:24] RECOVERY - puppet last run on mw1076 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [13:29:24] RECOVERY - puppet last run on elastic1019 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [13:29:25] RECOVERY - puppet last run on mw1175 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [13:29:25] RECOVERY - puppet last run on lvs2004 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [13:29:25] RECOVERY - puppet last run on mw1044 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [13:29:26] RECOVERY - puppet last run on cp1050 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [13:29:26] RECOVERY - puppet last run on db1060 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [13:29:27] RECOVERY - puppet last run on db1039 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [13:29:33] RECOVERY - puppet last run on ms-be2011 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [13:29:36] RECOVERY - puppet last run on cp4018 is OK: OK: Puppet is currently enabled, last run 16 seconds ago with 0 failures [13:29:36] RECOVERY - puppet last run on cp4005 is OK: OK: Puppet is currently enabled, last run 9 seconds ago with 0 failures [13:29:46] RECOVERY - puppet last run on mw1237 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [13:29:46] RECOVERY - puppet last run on antimony is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [13:29:46] RECOVERY - puppet last run on mc1005 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [13:29:57] RECOVERY - puppet last run on es1007 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [13:29:57] RECOVERY - puppet last run on sodium is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [13:29:58] RECOVERY - puppet last run on mw1151 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [13:30:03] RECOVERY - puppet last run on plutonium is OK: OK: Puppet is currently enabled, last run 20 seconds ago with 0 failures [13:30:15] RECOVERY - puppet last run on virt1001 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [13:30:15] RECOVERY - puppet last run on mw1172 is OK: OK: Puppet is currently enabled, last run 3 minutes ago with 0 failures [13:30:15] RECOVERY - puppet last run on search1002 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [13:30:15] RECOVERY - puppet last run on ms-be1012 is OK: OK: Puppet is currently enabled, last run 18 seconds ago with 0 failures [13:30:15] RECOVERY - puppet last run on search1005 is OK: OK: Puppet is currently enabled, last run 36 seconds ago with 0 failures [13:30:15] RECOVERY - puppet last run on wtp1005 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [13:30:15] RECOVERY - puppet last run on db1020 is OK: OK: Puppet is currently enabled, last run 9 seconds ago with 0 failures [13:30:16] RECOVERY - puppet last run on mw1079 is OK: OK: Puppet is currently enabled, last run 4 seconds ago with 0 failures [13:30:16] RECOVERY - puppet last run on mw1057 is OK: OK: Puppet is currently enabled, last run 15 seconds ago with 0 failures [13:30:17] RECOVERY - puppet last run on amssq40 is OK: OK: Puppet is currently enabled, last run 32 seconds ago with 0 failures [13:30:17] RECOVERY - puppet last run on amssq55 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [13:30:18] RECOVERY - puppet last run on amssq60 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [13:30:18] RECOVERY - puppet last run on amssq36 is OK: OK: Puppet is currently enabled, last run 40 seconds ago with 0 failures [13:30:19] RECOVERY - puppet last run on ms-fe3002 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [13:30:19] RECOVERY - puppet last run on db1036 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [13:30:20] RECOVERY - puppet last run on cp1046 is OK: OK: Puppet is currently enabled, last run 0 seconds ago with 0 failures [13:30:20] RECOVERY - puppet last run on argon is OK: OK: Puppet is currently enabled, last run 46 seconds ago with 0 failures [13:30:21] RECOVERY - puppet last run on amssq47 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [13:30:21] RECOVERY - puppet last run on amssq48 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [13:30:22] RECOVERY - puppet last run on ms-be3001 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [13:30:22] RECOVERY - puppet last run on cp3010 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [13:30:23] RECOVERY - puppet last run on ms-be1008 is OK: OK: Puppet is currently enabled, last run 0 seconds ago with 0 failures [13:30:23] RECOVERY - puppet last run on db1062 is OK: OK: Puppet is currently enabled, last run 21 seconds ago with 0 failures [13:30:24] RECOVERY - puppet last run on cp1062 is OK: OK: Puppet is currently enabled, last run 34 seconds ago with 0 failures [13:30:24] RECOVERY - puppet last run on mw1098 is OK: OK: Puppet is currently enabled, last run 15 seconds ago with 0 failures [13:30:25] RECOVERY - puppet last run on mw1084 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [13:30:25] RECOVERY - puppet last run on rdb1001 is OK: OK: Puppet is currently enabled, last run 23 seconds ago with 0 failures [13:30:26] RECOVERY - puppet last run on polonium is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [13:30:26] RECOVERY - puppet last run on tin is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [13:30:27] RECOVERY - puppet last run on analytics1037 is OK: OK: Puppet is currently enabled, last run 19 seconds ago with 0 failures [13:30:27] RECOVERY - puppet last run on mw1056 is OK: OK: Puppet is currently enabled, last run 1 second ago with 0 failures [13:30:28] RECOVERY - puppet last run on analytics1022 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [13:30:28] RECOVERY - puppet last run on mw1049 is OK: OK: Puppet is currently enabled, last run 39 seconds ago with 0 failures [13:30:29] RECOVERY - puppet last run on analytics1026 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [13:30:33] RECOVERY - puppet last run on mc1014 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [13:30:33] RECOVERY - puppet last run on mw1258 is OK: OK: Puppet is currently enabled, last run 58 seconds ago with 0 failures [13:30:33] RECOVERY - puppet last run on db1026 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [13:30:33] RECOVERY - puppet last run on gadolinium is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [13:30:33] RECOVERY - puppet last run on mw1212 is OK: OK: Puppet is currently enabled, last run 9 seconds ago with 0 failures [13:30:34] RECOVERY - puppet last run on mw1133 is OK: OK: Puppet is currently enabled, last run 13 seconds ago with 0 failures [13:30:45] RECOVERY - puppet last run on mw1202 is OK: OK: Puppet is currently enabled, last run 40 seconds ago with 0 failures [13:30:48] RECOVERY - puppet last run on mw1162 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [13:30:48] RECOVERY - puppet last run on cp1063 is OK: OK: Puppet is currently enabled, last run 50 seconds ago with 0 failures [13:30:48] RECOVERY - puppet last run on db1069 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [13:30:48] RECOVERY - puppet last run on mw1111 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [13:30:48] RECOVERY - puppet last run on wtp1023 is OK: OK: Puppet is currently enabled, last run 53 seconds ago with 0 failures [13:30:49] RECOVERY - puppet last run on mw1116 is OK: OK: Puppet is currently enabled, last run 14 seconds ago with 0 failures [13:30:49] RECOVERY - puppet last run on mw1206 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [13:30:50] RECOVERY - puppet last run on mw1168 is OK: OK: Puppet is currently enabled, last run 42 seconds ago with 0 failures [13:30:56] RECOVERY - puppet last run on mw1029 is OK: OK: Puppet is currently enabled, last run 14 seconds ago with 0 failures [13:30:56] RECOVERY - puppet last run on mc1001 is OK: OK: Puppet is currently enabled, last run 27 seconds ago with 0 failures [13:30:56] RECOVERY - puppet last run on db1071 is OK: OK: Puppet is currently enabled, last run 47 seconds ago with 0 failures [13:30:56] RECOVERY - puppet last run on mw1014 is OK: OK: Puppet is currently enabled, last run 38 seconds ago with 0 failures [13:30:56] RECOVERY - puppet last run on virt1004 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [13:30:57] RECOVERY - puppet last run on mw1190 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [13:30:57] RECOVERY - puppet last run on ms-be1009 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [13:30:57] RECOVERY - puppet last run on mw1227 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [13:30:58] RECOVERY - puppet last run on mw1163 is OK: OK: Puppet is currently enabled, last run 33 seconds ago with 0 failures [13:30:58] RECOVERY - puppet last run on ms-be2008 is OK: OK: Puppet is currently enabled, last run 21 seconds ago with 0 failures [13:31:03] RECOVERY - puppet last run on search1023 is OK: OK: Puppet is currently enabled, last run 10 seconds ago with 0 failures [13:31:03] RECOVERY - puppet last run on mw1125 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [13:31:03] RECOVERY - puppet last run on ssl1005 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [13:31:04] RECOVERY - puppet last run on ssl1009 is OK: OK: Puppet is currently enabled, last run 55 seconds ago with 0 failures [13:31:04] RECOVERY - puppet last run on ms-be2001 is OK: OK: Puppet is currently enabled, last run 23 seconds ago with 0 failures [13:31:04] RECOVERY - puppet last run on mw1165 is OK: OK: Puppet is currently enabled, last run 37 seconds ago with 0 failures [13:31:04] RECOVERY - puppet last run on db2001 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [13:31:05] RECOVERY - puppet last run on mw1081 is OK: OK: Puppet is currently enabled, last run 27 seconds ago with 0 failures [13:31:05] RECOVERY - puppet last run on mw1156 is OK: OK: Puppet is currently enabled, last run 44 seconds ago with 0 failures [13:31:15] RECOVERY - puppet last run on amssq56 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [13:31:15] RECOVERY - puppet last run on mw1074 is OK: OK: Puppet is currently enabled, last run 38 seconds ago with 0 failures [13:31:15] RECOVERY - puppet last run on hafnium is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [13:31:15] RECOVERY - puppet last run on search1017 is OK: OK: Puppet is currently enabled, last run 43 seconds ago with 0 failures [13:31:15] RECOVERY - puppet last run on db1004 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [13:31:16] RECOVERY - puppet last run on mw1183 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [13:31:16] RECOVERY - puppet last run on mw1188 is OK: OK: Puppet is currently enabled, last run 7 seconds ago with 0 failures [13:31:16] RECOVERY - puppet last run on mw1004 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [13:31:33] RECOVERY - puppet last run on mw1146 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [13:31:37] RECOVERY - puppet last run on mw1034 is OK: OK: Puppet is currently enabled, last run 52 seconds ago with 0 failures [13:31:38] RECOVERY - puppet last run on db1001 is OK: OK: Puppet is currently enabled, last run 44 seconds ago with 0 failures [13:31:40] RECOVERY - puppet last run on db2037 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [13:31:41] RECOVERY - puppet last run on elastic1015 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [13:31:41] RECOVERY - puppet last run on db2023 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [13:31:41] RECOVERY - puppet last run on mw1050 is OK: OK: Puppet is currently enabled, last run 57 seconds ago with 0 failures [13:31:41] RECOVERY - puppet last run on wtp1011 is OK: OK: Puppet is currently enabled, last run 34 seconds ago with 0 failures [13:31:41] RECOVERY - puppet last run on db2004 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [13:31:42] RECOVERY - puppet last run on wtp1018 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [13:31:42] RECOVERY - puppet last run on elastic1014 is OK: OK: Puppet is currently enabled, last run 58 seconds ago with 0 failures [13:31:43] RECOVERY - puppet last run on acamar is OK: OK: Puppet is currently enabled, last run 28 seconds ago with 0 failures [13:31:43] RECOVERY - puppet last run on db2003 is OK: OK: Puppet is currently enabled, last run 22 seconds ago with 0 failures [13:31:44] RECOVERY - puppet last run on analytics1023 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [13:31:44] RECOVERY - puppet last run on mw1053 is OK: OK: Puppet is currently enabled, last run 57 seconds ago with 0 failures [13:31:45] RECOVERY - puppet last run on search1015 is OK: OK: Puppet is currently enabled, last run 16 seconds ago with 0 failures [13:31:45] RECOVERY - puppet last run on rubidium is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [13:31:46] RECOVERY - puppet last run on mw1181 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [13:31:46] RECOVERY - puppet last run on snapshot1004 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [13:31:47] RECOVERY - puppet last run on hooft is OK: OK: Puppet is currently enabled, last run 20 seconds ago with 0 failures [13:31:54] RECOVERY - puppet last run on lvs1001 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [13:31:55] RECOVERY - puppet last run on virt1003 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [13:31:55] RECOVERY - puppet last run on amssq42 is OK: OK: Puppet is currently enabled, last run 44 seconds ago with 0 failures [13:31:55] RECOVERY - puppet last run on mw1159 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [13:31:55] RECOVERY - puppet last run on amssq34 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [13:31:55] RECOVERY - puppet last run on amssq51 is OK: OK: Puppet is currently enabled, last run 53 seconds ago with 0 failures [13:31:56] RECOVERY - puppet last run on mw1171 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [13:31:56] RECOVERY - puppet last run on labsdb1006 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [13:31:57] RECOVERY - puppet last run on wtp1002 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [13:31:57] RECOVERY - puppet last run on mw1180 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [13:31:58] RECOVERY - puppet last run on mw1087 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [13:31:58] RECOVERY - puppet last run on ms-be2012 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [13:31:59] RECOVERY - puppet last run on ssl3002 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [13:32:04] RECOVERY - puppet last run on labsdb1004 is OK: OK: Puppet is currently enabled, last run 44 seconds ago with 0 failures [13:32:04] RECOVERY - puppet last run on db1057 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [13:32:04] RECOVERY - puppet last run on osmium is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [13:32:04] RECOVERY - puppet last run on db1055 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [13:32:05] RECOVERY - puppet last run on mw1051 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [13:32:05] RECOVERY - puppet last run on wtp1004 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [13:32:05] RECOVERY - puppet last run on ms-be1007 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [13:32:06] RECOVERY - puppet last run on cp1048 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [13:32:15] RECOVERY - puppet last run on protactinium is OK: OK: Puppet is currently enabled, last run 54 seconds ago with 0 failures [13:32:17] RECOVERY - puppet last run on wtp1013 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [13:32:17] RECOVERY - puppet last run on wtp1015 is OK: OK: Puppet is currently enabled, last run 55 seconds ago with 0 failures [13:32:17] RECOVERY - puppet last run on virt1007 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [13:32:18] RECOVERY - puppet last run on elastic1017 is OK: OK: Puppet is currently enabled, last run 32 seconds ago with 0 failures [13:32:18] RECOVERY - puppet last run on ms-be2005 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [13:32:18] RECOVERY - puppet last run on ms-be2007 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [13:32:23] RECOVERY - puppet last run on db1063 is OK: OK: Puppet is currently enabled, last run 10 seconds ago with 0 failures [13:32:23] RECOVERY - puppet last run on mc1013 is OK: OK: Puppet is currently enabled, last run 56 seconds ago with 0 failures [13:32:23] RECOVERY - puppet last run on mw1225 is OK: OK: Puppet is currently enabled, last run 22 seconds ago with 0 failures [13:32:23] RECOVERY - puppet last run on lvs3004 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [13:32:23] RECOVERY - puppet last run on mw1167 is OK: OK: Puppet is currently enabled, last run 3 seconds ago with 0 failures [13:32:24] RECOVERY - puppet last run on mw1210 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [13:32:24] RECOVERY - puppet last run on cp1060 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [13:32:25] RECOVERY - puppet last run on mw1097 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [13:32:34] RECOVERY - puppet last run on wtp1022 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [13:32:35] RECOVERY - puppet last run on thallium is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [13:32:35] RECOVERY - puppet last run on berkelium is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [13:32:35] RECOVERY - puppet last run on elastic1011 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [13:32:35] RECOVERY - puppet last run on mw1023 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [13:32:35] RECOVERY - puppet last run on ssl1008 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [13:32:36] RECOVERY - puppet last run on search1024 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [13:32:36] RECOVERY - puppet last run on ocg1003 is OK: OK: Puppet is currently enabled, last run 19 seconds ago with 0 failures [13:32:37] RECOVERY - puppet last run on analytics1032 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [13:32:43] RECOVERY - puppet last run on elastic1023 is OK: OK: Puppet is currently enabled, last run 3 seconds ago with 0 failures [13:32:43] RECOVERY - puppet last run on rhenium is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [13:32:43] RECOVERY - puppet last run on rcs1001 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [13:32:43] RECOVERY - puppet last run on elastic1002 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [13:32:43] RECOVERY - puppet last run on rdb1003 is OK: OK: Puppet is currently enabled, last run 32 seconds ago with 0 failures [13:32:44] RECOVERY - puppet last run on cp3012 is OK: OK: Puppet is currently enabled, last run 10 seconds ago with 0 failures [13:32:44] RECOVERY - puppet last run on ms-fe3001 is OK: OK: Puppet is currently enabled, last run 2 seconds ago with 0 failures [13:32:45] RECOVERY - puppet last run on amssq41 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [13:32:45] RECOVERY - puppet last run on cp3009 is OK: OK: Puppet is currently enabled, last run 50 seconds ago with 0 failures [13:32:46] RECOVERY - puppet last run on amssq62 is OK: OK: Puppet is currently enabled, last run 46 seconds ago with 0 failures [13:32:46] RECOVERY - puppet last run on mw1229 is OK: OK: Puppet is currently enabled, last run 27 seconds ago with 0 failures [13:32:47] RECOVERY - puppet last run on amslvs3 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [13:32:53] RECOVERY - puppet last run on mw1248 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [13:32:54] RECOVERY - puppet last run on labmon1001 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [13:32:54] RECOVERY - puppet last run on cp1038 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [13:32:54] RECOVERY - puppet last run on rdb1002 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [13:32:54] RECOVERY - puppet last run on mw1030 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [13:32:54] RECOVERY - puppet last run on mw1043 is OK: OK: Puppet is currently enabled, last run 9 seconds ago with 0 failures [13:32:54] RECOVERY - puppet last run on mw1239 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [13:33:11] RECOVERY - puppet last run on elastic1005 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [13:33:13] RECOVERY - puppet last run on db1064 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [13:33:13] RECOVERY - puppet last run on oxygen is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [13:33:14] RECOVERY - puppet last run on mw1198 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [13:33:16] RECOVERY - puppet last run on stat1002 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [13:33:17] RECOVERY - puppet last run on db1030 is OK: OK: Puppet is currently enabled, last run 30 seconds ago with 0 failures [13:33:17] RECOVERY - puppet last run on mw1032 is OK: OK: Puppet is currently enabled, last run 49 seconds ago with 0 failures [13:33:18] RECOVERY - puppet last run on db1070 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [13:33:18] RECOVERY - puppet last run on mc1007 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [13:33:18] RECOVERY - puppet last run on mw1077 is OK: OK: Puppet is currently enabled, last run 36 seconds ago with 0 failures [13:33:28] RECOVERY - puppet last run on rcs1002 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [13:33:28] RECOVERY - puppet last run on mw1186 is OK: OK: Puppet is currently enabled, last run 41 seconds ago with 0 failures [13:33:28] RECOVERY - puppet last run on wtp1003 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [13:33:28] RECOVERY - puppet last run on pc1003 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [13:33:39] RECOVERY - puppet last run on lvs2003 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [13:33:39] RECOVERY - puppet last run on db1044 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [13:33:39] RECOVERY - puppet last run on mw1148 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [13:33:40] RECOVERY - puppet last run on db1054 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [13:33:50] RECOVERY - puppet last run on mw1243 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [13:33:51] RECOVERY - puppet last run on es1002 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [13:33:55] RECOVERY - puppet last run on titanium is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [13:34:08] RECOVERY - puppet last run on wtp1007 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [13:34:08] RECOVERY - puppet last run on nitrogen is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [13:34:19] RECOVERY - puppet last run on mw1122 is OK: OK: Puppet is currently enabled, last run 8 seconds ago with 0 failures [13:34:19] RECOVERY - puppet last run on tungsten is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [13:34:19] RECOVERY - puppet last run on analytics1014 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [13:34:19] RECOVERY - puppet last run on mw1091 is OK: OK: Puppet is currently enabled, last run 11 seconds ago with 0 failures [13:34:19] RECOVERY - puppet last run on analytics1011 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [13:34:19] RECOVERY - puppet last run on mw1209 is OK: OK: Puppet is currently enabled, last run 21 seconds ago with 0 failures [13:34:20] RECOVERY - puppet last run on db2028 is OK: OK: Puppet is currently enabled, last run 23 seconds ago with 0 failures [13:34:20] RECOVERY - puppet last run on mw1201 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [13:34:28] RECOVERY - puppet last run on mw1105 is OK: OK: Puppet is currently enabled, last run 21 seconds ago with 0 failures [13:34:30] RECOVERY - puppet last run on mw1024 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [13:34:35] RECOVERY - puppet last run on db1061 is OK: OK: Puppet is currently enabled, last run 54 seconds ago with 0 failures [13:34:39] RECOVERY - puppet last run on mw1001 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [13:34:39] RECOVERY - puppet last run on mw1185 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [13:34:40] RECOVERY - puppet last run on fluorine is OK: OK: Puppet is currently enabled, last run 28 seconds ago with 0 failures [13:34:51] RECOVERY - puppet last run on iodine is OK: OK: Puppet is currently enabled, last run 3 seconds ago with 0 failures [13:34:51] RECOVERY - puppet last run on analytics1028 is OK: OK: Puppet is currently enabled, last run 49 seconds ago with 0 failures [13:34:51] (03PS1) 10Yuvipanda: tools: Add postgis package to exec_environ [puppet] - 10https://gerrit.wikimedia.org/r/176350 [13:34:51] RECOVERY - puppet last run on mw1231 is OK: OK: Puppet is currently enabled, last run 46 seconds ago with 0 failures [13:34:51] RECOVERY - puppet last run on copper is OK: OK: Puppet is currently enabled, last run 47 seconds ago with 0 failures [13:34:51] RECOVERY - puppet last run on mw1022 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [13:34:59] RECOVERY - puppet last run on ssl1007 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [13:35:00] RECOVERY - puppet last run on search1004 is OK: OK: Puppet is currently enabled, last run 5 seconds ago with 0 failures [13:35:21] RECOVERY - puppet last run on mw1108 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [13:35:21] RECOVERY - puppet last run on ssl1006 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [13:35:21] RECOVERY - puppet last run on cp1044 is OK: OK: Puppet is currently enabled, last run 40 seconds ago with 0 failures [13:35:21] RECOVERY - puppet last run on db1038 is OK: OK: Puppet is currently enabled, last run 15 seconds ago with 0 failures [13:35:22] RECOVERY - puppet last run on mw1236 is OK: OK: Puppet is currently enabled, last run 8 seconds ago with 0 failures [13:35:22] RECOVERY - puppet last run on mw1064 is OK: OK: Puppet is currently enabled, last run 8 seconds ago with 0 failures [13:35:22] RECOVERY - puppet last run on elastic1020 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [13:35:23] RECOVERY - puppet last run on cp1037 is OK: OK: Puppet is currently enabled, last run 6 seconds ago with 0 failures [13:35:31] RECOVERY - puppet last run on db1037 is OK: OK: Puppet is currently enabled, last run 15 seconds ago with 0 failures [13:35:31] RECOVERY - puppet last run on osm-cp1001 is OK: OK: Puppet is currently enabled, last run 27 seconds ago with 0 failures [13:35:31] RECOVERY - puppet last run on mw1066 is OK: OK: Puppet is currently enabled, last run 15 seconds ago with 0 failures [13:35:31] RECOVERY - puppet last run on analytics1004 is OK: OK: Puppet is currently enabled, last run 30 seconds ago with 0 failures [13:35:31] RECOVERY - puppet last run on mw1204 is OK: OK: Puppet is currently enabled, last run 26 seconds ago with 0 failures [13:35:32] RECOVERY - puppet last run on ssl3001 is OK: OK: Puppet is currently enabled, last run 15 seconds ago with 0 failures [13:35:32] RECOVERY - puppet last run on mw1142 is OK: OK: Puppet is currently enabled, last run 33 seconds ago with 0 failures [13:35:33] RECOVERY - puppet last run on search1011 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [13:35:33] RECOVERY - puppet last run on amssq38 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [13:35:34] RECOVERY - puppet last run on calcium is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [13:35:34] RECOVERY - puppet last run on virt1000 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [13:35:41] RECOVERY - puppet last run on mw1121 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [13:35:43] RECOVERY - puppet last run on db1072 is OK: OK: Puppet is currently enabled, last run 50 seconds ago with 0 failures [13:35:44] RECOVERY - puppet last run on analytics1018 is OK: OK: Puppet is currently enabled, last run 40 seconds ago with 0 failures [13:35:44] RECOVERY - puppet last run on cp1052 is OK: OK: Puppet is currently enabled, last run 33 seconds ago with 0 failures [13:35:44] RECOVERY - puppet last run on mw1255 is OK: OK: Puppet is currently enabled, last run 13 seconds ago with 0 failures [13:35:44] RECOVERY - puppet last run on search1022 is OK: OK: Puppet is currently enabled, last run 41 seconds ago with 0 failures [13:35:44] RECOVERY - puppet last run on ms-fe1002 is OK: OK: Puppet is currently enabled, last run 35 seconds ago with 0 failures [13:35:45] RECOVERY - puppet last run on search1013 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [13:35:45] RECOVERY - puppet last run on lvs1004 is OK: OK: Puppet is currently enabled, last run 12 seconds ago with 0 failures [13:35:46] RECOVERY - puppet last run on mw1016 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [13:35:51] RECOVERY - puppet last run on ssl1001 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [13:35:52] RECOVERY - puppet last run on mw1010 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [13:35:52] RECOVERY - puppet last run on dbstore1002 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [13:35:53] RECOVERY - puppet last run on mw1219 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [13:35:53] RECOVERY - puppet last run on db1033 is OK: OK: Puppet is currently enabled, last run 59 seconds ago with 0 failures [13:35:53] RECOVERY - puppet last run on es1010 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [13:35:53] RECOVERY - puppet last run on mw1090 is OK: OK: Puppet is currently enabled, last run 45 seconds ago with 0 failures [13:35:53] RECOVERY - puppet last run on mw1086 is OK: OK: Puppet is currently enabled, last run 43 seconds ago with 0 failures [13:36:05] RECOVERY - puppet last run on strontium is OK: OK: Puppet is currently enabled, last run 57 seconds ago with 0 failures [13:36:07] RECOVERY - puppet last run on wtp1008 is OK: OK: Puppet is currently enabled, last run 59 seconds ago with 0 failures [13:36:13] RECOVERY - puppet last run on db1047 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [13:36:18] RECOVERY - puppet last run on db1011 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [13:36:18] RECOVERY - puppet last run on mw1071 is OK: OK: Puppet is currently enabled, last run 51 seconds ago with 0 failures [13:36:18] RECOVERY - puppet last run on mw1093 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [13:36:18] RECOVERY - puppet last run on rbf1001 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [13:36:18] RECOVERY - puppet last run on db1006 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [13:36:19] RECOVERY - puppet last run on mw1223 is OK: OK: Puppet is currently enabled, last run 50 seconds ago with 0 failures [13:36:19] RECOVERY - puppet last run on elastic1003 is OK: OK: Puppet is currently enabled, last run 56 seconds ago with 0 failures [13:36:28] RECOVERY - puppet last run on mw1241 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [13:36:28] RECOVERY - puppet last run on bast1001 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [13:36:28] RECOVERY - puppet last run on virt1008 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [13:36:28] RECOVERY - puppet last run on mw1135 is OK: OK: Puppet is currently enabled, last run 23 seconds ago with 0 failures [13:36:28] RECOVERY - puppet last run on mw1139 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [13:36:29] RECOVERY - puppet last run on cp4009 is OK: OK: Puppet is currently enabled, last run 10 seconds ago with 0 failures [13:36:29] RECOVERY - puppet last run on analytics1027 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [13:36:30] RECOVERY - puppet last run on cp1067 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [13:36:30] RECOVERY - puppet last run on mw1193 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [13:36:31] RECOVERY - puppet last run on radium is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [13:36:31] RECOVERY - puppet last run on search1012 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [13:36:32] RECOVERY - puppet last run on search1006 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [13:36:32] RECOVERY - puppet last run on db2030 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [13:36:39] RECOVERY - puppet last run on cp3004 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [13:36:42] RECOVERY - puppet last run on labsdb1005 is OK: OK: Puppet is currently enabled, last run 5 seconds ago with 0 failures [13:36:42] RECOVERY - puppet last run on mw1033 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [13:36:42] RECOVERY - puppet last run on chromium is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [13:36:42] RECOVERY - puppet last run on cp1040 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [13:36:42] RECOVERY - puppet last run on terbium is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [13:36:43] RECOVERY - puppet last run on wtp1010 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [13:37:00] RECOVERY - puppet last run on hydrogen is OK: OK: Puppet is currently enabled, last run 2 seconds ago with 0 failures [13:37:00] RECOVERY - puppet last run on mw1107 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [13:37:00] RECOVERY - puppet last run on ms-be1011 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [13:37:00] RECOVERY - puppet last run on achernar is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [13:37:08] RECOVERY - puppet last run on mw1215 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [13:37:08] RECOVERY - puppet last run on sca1001 is OK: OK: Puppet is currently enabled, last run 17 seconds ago with 0 failures [13:37:09] RECOVERY - puppet last run on ms-fe2002 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [13:37:09] RECOVERY - puppet last run on db2012 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [13:37:09] RECOVERY - puppet last run on ms-be1015 is OK: OK: Puppet is currently enabled, last run 26 seconds ago with 0 failures [13:37:10] RECOVERY - puppet last run on cp4002 is OK: OK: Puppet is currently enabled, last run 51 seconds ago with 0 failures [13:37:10] RECOVERY - puppet last run on db1049 is OK: OK: Puppet is currently enabled, last run 6 seconds ago with 0 failures [13:37:18] RECOVERY - puppet last run on stat1003 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [13:37:18] RECOVERY - puppet last run on mw1220 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [13:37:18] RECOVERY - puppet last run on mw1203 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [13:37:18] RECOVERY - puppet last run on ms-be1002 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [13:37:18] RECOVERY - puppet last run on mw1194 is OK: OK: Puppet is currently enabled, last run 9 seconds ago with 0 failures [13:37:19] RECOVERY - puppet last run on lvs3003 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [13:37:19] RECOVERY - puppet last run on amssq31 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [13:37:20] RECOVERY - puppet last run on cp3018 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [13:37:24] (03CR) 10Yuvipanda: [C: 032] tools: Add postgis package to exec_environ [puppet] - 10https://gerrit.wikimedia.org/r/176350 (owner: 10Yuvipanda) [13:37:28] RECOVERY - puppet last run on mw1152 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [13:37:43] RECOVERY - puppet last run on cp1045 is OK: OK: Puppet is currently enabled, last run 37 seconds ago with 0 failures [13:37:45] RECOVERY - puppet last run on mw1131 is OK: OK: Puppet is currently enabled, last run 4 seconds ago with 0 failures [13:37:45] RECOVERY - puppet last run on mercury is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [13:37:47] RECOVERY - puppet last run on elastic1029 is OK: OK: Puppet is currently enabled, last run 35 seconds ago with 0 failures [13:37:48] RECOVERY - puppet last run on analytics1021 is OK: OK: Puppet is currently enabled, last run 18 seconds ago with 0 failures [13:37:48] RECOVERY - puppet last run on ms-be1014 is OK: OK: Puppet is currently enabled, last run 14 seconds ago with 0 failures [13:37:48] RECOVERY - puppet last run on mw1112 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [13:37:49] RECOVERY - puppet last run on mw1037 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [13:37:49] RECOVERY - puppet last run on lvs4004 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [13:37:49] RECOVERY - puppet last run on cp4020 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [13:37:49] RECOVERY - puppet last run on cp4012 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [13:37:50] RECOVERY - puppet last run on lvs2002 is OK: OK: Puppet is currently enabled, last run 4 seconds ago with 0 failures [13:37:50] RECOVERY - puppet last run on baham is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [13:37:58] RECOVERY - puppet last run on db1056 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [13:37:58] RECOVERY - puppet last run on mw1027 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [13:37:58] RECOVERY - puppet last run on lvs1003 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [13:37:59] RECOVERY - puppet last run on db1005 is OK: OK: Puppet is currently enabled, last run 49 seconds ago with 0 failures [13:37:59] RECOVERY - puppet last run on mw1085 is OK: OK: Puppet is currently enabled, last run 19 seconds ago with 0 failures [13:37:59] RECOVERY - puppet last run on es1004 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [13:37:59] RECOVERY - puppet last run on analytics1029 is OK: OK: Puppet is currently enabled, last run 3 seconds ago with 0 failures [13:38:00] RECOVERY - puppet last run on mw1104 is OK: OK: Puppet is currently enabled, last run 38 seconds ago with 0 failures [13:38:00] RECOVERY - puppet last run on mw1143 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [13:38:08] RECOVERY - puppet last run on mw1113 is OK: OK: Puppet is currently enabled, last run 55 seconds ago with 0 failures [13:38:08] RECOVERY - puppet last run on dysprosium is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [13:38:19] RECOVERY - puppet last run on mc1015 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [13:38:27] RECOVERY - puppet last run on eeden is OK: OK: Puppet is currently enabled, last run 58 seconds ago with 0 failures [13:38:27] RECOVERY - puppet last run on cp3011 is OK: OK: Puppet is currently enabled, last run 1 second ago with 0 failures [13:38:27] RECOVERY - puppet last run on ms-be3004 is OK: OK: Puppet is currently enabled, last run 17 seconds ago with 0 failures [13:38:32] RECOVERY - puppet last run on mw1230 is OK: OK: Puppet is currently enabled, last run 22 seconds ago with 0 failures [13:38:32] RECOVERY - puppet last run on amssq39 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [13:38:32] RECOVERY - puppet last run on amssq33 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [13:38:32] RECOVERY - puppet last run on lanthanum is OK: OK: Puppet is currently enabled, last run 54 seconds ago with 0 failures [13:38:33] RECOVERY - puppet last run on search1003 is OK: OK: Puppet is currently enabled, last run 59 seconds ago with 0 failures [13:38:33] RECOVERY - puppet last run on mw1078 is OK: OK: Puppet is currently enabled, last run 17 seconds ago with 0 failures [13:38:33] RECOVERY - puppet last run on elastic1025 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [13:38:34] RECOVERY - puppet last run on mw1047 is OK: OK: Puppet is currently enabled, last run 54 seconds ago with 0 failures [13:38:34] RECOVERY - puppet last run on mw1021 is OK: OK: Puppet is currently enabled, last run 56 seconds ago with 0 failures [13:38:35] RECOVERY - puppet last run on labsdb1002 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [13:38:35] RECOVERY - puppet last run on mw1110 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [13:38:42] RECOVERY - puppet last run on mw1073 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [13:38:43] RECOVERY - puppet last run on mw1075 is OK: OK: Puppet is currently enabled, last run 39 seconds ago with 0 failures [13:38:43] RECOVERY - puppet last run on mw1207 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [13:38:43] RECOVERY - puppet last run on mw1158 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [13:38:43] RECOVERY - puppet last run on ms-be1010 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [13:38:43] RECOVERY - puppet last run on tmh1001 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [13:38:44] RECOVERY - puppet last run on elastic1010 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [13:38:44] RECOVERY - puppet last run on cp1053 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [13:38:52] RECOVERY - puppet last run on radon is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [13:38:52] RECOVERY - puppet last run on wtp1001 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [13:38:53] RECOVERY - puppet last run on db1009 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [13:38:53] RECOVERY - puppet last run on mw1253 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [13:39:05] RECOVERY - puppet last run on ssl1003 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [13:39:08] RECOVERY - puppet last run on tmh1002 is OK: OK: Puppet is currently enabled, last run 1 second ago with 0 failures [13:39:09] RECOVERY - puppet last run on analytics1024 is OK: OK: Puppet is currently enabled, last run 36 seconds ago with 0 failures [13:39:10] RECOVERY - puppet last run on cp1054 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [13:39:10] RECOVERY - puppet last run on zinc is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [13:39:11] RECOVERY - puppet last run on db1068 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [13:39:12] RECOVERY - puppet last run on mw1128 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [13:39:23] RECOVERY - puppet last run on ocg1002 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [13:39:23] RECOVERY - puppet last run on virt1009 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [13:39:24] RECOVERY - puppet last run on db1027 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [13:39:24] RECOVERY - puppet last run on search1008 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [13:39:24] RECOVERY - puppet last run on lvs1006 is OK: OK: Puppet is currently enabled, last run 26 seconds ago with 0 failures [13:39:24] RECOVERY - puppet last run on db1035 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [13:39:24] RECOVERY - puppet last run on analytics1031 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [13:39:25] RECOVERY - puppet last run on es1003 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [13:39:25] RECOVERY - puppet last run on db2010 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [13:39:26] RECOVERY - puppet last run on lvs2005 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [13:39:26] RECOVERY - puppet last run on cp4013 is OK: OK: Puppet is currently enabled, last run 17 seconds ago with 0 failures [13:39:27] RECOVERY - puppet last run on cp4011 is OK: OK: Puppet is currently enabled, last run 56 seconds ago with 0 failures [13:39:27] RECOVERY - puppet last run on mw1103 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [13:39:28] RECOVERY - puppet last run on mw1155 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [13:39:28] RECOVERY - puppet last run on ssl1004 is OK: OK: Puppet is currently enabled, last run 53 seconds ago with 0 failures [13:39:32] RECOVERY - puppet last run on pollux is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [13:39:32] RECOVERY - puppet last run on mw1058 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [13:39:33] RECOVERY - puppet last run on cp3021 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [13:39:33] RECOVERY - puppet last run on cp3005 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [13:39:42] RECOVERY - puppet last run on search1021 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [13:39:50] RECOVERY - puppet last run on cp1057 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [13:39:51] RECOVERY - puppet last run on elastic1016 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [13:39:52] RECOVERY - puppet last run on analytics1012 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [13:39:53] RECOVERY - puppet last run on mw1102 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [13:39:53] RECOVERY - puppet last run on mw1137 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [13:39:53] RECOVERY - puppet last run on search1009 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [13:39:53] RECOVERY - puppet last run on search1019 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [13:39:53] RECOVERY - puppet last run on labsdb1007 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [13:39:54] RECOVERY - puppet last run on db1045 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [13:40:02] RECOVERY - puppet last run on praseodymium is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [13:40:02] RECOVERY - puppet last run on es1005 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [13:40:12] RECOVERY - puppet last run on es2006 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [13:40:12] RECOVERY - puppet last run on ms-be2009 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [13:40:12] RECOVERY - puppet last run on db2017 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [13:40:12] RECOVERY - puppet last run on ms-be2010 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [13:40:13] RECOVERY - puppet last run on cp4015 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [13:40:13] RECOVERY - puppet last run on bast2001 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [13:40:13] RECOVERY - puppet last run on mw1191 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [13:40:14] RECOVERY - puppet last run on mw1017 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [13:40:14] RECOVERY - puppet last run on mw1154 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [13:40:23] RECOVERY - puppet last run on virt1005 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [13:40:23] RECOVERY - puppet last run on amssq37 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [13:40:23] RECOVERY - puppet last run on cp3007 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [13:40:23] RECOVERY - puppet last run on cp1066 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [13:40:32] RECOVERY - puppet last run on amssq50 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [13:40:34] RECOVERY - puppet last run on db1058 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [13:40:34] RECOVERY - puppet last run on mw1101 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [13:40:34] RECOVERY - puppet last run on db1019 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [13:40:34] RECOVERY - puppet last run on mw1136 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [13:40:50] RECOVERY - puppet last run on mw1199 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [13:40:51] RECOVERY - puppet last run on cp4016 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [13:40:51] RECOVERY - puppet last run on cp4010 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [13:41:03] RECOVERY - puppet last run on mw1019 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [13:41:18] RECOVERY - puppet last run on cp3017 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [13:41:22] RECOVERY - puppet last run on cp3019 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [13:41:22] RECOVERY - puppet last run on amssq45 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [13:41:23] RECOVERY - puppet last run on mw1083 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [13:41:23] RECOVERY - puppet last run on wtp1014 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [13:41:44] RECOVERY - puppet last run on erbium is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [13:42:22] RECOVERY - puppet last run on cp4017 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [13:42:35] RECOVERY - puppet last run on mw1094 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [13:42:59] RECOVERY - puppet last run on neon is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [13:42:59] RECOVERY - puppet last run on ms-be2015 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [13:44:44] greg-g: hey, do you have a second? [14:02:33] PROBLEM - Unmerged changes on repository puppet on strontium is CRITICAL: There is one unmerged change in puppet (dir /var/lib/git/operations/puppet). [14:04:12] PROBLEM - Unmerged changes on repository puppet on palladium is CRITICAL: There is one unmerged change in puppet (dir /var/lib/git/operations/puppet). [14:10:14] <_joe_> YuviPanda ^^ [14:10:55] bah [14:11:01] forgot to hit 'yes' [14:11:02] sorry [14:12:22] RECOVERY - Unmerged changes on repository puppet on palladium is OK: No changes to merge. [14:13:22] RECOVERY - Unmerged changes on repository puppet on strontium is OK: No changes to merge. [14:43:24] PROBLEM - puppet last run on bast2001 is CRITICAL: CRITICAL: puppet fail [14:59:08] RECOVERY - puppet last run on bast2001 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [15:19:14] PROBLEM - HHVM busy threads on mw1230 is CRITICAL: CRITICAL: 11.11% of data above the critical threshold [115.2] [15:22:06] RECOVERY - HHVM busy threads on mw1230 is OK: OK: Less than 1.00% above the threshold [76.8] [15:39:26] (03CR) 10Hashar: [C: 031] "Thanks for the explanation on https://phabricator.wikimedia.org/T1295 . I guess we will then be able to drop the udplog instance :]" [puppet] - 10https://gerrit.wikimedia.org/r/176191 (owner: 10BryanDavis) [15:48:22] (03Abandoned) 10BBlack: Enable (non-EC)DHE key exchange in compat cipher list [puppet] - 10https://gerrit.wikimedia.org/r/170879 (owner: 10BBlack) [15:48:33] (03PS4) 10BBlack: Remove old protoproxy / ssl[13]00x config / star certs [puppet] - 10https://gerrit.wikimedia.org/r/175466 [16:04:32] PROBLEM - HHVM busy threads on mw1231 is CRITICAL: CRITICAL: 10.00% of data above the critical threshold [115.2] [16:04:42] PROBLEM - HHVM busy threads on mw1221 is CRITICAL: CRITICAL: 11.11% of data above the critical threshold [115.2] [16:07:12] RECOVERY - HHVM busy threads on mw1231 is OK: OK: Less than 1.00% above the threshold [76.8] [16:07:23] RECOVERY - HHVM busy threads on mw1221 is OK: OK: Less than 1.00% above the threshold [76.8] [16:43:58] PROBLEM - HTTP 5xx req/min on tungsten is CRITICAL: CRITICAL: 13.33% of data above the critical threshold [500.0] [16:54:31] RECOVERY - HTTP 5xx req/min on tungsten is OK: OK: Less than 1.00% above the threshold [250.0] [16:55:02] PROBLEM - puppet last run on virt1006 is CRITICAL: CRITICAL: Puppet has 1 failures [17:08:33] RECOVERY - puppet last run on virt1006 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [17:18:13] (03PS2) 10Giuseppe Lavagetto: hiera: role-based backend, role keyword [puppet] - 10https://gerrit.wikimedia.org/r/176334 [17:23:55] (03CR) 10Ebrahim: "Thank you" [debs/librsvg] - 10https://gerrit.wikimedia.org/r/173639 (owner: 10Ebrahim) [19:39:26] PROBLEM - HHVM busy threads on mw1114 is CRITICAL: CRITICAL: 11.11% of data above the critical threshold [86.4] [19:40:05] PROBLEM - puppet last run on cp3004 is CRITICAL: CRITICAL: puppet fail [19:53:32] RECOVERY - puppet last run on cp3004 is OK: OK: Puppet is currently enabled, last run 25 seconds ago with 0 failures [19:55:45] RECOVERY - HHVM busy threads on mw1114 is OK: OK: Less than 1.00% above the threshold [57.6] [20:06:44] PROBLEM - HHVM busy threads on mw1114 is CRITICAL: CRITICAL: 11.11% of data above the critical threshold [86.4] [20:20:07] RECOVERY - HHVM busy threads on mw1114 is OK: OK: Less than 1.00% above the threshold [57.6] [20:55:02] PROBLEM - HHVM busy threads on mw1114 is CRITICAL: CRITICAL: 10.00% of data above the critical threshold [86.4] [21:06:16] RECOVERY - HHVM busy threads on mw1114 is OK: OK: Less than 1.00% above the threshold [57.6] [22:04:09] (03CR) 10Ori.livneh: "What about this did you find obtrusive?" [puppet] - 10https://gerrit.wikimedia.org/r/176307 (owner: 10Tim Starling) [22:16:05] PROBLEM - puppet last run on virt1001 is CRITICAL: CRITICAL: Puppet has 1 failures [22:23:28] PROBLEM - HTTP 5xx req/min on tungsten is CRITICAL: CRITICAL: 6.67% of data above the critical threshold [500.0] [22:29:54] RECOVERY - puppet last run on virt1001 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [22:31:40] RECOVERY - HTTP 5xx req/min on tungsten is OK: OK: Less than 1.00% above the threshold [250.0] [23:00:06] PROBLEM - puppet last run on ms-be1009 is CRITICAL: CRITICAL: Puppet has 1 failures [23:01:00] PROBLEM - puppet last run on mw1097 is CRITICAL: CRITICAL: Puppet has 1 failures [23:01:14] PROBLEM - puppet last run on labsdb1004 is CRITICAL: CRITICAL: Puppet has 1 failures [23:01:32] PROBLEM - puppet last run on mw1198 is CRITICAL: CRITICAL: Puppet has 1 failures [23:01:42] PROBLEM - puppet last run on titanium is CRITICAL: CRITICAL: Puppet has 1 failures [23:02:10] PROBLEM - puppet last run on search1015 is CRITICAL: CRITICAL: Puppet has 1 failures [23:03:06] PROBLEM - puppet last run on cp1038 is CRITICAL: CRITICAL: Puppet has 1 failures [23:04:32] (03PS1) 10Ori.livneh: HHVM: provision hhvm-tidy and load tidy.so [puppet] - 10https://gerrit.wikimedia.org/r/176385 [23:07:18] (03CR) 10Ori.livneh: [C: 032 V: 032] HHVM: provision hhvm-tidy and load tidy.so [puppet] - 10https://gerrit.wikimedia.org/r/176385 (owner: 10Ori.livneh) [23:11:22] RECOVERY - puppet last run on ms-be1009 is OK: OK: Puppet is currently enabled, last run 23 seconds ago with 0 failures [23:11:57] RECOVERY - puppet last run on mw1097 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [23:12:06] RECOVERY - puppet last run on labsdb1004 is OK: OK: Puppet is currently enabled, last run 57 seconds ago with 0 failures [23:12:26] RECOVERY - puppet last run on mw1198 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [23:12:56] RECOVERY - puppet last run on titanium is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [23:13:08] RECOVERY - puppet last run on search1015 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [23:14:05] RECOVERY - puppet last run on cp1038 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [23:19:25] PROBLEM - HTTP error ratio anomaly detection on tungsten is CRITICAL: CRITICAL: Anomaly detected: 11 data above and 0 below the confidence bounds [23:19:55] PROBLEM - puppet last run on mw1229 is CRITICAL: CRITICAL: Puppet has 1 failures [23:20:29] ooh 503s [23:20:37] PROBLEM - Apache HTTP on mw1252 is CRITICAL: HTTP CRITICAL: HTTP/1.1 503 Service Unavailable - 50350 bytes in 0.018 second response time [23:21:28] PROBLEM - HTTP 5xx req/min on tungsten is CRITICAL: CRITICAL: 53.33% of data above the critical threshold [500.0] [23:22:37] PROBLEM - Apache HTTP on mw1189 is CRITICAL: HTTP CRITICAL: HTTP/1.1 503 Service Unavailable - 50350 bytes in 0.095 second response time [23:23:19] (03PS1) 10Ori.livneh: hhvm: Don't load tidy.so [puppet] - 10https://gerrit.wikimedia.org/r/176388 [23:23:26] RECOVERY - Apache HTTP on mw1252 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 440 bytes in 0.048 second response time [23:23:38] (03CR) 10Ori.livneh: [C: 032 V: 032] hhvm: Don't load tidy.so [puppet] - 10https://gerrit.wikimedia.org/r/176388 (owner: 10Ori.livneh) [23:23:41] seriously? [23:23:53] on a friday night, on what is a wmf holiday? [23:24:05] PROBLEM - puppet last run on mw1255 is CRITICAL: CRITICAL: Puppet has 1 failures [23:24:05] come on [23:24:09] logstash appears empty [23:24:31] it's just ori's change [23:24:35] and he's reverting it apparently [23:24:45] yeah, it caused segfaults [23:24:58] but seriously, fix it and stop deploying changes all by yourself on a holiday [23:25:12] it's fixed [23:25:13] it'd really suck to start paging people at this day/hour [23:25:25] RECOVERY - Apache HTTP on mw1189 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 440 bytes in 0.054 second response time [23:26:38] are you also going to send a postmortem then? [23:26:52] PROBLEM - puppet last run on mw1232 is CRITICAL: CRITICAL: Puppet has 1 failures [23:27:00] it was quick, but we've sent postmortems for less than that in the past [23:27:33] it wasn't an outage [23:29:05] ori: it made me check why its down [23:29:20] PROBLEM - puppet last run on mw1254 is CRITICAL: CRITICAL: Puppet has 1 failures [23:29:38] PROBLEM - HHVM queue size on mw1222 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [80.0] [23:33:49] RECOVERY - puppet last run on mw1229 is OK: OK: Puppet is currently enabled, last run 44 seconds ago with 0 failures [23:35:14] RECOVERY - HHVM queue size on mw1222 is OK: OK: Less than 1.00% above the threshold [10.0] [23:35:42] PROBLEM - HHVM rendering on mw1189 is CRITICAL: HTTP CRITICAL: HTTP/1.1 503 Service Unavailable - 50350 bytes in 0.196 second response time [23:37:43] RECOVERY - puppet last run on mw1255 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [23:39:28] it seems logstash stopped ingesting data after 1515UTC today [23:40:46] RECOVERY - puppet last run on mw1232 is OK: OK: Puppet is currently enabled, last run 48 seconds ago with 0 failures [23:44:06] RECOVERY - HHVM rendering on mw1189 is OK: HTTP OK: HTTP/1.1 200 OK - 73874 bytes in 0.517 second response time [23:45:54] RECOVERY - puppet last run on mw1254 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [23:57:48] RECOVERY - HTTP 5xx req/min on tungsten is OK: OK: Less than 1.00% above the threshold [250.0]