[00:15:01] RECOVERY - Kafka Broker Under Replicated Partitions on analytics1012 is OK: kafka.server.ReplicaManager.UnderReplicatedPartitions.Value OKAY: 0.0 [00:35:15] (03PS1) 10BryanDavis: Rewrite robots.php [mediawiki-config] - 10https://gerrit.wikimedia.org/r/177948 [00:35:47] (03PS2) 10BryanDavis: Rewrite robots.php [mediawiki-config] - 10https://gerrit.wikimedia.org/r/177948 [00:47:27] (03PS1) 10Ori.livneh: Add preview log group [mediawiki-config] - 10https://gerrit.wikimedia.org/r/177950 [00:48:28] (03CR) 10Ori.livneh: [C: 032] Add preview log group [mediawiki-config] - 10https://gerrit.wikimedia.org/r/177950 (owner: 10Ori.livneh) [00:48:43] (03Merged) 10jenkins-bot: Add preview log group [mediawiki-config] - 10https://gerrit.wikimedia.org/r/177950 (owner: 10Ori.livneh) [00:49:13] !log ori Synchronized wmf-config/InitialiseSettings.php: Add preview log group (duration: 00m 06s) [00:49:17] Logged the message, Master [00:52:02] RECOVERY - Unmerged changes on repository mediawiki_config on tin is OK: No changes to merge. [00:58:50] (03CR) 10Chad: Rewrite robots.php (032 comments) [mediawiki-config] - 10https://gerrit.wikimedia.org/r/177948 (owner: 10BryanDavis) [00:59:30] (03CR) 10Chad: [C: 032] robots.txt: Use max lastmod time [mediawiki-config] - 10https://gerrit.wikimedia.org/r/177940 (owner: 10BryanDavis) [00:59:40] (03Merged) 10jenkins-bot: robots.txt: Use max lastmod time [mediawiki-config] - 10https://gerrit.wikimedia.org/r/177940 (owner: 10BryanDavis) [01:00:50] !log demon Synchronized w/robots.php: better mtime (duration: 00m 06s) [01:00:52] <^d> bd808: ^ [01:00:54] Logged the message, Master [01:01:47] * bd808 crosses fingers [01:18:41] (03PS3) 10BryanDavis: Rewrite robots.php [mediawiki-config] - 10https://gerrit.wikimedia.org/r/177948 [01:18:50] (03CR) 10BryanDavis: Rewrite robots.php (032 comments) [mediawiki-config] - 10https://gerrit.wikimedia.org/r/177948 (owner: 10BryanDavis) [01:21:52] (03PS4) 10BryanDavis: Rewrite robots.php [mediawiki-config] - 10https://gerrit.wikimedia.org/r/177948 [01:23:03] PROBLEM - puppet last run on mw1078 is CRITICAL: CRITICAL: puppet fail [01:24:57] (03CR) 10Chad: Rewrite robots.php (031 comment) [mediawiki-config] - 10https://gerrit.wikimedia.org/r/177948 (owner: 10BryanDavis) [01:25:58] (03PS5) 10BryanDavis: Rewrite robots.php [mediawiki-config] - 10https://gerrit.wikimedia.org/r/177948 [01:28:45] PROBLEM - Unmerged changes on repository mediawiki_config on tin is CRITICAL: There are 2 unmerged changes in mediawiki_config (dir /srv/mediawiki-staging/). [01:38:12] RECOVERY - puppet last run on mw1078 is OK: OK: Puppet is currently enabled, last run 12 seconds ago with 0 failures [01:49:04] PROBLEM - puppet last run on mw1221 is CRITICAL: CRITICAL: Puppet has 1 failures [02:01:28] RECOVERY - puppet last run on mw1221 is OK: OK: Puppet is currently enabled, last run 50 seconds ago with 0 failures [02:13:08] !log l10nupdate Synchronized php-1.25wmf10/cache/l10n: (no message) (duration: 00m 01s) [02:13:13] !log LocalisationUpdate completed (1.25wmf10) at 2014-12-06 02:13:12+00:00 [02:13:15] Logged the message, Master [02:13:21] Logged the message, Master [02:18:48] !log l10nupdate Synchronized php-1.25wmf11/cache/l10n: (no message) (duration: 00m 01s) [02:18:52] !log LocalisationUpdate completed (1.25wmf11) at 2014-12-06 02:18:52+00:00 [02:18:52] Logged the message, Master [02:18:56] Logged the message, Master [03:15:31] (03PS1) 10Ori.livneh: Don't set $wgTidyInternal on Labs [mediawiki-config] - 10https://gerrit.wikimedia.org/r/177958 [03:16:21] (03CR) 10Ori.livneh: [C: 032] Don't set $wgTidyInternal on Labs [mediawiki-config] - 10https://gerrit.wikimedia.org/r/177958 (owner: 10Ori.livneh) [03:16:29] (03Merged) 10jenkins-bot: Don't set $wgTidyInternal on Labs [mediawiki-config] - 10https://gerrit.wikimedia.org/r/177958 (owner: 10Ori.livneh) [03:18:25] RECOVERY - Unmerged changes on repository mediawiki_config on tin is OK: No changes to merge. [03:42:37] !log LocalisationUpdate ResourceLoader cache refresh completed at Sat Dec 6 03:42:37 UTC 2014 (duration 42m 36s) [03:42:41] Logged the message, Master [04:06:21] RECOVERY - Kafka Broker Under Replicated Partitions on analytics1018 is OK: kafka.server.ReplicaManager.UnderReplicatedPartitions.Value OKAY: 0.0 [06:05:22] (03CR) 10PleaseStand: Rewrite robots.php (035 comments) [mediawiki-config] - 10https://gerrit.wikimedia.org/r/177948 (owner: 10BryanDavis) [06:30:02] PROBLEM - HTTP 5xx req/min on graphite1001 is CRITICAL: CRITICAL: 6.67% of data above the critical threshold [500.0] [06:30:12] PROBLEM - HTTP 5xx req/min on tungsten is CRITICAL: CRITICAL: 7.14% of data above the critical threshold [500.0] [06:34:17] PROBLEM - puppet last run on db2036 is CRITICAL: CRITICAL: Puppet has 1 failures [06:34:17] PROBLEM - puppet last run on db1002 is CRITICAL: CRITICAL: Puppet has 2 failures [06:34:29] PROBLEM - puppet last run on cp3016 is CRITICAL: CRITICAL: Puppet has 3 failures [06:35:19] PROBLEM - puppet last run on mw1025 is CRITICAL: CRITICAL: Puppet has 1 failures [06:36:15] PROBLEM - puppet last run on elastic1022 is CRITICAL: CRITICAL: Puppet has 1 failures [06:36:55] RECOVERY - HTTP 5xx req/min on graphite1001 is OK: OK: Less than 1.00% above the threshold [250.0] [06:37:08] RECOVERY - HTTP 5xx req/min on tungsten is OK: OK: Less than 1.00% above the threshold [250.0] [06:37:19] PROBLEM - puppet last run on mw1144 is CRITICAL: CRITICAL: Puppet has 1 failures [06:46:59] RECOVERY - puppet last run on elastic1022 is OK: OK: Puppet is currently enabled, last run 54 seconds ago with 0 failures [06:47:40] RECOVERY - puppet last run on mw1144 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [06:48:09] RECOVERY - puppet last run on db2036 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [06:48:10] RECOVERY - puppet last run on db1002 is OK: OK: Puppet is currently enabled, last run 3 minutes ago with 0 failures [06:48:31] RECOVERY - puppet last run on cp3016 is OK: OK: Puppet is currently enabled, last run 3 minutes ago with 0 failures [06:49:12] RECOVERY - puppet last run on mw1025 is OK: OK: Puppet is currently enabled, last run 3 minutes ago with 0 failures [06:56:54] PROBLEM - puppet last run on lvs4003 is CRITICAL: CRITICAL: puppet fail [07:09:11] RECOVERY - puppet last run on lvs4003 is OK: OK: Puppet is currently enabled, last run 28 seconds ago with 0 failures [09:30:16] PROBLEM - check_mysql on db1008 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 694 [09:35:26] RECOVERY - check_mysql on db1008 is OK: Uptime: 4478906 Threads: 61 Questions: 96024911 Slow queries: 29071 Opens: 85582 Flush tables: 2 Open tables: 64 Queries per second avg: 21.439 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 0 [09:53:47] (03PS1) 10Yuvipanda: Revert "nagios_common: Add compat checkcommand definition temporarily" [puppet] - 10https://gerrit.wikimedia.org/r/177963 [09:54:00] (03PS2) 10Yuvipanda: Revert "nagios_common: Add compat checkcommand definition temporarily" [puppet] - 10https://gerrit.wikimedia.org/r/177963 [09:55:14] (03CR) 10Yuvipanda: [C: 032] Revert "nagios_common: Add compat checkcommand definition temporarily" [puppet] - 10https://gerrit.wikimedia.org/r/177963 (owner: 10Yuvipanda) [10:10:21] PROBLEM - check_puppetrun on db1008 is CRITICAL: CRITICAL: Puppet has 2 failures [10:15:27] PROBLEM - check_puppetrun on db1008 is CRITICAL: CRITICAL: Puppet has 2 failures [10:20:22] PROBLEM - check_puppetrun on db1008 is CRITICAL: CRITICAL: Puppet has 2 failures [10:25:20] PROBLEM - check_puppetrun on db1008 is CRITICAL: CRITICAL: Puppet has 2 failures [10:30:22] RECOVERY - check_puppetrun on db1008 is OK: OK: Puppet is currently enabled, last run 122 seconds ago with 0 failures [11:21:23] (03CR) 10Hashar: [C: 031] Minor lint in base manifests [puppet] - 10https://gerrit.wikimedia.org/r/177864 (owner: 10Alexandros Kosiaris) [13:35:49] (03PS2) 10Alexandros Kosiaris: Fix Parsoid in beta [puppet] - 10https://gerrit.wikimedia.org/r/169622 (owner: 10Catrope) [13:36:29] (03CR) 10Alexandros Kosiaris: [C: 032] "This has been merged in Beta puppetmaster so I am assuming it is working fine. Merging" [puppet] - 10https://gerrit.wikimedia.org/r/169622 (owner: 10Catrope) [13:37:19] (03CR) 10Alexandros Kosiaris: [C: 032] Minor lint in base manifests [puppet] - 10https://gerrit.wikimedia.org/r/177864 (owner: 10Alexandros Kosiaris) [14:23:08] PROBLEM - puppet last run on mw1047 is CRITICAL: CRITICAL: Puppet has 1 failures [14:37:32] RECOVERY - puppet last run on mw1047 is OK: OK: Puppet is currently enabled, last run 28 seconds ago with 0 failures [16:23:45] PROBLEM - git.wikimedia.org on antimony is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:35:07] RECOVERY - git.wikimedia.org on antimony is OK: HTTP OK: HTTP/1.1 200 OK - 57850 bytes in 0.244 second response time [17:24:51] (03CR) 10MZMcBride: "Eh, if we're rewriting robots.php, I'd kind of prefer that we centralize the rules by eliminating the /srv/mediawiki/robots.txt file. A fe" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/177948 (owner: 10BryanDavis) [18:35:36] (03PS1) 10Krinkle: robots.php: Use max() time and clean up [mediawiki-config] - 10https://gerrit.wikimedia.org/r/177996 [18:36:16] (03CR) 10Krinkle: "Following the mailing list thread I also wrote a patch yesterday, see https://gerrit.wikimedia.org/r/#/c/177996/ . Feel free to integrate " [mediawiki-config] - 10https://gerrit.wikimedia.org/r/177948 (owner: 10BryanDavis) [18:36:42] (03CR) 10Krinkle: "Rebased since I495ad95f35c3621a41e already did the max() issue.." [mediawiki-config] - 10https://gerrit.wikimedia.org/r/177996 (owner: 10Krinkle) [19:39:45] !log Jenkins has been conquered by the French again [19:39:50] Logged the message, Master [19:48:20] !log Made trivial edit to Jenkins language config to purge the French invasion (default language: en-us -> en-US) [19:48:25] Logged the message, Master [22:29:22] PROBLEM - puppet last run on ms-fe1001 is CRITICAL: CRITICAL: puppet fail [22:29:34] PROBLEM - puppet last run on elastic1007 is CRITICAL: CRITICAL: puppet fail [22:29:42] PROBLEM - puppet last run on analytics1041 is CRITICAL: CRITICAL: puppet fail [22:29:48] PROBLEM - puppet last run on db2005 is CRITICAL: CRITICAL: puppet fail [22:30:02] PROBLEM - puppet last run on lead is CRITICAL: CRITICAL: puppet fail [22:30:02] PROBLEM - puppet last run on ms-be1006 is CRITICAL: CRITICAL: puppet fail [22:30:03] PROBLEM - puppet last run on mw1217 is CRITICAL: CRITICAL: puppet fail [22:30:03] PROBLEM - puppet last run on mc1006 is CRITICAL: CRITICAL: puppet fail [22:30:15] PROBLEM - puppet last run on db2039 is CRITICAL: CRITICAL: puppet fail [22:30:15] PROBLEM - puppet last run on ms-be2006 is CRITICAL: CRITICAL: puppet fail [22:30:15] PROBLEM - puppet last run on lvs3001 is CRITICAL: CRITICAL: puppet fail [22:30:15] PROBLEM - puppet last run on amssq35 is CRITICAL: CRITICAL: puppet fail [22:30:26] PROBLEM - puppet last run on mw1153 is CRITICAL: CRITICAL: puppet fail [22:30:26] PROBLEM - puppet last run on db1073 is CRITICAL: CRITICAL: puppet fail [22:30:26] PROBLEM - puppet last run on elastic1018 is CRITICAL: CRITICAL: puppet fail [22:30:26] PROBLEM - puppet last run on mw1226 is CRITICAL: CRITICAL: puppet fail [22:30:26] PROBLEM - puppet last run on mw1046 is CRITICAL: CRITICAL: puppet fail [22:30:32] PROBLEM - puppet last run on mw1242 is CRITICAL: CRITICAL: puppet fail [22:30:32] PROBLEM - puppet last run on analytics1035 is CRITICAL: CRITICAL: puppet fail [22:30:44] PROBLEM - puppet last run on db1066 is CRITICAL: CRITICAL: puppet fail [22:30:44] PROBLEM - puppet last run on mw1123 is CRITICAL: CRITICAL: puppet fail [22:30:44] PROBLEM - puppet last run on mw1189 is CRITICAL: CRITICAL: puppet fail [22:30:44] PROBLEM - puppet last run on cp1039 is CRITICAL: CRITICAL: puppet fail [22:30:44] PROBLEM - puppet last run on cp1061 is CRITICAL: CRITICAL: puppet fail [22:30:54] PROBLEM - puppet last run on db1015 is CRITICAL: CRITICAL: puppet fail [22:30:59] PROBLEM - puppet last run on ms-be2004 is CRITICAL: CRITICAL: puppet fail [22:31:00] PROBLEM - puppet last run on cp4008 is CRITICAL: CRITICAL: puppet fail [22:31:02] PROBLEM - puppet last run on cp4003 is CRITICAL: CRITICAL: puppet fail [22:31:03] PROBLEM - puppet last run on cp3016 is CRITICAL: CRITICAL: puppet fail [22:31:03] PROBLEM - puppet last run on elastic1008 is CRITICAL: CRITICAL: puppet fail [22:31:13] PROBLEM - puppet last run on db1059 is CRITICAL: CRITICAL: puppet fail [22:31:14] PROBLEM - puppet last run on mc1003 is CRITICAL: CRITICAL: puppet fail [22:31:14] PROBLEM - puppet last run on ms-be1003 is CRITICAL: CRITICAL: puppet fail [22:31:14] PROBLEM - puppet last run on mw1118 is CRITICAL: CRITICAL: puppet fail [22:31:15] PROBLEM - puppet last run on mw1092 is CRITICAL: CRITICAL: puppet fail [22:31:15] PROBLEM - puppet last run on mw1008 is CRITICAL: CRITICAL: puppet fail [22:31:22] PROBLEM - puppet last run on labsdb1003 is CRITICAL: CRITICAL: puppet fail [22:31:22] PROBLEM - puppet last run on lvs1005 is CRITICAL: CRITICAL: puppet fail [22:31:22] PROBLEM - puppet last run on mw1173 is CRITICAL: CRITICAL: puppet fail [22:31:22] PROBLEM - puppet last run on mw1099 is CRITICAL: CRITICAL: puppet fail [22:31:22] PROBLEM - puppet last run on mw1205 is CRITICAL: CRITICAL: puppet fail [22:31:23] PROBLEM - puppet last run on analytics1030 is CRITICAL: CRITICAL: puppet fail [22:31:32] PROBLEM - puppet last run on mw1150 is CRITICAL: CRITICAL: puppet fail [22:31:32] PROBLEM - puppet last run on mw1254 is CRITICAL: CRITICAL: puppet fail [22:31:32] PROBLEM - puppet last run on mw1117 is CRITICAL: CRITICAL: puppet fail [22:31:32] PROBLEM - puppet last run on mw1100 is CRITICAL: CRITICAL: puppet fail [22:31:32] PROBLEM - puppet last run on gold is CRITICAL: CRITICAL: puppet fail [22:31:36] PROBLEM - puppet last run on mw1065 is CRITICAL: CRITICAL: puppet fail [22:31:42] PROBLEM - puppet last run on mw1061 is CRITICAL: CRITICAL: puppet fail [22:31:55] PROBLEM - puppet last run on wtp1020 is CRITICAL: CRITICAL: puppet fail [22:31:56] PROBLEM - puppet last run on mw1144 is CRITICAL: CRITICAL: puppet fail [22:31:56] PROBLEM - puppet last run on db2002 is CRITICAL: CRITICAL: puppet fail [22:31:56] PROBLEM - puppet last run on ms-fe2001 is CRITICAL: CRITICAL: puppet fail [22:31:56] PROBLEM - puppet last run on ruthenium is CRITICAL: CRITICAL: puppet fail [22:32:05] PROBLEM - puppet last run on mw1222 is CRITICAL: CRITICAL: puppet fail [22:32:06] PROBLEM - puppet last run on mw1009 is CRITICAL: CRITICAL: puppet fail [22:32:07] :o [22:32:07] PROBLEM - puppet last run on gallium is CRITICAL: CRITICAL: puppet fail [22:32:08] PROBLEM - puppet last run on mw1164 is CRITICAL: CRITICAL: puppet fail [22:32:11] PROBLEM - puppet last run on mw1069 is CRITICAL: CRITICAL: puppet fail [22:32:12] PROBLEM - puppet last run on elastic1021 is CRITICAL: CRITICAL: puppet fail [22:32:13] PROBLEM - puppet last run on mw1052 is CRITICAL: CRITICAL: puppet fail [22:32:13] PROBLEM - puppet last run on mc1002 is CRITICAL: CRITICAL: puppet fail [22:32:13] PROBLEM - puppet last run on mw1175 is CRITICAL: CRITICAL: puppet fail [22:32:15] PROBLEM - puppet last run on wtp1016 is CRITICAL: CRITICAL: puppet fail [22:32:16] PROBLEM - puppet last run on db1002 is CRITICAL: CRITICAL: puppet fail [22:32:16] PROBLEM - puppet last run on mw1176 is CRITICAL: CRITICAL: puppet fail [22:32:16] PROBLEM - puppet last run on search1007 is CRITICAL: CRITICAL: puppet fail [22:32:16] PROBLEM - puppet last run on mw1003 is CRITICAL: CRITICAL: puppet fail [22:32:16] PROBLEM - puppet last run on ms-fe1004 is CRITICAL: CRITICAL: puppet fail [22:32:17] PROBLEM - puppet last run on db1034 is CRITICAL: CRITICAL: puppet fail [22:32:17] PROBLEM - puppet last run on mw1228 is CRITICAL: CRITICAL: puppet fail [22:32:18] PROBLEM - puppet last run on mw1060 is CRITICAL: CRITICAL: puppet fail [22:32:18] PROBLEM - puppet last run on mw1129 is CRITICAL: CRITICAL: puppet fail [22:32:19] PROBLEM - puppet last run on dbproxy1001 is CRITICAL: CRITICAL: puppet fail [22:32:19] PROBLEM - puppet last run on elastic1022 is CRITICAL: CRITICAL: puppet fail [22:32:20] PROBLEM - puppet last run on lvs2001 is CRITICAL: CRITICAL: puppet fail [22:32:20] PROBLEM - puppet last run on virt1006 is CRITICAL: CRITICAL: puppet fail [22:32:26] PROBLEM - puppet last run on mw1119 is CRITICAL: CRITICAL: puppet fail [22:32:29] PROBLEM - puppet last run on es2001 is CRITICAL: CRITICAL: puppet fail [22:32:29] PROBLEM - puppet last run on mw1088 is CRITICAL: CRITICAL: puppet fail [22:32:31] PROBLEM - puppet last run on mw1025 is CRITICAL: CRITICAL: puppet fail [22:32:31] PROBLEM - puppet last run on db2018 is CRITICAL: CRITICAL: puppet fail [22:32:31] PROBLEM - puppet last run on cp4004 is CRITICAL: CRITICAL: puppet fail [22:32:31] PROBLEM - puppet last run on cp4014 is CRITICAL: CRITICAL: puppet fail [22:32:31] PROBLEM - puppet last run on db1050 is CRITICAL: CRITICAL: puppet fail [22:32:32] PROBLEM - puppet last run on iron is CRITICAL: CRITICAL: puppet fail [22:32:32] PROBLEM - puppet last run on elastic1012 is CRITICAL: CRITICAL: puppet fail [22:32:33] PROBLEM - puppet last run on sca1002 is CRITICAL: CRITICAL: puppet fail [22:32:33] PROBLEM - puppet last run on search1001 is CRITICAL: CRITICAL: puppet fail [22:32:34] PROBLEM - puppet last run on elastic1027 is CRITICAL: CRITICAL: puppet fail [22:32:34] PROBLEM - puppet last run on cp3003 is CRITICAL: CRITICAL: puppet fail [22:32:35] PROBLEM - puppet last run on cp3014 is CRITICAL: CRITICAL: puppet fail [22:32:35] PROBLEM - puppet last run on db1042 is CRITICAL: CRITICAL: puppet fail [22:32:36] PROBLEM - puppet last run on mw1170 is CRITICAL: CRITICAL: puppet fail [22:32:48] PROBLEM - puppet last run on search1010 is CRITICAL: CRITICAL: puppet fail [22:32:50] PROBLEM - puppet last run on mw1126 is CRITICAL: CRITICAL: puppet fail [22:32:54] PROBLEM - puppet last run on mw1039 is CRITICAL: CRITICAL: puppet fail [22:32:57] PROBLEM - puppet last run on mw1120 is CRITICAL: CRITICAL: puppet fail [22:33:02] PROBLEM - puppet last run on labstore1003 is CRITICAL: CRITICAL: puppet fail [22:33:03] PROBLEM - puppet last run on mw1042 is CRITICAL: CRITICAL: puppet fail [22:33:03] PROBLEM - puppet last run on mw1068 is CRITICAL: CRITICAL: puppet fail [22:33:13] PROBLEM - puppet last run on db1067 is CRITICAL: CRITICAL: puppet fail [22:33:15] PROBLEM - puppet last run on cp1056 is CRITICAL: CRITICAL: puppet fail [22:33:16] PROBLEM - puppet last run on db1022 is CRITICAL: CRITICAL: puppet fail [22:33:18] PROBLEM - puppet last run on db1046 is CRITICAL: CRITICAL: puppet fail [22:33:18] PROBLEM - puppet last run on search1018 is CRITICAL: CRITICAL: puppet fail [22:33:19] PROBLEM - puppet last run on pc1002 is CRITICAL: CRITICAL: puppet fail [22:33:19] PROBLEM - puppet last run on ms1004 is CRITICAL: CRITICAL: puppet fail [22:33:19] PROBLEM - puppet last run on mw1177 is CRITICAL: CRITICAL: puppet fail [22:33:19] PROBLEM - puppet last run on dataset1001 is CRITICAL: CRITICAL: puppet fail [22:33:19] PROBLEM - puppet last run on ms-fe2003 is CRITICAL: CRITICAL: puppet fail [22:33:20] PROBLEM - puppet last run on lvs2004 is CRITICAL: CRITICAL: puppet fail [22:33:20] PROBLEM - puppet last run on db2036 is CRITICAL: CRITICAL: puppet fail [22:33:21] PROBLEM - puppet last run on db2038 is CRITICAL: CRITICAL: puppet fail [22:33:21] PROBLEM - puppet last run on ms-be2011 is CRITICAL: CRITICAL: puppet fail [22:33:22] PROBLEM - puppet last run on db2007 is CRITICAL: CRITICAL: puppet fail [22:33:26] PROBLEM - puppet last run on amssq60 is CRITICAL: CRITICAL: puppet fail [22:33:26] PROBLEM - puppet last run on amssq47 is CRITICAL: CRITICAL: puppet fail [22:33:26] PROBLEM - puppet last run on cp3010 is CRITICAL: CRITICAL: puppet fail [22:33:26] PROBLEM - puppet last run on amssq55 is CRITICAL: CRITICAL: puppet fail [22:33:35] PROBLEM - puppet last run on search1002 is CRITICAL: CRITICAL: puppet fail [22:33:35] PROBLEM - puppet last run on virt1004 is CRITICAL: CRITICAL: puppet fail [22:33:35] PROBLEM - puppet last run on mw1235 is CRITICAL: CRITICAL: puppet fail [22:33:35] PROBLEM - puppet last run on mw1227 is CRITICAL: CRITICAL: puppet fail [22:33:43] PROBLEM - puppet last run on lithium is CRITICAL: CRITICAL: puppet fail [22:33:43] PROBLEM - puppet last run on analytics1010 is CRITICAL: CRITICAL: puppet fail [22:33:44] PROBLEM - puppet last run on es1007 is CRITICAL: CRITICAL: puppet fail [22:33:44] PROBLEM - puppet last run on mw1251 is CRITICAL: CRITICAL: puppet fail [22:33:44] PROBLEM - puppet last run on logstash1002 is CRITICAL: CRITICAL: puppet fail [22:33:44] PROBLEM - puppet last run on mw1211 is CRITICAL: CRITICAL: puppet fail [22:33:44] PROBLEM - puppet last run on elastic1024 is CRITICAL: CRITICAL: puppet fail [22:33:45] PROBLEM - puppet last run on db1021 is CRITICAL: CRITICAL: puppet fail [22:33:45] PROBLEM - puppet last run on mw1166 is CRITICAL: CRITICAL: puppet fail [22:33:46] PROBLEM - puppet last run on mw1002 is CRITICAL: CRITICAL: puppet fail [22:33:46] PROBLEM - puppet last run on mw1151 is CRITICAL: CRITICAL: puppet fail [22:33:47] PROBLEM - puppet last run on db1040 is CRITICAL: CRITICAL: puppet fail [22:33:47] PROBLEM - puppet last run on cp4019 is CRITICAL: CRITICAL: puppet fail [22:33:48] PROBLEM - puppet last run on amssq46 is CRITICAL: CRITICAL: puppet fail [22:33:48] PROBLEM - puppet last run on amssq51 is CRITICAL: CRITICAL: puppet fail [22:33:49] PROBLEM - puppet last run on amssq34 is CRITICAL: CRITICAL: puppet fail [22:34:03] PROBLEM - puppet last run on db1028 is CRITICAL: CRITICAL: puppet fail [22:34:03] PROBLEM - puppet last run on mw1195 is CRITICAL: CRITICAL: puppet fail [22:34:03] PROBLEM - puppet last run on db1043 is CRITICAL: CRITICAL: puppet fail [22:34:03] PROBLEM - puppet last run on db1018 is CRITICAL: CRITICAL: puppet fail [22:34:03] PROBLEM - puppet last run on wtp1005 is CRITICAL: CRITICAL: puppet fail [22:34:04] PROBLEM - puppet last run on db1039 is CRITICAL: CRITICAL: puppet fail [22:34:04] PROBLEM - puppet last run on antimony is CRITICAL: CRITICAL: puppet fail [22:34:05] PROBLEM - puppet last run on mw1076 is CRITICAL: CRITICAL: puppet fail [22:34:05] PROBLEM - puppet last run on analytics1038 is CRITICAL: CRITICAL: puppet fail [22:34:06] PROBLEM - puppet last run on db1052 is CRITICAL: CRITICAL: puppet fail [22:34:06] PROBLEM - puppet last run on db1051 is CRITICAL: CRITICAL: puppet fail [22:34:13] PROBLEM - puppet last run on labnet1001 is CRITICAL: CRITICAL: puppet fail [22:34:16] PROBLEM - puppet last run on cp1050 is CRITICAL: CRITICAL: puppet fail [22:34:18] PROBLEM - puppet last run on lvs4003 is CRITICAL: CRITICAL: puppet fail [22:34:19] PROBLEM - puppet last run on mc1014 is CRITICAL: CRITICAL: puppet fail [22:34:19] PROBLEM - puppet last run on mw1054 is CRITICAL: CRITICAL: puppet fail [22:34:19] PROBLEM - puppet last run on bast4001 is CRITICAL: CRITICAL: puppet fail [22:34:19] PROBLEM - puppet last run on ms-be3002 is CRITICAL: CRITICAL: puppet fail [22:34:19] PROBLEM - puppet last run on amssq48 is CRITICAL: CRITICAL: puppet fail [22:34:20] PROBLEM - puppet last run on amslvs1 is CRITICAL: CRITICAL: puppet fail [22:34:24] PROBLEM - puppet last run on db1023 is CRITICAL: CRITICAL: puppet fail [22:34:26] PROBLEM - puppet last run on mw1213 is CRITICAL: CRITICAL: puppet fail [22:34:26] PROBLEM - puppet last run on db1036 is CRITICAL: CRITICAL: puppet fail [22:34:26] PROBLEM - puppet last run on db1016 is CRITICAL: CRITICAL: puppet fail [22:34:26] PROBLEM - puppet last run on mw1079 is CRITICAL: CRITICAL: puppet fail [22:34:26] PROBLEM - puppet last run on analytics1022 is CRITICAL: CRITICAL: puppet fail [22:34:27] PROBLEM - puppet last run on mw1172 is CRITICAL: CRITICAL: puppet fail [22:34:28] PROBLEM - puppet last run on snapshot1001 is CRITICAL: CRITICAL: puppet fail [22:34:28] PROBLEM - puppet last run on lvs3004 is CRITICAL: CRITICAL: puppet fail [22:34:28] PROBLEM - puppet last run on cp1046 is CRITICAL: CRITICAL: puppet fail [22:34:34] PROBLEM - puppet last run on cp4018 is CRITICAL: CRITICAL: puppet fail [22:34:35] PROBLEM - puppet last run on install2001 is CRITICAL: CRITICAL: puppet fail [22:34:35] PROBLEM - puppet last run on cp4001 is CRITICAL: CRITICAL: puppet fail [22:34:44] PROBLEM - puppet last run on mw1011 is CRITICAL: CRITICAL: puppet fail [22:34:46] PROBLEM - puppet last run on elastic1030 is CRITICAL: CRITICAL: puppet fail [22:34:46] PROBLEM - puppet last run on rhenium is CRITICAL: CRITICAL: puppet fail [22:34:46] PROBLEM - puppet last run on mw1114 is CRITICAL: CRITICAL: puppet fail [22:34:46] PROBLEM - puppet last run on mw1014 is CRITICAL: CRITICAL: puppet fail [22:34:46] PROBLEM - puppet last run on elastic1006 is CRITICAL: CRITICAL: puppet fail [22:34:47] PROBLEM - puppet last run on mc1012 is CRITICAL: CRITICAL: puppet fail [22:34:47] PROBLEM - puppet last run on labmon1001 is CRITICAL: CRITICAL: puppet fail [22:34:48] PROBLEM - puppet last run on cp1063 is CRITICAL: CRITICAL: puppet fail [22:34:55] PROBLEM - puppet last run on mw1206 is CRITICAL: CRITICAL: puppet fail [22:34:59] PROBLEM - puppet last run on db1048 is CRITICAL: CRITICAL: puppet fail [22:34:59] PROBLEM - puppet last run on labstore1001 is CRITICAL: CRITICAL: puppet fail [22:35:00] PROBLEM - puppet last run on search1017 is CRITICAL: CRITICAL: puppet fail [22:35:00] PROBLEM - puppet last run on db1003 is CRITICAL: CRITICAL: puppet fail [22:35:00] PROBLEM - puppet last run on mw1249 is CRITICAL: CRITICAL: puppet fail [22:35:00] PROBLEM - puppet last run on mw1049 is CRITICAL: CRITICAL: puppet fail [22:35:00] PROBLEM - puppet last run on labcontrol2001 is CRITICAL: CRITICAL: puppet fail [22:35:01] PROBLEM - puppet last run on cp4005 is CRITICAL: CRITICAL: puppet fail [22:35:01] PROBLEM - puppet last run on amssq36 is CRITICAL: CRITICAL: puppet fail [22:35:02] PROBLEM - puppet last run on amssq42 is CRITICAL: CRITICAL: puppet fail [22:35:03] PROBLEM - puppet last run on mw1125 is CRITICAL: CRITICAL: puppet fail [22:35:03] PROBLEM - puppet last run on polonium is CRITICAL: CRITICAL: puppet fail [22:35:13] PROBLEM - puppet last run on sodium is CRITICAL: CRITICAL: puppet fail [22:35:18] PROBLEM - puppet last run on mw1044 is CRITICAL: CRITICAL: puppet fail [22:35:20] PROBLEM - puppet last run on mw1162 is CRITICAL: CRITICAL: puppet fail [22:35:20] PROBLEM - puppet last run on osmium is CRITICAL: CRITICAL: puppet fail [22:35:24] PROBLEM - puppet last run on plutonium is CRITICAL: CRITICAL: puppet fail [22:35:24] PROBLEM - puppet last run on mw1180 is CRITICAL: CRITICAL: puppet fail [22:35:24] PROBLEM - puppet last run on mw1247 is CRITICAL: CRITICAL: puppet fail [22:35:24] PROBLEM - puppet last run on elastic1015 is CRITICAL: CRITICAL: puppet fail [22:35:24] PROBLEM - puppet last run on mw1149 is CRITICAL: CRITICAL: puppet fail [22:35:25] PROBLEM - puppet last run on db1026 is CRITICAL: CRITICAL: puppet fail [22:35:25] PROBLEM - puppet last run on snapshot1004 is CRITICAL: CRITICAL: puppet fail [22:35:26] PROBLEM - puppet last run on snapshot1002 is CRITICAL: CRITICAL: puppet fail [22:35:26] PROBLEM - puppet last run on tin is CRITICAL: CRITICAL: puppet fail [22:35:27] PROBLEM - puppet last run on stat1002 is CRITICAL: CRITICAL: puppet fail [22:35:27] PROBLEM - puppet last run on analytics1016 is CRITICAL: CRITICAL: puppet fail [22:35:28] PROBLEM - puppet last run on virt1003 is CRITICAL: CRITICAL: puppet fail [22:35:28] PROBLEM - puppet last run on mw1004 is CRITICAL: CRITICAL: puppet fail [22:35:29] PROBLEM - puppet last run on mw1051 is CRITICAL: CRITICAL: puppet fail [22:35:29] PROBLEM - puppet last run on db2029 is CRITICAL: CRITICAL: puppet fail [22:35:30] PROBLEM - puppet last run on mw1081 is CRITICAL: CRITICAL: puppet fail [22:35:35] PROBLEM - puppet last run on db1060 is CRITICAL: CRITICAL: puppet fail [22:35:35] PROBLEM - puppet last run on mw1055 is CRITICAL: CRITICAL: puppet fail [22:35:35] PROBLEM - puppet last run on db2023 is CRITICAL: CRITICAL: puppet fail [22:35:35] PROBLEM - puppet last run on ms-be2001 is CRITICAL: CRITICAL: puppet fail [22:35:35] PROBLEM - puppet last run on amssq56 is CRITICAL: CRITICAL: puppet fail [22:35:36] PROBLEM - puppet last run on wtp1012 is CRITICAL: CRITICAL: puppet fail [22:35:37] PROBLEM - puppet last run on virt1001 is CRITICAL: CRITICAL: puppet fail [22:35:37] PROBLEM - puppet last run on nescio is CRITICAL: CRITICAL: puppet fail [22:35:46] PROBLEM - puppet last run on cp1058 is CRITICAL: CRITICAL: puppet fail [22:35:48] PROBLEM - puppet last run on elastic1019 is CRITICAL: CRITICAL: puppet fail [22:35:48] PROBLEM - puppet last run on mw1111 is CRITICAL: CRITICAL: puppet fail [22:35:48] PROBLEM - puppet last run on amssq41 is CRITICAL: CRITICAL: puppet fail [22:35:49] PROBLEM - puppet last run on hooft is CRITICAL: CRITICAL: puppet fail [22:35:49] PROBLEM - puppet last run on oxygen is CRITICAL: CRITICAL: puppet fail [22:35:57] PROBLEM - puppet last run on mw1238 is CRITICAL: CRITICAL: puppet fail [22:35:58] PROBLEM - puppet last run on mw1034 is CRITICAL: CRITICAL: puppet fail [22:36:00] PROBLEM - puppet last run on elastic1014 is CRITICAL: CRITICAL: puppet fail [22:36:06] PROBLEM - puppet last run on analytics1023 is CRITICAL: CRITICAL: puppet fail [22:36:07] PROBLEM - puppet last run on mc1005 is CRITICAL: CRITICAL: puppet fail [22:36:09] PROBLEM - puppet last run on mw1030 is CRITICAL: CRITICAL: puppet fail [22:36:20] PROBLEM - puppet last run on rubidium is CRITICAL: CRITICAL: puppet fail [22:36:21] PROBLEM - puppet last run on mw1084 is CRITICAL: CRITICAL: puppet fail [22:36:21] PROBLEM - puppet last run on mw1202 is CRITICAL: CRITICAL: puppet fail [22:36:21] PROBLEM - puppet last run on mw1050 is CRITICAL: CRITICAL: puppet fail [22:36:21] PROBLEM - puppet last run on search1023 is CRITICAL: CRITICAL: puppet fail [22:36:21] PROBLEM - puppet last run on analytics1013 is CRITICAL: CRITICAL: puppet fail [22:36:21] PROBLEM - puppet last run on mw1098 is CRITICAL: CRITICAL: puppet fail [22:36:22] PROBLEM - puppet last run on db2016 is CRITICAL: CRITICAL: puppet fail [22:36:22] PROBLEM - puppet last run on tungsten is CRITICAL: CRITICAL: puppet fail [22:36:23] PROBLEM - puppet last run on wtp1022 is CRITICAL: CRITICAL: puppet fail [22:36:32] PROBLEM - puppet last run on cp1038 is CRITICAL: CRITICAL: puppet fail [22:36:34] PROBLEM - puppet last run on mw1208 is CRITICAL: CRITICAL: puppet fail [22:36:35] PROBLEM - puppet last run on mw1237 is CRITICAL: CRITICAL: puppet fail [22:36:35] PROBLEM - puppet last run on analytics1026 is CRITICAL: CRITICAL: puppet fail [22:36:36] PROBLEM - puppet last run on db2004 is CRITICAL: CRITICAL: puppet fail [22:36:36] PROBLEM - puppet last run on lvs2006 is CRITICAL: CRITICAL: puppet fail [22:36:36] PROBLEM - puppet last run on ms-be2012 is CRITICAL: CRITICAL: puppet fail [22:36:50] PROBLEM - puppet last run on ms-be3001 is CRITICAL: CRITICAL: puppet fail [22:36:51] PROBLEM - puppet last run on amssq40 is CRITICAL: CRITICAL: puppet fail [22:36:52] PROBLEM - puppet last run on ms-fe3002 is CRITICAL: CRITICAL: puppet fail [22:36:52] PROBLEM - puppet last run on mw1156 is CRITICAL: CRITICAL: puppet fail [22:36:52] PROBLEM - puppet last run on mw1190 is CRITICAL: CRITICAL: puppet fail [22:36:52] PROBLEM - puppet last run on mw1183 is CRITICAL: CRITICAL: puppet fail [22:36:52] PROBLEM - puppet last run on mw1097 is CRITICAL: CRITICAL: puppet fail [22:36:53] PROBLEM - puppet last run on mw1159 is CRITICAL: CRITICAL: puppet fail [22:36:53] PROBLEM - puppet last run on mw1165 is CRITICAL: CRITICAL: puppet fail [22:36:54] PROBLEM - puppet last run on mc1007 is CRITICAL: CRITICAL: puppet fail [22:37:00] PROBLEM - puppet last run on mw1029 is CRITICAL: CRITICAL: puppet fail [22:37:00] PROBLEM - puppet last run on mw1087 is CRITICAL: CRITICAL: puppet fail [22:37:00] PROBLEM - puppet last run on mw1210 is CRITICAL: CRITICAL: puppet fail [22:37:01] PROBLEM - puppet last run on search1005 is CRITICAL: CRITICAL: puppet fail [22:37:01] PROBLEM - puppet last run on labsdb1006 is CRITICAL: CRITICAL: puppet fail [22:37:01] PROBLEM - puppet last run on protactinium is CRITICAL: CRITICAL: puppet fail [22:37:01] PROBLEM - puppet last run on db1071 is CRITICAL: CRITICAL: puppet fail [22:37:02] PROBLEM - puppet last run on thallium is CRITICAL: CRITICAL: puppet fail [22:37:02] PROBLEM - puppet last run on mw1198 is CRITICAL: CRITICAL: puppet fail [22:37:03] PROBLEM - puppet last run on cp1062 is CRITICAL: CRITICAL: puppet fail [22:37:03] PROBLEM - puppet last run on cp1060 is CRITICAL: CRITICAL: puppet fail [22:37:10] PROBLEM - puppet last run on mw1133 is CRITICAL: CRITICAL: puppet fail [22:37:10] PROBLEM - puppet last run on mw1168 is CRITICAL: CRITICAL: puppet fail [22:37:10] PROBLEM - puppet last run on search1024 is CRITICAL: CRITICAL: puppet fail [22:37:10] PROBLEM - puppet last run on ms-be1007 is CRITICAL: CRITICAL: puppet fail [22:37:10] PROBLEM - puppet last run on cp1048 is CRITICAL: CRITICAL: puppet fail [22:37:11] PROBLEM - puppet last run on mc1001 is CRITICAL: CRITICAL: puppet fail [22:37:11] PROBLEM - puppet last run on mw1212 is CRITICAL: CRITICAL: puppet fail [22:37:23] PROBLEM - puppet last run on db2001 is CRITICAL: CRITICAL: puppet fail [22:37:29] PROBLEM - puppet last run on db1069 is CRITICAL: CRITICAL: puppet fail [22:37:29] PROBLEM - puppet last run on hafnium is CRITICAL: CRITICAL: puppet fail [22:37:29] PROBLEM - puppet last run on mw1146 is CRITICAL: CRITICAL: puppet fail [22:37:29] PROBLEM - puppet last run on analytics1011 is CRITICAL: CRITICAL: puppet fail [22:37:30] PROBLEM - puppet last run on rdb1001 is CRITICAL: CRITICAL: puppet fail [22:37:30] PROBLEM - puppet last run on gadolinium is CRITICAL: CRITICAL: puppet fail [22:37:30] PROBLEM - puppet last run on wtp1011 is CRITICAL: CRITICAL: puppet fail [22:37:31] PROBLEM - puppet last run on mw1258 is CRITICAL: CRITICAL: puppet fail [22:37:31] PROBLEM - puppet last run on analytics1037 is CRITICAL: CRITICAL: puppet fail [22:37:32] PROBLEM - puppet last run on mw1181 is CRITICAL: CRITICAL: puppet fail [22:37:32] PROBLEM - puppet last run on mw1053 is CRITICAL: CRITICAL: puppet fail [22:37:33] PROBLEM - puppet last run on rcs1001 is CRITICAL: CRITICAL: puppet fail [22:37:33] PROBLEM - puppet last run on lvs1001 is CRITICAL: CRITICAL: puppet fail [22:37:42] PROBLEM - puppet last run on db1001 is CRITICAL: CRITICAL: puppet fail [22:37:42] PROBLEM - puppet last run on wtp1018 is CRITICAL: CRITICAL: puppet fail [22:37:42] PROBLEM - puppet last run on db1020 is CRITICAL: CRITICAL: puppet fail [22:37:42] PROBLEM - puppet last run on db2037 is CRITICAL: CRITICAL: puppet fail [22:37:42] PROBLEM - puppet last run on ms-be2007 is CRITICAL: CRITICAL: puppet fail [22:37:43] PROBLEM - puppet last run on mw1188 is CRITICAL: CRITICAL: puppet fail [22:37:43] PROBLEM - puppet last run on analytics1032 is CRITICAL: CRITICAL: puppet fail [22:37:44] PROBLEM - puppet last run on db1004 is CRITICAL: CRITICAL: puppet fail [22:37:44] PROBLEM - puppet last run on argon is CRITICAL: CRITICAL: puppet fail [22:37:45] PROBLEM - puppet last run on db1062 is CRITICAL: CRITICAL: puppet fail [22:37:45] PROBLEM - puppet last run on titanium is CRITICAL: CRITICAL: puppet fail [22:37:46] PROBLEM - puppet last run on mw1057 is CRITICAL: CRITICAL: puppet fail [22:37:51] PROBLEM - puppet last run on analytics1014 is CRITICAL: CRITICAL: puppet fail [22:37:51] PROBLEM - puppet last run on wtp1002 is CRITICAL: CRITICAL: puppet fail [22:37:51] PROBLEM - puppet last run on labsdb1004 is CRITICAL: CRITICAL: puppet fail [22:37:51] PROBLEM - puppet last run on ms-be1008 is CRITICAL: CRITICAL: puppet fail [22:37:52] PROBLEM - puppet last run on mw1163 is CRITICAL: CRITICAL: puppet fail [22:37:52] PROBLEM - puppet last run on mw1032 is CRITICAL: CRITICAL: puppet fail [22:37:52] PROBLEM - puppet last run on elastic1011 is CRITICAL: CRITICAL: puppet fail [22:38:04] PROBLEM - puppet last run on mw1074 is CRITICAL: CRITICAL: puppet fail [22:38:06] PROBLEM - puppet last run on ms-be2005 is CRITICAL: CRITICAL: puppet fail [22:38:13]