[00:54:26] PROBLEM - puppet last run on ms-fe3001 is CRITICAL: CRITICAL: puppet fail [01:12:23] RECOVERY - puppet last run on ms-fe3001 is OK: OK: Puppet is currently enabled, last run 7 seconds ago with 0 failures [02:09:47] PROBLEM - puppet last run on cp3010 is CRITICAL: CRITICAL: puppet fail [02:10:16] PROBLEM - Disk space on virt0 is CRITICAL: DISK CRITICAL - free space: /a 3612 MB (3% inode=99%): [02:17:01] !log LocalisationUpdate completed (1.24wmf22) at 2014-09-28 02:17:01+00:00 [02:17:10] Logged the message, Master [02:28:00] RECOVERY - puppet last run on cp3010 is OK: OK: Puppet is currently enabled, last run 18 seconds ago with 0 failures [02:28:14] !log LocalisationUpdate completed (1.25wmf1) at 2014-09-28 02:28:14+00:00 [02:28:19] Logged the message, Master [03:00:57] RECOVERY - Disk space on virt0 is OK: DISK OK [03:24:20] !log LocalisationUpdate ResourceLoader cache refresh completed at Sun Sep 28 03:24:20 UTC 2014 (duration 24m 19s) [03:24:28] Logged the message, Master [03:38:48] PROBLEM - puppet last run on mw1158 is CRITICAL: CRITICAL: Puppet has 1 failures [03:56:26] RECOVERY - puppet last run on mw1158 is OK: OK: Puppet is currently enabled, last run 43 seconds ago with 0 failures [05:39:31] PROBLEM - Kafka Broker Messages In on analytics1021 is CRITICAL: kafka.server.BrokerTopicMetrics.AllTopicsMessagesInPerSec.FifteenMinuteRate CRITICAL: 909.902020968 [06:27:38] PROBLEM - Apache HTTP on mw1123 is CRITICAL: HTTP CRITICAL: HTTP/1.0 500 Internal Server Error - 50413 bytes in 0.039 second response time [06:28:19] PROBLEM - puppet last run on mw1213 is CRITICAL: CRITICAL: puppet fail [06:28:29] PROBLEM - Apache HTTP on mw1196 is CRITICAL: HTTP CRITICAL: HTTP/1.0 500 Internal Server Error - 50413 bytes in 0.033 second response time [06:28:30] PROBLEM - puppet last run on mw1114 is CRITICAL: CRITICAL: puppet fail [06:28:38] PROBLEM - puppet last run on labcontrol2001 is CRITICAL: CRITICAL: puppet fail [06:28:39] PROBLEM - puppet last run on mw1039 is CRITICAL: CRITICAL: puppet fail [06:28:49] PROBLEM - puppet last run on mw1054 is CRITICAL: CRITICAL: puppet fail [06:28:50] PROBLEM - puppet last run on analytics1010 is CRITICAL: CRITICAL: puppet fail [06:28:59] PROBLEM - puppet last run on cp3016 is CRITICAL: CRITICAL: Puppet has 1 failures [06:29:48] PROBLEM - puppet last run on cp1056 is CRITICAL: CRITICAL: Puppet has 2 failures [06:29:48] PROBLEM - puppet last run on iron is CRITICAL: CRITICAL: Puppet has 1 failures [06:30:18] PROBLEM - puppet last run on analytics1030 is CRITICAL: CRITICAL: Puppet has 2 failures [06:30:19] PROBLEM - puppet last run on cp1061 is CRITICAL: CRITICAL: Puppet has 1 failures [06:30:29] PROBLEM - puppet last run on db2018 is CRITICAL: CRITICAL: Puppet has 2 failures [06:30:38] PROBLEM - puppet last run on mw1061 is CRITICAL: CRITICAL: Puppet has 1 failures [06:30:48] PROBLEM - puppet last run on mw1123 is CRITICAL: CRITICAL: Puppet has 1 failures [06:30:49] PROBLEM - puppet last run on mw1025 is CRITICAL: CRITICAL: Puppet has 1 failures [06:30:49] PROBLEM - puppet last run on mw1118 is CRITICAL: CRITICAL: Puppet has 1 failures [06:30:58] PROBLEM - puppet last run on mw1092 is CRITICAL: CRITICAL: Puppet has 1 failures [06:30:59] PROBLEM - puppet last run on mw1166 is CRITICAL: CRITICAL: Puppet has 1 failures [06:31:11] PROBLEM - puppet last run on db1046 is CRITICAL: CRITICAL: Puppet has 2 failures [06:31:11] PROBLEM - puppet last run on mw1052 is CRITICAL: CRITICAL: Puppet has 2 failures [06:31:11] PROBLEM - puppet last run on db1018 is CRITICAL: CRITICAL: Puppet has 2 failures [06:31:19] PROBLEM - puppet last run on db1028 is CRITICAL: CRITICAL: Puppet has 2 failures [06:31:19] PROBLEM - puppet last run on cp4003 is CRITICAL: CRITICAL: Puppet has 3 failures [06:31:19] PROBLEM - puppet last run on cp3014 is CRITICAL: CRITICAL: Puppet has 2 failures [06:31:20] PROBLEM - puppet last run on mw1042 is CRITICAL: CRITICAL: Puppet has 2 failures [06:31:20] PROBLEM - puppet last run on mw1119 is CRITICAL: CRITICAL: Puppet has 1 failures [06:31:28] PROBLEM - puppet last run on mw1065 is CRITICAL: CRITICAL: Puppet has 2 failures [06:31:50] PROBLEM - puppet last run on cp4008 is CRITICAL: CRITICAL: Puppet has 3 failures [06:40:08] PROBLEM - ElasticSearch health check on elastic1015 is CRITICAL: CRITICAL - elasticsearch (production-search-eqiad) is running. status: red: timed_out: false: number_of_nodes: 18: number_of_data_nodes: 18: active_primary_shards: 2012: active_shards: 6031: relocating_shards: 2: initializing_shards: 1: unassigned_shards: 1 [06:40:09] PROBLEM - ElasticSearch health check on elastic1016 is CRITICAL: CRITICAL - elasticsearch (production-search-eqiad) is running. status: red: timed_out: false: number_of_nodes: 18: number_of_data_nodes: 18: active_primary_shards: 2012: active_shards: 6031: relocating_shards: 2: initializing_shards: 1: unassigned_shards: 1 [06:40:09] PROBLEM - ElasticSearch health check on elastic1005 is CRITICAL: CRITICAL - elasticsearch (production-search-eqiad) is running. status: red: timed_out: false: number_of_nodes: 18: number_of_data_nodes: 18: active_primary_shards: 2012: active_shards: 6031: relocating_shards: 2: initializing_shards: 1: unassigned_shards: 1 [06:41:11] RECOVERY - ElasticSearch health check on elastic1016 is OK: OK - elasticsearch (production-search-eqiad) is running. status: green: timed_out: false: number_of_nodes: 18: number_of_data_nodes: 18: active_primary_shards: 2013: active_shards: 6034: relocating_shards: 2: initializing_shards: 0: unassigned_shards: 0 [06:41:11] RECOVERY - ElasticSearch health check on elastic1015 is OK: OK - elasticsearch (production-search-eqiad) is running. status: green: timed_out: false: number_of_nodes: 18: number_of_data_nodes: 18: active_primary_shards: 2013: active_shards: 6034: relocating_shards: 2: initializing_shards: 0: unassigned_shards: 0 [06:41:11] RECOVERY - ElasticSearch health check on elastic1005 is OK: OK - elasticsearch (production-search-eqiad) is running. status: green: timed_out: false: number_of_nodes: 18: number_of_data_nodes: 18: active_primary_shards: 2013: active_shards: 6034: relocating_shards: 2: initializing_shards: 0: unassigned_shards: 0 [06:45:10] RECOVERY - puppet last run on cp3016 is OK: OK: Puppet is currently enabled, last run 1 seconds ago with 0 failures [06:45:20] RECOVERY - puppet last run on db1046 is OK: OK: Puppet is currently enabled, last run 20 seconds ago with 0 failures [06:45:20] RECOVERY - puppet last run on db1018 is OK: OK: Puppet is currently enabled, last run 19 seconds ago with 0 failures [06:45:20] RECOVERY - puppet last run on mw1052 is OK: OK: Puppet is currently enabled, last run 8 seconds ago with 0 failures [06:45:30] RECOVERY - puppet last run on analytics1030 is OK: OK: Puppet is currently enabled, last run 39 seconds ago with 0 failures [06:45:30] RECOVERY - puppet last run on mw1042 is OK: OK: Puppet is currently enabled, last run 13 seconds ago with 0 failures [06:45:30] RECOVERY - puppet last run on cp1061 is OK: OK: Puppet is currently enabled, last run 12 seconds ago with 0 failures [06:45:31] RECOVERY - puppet last run on iron is OK: OK: Puppet is currently enabled, last run 8 seconds ago with 0 failures [06:45:50] RECOVERY - puppet last run on mw1119 is OK: OK: Puppet is currently enabled, last run 2 seconds ago with 0 failures [06:45:51] RECOVERY - puppet last run on db2018 is OK: OK: Puppet is currently enabled, last run 44 seconds ago with 0 failures [06:46:00] RECOVERY - puppet last run on mw1123 is OK: OK: Puppet is currently enabled, last run 11 seconds ago with 0 failures [06:46:00] RECOVERY - puppet last run on cp1056 is OK: OK: Puppet is currently enabled, last run 40 seconds ago with 0 failures [06:46:00] RECOVERY - puppet last run on mw1061 is OK: OK: Puppet is currently enabled, last run 44 seconds ago with 0 failures [06:46:00] RECOVERY - puppet last run on mw1025 is OK: OK: Puppet is currently enabled, last run 36 seconds ago with 0 failures [06:46:10] RECOVERY - puppet last run on mw1092 is OK: OK: Puppet is currently enabled, last run 35 seconds ago with 0 failures [06:46:10] RECOVERY - puppet last run on mw1166 is OK: OK: Puppet is currently enabled, last run 25 seconds ago with 0 failures [06:46:30] RECOVERY - puppet last run on cp4003 is OK: OK: Puppet is currently enabled, last run 61 seconds ago with 0 failures [06:46:31] RECOVERY - puppet last run on cp3014 is OK: OK: Puppet is currently enabled, last run 52 seconds ago with 0 failures [06:46:40] RECOVERY - puppet last run on mw1213 is OK: OK: Puppet is currently enabled, last run 23 seconds ago with 0 failures [06:46:40] RECOVERY - puppet last run on cp4008 is OK: OK: Puppet is currently enabled, last run 20 seconds ago with 0 failures [06:46:40] RECOVERY - puppet last run on mw1065 is OK: OK: Puppet is currently enabled, last run 48 seconds ago with 0 failures [06:46:40] RECOVERY - puppet last run on mw1114 is OK: OK: Puppet is currently enabled, last run 26 seconds ago with 0 failures [06:46:41] RECOVERY - puppet last run on mw1118 is OK: OK: Puppet is currently enabled, last run 58 seconds ago with 0 failures [06:46:51] RECOVERY - puppet last run on labcontrol2001 is OK: OK: Puppet is currently enabled, last run 0 seconds ago with 0 failures [06:47:00] RECOVERY - puppet last run on mw1054 is OK: OK: Puppet is currently enabled, last run 16 seconds ago with 0 failures [06:47:20] RECOVERY - puppet last run on analytics1010 is OK: OK: Puppet is currently enabled, last run 51 seconds ago with 0 failures [06:47:20] RECOVERY - puppet last run on db1028 is OK: OK: Puppet is currently enabled, last run 57 seconds ago with 0 failures [06:48:01] RECOVERY - puppet last run on mw1039 is OK: OK: Puppet is currently enabled, last run 62 seconds ago with 0 failures [06:55:50] PROBLEM - RAID on searchidx1001 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [06:56:40] RECOVERY - RAID on searchidx1001 is OK: OK: optimal, 1 logical, 4 physical [10:05:26] PROBLEM - puppet last run on lvs4002 is CRITICAL: CRITICAL: Puppet has 1 failures [10:09:43] <_joe_> !log updated bash (again) across the whole cluster [10:09:48] Logged the message, Master [10:23:35] RECOVERY - puppet last run on lvs4002 is OK: OK: Puppet is currently enabled, last run 37 seconds ago with 0 failures [14:01:49] (03PS1) 10Andrew Bogott: For labs, give mathoid and empty lvs_service_ips list. [puppet] - 10https://gerrit.wikimedia.org/r/163464 [14:06:09] (03CR) 10Andrew Bogott: [C: 032] For labs, give mathoid and empty lvs_service_ips list. [puppet] - 10https://gerrit.wikimedia.org/r/163464 (owner: 10Andrew Bogott) [14:15:38] PROBLEM - Unmerged changes on repository puppet on strontium is CRITICAL: There is one unmerged change in puppet (dir /var/lib/git/operations/puppet). [14:15:58] PROBLEM - Unmerged changes on repository puppet on palladium is CRITICAL: There is one unmerged change in puppet (dir /var/lib/git/operations/puppet). [14:16:23] (03PS12) 10Andrew Bogott: WIP Citoid puppetization [puppet] - 10https://gerrit.wikimedia.org/r/163068 (owner: 10Catrope) [14:17:08] (03CR) 10Andrew Bogott: "In order to fix a puppet compile issue on beta I added a line to this patch defining an empty service_ip list for citoid on labs." [puppet] - 10https://gerrit.wikimedia.org/r/163068 (owner: 10Catrope) [14:26:07] ori: for labs instances that include the old puppet class webserver::apache2, what should they include now instead? [14:29:39] hm… perhaps the answer is 'webserver::apache' ? [15:56:13] PROBLEM - puppet last run on analytics1021 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [17:04:06] (03PS3) 10Glaisher: Initial setup for Persian Wikivoyage [mediawiki-config] - 10https://gerrit.wikimedia.org/r/163356 (https://bugzilla.wikimedia.org/71382) [17:06:15] (03CR) 10Glaisher: "Remove commonsuploads.dblist as it is for soft-upload." [mediawiki-config] - 10https://gerrit.wikimedia.org/r/163356 (https://bugzilla.wikimedia.org/71382) (owner: 10Glaisher) [17:10:38] (03CR) 10Calak: [C: 031] Initial setup for Persian Wikivoyage [mediawiki-config] - 10https://gerrit.wikimedia.org/r/163356 (https://bugzilla.wikimedia.org/71382) (owner: 10Glaisher) [18:56:43] PROBLEM - Varnishkafka Delivery Errors on cp3019 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 798.0 [19:07:36] PROBLEM - Varnishkafka Delivery Errors on cp3020 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 336.0 [19:08:46] PROBLEM - Varnishkafka Delivery Errors on cp3019 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 587.366638 [19:16:46] RECOVERY - Varnishkafka Delivery Errors on cp3020 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0 [19:25:45] PROBLEM - Varnishkafka Delivery Errors on cp3020 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 1770.333374 [19:37:45] RECOVERY - Varnishkafka Delivery Errors on cp3020 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0 [19:51:31] RECOVERY - Varnishkafka Delivery Errors on cp3019 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0 [20:14:01] PROBLEM - puppet last run on db2003 is CRITICAL: CRITICAL: Puppet has 1 failures [20:17:12] PROBLEM - puppet last run on db1057 is CRITICAL: CRITICAL: Puppet has 1 failures [20:17:21] PROBLEM - puppet last run on titanium is CRITICAL: CRITICAL: Puppet has 1 failures [20:17:27] (03CR) 10Ebrahim: [C: 031] Initial setup for Persian Wikivoyage [mediawiki-config] - 10https://gerrit.wikimedia.org/r/163356 (https://bugzilla.wikimedia.org/71382) (owner: 10Glaisher) [20:17:32] PROBLEM - puppet last run on db1001 is CRITICAL: CRITICAL: Puppet has 1 failures [20:17:32] PROBLEM - puppet last run on mc1007 is CRITICAL: CRITICAL: Puppet has 1 failures [20:17:32] PROBLEM - puppet last run on cp1060 is CRITICAL: CRITICAL: Puppet has 1 failures [20:17:41] PROBLEM - puppet last run on db1044 is CRITICAL: CRITICAL: Puppet has 1 failures [20:17:41] PROBLEM - puppet last run on lvs2003 is CRITICAL: CRITICAL: Puppet has 1 failures [20:17:41] PROBLEM - puppet last run on ssl1008 is CRITICAL: CRITICAL: Puppet has 1 failures [20:17:42] PROBLEM - puppet last run on ocg1003 is CRITICAL: CRITICAL: Puppet has 1 failures [20:17:51] PROBLEM - puppet last run on mw1032 is CRITICAL: CRITICAL: Puppet has 1 failures [20:17:54] PROBLEM - puppet last run on db1070 is CRITICAL: CRITICAL: Puppet has 1 failures [20:17:55] PROBLEM - puppet last run on elastic1017 is CRITICAL: CRITICAL: Puppet has 1 failures [20:17:55] PROBLEM - puppet last run on mw1108 is CRITICAL: CRITICAL: Puppet has 1 failures [20:18:02] that looks bad. [20:18:03] PROBLEM - puppet last run on mw1022 is CRITICAL: CRITICAL: Puppet has 1 failures [20:18:03] PROBLEM - puppet last run on search1013 is CRITICAL: CRITICAL: Puppet has 1 failures [20:18:03] PROBLEM - puppet last run on mw1201 is CRITICAL: CRITICAL: Puppet has 1 failures [20:18:03] PROBLEM - puppet last run on virt1000 is CRITICAL: CRITICAL: Puppet has 1 failures [20:18:03] PROBLEM - puppet last run on elastic1005 is CRITICAL: CRITICAL: Puppet has 1 failures [20:18:03] PROBLEM - puppet last run on wtp1013 is CRITICAL: CRITICAL: Puppet has 1 failures [20:18:11] PROBLEM - puppet last run on rdb1002 is CRITICAL: CRITICAL: Puppet has 1 failures [20:18:13] PROBLEM - puppet last run on amssq38 is CRITICAL: CRITICAL: Puppet has 1 failures [20:18:13] PROBLEM - puppet last run on cp3009 is CRITICAL: CRITICAL: Puppet has 1 failures [20:18:14] PROBLEM - puppet last run on wtp1003 is CRITICAL: CRITICAL: Puppet has 1 failures [20:18:14] PROBLEM - puppet last run on mw1185 is CRITICAL: CRITICAL: Puppet has 1 failures [20:18:14] PROBLEM - puppet last run on ssl1007 is CRITICAL: CRITICAL: Puppet has 1 failures [20:18:14] PROBLEM - puppet last run on pc1003 is CRITICAL: CRITICAL: Puppet has 1 failures [20:18:14] PROBLEM - puppet last run on mw1148 is CRITICAL: CRITICAL: Puppet has 1 failures [20:18:21] PROBLEM - puppet last run on db1064 is CRITICAL: CRITICAL: Puppet has 1 failures [20:18:23] PROBLEM - puppet last run on db2011 is CRITICAL: CRITICAL: Puppet has 1 failures [20:18:23] PROBLEM - puppet last run on mw1024 is CRITICAL: CRITICAL: Puppet has 1 failures [20:18:23] PROBLEM - puppet last run on amslvs3 is CRITICAL: CRITICAL: Puppet has 1 failures [20:18:23] PROBLEM - puppet last run on nitrogen is CRITICAL: CRITICAL: Puppet has 1 failures [20:18:23] PROBLEM - puppet last run on db1063 is CRITICAL: CRITICAL: Puppet has 1 failures [20:18:31] PROBLEM - puppet last run on wtp1007 is CRITICAL: CRITICAL: Puppet has 1 failures [20:18:31] PROBLEM - puppet last run on mw1001 is CRITICAL: CRITICAL: Puppet has 1 failures [20:18:32] PROBLEM - puppet last run on rdb1003 is CRITICAL: CRITICAL: Puppet has 1 failures [20:18:41] PROBLEM - puppet last run on sanger is CRITICAL: CRITICAL: Puppet has 1 failures [20:18:41] PROBLEM - puppet last run on mw1167 is CRITICAL: CRITICAL: Puppet has 1 failures [20:18:41] PROBLEM - puppet last run on mw1122 is CRITICAL: CRITICAL: Puppet has 1 failures [20:18:42] PROBLEM - puppet last run on cp3012 is CRITICAL: CRITICAL: Puppet has 1 failures [20:18:42] PROBLEM - puppet last run on mw1186 is CRITICAL: CRITICAL: Puppet has 1 failures [20:18:42] PROBLEM - puppet last run on mw1121 is CRITICAL: CRITICAL: Puppet has 1 failures [20:18:42] PROBLEM - puppet last run on mw1105 is CRITICAL: CRITICAL: Puppet has 1 failures [20:18:43] PROBLEM - puppet last run on ms-fe3001 is CRITICAL: CRITICAL: Puppet has 1 failures [20:18:54] PROBLEM - puppet last run on mercury is CRITICAL: CRITICAL: Puppet has 1 failures [20:18:57] PROBLEM - puppet last run on mw1033 is CRITICAL: CRITICAL: Puppet has 1 failures [20:18:57] PROBLEM - puppet last run on mw1077 is CRITICAL: CRITICAL: Puppet has 1 failures [20:18:58] PROBLEM - puppet last run on cp3004 is CRITICAL: CRITICAL: Puppet has 1 failures [20:18:58] PROBLEM - puppet last run on analytics1027 is CRITICAL: CRITICAL: Puppet has 1 failures [20:18:58] PROBLEM - puppet last run on mw1091 is CRITICAL: CRITICAL: Puppet has 1 failures [20:19:12] PROBLEM - puppet last run on mw1139 is CRITICAL: CRITICAL: Puppet has 1 failures [20:19:12] PROBLEM - puppet last run on cp1052 is CRITICAL: CRITICAL: Puppet has 1 failures [20:19:12] PROBLEM - puppet last run on iridium is CRITICAL: CRITICAL: Puppet has 1 failures [20:19:13] PROBLEM - puppet last run on db1030 is CRITICAL: CRITICAL: Puppet has 1 failures [20:19:13] PROBLEM - puppet last run on stat1003 is CRITICAL: CRITICAL: Puppet has 1 failures [20:19:13] PROBLEM - puppet last run on search1011 is CRITICAL: CRITICAL: Puppet has 1 failures [20:19:13] PROBLEM - puppet last run on search1022 is CRITICAL: CRITICAL: Puppet has 1 failures [20:19:31] PROBLEM - puppet last run on ssl1001 is CRITICAL: CRITICAL: Puppet has 1 failures [20:19:36] that's more than the usual mod_passenger silliness [20:19:41] PROBLEM - puppet last run on mw1043 is CRITICAL: CRITICAL: Puppet has 1 failures [20:19:42] PROBLEM - puppet last run on mw1209 is CRITICAL: CRITICAL: Puppet has 1 failures [20:19:42] PROBLEM - puppet last run on iodine is CRITICAL: CRITICAL: Puppet has 1 failures [20:19:42] PROBLEM - puppet last run on db1006 is CRITICAL: CRITICAL: Puppet has 1 failures [20:19:42] PROBLEM - puppet last run on db1047 is CRITICAL: CRITICAL: Puppet has 1 failures [20:19:43] PROBLEM - puppet last run on mw1010 is CRITICAL: CRITICAL: Puppet has 1 failures [20:19:43] PROBLEM - puppet last run on db2028 is CRITICAL: CRITICAL: Puppet has 1 failures [20:19:44] PROBLEM - puppet last run on mw1093 is CRITICAL: CRITICAL: Puppet has 1 failures [20:19:44] PROBLEM - puppet last run on search1006 is CRITICAL: CRITICAL: Puppet has 1 failures [20:19:44] PROBLEM - puppet last run on dbstore1002 is CRITICAL: CRITICAL: Puppet has 1 failures [20:19:45] PROBLEM - puppet last run on analytics1004 is CRITICAL: CRITICAL: Puppet has 1 failures [20:19:45] PROBLEM - puppet last run on db2030 is CRITICAL: CRITICAL: Puppet has 1 failures [20:19:46] PROBLEM - puppet last run on db2012 is CRITICAL: CRITICAL: Puppet has 1 failures [20:19:46] PROBLEM - puppet last run on mw1016 is CRITICAL: CRITICAL: Puppet has 1 failures [20:19:51] PROBLEM - puppet last run on analytics1028 is CRITICAL: CRITICAL: Puppet has 1 failures [20:19:51] PROBLEM - puppet last run on db1011 is CRITICAL: CRITICAL: Puppet has 1 failures [20:19:51] PROBLEM - puppet last run on mw1142 is CRITICAL: CRITICAL: Puppet has 1 failures [20:19:52] PROBLEM - puppet last run on cp1037 is CRITICAL: CRITICAL: Puppet has 1 failures [20:19:52] PROBLEM - puppet last run on calcium is CRITICAL: CRITICAL: Puppet has 1 failures [20:19:52] PROBLEM - puppet last run on db1033 is CRITICAL: CRITICAL: Puppet has 1 failures [20:19:52] PROBLEM - puppet last run on ms-be1011 is CRITICAL: CRITICAL: Puppet has 1 failures [20:19:53] PROBLEM - puppet last run on db1061 is CRITICAL: CRITICAL: Puppet has 1 failures [20:20:01] PROBLEM - puppet last run on mw1152 is CRITICAL: CRITICAL: Puppet has 1 failures [20:20:07] PROBLEM - puppet last run on mw1219 is CRITICAL: CRITICAL: Puppet has 1 failures [20:20:07] PROBLEM - puppet last run on achernar is CRITICAL: CRITICAL: Puppet has 1 failures [20:20:07] PROBLEM - puppet last run on cp4020 is CRITICAL: CRITICAL: Puppet has 1 failures [20:20:11] PROBLEM - puppet last run on es1010 is CRITICAL: CRITICAL: Puppet has 1 failures [20:20:11] PROBLEM - puppet last run on db1038 is CRITICAL: CRITICAL: Puppet has 1 failures [20:20:11] PROBLEM - puppet last run on ssl1006 is CRITICAL: CRITICAL: Puppet has 1 failures [20:20:11] PROBLEM - puppet last run on ms-be1002 is CRITICAL: CRITICAL: Puppet has 1 failures [20:20:11] PROBLEM - puppet last run on db1072 is CRITICAL: CRITICAL: Puppet has 1 failures [20:20:12] PROBLEM - puppet last run on fluorine is CRITICAL: CRITICAL: Puppet has 1 failures [20:20:12] PROBLEM - puppet last run on wtp1010 is CRITICAL: CRITICAL: Puppet has 1 failures [20:20:21] PROBLEM - puppet last run on copper is CRITICAL: CRITICAL: Puppet has 1 failures [20:20:22] PROBLEM - puppet last run on mw1066 is CRITICAL: CRITICAL: Puppet has 1 failures [20:20:22] PROBLEM - puppet last run on search1012 is CRITICAL: CRITICAL: Puppet has 1 failures [20:20:22] PROBLEM - puppet last run on search1004 is CRITICAL: CRITICAL: Puppet has 1 failures [20:20:22] PROBLEM - puppet last run on rbf1001 is CRITICAL: CRITICAL: Puppet has 1 failures [20:20:22] PROBLEM - puppet last run on mw1193 is CRITICAL: CRITICAL: Puppet has 1 failures [20:20:23] PROBLEM - puppet last run on amssq31 is CRITICAL: CRITICAL: Puppet has 1 failures [20:20:31] PROBLEM - puppet last run on ssl3001 is CRITICAL: CRITICAL: Puppet has 1 failures [20:20:31] PROBLEM - puppet last run on mw1090 is CRITICAL: CRITICAL: Puppet has 1 failures [20:20:31] PROBLEM - puppet last run on cp1067 is CRITICAL: CRITICAL: Puppet has 1 failures [20:20:31] PROBLEM - puppet last run on analytics1018 is CRITICAL: CRITICAL: Puppet has 1 failures [20:20:32] PROBLEM - puppet last run on cp1040 is CRITICAL: CRITICAL: Puppet has 1 failures [20:20:41] PROBLEM - puppet last run on mw1071 is CRITICAL: CRITICAL: Puppet has 1 failures [20:20:41] PROBLEM - puppet last run on db60 is CRITICAL: CRITICAL: Puppet has 1 failures [20:20:41] PROBLEM - puppet last run on cp1044 is CRITICAL: CRITICAL: Puppet has 1 failures [20:20:41] PROBLEM - puppet last run on dysprosium is CRITICAL: CRITICAL: Puppet has 1 failures [20:20:42] PROBLEM - puppet last run on mw1086 is CRITICAL: CRITICAL: Puppet has 1 failures [20:20:42] PROBLEM - puppet last run on cp3018 is CRITICAL: CRITICAL: Puppet has 1 failures [20:20:42] PROBLEM - puppet last run on mw1220 is CRITICAL: CRITICAL: Puppet has 1 failures [20:20:43] PROBLEM - puppet last run on osm-cp1001 is CRITICAL: CRITICAL: Puppet has 1 failures [20:20:43] PROBLEM - puppet last run on terbium is CRITICAL: CRITICAL: Puppet has 1 failures [20:20:44] PROBLEM - puppet last run on mw1207 is CRITICAL: CRITICAL: Puppet has 1 failures [20:20:51] PROBLEM - puppet last run on cp1045 is CRITICAL: CRITICAL: Puppet has 1 failures [20:20:51] PROBLEM - puppet last run on mw1064 is CRITICAL: CRITICAL: Puppet has 1 failures [20:20:51] PROBLEM - puppet last run on mw1104 is CRITICAL: CRITICAL: Puppet has 1 failures [20:20:51] PROBLEM - puppet last run on db1056 is CRITICAL: CRITICAL: Puppet has 1 failures [20:20:52] PROBLEM - puppet last run on mw1203 is CRITICAL: CRITICAL: Puppet has 1 failures [20:20:52] PROBLEM - puppet last run on elastic1003 is CRITICAL: CRITICAL: Puppet has 1 failures [20:20:52] PROBLEM - puppet last run on mw1027 is CRITICAL: CRITICAL: Puppet has 1 failures [20:20:53] PROBLEM - puppet last run on mw1215 is CRITICAL: CRITICAL: Puppet has 1 failures [20:20:53] PROBLEM - puppet last run on db71 is CRITICAL: CRITICAL: Puppet has 1 failures [20:20:54] PROBLEM - puppet last run on mw1143 is CRITICAL: CRITICAL: Puppet has 1 failures [20:20:54] PROBLEM - puppet last run on cp1053 is CRITICAL: CRITICAL: Puppet has 1 failures [20:21:01] PROBLEM - puppet last run on db1035 is CRITICAL: CRITICAL: Puppet has 1 failures [20:21:04] PROBLEM - puppet last run on strontium is CRITICAL: CRITICAL: Puppet has 1 failures [20:21:05] PROBLEM - puppet last run on mw1204 is CRITICAL: CRITICAL: Puppet has 1 failures [20:21:05] PROBLEM - puppet last run on virt1008 is CRITICAL: CRITICAL: Puppet has 1 failures [20:21:05] PROBLEM - puppet last run on cp4002 is CRITICAL: CRITICAL: Puppet has 1 failures [20:21:05] PROBLEM - puppet last run on cp4009 is CRITICAL: CRITICAL: Puppet has 1 failures [20:21:05] PROBLEM - puppet last run on chromium is CRITICAL: CRITICAL: Puppet has 1 failures [20:21:21] PROBLEM - puppet last run on mw1107 is CRITICAL: CRITICAL: Puppet has 1 failures [20:21:21] PROBLEM - puppet last run on mw1135 is CRITICAL: CRITICAL: Puppet has 1 failures [20:21:21] PROBLEM - puppet last run on amssq39 is CRITICAL: CRITICAL: Puppet has 1 failures [20:21:21] PROBLEM - puppet last run on db1045 is CRITICAL: CRITICAL: Puppet has 1 failures [20:21:21] PROBLEM - puppet last run on ms-be1015 is CRITICAL: CRITICAL: Puppet has 1 failures [20:21:22] PROBLEM - puppet last run on sca1001 is CRITICAL: CRITICAL: Puppet has 1 failures [20:21:22] PROBLEM - puppet last run on wtp1008 is CRITICAL: CRITICAL: Puppet has 1 failures [20:21:23] PROBLEM - puppet last run on mexia is CRITICAL: CRITICAL: Puppet has 1 failures [20:21:23] PROBLEM - puppet last run on lvs3003 is CRITICAL: CRITICAL: Puppet has 1 failures [20:21:24] PROBLEM - puppet last run on cp1054 is CRITICAL: CRITICAL: Puppet has 1 failures [20:21:24] PROBLEM - puppet last run on cp4012 is CRITICAL: CRITICAL: Puppet has 1 failures [20:21:25] PROBLEM - puppet last run on lvs4004 is CRITICAL: CRITICAL: Puppet has 1 failures [20:21:31] PROBLEM - puppet last run on cp3011 is CRITICAL: CRITICAL: Puppet has 1 failures [20:21:31] PROBLEM - puppet last run on linne is CRITICAL: CRITICAL: Puppet has 1 failures [20:21:31] PROBLEM - puppet last run on lvs1003 is CRITICAL: CRITICAL: Puppet has 1 failures [20:21:31] PROBLEM - puppet last run on bast1001 is CRITICAL: CRITICAL: Puppet has 1 failures [20:21:31] PROBLEM - puppet last run on mw1021 is CRITICAL: CRITICAL: Puppet has 1 failures [20:21:32] PROBLEM - puppet last run on wtp1001 is CRITICAL: CRITICAL: Puppet has 1 failures [20:21:41] PROBLEM - puppet last run on ms-fe1002 is CRITICAL: CRITICAL: Puppet has 1 failures [20:21:42] PROBLEM - puppet last run on mw1110 is CRITICAL: CRITICAL: Puppet has 1 failures [20:21:42] PROBLEM - puppet last run on db1037 is CRITICAL: CRITICAL: Puppet has 1 failures [20:21:42] PROBLEM - puppet last run on cp3005 is CRITICAL: CRITICAL: Puppet has 1 failures [20:21:42] PROBLEM - puppet last run on lvs1004 is CRITICAL: CRITICAL: Puppet has 1 failures [20:21:42] PROBLEM - puppet last run on mw1112 is CRITICAL: CRITICAL: Puppet has 1 failures [20:21:42] PROBLEM - puppet last run on fenari is CRITICAL: CRITICAL: Puppet has 1 failures [20:21:43] PROBLEM - puppet last run on mw1113 is CRITICAL: CRITICAL: Puppet has 1 failures [20:21:43] PROBLEM - puppet last run on cp1068 is CRITICAL: CRITICAL: Puppet has 1 failures [20:21:44] PROBLEM - puppet last run on mw1154 is CRITICAL: CRITICAL: Puppet has 1 failures [20:21:51] PROBLEM - puppet last run on tmh1001 is CRITICAL: CRITICAL: Puppet has 1 failures [20:21:51] PROBLEM - puppet last run on mw1131 is CRITICAL: CRITICAL: Puppet has 1 failures [20:21:52] PROBLEM - puppet last run on mw1155 is CRITICAL: CRITICAL: Puppet has 1 failures [20:21:52] PROBLEM - puppet last run on mw1037 is CRITICAL: CRITICAL: Puppet has 1 failures [20:21:52] PROBLEM - puppet last run on mw1158 is CRITICAL: CRITICAL: Puppet has 1 failures [20:21:52] PROBLEM - puppet last run on eeden is CRITICAL: CRITICAL: Puppet has 1 failures [20:22:01] PROBLEM - puppet last run on bast2001 is CRITICAL: CRITICAL: Puppet has 1 failures [20:22:01] PROBLEM - puppet last run on cp4016 is CRITICAL: CRITICAL: Puppet has 1 failures [20:22:02] PROBLEM - puppet last run on lvs2002 is CRITICAL: CRITICAL: Puppet has 1 failures [20:22:11] PROBLEM - puppet last run on analytics1012 is CRITICAL: CRITICAL: Puppet has 1 failures [20:22:11] PROBLEM - puppet last run on cp1066 is CRITICAL: CRITICAL: Puppet has 1 failures [20:22:11] PROBLEM - puppet last run on amssq37 is CRITICAL: CRITICAL: Puppet has 1 failures [20:22:11] PROBLEM - puppet last run on db1027 is CRITICAL: CRITICAL: Puppet has 1 failures [20:22:21] PROBLEM - puppet last run on mw1073 is CRITICAL: CRITICAL: Puppet has 1 failures [20:22:27] PROBLEM - puppet last run on mw1047 is CRITICAL: CRITICAL: Puppet has 1 failures [20:22:27] PROBLEM - puppet last run on mw1128 is CRITICAL: CRITICAL: Puppet has 1 failures [20:22:27] PROBLEM - puppet last run on cp3017 is CRITICAL: CRITICAL: Puppet has 1 failures [20:22:27] PROBLEM - puppet last run on analytics1031 is CRITICAL: CRITICAL: Puppet has 1 failures [20:22:27] PROBLEM - puppet last run on cp4011 is CRITICAL: CRITICAL: Puppet has 1 failures [20:22:28] PROBLEM - puppet last run on mc1015 is CRITICAL: CRITICAL: Puppet has 1 failures [20:22:28] PROBLEM - puppet last run on db2017 is CRITICAL: CRITICAL: Puppet has 1 failures [20:22:29] PROBLEM - puppet last run on virt1005 is CRITICAL: CRITICAL: Puppet has 1 failures [20:22:31] PROBLEM - puppet last run on cp3021 is CRITICAL: CRITICAL: Puppet has 1 failures [20:22:31] PROBLEM - puppet last run on cp3019 is CRITICAL: CRITICAL: Puppet has 1 failures [20:22:41] PROBLEM - puppet last run on labsdb1005 is CRITICAL: CRITICAL: Puppet has 1 failures [20:22:42] PROBLEM - puppet last run on amssq50 is CRITICAL: CRITICAL: Puppet has 1 failures [20:22:42] PROBLEM - puppet last run on search1003 is CRITICAL: CRITICAL: Puppet has 1 failures [20:22:51] PROBLEM - puppet last run on ms-be1014 is CRITICAL: CRITICAL: Puppet has 1 failures [20:22:51] PROBLEM - puppet last run on cp4010 is CRITICAL: CRITICAL: Puppet has 1 failures [20:22:52] PROBLEM - puppet last run on mw1199 is CRITICAL: CRITICAL: Puppet has 1 failures [20:22:52] PROBLEM - puppet last run on mw1103 is CRITICAL: CRITICAL: Puppet has 1 failures [20:22:52] PROBLEM - puppet last run on elastic1010 is CRITICAL: CRITICAL: Puppet has 1 failures [20:22:52] PROBLEM - puppet last run on db1005 is CRITICAL: CRITICAL: Puppet has 1 failures [20:22:52] PROBLEM - puppet last run on ms-be1010 is CRITICAL: CRITICAL: Puppet has 1 failures [20:22:53] PROBLEM - puppet last run on mw1018 is CRITICAL: CRITICAL: Puppet has 1 failures [20:23:11] PROBLEM - puppet last run on palladium is CRITICAL: CRITICAL: Puppet has 1 failures [20:23:12] PROBLEM - puppet last run on mw1058 is CRITICAL: CRITICAL: Puppet has 1 failures [20:23:12] PROBLEM - puppet last run on elastic1013 is CRITICAL: CRITICAL: Puppet has 1 failures [20:23:12] PROBLEM - puppet last run on mw1095 is CRITICAL: CRITICAL: Puppet has 1 failures [20:23:21] PROBLEM - puppet last run on ssl1003 is CRITICAL: CRITICAL: Puppet has 1 failures [20:23:22] PROBLEM - puppet last run on db1009 is CRITICAL: CRITICAL: Puppet has 1 failures [20:23:22] PROBLEM - puppet last run on mw1194 is CRITICAL: CRITICAL: Puppet has 1 failures [20:23:22] PROBLEM - puppet last run on es1004 is CRITICAL: CRITICAL: Puppet has 1 failures [20:23:22] PROBLEM - puppet last run on db1068 is CRITICAL: CRITICAL: Puppet has 1 failures [20:23:22] PROBLEM - puppet last run on elastic1009 is CRITICAL: CRITICAL: Puppet has 1 failures [20:23:22] PROBLEM - puppet last run on labsdb1002 is CRITICAL: CRITICAL: Puppet has 1 failures [20:23:31] PROBLEM - puppet last run on ms-be2009 is CRITICAL: CRITICAL: Puppet has 1 failures [20:23:31] PROBLEM - puppet last run on db1049 is CRITICAL: CRITICAL: Puppet has 1 failures [20:23:34] PROBLEM - puppet last run on cp4015 is CRITICAL: CRITICAL: Puppet has 1 failures [20:23:41] PROBLEM - puppet last run on amssq52 is CRITICAL: CRITICAL: Puppet has 1 failures [20:23:41] PROBLEM - puppet last run on mw1102 is CRITICAL: CRITICAL: Puppet has 1 failures [20:23:41] PROBLEM - puppet last run on search1019 is CRITICAL: CRITICAL: Puppet has 1 failures [20:23:42] PROBLEM - puppet last run on mw1101 is CRITICAL: CRITICAL: Puppet has 1 failures [20:23:42] PROBLEM - puppet last run on mw1070 is CRITICAL: CRITICAL: Puppet has 1 failures [20:23:42] PROBLEM - puppet last run on cp1064 is CRITICAL: CRITICAL: Puppet has 1 failures [20:23:42] PROBLEM - puppet last run on zinc is CRITICAL: CRITICAL: Puppet has 1 failures [20:23:43] PROBLEM - puppet last run on mw1075 is CRITICAL: CRITICAL: Puppet has 1 failures [20:23:47] PROBLEM - puppet last run on lvs1006 is CRITICAL: CRITICAL: Puppet has 1 failures [20:23:47] PROBLEM - puppet last run on lvs2005 is CRITICAL: CRITICAL: Puppet has 1 failures [20:23:47] PROBLEM - puppet last run on amssq45 is CRITICAL: CRITICAL: Puppet has 1 failures [20:23:47] PROBLEM - puppet last run on cp3007 is CRITICAL: CRITICAL: Puppet has 1 failures [20:23:51] PROBLEM - puppet last run on labsdb1007 is CRITICAL: CRITICAL: Puppet has 1 failures [20:23:51] PROBLEM - puppet last run on mw1085 is CRITICAL: CRITICAL: Puppet has 1 failures [20:23:51] PROBLEM - puppet last run on mw1157 is CRITICAL: CRITICAL: Puppet has 1 failures [20:23:52] PROBLEM - puppet last run on mw1019 is CRITICAL: CRITICAL: Puppet has 1 failures [20:23:52] PROBLEM - puppet last run on hydrogen is CRITICAL: CRITICAL: Puppet has 1 failures [20:24:01] PROBLEM - puppet last run on lvs3002 is CRITICAL: CRITICAL: Puppet has 1 failures [20:24:01] PROBLEM - puppet last run on ms-be3004 is CRITICAL: CRITICAL: Puppet has 1 failures [20:24:01] PROBLEM - puppet last run on mw1078 is CRITICAL: CRITICAL: Puppet has 1 failures [20:24:02] PROBLEM - puppet last run on mw1179 is CRITICAL: CRITICAL: Puppet has 1 failures [20:24:02] PROBLEM - puppet last run on search1008 is CRITICAL: CRITICAL: Puppet has 1 failures [20:24:02] PROBLEM - puppet last run on wtp1014 is CRITICAL: CRITICAL: Puppet has 1 failures [20:24:02] PROBLEM - puppet last run on virt1009 is CRITICAL: CRITICAL: Puppet has 1 failures [20:24:03] PROBLEM - puppet last run on db1019 is CRITICAL: CRITICAL: Puppet has 1 failures [20:24:03] PROBLEM - puppet last run on mw1182 is CRITICAL: CRITICAL: Puppet has 1 failures [20:24:04] PROBLEM - puppet last run on es1005 is CRITICAL: CRITICAL: Puppet has 1 failures [20:24:04] PROBLEM - puppet last run on analytics1029 is CRITICAL: CRITICAL: Puppet has 1 failures [20:24:05] PROBLEM - puppet last run on mw1137 is CRITICAL: CRITICAL: Puppet has 1 failures [20:24:05] PROBLEM - puppet last run on lanthanum is CRITICAL: CRITICAL: Puppet has 1 failures [20:24:21] PROBLEM - puppet last run on erbium is CRITICAL: CRITICAL: Puppet has 1 failures [20:24:21] PROBLEM - puppet last run on mw1020 is CRITICAL: CRITICAL: Puppet has 1 failures [20:24:21] PROBLEM - puppet last run on ms-be2010 is CRITICAL: CRITICAL: Puppet has 1 failures [20:24:21] PROBLEM - puppet last run on es1003 is CRITICAL: CRITICAL: Puppet has 1 failures [20:24:22] PROBLEM - puppet last run on ms-be1005 is CRITICAL: CRITICAL: Puppet has 1 failures [20:24:22] PROBLEM - puppet last run on cp1057 is CRITICAL: CRITICAL: Puppet has 1 failures [20:24:22] PROBLEM - puppet last run on mw1136 is CRITICAL: CRITICAL: Puppet has 1 failures [20:24:31] PROBLEM - puppet last run on nickel is CRITICAL: CRITICAL: Puppet has 1 failures [20:24:31] PROBLEM - puppet last run on mw1094 is CRITICAL: CRITICAL: Puppet has 1 failures [20:24:31] PROBLEM - puppet last run on ocg1002 is CRITICAL: CRITICAL: Puppet has 1 failures [20:24:31] PROBLEM - puppet last run on mw1015 is CRITICAL: CRITICAL: Puppet has 1 failures [20:24:31] PROBLEM - puppet last run on mw1017 is CRITICAL: CRITICAL: Puppet has 1 failures [20:24:42] PROBLEM - puppet last run on search1009 is CRITICAL: CRITICAL: Puppet has 1 failures [20:24:43] PROBLEM - puppet last run on zirconium is CRITICAL: CRITICAL: Puppet has 1 failures [20:24:51] PROBLEM - puppet last run on cp4017 is CRITICAL: CRITICAL: Puppet has 1 failures [20:24:51] PROBLEM - puppet last run on analytics1024 is CRITICAL: CRITICAL: Puppet has 1 failures [20:24:51] PROBLEM - puppet last run on ssl1004 is CRITICAL: CRITICAL: Puppet has 1 failures [20:24:51] PROBLEM - puppet last run on tmh1002 is CRITICAL: CRITICAL: Puppet has 1 failures [20:24:52] PROBLEM - puppet last run on praseodymium is CRITICAL: CRITICAL: Puppet has 1 failures [20:25:01] PROBLEM - puppet last run on mc1008 is CRITICAL: CRITICAL: Puppet has 1 failures [20:25:01] PROBLEM - puppet last run on mw1083 is CRITICAL: CRITICAL: Puppet has 1 failures [20:25:01] PROBLEM - puppet last run on search1021 is CRITICAL: CRITICAL: Puppet has 1 failures [20:25:02] PROBLEM - puppet last run on db1024 is CRITICAL: CRITICAL: Puppet has 1 failures [20:25:02] PROBLEM - puppet last run on elastic1016 is CRITICAL: CRITICAL: Puppet has 1 failures [20:25:02] PROBLEM - puppet last run on mw1127 is CRITICAL: CRITICAL: Puppet has 1 failures [20:25:02] PROBLEM - puppet last run on mw1169 is CRITICAL: CRITICAL: Puppet has 1 failures [20:25:03] PROBLEM - puppet last run on analytics1034 is CRITICAL: CRITICAL: Puppet has 1 failures [20:25:03] PROBLEM - puppet last run on analytics1019 is CRITICAL: CRITICAL: Puppet has 1 failures [20:25:04] PROBLEM - puppet last run on cp1051 is CRITICAL: CRITICAL: Puppet has 1 failures [20:25:04] PROBLEM - puppet last run on cp4013 is CRITICAL: CRITICAL: Puppet has 1 failures [20:25:05] PROBLEM - puppet last run on db1058 is CRITICAL: CRITICAL: Puppet has 1 failures [20:25:25] PROBLEM - puppet last run on mw1013 is CRITICAL: CRITICAL: Puppet has 1 failures [20:25:25] PROBLEM - puppet last run on db2033 is CRITICAL: CRITICAL: Puppet has 1 failures [20:25:26] PROBLEM - puppet last run on amssq57 is CRITICAL: CRITICAL: Puppet has 1 failures [20:25:26] PROBLEM - puppet last run on mw1214 is CRITICAL: CRITICAL: Puppet has 1 failures [20:25:26] PROBLEM - puppet last run on mw1138 is CRITICAL: CRITICAL: Puppet has 1 failures [20:25:28] PROBLEM - puppet last run on cp1069 is CRITICAL: CRITICAL: Puppet has 1 failures [20:25:31] PROBLEM - puppet last run on mw1184 is CRITICAL: CRITICAL: Puppet has 1 failures [20:25:31] PROBLEM - puppet last run on mw1035 is CRITICAL: CRITICAL: Puppet has 1 failures [20:25:32] PROBLEM - puppet last run on lvs4001 is CRITICAL: CRITICAL: Puppet has 1 failures [20:25:41] PROBLEM - puppet last run on cp1065 is CRITICAL: CRITICAL: Puppet has 1 failures [20:25:43] PROBLEM - puppet last run on labsdb1001 is CRITICAL: CRITICAL: Puppet has 1 failures [20:25:43] PROBLEM - puppet last run on amslvs4 is CRITICAL: CRITICAL: Puppet has 1 failures [20:25:51] PROBLEM - puppet last run on db1007 is CRITICAL: CRITICAL: Puppet has 1 failures [20:25:51] PROBLEM - puppet last run on es7 is CRITICAL: CRITICAL: Puppet has 1 failures [20:25:52] PROBLEM - puppet last run on es10 is CRITICAL: CRITICAL: Puppet has 1 failures [20:25:52] PROBLEM - puppet last run on mw1096 is CRITICAL: CRITICAL: Puppet has 1 failures [20:25:52] PROBLEM - puppet last run on neon is CRITICAL: CRITICAL: Puppet has 1 failures [20:26:04] PROBLEM - puppet last run on mw1036 is CRITICAL: CRITICAL: Puppet has 1 failures [20:26:04] PROBLEM - puppet last run on search1014 is CRITICAL: CRITICAL: Puppet has 1 failures [20:26:04] PROBLEM - puppet last run on wtp1021 is CRITICAL: CRITICAL: Puppet has 1 failures [20:26:07] PROBLEM - puppet last run on db1041 is CRITICAL: CRITICAL: Puppet has 1 failures [20:26:08] PROBLEM - puppet last run on es4 is CRITICAL: CRITICAL: Puppet has 1 failures [20:26:08] PROBLEM - puppet last run on db2035 is CRITICAL: CRITICAL: Puppet has 1 failures [20:26:08] PROBLEM - puppet last run on tarin is CRITICAL: CRITICAL: Puppet has 1 failures [20:26:09] PROBLEM - puppet last run on es1006 is CRITICAL: CRITICAL: Puppet has 1 failures [20:26:21] PROBLEM - puppet last run on wtp1024 is CRITICAL: CRITICAL: Puppet has 1 failures [20:26:21] PROBLEM - puppet last run on mw1062 is CRITICAL: CRITICAL: Puppet has 1 failures [20:26:22] PROBLEM - puppet last run on mw1196 is CRITICAL: CRITICAL: Puppet has 1 failures [20:26:22] PROBLEM - puppet last run on virt1002 is CRITICAL: CRITICAL: Puppet has 1 failures [20:26:22] PROBLEM - puppet last run on mw1192 is CRITICAL: CRITICAL: Puppet has 1 failures [20:26:22] PROBLEM - puppet last run on ytterbium is CRITICAL: CRITICAL: Puppet has 1 failures [20:26:22] PROBLEM - puppet last run on cp3013 is CRITICAL: CRITICAL: Puppet has 1 failures [20:26:23] PROBLEM - puppet last run on cp1043 is CRITICAL: CRITICAL: Puppet has 1 failures [20:26:23] PROBLEM - puppet last run on ocg1001 is CRITICAL: CRITICAL: Puppet has 1 failures [20:26:31] PROBLEM - puppet last run on mw1216 is CRITICAL: CRITICAL: Puppet has 1 failures [20:26:31] PROBLEM - puppet last run on es1009 is CRITICAL: CRITICAL: Puppet has 1 failures [20:26:31] PROBLEM - puppet last run on mc1010 is CRITICAL: CRITICAL: Puppet has 1 failures [20:26:31] PROBLEM - puppet last run on mw1109 is CRITICAL: CRITICAL: Puppet has 1 failures [20:26:35] That looks bad. [20:26:41] PROBLEM - puppet last run on amssq58 is CRITICAL: CRITICAL: Puppet has 1 failures [20:26:41] PROBLEM - puppet last run on analytics1039 is CRITICAL: CRITICAL: Puppet has 1 failures [20:26:41] PROBLEM - puppet last run on analytics1015 is CRITICAL: CRITICAL: Puppet has 1 failures [20:26:42] PROBLEM - puppet last run on cp3022 is CRITICAL: CRITICAL: Puppet has 1 failures [20:26:42] PROBLEM - puppet last run on ssl3003 is CRITICAL: CRITICAL: Puppet has 1 failures [20:26:42] PROBLEM - puppet last run on mc1009 is CRITICAL: CRITICAL: Puppet has 1 failures [20:26:42] PROBLEM - puppet last run on mw1161 is CRITICAL: CRITICAL: Puppet has 1 failures [20:26:43] PROBLEM - puppet last run on db1010 is CRITICAL: CRITICAL: Puppet has 1 failures [20:26:45] PROBLEM - puppet last run on vanadium is CRITICAL: CRITICAL: Puppet has 1 failures [20:26:51] PROBLEM - puppet last run on mw1124 is CRITICAL: CRITICAL: Puppet has 1 failures [20:26:58] PROBLEM - puppet last run on mw1191 is CRITICAL: CRITICAL: Puppet has 1 failures [20:26:58] PROBLEM - puppet last run on mw1147 is CRITICAL: CRITICAL: Puppet has 1 failures [20:26:58] PROBLEM - puppet last run on mw1218 is CRITICAL: CRITICAL: Puppet has 1 failures [20:27:02] PROBLEM - puppet last run on mw1130 is CRITICAL: CRITICAL: Puppet has 1 failures [20:27:02] PROBLEM - puppet last run on ms-fe1003 is CRITICAL: CRITICAL: Puppet has 1 failures [20:27:02] PROBLEM - puppet last run on rdb1004 is CRITICAL: CRITICAL: Puppet has 1 failures [20:27:11] PROBLEM - puppet last run on cp1059 is CRITICAL: CRITICAL: Puppet has 1 failures [20:27:11] PROBLEM - puppet last run on analytics1036 is CRITICAL: CRITICAL: Puppet has 1 failures [20:27:12] PROBLEM - puppet last run on dbstore1001 is CRITICAL: CRITICAL: Puppet has 1 failures [20:27:12] PROBLEM - puppet last run on mw1040 is CRITICAL: CRITICAL: Puppet has 1 failures [20:27:12] PROBLEM - puppet last run on cp4007 is CRITICAL: CRITICAL: Puppet has 1 failures [20:27:21] PROBLEM - puppet last run on mw1178 is CRITICAL: CRITICAL: Puppet has 1 failures [20:27:21] PROBLEM - puppet last run on ms-be1001 is CRITICAL: CRITICAL: Puppet has 1 failures [20:27:22] PROBLEM - puppet last run on mw1132 is CRITICAL: CRITICAL: Puppet has 1 failures [20:27:22] PROBLEM - puppet last run on amssq44 is CRITICAL: CRITICAL: Puppet has 1 failures [20:27:22] PROBLEM - puppet last run on wtp1019 is CRITICAL: CRITICAL: Puppet has 1 failures [20:27:22] PROBLEM - puppet last run on mw1048 is CRITICAL: CRITICAL: Puppet has 1 failures [20:27:22] PROBLEM - puppet last run on cp3015 is CRITICAL: CRITICAL: Puppet has 1 failures [20:27:31] PROBLEM - puppet last run on mc1011 is CRITICAL: CRITICAL: Puppet has 1 failures [20:27:41] PROBLEM - puppet last run on mw1067 is CRITICAL: CRITICAL: Puppet has 1 failures [20:27:42] PROBLEM - puppet last run on mw1072 is CRITICAL: CRITICAL: Puppet has 1 failures [20:27:42] PROBLEM - puppet last run on mw1038 is CRITICAL: CRITICAL: Puppet has 1 failures [20:27:42] PROBLEM - puppet last run on ms1001 is CRITICAL: CRITICAL: Puppet has 1 failures [20:27:42] PROBLEM - puppet last run on mw1028 is CRITICAL: CRITICAL: Puppet has 1 failures [20:27:42] PROBLEM - puppet last run on es1001 is CRITICAL: CRITICAL: Puppet has 1 failures [20:27:42] PROBLEM - puppet last run on mc1004 is CRITICAL: CRITICAL: Puppet has 1 failures [20:27:43] PROBLEM - puppet last run on amssq43 is CRITICAL: CRITICAL: Puppet has 1 failures [20:27:43] PROBLEM - puppet last run on search1020 is CRITICAL: CRITICAL: Puppet has 1 failures [20:27:44] PROBLEM - puppet last run on mw1005 is CRITICAL: CRITICAL: Puppet has 1 failures [20:27:51] PROBLEM - puppet last run on pc1001 is CRITICAL: CRITICAL: Puppet has 1 failures [20:27:51] PROBLEM - puppet last run on mw1134 is CRITICAL: CRITICAL: Puppet has 1 failures [20:27:52] PROBLEM - puppet last run on logstash1003 is CRITICAL: CRITICAL: Puppet has 1 failures [20:27:52] PROBLEM - puppet last run on mw1080 is CRITICAL: CRITICAL: Puppet has 1 failures [20:27:52] PROBLEM - puppet last run on wtp1017 is CRITICAL: CRITICAL: Puppet has 1 failures [20:27:52] PROBLEM - puppet last run on magnesium is CRITICAL: CRITICAL: Puppet has 1 failures [20:27:52] PROBLEM - puppet last run on ms-be2002 is CRITICAL: CRITICAL: Puppet has 1 failures [20:27:53] PROBLEM - puppet last run on uranium is CRITICAL: CRITICAL: Puppet has 1 failures [20:28:01] PROBLEM - puppet last run on cp3006 is CRITICAL: CRITICAL: Puppet has 1 failures [20:28:01] PROBLEM - puppet last run on caesium is CRITICAL: CRITICAL: Puppet has 1 failures [20:28:01] PROBLEM - puppet last run on mw1063 is CRITICAL: CRITICAL: Puppet has 1 failures [20:28:11] PROBLEM - puppet last run on mw1106 is CRITICAL: CRITICAL: Puppet has 1 failures [20:28:15] PROBLEM - puppet last run on mw1007 is CRITICAL: CRITICAL: Puppet has 1 failures [20:28:15] PROBLEM - puppet last run on ms-be1013 is CRITICAL: CRITICAL: Puppet has 1 failures [20:28:15] PROBLEM - puppet last run on mw1115 is CRITICAL: CRITICAL: Puppet has 1 failures [20:28:22] PROBLEM - puppet last run on carbon is CRITICAL: CRITICAL: Puppet has 1 failures [20:28:22] PROBLEM - puppet last run on db1017 is CRITICAL: CRITICAL: Puppet has 1 failures [20:28:22] PROBLEM - puppet last run on cp1049 is CRITICAL: CRITICAL: Puppet has 1 failures [20:28:32] PROBLEM - puppet last run on ms-be1004 is CRITICAL: CRITICAL: Puppet has 1 failures [20:28:33] PROBLEM - puppet last run on mw1089 is CRITICAL: CRITICAL: Puppet has 1 failures [20:28:33] PROBLEM - puppet last run on mw1006 is CRITICAL: CRITICAL: Puppet has 1 failures [20:28:33] PROBLEM - puppet last run on db1073 is CRITICAL: CRITICAL: Puppet has 1 failures [20:28:33] PROBLEM - puppet last run on mc1016 is CRITICAL: CRITICAL: Puppet has 1 failures [20:28:33] PROBLEM - puppet last run on db73 is CRITICAL: CRITICAL: Puppet has 1 failures [20:28:33] PROBLEM - puppet last run on cp3008 is CRITICAL: CRITICAL: Puppet has 1 failures [20:28:34] PROBLEM - puppet last run on mw1031 is CRITICAL: CRITICAL: Puppet has 1 failures [20:28:37] !log Puppet failures appear to be caused by apt-get timeouts [20:28:41] PROBLEM - puppet last run on mw1197 is CRITICAL: CRITICAL: Puppet has 1 failures [20:28:41] PROBLEM - puppet last run on wtp1006 is CRITICAL: CRITICAL: Puppet has 1 failures [20:28:41] PROBLEM - puppet last run on cp4006 is CRITICAL: CRITICAL: Puppet has 1 failures [20:28:41] PROBLEM - puppet last run on dbproxy1002 is CRITICAL: CRITICAL: Puppet has 1 failures [20:28:41] PROBLEM - puppet last run on netmon1001 is CRITICAL: CRITICAL: Puppet has 1 failures [20:28:42] PROBLEM - puppet last run on mw1145 is CRITICAL: CRITICAL: Puppet has 1 failures [20:28:42] PROBLEM - puppet last run on amssq59 is CRITICAL: CRITICAL: Puppet has 1 failures [20:28:42] Logged the message, Master [20:28:43] PROBLEM - puppet last run on db1029 is CRITICAL: CRITICAL: Puppet has 1 failures [20:28:51] PROBLEM - puppet last run on db1053 is CRITICAL: CRITICAL: Puppet has 1 failures [20:28:51] PROBLEM - puppet last run on platinum is CRITICAL: CRITICAL: Puppet has 1 failures [20:28:51] PROBLEM - puppet last run on db74 is CRITICAL: CRITICAL: Puppet has 1 failures [20:28:51] PROBLEM - puppet last run on es1008 is CRITICAL: CRITICAL: Puppet has 1 failures [20:28:51] PROBLEM - puppet last run on analytics1041 is CRITICAL: CRITICAL: Puppet has 1 failures [20:28:52] PROBLEM - puppet last run on mw1200 is CRITICAL: CRITICAL: Puppet has 1 failures [20:28:52] PROBLEM - puppet last run on amssq49 is CRITICAL: CRITICAL: Puppet has 1 failures [20:28:53] PROBLEM - puppet last run on mw1059 is CRITICAL: CRITICAL: Puppet has 1 failures [20:28:53] PROBLEM - puppet last run on analytics1040 is CRITICAL: CRITICAL: Puppet has 1 failures [20:28:54] PROBLEM - puppet last run on mw1141 is CRITICAL: CRITICAL: Puppet has 1 failures [20:28:54] PROBLEM - puppet last run on amslvs2 is CRITICAL: CRITICAL: Puppet has 1 failures [20:28:55] PROBLEM - puppet last run on potassium is CRITICAL: CRITICAL: Puppet has 1 failures [20:28:55] PROBLEM - puppet last run on cp1039 is CRITICAL: CRITICAL: Puppet has 1 failures [20:28:56] PROBLEM - puppet last run on cerium is CRITICAL: CRITICAL: Puppet has 1 failures [20:28:56] PROBLEM - puppet last run on neptunium is CRITICAL: CRITICAL: Puppet has 1 failures [20:28:57] PROBLEM - puppet last run on lvs4002 is CRITICAL: CRITICAL: Puppet has 1 failures [20:29:01] PROBLEM - puppet last run on amssq54 is CRITICAL: CRITICAL: Puppet has 1 failures [20:29:01] PROBLEM - puppet last run on wtp1009 is CRITICAL: CRITICAL: Puppet has 1 failures [20:29:11] PROBLEM - puppet last run on analytics1017 is CRITICAL: CRITICAL: Puppet has 1 failures [20:29:15] PROBLEM - puppet last run on logstash1001 is CRITICAL: CRITICAL: Puppet has 1 failures [20:29:15] PROBLEM - puppet last run on mw1174 is CRITICAL: CRITICAL: Puppet has 1 failures [20:29:15] PROBLEM - puppet last run on mw1041 is CRITICAL: CRITICAL: Puppet has 1 failures [20:29:15] PROBLEM - puppet last run on mw1045 is CRITICAL: CRITICAL: Puppet has 1 failures [20:29:15] PROBLEM - puppet last run on mw1140 is CRITICAL: CRITICAL: Puppet has 1 failures [20:29:16] PROBLEM - puppet last run on wtp1020 is CRITICAL: CRITICAL: Puppet has 1 failures [20:29:16] PROBLEM - puppet last run on ms-be1003 is CRITICAL: CRITICAL: Puppet has 1 failures [20:29:17] PROBLEM - puppet last run on analytics1033 is CRITICAL: CRITICAL: Puppet has 1 failures [20:29:17] PROBLEM - puppet last run on amssq32 is CRITICAL: CRITICAL: Puppet has 1 failures [20:29:18] PROBLEM - puppet last run on mw1026 is CRITICAL: CRITICAL: Puppet has 1 failures [20:29:18] PROBLEM - puppet last run on cp1070 is CRITICAL: CRITICAL: Puppet has 1 failures [20:29:19] PROBLEM - puppet last run on db1065 is CRITICAL: CRITICAL: Puppet has 1 failures [20:29:21] PROBLEM - puppet last run on searchidx1001 is CRITICAL: CRITICAL: Puppet has 1 failures [20:29:21] PROBLEM - puppet last run on cp1055 is CRITICAL: CRITICAL: Puppet has 1 failures [20:29:21] PROBLEM - puppet last run on snapshot1003 is CRITICAL: CRITICAL: Puppet has 1 failures [20:29:21] PROBLEM - puppet last run on elastic1004 is CRITICAL: CRITICAL: Puppet has 1 failures [20:29:22] PROBLEM - puppet last run on mw1160 is CRITICAL: CRITICAL: Puppet has 1 failures [20:29:22] PROBLEM - puppet last run on xenon is CRITICAL: CRITICAL: Puppet has 1 failures [20:29:22] PROBLEM - puppet last run on gold is CRITICAL: CRITICAL: Puppet has 1 failures [20:29:23] PROBLEM - puppet last run on elastic1001 is CRITICAL: CRITICAL: Puppet has 1 failures [20:29:23] PROBLEM - puppet last run on cp3020 is CRITICAL: CRITICAL: Puppet has 1 failures [20:29:24] PROBLEM - puppet last run on mw1012 is CRITICAL: CRITICAL: Puppet has 1 failures [20:29:31] PROBLEM - puppet last run on db2039 is CRITICAL: CRITICAL: Puppet has 1 failures [20:29:37] PROBLEM - puppet last run on db2034 is CRITICAL: CRITICAL: Puppet has 1 failures [20:29:37] PROBLEM - puppet last run on rbf1002 is CRITICAL: CRITICAL: Puppet has 1 failures [20:29:37] PROBLEM - puppet last run on analytics1020 is CRITICAL: CRITICAL: Puppet has 1 failures [20:29:37] PROBLEM - puppet last run on amssq53 is CRITICAL: CRITICAL: Puppet has 1 failures [20:29:37] PROBLEM - puppet last run on db2005 is CRITICAL: CRITICAL: Puppet has 1 failures [20:29:37] PROBLEM - puppet last run on ms-be1006 is CRITICAL: CRITICAL: Puppet has 1 failures [20:29:46] PROBLEM - puppet last run on elastic1007 is CRITICAL: CRITICAL: Puppet has 1 failures [20:29:51] PROBLEM - puppet last run on amssq61 is CRITICAL: CRITICAL: Puppet has 1 failures [20:29:52] PROBLEM - puppet last run on helium is CRITICAL: CRITICAL: Puppet has 1 failures [20:29:52] PROBLEM - puppet last run on mc1003 is CRITICAL: CRITICAL: Puppet has 1 failures [20:29:52] PROBLEM - puppet last run on elastic1012 is CRITICAL: CRITICAL: Puppet has 1 failures [20:29:52] PROBLEM - puppet last run on lead is CRITICAL: CRITICAL: Puppet has 1 failures [20:30:02] PROBLEM - puppet last run on cp1047 is CRITICAL: CRITICAL: Puppet has 1 failures [20:30:02] PROBLEM - puppet last run on db1031 is CRITICAL: CRITICAL: Puppet has 1 failures [20:30:02] PROBLEM - puppet last run on analytics1025 is CRITICAL: CRITICAL: Puppet has 1 failures [20:30:02] PROBLEM - puppet last run on wtp1016 is CRITICAL: CRITICAL: Puppet has 1 failures [20:30:02] PROBLEM - puppet last run on mc1006 is CRITICAL: CRITICAL: Puppet has 1 failures [20:30:02] PROBLEM - puppet last run on lvs1002 is CRITICAL: CRITICAL: Puppet has 1 failures [20:30:02] PROBLEM - puppet last run on search1016 is CRITICAL: CRITICAL: Puppet has 1 failures [20:30:11] PROBLEM - puppet last run on db2019 is CRITICAL: CRITICAL: Puppet has 1 failures [20:30:11] PROBLEM - puppet last run on ssl1002 is CRITICAL: CRITICAL: Puppet has 1 failures [20:30:12] PROBLEM - puppet last run on mw1164 is CRITICAL: CRITICAL: Puppet has 1 failures [20:30:12] PROBLEM - puppet last run on mw1009 is CRITICAL: CRITICAL: Puppet has 1 failures [20:30:12] PROBLEM - puppet last run on mw1069 is CRITICAL: CRITICAL: Puppet has 1 failures [20:30:12] PROBLEM - puppet last run on mw1117 is CRITICAL: CRITICAL: Puppet has 1 failures [20:30:12] PROBLEM - puppet last run on mw1082 is CRITICAL: CRITICAL: Puppet has 1 failures [20:30:21] PROBLEM - puppet last run on lvs1005 is CRITICAL: CRITICAL: Puppet has 1 failures [20:30:21] PROBLEM - puppet last run on db2009 is CRITICAL: CRITICAL: Puppet has 1 failures [20:30:21] PROBLEM - puppet last run on ms1004 is CRITICAL: CRITICAL: Puppet has 1 failures [20:30:22] PROBLEM - puppet last run on mw1176 is CRITICAL: CRITICAL: Puppet has 1 failures [20:30:22] PROBLEM - puppet last run on mw1187 is CRITICAL: CRITICAL: Puppet has 1 failures [20:30:22] PROBLEM - puppet last run on mw1088 is CRITICAL: CRITICAL: Puppet has 1 failures [20:30:32] PROBLEM - puppet last run on db1050 is CRITICAL: CRITICAL: Puppet has 1 failures [20:30:32] PROBLEM - puppet last run on ms-be2004 is CRITICAL: CRITICAL: Puppet has 1 failures [20:30:32] PROBLEM - puppet last run on mw1068 is CRITICAL: CRITICAL: Puppet has 1 failures [20:30:32] PROBLEM - puppet last run on mw1046 is CRITICAL: CRITICAL: Puppet has 1 failures [20:30:42] PROBLEM - puppet last run on search1010 is CRITICAL: CRITICAL: Puppet has 1 failures [20:30:42] PROBLEM - puppet last run on mw1100 is CRITICAL: CRITICAL: Puppet has 1 failures [20:30:42] PROBLEM - puppet last run on mw1120 is CRITICAL: CRITICAL: Puppet has 1 failures [20:30:43] PROBLEM - puppet last run on lvs3001 is CRITICAL: CRITICAL: Puppet has 1 failures [20:30:43] PROBLEM - puppet last run on labstore1003 is CRITICAL: CRITICAL: Puppet has 1 failures [20:30:43] PROBLEM - puppet last run on amssq35 is CRITICAL: CRITICAL: Puppet has 1 failures [20:30:43] PROBLEM - puppet last run on cp3003 is CRITICAL: CRITICAL: Puppet has 1 failures [20:30:44] PROBLEM - puppet last run on analytics1035 is CRITICAL: CRITICAL: Puppet has 1 failures [20:30:51] PROBLEM - puppet last run on mw1003 is CRITICAL: CRITICAL: Puppet has 1 failures [20:30:51] PROBLEM - puppet last run on mc1002 is CRITICAL: CRITICAL: Puppet has 1 failures [20:30:51] PROBLEM - puppet last run on elastic1008 is CRITICAL: CRITICAL: Puppet has 1 failures [20:30:51] PROBLEM - puppet last run on mw1042 is CRITICAL: CRITICAL: Puppet has 1 failures [20:30:52] PROBLEM - puppet last run on sca1002 is CRITICAL: CRITICAL: Puppet has 1 failures [20:31:01] PROBLEM - puppet last run on lvs2004 is CRITICAL: CRITICAL: Puppet has 1 failures [20:31:01] PROBLEM - puppet last run on db2002 is CRITICAL: CRITICAL: Puppet has 1 failures [20:31:01] PROBLEM - puppet last run on analytics1030 is CRITICAL: CRITICAL: Puppet has 1 failures [20:31:01] PROBLEM - puppet last run on mw1153 is CRITICAL: CRITICAL: Puppet has 1 failures [20:31:01] PROBLEM - puppet last run on search1018 is CRITICAL: CRITICAL: Puppet has 1 failures [20:31:02] PROBLEM - puppet last run on mw1099 is CRITICAL: CRITICAL: Puppet has 1 failures [20:31:02] PROBLEM - puppet last run on db1018 is CRITICAL: CRITICAL: Puppet has 1 failures [20:31:03] PROBLEM - puppet last run on cp3016 is CRITICAL: CRITICAL: Puppet has 1 failures [20:31:03] PROBLEM - puppet last run on ms-fe1001 is CRITICAL: CRITICAL: Puppet has 1 failures [20:31:04] PROBLEM - puppet last run on cp1061 is CRITICAL: CRITICAL: Puppet has 1 failures [20:31:04] PROBLEM - puppet last run on mw1217 is CRITICAL: CRITICAL: Puppet has 1 failures [20:31:05] PROBLEM - puppet last run on mw1173 is CRITICAL: CRITICAL: Puppet has 1 failures [20:31:14] PROBLEM - puppet last run on mw1025 is CRITICAL: CRITICAL: Puppet has 1 failures [20:31:14] PROBLEM - puppet last run on db1022 is CRITICAL: CRITICAL: Puppet has 1 failures [20:31:14] PROBLEM - puppet last run on mw1189 is CRITICAL: CRITICAL: Puppet has 1 failures [20:31:14] PROBLEM - puppet last run on mw1060 is CRITICAL: CRITICAL: Puppet has 1 failures [20:31:15] PROBLEM - puppet last run on mw1150 is CRITICAL: CRITICAL: Puppet has 1 failures [20:31:15] PROBLEM - puppet last run on mw1205 is CRITICAL: CRITICAL: Puppet has 1 failures [20:31:21] PROBLEM - puppet last run on mw1008 is CRITICAL: CRITICAL: Puppet has 1 failures [20:31:26] RECOVERY - puppet last run on wtp1013 is OK: OK: Puppet is currently enabled, last run 22 seconds ago with 0 failures [20:31:26] PROBLEM - puppet last run on db1066 is CRITICAL: CRITICAL: Puppet has 1 failures [20:31:26] PROBLEM - puppet last run on ms-fe1004 is CRITICAL: CRITICAL: Puppet has 1 failures [20:31:32] PROBLEM - puppet last run on cp1056 is CRITICAL: CRITICAL: Puppet has 1 failures [20:31:32] RECOVERY - puppet last run on rdb1002 is OK: OK: Puppet is currently enabled, last run 16 seconds ago with 0 failures [20:31:32] PROBLEM - puppet last run on mw1166 is CRITICAL: CRITICAL: Puppet has 1 failures [20:31:32] PROBLEM - puppet last run on mw1170 is CRITICAL: CRITICAL: Puppet has 1 failures [20:31:32] PROBLEM - puppet last run on ruthenium is CRITICAL: CRITICAL: Puppet has 1 failures [20:31:33] PROBLEM - puppet last run on db1015 is CRITICAL: CRITICAL: Puppet has 1 failures [20:31:33] RECOVERY - puppet last run on db1057 is OK: OK: Puppet is currently enabled, last run 24 seconds ago with 0 failures [20:31:33] PROBLEM - puppet last run on iron is CRITICAL: CRITICAL: Puppet has 1 failures [20:31:33] PROBLEM - puppet last run on cp3014 is CRITICAL: CRITICAL: Puppet has 1 failures [20:31:41] PROBLEM - puppet last run on mw1123 is CRITICAL: CRITICAL: Puppet has 1 failures [20:31:41] PROBLEM - puppet last run on db1046 is CRITICAL: CRITICAL: Puppet has 1 failures [20:31:41] PROBLEM - puppet last run on db1040 is CRITICAL: CRITICAL: Puppet has 1 failures [20:31:41] PROBLEM - puppet last run on cp4003 is CRITICAL: CRITICAL: Puppet has 1 failures [20:31:51] RECOVERY - puppet last run on db1001 is OK: OK: Puppet is currently enabled, last run 35 seconds ago with 0 failures [20:31:57] PROBLEM - puppet last run on mw1118 is CRITICAL: CRITICAL: Puppet has 1 failures [20:31:58] PROBLEM - puppet last run on db1002 is CRITICAL: CRITICAL: Puppet has 1 failures [20:32:01] PROBLEM - puppet last run on db1059 is CRITICAL: CRITICAL: Puppet has 1 failures [20:32:10] PROBLEM - puppet last run on mw1061 is CRITICAL: CRITICAL: Puppet has 1 failures [20:32:10] PROBLEM - puppet last run on cp4008 is CRITICAL: CRITICAL: Puppet has 1 failures [20:32:11] PROBLEM - puppet last run on mw1119 is CRITICAL: CRITICAL: Puppet has 1 failures [20:32:11] PROBLEM - puppet last run on search1001 is CRITICAL: CRITICAL: Puppet has 1 failures [20:32:11] PROBLEM - puppet last run on mw1092 is CRITICAL: CRITICAL: Puppet has 1 failures [20:32:11] PROBLEM - puppet last run on db2018 is CRITICAL: CRITICAL: Puppet has 1 failures [20:32:21] PROBLEM - puppet last run on db1034 is CRITICAL: CRITICAL: Puppet has 1 failures [20:32:21] RECOVERY - puppet last run on mw1032 is OK: OK: Puppet is currently enabled, last run 49 seconds ago with 0 failures [20:32:21] PROBLEM - puppet last run on db1067 is CRITICAL: CRITICAL: Puppet has 1 failures [20:32:21] PROBLEM - puppet last run on mw1052 is CRITICAL: CRITICAL: Puppet has 1 failures [20:32:22] PROBLEM - puppet last run on mw1065 is CRITICAL: CRITICAL: Puppet has 1 failures [20:32:22] PROBLEM - puppet last run on db1042 is CRITICAL: CRITICAL: Puppet has 1 failures [20:32:22] PROBLEM - puppet last run on amssq47 is CRITICAL: CRITICAL: Puppet has 1 failures [20:32:23] RECOVERY - puppet last run on db2003 is OK: OK: Puppet is currently enabled, last run 60 seconds ago with 0 failures [20:32:23] PROBLEM - puppet last run on dbproxy1001 is CRITICAL: CRITICAL: Puppet has 1 failures [20:32:31] PROBLEM - puppet last run on db2036 is CRITICAL: CRITICAL: Puppet has 1 failures [20:32:31] PROBLEM - puppet last run on mw1144 is CRITICAL: CRITICAL: Puppet has 1 failures [20:32:35] PROBLEM - puppet last run on db1021 is CRITICAL: CRITICAL: Puppet has 1 failures [20:32:36] PROBLEM - puppet last run on logstash1002 is CRITICAL: CRITICAL: Puppet has 1 failures [20:32:36] PROBLEM - puppet last run on search1007 is CRITICAL: CRITICAL: Puppet has 1 failures [20:32:36] PROBLEM - puppet last run on lvs2001 is CRITICAL: CRITICAL: Puppet has 1 failures [20:32:37] PROBLEM - puppet last run on amssq60 is CRITICAL: CRITICAL: Puppet has 1 failures [20:32:41] PROBLEM - puppet last run on labcontrol2001 is CRITICAL: CRITICAL: Puppet has 1 failures [20:33:01] PROBLEM - puppet last run on mw1175 is CRITICAL: CRITICAL: Puppet has 1 failures [20:33:01] PROBLEM - puppet last run on mw1213 is CRITICAL: CRITICAL: Puppet has 1 failures [20:33:02] PROBLEM - puppet last run on mw1129 is CRITICAL: CRITICAL: Puppet has 1 failures [20:33:11] PROBLEM - puppet last run on amssq55 is CRITICAL: CRITICAL: Puppet has 1 failures [20:33:21] PROBLEM - puppet last run on mw1114 is CRITICAL: CRITICAL: Puppet has 1 failures [20:33:30] PROBLEM - puppet last run on mw1054 is CRITICAL: CRITICAL: Puppet has 1 failures [20:33:42] PROBLEM - puppet last run on ms-be2011 is CRITICAL: CRITICAL: Puppet has 1 failures [20:33:42] PROBLEM - puppet last run on silver is CRITICAL: CRITICAL: Puppet has 1 failures [20:35:53] RECOVERY - puppet last run on copper is OK: OK: Puppet is currently enabled, last run 4 seconds ago with 0 failures [20:35:53] RECOVERY - puppet last run on mw1193 is OK: OK: Puppet is currently enabled, last run 2 seconds ago with 0 failures [20:35:54] RECOVERY - puppet last run on cp1067 is OK: OK: Puppet is currently enabled, last run 7 seconds ago with 0 failures [20:35:54] RECOVERY - puppet last run on bast1001 is OK: OK: Puppet is currently enabled, last run 4 seconds ago with 0 failures [20:35:54] RECOVERY - puppet last run on wtp1001 is OK: OK: Puppet is currently enabled, last run 3 seconds ago with 0 failures [20:35:54] RECOVERY - puppet last run on iodine is OK: OK: Puppet is currently enabled, last run 9 seconds ago with 0 failures [20:35:54] RECOVERY - puppet last run on analytics1018 is OK: OK: Puppet is currently enabled, last run 2 seconds ago with 0 failures [20:35:55] RECOVERY - puppet last run on cp1040 is OK: OK: Puppet is currently enabled, last run 9 seconds ago with 0 failures [20:35:55] RECOVERY - puppet last run on mw1071 is OK: OK: Puppet is currently enabled, last run 5 seconds ago with 0 failures [20:35:56] RECOVERY - puppet last run on db1047 is OK: OK: Puppet is currently enabled, last run 11 seconds ago with 0 failures [20:36:01] RECOVERY - puppet last run on db1037 is OK: OK: Puppet is currently enabled, last run 2 seconds ago with 0 failures [20:36:01] RECOVERY - puppet last run on lvs1004 is OK: OK: Puppet is currently enabled, last run 3 seconds ago with 0 failures [20:36:01] RECOVERY - puppet last run on dbstore1002 is OK: OK: Puppet is currently enabled, last run 20 seconds ago with 0 failures [20:36:02] RECOVERY - puppet last run on mw1220 is OK: OK: Puppet is currently enabled, last run 14 seconds ago with 0 failures [20:36:02] RECOVERY - puppet last run on mw1121 is OK: OK: Puppet is currently enabled, last run 10 seconds ago with 0 failures [20:36:02] RECOVERY - puppet last run on analytics1028 is OK: OK: Puppet is currently enabled, last run 23 seconds ago with 0 failures [20:36:02] RECOVERY - puppet last run on osm-cp1001 is OK: OK: Puppet is currently enabled, last run 10 seconds ago with 0 failures [20:36:21] RECOVERY - puppet last run on db1070 is OK: OK: Puppet is currently enabled, last run 53 seconds ago with 0 failures [20:36:22] RECOVERY - puppet last run on cp1045 is OK: OK: Puppet is currently enabled, last run 27 seconds ago with 0 failures [20:36:22] RECOVERY - puppet last run on mercury is OK: OK: Puppet is currently enabled, last run 28 seconds ago with 0 failures [20:36:22] RECOVERY - puppet last run on db1061 is OK: OK: Puppet is currently enabled, last run 45 seconds ago with 0 failures [20:36:22] RECOVERY - puppet last run on palladium is OK: OK: Puppet is currently enabled, last run 0 seconds ago with 0 failures [20:36:23] RECOVERY - puppet last run on db71 is OK: OK: Puppet is currently enabled, last run 28 seconds ago with 0 failures [20:36:23] RECOVERY - puppet last run on virt1008 is OK: OK: Puppet is currently enabled, last run 33 seconds ago with 0 failures [20:36:31] RECOVERY - puppet last run on db1038 is OK: OK: Puppet is currently enabled, last run 32 seconds ago with 0 failures [20:36:41] RECOVERY - puppet last run on iridium is OK: OK: Puppet is currently enabled, last run 7 seconds ago with 0 failures [20:36:41] RECOVERY - puppet last run on mw1107 is OK: OK: Puppet is currently enabled, last run 48 seconds ago with 0 failures [20:36:41] RECOVERY - puppet last run on ms-be1015 is OK: OK: Puppet is currently enabled, last run 5 seconds ago with 0 failures [20:36:41] RECOVERY - puppet last run on cp1066 is OK: OK: Puppet is currently enabled, last run 7 seconds ago with 0 failures [20:36:41] RECOVERY - puppet last run on ms-be1002 is OK: OK: Puppet is currently enabled, last run 15 seconds ago with 0 failures [20:36:42] RECOVERY - puppet last run on fluorine is OK: OK: Puppet is currently enabled, last run 7 seconds ago with 0 failures [20:36:42] RECOVERY - puppet last run on db1045 is OK: OK: Puppet is currently enabled, last run 45 seconds ago with 0 failures [20:36:43] RECOVERY - puppet last run on db1072 is OK: OK: Puppet is currently enabled, last run 57 seconds ago with 0 failures [20:36:43] RECOVERY - puppet last run on ssl1003 is OK: OK: Puppet is currently enabled, last run 6 seconds ago with 0 failures [20:36:44] RECOVERY - puppet last run on db1027 is OK: OK: Puppet is currently enabled, last run 50 seconds ago with 0 failures [20:36:44] RECOVERY - puppet last run on sca1001 is OK: OK: Puppet is currently enabled, last run 10 seconds ago with 0 failures [20:36:45] RECOVERY - puppet last run on wtp1008 is OK: OK: Puppet is currently enabled, last run 44 seconds ago with 0 failures [20:36:45] RECOVERY - puppet last run on wtp1010 is OK: OK: Puppet is currently enabled, last run 18 seconds ago with 0 failures [20:36:46] RECOVERY - puppet last run on es1004 is OK: OK: Puppet is currently enabled, last run 46 seconds ago with 0 failures [20:36:46] RECOVERY - puppet last run on cp1054 is OK: OK: Puppet is currently enabled, last run 17 seconds ago with 0 failures [20:36:47] RECOVERY - puppet last run on mw1194 is OK: OK: Puppet is currently enabled, last run 1 seconds ago with 0 failures [20:36:47] RECOVERY - puppet last run on search1011 is OK: OK: Puppet is currently enabled, last run 11 seconds ago with 0 failures [20:36:48] RECOVERY - puppet last run on amssq39 is OK: OK: Puppet is currently enabled, last run 11 seconds ago with 0 failures [20:36:48] RECOVERY - puppet last run on search1022 is OK: OK: Puppet is currently enabled, last run 59 seconds ago with 0 failures [20:36:49] RECOVERY - puppet last run on analytics1031 is OK: OK: Puppet is currently enabled, last run 21 seconds ago with 0 failures [20:36:49] RECOVERY - puppet last run on mc1015 is OK: OK: Puppet is currently enabled, last run 7 seconds ago with 0 failures [20:36:50] RECOVERY - puppet last run on rbf1001 is OK: OK: Puppet is currently enabled, last run 14 seconds ago with 0 failures [20:36:51] RECOVERY - puppet last run on search1004 is OK: OK: Puppet is currently enabled, last run 16 seconds ago with 0 failures [20:36:51] RECOVERY - puppet last run on search1012 is OK: OK: Puppet is currently enabled, last run 9 seconds ago with 0 failures [20:36:51] RECOVERY - puppet last run on db2011 is OK: OK: Puppet is currently enabled, last run 59 seconds ago with 0 failures [20:36:52] RECOVERY - puppet last run on ssl1001 is OK: OK: Puppet is currently enabled, last run 16 seconds ago with 0 failures [20:36:52] RECOVERY - puppet last run on mw1090 is OK: OK: Puppet is currently enabled, last run 9 seconds ago with 0 failures [20:36:53] RECOVERY - puppet last run on linne is OK: OK: Puppet is currently enabled, last run 10 seconds ago with 0 failures [20:36:53] RECOVERY - puppet last run on lvs1003 is OK: OK: Puppet is currently enabled, last run 22 seconds ago with 0 failures [20:36:54] RECOVERY - puppet last run on mw1209 is OK: OK: Puppet is currently enabled, last run 7 seconds ago with 0 failures [20:36:54] RECOVERY - puppet last run on amssq31 is OK: OK: Puppet is currently enabled, last run 9 seconds ago with 0 failures [20:36:55] RECOVERY - puppet last run on cp3011 is OK: OK: Puppet is currently enabled, last run 20 seconds ago with 0 failures [20:36:55] RECOVERY - puppet last run on mw1021 is OK: OK: Puppet is currently enabled, last run 15 seconds ago with 0 failures [20:36:56] RECOVERY - puppet last run on db1006 is OK: OK: Puppet is currently enabled, last run 59 seconds ago with 0 failures [20:36:56] RECOVERY - puppet last run on ms-fe1002 is OK: OK: Puppet is currently enabled, last run 24 seconds ago with 0 failures [20:37:01] RECOVERY - puppet last run on mw1010 is OK: OK: Puppet is currently enabled, last run 54 seconds ago with 0 failures [20:37:01] RECOVERY - puppet last run on mw1110 is OK: OK: Puppet is currently enabled, last run 16 seconds ago with 0 failures [20:37:01] RECOVERY - puppet last run on labsdb1005 is OK: OK: Puppet is currently enabled, last run 36 seconds ago with 0 failures [20:37:01] RECOVERY - puppet last run on mw1093 is OK: OK: Puppet is currently enabled, last run 64 seconds ago with 0 failures [20:37:01] RECOVERY - puppet last run on search1006 is OK: OK: Puppet is currently enabled, last run 19 seconds ago with 0 failures [20:37:02] RECOVERY - puppet last run on mw1112 is OK: OK: Puppet is currently enabled, last run 20 seconds ago with 0 failures [20:37:02] RECOVERY - puppet last run on cp1044 is OK: OK: Puppet is currently enabled, last run 23 seconds ago with 0 failures [20:37:03] RECOVERY - puppet last run on db60 is OK: OK: Puppet is currently enabled, last run 16 seconds ago with 0 failures [20:37:03] RECOVERY - puppet last run on analytics1004 is OK: OK: Puppet is currently enabled, last run 26 seconds ago with 0 failures [20:37:04] RECOVERY - puppet last run on dysprosium is OK: OK: Puppet is currently enabled, last run 40 seconds ago with 0 failures [20:37:04] RECOVERY - puppet last run on db2028 is OK: OK: Puppet is currently enabled, last run 20 seconds ago with 0 failures [20:37:05] RECOVERY - puppet last run on mw1167 is OK: OK: Puppet is currently enabled, last run 40 seconds ago with 0 failures [20:37:05] RECOVERY - puppet last run on mw1086 is OK: OK: Puppet is currently enabled, last run 10 seconds ago with 0 failures [20:37:06] RECOVERY - puppet last run on mw1016 is OK: OK: Puppet is currently enabled, last run 11 seconds ago with 0 failures [20:37:06] RECOVERY - puppet last run on search1003 is OK: OK: Puppet is currently enabled, last run 39 seconds ago with 0 failures [20:37:07] RECOVERY - puppet last run on db2012 is OK: OK: Puppet is currently enabled, last run 27 seconds ago with 0 failures [20:37:07] RECOVERY - puppet last run on db2030 is OK: OK: Puppet is currently enabled, last run 22 seconds ago with 0 failures [20:37:08] RECOVERY - puppet last run on terbium is OK: OK: Puppet is currently enabled, last run 15 seconds ago with 0 failures [20:37:08] RECOVERY - puppet last run on mw1154 is OK: OK: Puppet is currently enabled, last run 30 seconds ago with 0 failures [20:37:09] RECOVERY - puppet last run on cp1068 is OK: OK: Puppet is currently enabled, last run 26 seconds ago with 0 failures [20:37:09] RECOVERY - puppet last run on mw1113 is OK: OK: Puppet is currently enabled, last run 19 seconds ago with 0 failures [20:37:21] RECOVERY - puppet last run on ms-be1014 is OK: OK: Puppet is currently enabled, last run 1 seconds ago with 0 failures [20:37:21] RECOVERY - puppet last run on ms-be1010 is OK: OK: Puppet is currently enabled, last run 35 seconds ago with 0 failures [20:37:21] RECOVERY - puppet last run on mw1207 is OK: OK: Puppet is currently enabled, last run 7 seconds ago with 0 failures [20:37:21] RECOVERY - puppet last run on tmh1001 is OK: OK: Puppet is currently enabled, last run 31 seconds ago with 0 failures [20:37:22] RECOVERY - puppet last run on amssq50 is OK: OK: Puppet is currently enabled, last run 20 seconds ago with 0 failures [20:37:22] RECOVERY - puppet last run on cp3005 is OK: OK: Puppet is currently enabled, last run 15 seconds ago with 0 failures [20:37:22] RECOVERY - puppet last run on mw1131 is OK: OK: Puppet is currently enabled, last run 28 seconds ago with 0 failures [20:37:23] RECOVERY - puppet last run on cp3018 is OK: OK: Puppet is currently enabled, last run 16 seconds ago with 0 failures [20:37:23] RECOVERY - puppet last run on cp4010 is OK: OK: Puppet is currently enabled, last run 8 seconds ago with 0 failures [20:37:24] RECOVERY - puppet last run on calcium is OK: OK: Puppet is currently enabled, last run 50 seconds ago with 0 failures [20:37:24] RECOVERY - puppet last run on mw1155 is OK: OK: Puppet is currently enabled, last run 34 seconds ago with 0 failures [20:37:25] RECOVERY - puppet last run on mw1142 is OK: OK: Puppet is currently enabled, last run 39 seconds ago with 0 failures [20:37:25] RECOVERY - puppet last run on mw1064 is OK: OK: Puppet is currently enabled, last run 45 seconds ago with 0 failures [20:38:41] RECOVERY - puppet last run on lanthanum is OK: OK: Puppet is currently enabled, last run 49 seconds ago with 0 failures [20:38:47] RECOVERY - puppet last run on mw1095 is OK: OK: Puppet is currently enabled, last run 28 seconds ago with 0 failures [20:38:48] RECOVERY - puppet last run on mw1137 is OK: OK: Puppet is currently enabled, last run 23 seconds ago with 0 failures [20:38:48] RECOVERY - puppet last run on mw1020 is OK: OK: Puppet is currently enabled, last run 34 seconds ago with 0 failures [20:38:48] RECOVERY - puppet last run on es1003 is OK: OK: Puppet is currently enabled, last run 31 seconds ago with 0 failures [20:38:51] RECOVERY - puppet last run on ms-be2010 is OK: OK: Puppet is currently enabled, last run 6 seconds ago with 0 failures [20:38:51] RECOVERY - puppet last run on mw1214 is OK: OK: Puppet is currently enabled, last run 7 seconds ago with 0 failures [20:38:51] RECOVERY - puppet last run on ms-be1005 is OK: OK: Puppet is currently enabled, last run 24 seconds ago with 0 failures [20:38:51] RECOVERY - puppet last run on cp1057 is OK: OK: Puppet is currently enabled, last run 27 seconds ago with 0 failures [20:38:52] RECOVERY - puppet last run on virt1005 is OK: OK: Puppet is currently enabled, last run 55 seconds ago with 0 failures [20:38:52] RECOVERY - puppet last run on ocg1002 is OK: OK: Puppet is currently enabled, last run 60 seconds ago with 0 failures [20:38:52] RECOVERY - puppet last run on mw1015 is OK: OK: Puppet is currently enabled, last run 62 seconds ago with 0 failures [20:39:01] RECOVERY - puppet last run on nickel is OK: OK: Puppet is currently enabled, last run 48 seconds ago with 0 failures [20:39:01] RECOVERY - puppet last run on lvs4001 is OK: OK: Puppet is currently enabled, last run 49 seconds ago with 0 failures [20:39:02] RECOVERY - puppet last run on cp4015 is OK: OK: Puppet is currently enabled, last run 57 seconds ago with 0 failures [20:39:11] RECOVERY - puppet last run on amssq52 is OK: OK: Puppet is currently enabled, last run 59 seconds ago with 0 failures [20:39:11] RECOVERY - puppet last run on search1009 is OK: OK: Puppet is currently enabled, last run 37 seconds ago with 0 failures [20:39:11] RECOVERY - puppet last run on lvs1006 is OK: OK: Puppet is currently enabled, last run 35 seconds ago with 0 failures [20:39:12] RECOVERY - puppet last run on cp4017 is OK: OK: Puppet is currently enabled, last run 12 seconds ago with 0 failures [20:39:12] RECOVERY - puppet last run on analytics1024 is OK: OK: Puppet is currently enabled, last run 38 seconds ago with 0 failures [20:39:21] RECOVERY - puppet last run on praseodymium is OK: OK: Puppet is currently enabled, last run 23 seconds ago with 0 failures [20:39:21] RECOVERY - puppet last run on mc1008 is OK: OK: Puppet is currently enabled, last run 2 seconds ago with 0 failures [20:39:21] RECOVERY - puppet last run on mw1083 is OK: OK: Puppet is currently enabled, last run 43 seconds ago with 0 failures [20:39:21] RECOVERY - puppet last run on search1021 is OK: OK: Puppet is currently enabled, last run 39 seconds ago with 0 failures [20:39:26] RECOVERY - puppet last run on wtp1014 is OK: OK: Puppet is currently enabled, last run 44 seconds ago with 0 failures [20:39:27] RECOVERY - puppet last run on db1019 is OK: OK: Puppet is currently enabled, last run 57 seconds ago with 0 failures [20:39:31] RECOVERY - puppet last run on mw1182 is OK: OK: Puppet is currently enabled, last run 62 seconds ago with 0 failures [20:39:31] RECOVERY - puppet last run on elastic1016 is OK: OK: Puppet is currently enabled, last run 62 seconds ago with 0 failures [20:39:31] RECOVERY - puppet last run on analytics1034 is OK: OK: Puppet is currently enabled, last run 21 seconds ago with 0 failures [20:39:31] RECOVERY - puppet last run on mw1169 is OK: OK: Puppet is currently enabled, last run 53 seconds ago with 0 failures [20:39:31] RECOVERY - puppet last run on mw1127 is OK: OK: Puppet is currently enabled, last run 25 seconds ago with 0 failures [20:39:31] RECOVERY - puppet last run on db1041 is OK: OK: Puppet is currently enabled, last run 14 seconds ago with 0 failures [20:39:31] RECOVERY - puppet last run on analytics1019 is OK: OK: Puppet is currently enabled, last run 4 seconds ago with 0 failures [20:39:32] RECOVERY - puppet last run on cp1051 is OK: OK: Puppet is currently enabled, last run 0 seconds ago with 0 failures [20:39:32] RECOVERY - puppet last run on es4 is OK: OK: Puppet is currently enabled, last run 35 seconds ago with 0 failures [20:39:33] RECOVERY - puppet last run on analytics1029 is OK: OK: Puppet is currently enabled, last run 54 seconds ago with 0 failures [20:39:33] RECOVERY - puppet last run on db1058 is OK: OK: Puppet is currently enabled, last run 60 seconds ago with 0 failures [20:39:42] RECOVERY - puppet last run on erbium is OK: OK: Puppet is currently enabled, last run 53 seconds ago with 0 failures [20:39:42] RECOVERY - puppet last run on cp4013 is OK: OK: Puppet is currently enabled, last run 21 seconds ago with 0 failures [20:39:46] RECOVERY - puppet last run on mw1013 is OK: OK: Puppet is currently enabled, last run 3 seconds ago with 0 failures [20:39:46] RECOVERY - puppet last run on mw1196 is OK: OK: Puppet is currently enabled, last run 17 seconds ago with 0 failures [20:39:46] RECOVERY - puppet last run on db2033 is OK: OK: Puppet is currently enabled, last run 14 seconds ago with 0 failures [20:39:46] RECOVERY - puppet last run on virt1002 is OK: OK: Puppet is currently enabled, last run 1 seconds ago with 0 failures [20:39:46] RECOVERY - puppet last run on mw1138 is OK: OK: Puppet is currently enabled, last run 5 seconds ago with 0 failures [20:39:51] RECOVERY - puppet last run on cp1069 is OK: OK: Puppet is currently enabled, last run 2 seconds ago with 0 failures [20:39:51] RECOVERY - puppet last run on amssq57 is OK: OK: Puppet is currently enabled, last run 7 seconds ago with 0 failures [20:39:51] RECOVERY - puppet last run on mw1136 is OK: OK: Puppet is currently enabled, last run 52 seconds ago with 0 failures [20:39:51] RECOVERY - puppet last run on mw1184 is OK: OK: Puppet is currently enabled, last run 50 seconds ago with 0 failures [20:40:01] RECOVERY - puppet last run on mw1094 is OK: OK: Puppet is currently enabled, last run 43 seconds ago with 0 failures [20:40:02] RECOVERY - puppet last run on analytics1039 is OK: OK: Puppet is currently enabled, last run 9 seconds ago with 0 failures [20:40:02] RECOVERY - puppet last run on cp1065 is OK: OK: Puppet is currently enabled, last run 37 seconds ago with 0 failures [20:40:11] RECOVERY - puppet last run on amssq58 is OK: OK: Puppet is currently enabled, last run 7 seconds ago with 0 failures [20:40:11] RECOVERY - puppet last run on labsdb1001 is OK: OK: Puppet is currently enabled, last run 21 seconds ago with 0 failures [20:40:12] RECOVERY - puppet last run on zirconium is OK: OK: Puppet is currently enabled, last run 50 seconds ago with 0 failures [20:40:18] RECOVERY - puppet last run on db1010 is OK: OK: Puppet is currently enabled, last run 2 seconds ago with 0 failures [20:40:18] RECOVERY - puppet last run on es7 is OK: OK: Puppet is currently enabled, last run 55 seconds ago with 0 failures [20:40:18] RECOVERY - puppet last run on mw1191 is OK: OK: Puppet is currently enabled, last run 1 seconds ago with 0 failures [20:40:18] RECOVERY - puppet last run on es10 is OK: OK: Puppet is currently enabled, last run 49 seconds ago with 0 failures [20:40:18] RECOVERY - puppet last run on amslvs4 is OK: OK: Puppet is currently enabled, last run 31 seconds ago with 0 failures [20:40:21] RECOVERY - puppet last run on mw1096 is OK: OK: Puppet is currently enabled, last run 39 seconds ago with 0 failures [20:40:32] RECOVERY - puppet last run on mw1036 is OK: OK: Puppet is currently enabled, last run 41 seconds ago with 0 failures [20:40:33] RECOVERY - puppet last run on search1014 is OK: OK: Puppet is currently enabled, last run 49 seconds ago with 0 failures [20:40:33] RECOVERY - puppet last run on wtp1021 is OK: OK: Puppet is currently enabled, last run 9 seconds ago with 0 failures [20:40:33] RECOVERY - puppet last run on db1024 is OK: OK: Puppet is currently enabled, last run 56 seconds ago with 0 failures [20:40:55] RECOVERY - puppet last run on analytics1036 is OK: OK: Puppet is currently enabled, last run 48 seconds ago with 0 failures [20:40:59] RECOVERY - puppet last run on es1006 is OK: OK: Puppet is currently enabled, last run 38 seconds ago with 0 failures [20:40:59] RECOVERY - puppet last run on ytterbium is OK: OK: Puppet is currently enabled, last run 43 seconds ago with 0 failures [20:41:00] RECOVERY - puppet last run on cp1043 is OK: OK: Puppet is currently enabled, last run 45 seconds ago with 0 failures [20:41:00] RECOVERY - puppet last run on mc1010 is OK: OK: Puppet is currently enabled, last run 40 seconds ago with 0 failures [20:41:00] RECOVERY - puppet last run on mw1035 is OK: OK: Puppet is currently enabled, last run 58 seconds ago with 0 failures [20:41:00] RECOVERY - puppet last run on mw1216 is OK: OK: Puppet is currently enabled, last run 19 seconds ago with 0 failures [20:41:00] RECOVERY - puppet last run on es1009 is OK: OK: Puppet is currently enabled, last run 40 seconds ago with 0 failures [20:41:00] RECOVERY - puppet last run on cp3013 is OK: OK: Puppet is currently enabled, last run 5 seconds ago with 0 failures [20:41:09] RECOVERY - puppet last run on mw1109 is OK: OK: Puppet is currently enabled, last run 51 seconds ago with 0 failures [20:41:10] RECOVERY - puppet last run on analytics1015 is OK: OK: Puppet is currently enabled, last run 21 seconds ago with 0 failures [20:41:15] RECOVERY - puppet last run on mc1009 is OK: OK: Puppet is currently enabled, last run 14 seconds ago with 0 failures [20:41:19] RECOVERY - puppet last run on mw1161 is OK: OK: Puppet is currently enabled, last run 30 seconds ago with 0 failures [20:41:19] RECOVERY - puppet last run on db1007 is OK: OK: Puppet is currently enabled, last run 36 seconds ago with 0 failures [20:41:19] RECOVERY - puppet last run on vanadium is OK: OK: Puppet is currently enabled, last run 30 seconds ago with 0 failures [20:41:19] RECOVERY - puppet last run on mw1124 is OK: OK: Puppet is currently enabled, last run 3 seconds ago with 0 failures [20:41:20] RECOVERY - puppet last run on mw1147 is OK: OK: Puppet is currently enabled, last run 36 seconds ago with 0 failures [20:41:20] RECOVERY - puppet last run on pc1001 is OK: OK: Puppet is currently enabled, last run 2 seconds ago with 0 failures [20:41:20] RECOVERY - puppet last run on mw1218 is OK: OK: Puppet is currently enabled, last run 34 seconds ago with 0 failures [20:41:20] RECOVERY - puppet last run on cp3022 is OK: OK: Puppet is currently enabled, last run 43 seconds ago with 0 failures [20:41:20] RECOVERY - puppet last run on ssl3003 is OK: OK: Puppet is currently enabled, last run 44 seconds ago with 0 failures [20:41:25] RECOVERY - puppet last run on logstash1003 is OK: OK: Puppet is currently enabled, last run 1 seconds ago with 0 failures [20:41:25] RECOVERY - puppet last run on neon is OK: OK: Puppet is currently enabled, last run 38 seconds ago with 0 failures [20:41:29] RECOVERY - puppet last run on wtp1017 is OK: OK: Puppet is currently enabled, last run 5 seconds ago with 0 failures [20:41:30] RECOVERY - puppet last run on uranium is OK: OK: Puppet is currently enabled, last run 5 seconds ago with 0 failures [20:41:35] RECOVERY - puppet last run on mw1130 is OK: OK: Puppet is currently enabled, last run 18 seconds ago with 0 failures [20:41:35] RECOVERY - puppet last run on rdb1004 is OK: OK: Puppet is currently enabled, last run 19 seconds ago with 0 failures [20:41:41] RECOVERY - puppet last run on mw1040 is OK: OK: Puppet is currently enabled, last run 60 seconds ago with 0 failures [20:41:46] RECOVERY - puppet last run on cp1059 is OK: OK: Puppet is currently enabled, last run 11 seconds ago with 0 failures [20:41:56] RECOVERY - puppet last run on mw1115 is OK: OK: Puppet is currently enabled, last run 2 seconds ago with 0 failures [20:41:56] RECOVERY - puppet last run on dbstore1001 is OK: OK: Puppet is currently enabled, last run 60 seconds ago with 0 failures [20:41:57] RECOVERY - puppet last run on mw1192 is OK: OK: Puppet is currently enabled, last run 56 seconds ago with 0 failures [20:42:01] RECOVERY - puppet last run on mc1011 is OK: OK: Puppet is currently enabled, last run 17 seconds ago with 0 failures [20:42:01] RECOVERY - puppet last run on ocg1001 is OK: OK: Puppet is currently enabled, last run 58 seconds ago with 0 failures [20:42:01] RECOVERY - puppet last run on cp4007 is OK: OK: Puppet is currently enabled, last run 52 seconds ago with 0 failures [20:42:01] RECOVERY - puppet last run on cp3015 is OK: OK: Puppet is currently enabled, last run 7 seconds ago with 0 failures [20:42:01] RECOVERY - puppet last run on ms-be1001 is OK: OK: Puppet is currently enabled, last run 3 seconds ago with 0 failures [20:42:06] RECOVERY - puppet last run on dbproxy1002 is OK: OK: Puppet is currently enabled, last run 1 seconds ago with 0 failures [20:42:06] RECOVERY - puppet last run on mw1067 is OK: OK: Puppet is currently enabled, last run 7 seconds ago with 0 failures [20:42:06] RECOVERY - puppet last run on mw1072 is OK: OK: Puppet is currently enabled, last run 6 seconds ago with 0 failures [20:42:06] RECOVERY - puppet last run on db1029 is OK: OK: Puppet is currently enabled, last run 12 seconds ago with 0 failures [20:42:06] RECOVERY - puppet last run on ms1001 is OK: OK: Puppet is currently enabled, last run 52 seconds ago with 0 failures [20:42:06] RECOVERY - puppet last run on mw1038 is OK: OK: Puppet is currently enabled, last run 27 seconds ago with 0 failures [20:42:16] RECOVERY - puppet last run on amssq59 is OK: OK: Puppet is currently enabled, last run 24 seconds ago with 0 failures [20:42:16] RECOVERY - puppet last run on mw1028 is OK: OK: Puppet is currently enabled, last run 46 seconds ago with 0 failures [20:42:16] RECOVERY - puppet last run on db1053 is OK: OK: Puppet is currently enabled, last run 20 seconds ago with 0 failures [20:42:16] RECOVERY - puppet last run on mc1004 is OK: OK: Puppet is currently enabled, last run 53 seconds ago with 0 failures [20:42:17] RECOVERY - puppet last run on search1020 is OK: OK: Puppet is currently enabled, last run 24 seconds ago with 0 failures [20:42:17] RECOVERY - puppet last run on db1065 is OK: OK: Puppet is currently enabled, last run 11 seconds ago with 0 failures [20:42:17] RECOVERY - puppet last run on mw1005 is OK: OK: Puppet is currently enabled, last run 28 seconds ago with 0 failures [20:42:26] RECOVERY - puppet last run on carbon is OK: OK: Puppet is currently enabled, last run 10 seconds ago with 0 failures [20:42:26] RECOVERY - puppet last run on mw1048 is OK: OK: Puppet is currently enabled, last run 58 seconds ago with 0 failures [20:42:26] RECOVERY - puppet last run on mw1132 is OK: OK: Puppet is currently enabled, last run 90 seconds ago with 0 failures [20:42:26] RECOVERY - puppet last run on wtp1024 is OK: OK: Puppet is currently enabled, last run 162 seconds ago with 0 failures [20:42:26] RECOVERY - puppet last run on mw1059 is OK: OK: Puppet is currently enabled, last run 9 seconds ago with 0 failures [20:42:26] RECOVERY - puppet last run on mw1080 is OK: OK: Puppet is currently enabled, last run 52 seconds ago with 0 failures [20:42:26] RECOVERY - puppet last run on mw1134 is OK: OK: Puppet is currently enabled, last run 7 seconds ago with 0 failures [20:42:27] RECOVERY - puppet last run on magnesium is OK: OK: Puppet is currently enabled, last run 16 seconds ago with 0 failures [20:42:27] RECOVERY - puppet last run on cp3006 is OK: OK: Puppet is currently enabled, last run 58 seconds ago with 0 failures [20:42:28] RECOVERY - puppet last run on neptunium is OK: OK: Puppet is currently enabled, last run 6 seconds ago with 0 failures [20:42:28] RECOVERY - puppet last run on ms-be2002 is OK: OK: Puppet is currently enabled, last run 60 seconds ago with 0 failures [20:42:36] RECOVERY - puppet last run on ms-fe1003 is OK: OK: Puppet is currently enabled, last run 50 seconds ago with 0 failures [20:42:36] RECOVERY - puppet last run on wtp1009 is OK: OK: Puppet is currently enabled, last run 34 seconds ago with 0 failures [20:42:36] RECOVERY - puppet last run on analytics1017 is OK: OK: Puppet is currently enabled, last run 11 seconds ago with 0 failures [20:42:36] RECOVERY - puppet last run on caesium is OK: OK: Puppet is currently enabled, last run 24 seconds ago with 0 failures [20:42:36] RECOVERY - puppet last run on mw1063 is OK: OK: Puppet is currently enabled, last run 3 seconds ago with 0 failures [20:42:37] RECOVERY - puppet last run on mw1106 is OK: OK: Puppet is currently enabled, last run 8 seconds ago with 0 failures [20:42:37] RECOVERY - puppet last run on logstash1001 is OK: OK: Puppet is currently enabled, last run 28 seconds ago with 0 failures [20:42:37] RECOVERY - puppet last run on mw1045 is OK: OK: Puppet is currently enabled, last run 10 seconds ago with 0 failures [20:42:37] RECOVERY - puppet last run on mw1041 is OK: OK: Puppet is currently enabled, last run 20 seconds ago with 0 failures [20:42:56] RECOVERY - puppet last run on db2035 is OK: OK: Puppet is currently enabled, last run 126 seconds ago with 0 failures [20:42:56] RECOVERY - puppet last run on mw1062 is OK: OK: Puppet is currently enabled, last run 131 seconds ago with 0 failures [20:42:56] RECOVERY - puppet last run on mw1089 is OK: OK: Puppet is currently enabled, last run 69 seconds ago with 0 failures [20:42:57] RECOVERY - puppet last run on cp1070 is OK: OK: Puppet is currently enabled, last run 53 seconds ago with 0 failures [20:42:57] RECOVERY - puppet last run on analytics1033 is OK: OK: Puppet is currently enabled, last run 20 seconds ago with 0 failures [20:42:57] RECOVERY - puppet last run on wtp1019 is OK: OK: Puppet is currently enabled, last run 108 seconds ago with 0 failures [20:42:57] RECOVERY - puppet last run on mw1012 is OK: OK: Puppet is currently enabled, last run 17 seconds ago with 0 failures [20:42:57] RECOVERY - puppet last run on mc1016 is OK: OK: Puppet is currently enabled, last run 20 seconds ago with 0 failures [20:42:58] RECOVERY - puppet last run on ms-be1004 is OK: OK: Puppet is currently enabled, last run 20 seconds ago with 0 failures [20:42:58] RECOVERY - puppet last run on db73 is OK: OK: Puppet is currently enabled, last run 47 seconds ago with 0 failures [20:42:59] RECOVERY - puppet last run on amssq44 is OK: OK: Puppet is currently enabled, last run 46 seconds ago with 0 failures [20:42:59] RECOVERY - puppet last run on cp3020 is OK: OK: Puppet is currently enabled, last run 8 seconds ago with 0 failures [20:43:00] RECOVERY - puppet last run on db1017 is OK: OK: Puppet is currently enabled, last run 29 seconds ago with 0 failures [20:43:06] RECOVERY - puppet last run on mw1031 is OK: OK: Puppet is currently enabled, last run 51 seconds ago with 0 failures [20:43:06] RECOVERY - puppet last run on cp3008 is OK: OK: Puppet is currently enabled, last run 37 seconds ago with 0 failures [20:43:07] RECOVERY - puppet last run on mw1178 is OK: OK: Puppet is currently enabled, last run 109 seconds ago with 0 failures [20:43:07] RECOVERY - puppet last run on tarin is OK: OK: Puppet is currently enabled, last run 190 seconds ago with 0 failures [20:43:07] RECOVERY - puppet last run on netmon1001 is OK: OK: Puppet is currently enabled, last run 16 seconds ago with 0 failures [20:43:26] RECOVERY - puppet last run on es1001 is OK: OK: Puppet is currently enabled, last run 44 seconds ago with 0 failures [20:43:29] RECOVERY - puppet last run on amssq43 is OK: OK: Puppet is currently enabled, last run 63 seconds ago with 0 failures [20:43:30] RECOVERY - puppet last run on es1008 is OK: OK: Puppet is currently enabled, last run 31 seconds ago with 0 failures [20:43:30] RECOVERY - puppet last run on platinum is OK: OK: Puppet is currently enabled, last run 34 seconds ago with 0 failures [20:43:30] RECOVERY - puppet last run on mw1200 is OK: OK: Puppet is currently enabled, last run 65 seconds ago with 0 failures [20:43:30] RECOVERY - puppet last run on cp1049 is OK: OK: Puppet is currently enabled, last run 36 seconds ago with 0 failures [20:43:30] RECOVERY - puppet last run on mw1187 is OK: OK: Puppet is currently enabled, last run 1 seconds ago with 0 failures [20:43:30] RECOVERY - puppet last run on elastic1001 is OK: OK: Puppet is currently enabled, last run 39 seconds ago with 0 failures [20:43:31] RECOVERY - puppet last run on analytics1040 is OK: OK: Puppet is currently enabled, last run 35 seconds ago with 0 failures [20:43:31] RECOVERY - puppet last run on mw1141 is OK: OK: Puppet is currently enabled, last run 33 seconds ago with 0 failures [20:43:32] RECOVERY - puppet last run on analytics1025 is OK: OK: Puppet is currently enabled, last run 26 seconds ago with 0 failures [20:43:32] RECOVERY - puppet last run on amslvs2 is OK: OK: Puppet is currently enabled, last run 38 seconds ago with 0 failures [20:43:33] RECOVERY - puppet last run on amssq49 is OK: OK: Puppet is currently enabled, last run 36 seconds ago with 0 failures [20:43:33] RECOVERY - puppet last run on lvs4002 is OK: OK: Puppet is currently enabled, last run 55 seconds ago with 0 failures [20:43:34] RECOVERY - puppet last run on amssq54 is OK: OK: Puppet is currently enabled, last run 52 seconds ago with 0 failures [20:43:36] RECOVERY - puppet last run on mc1006 is OK: OK: Puppet is currently enabled, last run 9 seconds ago with 0 failures [20:43:36] RECOVERY - puppet last run on db2009 is OK: OK: Puppet is currently enabled, last run 31 seconds ago with 0 failures [20:43:46] RECOVERY - puppet last run on search1016 is OK: OK: Puppet is currently enabled, last run 25 seconds ago with 0 failures [20:43:46] RECOVERY - puppet last run on wtp1020 is OK: OK: Puppet is currently enabled, last run 20 seconds ago with 0 failures [20:43:56] RECOVERY - puppet last run on mw1174 is OK: OK: Puppet is currently enabled, last run 33 seconds ago with 0 failures [20:43:59] RECOVERY - puppet last run on mw1007 is OK: OK: Puppet is currently enabled, last run 57 seconds ago with 0 failures [20:43:59] RECOVERY - puppet last run on mw1140 is OK: OK: Puppet is currently enabled, last run 57 seconds ago with 0 failures [20:44:00] RECOVERY - puppet last run on ms-be1013 is OK: OK: Puppet is currently enabled, last run 60 seconds ago with 0 failures [20:44:00] RECOVERY - puppet last run on ssl1002 is OK: OK: Puppet is currently enabled, last run 15 seconds ago with 0 failures [20:44:00] RECOVERY - puppet last run on mw1026 is OK: OK: Puppet is currently enabled, last run 9 seconds ago with 0 failures [20:44:00] RECOVERY - puppet last run on db2019 is OK: OK: Puppet is currently enabled, last run 9 seconds ago with 0 failures [20:44:00] RECOVERY - puppet last run on searchidx1001 is OK: OK: Puppet is currently enabled, last run 52 seconds ago with 0 failures [20:44:01] RECOVERY - puppet last run on cp1055 is OK: OK: Puppet is currently enabled, last run 44 seconds ago with 0 failures [20:44:01] RECOVERY - puppet last run on snapshot1003 is OK: OK: Puppet is currently enabled, last run 15 seconds ago with 0 failures [20:44:02] RECOVERY - puppet last run on mw1006 is OK: OK: Puppet is currently enabled, last run 56 seconds ago with 0 failures [20:44:02] RECOVERY - puppet last run on analytics1020 is OK: OK: Puppet is currently enabled, last run 56 seconds ago with 0 failures [20:44:03] RECOVERY - puppet last run on rbf1002 is OK: OK: Puppet is currently enabled, last run 34 seconds ago with 0 failures [20:44:03] RECOVERY - puppet last run on db2034 is OK: OK: Puppet is currently enabled, last run 14 seconds ago with 0 failures [20:44:04] RECOVERY - puppet last run on ms-be1006 is OK: OK: Puppet is currently enabled, last run 47 seconds ago with 0 failures [20:44:04] RECOVERY - puppet last run on db2039 is OK: OK: Puppet is currently enabled, last run 49 seconds ago with 0 failures [20:44:05] RECOVERY - puppet last run on db1073 is OK: OK: Puppet is currently enabled, last run 51 seconds ago with 0 failures [20:44:05] RECOVERY - puppet last run on elastic1007 is OK: OK: Puppet is currently enabled, last run 45 seconds ago with 0 failures [20:44:06] RECOVERY - puppet last run on db2005 is OK: OK: Puppet is currently enabled, last run 48 seconds ago with 0 failures [20:44:06] RECOVERY - puppet last run on mw1197 is OK: OK: Puppet is currently enabled, last run 58 seconds ago with 0 failures [20:44:07] RECOVERY - puppet last run on wtp1006 is OK: OK: Puppet is currently enabled, last run 57 seconds ago with 0 failures [20:44:07] RECOVERY - puppet last run on labstore1003 is OK: OK: Puppet is currently enabled, last run 2 seconds ago with 0 failures [20:44:08] RECOVERY - puppet last run on mw1145 is OK: OK: Puppet is currently enabled, last run 55 seconds ago with 0 failures [20:44:08] RECOVERY - puppet last run on helium is OK: OK: Puppet is currently enabled, last run 41 seconds ago with 0 failures [20:44:09] RECOVERY - puppet last run on cp4006 is OK: OK: Puppet is currently enabled, last run 45 seconds ago with 0 failures [20:44:16] RECOVERY - puppet last run on analytics1035 is OK: OK: Puppet is currently enabled, last run 17 seconds ago with 0 failures [20:44:17] RECOVERY - puppet last run on mc1003 is OK: OK: Puppet is currently enabled, last run 15 seconds ago with 0 failures [20:44:17] RECOVERY - puppet last run on amssq61 is OK: OK: Puppet is currently enabled, last run 51 seconds ago with 0 failures [20:44:17] RECOVERY - puppet last run on mc1002 is OK: OK: Puppet is currently enabled, last run 29 seconds ago with 0 failures [20:44:17] RECOVERY - puppet last run on lead is OK: OK: Puppet is currently enabled, last run 4 seconds ago with 0 failures [20:44:26] RECOVERY - puppet last run on analytics1041 is OK: OK: Puppet is currently enabled, last run 49 seconds ago with 0 failures [20:44:27] RECOVERY - puppet last run on elastic1004 is OK: OK: Puppet is currently enabled, last run 57 seconds ago with 0 failures [20:44:27] RECOVERY - puppet last run on db74 is OK: OK: Puppet is currently enabled, last run 44 seconds ago with 0 failures [20:44:27] RECOVERY - puppet last run on cp1047 is OK: OK: Puppet is currently enabled, last run 38 seconds ago with 0 failures [20:44:27] RECOVERY - puppet last run on cerium is OK: OK: Puppet is currently enabled, last run 52 seconds ago with 0 failures [20:44:27] RECOVERY - puppet last run on sca1002 is OK: OK: Puppet is currently enabled, last run 25 seconds ago with 0 failures [20:44:27] RECOVERY - puppet last run on ms1004 is OK: OK: Puppet is currently enabled, last run 15 seconds ago with 0 failures [20:44:27] RECOVERY - puppet last run on mw1082 is OK: OK: Puppet is currently enabled, last run 37 seconds ago with 0 failures [20:44:36] RECOVERY - puppet last run on gold is OK: OK: Puppet is currently enabled, last run 61 seconds ago with 0 failures [20:44:36] RECOVERY - puppet last run on mw1153 is OK: OK: Puppet is currently enabled, last run 12 seconds ago with 0 failures [20:44:36] RECOVERY - puppet last run on potassium is OK: OK: Puppet is currently enabled, last run 60 seconds ago with 0 failures [20:44:37] RECOVERY - puppet last run on db1031 is OK: OK: Puppet is currently enabled, last run 51 seconds ago with 0 failures [20:44:37] RECOVERY - puppet last run on cp1039 is OK: OK: Puppet is currently enabled, last run 38 seconds ago with 0 failures [20:44:37] RECOVERY - puppet last run on ms-fe1001 is OK: OK: Puppet is currently enabled, last run 14 seconds ago with 0 failures [20:44:38] RECOVERY - puppet last run on mw1217 is OK: OK: Puppet is currently enabled, last run 14 seconds ago with 0 failures [20:44:38] RECOVERY - puppet last run on wtp1016 is OK: OK: Puppet is currently enabled, last run 54 seconds ago with 0 failures [20:44:38] RECOVERY - puppet last run on mw1173 is OK: OK: Puppet is currently enabled, last run 7 seconds ago with 0 failures [20:44:38] RECOVERY - puppet last run on amssq32 is OK: OK: Puppet is currently enabled, last run 30 seconds ago with 0 failures [20:44:46] RECOVERY - puppet last run on lvs1005 is OK: OK: Puppet is currently enabled, last run 30 seconds ago with 0 failures [20:44:47] RECOVERY - puppet last run on mw1160 is OK: OK: Puppet is currently enabled, last run 46 seconds ago with 0 failures [20:44:47] RECOVERY - puppet last run on db1022 is OK: OK: Puppet is currently enabled, last run 18 seconds ago with 0 failures [20:44:56] RECOVERY - puppet last run on mw1189 is OK: OK: Puppet is currently enabled, last run 16 seconds ago with 0 failures [20:44:56] RECOVERY - puppet last run on mw1060 is OK: OK: Puppet is currently enabled, last run 22 seconds ago with 0 failures [20:44:56] RECOVERY - puppet last run on mw1205 is OK: OK: Puppet is currently enabled, last run 33 seconds ago with 0 failures [20:44:56] RECOVERY - puppet last run on mw1009 is OK: OK: Puppet is currently enabled, last run 34 seconds ago with 0 failures [20:44:57] RECOVERY - puppet last run on ms-be1003 is OK: OK: Puppet is currently enabled, last run 42 seconds ago with 0 failures [20:44:57] RECOVERY - puppet last run on db1066 is OK: OK: Puppet is currently enabled, last run 34 seconds ago with 0 failures [20:44:57] RECOVERY - puppet last run on mw1069 is OK: OK: Puppet is currently enabled, last run 10 seconds ago with 0 failures [20:44:57] RECOVERY - puppet last run on mw1046 is OK: OK: Puppet is currently enabled, last run 30 seconds ago with 0 failures [20:44:58] RECOVERY - puppet last run on search1010 is OK: OK: Puppet is currently enabled, last run 39 seconds ago with 0 failures [20:44:58] RECOVERY - puppet last run on iron is OK: OK: Puppet is currently enabled, last run 9 seconds ago with 0 failures [20:44:59] RECOVERY - puppet last run on db1050 is OK: OK: Puppet is currently enabled, last run 21 seconds ago with 0 failures [20:44:59] RECOVERY - puppet last run on ms-be2004 is OK: OK: Puppet is currently enabled, last run 6 seconds ago with 0 failures [20:45:06] RECOVERY - puppet last run on amssq53 is OK: OK: Puppet is currently enabled, last run 40 seconds ago with 0 failures [20:45:06] RECOVERY - puppet last run on xenon is OK: OK: Puppet is currently enabled, last run 44 seconds ago with 0 failures [20:45:06] RECOVERY - puppet last run on mw1120 is OK: OK: Puppet is currently enabled, last run 42 seconds ago with 0 failures [20:45:16] RECOVERY - puppet last run on lvs3001 is OK: OK: Puppet is currently enabled, last run 58 seconds ago with 0 failures [20:45:16] RECOVERY - puppet last run on mw1003 is OK: OK: Puppet is currently enabled, last run 22 seconds ago with 0 failures [20:45:16] RECOVERY - puppet last run on cp3003 is OK: OK: Puppet is currently enabled, last run 18 seconds ago with 0 failures [20:45:16] RECOVERY - puppet last run on ms-fe1004 is OK: OK: Puppet is currently enabled, last run 5 seconds ago with 0 failures [20:45:17] RECOVERY - puppet last run on mw1117 is OK: OK: Puppet is currently enabled, last run 16 seconds ago with 0 failures [20:45:17] RECOVERY - puppet last run on mw1088 is OK: OK: Puppet is currently enabled, last run 51 seconds ago with 0 failures [20:45:17] RECOVERY - puppet last run on elastic1008 is OK: OK: Puppet is currently enabled, last run 55 seconds ago with 0 failures [20:45:17] RECOVERY - puppet last run on elastic1012 is OK: OK: Puppet is currently enabled, last run 52 seconds ago with 0 failures [20:45:26] RECOVERY - puppet last run on db1059 is OK: OK: Puppet is currently enabled, last run 17 seconds ago with 0 failures [20:45:28] RECOVERY - puppet last run on mw1176 is OK: OK: Puppet is currently enabled, last run 59 seconds ago with 0 failures [20:45:38] RECOVERY - puppet last run on analytics1030 is OK: OK: Puppet is currently enabled, last run 30 seconds ago with 0 failures [20:45:38] RECOVERY - puppet last run on db1018 is OK: OK: Puppet is currently enabled, last run 22 seconds ago with 0 failures [20:45:38] RECOVERY - puppet last run on db2002 is OK: OK: Puppet is currently enabled, last run 41 seconds ago with 0 failures [20:45:38] RECOVERY - puppet last run on mw1099 is OK: OK: Puppet is currently enabled, last run 41 seconds ago with 0 failures [20:45:38] RECOVERY - puppet last run on search1018 is OK: OK: Puppet is currently enabled, last run 20 seconds ago with 0 failures [20:45:38] RECOVERY - puppet last run on lvs2004 is OK: OK: Puppet is currently enabled, last run 6 seconds ago with 0 failures [20:45:38] RECOVERY - puppet last run on search1001 is OK: OK: Puppet is currently enabled, last run 7 seconds ago with 0 failures [20:45:39] RECOVERY - puppet last run on db2018 is OK: OK: Puppet is currently enabled, last run 8 seconds ago with 0 failures [20:45:39] RECOVERY - puppet last run on cp3016 is OK: OK: Puppet is currently enabled, last run 30 seconds ago with 0 failures [20:45:40] RECOVERY - puppet last run on cp1061 is OK: OK: Puppet is currently enabled, last run 30 seconds ago with 0 failures [20:45:46] RECOVERY - puppet last run on lvs1002 is OK: OK: Puppet is currently enabled, last run 48 seconds ago with 0 failures [20:45:46] RECOVERY - puppet last run on mw1025 is OK: OK: Puppet is currently enabled, last run 21 seconds ago with 0 failures [20:45:46] RECOVERY - puppet last run on cp1056 is OK: OK: Puppet is currently enabled, last run 14 seconds ago with 0 failures [20:45:56] RECOVERY - puppet last run on mw1150 is OK: OK: Puppet is currently enabled, last run 50 seconds ago with 0 failures [20:45:56] RECOVERY - puppet last run on mw1065 is OK: OK: Puppet is currently enabled, last run 8 seconds ago with 0 failures [20:45:56] RECOVERY - puppet last run on mw1164 is OK: OK: Puppet is currently enabled, last run 56 seconds ago with 0 failures [20:45:56] RECOVERY - puppet last run on mw1008 is OK: OK: Puppet is currently enabled, last run 50 seconds ago with 0 failures [20:45:57] RECOVERY - puppet last run on logstash1002 is OK: OK: Puppet is currently enabled, last run 12 seconds ago with 0 failures [20:45:57] RECOVERY - puppet last run on db1015 is OK: OK: Puppet is currently enabled, last run 51 seconds ago with 0 failures [20:45:57] RECOVERY - puppet last run on mw1068 is OK: OK: Puppet is currently enabled, last run 44 seconds ago with 0 failures [20:45:57] RECOVERY - puppet last run on mw1170 is OK: OK: Puppet is currently enabled, last run 9 seconds ago with 0 failures [20:45:58] RECOVERY - puppet last run on mw1100 is OK: OK: Puppet is currently enabled, last run 56 seconds ago with 0 failures [20:45:58] RECOVERY - puppet last run on mw1123 is OK: OK: Puppet is currently enabled, last run 39 seconds ago with 0 failures [20:45:59] RECOVERY - puppet last run on db1046 is OK: OK: Puppet is currently enabled, last run 50 seconds ago with 0 failures [20:46:06] RECOVERY - puppet last run on db1040 is OK: OK: Puppet is currently enabled, last run 60 seconds ago with 0 failures [20:46:18] RECOVERY - puppet last run on cp4003 is OK: OK: Puppet is currently enabled, last run 2 seconds ago with 0 failures [20:46:26] RECOVERY - puppet last run on amssq35 is OK: OK: Puppet is currently enabled, last run 52 seconds ago with 0 failures [20:46:27] RECOVERY - puppet last run on mw1042 is OK: OK: Puppet is currently enabled, last run 35 seconds ago with 0 failures [20:46:27] RECOVERY - puppet last run on mw1118 is OK: OK: Puppet is currently enabled, last run 15 seconds ago with 0 failures [20:46:27] RECOVERY - puppet last run on db1002 is OK: OK: Puppet is currently enabled, last run 53 seconds ago with 0 failures [20:46:27] RECOVERY - puppet last run on mw1061 is OK: OK: Puppet is currently enabled, last run 50 seconds ago with 0 failures [20:46:27] RECOVERY - puppet last run on db2036 is OK: OK: Puppet is currently enabled, last run 11 seconds ago with 0 failures [20:46:27] RECOVERY - puppet last run on cp4008 is OK: OK: Puppet is currently enabled, last run 3 seconds ago with 0 failures [20:46:36] RECOVERY - puppet last run on mw1119 is OK: OK: Puppet is currently enabled, last run 26 seconds ago with 0 failures [20:46:36] RECOVERY - puppet last run on mw1092 is OK: OK: Puppet is currently enabled, last run 27 seconds ago with 0 failures [20:46:37] RECOVERY - puppet last run on mw1114 is OK: OK: Puppet is currently enabled, last run 8 seconds ago with 0 failures [20:46:37] RECOVERY - puppet last run on db1021 is OK: OK: Puppet is currently enabled, last run 30 seconds ago with 0 failures [20:46:46] RECOVERY - puppet last run on mw1144 is OK: OK: Puppet is currently enabled, last run 31 seconds ago with 0 failures [20:46:46] RECOVERY - puppet last run on db1067 is OK: OK: Puppet is currently enabled, last run 39 seconds ago with 0 failures [20:46:56] RECOVERY - puppet last run on mw1052 is OK: OK: Puppet is currently enabled, last run 60 seconds ago with 0 failures [20:46:57] RECOVERY - puppet last run on mw1054 is OK: OK: Puppet is currently enabled, last run 31 seconds ago with 0 failures [20:46:57] RECOVERY - puppet last run on search1007 is OK: OK: Puppet is currently enabled, last run 38 seconds ago with 0 failures [20:46:57] RECOVERY - puppet last run on mw1166 is OK: OK: Puppet is currently enabled, last run 55 seconds ago with 0 failures [20:46:57] RECOVERY - puppet last run on lvs2001 is OK: OK: Puppet is currently enabled, last run 33 seconds ago with 0 failures [20:46:57] RECOVERY - puppet last run on silver is OK: OK: Puppet is currently enabled, last run 3 seconds ago with 0 failures [20:47:07] RECOVERY - puppet last run on labcontrol2001 is OK: OK: Puppet is currently enabled, last run 9 seconds ago with 0 failures [20:47:08] RECOVERY - puppet last run on cp3014 is OK: OK: Puppet is currently enabled, last run 34 seconds ago with 0 failures [20:47:16] RECOVERY - puppet last run on db1042 is OK: OK: Puppet is currently enabled, last run 36 seconds ago with 0 failures [20:47:18] RECOVERY - puppet last run on dbproxy1001 is OK: OK: Puppet is currently enabled, last run 45 seconds ago with 0 failures [20:47:26] RECOVERY - puppet last run on mw1175 is OK: OK: Puppet is currently enabled, last run 42 seconds ago with 0 failures [20:47:27] RECOVERY - puppet last run on mw1213 is OK: OK: Puppet is currently enabled, last run 27 seconds ago with 0 failures [20:47:36] RECOVERY - puppet last run on amssq47 is OK: OK: Puppet is currently enabled, last run 34 seconds ago with 0 failures [20:47:36] RECOVERY - puppet last run on mw1129 is OK: OK: Puppet is currently enabled, last run 53 seconds ago with 0 failures [20:47:37] RECOVERY - puppet last run on amssq55 is OK: OK: Puppet is currently enabled, last run 37 seconds ago with 0 failures [20:47:37] RECOVERY - puppet last run on db1034 is OK: OK: Puppet is currently enabled, last run 53 seconds ago with 0 failures [20:47:57] RECOVERY - puppet last run on ruthenium is OK: OK: Puppet is currently enabled, last run 26 seconds ago with 0 failures [20:47:57] RECOVERY - puppet last run on ms-be2011 is OK: OK: Puppet is currently enabled, last run 42 seconds ago with 0 failures [20:48:07] RECOVERY - puppet last run on amssq60 is OK: OK: Puppet is currently enabled, last run 48 seconds ago with 0 failures [20:51:07] RECOVERY - puppet last run on titanium is OK: OK: Puppet is currently enabled, last run 1 seconds ago with 0 failures [20:51:07] RECOVERY - puppet last run on analytics1014 is OK: OK: Puppet is currently enabled, last run 13 seconds ago with 0 failures [20:51:16] RECOVERY - puppet last run on mw1116 is OK: OK: Puppet is currently enabled, last run 1 seconds ago with 0 failures [20:51:16] RECOVERY - puppet last run on cp1060 is OK: OK: Puppet is currently enabled, last run 7 seconds ago with 0 failures [20:51:26] RECOVERY - puppet last run on wtp1011 is OK: OK: Puppet is currently enabled, last run 11 seconds ago with 0 failures [20:51:36] RECOVERY - puppet last run on mw1171 is OK: OK: Puppet is currently enabled, last run 30 seconds ago with 0 failures [20:51:42] RECOVERY - puppet last run on ssl1008 is OK: OK: Puppet is currently enabled, last run 15 seconds ago with 0 failures [20:51:56] RECOVERY - puppet last run on rcs1002 is OK: OK: Puppet is currently enabled, last run 6 seconds ago with 0 failures [20:51:56] RECOVERY - puppet last run on rcs1001 is OK: OK: Puppet is currently enabled, last run 21 seconds ago with 0 failures [20:51:57] RECOVERY - puppet last run on mw1148 is OK: OK: Puppet is currently enabled, last run 16 seconds ago with 0 failures [20:52:07] RECOVERY - puppet last run on amssq38 is OK: OK: Puppet is currently enabled, last run 28 seconds ago with 0 failures [20:52:09] RECOVERY - puppet last run on protactinium is OK: OK: Puppet is currently enabled, last run 34 seconds ago with 0 failures [20:52:19] RECOVERY - puppet last run on nitrogen is OK: OK: Puppet is currently enabled, last run 7 seconds ago with 0 failures [20:52:26] RECOVERY - puppet last run on db1063 is OK: OK: Puppet is currently enabled, last run 5 seconds ago with 0 failures [20:52:29] RECOVERY - puppet last run on mc1007 is OK: OK: Puppet is currently enabled, last run 38 seconds ago with 0 failures [20:52:29] RECOVERY - puppet last run on mw1186 is OK: OK: Puppet is currently enabled, last run 15 seconds ago with 0 failures [20:52:29] RECOVERY - puppet last run on pc1003 is OK: OK: Puppet is currently enabled, last run 38 seconds ago with 0 failures [20:52:29] RECOVERY - puppet last run on amslvs3 is OK: OK: Puppet is currently enabled, last run 42 seconds ago with 0 failures [20:52:29] RECOVERY - puppet last run on rdb1003 is OK: OK: Puppet is currently enabled, last run 27 seconds ago with 0 failures [20:52:30] RECOVERY - puppet last run on elastic1002 is OK: OK: Puppet is currently enabled, last run 58 seconds ago with 0 failures [20:52:36] RECOVERY - puppet last run on db1044 is OK: OK: Puppet is currently enabled, last run 25 seconds ago with 0 failures [20:52:36] RECOVERY - puppet last run on db1030 is OK: OK: Puppet is currently enabled, last run 28 seconds ago with 0 failures [20:52:36] RECOVERY - puppet last run on wtp1003 is OK: OK: Puppet is currently enabled, last run 19 seconds ago with 0 failures [20:52:36] RECOVERY - puppet last run on elastic1005 is OK: OK: Puppet is currently enabled, last run 53 seconds ago with 0 failures [20:52:37] RECOVERY - puppet last run on ocg1003 is OK: OK: Puppet is currently enabled, last run 31 seconds ago with 0 failures [20:52:37] RECOVERY - puppet last run on lvs2003 is OK: OK: Puppet is currently enabled, last run 33 seconds ago with 0 failures [20:52:37] RECOVERY - puppet last run on analytics1011 is OK: OK: Puppet is currently enabled, last run 45 seconds ago with 0 failures [20:52:37] RECOVERY - puppet last run on cp3012 is OK: OK: Puppet is currently enabled, last run 52 seconds ago with 0 failures [20:52:38] RECOVERY - puppet last run on mw1033 is OK: OK: Puppet is currently enabled, last run 7 seconds ago with 0 failures [20:52:46] RECOVERY - puppet last run on ms-fe3001 is OK: OK: Puppet is currently enabled, last run 5 seconds ago with 0 failures [20:52:46] RECOVERY - puppet last run on mw1077 is OK: OK: Puppet is currently enabled, last run 12 seconds ago with 0 failures [20:52:46] RECOVERY - puppet last run on elastic1017 is OK: OK: Puppet is currently enabled, last run 15 seconds ago with 0 failures [20:52:46] RECOVERY - puppet last run on ssl1007 is OK: OK: Puppet is currently enabled, last run 44 seconds ago with 0 failures [20:52:56] RECOVERY - puppet last run on mw1108 is OK: OK: Puppet is currently enabled, last run 20 seconds ago with 0 failures [20:52:57] RECOVERY - puppet last run on mw1022 is OK: OK: Puppet is currently enabled, last run 27 seconds ago with 0 failures [20:52:57] RECOVERY - puppet last run on db1064 is OK: OK: Puppet is currently enabled, last run 41 seconds ago with 0 failures [20:53:06] RECOVERY - puppet last run on cp3009 is OK: OK: Puppet is currently enabled, last run 58 seconds ago with 0 failures [20:53:06] RECOVERY - puppet last run on mw1024 is OK: OK: Puppet is currently enabled, last run 52 seconds ago with 0 failures [20:53:16] RECOVERY - puppet last run on mw1043 is OK: OK: Puppet is currently enabled, last run 46 seconds ago with 0 failures [20:53:19] RECOVERY - puppet last run on wtp1007 is OK: OK: Puppet is currently enabled, last run 55 seconds ago with 0 failures [20:53:19] RECOVERY - puppet last run on mw1001 is OK: OK: Puppet is currently enabled, last run 24 seconds ago with 0 failures [20:53:27] RECOVERY - puppet last run on mw1122 is OK: OK: Puppet is currently enabled, last run 18 seconds ago with 0 failures [20:53:42] RECOVERY - puppet last run on sanger is OK: OK: Puppet is currently enabled, last run 46 seconds ago with 0 failures [20:53:42] RECOVERY - puppet last run on virt1000 is OK: OK: Puppet is currently enabled, last run 21 seconds ago with 0 failures [20:53:43] RECOVERY - puppet last run on mw1105 is OK: OK: Puppet is currently enabled, last run 36 seconds ago with 0 failures [20:53:43] RECOVERY - puppet last run on db1011 is OK: OK: Puppet is currently enabled, last run 22 seconds ago with 0 failures [20:53:58] RECOVERY - puppet last run on mw1185 is OK: OK: Puppet is currently enabled, last run 51 seconds ago with 0 failures [20:54:00] I hope everyone is enjoying the ride [20:54:08] RECOVERY - puppet last run on mw1201 is OK: OK: Puppet is currently enabled, last run 62 seconds ago with 0 failures [20:54:40] WHAT I CAN'T HEAR YOU [20:54:56] sjoerddebruin: /ignore icinga-wm ;) [20:55:07] * greg-g hasn't, but I don't begrudge those who do [20:55:16] I like him. [20:56:58] so verbose and redundant though [21:10:24] PROBLEM - Too many prints [21:10:27] :) [21:22:14] PROBLEM - puppet last run on ssl1003 is CRITICAL: CRITICAL: Puppet has 1 failures [21:22:45] PROBLEM - puppet last run on db2017 is CRITICAL: CRITICAL: Puppet has 1 failures [21:24:34] PROBLEM - puppet last run on amssq50 is CRITICAL: CRITICAL: Puppet has 1 failures [21:37:05] RECOVERY - puppet last run on db2017 is OK: OK: Puppet is currently enabled, last run 28 seconds ago with 0 failures [21:37:15] RECOVERY - puppet last run on ssl1003 is OK: OK: Puppet is currently enabled, last run 43 seconds ago with 0 failures [21:37:34] RECOVERY - puppet last run on amssq50 is OK: OK: Puppet is currently enabled, last run 29 seconds ago with 0 failures [22:00:13] PROBLEM - puppet last run on virt1008 is CRITICAL: CRITICAL: Puppet has 1 failures [22:00:42] PROBLEM - puppet last run on cp1045 is CRITICAL: CRITICAL: Puppet has 1 failures [22:01:02] PROBLEM - puppet last run on db1045 is CRITICAL: CRITICAL: Puppet has 1 failures [22:01:02] PROBLEM - puppet last run on ms-be1015 is CRITICAL: CRITICAL: Puppet has 1 failures [22:01:03] PROBLEM - puppet last run on cp1053 is CRITICAL: CRITICAL: Puppet has 1 failures [22:01:12] PROBLEM - puppet last run on mw1158 is CRITICAL: CRITICAL: Puppet has 1 failures [22:01:12] PROBLEM - puppet last run on linne is CRITICAL: CRITICAL: Puppet has 1 failures [22:01:12] PROBLEM - puppet last run on lvs1003 is CRITICAL: CRITICAL: Puppet has 1 failures [22:01:12] PROBLEM - puppet last run on mw1090 is CRITICAL: CRITICAL: Puppet has 1 failures [22:01:12] PROBLEM - puppet last run on lvs1004 is CRITICAL: CRITICAL: Puppet has 1 failures [22:01:22] PROBLEM - puppet last run on mw1021 is CRITICAL: CRITICAL: Puppet has 1 failures [22:01:22] PROBLEM - puppet last run on cp3005 is CRITICAL: CRITICAL: Puppet has 1 failures [22:01:22] PROBLEM - puppet last run on mw1071 is CRITICAL: CRITICAL: Puppet has 1 failures [22:01:22] PROBLEM - puppet last run on wtp1001 is CRITICAL: CRITICAL: Puppet has 1 failures [22:01:22] PROBLEM - puppet last run on lvs4004 is CRITICAL: CRITICAL: Puppet has 1 failures [22:01:23] PROBLEM - puppet last run on db1035 is CRITICAL: CRITICAL: Puppet has 1 failures [22:01:23] PROBLEM - puppet last run on dysprosium is CRITICAL: CRITICAL: Puppet has 1 failures [22:01:32] PROBLEM - puppet last run on mw1204 is CRITICAL: CRITICAL: Puppet has 1 failures [22:01:33] PROBLEM - puppet last run on mw1155 is CRITICAL: CRITICAL: Puppet has 1 failures [22:01:33] PROBLEM - puppet last run on osm-cp1001 is CRITICAL: CRITICAL: Puppet has 1 failures [22:01:33] PROBLEM - puppet last run on cp1068 is CRITICAL: CRITICAL: Puppet has 1 failures [22:01:33] PROBLEM - puppet last run on mw1154 is CRITICAL: CRITICAL: Puppet has 1 failures [22:01:42] PROBLEM - puppet last run on mw1113 is CRITICAL: CRITICAL: Puppet has 1 failures [22:01:43] PROBLEM - puppet last run on mw1207 is CRITICAL: CRITICAL: Puppet has 1 failures [22:01:43] PROBLEM - puppet last run on mw1037 is CRITICAL: CRITICAL: Puppet has 1 failures [22:01:43] PROBLEM - puppet last run on mw1131 is CRITICAL: CRITICAL: Puppet has 1 failures [22:01:44] PROBLEM - puppet last run on mw1110 is CRITICAL: CRITICAL: Puppet has 1 failures [22:01:52] PROBLEM - puppet last run on mw1104 is CRITICAL: CRITICAL: Puppet has 1 failures [22:01:52] PROBLEM - puppet last run on db1056 is CRITICAL: CRITICAL: Puppet has 1 failures [22:01:52] PROBLEM - puppet last run on sca1001 is CRITICAL: CRITICAL: Puppet has 1 failures [22:01:52] PROBLEM - puppet last run on ms-be1010 is CRITICAL: CRITICAL: Puppet has 1 failures [22:01:52] PROBLEM - puppet last run on analytics1031 is CRITICAL: CRITICAL: Puppet has 1 failures [22:01:53] PROBLEM - puppet last run on amssq37 is CRITICAL: CRITICAL: Puppet has 1 failures [22:01:53] PROBLEM - puppet last run on mw1128 is CRITICAL: CRITICAL: Puppet has 1 failures [22:02:02] PROBLEM - puppet last run on bast2001 is CRITICAL: CRITICAL: Puppet has 1 failures [22:02:02] PROBLEM - puppet last run on mw1073 is CRITICAL: CRITICAL: Puppet has 1 failures [22:02:03] PROBLEM - puppet last run on mw1135 is CRITICAL: CRITICAL: Puppet has 1 failures [22:02:03] PROBLEM - puppet last run on wtp1008 is CRITICAL: CRITICAL: Puppet has 1 failures [22:02:03] PROBLEM - puppet last run on bast1001 is CRITICAL: CRITICAL: Puppet has 1 failures [22:02:03] PROBLEM - puppet last run on palladium is CRITICAL: CRITICAL: Puppet has 1 failures [22:02:03] PROBLEM - puppet last run on cp1054 is CRITICAL: CRITICAL: Puppet has 1 failures [22:02:06] .... [22:02:12] PROBLEM - puppet last run on lvs2002 is CRITICAL: CRITICAL: Puppet has 1 failures [22:02:12] PROBLEM - puppet last run on mw1199 is CRITICAL: CRITICAL: Puppet has 1 failures [22:02:12] PROBLEM - puppet last run on es1004 is CRITICAL: CRITICAL: Puppet has 1 failures [22:02:12] PROBLEM - puppet last run on db2017 is CRITICAL: CRITICAL: Puppet has 1 failures [22:02:25] PROBLEM - puppet last run on cp4011 is CRITICAL: CRITICAL: Puppet has 1 failures [22:02:25] PROBLEM - puppet last run on cp1066 is CRITICAL: CRITICAL: Puppet has 1 failures [22:02:25] PROBLEM - puppet last run on analytics1012 is CRITICAL: CRITICAL: Puppet has 1 failures [22:02:25] PROBLEM - puppet last run on search1003 is CRITICAL: CRITICAL: Puppet has 1 failures [22:02:35] PROBLEM - puppet last run on mw1194 is CRITICAL: CRITICAL: Puppet has 1 failures [22:02:37] greg-g, hm, it's back :/ [22:02:37] PROBLEM - puppet last run on ssl1003 is CRITICAL: CRITICAL: Puppet has 1 failures [22:02:40] PROBLEM - puppet last run on cp4016 is CRITICAL: CRITICAL: Puppet has 1 failures [22:02:41] PROBLEM - puppet last run on cp3011 is CRITICAL: CRITICAL: Puppet has 1 failures [22:02:41] PROBLEM - puppet last run on cp3019 is CRITICAL: CRITICAL: Puppet has 1 failures [22:02:41] PROBLEM - puppet last run on ms-be1014 is CRITICAL: CRITICAL: Puppet has 1 failures [22:02:45] PROBLEM - puppet last run on tmh1001 is CRITICAL: CRITICAL: Puppet has 1 failures [22:02:45] PROBLEM - puppet last run on mw1047 is CRITICAL: CRITICAL: Puppet has 1 failures [22:02:46] PROBLEM - puppet last run on fenari is CRITICAL: CRITICAL: Puppet has 1 failures [22:02:46] PROBLEM - puppet last run on db1027 is CRITICAL: CRITICAL: Puppet has 1 failures [22:02:46] PROBLEM - puppet last run on lvs2005 is CRITICAL: CRITICAL: Puppet has 1 failures [22:02:46] PROBLEM - puppet last run on mw1103 is CRITICAL: CRITICAL: Puppet has 1 failures [22:02:46] PROBLEM - puppet last run on mw1137 is CRITICAL: CRITICAL: Puppet has 1 failures [22:02:55] PROBLEM - puppet last run on cp4010 is CRITICAL: CRITICAL: Puppet has 1 failures [22:02:55] PROBLEM - puppet last run on db1068 is CRITICAL: CRITICAL: Puppet has 1 failures [22:02:56] PROBLEM - puppet last run on amssq50 is CRITICAL: CRITICAL: Puppet has 1 failures [22:02:56] PROBLEM - puppet last run on labsdb1002 is CRITICAL: CRITICAL: Puppet has 1 failures [22:02:56] PROBLEM - puppet last run on mc1015 is CRITICAL: CRITICAL: Puppet has 1 failures [22:02:56] PROBLEM - puppet last run on mw1018 is CRITICAL: CRITICAL: Puppet has 1 failures [22:02:56] PROBLEM - puppet last run on mw1015 is CRITICAL: CRITICAL: Puppet has 1 failures [22:02:57] PROBLEM - puppet last run on virt1005 is CRITICAL: CRITICAL: Puppet has 1 failures [22:03:07] PROBLEM - puppet last run on mw1020 is CRITICAL: CRITICAL: Puppet has 1 failures [22:03:08] PROBLEM - puppet last run on mw1095 is CRITICAL: CRITICAL: Puppet has 1 failures [22:03:15] PROBLEM - puppet last run on cp4015 is CRITICAL: CRITICAL: Puppet has 1 failures [22:03:25] PROBLEM - puppet last run on mw1101 is CRITICAL: CRITICAL: Puppet has 1 failures [22:03:25] PROBLEM - puppet last run on cp1064 is CRITICAL: CRITICAL: Puppet has 1 failures [22:03:25] PROBLEM - puppet last run on cp3017 is CRITICAL: CRITICAL: Puppet has 1 failures [22:03:25] PROBLEM - puppet last run on elastic1010 is CRITICAL: CRITICAL: Puppet has 1 failures [22:03:25] PROBLEM - puppet last run on ms-be2009 is CRITICAL: CRITICAL: Puppet has 1 failures [22:03:25] PROBLEM - puppet last run on virt1009 is CRITICAL: CRITICAL: Puppet has 1 failures [22:03:25] PROBLEM - puppet last run on elastic1013 is CRITICAL: CRITICAL: Puppet has 1 failures [22:03:26] PROBLEM - puppet last run on mw1058 is CRITICAL: CRITICAL: Puppet has 1 failures [22:03:35] PROBLEM - puppet last run on mw1085 is CRITICAL: CRITICAL: Puppet has 1 failures [22:03:35] PROBLEM - puppet last run on mw1157 is CRITICAL: CRITICAL: Puppet has 1 failures [22:03:35] PROBLEM - puppet last run on hydrogen is CRITICAL: CRITICAL: Puppet has 1 failures [22:03:36] PROBLEM - puppet last run on cp3021 is CRITICAL: CRITICAL: Puppet has 1 failures [22:03:36] PROBLEM - puppet last run on amssq52 is CRITICAL: CRITICAL: Puppet has 1 failures [22:03:45] PROBLEM - puppet last run on mw1019 is CRITICAL: CRITICAL: Puppet has 1 failures [22:03:46] PROBLEM - puppet last run on db1009 is CRITICAL: CRITICAL: Puppet has 1 failures [22:03:46] PROBLEM - puppet last run on lvs3002 is CRITICAL: CRITICAL: Puppet has 1 failures [22:03:55] PROBLEM - puppet last run on amssq45 is CRITICAL: CRITICAL: Puppet has 1 failures [22:03:57] PROBLEM - puppet last run on elastic1016 is CRITICAL: CRITICAL: Puppet has 1 failures [22:03:58] PROBLEM - puppet last run on mw1179 is CRITICAL: CRITICAL: Puppet has 1 failures [22:03:58] PROBLEM - puppet last run on wtp1014 is CRITICAL: CRITICAL: Puppet has 1 failures [22:03:58] PROBLEM - puppet last run on search1008 is CRITICAL: CRITICAL: Puppet has 1 failures [22:04:05] PROBLEM - puppet last run on chromium is CRITICAL: CRITICAL: Puppet has 1 failures [22:04:06] PROBLEM - puppet last run on labsdb1005 is CRITICAL: CRITICAL: Puppet has 1 failures [22:04:06] PROBLEM - puppet last run on elastic1009 is CRITICAL: CRITICAL: Puppet has 1 failures [22:04:06] PROBLEM - puppet last run on db1049 is CRITICAL: CRITICAL: Puppet has 1 failures [22:04:06] PROBLEM - puppet last run on mw1075 is CRITICAL: CRITICAL: Puppet has 1 failures [22:04:08] PROBLEM - puppet last run on mw1136 is CRITICAL: CRITICAL: Puppet has 1 failures [22:04:08] PROBLEM - puppet last run on zinc is CRITICAL: CRITICAL: Puppet has 1 failures [22:04:08] PROBLEM - puppet last run on mw1017 is CRITICAL: CRITICAL: Puppet has 1 failures [22:04:08] PROBLEM - puppet last run on lvs1006 is CRITICAL: CRITICAL: Puppet has 1 failures [22:04:16] PROBLEM - puppet last run on lanthanum is CRITICAL: CRITICAL: Puppet has 1 failures [22:04:25] PROBLEM - puppet last run on lvs4001 is CRITICAL: CRITICAL: Puppet has 1 failures [22:04:25] PROBLEM - puppet last run on search1019 is CRITICAL: CRITICAL: Puppet has 1 failures [22:04:36] PROBLEM - puppet last run on ssl1004 is CRITICAL: CRITICAL: Puppet has 1 failures [22:04:45] PROBLEM - puppet last run on cp3007 is CRITICAL: CRITICAL: Puppet has 1 failures [22:04:55] PROBLEM - puppet last run on eeden is CRITICAL: CRITICAL: Puppet has 1 failures [22:04:56] PROBLEM - puppet last run on ms-be3004 is CRITICAL: CRITICAL: Puppet has 1 failures [22:04:56] PROBLEM - puppet last run on ms-be1005 is CRITICAL: CRITICAL: Puppet has 1 failures [22:05:05] PROBLEM - puppet last run on mw1094 is CRITICAL: CRITICAL: Puppet has 1 failures [22:05:47] PROBLEM - puppet last run on mw1070 is CRITICAL: CRITICAL: Puppet has 1 failures [22:06:45] PROBLEM - puppet last run on mw1102 is CRITICAL: CRITICAL: Puppet has 1 failures [22:07:42] Krenair: "great" [22:10:15] greg-g: anything in syslog ? [22:14:36] RECOVERY - puppet last run on mw1071 is OK: OK: Puppet is currently enabled, last run 13 seconds ago with 0 failures [22:14:56] RECOVERY - puppet last run on cp1045 is OK: OK: Puppet is currently enabled, last run 2 seconds ago with 0 failures [22:15:14] RECOVERY - puppet last run on sca1001 is OK: OK: Puppet is currently enabled, last run 0 seconds ago with 0 failures [22:15:26] RECOVERY - puppet last run on db1045 is OK: OK: Puppet is currently enabled, last run 29 seconds ago with 0 failures [22:15:27] RECOVERY - puppet last run on ms-be1015 is OK: OK: Puppet is currently enabled, last run 3 seconds ago with 0 failures [22:15:27] RECOVERY - puppet last run on wtp1008 is OK: OK: Puppet is currently enabled, last run 15 seconds ago with 0 failures [22:15:36] RECOVERY - puppet last run on cp1053 is OK: OK: Puppet is currently enabled, last run 31 seconds ago with 0 failures [22:15:41] RECOVERY - puppet last run on mw1021 is OK: OK: Puppet is currently enabled, last run 22 seconds ago with 0 failures [22:15:42] RECOVERY - puppet last run on mw1090 is OK: OK: Puppet is currently enabled, last run 46 seconds ago with 0 failures [22:15:42] RECOVERY - puppet last run on lvs1004 is OK: OK: Puppet is currently enabled, last run 18 seconds ago with 0 failures [22:15:42] RECOVERY - puppet last run on virt1008 is OK: OK: Puppet is currently enabled, last run 37 seconds ago with 0 failures [22:15:42] RECOVERY - puppet last run on linne is OK: OK: Puppet is currently enabled, last run 6 seconds ago with 0 failures [22:15:42] RECOVERY - puppet last run on db1035 is OK: OK: Puppet is currently enabled, last run 47 seconds ago with 0 failures [22:15:43] RECOVERY - puppet last run on dysprosium is OK: OK: Puppet is currently enabled, last run 25 seconds ago with 0 failures [22:15:46] RECOVERY - puppet last run on wtp1001 is OK: OK: Puppet is currently enabled, last run 56 seconds ago with 0 failures [22:15:46] RECOVERY - puppet last run on mw1204 is OK: OK: Puppet is currently enabled, last run 53 seconds ago with 0 failures [22:15:56] RECOVERY - puppet last run on mw1155 is OK: OK: Puppet is currently enabled, last run 19 seconds ago with 0 failures [22:16:06] RECOVERY - puppet last run on cp1068 is OK: OK: Puppet is currently enabled, last run 40 seconds ago with 0 failures [22:16:06] RECOVERY - puppet last run on osm-cp1001 is OK: OK: Puppet is currently enabled, last run 45 seconds ago with 0 failures [22:16:06] RECOVERY - puppet last run on mw1154 is OK: OK: Puppet is currently enabled, last run 20 seconds ago with 0 failures [22:16:06] RECOVERY - puppet last run on cp3011 is OK: OK: Puppet is currently enabled, last run 36 seconds ago with 0 failures [22:16:07] RECOVERY - puppet last run on mw1113 is OK: OK: Puppet is currently enabled, last run 8 seconds ago with 0 failures [22:16:07] RECOVERY - puppet last run on mw1110 is OK: OK: Puppet is currently enabled, last run 15 seconds ago with 0 failures [22:16:07] RECOVERY - puppet last run on mw1131 is OK: OK: Puppet is currently enabled, last run 25 seconds ago with 0 failures [22:16:07] RECOVERY - puppet last run on eeden is OK: OK: Puppet is currently enabled, last run 23 seconds ago with 0 failures [22:16:07] RECOVERY - puppet last run on cp3005 is OK: OK: Puppet is currently enabled, last run 8 seconds ago with 0 failures [22:16:08] RECOVERY - puppet last run on mw1104 is OK: OK: Puppet is currently enabled, last run 9 seconds ago with 0 failures [22:16:08] RECOVERY - puppet last run on db1056 is OK: OK: Puppet is currently enabled, last run 56 seconds ago with 0 failures [22:16:09] RECOVERY - puppet last run on db1027 is OK: OK: Puppet is currently enabled, last run 4 seconds ago with 0 failures [22:16:16] RECOVERY - puppet last run on chromium is OK: OK: Puppet is currently enabled, last run 44 seconds ago with 0 failures [22:16:16] RECOVERY - puppet last run on mc1015 is OK: OK: Puppet is currently enabled, last run 5 seconds ago with 0 failures [22:16:26] RECOVERY - puppet last run on labsdb1005 is OK: OK: Puppet is currently enabled, last run 29 seconds ago with 0 failures [22:16:26] RECOVERY - puppet last run on amssq37 is OK: OK: Puppet is currently enabled, last run 50 seconds ago with 0 failures [22:16:27] RECOVERY - puppet last run on mw1073 is OK: OK: Puppet is currently enabled, last run 0 seconds ago with 0 failures [22:16:27] RECOVERY - puppet last run on bast1001 is OK: OK: Puppet is currently enabled, last run 57 seconds ago with 0 failures [22:16:27] RECOVERY - puppet last run on mw1135 is OK: OK: Puppet is currently enabled, last run 30 seconds ago with 0 failures [22:16:39] RECOVERY - puppet last run on cp1054 is OK: OK: Puppet is currently enabled, last run 55 seconds ago with 0 failures [22:16:39] RECOVERY - puppet last run on mw1158 is OK: OK: Puppet is currently enabled, last run 59 seconds ago with 0 failures [22:16:39] RECOVERY - puppet last run on lvs1003 is OK: OK: Puppet is currently enabled, last run 27 seconds ago with 0 failures [22:16:39] RECOVERY - puppet last run on db2017 is OK: OK: Puppet is currently enabled, last run 16 seconds ago with 0 failures [22:16:40] RECOVERY - puppet last run on lvs4004 is OK: OK: Puppet is currently enabled, last run 41 seconds ago with 0 failures [22:16:46] RECOVERY - puppet last run on cp1066 is OK: OK: Puppet is currently enabled, last run 34 seconds ago with 0 failures [22:16:46] RECOVERY - puppet last run on analytics1012 is OK: OK: Puppet is currently enabled, last run 18 seconds ago with 0 failures [22:16:46] RECOVERY - puppet last run on mw1194 is OK: OK: Puppet is currently enabled, last run 27 seconds ago with 0 failures [22:16:56] RECOVERY - puppet last run on ssl1003 is OK: OK: Puppet is currently enabled, last run 25 seconds ago with 0 failures [22:16:56] RECOVERY - puppet last run on hydrogen is OK: OK: Puppet is currently enabled, last run 5 seconds ago with 0 failures [22:16:57] RECOVERY - puppet last run on ms-be1014 is OK: OK: Puppet is currently enabled, last run 5 seconds ago with 0 failures [22:16:57] RECOVERY - puppet last run on cp4016 is OK: OK: Puppet is currently enabled, last run 2 seconds ago with 0 failures [22:16:57] RECOVERY - puppet last run on mw1207 is OK: OK: Puppet is currently enabled, last run 56 seconds ago with 0 failures [22:16:57] RECOVERY - puppet last run on mw1037 is OK: OK: Puppet is currently enabled, last run 52 seconds ago with 0 failures [22:16:57] RECOVERY - puppet last run on tmh1001 is OK: OK: Puppet is currently enabled, last run 20 seconds ago with 0 failures [22:17:06] RECOVERY - puppet last run on mw1047 is OK: OK: Puppet is currently enabled, last run 5 seconds ago with 0 failures [22:17:08] RECOVERY - puppet last run on mw1179 is OK: OK: Puppet is currently enabled, last run 1 seconds ago with 0 failures [22:17:08] RECOVERY - puppet last run on fenari is OK: OK: Puppet is currently enabled, last run 41 seconds ago with 0 failures [22:17:16] RECOVERY - puppet last run on db1009 is OK: OK: Puppet is currently enabled, last run 6 seconds ago with 0 failures [22:17:16] RECOVERY - puppet last run on mw1103 is OK: OK: Puppet is currently enabled, last run 54 seconds ago with 0 failures [22:17:16] RECOVERY - puppet last run on ms-be1010 is OK: OK: Puppet is currently enabled, last run 33 seconds ago with 0 failures [22:17:16] RECOVERY - puppet last run on search1008 is OK: OK: Puppet is currently enabled, last run 9 seconds ago with 0 failures [22:17:16] RECOVERY - puppet last run on analytics1031 is OK: OK: Puppet is currently enabled, last run 44 seconds ago with 0 failures [22:17:17] RECOVERY - puppet last run on labsdb1002 is OK: OK: Puppet is currently enabled, last run 2 seconds ago with 0 failures [22:17:27] RECOVERY - puppet last run on mw1018 is OK: OK: Puppet is currently enabled, last run 58 seconds ago with 0 failures [22:17:27] RECOVERY - puppet last run on elastic1009 is OK: OK: Puppet is currently enabled, last run 17 seconds ago with 0 failures [22:17:27] RECOVERY - puppet last run on mw1128 is OK: OK: Puppet is currently enabled, last run 45 seconds ago with 0 failures [22:17:28] RECOVERY - puppet last run on amssq50 is OK: OK: Puppet is currently enabled, last run 20 seconds ago with 0 failures [22:17:28] RECOVERY - puppet last run on mw1075 is OK: OK: Puppet is currently enabled, last run 15 seconds ago with 0 failures [22:17:28] RECOVERY - puppet last run on virt1005 is OK: OK: Puppet is currently enabled, last run 10 seconds ago with 0 failures [22:17:36] RECOVERY - puppet last run on zinc is OK: OK: Puppet is currently enabled, last run 51 seconds ago with 0 failures [22:17:36] RECOVERY - puppet last run on bast2001 is OK: OK: Puppet is currently enabled, last run 45 seconds ago with 0 failures [22:17:37] RECOVERY - puppet last run on palladium is OK: OK: Puppet is currently enabled, last run 60 seconds ago with 0 failures [22:17:41] RECOVERY - puppet last run on lvs2002 is OK: OK: Puppet is currently enabled, last run 59 seconds ago with 0 failures [22:17:41] RECOVERY - puppet last run on es1004 is OK: OK: Puppet is currently enabled, last run 50 seconds ago with 0 failures [22:17:42] RECOVERY - puppet last run on mw1199 is OK: OK: Puppet is currently enabled, last run 58 seconds ago with 0 failures [22:17:42] RECOVERY - puppet last run on mw1101 is OK: OK: Puppet is currently enabled, last run 33 seconds ago with 0 failures [22:17:46] RECOVERY - puppet last run on elastic1010 is OK: OK: Puppet is currently enabled, last run 43 seconds ago with 0 failures [22:17:46] RECOVERY - puppet last run on search1019 is OK: OK: Puppet is currently enabled, last run 20 seconds ago with 0 failures [22:17:46] RECOVERY - puppet last run on cp4011 is OK: OK: Puppet is currently enabled, last run 26 seconds ago with 0 failures [22:17:46] RECOVERY - puppet last run on cp3017 is OK: OK: Puppet is currently enabled, last run 29 seconds ago with 0 failures [22:17:46] RECOVERY - puppet last run on elastic1013 is OK: OK: Puppet is currently enabled, last run 37 seconds ago with 0 failures [22:17:46] RECOVERY - puppet last run on mw1058 is OK: OK: Puppet is currently enabled, last run 7 seconds ago with 0 failures [22:17:46] RECOVERY - puppet last run on search1003 is OK: OK: Puppet is currently enabled, last run 61 seconds ago with 0 failures [22:17:56] RECOVERY - puppet last run on ms-be2009 is OK: OK: Puppet is currently enabled, last run 24 seconds ago with 0 failures [22:17:56] RECOVERY - puppet last run on mw1085 is OK: OK: Puppet is currently enabled, last run 7 seconds ago with 0 failures [22:17:57] RECOVERY - puppet last run on amssq52 is OK: OK: Puppet is currently enabled, last run 30 seconds ago with 0 failures [22:17:57] RECOVERY - puppet last run on cp3021 is OK: OK: Puppet is currently enabled, last run 48 seconds ago with 0 failures [22:17:57] RECOVERY - puppet last run on cp3019 is OK: OK: Puppet is currently enabled, last run 55 seconds ago with 0 failures [22:18:06] RECOVERY - puppet last run on mw1019 is OK: OK: Puppet is currently enabled, last run 13 seconds ago with 0 failures [22:18:06] RECOVERY - puppet last run on mw1102 is OK: OK: Puppet is currently enabled, last run 17 seconds ago with 0 failures [22:18:06] RECOVERY - puppet last run on mw1070 is OK: OK: Puppet is currently enabled, last run 20 seconds ago with 0 failures [22:18:07] RECOVERY - puppet last run on cp3007 is OK: OK: Puppet is currently enabled, last run 24 seconds ago with 0 failures [22:18:07] RECOVERY - puppet last run on mw1137 is OK: OK: Puppet is currently enabled, last run 43 seconds ago with 0 failures [22:18:09] RECOVERY - puppet last run on elastic1016 is OK: OK: Puppet is currently enabled, last run 7 seconds ago with 0 failures [22:18:26] RECOVERY - puppet last run on wtp1014 is OK: OK: Puppet is currently enabled, last run 1 seconds ago with 0 failures [22:18:26] RECOVERY - puppet last run on amssq45 is OK: OK: Puppet is currently enabled, last run 4 seconds ago with 0 failures [22:18:26] RECOVERY - puppet last run on lvs3002 is OK: OK: Puppet is currently enabled, last run 46 seconds ago with 0 failures [22:18:26] RECOVERY - puppet last run on db1068 is OK: OK: Puppet is currently enabled, last run 59 seconds ago with 0 failures [22:18:27] RECOVERY - puppet last run on lvs2005 is OK: OK: Puppet is currently enabled, last run 33 seconds ago with 0 failures [22:18:28] RECOVERY - puppet last run on ms-be3004 is OK: OK: Puppet is currently enabled, last run 64 seconds ago with 0 failures [22:18:28] RECOVERY - puppet last run on cp4010 is OK: OK: Puppet is currently enabled, last run 61 seconds ago with 0 failures [22:18:28] RECOVERY - puppet last run on mw1015 is OK: OK: Puppet is currently enabled, last run 25 seconds ago with 0 failures [22:18:28] RECOVERY - puppet last run on ms-be1005 is OK: OK: Puppet is currently enabled, last run 11 seconds ago with 0 failures [22:18:28] RECOVERY - puppet last run on db1049 is OK: OK: Puppet is currently enabled, last run 45 seconds ago with 0 failures [22:18:36] RECOVERY - puppet last run on mw1020 is OK: OK: Puppet is currently enabled, last run 30 seconds ago with 0 failures [22:18:36] RECOVERY - puppet last run on mw1017 is OK: OK: Puppet is currently enabled, last run 37 seconds ago with 0 failures [22:18:36] RECOVERY - puppet last run on mw1095 is OK: OK: Puppet is currently enabled, last run 61 seconds ago with 0 failures [22:18:38] RECOVERY - puppet last run on lanthanum is OK: OK: Puppet is currently enabled, last run 53 seconds ago with 0 failures [22:18:46] RECOVERY - puppet last run on cp4015 is OK: OK: Puppet is currently enabled, last run 53 seconds ago with 0 failures [22:18:46] RECOVERY - puppet last run on cp1064 is OK: OK: Puppet is currently enabled, last run 53 seconds ago with 0 failures [22:18:46] RECOVERY - puppet last run on virt1009 is OK: OK: Puppet is currently enabled, last run 58 seconds ago with 0 failures [22:19:01] RECOVERY - puppet last run on mw1157 is OK: OK: Puppet is currently enabled, last run 31 seconds ago with 0 failures [22:19:02] RECOVERY - puppet last run on ssl1004 is OK: OK: Puppet is currently enabled, last run 14 seconds ago with 0 failures [22:19:36] RECOVERY - puppet last run on mw1094 is OK: OK: Puppet is currently enabled, last run 60 seconds ago with 0 failures [22:19:36] RECOVERY - puppet last run on mw1136 is OK: OK: Puppet is currently enabled, last run 25 seconds ago with 0 failures [22:19:36] RECOVERY - puppet last run on lvs1006 is OK: OK: Puppet is currently enabled, last run 53 seconds ago with 0 failures [22:19:46] RECOVERY - puppet last run on lvs4001 is OK: OK: Puppet is currently enabled, last run 52 seconds ago with 0 failures [22:56:28] RECOVERY - Unmerged changes on repository puppet on palladium is OK: No changes to merge. [22:56:49] RECOVERY - Unmerged changes on repository puppet on strontium is OK: No changes to merge. [22:59:17] It's almost worth it for all the sweet, sweet recovery messages [23:01:35] afaics nothing is wrong now. no idea what may have been transiently wrong when apt was failing [23:02:15] also, these may have been missed in the spam or something (well they're ~16H old): [23:02:18] mw1123 HTTP CRITICAL: HTTP/1.0 500 Internal Server Error - 50413 bytes in 0.032 second response time [23:02:21] mw1196 HTTP CRITICAL: HTTP/1.0 500 Internal Server Error - 50413 bytes in 0.034 second response time [23:02:24] osmium PROCS CRITICAL: 0 processes with command name 'hhvm' [23:02:26] ^ still active problems in icinga [23:06:05] Sep 28 23:05:33 mw1196: PHP Fatal error: Base lambda function for closure not found in /srv/mediawiki/php-1.24wmf22/extensions/Wikidata/extensions/Wikibase/lib/config/WikibaseLib.default.php on line 18 [23:06:09] Sep 28 23:05:33 mw1123: PHP Fatal error: Base lambda function for closure not found in /srv/mediawiki/php-1.24wmf22/extensions/Wikidata/extensions/Wikibase/lib/config/WikibaseLib.default.php on line 18 [23:06:12] PROBLEM - MySQL Replication Heartbeat on db1042 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [23:06:13] PROBLEM - MySQL Idle Transactions on db1042 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [23:06:13] PROBLEM - MySQL disk space on db1042 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [23:06:13] PROBLEM - DPKG on db1042 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [23:06:13] PROBLEM - mysqld processes on db1042 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [23:06:22] PROBLEM - puppet last run on db1042 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [23:06:23] PROBLEM - MySQL InnoDB on db1042 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [23:06:23] PROBLEM - Full LVS Snapshot on db1042 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [23:06:32] PROBLEM - check if dhclient is running on db1042 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [23:06:32] PROBLEM - MySQL Recent Restart on db1042 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [23:06:32] PROBLEM - RAID on db1042 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [23:06:33] PROBLEM - Disk space on db1042 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [23:06:33] PROBLEM - check configured eth on db1042 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [23:06:42] PROBLEM - MySQL Slave Running on db1042 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [23:06:42] PROBLEM - SSH on db1042 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:06:42] PROBLEM - MySQL Processlist on db1042 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [23:06:54] PROBLEM - MySQL Slave Delay on db1042 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [23:09:05] bblack: Restart the apaches that is happening on... [23:09:42] RECOVERY - Apache HTTP on mw1123 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 400 bytes in 0.448 second response time [23:10:21] !log restarted apache on mw1123 + mw1196 [23:10:27] hoo: thanks :) [23:10:35] morebots: ? [23:10:53] RECOVERY - Apache HTTP on mw1196 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 400 bytes in 0.074 second response time [23:11:25] Logged the message, Master [23:11:25] I am a logbot running on tools-exec-07. [23:11:25] Messages are logged to wikitech.wikimedia.org/wiki/Server_Admin_Log. [23:11:25] To log a message, type !log . [23:11:52] !log test [23:11:58] Logged the message, Master [23:12:00] * hoo slaps himself [23:12:05] !log restarted apache on mw1123 + mw1196 [23:12:17] oh it did say it logged that one, eventually [23:13:31] bblack (or anyone): I need to purge a memcache key with a maintenance script that ebernhardson wrote, but I can't figure out how to run it [23:13:35] who is on db1042 mgmt? [23:13:49] springle: me, I just started to look, but I'll let you do it :) [23:13:49] spagewmf: where's the script? [23:14:00] bblack: oh cool :) tnx [23:14:06] tin:/home/ebernhardson/purge.php , doesn't run with mwscript because that lookw for it within $IP [23:14:26] (what I saw was an incomplete bit of a dmesg output, and no login-prompt response. it's probably dead, Jim) [23:14:38] spagewmf: just do: 'mwscript eval.php enwiki', then: 'require_once "/home/ebernhardson/purge.php";' [23:14:50] spagewmf: (replacing enwiki with the desired target wiki obviously) [23:15:10] ori excellent thanks [23:15:28] !log springle Synchronized wmf-config/db-eqiad.php: depool db1042, locked up (duration: 00m 07s) [23:15:33] Logged the message, Master [23:16:04] (03PS1) 10Hoo man: /usr/sbin/scap-2 no longer exists [puppet] - 10https://gerrit.wikimedia.org/r/163532 [23:16:11] easy peasy CR ;) [23:17:00] !log powercycle db1042 [23:17:06] Logged the message, Master [23:18:27] (03PS2) 10Hoo man: /usr/sbin/scap-2 no longer exists [puppet] - 10https://gerrit.wikimedia.org/r/163532 [23:18:31] (03CR) 10BBlack: [C: 032] "Yeah it doesn't seem to :)" [puppet] - 10https://gerrit.wikimedia.org/r/163532 (owner: 10Hoo man) [23:19:09] bblack: sorry... I realized the comment is also outdated and amended [23:19:12] PROBLEM - NTP on db1042 is CRITICAL: NTP CRITICAL: No response from NTP server [23:19:18] (03CR) 10BBlack: [C: 032] /usr/sbin/scap-2 no longer exists [puppet] - 10https://gerrit.wikimedia.org/r/163532 (owner: 10Hoo man) [23:19:35] it was kinda neat actually, you exploited a gerrit race condition :) [23:19:49] I reviewed PS1, hit +2, got an error box, and it left things looking like I +2'd PS2 [23:21:01] heh :) [23:23:58] RECOVERY - NTP on db1042 is OK: NTP OK: Offset -0.01498222351 secs [23:25:57] (03PS1) 10Springle: db1042 depooled from tin. make it stick. [mediawiki-config] - 10https://gerrit.wikimedia.org/r/163534 [23:26:27] (03CR) 10Springle: [C: 032] db1042 depooled from tin. make it stick. [mediawiki-config] - 10https://gerrit.wikimedia.org/r/163534 (owner: 10Springle) [23:26:32] (03Merged) 10jenkins-bot: db1042 depooled from tin. make it stick. [mediawiki-config] - 10https://gerrit.wikimedia.org/r/163534 (owner: 10Springle) [23:27:42] !log springle Synchronized wmf-config/db-eqiad.php: switch db1042 load groups to db1056 (duration: 00m 06s) [23:27:49] Logged the message, Master [23:28:39] RECOVERY - MySQL Recent Restart on db1042 is OK: OK seconds since restart [23:28:39] RECOVERY - Full LVS Snapshot on db1042 is OK: OK no full LVM snapshot volumes [23:28:39] RECOVERY - check if dhclient is running on db1042 is OK: PROCS OK: 0 processes with command name dhclient [23:28:39] RECOVERY - RAID on db1042 is OK: OK: optimal, 1 logical, 2 physical [23:28:48] RECOVERY - check configured eth on db1042 is OK: NRPE: Unable to read output [23:28:48] RECOVERY - Disk space on db1042 is OK: DISK OK [23:28:59] RECOVERY - SSH on db1042 is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1.4 (protocol 2.0) [23:28:59] RECOVERY - MySQL Slave Running on db1042 is OK: OK replication [23:29:08] RECOVERY - MySQL Slave Delay on db1042 is OK: OK replication delay 0 seconds [23:29:16] RECOVERY - MySQL Replication Heartbeat on db1042 is OK: OK replication delay 0 seconds [23:29:29] RECOVERY - MySQL Idle Transactions on db1042 is OK: OK longest blocking idle transaction sleeps for seconds [23:29:29] RECOVERY - MySQL disk space on db1042 is OK: DISK OK [23:29:38] RECOVERY - puppet last run on db1042 is OK: OK: Puppet is currently enabled, last run 12 seconds ago with 0 failures [23:29:38] RECOVERY - DPKG on db1042 is OK: All packages OK