[00:00:01] (03CR) 10Dzahn: [C: 031] "there, the only differences are some source pathes for files (you can check i moved those into the module) and class names (i call stuff '" [operations/puppet] - 10https://gerrit.wikimedia.org/r/116064 (owner: 10Dzahn) [00:00:14] (03PS1) 10BBlack: Change puppet splaylimit from 60 to 59 [operations/puppet] - 10https://gerrit.wikimedia.org/r/154367 [00:01:04] (03PS1) 10Chmarkine: wikitech - use ssl_ciphersuite to add HSTS [operations/puppet] - 10https://gerrit.wikimedia.org/r/154368 [00:01:15] (03CR) 10BBlack: [C: 032] Change puppet splaylimit from 60 to 59 [operations/puppet] - 10https://gerrit.wikimedia.org/r/154367 (owner: 10BBlack) [00:01:24] (03PS2) 10Chmarkine: wikitech - use ssl_ciphersuite to add HSTS [operations/puppet] - 10https://gerrit.wikimedia.org/r/154368 [00:02:09] RECOVERY - Unmerged changes on repository puppet on strontium is OK: No changes to merge. [00:04:27] (03CR) 10Ori.livneh: "Tested on Vagrant with both Ruby 1.8 and 1.9" [operations/puppet] - 10https://gerrit.wikimedia.org/r/154366 (owner: 10Ori.livneh) [00:12:58] PROBLEM - Puppet freshness on db1006 is CRITICAL: Last successful Puppet run was Fri 15 Aug 2014 22:12:47 UTC [00:13:29] (03PS2) 10Ori.livneh: ordered_yaml(): fix for Ruby 1.8 [operations/puppet] - 10https://gerrit.wikimedia.org/r/154366 [00:14:43] (03PS1) 10Dzahn: exim templates - deprecated variable syntax [operations/puppet] - 10https://gerrit.wikimedia.org/r/154371 [00:22:02] (03PS3) 10Ori.livneh: ordered_yaml(): fix for Ruby 1.8 [operations/puppet] - 10https://gerrit.wikimedia.org/r/154366 [00:23:32] (03PS1) 10Dzahn: sudoers.erb - deprecated variable access [operations/puppet] - 10https://gerrit.wikimedia.org/r/154372 [00:25:48] (03PS1) 10Dzahn: ferm rule.erb - deprecated variable access [operations/puppet] - 10https://gerrit.wikimedia.org/r/154373 [00:26:48] (03CR) 10Dzahn: "yeah, just comments. but the point is to clean up the puppet-compiler output or it's hard to see signal in the all the noise" [operations/puppet] - 10https://gerrit.wikimedia.org/r/154373 (owner: 10Dzahn) [00:33:28] RECOVERY - Puppet freshness on db1006 is OK: puppet ran at Sat Aug 16 00:33:20 UTC 2014 [00:45:08] PROBLEM - check_fundraising_jobs on db1025 is CRITICAL: CRITICAL missing_thank_yous=1: recurring_gc_contribs_missed=1003: recurring_gc_failures_missed=266 [critical =50]: recurring_gc_jobs_required=1014: recurring_gc_schedule_sanity=0 [00:49:58] PROBLEM - Puppet freshness on mw1053 is CRITICAL: Last successful Puppet run was Thu 14 Aug 2014 20:37:55 UTC [00:50:08] PROBLEM - check_fundraising_jobs on db1025 is CRITICAL: CRITICAL missing_thank_yous=1: recurring_gc_contribs_missed=1003: recurring_gc_failures_missed=266 [critical =50]: recurring_gc_jobs_required=1014: recurring_gc_schedule_sanity=0 [00:55:08] PROBLEM - check_fundraising_jobs on db1025 is CRITICAL: CRITICAL missing_thank_yous=1: recurring_gc_contribs_missed=1003: recurring_gc_failures_missed=266 [critical =50]: recurring_gc_jobs_required=1014: recurring_gc_schedule_sanity=0 [00:57:48] (03PS11) 10Dzahn: turn icinga into module [operations/puppet] - 10https://gerrit.wikimedia.org/r/145472 [01:00:08] PROBLEM - check_fundraising_jobs on db1025 is CRITICAL: CRITICAL missing_thank_yous=1: recurring_gc_contribs_missed=1003: recurring_gc_failures_missed=266 [critical =50]: recurring_gc_jobs_required=1014: recurring_gc_schedule_sanity=0 [01:02:19] (03CR) 10Dzahn: "puppet compiler cant run naggen2. as in content => generate('/usr/local/bin/naggen2'" [operations/puppet] - 10https://gerrit.wikimedia.org/r/145472 (owner: 10Dzahn) [01:02:34] mutante, is that fundraising check important? [01:03:52] jeremyb: i don't really know [01:04:24] let's see the history of it [01:05:08] PROBLEM - check_fundraising_jobs on db1025 is CRITICAL: CRITICAL missing_thank_yous=1: recurring_gc_contribs_missed=1003: recurring_gc_failures_missed=266 [critical =50]: recurring_gc_jobs_required=1014: recurring_gc_schedule_sanity=0 [01:05:57] at least the number isnt growing further [01:06:09] tells -fundraising [01:07:31] it's the recurring_gc_failures_missed [01:09:26] i don't see it puppetized [01:10:08] PROBLEM - check_fundraising_jobs on db1025 is CRITICAL: CRITICAL missing_thank_yous=1: recurring_gc_contribs_missed=1003: recurring_gc_failures_missed=266 [critical =50]: recurring_gc_jobs_required=1014: recurring_gc_schedule_sanity=0 [01:10:49] PROBLEM - puppet last run on mw1051 is CRITICAL: CRITICAL: Epic puppet fail [01:10:49] PROBLEM - puppet last run on snapshot1002 is CRITICAL: CRITICAL: Epic puppet fail [01:10:49] PROBLEM - puppet last run on nfs1 is CRITICAL: CRITICAL: Epic puppet fail [01:10:58] PROBLEM - puppet last run on mw1098 is CRITICAL: CRITICAL: Epic puppet fail [01:11:08] PROBLEM - puppet last run on mw1081 is CRITICAL: CRITICAL: Epic puppet fail [01:11:08] PROBLEM - puppet last run on cp1046 is CRITICAL: CRITICAL: Epic puppet fail [01:11:09] PROBLEM - puppet last run on mw1079 is CRITICAL: CRITICAL: Epic puppet fail [01:11:09] PROBLEM - puppet last run on ssl1009 is CRITICAL: CRITICAL: Epic puppet fail [01:14:38] PROBLEM - Disk space on ms-be1005 is CRITICAL: DISK CRITICAL - /srv/swift-storage/sdk1 is not accessible: Input/output error [01:14:38] PROBLEM - RAID on ms-be1005 is CRITICAL: CRITICAL: 1 failed LD(s) (Offline) [01:15:08] PROBLEM - check_fundraising_jobs on db1025 is CRITICAL: CRITICAL missing_thank_yous=1: recurring_gc_contribs_missed=1003: recurring_gc_failures_missed=266 [critical =50]: recurring_gc_jobs_required=1014: recurring_gc_schedule_sanity=0 [01:18:23] hello Jeff_Green [01:18:43] you here for the FR jobs ? [01:20:08] PROBLEM - check_fundraising_jobs on db1025 is CRITICAL: CRITICAL missing_thank_yous=1: recurring_gc_contribs_missed=1003: recurring_gc_failures_missed=266 [critical =50]: recurring_gc_jobs_required=1014: recurring_gc_schedule_sanity=0 [01:20:49] PROBLEM - puppet last run on ms-be1005 is CRITICAL: CRITICAL: Puppet has 1 failures [01:25:08] PROBLEM - check_fundraising_jobs on db1025 is CRITICAL: CRITICAL missing_thank_yous=1: recurring_gc_contribs_missed=1003: recurring_gc_failures_missed=266 [critical =50]: recurring_gc_jobs_required=1014: recurring_gc_schedule_sanity=0 [01:28:01] yeah [01:28:17] i bumped up the alert thresholds for now [01:28:47] ah, thanks, at least it seemed to stick to that value [01:28:48] 266 [01:28:48] RECOVERY - puppet last run on snapshot1002 is OK: OK: Puppet is currently enabled, last run 25 seconds ago with 0 failures [01:29:48] RECOVERY - puppet last run on nfs1 is OK: OK: Puppet is currently enabled, last run 16 seconds ago with 0 failures [01:29:58] RECOVERY - puppet last run on mw1098 is OK: OK: Puppet is currently enabled, last run 20 seconds ago with 0 failures [01:30:08] PROBLEM - check_fundraising_jobs on db1025 is CRITICAL: CRITICAL missing_thank_yous=1: recurring_gc_contribs_missed=1003: recurring_gc_failures_missed=266 [critical =50]: recurring_gc_jobs_required=1014: recurring_gc_schedule_sanity=0 [01:30:09] RECOVERY - puppet last run on cp1046 is OK: OK: Puppet is currently enabled, last run 19 seconds ago with 0 failures [01:30:09] RECOVERY - puppet last run on mw1079 is OK: OK: Puppet is currently enabled, last run 15 seconds ago with 0 failures [01:30:09] RECOVERY - puppet last run on ssl1009 is OK: OK: Puppet is currently enabled, last run 24 seconds ago with 0 failures [01:30:22] yeah. I don't think it's a big deal to sit at that level until the fr-tech folks return [01:30:48] RECOVERY - puppet last run on mw1051 is OK: OK: Puppet is currently enabled, last run 59 seconds ago with 0 failures [01:31:08] RECOVERY - puppet last run on mw1081 is OK: OK: Puppet is currently enabled, last run 39 seconds ago with 0 failures [01:32:58] RECOVERY - check_fundraising_jobs on db1025 is OK: OK missing_thank_yous=1: recurring_gc_contribs_missed=1003: recurring_gc_failures_missed=266: recurring_gc_jobs_required=1014: recurring_gc_schedule_sanity=0 [01:37:14] ori: That seems like it's the normal Passenger MO. I wish more daemons keeping pools of workers understood the importance of adding some randomness to their lifetimes though. [02:17:03] !log LocalisationUpdate completed (1.24wmf16) at 2014-08-16 02:16:00+00:00 [02:17:13] Logged the message, Master [02:23:45] Coren: "I wish more daemons keeping pools of workers understood the importance of adding some randomness to their lifetimes though." [02:23:55] Coren: ELI5? :) [02:24:13] (i.e.: what is the importance?) [02:25:13] (03CR) 10Ori.livneh: [C: 04-1] "Won't work; Puppet uses a custom gem for YAML serialization called "zaml". Of course." [operations/puppet] - 10https://gerrit.wikimedia.org/r/154366 (owner: 10Ori.livneh) [02:25:52] Hm. Say you configure the deamons to server a max of 50 requests. It means that they'll all die at pretty much the same time (after 50 requests each) and will all restart at the same time possibly causing contention. [02:26:43] Coren: ah, right. makes sense. [02:26:47] A smart deamon, when configured to have workers server max X request would actually have them restart after X-N where N is some small random number. As time progresses, the restarts get more and more distributed and you rarely end up with many dying and restarting all at once. [02:26:53] (03PS1) 10Anomie: Fix $wgRestrictionLevels ordering [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/154376 (https://bugzilla.wikimedia.org/69640) [02:27:06] !log LocalisationUpdate completed (1.24wmf17) at 2014-08-16 02:26:02+00:00 [02:27:11] Logged the message, Master [02:27:23] Coren: *nod* thanks! [02:27:52] It's also the reason why Ethernet specifies that you have to wait some random extra time after a collision before retrying; so that you don't end up with a bunch of trancievers colliding, waiting a fixed time, trying again at the same time and all failing again. :-) [02:28:51] yes, like two people trying to move out of each other's way in lock-step [02:28:58] * Coren chuckles. [02:29:00] Right. [02:30:41] hah [03:07:35] !log LocalisationUpdate ResourceLoader cache refresh completed at Sat Aug 16 03:06:29 UTC 2014 (duration 6m 28s) [03:07:40] Logged the message, Master [04:50:56] (03PS4) 10Ori.livneh: ordered_yaml(): fix for Ruby 1.8 [operations/puppet] - 10https://gerrit.wikimedia.org/r/154366 [04:51:58] PROBLEM - Puppet freshness on mw1053 is CRITICAL: Last successful Puppet run was Thu 14 Aug 2014 20:37:55 UTC [05:20:58] PROBLEM - Puppet freshness on db1010 is CRITICAL: Last successful Puppet run was Sat 16 Aug 2014 03:20:10 UTC [05:40:58] PROBLEM - Puppet freshness on db1007 is CRITICAL: Last successful Puppet run was Sat 16 Aug 2014 03:40:20 UTC [06:11:28] PROBLEM - MySQL Processlist on db1068 is CRITICAL: CRIT 145 unauthenticated, 0 locked, 0 copy to table, 0 statistics [06:12:28] RECOVERY - MySQL Processlist on db1068 is OK: OK 0 unauthenticated, 0 locked, 0 copy to table, 0 statistics [06:20:18] RECOVERY - Puppet freshness on db1007 is OK: puppet ran at Sat Aug 16 06:20:13 UTC 2014 [06:28:38] PROBLEM - puppet last run on mw1088 is CRITICAL: CRITICAL: Puppet has 1 failures [06:28:39] PROBLEM - puppet last run on mw1173 is CRITICAL: CRITICAL: Puppet has 1 failures [06:28:49] PROBLEM - puppet last run on mw1065 is CRITICAL: CRITICAL: Puppet has 1 failures [06:28:50] PROBLEM - puppet last run on mw1003 is CRITICAL: CRITICAL: Puppet has 1 failures [06:29:18] PROBLEM - puppet last run on mw1189 is CRITICAL: CRITICAL: Puppet has 1 failures [06:29:28] PROBLEM - puppet last run on db1059 is CRITICAL: CRITICAL: Puppet has 1 failures [06:29:29] PROBLEM - puppet last run on mw1205 is CRITICAL: CRITICAL: Puppet has 1 failures [06:30:09] PROBLEM - puppet last run on cp4003 is CRITICAL: CRITICAL: Puppet has 1 failures [06:33:09] PROBLEM - puppet last run on searchidx1001 is CRITICAL: CRITICAL: Puppet has 1 failures [06:33:58] PROBLEM - Puppet freshness on db1006 is CRITICAL: Last successful Puppet run was Sat 16 Aug 2014 04:33:13 UTC [06:40:08] RECOVERY - Puppet freshness on db1010 is OK: puppet ran at Sat Aug 16 06:39:59 UTC 2014 [06:44:18] RECOVERY - puppet last run on mw1189 is OK: OK: Puppet is currently enabled, last run 1 seconds ago with 0 failures [06:45:09] RECOVERY - puppet last run on searchidx1001 is OK: OK: Puppet is currently enabled, last run 2 seconds ago with 0 failures [06:45:28] RECOVERY - puppet last run on db1059 is OK: OK: Puppet is currently enabled, last run 18 seconds ago with 0 failures [06:45:39] RECOVERY - puppet last run on mw1173 is OK: OK: Puppet is currently enabled, last run 32 seconds ago with 0 failures [06:46:29] RECOVERY - puppet last run on mw1205 is OK: OK: Puppet is currently enabled, last run 40 seconds ago with 0 failures [06:46:38] RECOVERY - puppet last run on mw1088 is OK: OK: Puppet is currently enabled, last run 43 seconds ago with 0 failures [06:46:49] RECOVERY - puppet last run on mw1065 is OK: OK: Puppet is currently enabled, last run 32 seconds ago with 0 failures [06:46:49] RECOVERY - puppet last run on mw1003 is OK: OK: Puppet is currently enabled, last run 56 seconds ago with 0 failures [06:47:09] RECOVERY - puppet last run on cp4003 is OK: OK: Puppet is currently enabled, last run 55 seconds ago with 0 failures [06:52:58] PROBLEM - Puppet freshness on mw1053 is CRITICAL: Last successful Puppet run was Thu 14 Aug 2014 20:37:55 UTC [06:52:58] PROBLEM - puppet last run on db1020 is CRITICAL: CRITICAL: Puppet has 2 failures [06:57:38] PROBLEM - puppet last run on db60 is CRITICAL: CRITICAL: Puppet has 1 failures [07:09:58] RECOVERY - puppet last run on db1020 is OK: OK: Puppet is currently enabled, last run 40 seconds ago with 0 failures [07:16:39] RECOVERY - puppet last run on db60 is OK: OK: Puppet is currently enabled, last run 53 seconds ago with 0 failures [08:12:18] PROBLEM - Kafka Broker Messages In on analytics1021 is CRITICAL: kafka.server.BrokerTopicMetrics.AllTopicsMessagesInPerSec.FifteenMinuteRate CRITICAL: 967.091391665 [08:20:58] PROBLEM - Puppet freshness on db1007 is CRITICAL: Last successful Puppet run was Sat 16 Aug 2014 06:20:13 UTC [08:34:58] PROBLEM - Puppet freshness on db1006 is CRITICAL: Last successful Puppet run was Sat 16 Aug 2014 04:33:13 UTC [08:39:48] PROBLEM - ElasticSearch health check on elastic1007 is CRITICAL: CRITICAL - elasticsearch (production-search-eqiad) is running. status: red: timed_out: false: number_of_nodes: 17: number_of_data_nodes: 17: active_primary_shards: 2056: active_shards: 6167: relocating_shards: 0: initializing_shards: 1: unassigned_shards: 1 [08:42:48] PROBLEM - ElasticSearch health check on elastic1007 is CRITICAL: CRITICAL - elasticsearch (production-search-eqiad) is running. status: red: timed_out: false: number_of_nodes: 17: number_of_data_nodes: 17: active_primary_shards: 2056: active_shards: 6167: relocating_shards: 0: initializing_shards: 1: unassigned_shards: 1 [08:44:48] RECOVERY - ElasticSearch health check on elastic1007 is OK: OK - elasticsearch (production-search-eqiad) is running. status: green: timed_out: false: number_of_nodes: 17: number_of_data_nodes: 17: active_primary_shards: 2057: active_shards: 6170: relocating_shards: 0: initializing_shards: 0: unassigned_shards: 0 [08:53:58] PROBLEM - Puppet freshness on mw1053 is CRITICAL: Last successful Puppet run was Thu 14 Aug 2014 20:37:55 UTC [09:19:48] RECOVERY - Puppet freshness on db1007 is OK: puppet ran at Sat Aug 16 09:19:42 UTC 2014 [10:02:55] ACKNOWLEDGEMENT - Puppet freshness on mw1053 is CRITICAL: Last successful Puppet run was Thu 14 Aug 2014 20:37:55 UTC Filippo Giunchedi hhvm work [10:32:40] (03PS1) 10Gergő Tisza: Disable MediaViewer by default for logged-in users on Commons [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/154396 (https://bugzilla.wikimedia.org/69363) [10:35:58] PROBLEM - Puppet freshness on db1006 is CRITICAL: Last successful Puppet run was Sat 16 Aug 2014 04:33:13 UTC [10:53:08] RECOVERY - Puppet freshness on db1006 is OK: puppet ran at Sat Aug 16 10:53:02 UTC 2014 [10:53:33] (03CR) 10Ori.livneh: [C: 032] ordered_yaml(): fix for Ruby 1.8 [operations/puppet] - 10https://gerrit.wikimedia.org/r/154366 (owner: 10Ori.livneh) [11:01:25] (03CR) 10Odder: [C: 04-1] Fix $wgRestrictionLevels ordering [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/154376 (https://bugzilla.wikimedia.org/69640) (owner: 10Anomie) [11:12:21] (03CR) 10Steinsplitter: [C: 04-1] Fix $wgRestrictionLevels ordering [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/154376 (https://bugzilla.wikimedia.org/69640) (owner: 10Anomie) [11:17:08] PROBLEM - puppet last run on virt1000 is CRITICAL: CRITICAL: Puppet has 1 failures [11:19:41] (03CR) 10Bartosz Dziewoński: "By the way, this caused bug 69464 / bug 69640. Please don't make unrelated changes when changing configuration." [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/153302 (owner: 10Tim Starling) [11:19:58] PROBLEM - Puppet freshness on db1007 is CRITICAL: Last successful Puppet run was Sat 16 Aug 2014 09:19:42 UTC [11:21:41] (03CR) 10Bartosz Dziewoński: [C: 031] "Odder, Steinsplitter, what do the -1's mean? Has borderless hate dimmed your minds? This is fixing an obvious bug that is unrelated to the" [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/154376 (https://bugzilla.wikimedia.org/69640) (owner: 10Anomie) [11:21:54] fucking retards [11:21:59] fuck you [11:22:06] (03CR) 10Ricordisamoa: [C: 04-1] "Instead, Idfa211257dbacc7623d42393257de1525ff01e9e should be reverted." [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/154376 (https://bugzilla.wikimedia.org/69640) (owner: 10Anomie) [11:26:18] PROBLEM - puppet last run on cp3003 is CRITICAL: CRITICAL: Epic puppet fail [11:27:37] fuck fuck fuck. [11:34:08] RECOVERY - puppet last run on virt1000 is OK: OK: Puppet is currently enabled, last run 38 seconds ago with 0 failures [11:46:18] RECOVERY - puppet last run on cp3003 is OK: OK: Puppet is currently enabled, last run 20 seconds ago with 0 failures [11:59:08] PROBLEM - puppet last run on mw1095 is CRITICAL: CRITICAL: Epic puppet fail [11:59:09] PROBLEM - puppet last run on labsdb1005 is CRITICAL: CRITICAL: Puppet has 7 failures [11:59:09] PROBLEM - puppet last run on analytics1021 is CRITICAL: CRITICAL: Puppet has 7 failures [11:59:09] PROBLEM - puppet last run on mw1017 is CRITICAL: CRITICAL: Epic puppet fail [11:59:09] PROBLEM - puppet last run on cp4015 is CRITICAL: CRITICAL: Epic puppet fail [11:59:18] PROBLEM - puppetmaster backend https on palladium is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 8141: HTTP/1.1 500 Internal Server Error [11:59:18] PROBLEM - puppet last run on amssq52 is CRITICAL: CRITICAL: Epic puppet fail [11:59:28] PROBLEM - puppet last run on mw1184 is CRITICAL: CRITICAL: Epic puppet fail [11:59:38] PROBLEM - puppet last run on nickel is CRITICAL: CRITICAL: Epic puppet fail [11:59:38] PROBLEM - puppet last run on mw1131 is CRITICAL: CRITICAL: Puppet has 32 failures [11:59:38] PROBLEM - puppet last run on search1019 is CRITICAL: CRITICAL: Puppet has 40 failures [11:59:38] PROBLEM - puppet last run on mw1078 is CRITICAL: CRITICAL: Epic puppet fail [11:59:38] PROBLEM - puppet last run on lanthanum is CRITICAL: CRITICAL: Epic puppet fail [11:59:38] PROBLEM - puppet last run on labsdb1002 is CRITICAL: CRITICAL: Puppet has 18 failures [11:59:39] PROBLEM - puppet last run on ocg1002 is CRITICAL: CRITICAL: Epic puppet fail [11:59:39] PROBLEM - puppet last run on fenari is CRITICAL: CRITICAL: Puppet has 4 failures [11:59:40] PROBLEM - puppet last run on mexia is CRITICAL: CRITICAL: Puppet has 1 failures [11:59:40] PROBLEM - puppet last run on hydrogen is CRITICAL: CRITICAL: Puppet has 23 failures [11:59:41] PROBLEM - puppet last run on zinc is CRITICAL: CRITICAL: Puppet has 18 failures [11:59:48] PROBLEM - puppet last run on mw1179 is CRITICAL: CRITICAL: Epic puppet fail [11:59:48] PROBLEM - puppet last run on mw1085 is CRITICAL: CRITICAL: Epic puppet fail [11:59:48] PROBLEM - puppet last run on ms-be1014 is CRITICAL: CRITICAL: Epic puppet fail [11:59:48] PROBLEM - puppet last run on mw1019 is CRITICAL: CRITICAL: Puppet has 48 failures [11:59:48] PROBLEM - puppet last run on search1003 is CRITICAL: CRITICAL: Puppet has 40 failures [11:59:48] PROBLEM - puppet last run on search1008 is CRITICAL: CRITICAL: Puppet has 45 failures [11:59:49] PROBLEM - puppet last run on db1049 is CRITICAL: CRITICAL: Puppet has 19 failures [11:59:49] PROBLEM - puppet last run on mw1102 is CRITICAL: CRITICAL: Puppet has 62 failures [11:59:50] PROBLEM - puppet last run on elastic1013 is CRITICAL: CRITICAL: Epic puppet fail [11:59:50] PROBLEM - puppet last run on cp4017 is CRITICAL: CRITICAL: Epic puppet fail [11:59:51] PROBLEM - puppet last run on cp4011 is CRITICAL: CRITICAL: Puppet has 12 failures [11:59:58] PROBLEM - puppet last run on cp3007 is CRITICAL: CRITICAL: Epic puppet fail [11:59:58] PROBLEM - puppet last run on amssq45 is CRITICAL: CRITICAL: Epic puppet fail [11:59:58] PROBLEM - puppet last run on eeden is CRITICAL: CRITICAL: Puppet has 4 failures [11:59:59] PROBLEM - puppet last run on db1068 is CRITICAL: CRITICAL: Puppet has 22 failures [12:00:08] PROBLEM - puppet last run on mw1182 is CRITICAL: CRITICAL: Epic puppet fail [12:00:08] PROBLEM - puppet last run on mw1101 is CRITICAL: CRITICAL: Puppet has 62 failures [12:00:08] PROBLEM - puppet last run on mw1070 is CRITICAL: CRITICAL: Puppet has 62 failures [12:00:09] PROBLEM - puppet last run on palladium is CRITICAL: CRITICAL: Puppet has 35 failures [12:00:09] PROBLEM - puppet last run on mw1018 is CRITICAL: CRITICAL: Puppet has 38 failures [12:00:09] PROBLEM - puppet last run on db1009 is CRITICAL: CRITICAL: Puppet has 17 failures [12:00:09] PROBLEM - puppet last run on mw1094 is CRITICAL: CRITICAL: Epic puppet fail [12:00:10] PROBLEM - puppet last run on mw1021 is CRITICAL: CRITICAL: Puppet has 33 failures [12:00:10] PROBLEM - puppet last run on analytics1012 is CRITICAL: CRITICAL: Puppet has 19 failures [12:00:11] PROBLEM - puppet last run on mw1047 is CRITICAL: CRITICAL: Puppet has 57 failures [12:00:11] PROBLEM - puppet last run on mw1199 is CRITICAL: CRITICAL: Puppet has 36 failures [12:00:12] PROBLEM - puppet last run on mw1135 is CRITICAL: CRITICAL: Puppet has 37 failures [12:00:12] PROBLEM - puppet last run on cp4010 is CRITICAL: CRITICAL: Puppet has 16 failures [12:00:13] PROBLEM - puppet last run on cp4013 is CRITICAL: CRITICAL: Epic puppet fail [12:00:18] PROBLEM - puppet last run on lvs3002 is CRITICAL: CRITICAL: Puppet has 21 failures [12:00:18] PROBLEM - puppet last run on cp3019 is CRITICAL: CRITICAL: Puppet has 20 failures [12:00:18] PROBLEM - puppet last run on analytics1024 is CRITICAL: CRITICAL: Epic puppet fail [12:00:18] PROBLEM - puppet last run on mw1157 is CRITICAL: CRITICAL: Puppet has 75 failures [12:00:18] PROBLEM - puppet last run on cp1064 is CRITICAL: CRITICAL: Puppet has 26 failures [12:00:18] PROBLEM - puppet last run on mw1075 is CRITICAL: CRITICAL: Puppet has 64 failures [12:00:18] PROBLEM - puppet last run on cp3017 is CRITICAL: CRITICAL: Puppet has 14 failures [12:00:19] PROBLEM - puppet last run on db1005 is CRITICAL: CRITICAL: Puppet has 21 failures [12:00:19] PROBLEM - puppet last run on cp3021 is CRITICAL: CRITICAL: Puppet has 17 failures [12:00:20] PROBLEM - puppet last run on amssq50 is CRITICAL: CRITICAL: Puppet has 11 failures [12:00:28] PROBLEM - puppet last run on cp4016 is CRITICAL: CRITICAL: Puppet has 28 failures [12:00:28] PROBLEM - puppet last run on wtp1014 is CRITICAL: CRITICAL: Puppet has 23 failures [12:00:29] PROBLEM - puppet last run on db1058 is CRITICAL: CRITICAL: Puppet has 20 failures [12:00:29] PROBLEM - puppet last run on labsdb1007 is CRITICAL: CRITICAL: Epic puppet fail [12:00:29] PROBLEM - puppet last run on ssl1004 is CRITICAL: CRITICAL: Epic puppet fail [12:00:29] PROBLEM - puppet last run on cp1057 is CRITICAL: CRITICAL: Puppet has 24 failures [12:00:29] PROBLEM - puppet last run on es1005 is CRITICAL: CRITICAL: Puppet has 17 failures [12:00:30] PROBLEM - puppet last run on db1019 is CRITICAL: CRITICAL: Epic puppet fail [12:00:38] PROBLEM - puppet last run on elastic1016 is CRITICAL: CRITICAL: Puppet has 19 failures [12:00:38] PROBLEM - puppet last run on virt1005 is CRITICAL: CRITICAL: Puppet has 25 failures [12:00:38] PROBLEM - puppet last run on elastic1009 is CRITICAL: CRITICAL: Puppet has 18 failures [12:00:38] PROBLEM - puppet last run on mw1127 is CRITICAL: CRITICAL: Puppet has 57 failures [12:00:38] PROBLEM - puppet last run on elastic1010 is CRITICAL: CRITICAL: Puppet has 25 failures [12:00:39] PROBLEM - puppet last run on mw1103 is CRITICAL: CRITICAL: Puppet has 61 failures [12:00:39] PROBLEM - puppet last run on mw1015 is CRITICAL: CRITICAL: Puppet has 48 failures [12:00:40] PROBLEM - puppet last run on mw1136 is CRITICAL: CRITICAL: Epic puppet fail [12:00:48] PROBLEM - puppet last run on mw1169 is CRITICAL: CRITICAL: Puppet has 67 failures [12:00:48] PROBLEM - puppet last run on mw1214 is CRITICAL: CRITICAL: Puppet has 62 failures [12:00:48] PROBLEM - puppet last run on analytics1029 is CRITICAL: CRITICAL: Puppet has 22 failures [12:00:48] PROBLEM - puppet last run on mw1058 is CRITICAL: CRITICAL: Puppet has 65 failures [12:00:48] PROBLEM - puppet last run on mw1020 is CRITICAL: CRITICAL: Puppet has 58 failures [12:00:48] PROBLEM - puppet last run on cp1069 is CRITICAL: CRITICAL: Epic puppet fail [12:00:48] PROBLEM - puppet last run on erbium is CRITICAL: CRITICAL: Epic puppet fail [12:00:49] PROBLEM - puppet last run on zirconium is CRITICAL: CRITICAL: Epic puppet fail [12:00:49] PROBLEM - puppet last run on cp1051 is CRITICAL: CRITICAL: Epic puppet fail [12:00:50] PROBLEM - puppet last run on mw1083 is CRITICAL: CRITICAL: Puppet has 59 failures [12:00:50] PROBLEM - puppet last run on lvs1006 is CRITICAL: CRITICAL: Puppet has 17 failures [12:00:51] PROBLEM - puppet last run on search1021 is CRITICAL: CRITICAL: Epic puppet fail [12:00:51] PROBLEM - puppet last run on mw1137 is CRITICAL: CRITICAL: Puppet has 64 failures [12:00:58] PROBLEM - puppet last run on ms-be3004 is CRITICAL: CRITICAL: Puppet has 20 failures [12:00:58] PROBLEM - puppet last run on amslvs4 is CRITICAL: CRITICAL: Epic puppet fail [12:01:08] PROBLEM - puppet last run on es1003 is CRITICAL: CRITICAL: Epic puppet fail [12:01:09] PROBLEM - puppet last run on mw1036 is CRITICAL: CRITICAL: Epic puppet fail [12:01:09] PROBLEM - puppet last run on search1009 is CRITICAL: CRITICAL: Puppet has 41 failures [12:01:18] PROBLEM - puppet last run on mc1008 is CRITICAL: CRITICAL: Puppet has 17 failures [12:01:28] PROBLEM - puppet last run on tmh1002 is CRITICAL: CRITICAL: Puppet has 25 failures [12:01:28] PROBLEM - puppet last run on analytics1034 is CRITICAL: CRITICAL: Puppet has 21 failures [12:01:38] PROBLEM - puppet last run on mw1096 is CRITICAL: CRITICAL: Puppet has 66 failures [12:01:38] PROBLEM - puppet last run on praseodymium is CRITICAL: CRITICAL: Puppet has 17 failures [12:01:38] PROBLEM - puppet last run on analytics1019 is CRITICAL: CRITICAL: Puppet has 13 failures [12:01:39] PROBLEM - puppet last run on mw1132 is CRITICAL: CRITICAL: Epic puppet fail [12:01:39] PROBLEM - puppet last run on es7 is CRITICAL: CRITICAL: Puppet has 14 failures [12:01:39] PROBLEM - puppet last run on es10 is CRITICAL: CRITICAL: Puppet has 20 failures [12:01:49] PROBLEM - puppet last run on virt0 is CRITICAL: CRITICAL: Puppet has 60 failures [12:01:49] PROBLEM - puppet last run on mw1035 is CRITICAL: CRITICAL: Epic puppet fail [12:01:49] PROBLEM - puppet last run on cp1065 is CRITICAL: CRITICAL: Epic puppet fail [12:01:49] PROBLEM - puppet last run on db1024 is CRITICAL: CRITICAL: Epic puppet fail [12:01:49] PROBLEM - puppet last run on db1041 is CRITICAL: CRITICAL: Epic puppet fail [12:01:49] PROBLEM - puppet last run on es1006 is CRITICAL: CRITICAL: Epic puppet fail [12:01:49] PROBLEM - puppet last run on mw1124 is CRITICAL: CRITICAL: Epic puppet fail [12:01:50] PROBLEM - puppet last run on lvs4001 is CRITICAL: CRITICAL: Puppet has 22 failures [12:01:58] PROBLEM - puppet last run on amssq57 is CRITICAL: CRITICAL: Puppet has 25 failures [12:01:58] PROBLEM - puppet last run on cp3013 is CRITICAL: CRITICAL: Epic puppet fail [12:01:58] PROBLEM - puppet last run on ssl3003 is CRITICAL: CRITICAL: Epic puppet fail [12:02:08] PROBLEM - puppet last run on mw1138 is CRITICAL: CRITICAL: Puppet has 76 failures [12:02:08] PROBLEM - puppet last run on analytics1015 is CRITICAL: CRITICAL: Epic puppet fail [12:02:08] PROBLEM - puppet last run on mw1161 is CRITICAL: CRITICAL: Epic puppet fail [12:02:09] PROBLEM - puppet last run on mw1196 is CRITICAL: CRITICAL: Puppet has 69 failures [12:02:09] PROBLEM - puppet last run on mw1191 is CRITICAL: CRITICAL: Puppet has 72 failures [12:02:09] PROBLEM - puppet last run on mw1013 is CRITICAL: CRITICAL: Puppet has 47 failures [12:02:09] PROBLEM - puppet last run on neon is CRITICAL: CRITICAL: Puppet has 64 failures [12:02:09] PROBLEM - puppet last run on mw1192 is CRITICAL: CRITICAL: Epic puppet fail [12:02:18] PROBLEM - puppet last run on cp1043 is CRITICAL: CRITICAL: Puppet has 18 failures [12:02:29] PROBLEM - puppet last run on mw1089 is CRITICAL: CRITICAL: Epic puppet fail [12:02:29] PROBLEM - puppet last run on virt1002 is CRITICAL: CRITICAL: Epic puppet fail [12:02:29] PROBLEM - puppet last run on es1009 is CRITICAL: CRITICAL: Puppet has 19 failures [12:02:38] PROBLEM - puppet last run on mw1147 is CRITICAL: CRITICAL: Epic puppet fail [12:02:38] PROBLEM - puppet last run on search1014 is CRITICAL: CRITICAL: Puppet has 47 failures [12:02:39] PROBLEM - puppet last run on tarin is CRITICAL: CRITICAL: Puppet has 17 failures [12:02:39] PROBLEM - puppet last run on wtp1024 is CRITICAL: CRITICAL: Puppet has 27 failures [12:02:48] PROBLEM - puppet last run on ytterbium is CRITICAL: CRITICAL: Epic puppet fail [12:02:48] PROBLEM - puppet last run on wtp1021 is CRITICAL: CRITICAL: Puppet has 22 failures [12:02:48] PROBLEM - puppet last run on mw1216 is CRITICAL: CRITICAL: Puppet has 76 failures [12:02:48] PROBLEM - puppet last run on mc1010 is CRITICAL: CRITICAL: Puppet has 15 failures [12:02:48] PROBLEM - puppet last run on cp4007 is CRITICAL: CRITICAL: Epic puppet fail [12:02:48] PROBLEM - puppet last run on mw1028 is CRITICAL: CRITICAL: Epic puppet fail [12:02:48] PROBLEM - puppet last run on analytics1039 is CRITICAL: CRITICAL: Puppet has 15 failures [12:02:49] PROBLEM - puppet last run on mw1178 is CRITICAL: CRITICAL: Epic puppet fail [12:02:49] PROBLEM - puppet last run on db1007 is CRITICAL: CRITICAL: Puppet has 21 failures [12:02:58] PROBLEM - puppet last run on amssq59 is CRITICAL: CRITICAL: Epic puppet fail [12:02:58] PROBLEM - puppet last run on amssq58 is CRITICAL: CRITICAL: Puppet has 21 failures [12:02:58] PROBLEM - puppet last run on cp3006 is CRITICAL: CRITICAL: Epic puppet fail [12:03:08] PROBLEM - puppet last run on ms1001 is CRITICAL: CRITICAL: Epic puppet fail [12:03:08] PROBLEM - puppet last run on pc1001 is CRITICAL: CRITICAL: Epic puppet fail [12:03:08] PROBLEM - puppet last run on mw1072 is CRITICAL: CRITICAL: Epic puppet fail [12:03:09] PROBLEM - puppet last run on mc1004 is CRITICAL: CRITICAL: Epic puppet fail [12:03:09] PROBLEM - puppet last run on mw1134 is CRITICAL: CRITICAL: Epic puppet fail [12:03:18] PROBLEM - puppet last run on amssq44 is CRITICAL: CRITICAL: Epic puppet fail [12:03:18] PROBLEM - puppet last run on cp3015 is CRITICAL: CRITICAL: Epic puppet fail [12:03:18] PROBLEM - puppet last run on amssq43 is CRITICAL: CRITICAL: Epic puppet fail [12:03:29] PROBLEM - puppet last run on analytics1036 is CRITICAL: CRITICAL: Puppet has 20 failures [12:03:29] PROBLEM - puppet last run on search1020 is CRITICAL: CRITICAL: Epic puppet fail [12:03:29] PROBLEM - puppet last run on mw1109 is CRITICAL: CRITICAL: Puppet has 52 failures [12:03:29] PROBLEM - puppet last run on logstash1003 is CRITICAL: CRITICAL: Epic puppet fail [12:03:29] PROBLEM - puppet last run on mw1062 is CRITICAL: CRITICAL: Puppet has 66 failures [12:03:29] PROBLEM - puppet last run on mw1048 is CRITICAL: CRITICAL: Puppet has 61 failures [12:03:38] PROBLEM - puppet last run on wtp1017 is CRITICAL: CRITICAL: Puppet has 23 failures [12:03:38] PROBLEM - puppet last run on rdb1004 is CRITICAL: CRITICAL: Epic puppet fail [12:03:39] PROBLEM - puppet last run on ocg1001 is CRITICAL: CRITICAL: Puppet has 23 failures [12:03:39] PROBLEM - puppet last run on db1010 is CRITICAL: CRITICAL: Puppet has 20 failures [12:03:39] PROBLEM - puppet last run on mw1005 is CRITICAL: CRITICAL: Epic puppet fail [12:03:39] PROBLEM - puppet last run on mw1080 is CRITICAL: CRITICAL: Epic puppet fail [12:03:48] PROBLEM - puppet last run on vanadium is CRITICAL: CRITICAL: Epic puppet fail [12:03:48] PROBLEM - puppet last run on mc1009 is CRITICAL: CRITICAL: Puppet has 20 failures [12:03:48] PROBLEM - puppet last run on mw1218 is CRITICAL: CRITICAL: Puppet has 71 failures [12:03:48] PROBLEM - puppet last run on cp1059 is CRITICAL: CRITICAL: Puppet has 23 failures [12:03:48] PROBLEM - puppet last run on dbstore1001 is CRITICAL: CRITICAL: Puppet has 18 failures [12:03:48] PROBLEM - puppet last run on ms-be1001 is CRITICAL: CRITICAL: Puppet has 22 failures [12:03:48] PROBLEM - puppet last run on cp1070 is CRITICAL: CRITICAL: Epic puppet fail [12:03:49] PROBLEM - puppet last run on mw1040 is CRITICAL: CRITICAL: Puppet has 77 failures [12:03:49] PROBLEM - puppet last run on mw1059 is CRITICAL: CRITICAL: Epic puppet fail [12:03:50] PROBLEM - puppet last run on es1001 is CRITICAL: CRITICAL: Epic puppet fail [12:03:50] PROBLEM - puppet last run on mw1031 is CRITICAL: CRITICAL: Puppet has 64 failures [12:03:51] PROBLEM - puppet last run on lvs4002 is CRITICAL: CRITICAL: Epic puppet fail [12:03:58] PROBLEM - puppet last run on cp3008 is CRITICAL: CRITICAL: Epic puppet fail [12:03:58] PROBLEM - puppet last run on amssq49 is CRITICAL: CRITICAL: Epic puppet fail [12:03:58] PROBLEM - puppet last run on magnesium is CRITICAL: CRITICAL: Epic puppet fail [12:03:58] PROBLEM - puppet last run on caesium is CRITICAL: CRITICAL: Epic puppet fail [12:03:58] PROBLEM - puppet last run on cp3022 is CRITICAL: CRITICAL: Puppet has 20 failures [12:04:08] PROBLEM - puppet last run on labsdb1001 is CRITICAL: CRITICAL: Puppet has 17 failures [12:04:08] PROBLEM - puppet last run on mw1007 is CRITICAL: CRITICAL: Epic puppet fail [12:04:08] PROBLEM - puppet last run on mc1016 is CRITICAL: CRITICAL: Epic puppet fail [12:04:09] PROBLEM - puppet last run on mc1011 is CRITICAL: CRITICAL: Puppet has 16 failures [12:04:09] PROBLEM - puppet last run on ms-be1013 is CRITICAL: CRITICAL: Puppet has 32 failures [12:04:09] PROBLEM - puppet last run on analytics1033 is CRITICAL: CRITICAL: Puppet has 24 failures [12:04:09] PROBLEM - puppet last run on mw1045 is CRITICAL: CRITICAL: Epic puppet fail [12:04:10] PROBLEM - puppet last run on ms-fe1003 is CRITICAL: CRITICAL: Puppet has 22 failures [12:04:10] PROBLEM - puppet last run on wtp1019 is CRITICAL: CRITICAL: Puppet has 20 failures [12:04:11] PROBLEM - puppet last run on mw1041 is CRITICAL: CRITICAL: Epic puppet fail [12:04:11] PROBLEM - puppet last run on mw1067 is CRITICAL: CRITICAL: Puppet has 67 failures [12:04:12] PROBLEM - puppet last run on db1017 is CRITICAL: CRITICAL: Epic puppet fail [12:04:12] PROBLEM - puppet last run on netmon1001 is CRITICAL: CRITICAL: Puppet has 33 failures [12:04:13] PROBLEM - puppet last run on mw1140 is CRITICAL: CRITICAL: Epic puppet fail [12:04:18] PROBLEM - puppet last run on dbproxy1002 is CRITICAL: CRITICAL: Puppet has 19 failures [12:04:18] PROBLEM - puppet last run on db1029 is CRITICAL: CRITICAL: Epic puppet fail [12:04:18] PROBLEM - puppet last run on rbf1002 is CRITICAL: CRITICAL: Epic puppet fail [12:04:18] PROBLEM - puppet last run on cp4006 is CRITICAL: CRITICAL: Epic puppet fail [12:04:29] PROBLEM - puppet last run on mw1038 is CRITICAL: CRITICAL: Puppet has 71 failures [12:04:38] PROBLEM - puppet last run on ms-be1004 is CRITICAL: CRITICAL: Puppet has 25 failures [12:04:38] PROBLEM - puppet last run on analytics1017 is CRITICAL: CRITICAL: Puppet has 15 failures [12:04:38] PROBLEM - puppet last run on gold is CRITICAL: CRITICAL: Epic puppet fail [12:04:38] PROBLEM - puppet last run on cerium is CRITICAL: CRITICAL: Epic puppet fail [12:04:38] PROBLEM - puppet last run on mw1012 is CRITICAL: CRITICAL: Epic puppet fail [12:04:39] PROBLEM - puppet last run on platinum is CRITICAL: CRITICAL: Epic puppet fail [12:04:39] PROBLEM - puppet last run on db73 is CRITICAL: CRITICAL: Puppet has 17 failures [12:04:40] PROBLEM - puppet last run on wtp1009 is CRITICAL: CRITICAL: Epic puppet fail [12:04:40] PROBLEM - puppet last run on logstash1001 is CRITICAL: CRITICAL: Puppet has 25 failures [12:04:48] PROBLEM - puppet last run on db1053 is CRITICAL: CRITICAL: Epic puppet fail [12:04:48] PROBLEM - puppet last run on cp1047 is CRITICAL: CRITICAL: Epic puppet fail [12:04:49] PROBLEM - puppet last run on es1008 is CRITICAL: CRITICAL: Epic puppet fail [12:04:49] PROBLEM - puppet last run on xenon is CRITICAL: CRITICAL: Epic puppet fail [12:04:58] PROBLEM - puppet last run on amslvs2 is CRITICAL: CRITICAL: Puppet has 20 failures [12:04:58] PROBLEM - puppet last run on amssq32 is CRITICAL: CRITICAL: Epic puppet fail [12:05:08] PROBLEM - puppet last run on ms-be1006 is CRITICAL: CRITICAL: Epic puppet fail [12:05:08] PROBLEM - puppet last run on helium is CRITICAL: CRITICAL: Epic puppet fail [12:05:08] PROBLEM - puppet last run on db1065 is CRITICAL: CRITICAL: Puppet has 21 failures [12:05:09] PROBLEM - puppet last run on mw1082 is CRITICAL: CRITICAL: Epic puppet fail [12:05:09] PROBLEM - puppet last run on mw1115 is CRITICAL: CRITICAL: Puppet has 58 failures [12:05:09] PROBLEM - puppet last run on mw1106 is CRITICAL: CRITICAL: Puppet has 57 failures [12:05:09] PROBLEM - puppet last run on mw1145 is CRITICAL: CRITICAL: Puppet has 55 failures [12:05:09] PROBLEM - puppet last run on elastic1007 is CRITICAL: CRITICAL: Puppet has 25 failures [12:05:10] PROBLEM - puppet last run on wtp1020 is CRITICAL: CRITICAL: Puppet has 23 failures [12:05:10] PROBLEM - puppet last run on searchidx1001 is CRITICAL: CRITICAL: Puppet has 51 failures [12:05:11] PROBLEM - puppet last run on mw1026 is CRITICAL: CRITICAL: Epic puppet fail [12:05:11] PROBLEM - puppet last run on mw1160 is CRITICAL: CRITICAL: Epic puppet fail [12:05:18] PROBLEM - puppet last run on cp3020 is CRITICAL: CRITICAL: Puppet has 21 failures [12:05:18] PROBLEM - puppet last run on amssq53 is CRITICAL: CRITICAL: Epic puppet fail [12:05:18] PROBLEM - puppet last run on mc1006 is CRITICAL: CRITICAL: Epic puppet fail [12:05:18] PROBLEM - puppet last run on lvs1002 is CRITICAL: CRITICAL: Epic puppet fail [12:05:18] PROBLEM - puppet last run on carbon is CRITICAL: CRITICAL: Puppet has 25 failures [12:05:19] PROBLEM - puppet last run on amssq54 is CRITICAL: CRITICAL: Puppet has 21 failures [12:05:19] PROBLEM - puppet last run on cp1049 is CRITICAL: CRITICAL: Puppet has 23 failures [12:05:28] PROBLEM - puppet last run on mw1063 is CRITICAL: CRITICAL: Puppet has 65 failures [12:05:29] PROBLEM - puppet last run on wtp1006 is CRITICAL: CRITICAL: Puppet has 22 failures [12:05:29] PROBLEM - puppet last run on ms-be1003 is CRITICAL: CRITICAL: Epic puppet fail [12:05:29] PROBLEM - puppet last run on mw1006 is CRITICAL: CRITICAL: Puppet has 51 failures [12:05:29] PROBLEM - puppet last run on elastic1004 is CRITICAL: CRITICAL: Epic puppet fail [12:05:29] PROBLEM - puppet last run on mw1141 is CRITICAL: CRITICAL: Puppet has 70 failures [12:05:38] PROBLEM - puppet last run on mw1197 is CRITICAL: CRITICAL: Puppet has 61 failures [12:05:38] PROBLEM - puppet last run on search1016 is CRITICAL: CRITICAL: Epic puppet fail [12:05:38] PROBLEM - puppet last run on mw1200 is CRITICAL: CRITICAL: Puppet has 63 failures [12:05:38] PROBLEM - puppet last run on mw1009 is CRITICAL: CRITICAL: Epic puppet fail [12:05:39] PROBLEM - puppet last run on potassium is CRITICAL: CRITICAL: Puppet has 18 failures [12:05:39] PROBLEM - puppet last run on ssl1002 is CRITICAL: CRITICAL: Epic puppet fail [12:05:48] PROBLEM - puppet last run on db1073 is CRITICAL: CRITICAL: Epic puppet fail [12:05:48] PROBLEM - puppet last run on mw1187 is CRITICAL: CRITICAL: Puppet has 66 failures [12:05:48] PROBLEM - puppet last run on mw1174 is CRITICAL: CRITICAL: Puppet has 50 failures [12:05:48] PROBLEM - puppet last run on analytics1041 is CRITICAL: CRITICAL: Puppet has 18 failures [12:05:48] PROBLEM - puppet last run on analytics1040 is CRITICAL: CRITICAL: Puppet has 23 failures [12:05:48] PROBLEM - puppet last run on cp1039 is CRITICAL: CRITICAL: Epic puppet fail [12:05:48] PROBLEM - puppet last run on search1010 is CRITICAL: CRITICAL: Epic puppet fail [12:05:49] PROBLEM - puppet last run on db1031 is CRITICAL: CRITICAL: Puppet has 23 failures [12:05:49] PROBLEM - puppet last run on elastic1012 is CRITICAL: CRITICAL: Epic puppet fail [12:05:50] PROBLEM - puppet last run on mw1003 is CRITICAL: CRITICAL: Epic puppet fail [12:05:50] PROBLEM - puppet last run on analytics1020 is CRITICAL: CRITICAL: Puppet has 23 failures [12:05:58] PROBLEM - puppet last run on amssq61 is CRITICAL: CRITICAL: Puppet has 25 failures [12:05:58] PROBLEM - puppet last run on cp3016 is CRITICAL: CRITICAL: Epic puppet fail [12:05:58] PROBLEM - puppet last run on mw1069 is CRITICAL: CRITICAL: Epic puppet fail [12:06:08] PROBLEM - puppet last run on analytics1025 is CRITICAL: CRITICAL: Puppet has 24 failures [12:06:08] PROBLEM - puppet last run on mw1068 is CRITICAL: CRITICAL: Epic puppet fail [12:06:08] PROBLEM - puppet last run on mw1100 is CRITICAL: CRITICAL: Epic puppet fail [12:06:08] PROBLEM - puppet last run on mc1002 is CRITICAL: CRITICAL: Epic puppet fail [12:06:09] PROBLEM - puppet last run on db1022 is CRITICAL: CRITICAL: Puppet has 26 failures [12:06:09] PROBLEM - puppet last run on elastic1008 is CRITICAL: CRITICAL: Epic puppet fail [12:06:09] PROBLEM - puppet last run on lvs1005 is CRITICAL: CRITICAL: Epic puppet fail [12:06:09] PROBLEM - puppet last run on ms1004 is CRITICAL: CRITICAL: Epic puppet fail [12:06:09] PROBLEM - puppet last run on mw1060 is CRITICAL: CRITICAL: Epic puppet fail [12:06:10] PROBLEM - puppet last run on db74 is CRITICAL: CRITICAL: Puppet has 16 failures [12:06:15] (03CR) 10Anomie: "@Ricordisamoa: -1ing this patch that fixes an actual bug as a protest over political issues is not constructive. What you demand is not go" [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/154376 (https://bugzilla.wikimedia.org/69640) (owner: 10Anomie) [12:06:18] PROBLEM - puppet last run on cp3003 is CRITICAL: CRITICAL: Epic puppet fail [12:06:18] PROBLEM - puppet last run on analytics1035 is CRITICAL: CRITICAL: Epic puppet fail [12:06:18] PROBLEM - puppet last run on mw1120 is CRITICAL: CRITICAL: Epic puppet fail [12:06:18] PROBLEM - puppet last run on mw1099 is CRITICAL: CRITICAL: Epic puppet fail [12:06:28] PROBLEM - puppet last run on db1059 is CRITICAL: CRITICAL: Epic puppet fail [12:06:28] PROBLEM - puppet last run on db1002 is CRITICAL: CRITICAL: Epic puppet fail [12:06:29] PROBLEM - puppet last run on snapshot1003 is CRITICAL: CRITICAL: Puppet has 57 failures [12:06:38] PROBLEM - puppet last run on mw1088 is CRITICAL: CRITICAL: Puppet has 50 failures [12:06:38] PROBLEM - puppet last run on mw1205 is CRITICAL: CRITICAL: Puppet has 66 failures [12:06:38] PROBLEM - puppet last run on db1066 is CRITICAL: CRITICAL: Epic puppet fail [12:06:48] PROBLEM - puppet last run on mw1176 is CRITICAL: CRITICAL: Puppet has 65 failures [12:06:48] PROBLEM - puppet last run on mw1046 is CRITICAL: CRITICAL: Puppet has 74 failures [12:06:48] PROBLEM - puppet last run on mw1065 is CRITICAL: CRITICAL: Epic puppet fail [12:06:48] PROBLEM - puppet last run on cp1055 is CRITICAL: CRITICAL: Puppet has 27 failures [12:06:49] PROBLEM - puppet last run on db1050 is CRITICAL: CRITICAL: Puppet has 10 failures [12:06:49] PROBLEM - puppet last run on mw1117 is CRITICAL: CRITICAL: Epic puppet fail [12:06:49] PROBLEM - puppet last run on elastic1001 is CRITICAL: CRITICAL: Puppet has 23 failures [12:06:49] PROBLEM - puppet last run on analytics1030 is CRITICAL: CRITICAL: Epic puppet fail [12:06:50] PROBLEM - puppet last run on lead is CRITICAL: CRITICAL: Epic puppet fail [12:06:50] PROBLEM - puppet last run on cp4008 is CRITICAL: CRITICAL: Epic puppet fail [12:06:58] PROBLEM - puppet last run on lvs3001 is CRITICAL: CRITICAL: Puppet has 18 failures [12:06:58] PROBLEM - puppet last run on mw1150 is CRITICAL: CRITICAL: Puppet has 66 failures [12:06:58] PROBLEM - puppet last run on cp3014 is CRITICAL: CRITICAL: Epic puppet fail [12:07:08] PROBLEM - puppet last run on mw1164 is CRITICAL: CRITICAL: Puppet has 67 failures [12:07:08] PROBLEM - puppet last run on mw1217 is CRITICAL: CRITICAL: Puppet has 62 failures [12:07:09] PROBLEM - puppet last run on mw1170 is CRITICAL: CRITICAL: Epic puppet fail [12:07:09] PROBLEM - puppet last run on mw1166 is CRITICAL: CRITICAL: Epic puppet fail [12:07:09] PROBLEM - puppet last run on mw1061 is CRITICAL: CRITICAL: Epic puppet fail [12:07:09] PROBLEM - puppet last run on db1046 is CRITICAL: CRITICAL: Epic puppet fail [12:07:09] PROBLEM - puppet last run on mc1003 is CRITICAL: CRITICAL: Puppet has 16 failures [12:07:10] PROBLEM - puppet last run on wtp1016 is CRITICAL: CRITICAL: Puppet has 25 failures [12:07:10] PROBLEM - puppet last run on ms-fe1001 is CRITICAL: CRITICAL: Puppet has 27 failures [12:07:18] PROBLEM - puppet last run on amssq35 is CRITICAL: CRITICAL: Puppet has 24 failures [12:07:18] PROBLEM - puppet last run on mw1119 is CRITICAL: CRITICAL: Epic puppet fail [12:07:18] PROBLEM - puppet last run on mw1189 is CRITICAL: CRITICAL: Puppet has 63 failures [12:07:18] PROBLEM - puppet last run on mw1153 is CRITICAL: CRITICAL: Puppet has 64 failures [12:07:28] PROBLEM - puppet last run on mw1123 is CRITICAL: CRITICAL: Epic puppet fail [12:07:29] PROBLEM - puppet last run on ms-fe1004 is CRITICAL: CRITICAL: Puppet has 21 failures [12:07:29] PROBLEM - puppet last run on cp1056 is CRITICAL: CRITICAL: Puppet has 23 failures [12:07:29] PROBLEM - puppet last run on labsdb1003 is CRITICAL: CRITICAL: Epic puppet fail [12:07:29] PROBLEM - puppet last run on mw1092 is CRITICAL: CRITICAL: Epic puppet fail [12:07:38] PROBLEM - puppet last run on mw1008 is CRITICAL: CRITICAL: Puppet has 66 failures [12:07:39] PROBLEM - puppet last run on iron is CRITICAL: CRITICAL: Puppet has 28 failures [12:07:39] PROBLEM - puppet last run on mw1173 is CRITICAL: CRITICAL: Puppet has 66 failures [12:07:48] PROBLEM - puppet last run on mw1175 is CRITICAL: CRITICAL: Epic puppet fail [12:07:48] PROBLEM - puppet last run on mw1213 is CRITICAL: CRITICAL: Epic puppet fail [12:07:49] PROBLEM - puppet last run on mw1042 is CRITICAL: CRITICAL: Puppet has 56 failures [12:07:49] PROBLEM - puppet last run on db1040 is CRITICAL: CRITICAL: Puppet has 22 failures [12:07:49] PROBLEM - puppet last run on mw1054 is CRITICAL: CRITICAL: Epic puppet fail [12:07:49] PROBLEM - puppet last run on db1015 is CRITICAL: CRITICAL: Puppet has 18 failures [12:07:49] PROBLEM - puppet last run on db1021 is CRITICAL: CRITICAL: Epic puppet fail [12:07:49] PROBLEM - puppet last run on mw1144 is CRITICAL: CRITICAL: Puppet has 60 failures [12:07:49] PROBLEM - puppet last run on cp4004 is CRITICAL: CRITICAL: Epic puppet fail [12:07:50] PROBLEM - puppet last run on cp4014 is CRITICAL: CRITICAL: Epic puppet fail [12:07:58] PROBLEM - puppet last run on mw1052 is CRITICAL: CRITICAL: Puppet has 65 failures [12:07:58] PROBLEM - puppet last run on amssq60 is CRITICAL: CRITICAL: Epic puppet fail [12:08:08] PROBLEM - puppet last run on db1067 is CRITICAL: CRITICAL: Puppet has 22 failures [12:08:08] PROBLEM - puppet last run on analytics1010 is CRITICAL: CRITICAL: Epic puppet fail [12:08:08] PROBLEM - puppet last run on gallium is CRITICAL: CRITICAL: Epic puppet fail [12:08:08] PROBLEM - puppet last run on mw1172 is CRITICAL: CRITICAL: Epic puppet fail [12:08:08] PROBLEM - puppet last run on db1042 is CRITICAL: CRITICAL: Epic puppet fail [12:08:08] PROBLEM - puppet last run on logstash1002 is CRITICAL: CRITICAL: Puppet has 24 failures [12:08:09] PROBLEM - puppet last run on db1018 is CRITICAL: CRITICAL: Puppet has 22 failures [12:08:09] PROBLEM - puppet last run on search1018 is CRITICAL: CRITICAL: Puppet has 33 failures [12:08:09] PROBLEM - puppet last run on mw1055 is CRITICAL: CRITICAL: Epic puppet fail [12:08:10] PROBLEM - puppet last run on db1028 is CRITICAL: CRITICAL: Puppet has 21 failures [12:08:10] PROBLEM - puppet last run on cp4003 is CRITICAL: CRITICAL: Puppet has 25 failures [12:08:18] PROBLEM - puppet last run on cp1061 is CRITICAL: CRITICAL: Puppet has 19 failures [12:08:18] PROBLEM - puppet last run on mw1114 is CRITICAL: CRITICAL: Epic puppet fail [12:08:18] PROBLEM - puppet last run on amssq48 is CRITICAL: CRITICAL: Epic puppet fail [12:08:18] PROBLEM - puppet last run on amssq55 is CRITICAL: CRITICAL: Epic puppet fail [12:08:18] PROBLEM - puppet last run on mw1011 is CRITICAL: CRITICAL: Epic puppet fail [12:08:29] PROBLEM - puppet last run on search1001 is CRITICAL: CRITICAL: Puppet has 41 failures [12:08:29] PROBLEM - puppet last run on db1034 is CRITICAL: CRITICAL: Puppet has 25 failures [12:08:38] PROBLEM - puppet last run on mw1025 is CRITICAL: CRITICAL: Puppet has 66 failures [12:08:38] PROBLEM - puppet last run on mw1118 is CRITICAL: CRITICAL: Puppet has 70 failures [12:08:39] PROBLEM - puppet last run on holmium is CRITICAL: CRITICAL: Puppet has 30 failures [12:08:39] PROBLEM - puppet last run on mw1002 is CRITICAL: CRITICAL: Epic puppet fail [12:08:48] PROBLEM - puppet last run on mw1044 is CRITICAL: CRITICAL: Epic puppet fail [12:08:48] PROBLEM - puppet last run on virt1004 is CRITICAL: CRITICAL: Epic puppet fail [12:08:48] PROBLEM - puppet last run on mw1039 is CRITICAL: CRITICAL: Puppet has 66 failures [12:08:48] PROBLEM - puppet last run on dbproxy1001 is CRITICAL: CRITICAL: Epic puppet fail [12:08:49] PROBLEM - puppet last run on silver is CRITICAL: CRITICAL: Epic puppet fail [12:08:49] PROBLEM - puppet last run on mw1206 is CRITICAL: CRITICAL: Epic puppet fail [12:08:58] PROBLEM - puppet last run on amssq46 is CRITICAL: CRITICAL: Puppet has 21 failures [12:08:58] PROBLEM - puppet last run on amssq34 is CRITICAL: CRITICAL: Epic puppet fail [12:08:59] PROBLEM - puppet last run on cp3010 is CRITICAL: CRITICAL: Epic puppet fail [12:09:08] PROBLEM - puppet last run on db1051 is CRITICAL: CRITICAL: Puppet has 21 failures [12:09:08] PROBLEM - puppet last run on analytics1038 is CRITICAL: CRITICAL: Epic puppet fail [12:09:08] PROBLEM - puppet last run on search1007 is CRITICAL: CRITICAL: Puppet has 42 failures [12:09:09] PROBLEM - puppet last run on mw1162 is CRITICAL: CRITICAL: Epic puppet fail [12:09:09] PROBLEM - puppet last run on db1023 is CRITICAL: CRITICAL: Puppet has 21 failures [12:09:09] PROBLEM - puppet last run on mw1076 is CRITICAL: CRITICAL: Epic puppet fail [12:09:09] PROBLEM - puppet last run on cp4001 is CRITICAL: CRITICAL: Epic puppet fail [12:09:10] PROBLEM - puppet last run on cp4019 is CRITICAL: CRITICAL: Epic puppet fail [12:09:18] PROBLEM - puppet last run on mw1126 is CRITICAL: CRITICAL: Puppet has 72 failures [12:09:18] PROBLEM - puppet last run on es1007 is CRITICAL: CRITICAL: Epic puppet fail [12:09:18] PROBLEM - puppet last run on tin is CRITICAL: CRITICAL: Puppet has 58 failures [12:09:18] PROBLEM - puppet last run on cp4018 is CRITICAL: CRITICAL: Epic puppet fail [12:09:29] PROBLEM - puppet last run on wtp1012 is CRITICAL: CRITICAL: Epic puppet fail [12:09:29] PROBLEM - puppet last run on pc1002 is CRITICAL: CRITICAL: Epic puppet fail [12:09:38] PROBLEM - puppet last run on analytics1013 is CRITICAL: CRITICAL: Epic puppet fail [12:09:38] PROBLEM - puppet last run on labstore1001 is CRITICAL: CRITICAL: Epic puppet fail [12:09:38] PROBLEM - puppet last run on polonium is CRITICAL: CRITICAL: Puppet has 20 failures [12:09:38] PROBLEM - puppet last run on dataset1001 is CRITICAL: CRITICAL: Epic puppet fail [12:09:48] PROBLEM - puppet last run on mw1190 is CRITICAL: CRITICAL: Epic puppet fail [12:09:48] PROBLEM - puppet last run on mw1177 is CRITICAL: CRITICAL: Puppet has 61 failures [12:09:48] PROBLEM - puppet last run on mc1014 is CRITICAL: CRITICAL: Epic puppet fail [12:09:48] PROBLEM - puppet last run on virt1006 is CRITICAL: CRITICAL: Puppet has 22 failures [12:09:48] PROBLEM - puppet last run on mw1180 is CRITICAL: CRITICAL: Epic puppet fail [12:09:48] PROBLEM - puppet last run on labnet1001 is CRITICAL: CRITICAL: Puppet has 22 failures [12:09:48] PROBLEM - puppet last run on wtp1005 is CRITICAL: CRITICAL: Epic puppet fail [12:09:49] PROBLEM - puppet last run on virt1001 is CRITICAL: CRITICAL: Epic puppet fail [12:09:49] PROBLEM - puppet last run on ssl1005 is CRITICAL: CRITICAL: Epic puppet fail [12:09:50] PROBLEM - puppet last run on cp1058 is CRITICAL: CRITICAL: Epic puppet fail [12:09:50] PROBLEM - puppet last run on db1043 is CRITICAL: CRITICAL: Epic puppet fail [12:09:51] PROBLEM - puppet last run on antimony is CRITICAL: CRITICAL: Epic puppet fail [12:09:51] PROBLEM - puppet last run on snapshot1002 is CRITICAL: CRITICAL: Epic puppet fail [12:09:58] PROBLEM - puppet last run on ms-be3001 is CRITICAL: CRITICAL: Epic puppet fail [12:09:58] PROBLEM - puppet last run on amssq47 is CRITICAL: CRITICAL: Puppet has 26 failures [12:09:58] PROBLEM - puppet last run on amslvs1 is CRITICAL: CRITICAL: Puppet has 25 failures [12:09:58] PROBLEM - puppet last run on mw1129 is CRITICAL: CRITICAL: Puppet has 63 failures [12:10:08] PROBLEM - puppet last run on mw1125 is CRITICAL: CRITICAL: Epic puppet fail [12:10:08] PROBLEM - puppet last run on mw1211 is CRITICAL: CRITICAL: Puppet has 62 failures [12:10:08] PROBLEM - puppet last run on mc1012 is CRITICAL: CRITICAL: Puppet has 21 failures [12:10:08] PROBLEM - puppet last run on mw1195 is CRITICAL: CRITICAL: Puppet has 64 failures [12:10:08] PROBLEM - puppet last run on cp1046 is CRITICAL: CRITICAL: Puppet has 19 failures [12:10:08] PROBLEM - puppet last run on virt1003 is CRITICAL: CRITICAL: Puppet has 21 failures [12:10:09] PROBLEM - puppet last run on mw1151 is CRITICAL: CRITICAL: Epic puppet fail [12:10:09] PROBLEM - puppet last run on db1036 is CRITICAL: CRITICAL: Epic puppet fail [12:10:10] PROBLEM - puppet last run on db1003 is CRITICAL: CRITICAL: Puppet has 26 failures [12:10:10] PROBLEM - puppet last run on mw1208 is CRITICAL: CRITICAL: Puppet has 64 failures [12:10:11] PROBLEM - puppet last run on db1016 is CRITICAL: CRITICAL: Puppet has 17 failures [12:10:11] PROBLEM - puppet last run on mw1133 is CRITICAL: CRITICAL: Epic puppet fail [12:10:12] PROBLEM - puppet last run on snapshot1001 is CRITICAL: CRITICAL: Puppet has 51 failures [12:10:12] PROBLEM - puppet last run on analytics1016 is CRITICAL: CRITICAL: Puppet has 19 failures [12:10:13] PROBLEM - puppet last run on mw1014 is CRITICAL: CRITICAL: Epic puppet fail [12:10:13] PROBLEM - puppet last run on ruthenium is CRITICAL: CRITICAL: Puppet has 19 failures [12:10:18] PROBLEM - puppet last run on amssq36 is CRITICAL: CRITICAL: Epic puppet fail [12:10:29] PROBLEM - puppet last run on db1060 is CRITICAL: CRITICAL: Puppet has 21 failures [12:10:29] PROBLEM - puppet last run on db1052 is CRITICAL: CRITICAL: Puppet has 18 failures [12:10:29] PROBLEM - puppet last run on gadolinium is CRITICAL: CRITICAL: Puppet has 28 failures [12:10:29] PROBLEM - puppet last run on search1017 is CRITICAL: CRITICAL: Epic puppet fail [12:10:29] PROBLEM - puppet last run on cp1048 is CRITICAL: CRITICAL: Epic puppet fail [12:10:29] PROBLEM - puppet last run on mc1005 is CRITICAL: CRITICAL: Puppet has 19 failures [12:10:29] PROBLEM - puppet last run on mw1049 is CRITICAL: CRITICAL: Epic puppet fail [12:10:38] PROBLEM - puppet last run on thallium is CRITICAL: CRITICAL: Epic puppet fail [12:10:39] PROBLEM - puppet last run on mw1004 is CRITICAL: CRITICAL: Epic puppet fail [12:10:39] PROBLEM - puppet last run on analytics1022 is CRITICAL: CRITICAL: Epic puppet fail [12:10:39] PROBLEM - puppet last run on virt1007 is CRITICAL: CRITICAL: Epic puppet fail [12:10:48] PROBLEM - puppet last run on mw1181 is CRITICAL: CRITICAL: Epic puppet fail [12:10:49] PROBLEM - puppet last run on mw1149 is CRITICAL: CRITICAL: Puppet has 66 failures [12:10:49] PROBLEM - puppet last run on db1048 is CRITICAL: CRITICAL: Epic puppet fail [12:10:49] PROBLEM - puppet last run on search1005 is CRITICAL: CRITICAL: Epic puppet fail [12:10:49] PROBLEM - puppet last run on mw1051 is CRITICAL: CRITICAL: Puppet has 66 failures [12:10:49] PROBLEM - puppet last run on cp1050 is CRITICAL: CRITICAL: Puppet has 24 failures [12:10:49] PROBLEM - puppet last run on db1039 is CRITICAL: CRITICAL: Epic puppet fail [12:10:50] PROBLEM - puppet last run on lvs4003 is CRITICAL: CRITICAL: Puppet has 14 failures [12:10:50] PROBLEM - puppet last run on bast4001 is CRITICAL: CRITICAL: Epic puppet fail [12:10:58] PROBLEM - puppet last run on amssq40 is CRITICAL: CRITICAL: Epic puppet fail [12:10:58] PROBLEM - puppet last run on ms-fe3002 is CRITICAL: CRITICAL: Puppet has 15 failures [12:10:58] PROBLEM - puppet last run on mw1084 is CRITICAL: CRITICAL: Epic puppet fail [12:10:58] PROBLEM - puppet last run on mw1098 is CRITICAL: CRITICAL: Puppet has 58 failures [12:11:08] PROBLEM - puppet last run on db1071 is CRITICAL: CRITICAL: Puppet has 13 failures [12:11:08] PROBLEM - puppet last run on labsdb1006 is CRITICAL: CRITICAL: Epic puppet fail [12:11:08] PROBLEM - puppet last run on argon is CRITICAL: CRITICAL: Epic puppet fail [12:11:08] PROBLEM - puppet last run on mw1183 is CRITICAL: CRITICAL: Epic puppet fail [12:11:08] PROBLEM - puppet last run on mw1057 is CRITICAL: CRITICAL: Epic puppet fail [12:11:08] PROBLEM - puppet last run on mw1165 is CRITICAL: CRITICAL: Epic puppet fail [12:11:09] PROBLEM - puppet last run on rdb1001 is CRITICAL: CRITICAL: Puppet has 16 failures [12:11:09] PROBLEM - puppet last run on mw1111 is CRITICAL: CRITICAL: Puppet has 67 failures [12:11:09] PROBLEM - puppet last run on snapshot1004 is CRITICAL: CRITICAL: Epic puppet fail [12:11:10] PROBLEM - puppet last run on mw1079 is CRITICAL: CRITICAL: Puppet has 54 failures [12:11:10] PROBLEM - puppet last run on tungsten is CRITICAL: CRITICAL: Epic puppet fail [12:11:11] PROBLEM - puppet last run on search1002 is CRITICAL: CRITICAL: Puppet has 57 failures [12:11:11] PROBLEM - puppet last run on mw1056 is CRITICAL: CRITICAL: Epic puppet fail [12:11:12] PROBLEM - puppet last run on elastic1006 is CRITICAL: CRITICAL: Puppet has 25 failures [12:11:12] PROBLEM - puppet last run on mw1146 is CRITICAL: CRITICAL: Epic puppet fail [12:11:13] PROBLEM - puppet last run on analytics1023 is CRITICAL: CRITICAL: Epic puppet fail [12:11:13] PROBLEM - puppet last run on hafnium is CRITICAL: CRITICAL: Epic puppet fail [12:11:14] PROBLEM - puppet last run on ssl1009 is CRITICAL: CRITICAL: Puppet has 24 failures [12:11:14] PROBLEM - puppet last run on search1023 is CRITICAL: CRITICAL: Epic puppet fail [12:11:18] PROBLEM - puppet last run on hooft is CRITICAL: CRITICAL: Epic puppet fail [12:11:18] PROBLEM - puppet last run on amssq51 is CRITICAL: CRITICAL: Puppet has 23 failures [12:11:18] PROBLEM - puppet last run on ms-be3002 is CRITICAL: CRITICAL: Puppet has 28 failures [12:11:18] PROBLEM - puppet last run on db1026 is CRITICAL: CRITICAL: Puppet has 19 failures [12:11:18] PROBLEM - puppet last run on rhenium is CRITICAL: CRITICAL: Epic puppet fail [12:11:18] PROBLEM - puppet last run on mw1168 is CRITICAL: CRITICAL: Puppet has 63 failures [12:11:18] PROBLEM - puppet last run on mw1029 is CRITICAL: CRITICAL: Epic puppet fail [12:11:19] PROBLEM - puppet last run on labmon1001 is CRITICAL: CRITICAL: Puppet has 16 failures [12:11:19] PROBLEM - puppet last run on cp1063 is CRITICAL: CRITICAL: Puppet has 20 failures [12:11:20] PROBLEM - puppet last run on lvs3004 is CRITICAL: CRITICAL: Puppet has 24 failures [12:11:28] PROBLEM - puppet last run on wtp1018 is CRITICAL: CRITICAL: Epic puppet fail [12:11:29] PROBLEM - puppet last run on mw1159 is CRITICAL: CRITICAL: Puppet has 59 failures [12:11:29] PROBLEM - puppet last run on ms-be1012 is CRITICAL: CRITICAL: Puppet has 23 failures [12:11:29] PROBLEM - puppet last run on elastic1014 is CRITICAL: CRITICAL: Epic puppet fail [12:11:29] PROBLEM - puppet last run on mw1202 is CRITICAL: CRITICAL: Puppet has 48 failures [12:11:38] PROBLEM - puppet last run on wtp1022 is CRITICAL: CRITICAL: Epic puppet fail [12:11:38] PROBLEM - puppet last run on osmium is CRITICAL: CRITICAL: Puppet has 52 failures [12:11:39] PROBLEM - puppet last run on db1069 is CRITICAL: CRITICAL: Puppet has 20 failures [12:11:39] PROBLEM - puppet last run on wtp1002 is CRITICAL: CRITICAL: Epic puppet fail [12:11:48] PROBLEM - puppet last run on oxygen is CRITICAL: CRITICAL: Puppet has 26 failures [12:11:48] PROBLEM - puppet last run on mw1188 is CRITICAL: CRITICAL: Epic puppet fail [12:11:48] PROBLEM - puppet last run on amssq56 is CRITICAL: CRITICAL: Puppet has 23 failures [12:11:48] PROBLEM - puppet last run on elastic1015 is CRITICAL: CRITICAL: Puppet has 23 failures [12:11:48] PROBLEM - puppet last run on mw1156 is CRITICAL: CRITICAL: Epic puppet fail [12:11:48] PROBLEM - puppet last run on ms-be1007 is CRITICAL: CRITICAL: Epic puppet fail [12:11:48] PROBLEM - puppet last run on cp1062 is CRITICAL: CRITICAL: Puppet has 23 failures [12:11:49] PROBLEM - puppet last run on ms-be1008 is CRITICAL: CRITICAL: Puppet has 22 failures [12:11:49] PROBLEM - puppet last run on db1004 is CRITICAL: CRITICAL: Puppet has 23 failures [12:11:50] PROBLEM - puppet last run on mw1116 is CRITICAL: CRITICAL: Epic puppet fail [12:11:58] PROBLEM - puppet last run on db1020 is CRITICAL: CRITICAL: Puppet has 16 failures [12:11:58] PROBLEM - puppet last run on db72 is CRITICAL: CRITICAL: Puppet has 20 failures [12:12:08] PROBLEM - puppet last run on mw1030 is CRITICAL: CRITICAL: Puppet has 68 failures [12:12:08] PROBLEM - puppet last run on mw1034 is CRITICAL: CRITICAL: Puppet has 62 failures [12:12:09] PROBLEM - puppet last run on cp4005 is CRITICAL: CRITICAL: Puppet has 28 failures [12:12:09] PROBLEM - puppet last run on db1055 is CRITICAL: CRITICAL: Epic puppet fail [12:12:09] PROBLEM - puppet last run on search1024 is CRITICAL: CRITICAL: Epic puppet fail [12:12:18] PROBLEM - puppet last run on cp3009 is CRITICAL: CRITICAL: Epic puppet fail [12:12:18] PROBLEM - puppet last run on amssq62 is CRITICAL: CRITICAL: Epic puppet fail [12:12:29] PROBLEM - puppet last run on wtp1023 is CRITICAL: CRITICAL: Epic puppet fail [12:12:29] PROBLEM - puppet last run on mw1210 is CRITICAL: CRITICAL: Epic puppet fail [12:12:29] PROBLEM - puppet last run on rubidium is CRITICAL: CRITICAL: Puppet has 24 failures [12:12:29] PROBLEM - puppet last run on mw1163 is CRITICAL: CRITICAL: Puppet has 61 failures [12:12:38] PROBLEM - puppet last run on db1062 is CRITICAL: CRITICAL: Epic puppet fail [12:12:38] PROBLEM - puppet last run on mw1087 is CRITICAL: CRITICAL: Puppet has 60 failures [12:12:38] PROBLEM - puppet last run on cp1038 is CRITICAL: CRITICAL: Puppet has 28 failures [12:12:38] PROBLEM - puppet last run on analytics1037 is CRITICAL: CRITICAL: Puppet has 22 failures [12:12:39] PROBLEM - puppet last run on db69 is CRITICAL: CRITICAL: Puppet has 19 failures [12:12:39] PROBLEM - puppet last run on wtp1004 is CRITICAL: CRITICAL: Puppet has 23 failures [12:12:48] PROBLEM - puppet last run on lvs1001 is CRITICAL: CRITICAL: Puppet has 20 failures [12:12:48] PROBLEM - puppet last run on mw1171 is CRITICAL: CRITICAL: Epic puppet fail [12:12:48] PROBLEM - puppet last run on mw1212 is CRITICAL: CRITICAL: Puppet has 58 failures [12:12:48] PROBLEM - puppet last run on analytics1032 is CRITICAL: CRITICAL: Puppet has 22 failures [12:12:48] PROBLEM - puppet last run on mw1074 is CRITICAL: CRITICAL: Puppet has 61 failures [12:12:48] PROBLEM - puppet last run on mc1001 is CRITICAL: CRITICAL: Epic puppet fail [12:12:49] PROBLEM - puppet last run on mw1050 is CRITICAL: CRITICAL: Puppet has 52 failures [12:12:58] PROBLEM - puppet last run on ssl3002 is CRITICAL: CRITICAL: Puppet has 19 failures [12:12:58] PROBLEM - puppet last run on ms-be1009 is CRITICAL: CRITICAL: Puppet has 22 failures [12:12:58] PROBLEM - puppet last run on elastic1005 is CRITICAL: CRITICAL: Epic puppet fail [12:12:58] PROBLEM - puppet last run on amssq41 is CRITICAL: CRITICAL: Puppet has 19 failures [12:13:08] PROBLEM - puppet last run on db1054 is CRITICAL: CRITICAL: Epic puppet fail [12:13:08] PROBLEM - puppet last run on mw1081 is CRITICAL: CRITICAL: Puppet has 55 failures [12:13:08] PROBLEM - puppet last run on rdb1002 is CRITICAL: CRITICAL: Epic puppet fail [12:13:08] PROBLEM - puppet last run on protactinium is CRITICAL: CRITICAL: Puppet has 25 failures [12:13:09] PROBLEM - puppet last run on elastic1011 is CRITICAL: CRITICAL: Puppet has 22 failures [12:13:09] PROBLEM - puppet last run on mw1198 is CRITICAL: CRITICAL: Puppet has 61 failures [12:13:09] PROBLEM - puppet last run on stat1002 is CRITICAL: CRITICAL: Puppet has 36 failures [12:13:09] PROBLEM - puppet last run on cp1060 is CRITICAL: CRITICAL: Epic puppet fail [12:13:18] PROBLEM - puppet last run on cp3012 is CRITICAL: CRITICAL: Epic puppet fail [12:13:18] PROBLEM - puppet last run on amssq42 is CRITICAL: CRITICAL: Puppet has 27 failures [12:13:18] PROBLEM - puppet last run on ssl1008 is CRITICAL: CRITICAL: Puppet has 21 failures [12:13:18] PROBLEM - puppet last run on amslvs3 is CRITICAL: CRITICAL: Epic puppet fail [12:13:18] PROBLEM - puppet last run on mw1023 is CRITICAL: CRITICAL: Puppet has 65 failures [12:13:29] PROBLEM - puppet last run on db1001 is CRITICAL: CRITICAL: Epic puppet fail [12:13:29] PROBLEM - puppet last run on mw1097 is CRITICAL: CRITICAL: Puppet has 76 failures [12:13:38] PROBLEM - puppet last run on analytics1014 is CRITICAL: CRITICAL: Puppet has 16 failures [12:13:39] PROBLEM - puppet last run on titanium is CRITICAL: CRITICAL: Puppet has 24 failures [12:13:39] PROBLEM - puppet last run on wtp1003 is CRITICAL: CRITICAL: Epic puppet fail [12:13:48] PROBLEM - puppet last run on wtp1013 is CRITICAL: CRITICAL: Epic puppet fail [12:13:48] PROBLEM - puppet last run on wtp1007 is CRITICAL: CRITICAL: Puppet has 25 failures [12:13:49] PROBLEM - puppet last run on mc1013 is CRITICAL: CRITICAL: Puppet has 21 failures [12:13:49] PROBLEM - puppet last run on mw1001 is CRITICAL: CRITICAL: Epic puppet fail [12:13:49] PROBLEM - puppet last run on wtp1015 is CRITICAL: CRITICAL: Epic puppet fail [12:13:49] PROBLEM - puppet last run on nitrogen is CRITICAL: CRITICAL: Epic puppet fail [12:13:49] PROBLEM - puppet last run on es1002 is CRITICAL: CRITICAL: Puppet has 20 failures [12:13:50] PROBLEM - puppet last run on mc1007 is CRITICAL: CRITICAL: Epic puppet fail [12:13:50] PROBLEM - puppet last run on rcs1001 is CRITICAL: CRITICAL: Epic puppet fail [12:13:51] PROBLEM - puppet last run on rcs1002 is CRITICAL: CRITICAL: Epic puppet fail [12:13:58] PROBLEM - puppet last run on db1044 is CRITICAL: CRITICAL: Puppet has 21 failures [12:14:08] PROBLEM - puppet last run on labsdb1004 is CRITICAL: CRITICAL: Puppet has 24 failures [12:14:08] PROBLEM - puppet last run on db1057 is CRITICAL: CRITICAL: Puppet has 19 failures [12:14:08] PROBLEM - puppet last run on wtp1011 is CRITICAL: CRITICAL: Puppet has 21 failures [12:14:08] PROBLEM - puppet last run on mw1185 is CRITICAL: CRITICAL: Epic puppet fail [12:14:08] PROBLEM - puppet last run on mw1105 is CRITICAL: CRITICAL: Epic puppet fail [12:14:08] PROBLEM - puppet last run on rdb1003 is CRITICAL: CRITICAL: Epic puppet fail [12:14:09] PROBLEM - puppet last run on ssl1007 is CRITICAL: CRITICAL: Epic puppet fail [12:14:09] PROBLEM - puppet last run on mw1148 is CRITICAL: CRITICAL: Puppet has 70 failures [12:14:09] PROBLEM - puppet last run on elastic1002 is CRITICAL: CRITICAL: Puppet has 16 failures [12:14:10] PROBLEM - puppet last run on search1015 is CRITICAL: CRITICAL: Puppet has 50 failures [12:14:10] PROBLEM - puppet last run on mw1122 is CRITICAL: CRITICAL: Epic puppet fail [12:14:18] PROBLEM - puppet last run on mw1022 is CRITICAL: CRITICAL: Epic puppet fail [12:14:18] PROBLEM - puppet last run on ms-fe3001 is CRITICAL: CRITICAL: Epic puppet fail [12:14:18] PROBLEM - puppet last run on mw1043 is CRITICAL: CRITICAL: Epic puppet fail [12:14:28] PROBLEM - puppet last run on db1063 is CRITICAL: CRITICAL: Epic puppet fail [12:14:28] PROBLEM - puppet last run on mw1024 is CRITICAL: CRITICAL: Epic puppet fail [12:14:29] PROBLEM - puppet last run on mw1032 is CRITICAL: CRITICAL: Puppet has 67 failures [12:14:38] PROBLEM - puppet last run on db1070 is CRITICAL: CRITICAL: Puppet has 22 failures [12:14:38] PROBLEM - puppet last run on analytics1011 is CRITICAL: CRITICAL: Puppet has 17 failures [12:14:38] PROBLEM - puppet last run on iodine is CRITICAL: CRITICAL: Epic puppet fail [12:14:38] PROBLEM - puppet last run on ocg1003 is CRITICAL: CRITICAL: Puppet has 19 failures [12:14:39] PROBLEM - puppet last run on db1061 is CRITICAL: CRITICAL: Epic puppet fail [12:14:39] PROBLEM - puppet last run on sanger is CRITICAL: CRITICAL: Epic puppet fail [12:14:48] PROBLEM - puppet last run on db1064 is CRITICAL: CRITICAL: Puppet has 21 failures [12:14:48] PROBLEM - puppet last run on pc1003 is CRITICAL: CRITICAL: Puppet has 27 failures [12:14:48] PROBLEM - puppet last run on elastic1017 is CRITICAL: CRITICAL: Puppet has 26 failures [12:14:48] PROBLEM - puppet last run on stat1003 is CRITICAL: CRITICAL: Epic puppet fail [12:14:48] PROBLEM - puppet last run on db1030 is CRITICAL: CRITICAL: Puppet has 23 failures [12:14:48] PROBLEM - puppet last run on db1006 is CRITICAL: CRITICAL: Epic puppet fail [12:14:48] PROBLEM - puppet last run on mw1139 is CRITICAL: CRITICAL: Epic puppet fail [12:14:58] PROBLEM - puppet last run on amssq38 is CRITICAL: CRITICAL: Puppet has 23 failures [12:15:08] PROBLEM - puppet last run on mw1209 is CRITICAL: CRITICAL: Epic puppet fail [12:15:08] PROBLEM - puppet last run on db1038 is CRITICAL: CRITICAL: Epic puppet fail [12:15:08] PROBLEM - puppet last run on mw1219 is CRITICAL: CRITICAL: Epic puppet fail [12:15:08] PROBLEM - puppet last run on calcium is CRITICAL: CRITICAL: Puppet has 18 failures [12:15:08] PROBLEM - puppet last run on mw1093 is CRITICAL: CRITICAL: Epic puppet fail [12:15:08] PROBLEM - puppet last run on db1047 is CRITICAL: CRITICAL: Epic puppet fail [12:15:08] PROBLEM - puppet last run on mw1186 is CRITICAL: CRITICAL: Puppet has 73 failures [12:15:09] PROBLEM - puppet last run on mw1077 is CRITICAL: CRITICAL: Puppet has 59 failures [12:15:09] PROBLEM - puppet last run on es1010 is CRITICAL: CRITICAL: Puppet has 18 failures [12:15:10] PROBLEM - puppet last run on ms-be1011 is CRITICAL: CRITICAL: Epic puppet fail [12:15:10] PROBLEM - puppet last run on search1013 is CRITICAL: CRITICAL: Puppet has 41 failures [12:15:11] PROBLEM - puppet last run on search1006 is CRITICAL: CRITICAL: Epic puppet fail [12:15:18] PROBLEM - puppet last run on cp3004 is CRITICAL: CRITICAL: Puppet has 20 failures [12:15:18] PROBLEM - puppet last run on mw1108 is CRITICAL: CRITICAL: Puppet has 59 failures [12:15:18] PROBLEM - puppet last run on mercury is CRITICAL: CRITICAL: Epic puppet fail [12:15:18] PROBLEM - puppet last run on ssl1001 is CRITICAL: CRITICAL: Puppet has 22 failures [12:15:29] PROBLEM - puppet last run on analytics1027 is CRITICAL: CRITICAL: Puppet has 17 failures [12:15:29] PROBLEM - puppet last run on mw1167 is CRITICAL: CRITICAL: Puppet has 64 failures [12:15:29] PROBLEM - puppet last run on terbium is CRITICAL: CRITICAL: Epic puppet fail [12:15:29] PROBLEM - puppet last run on db1033 is CRITICAL: CRITICAL: Puppet has 20 failures [12:15:38] PROBLEM - puppet last run on db1011 is CRITICAL: CRITICAL: Epic puppet fail [12:15:39] PROBLEM - puppet last run on mw1121 is CRITICAL: CRITICAL: Puppet has 56 failures [12:15:39] PROBLEM - puppet last run on mw1142 is CRITICAL: CRITICAL: Puppet has 73 failures [12:15:39] PROBLEM - puppet last run on db60 is CRITICAL: CRITICAL: Epic puppet fail [12:15:48] PROBLEM - puppet last run on db1072 is CRITICAL: CRITICAL: Epic puppet fail [12:15:48] PROBLEM - puppet last run on analytics1004 is CRITICAL: CRITICAL: Epic puppet fail [12:15:48] PROBLEM - puppet last run on mw1215 is CRITICAL: CRITICAL: Epic puppet fail [12:15:48] PROBLEM - puppet last run on cp1067 is CRITICAL: CRITICAL: Epic puppet fail [12:15:48] PROBLEM - puppet last run on mw1064 is CRITICAL: CRITICAL: Epic puppet fail [12:15:48] PROBLEM - puppet last run on mw1152 is CRITICAL: CRITICAL: Epic puppet fail [12:15:48] PROBLEM - puppet last run on search1022 is CRITICAL: CRITICAL: Epic puppet fail [12:15:49] PROBLEM - puppet last run on cp1052 is CRITICAL: CRITICAL: Epic puppet fail [12:15:49] PROBLEM - puppet last run on stat1001 is CRITICAL: CRITICAL: Epic puppet fail [12:15:50] PROBLEM - puppet last run on cp1040 is CRITICAL: CRITICAL: Epic puppet fail [12:15:50] PROBLEM - puppet last run on mw1091 is CRITICAL: CRITICAL: Epic puppet fail [12:15:51] PROBLEM - puppet last run on mw1204 is CRITICAL: CRITICAL: Epic puppet fail [12:15:51] PROBLEM - puppet last run on mw1010 is CRITICAL: CRITICAL: Epic puppet fail [12:15:52] PROBLEM - puppet last run on dbstore1002 is CRITICAL: CRITICAL: Epic puppet fail [12:15:52] PROBLEM - puppet last run on cp4002 is CRITICAL: CRITICAL: Epic puppet fail [12:15:58] PROBLEM - puppet last run on ssl3001 is CRITICAL: CRITICAL: Epic puppet fail [12:15:58] PROBLEM - puppet last run on mw1033 is CRITICAL: CRITICAL: Puppet has 67 failures [12:15:58] PROBLEM - puppet last run on mw1016 is CRITICAL: CRITICAL: Epic puppet fail [12:15:58] PROBLEM - puppet last run on lvs3003 is CRITICAL: CRITICAL: Epic puppet fail [12:16:08] PROBLEM - puppet last run on analytics1003 is CRITICAL: CRITICAL: Epic puppet fail [12:16:08] PROBLEM - puppet last run on bast1001 is CRITICAL: CRITICAL: Epic puppet fail [12:16:08] PROBLEM - puppet last run on search1011 is CRITICAL: CRITICAL: Puppet has 43 failures [12:16:08] PROBLEM - puppet last run on virt1008 is CRITICAL: CRITICAL: Epic puppet fail [12:16:09] PROBLEM - puppet last run on elastic1003 is CRITICAL: CRITICAL: Epic puppet fail [12:16:09] PROBLEM - puppet last run on analytics1028 is CRITICAL: CRITICAL: Puppet has 19 failures [12:16:09] PROBLEM - puppet last run on cp1037 is CRITICAL: CRITICAL: Puppet has 32 failures [12:16:09] PROBLEM - puppet last run on ms-be1002 is CRITICAL: CRITICAL: Puppet has 26 failures [12:16:09] PROBLEM - puppet last run on virt1000 is CRITICAL: CRITICAL: Puppet has 49 failures [12:16:10] PROBLEM - puppet last run on mw1203 is CRITICAL: CRITICAL: Epic puppet fail [12:16:10] PROBLEM - puppet last run on cp4009 is CRITICAL: CRITICAL: Epic puppet fail [12:16:11] PROBLEM - puppet last run on ssl1006 is CRITICAL: CRITICAL: Puppet has 19 failures [12:16:11] PROBLEM - puppet last run on mw1090 is CRITICAL: CRITICAL: Epic puppet fail [12:16:12] PROBLEM - puppet last run on mw1201 is CRITICAL: CRITICAL: Puppet has 62 failures [12:16:18] PROBLEM - puppet last run on amssq31 is CRITICAL: CRITICAL: Puppet has 21 failures [12:16:18] PROBLEM - puppet last run on amssq39 is CRITICAL: CRITICAL: Epic puppet fail [12:16:18] PROBLEM - puppet last run on cp3018 is CRITICAL: CRITICAL: Epic puppet fail [12:16:18] PROBLEM - puppet last run on cp4020 is CRITICAL: CRITICAL: Puppet has 26 failures [12:16:18] PROBLEM - puppet last run on wtp1010 is CRITICAL: CRITICAL: Puppet has 22 failures [12:16:28] PROBLEM - puppet last run on copper is CRITICAL: CRITICAL: Puppet has 18 failures [12:16:29] PROBLEM - puppet last run on cp1044 is CRITICAL: CRITICAL: Epic puppet fail [12:16:29] PROBLEM - puppet last run on strontium is CRITICAL: CRITICAL: Puppet has 30 failures [12:16:38] PROBLEM - puppet last run on mw1113 is CRITICAL: CRITICAL: Epic puppet fail [12:16:38] PROBLEM - puppet last run on mw1107 is CRITICAL: CRITICAL: Puppet has 65 failures [12:16:38] PROBLEM - puppet last run on mw1086 is CRITICAL: CRITICAL: Puppet has 56 failures [12:16:38] PROBLEM - puppet last run on mw1027 is CRITICAL: CRITICAL: Epic puppet fail [12:16:39] PROBLEM - puppet last run on wtp1001 is CRITICAL: CRITICAL: Epic puppet fail [12:16:48] PROBLEM - puppet last run on mw1104 is CRITICAL: CRITICAL: Epic puppet fail [12:16:48] PROBLEM - puppet last run on dysprosium is CRITICAL: CRITICAL: Epic puppet fail [12:16:49] PROBLEM - puppet last run on fluorine is CRITICAL: CRITICAL: Puppet has 68 failures [12:16:49] PROBLEM - puppet last run on ms-fe1002 is CRITICAL: CRITICAL: Epic puppet fail [12:17:08] PROBLEM - puppet last run on cp1068 is CRITICAL: CRITICAL: Epic puppet fail [12:17:08] PROBLEM - puppet last run on wtp1008 is CRITICAL: CRITICAL: Epic puppet fail [12:17:08] PROBLEM - puppet last run on chromium is CRITICAL: CRITICAL: Epic puppet fail [12:17:08] PROBLEM - puppet last run on mw1110 is CRITICAL: CRITICAL: Puppet has 58 failures [12:17:09] PROBLEM - puppet last run on rbf1001 is CRITICAL: CRITICAL: Puppet has 18 failures [12:17:09] PROBLEM - puppet last run on ms-be1015 is CRITICAL: CRITICAL: Epic puppet fail [12:17:09] PROBLEM - puppet last run on db1035 is CRITICAL: CRITICAL: Epic puppet fail [12:17:09] PROBLEM - puppet last run on db1056 is CRITICAL: CRITICAL: Puppet has 20 failures [12:17:10] PROBLEM - puppet last run on search1004 is CRITICAL: CRITICAL: Puppet has 46 failures [12:17:10] PROBLEM - puppet last run on mw1112 is CRITICAL: CRITICAL: Puppet has 65 failures [12:17:11] PROBLEM - puppet last run on cp4012 is CRITICAL: CRITICAL: Puppet has 27 failures [12:17:18] PROBLEM - puppet last run on amssq33 is CRITICAL: CRITICAL: Puppet has 29 failures [12:17:18] PROBLEM - puppet last run on db1045 is CRITICAL: CRITICAL: Epic puppet fail [12:17:18] PROBLEM - puppet last run on cp3005 is CRITICAL: CRITICAL: Epic puppet fail [12:17:18] PROBLEM - puppet last run on analytics1018 is CRITICAL: CRITICAL: Puppet has 16 failures [12:17:29] PROBLEM - puppet last run on osm-cp1001 is CRITICAL: CRITICAL: Puppet has 20 failures [12:17:29] PROBLEM - puppet last run on cp1045 is CRITICAL: CRITICAL: Puppet has 25 failures [12:17:29] PROBLEM - puppet last run on search1012 is CRITICAL: CRITICAL: Puppet has 40 failures [12:17:29] PROBLEM - puppet last run on lvs1004 is CRITICAL: CRITICAL: Epic puppet fail [12:17:29] PROBLEM - puppet last run on mw1193 is CRITICAL: CRITICAL: Puppet has 59 failures [12:17:38] PROBLEM - puppet last run on mw1220 is CRITICAL: CRITICAL: Puppet has 59 failures [12:17:38] PROBLEM - puppet last run on mw1143 is CRITICAL: CRITICAL: Puppet has 64 failures [12:17:38] PROBLEM - puppet last run on ms-be1010 is CRITICAL: CRITICAL: Epic puppet fail [12:17:38] PROBLEM - puppet last run on es1004 is CRITICAL: CRITICAL: Epic puppet fail [12:17:48] PROBLEM - puppet last run on db71 is CRITICAL: CRITICAL: Puppet has 22 failures [12:17:48] PROBLEM - puppet last run on cp1053 is CRITICAL: CRITICAL: Puppet has 23 failures [12:17:48] PROBLEM - puppet last run on mw1037 is CRITICAL: CRITICAL: Puppet has 66 failures [12:17:48] PROBLEM - puppet last run on db1037 is CRITICAL: CRITICAL: Puppet has 17 failures [12:17:48] PROBLEM - puppet last run on mw1071 is CRITICAL: CRITICAL: Puppet has 60 failures [12:17:48] PROBLEM - puppet last run on mw1155 is CRITICAL: CRITICAL: Epic puppet fail [12:17:49] PROBLEM - puppet last run on mw1066 is CRITICAL: CRITICAL: Puppet has 54 failures [12:17:58] PROBLEM - puppet last run on mw1207 is CRITICAL: CRITICAL: Puppet has 70 failures [12:18:08] PROBLEM - puppet last run on cp1054 is CRITICAL: CRITICAL: Epic puppet fail [12:18:08] PROBLEM - puppet last run on db1027 is CRITICAL: CRITICAL: Puppet has 19 failures [12:18:08] PROBLEM - puppet last run on tmh1001 is CRITICAL: CRITICAL: Epic puppet fail [12:18:18] PROBLEM - puppet last run on cp3011 is CRITICAL: CRITICAL: Puppet has 19 failures [12:18:18] PROBLEM - puppet last run on lvs1003 is CRITICAL: CRITICAL: Puppet has 13 failures [12:18:29] PROBLEM - puppet last run on mw1158 is CRITICAL: CRITICAL: Puppet has 72 failures [12:18:48] PROBLEM - puppet last run on mc1015 is CRITICAL: CRITICAL: Puppet has 15 failures [12:18:48] PROBLEM - puppet last run on ssl1003 is CRITICAL: CRITICAL: Puppet has 21 failures [12:18:48] PROBLEM - puppet last run on mw1128 is CRITICAL: CRITICAL: Epic puppet fail [12:18:49] PROBLEM - puppet last run on lvs4004 is CRITICAL: CRITICAL: Puppet has 17 failures [12:18:58] PROBLEM - puppet last run on amssq37 is CRITICAL: CRITICAL: Puppet has 30 failures [12:19:08] PROBLEM - puppet last run on analytics1031 is CRITICAL: CRITICAL: Puppet has 23 failures [12:19:08] PROBLEM - puppet last run on mw1154 is CRITICAL: CRITICAL: Puppet has 70 failures [12:19:29] PROBLEM - puppet last run on cp1066 is CRITICAL: CRITICAL: Puppet has 23 failures [12:19:37] (03PS1) 10Hashar: diamond dupe def with apache::monitoring on labs [operations/puppet] - 10https://gerrit.wikimedia.org/r/154401 [12:19:38] RECOVERY - puppet last run on nickel is OK: OK: Puppet is currently enabled, last run 52 seconds ago with 0 failures [12:19:58] PROBLEM - puppet last run on mw1194 is CRITICAL: CRITICAL: Puppet has 53 failures [12:20:09] PROBLEM - puppet last run on mw1073 is CRITICAL: CRITICAL: Puppet has 66 failures [12:21:18] PROBLEM - puppet last run on es4 is CRITICAL: CRITICAL: Epic puppet fail [12:21:30] (03CR) 10Hashar: "That fixed the dupe definition on integration-dev.eqiad.wmflabs." [operations/puppet] - 10https://gerrit.wikimedia.org/r/154401 (owner: 10Hashar) [12:22:08] RECOVERY - puppet last run on ms1001 is OK: OK: Puppet is currently enabled, last run 25 seconds ago with 0 failures [12:24:09] RECOVERY - puppet last run on ms1004 is OK: OK: Puppet is currently enabled, last run 7 seconds ago with 0 failures [12:24:45] (03CR) 10Steinsplitter: [C: 031] "@Anomie: Superprotect is in danger to get switched off. Therfore no fixes needed now. AND not sure why you talk about political issues - i" [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/154376 (https://bugzilla.wikimedia.org/69640) (owner: 10Anomie) [12:25:00] (03CR) 10Steinsplitter: [C: 04-1] Fix $wgRestrictionLevels ordering [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/154376 (https://bugzilla.wikimedia.org/69640) (owner: 10Anomie) [12:25:09] PROBLEM - puppet last run on tridge is CRITICAL: CRITICAL: Epic puppet fail [12:30:48] PROBLEM - puppet last run on sodium is CRITICAL: CRITICAL: Epic puppet fail [12:30:48] PROBLEM - puppet last run on nfs1 is CRITICAL: CRITICAL: Epic puppet fail [12:33:39] RECOVERY - puppet last run on sanger is OK: OK: Puppet is currently enabled, last run 41 seconds ago with 0 failures [12:36:39] PROBLEM - puppet last run on linne is CRITICAL: CRITICAL: Epic puppet fail [12:37:38] PROBLEM - puppetmaster https on palladium is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 8140: HTTP/1.1 500 Internal Server Error [12:39:18] RECOVERY - puppet last run on es4 is OK: OK: Puppet is currently enabled, last run 11 seconds ago with 0 failures [12:39:33] (03CR) 10Bartosz Dziewoński: "Steinsplitter, please read the bug before spewing further nonsense. This patch is not related to superprotection, like it was already said" [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/154376 (https://bugzilla.wikimedia.org/69640) (owner: 10Anomie) [12:39:38] PROBLEM - puppet last run on nickel is CRITICAL: CRITICAL: Epic puppet fail [12:39:39] RECOVERY - puppetmaster https on palladium is OK: HTTP OK: Status line output matched 400 - 335 bytes in 0.109 second response time [12:43:09] RECOVERY - puppet last run on tridge is OK: OK: Puppet is currently enabled, last run 8 seconds ago with 0 failures [12:45:38] PROBLEM - puppetmaster https on palladium is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 8140: HTTP/1.1 500 Internal Server Error [12:47:38] RECOVERY - puppetmaster https on palladium is OK: HTTP OK: Status line output matched 400 - 335 bytes in 0.036 second response time [12:48:49] RECOVERY - puppet last run on nfs1 is OK: OK: Puppet is currently enabled, last run 18 seconds ago with 0 failures [12:49:18] PROBLEM - puppet last run on nescio is CRITICAL: CRITICAL: Epic puppet fail [12:52:38] PROBLEM - puppetmaster https on palladium is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 8140: HTTP/1.1 500 Internal Server Error [12:53:56] (03CR) 10Anomie: "@Steinsplitter: Even if superprotection were going to be removed today (which seems unlikely despite your assertion), in the absence of bu" [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/154376 (https://bugzilla.wikimedia.org/69640) (owner: 10Anomie) [12:54:48] RECOVERY - puppetmaster https on palladium is OK: HTTP OK: Status line output matched 400 - 335 bytes in 4.756 second response time [12:55:19] (03CR) 10Anomie: "@Bartosz: The DRY cleanup would have been a good thing here, except for the fact that order in this array matters. I don't think criticism" [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/153302 (owner: 10Tim Starling) [12:56:38] RECOVERY - puppet last run on linne is OK: OK: Puppet is currently enabled, last run 46 seconds ago with 0 failures [13:01:18] PROBLEM - puppet last run on es4 is CRITICAL: CRITICAL: Epic puppet fail [13:03:08] PROBLEM - puppet last run on ms1001 is CRITICAL: CRITICAL: Epic puppet fail [13:03:45] (03PS2) 10Helder.wiki: Fix $wgRestrictionLevels ordering [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/154376 (https://bugzilla.wikimedia.org/69640) (owner: 10Anomie) [13:04:04] (03CR) 10Helder.wiki: [C: 031] Fix $wgRestrictionLevels ordering [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/154376 (https://bugzilla.wikimedia.org/69640) (owner: 10Anomie) [13:06:09] PROBLEM - puppet last run on ms1004 is CRITICAL: CRITICAL: Epic puppet fail [13:07:38] PROBLEM - puppetmaster https on palladium is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 8140: HTTP/1.1 500 Internal Server Error [13:08:18] RECOVERY - puppet last run on nescio is OK: OK: Puppet is currently enabled, last run 12 seconds ago with 0 failures [13:08:38] RECOVERY - puppetmaster https on palladium is OK: HTTP OK: Status line output matched 400 - 335 bytes in 0.182 second response time [13:10:49] PROBLEM - puppet last run on nfs1 is CRITICAL: CRITICAL: Epic puppet fail [13:16:39] PROBLEM - puppet last run on linne is CRITICAL: CRITICAL: Epic puppet fail [13:17:58] PROBLEM - Puppet freshness on db1009 is CRITICAL: Last successful Puppet run was Sat 16 Aug 2014 11:17:22 UTC [13:19:18] RECOVERY - puppet last run on es4 is OK: OK: Puppet is currently enabled, last run 3 seconds ago with 0 failures [13:20:58] PROBLEM - Puppet freshness on db1007 is CRITICAL: Last successful Puppet run was Sat 16 Aug 2014 09:19:42 UTC [13:34:38] PROBLEM - puppetmaster https on palladium is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 8140: HTTP/1.1 500 Internal Server Error [13:35:38] RECOVERY - puppet last run on linne is OK: OK: Puppet is currently enabled, last run 17 seconds ago with 0 failures [13:36:38] RECOVERY - puppetmaster https on palladium is OK: HTTP OK: Status line output matched 400 - 335 bytes in 0.064 second response time [13:37:28] RECOVERY - Puppet freshness on db1009 is OK: puppet ran at Sat Aug 16 13:37:26 UTC 2014 [13:39:58] PROBLEM - Puppet freshness on mw1036 is CRITICAL: Last successful Puppet run was Sat 16 Aug 2014 11:39:14 UTC [13:41:58] PROBLEM - Puppet freshness on rdb1004 is CRITICAL: Last successful Puppet run was Sat 16 Aug 2014 11:41:15 UTC [13:42:58] PROBLEM - Puppet freshness on cp3008 is CRITICAL: Last successful Puppet run was Sat 16 Aug 2014 11:42:37 UTC [13:42:58] PROBLEM - Puppet freshness on mc1016 is CRITICAL: Last successful Puppet run was Sat 16 Aug 2014 11:42:11 UTC [13:44:58] PROBLEM - Puppet freshness on mw1009 is CRITICAL: Last successful Puppet run was Sat 16 Aug 2014 11:44:18 UTC [13:44:58] PROBLEM - Puppet freshness on mw1099 is CRITICAL: Last successful Puppet run was Sat 16 Aug 2014 11:44:38 UTC [13:46:58] PROBLEM - Puppet freshness on amssq55 is CRITICAL: Last successful Puppet run was Sat 16 Aug 2014 11:46:35 UTC [13:48:58] PROBLEM - Puppet freshness on db1048 is CRITICAL: Last successful Puppet run was Sat 16 Aug 2014 11:48:06 UTC [13:48:58] PROBLEM - Puppet freshness on search1005 is CRITICAL: Last successful Puppet run was Sat 16 Aug 2014 11:48:21 UTC [13:49:49] RECOVERY - puppet last run on nfs1 is OK: OK: Puppet is currently enabled, last run 54 seconds ago with 0 failures [13:50:58] PROBLEM - Puppet freshness on amssq62 is CRITICAL: Last successful Puppet run was Sat 16 Aug 2014 11:50:18 UTC [13:50:58] PROBLEM - Puppet freshness on hooft is CRITICAL: Last successful Puppet run was Sat 16 Aug 2014 11:50:13 UTC [13:52:58] PROBLEM - Puppet freshness on mw1024 is CRITICAL: Last successful Puppet run was Sat 16 Aug 2014 11:52:30 UTC [13:53:38] PROBLEM - puppetmaster https on palladium is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 8140: HTTP/1.1 500 Internal Server Error [13:54:39] RECOVERY - puppetmaster https on palladium is OK: HTTP OK: Status line output matched 400 - 335 bytes in 0.091 second response time [13:54:58] PROBLEM - Puppet freshness on cp1067 is CRITICAL: Last successful Puppet run was Sat 16 Aug 2014 11:54:32 UTC [13:58:38] RECOVERY - puppet last run on nickel is OK: OK: Puppet is currently enabled, last run 8 seconds ago with 0 failures [13:59:38] RECOVERY - Puppet freshness on mw1036 is OK: puppet ran at Sat Aug 16 13:59:30 UTC 2014 [14:01:08] RECOVERY - puppet last run on ms1001 is OK: OK: Puppet is currently enabled, last run 29 seconds ago with 0 failures [14:01:39] RECOVERY - Puppet freshness on rdb1004 is OK: puppet ran at Sat Aug 16 14:01:36 UTC 2014 [14:01:58] RECOVERY - Puppet freshness on cp3008 is OK: puppet ran at Sat Aug 16 14:01:51 UTC 2014 [14:02:51] (03CR) 10MZMcBride: "Super-protection is a bad idea and should be removed. But using -1 on vaguely related changesets probably does more harm than good." (031 comment) [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/154376 (https://bugzilla.wikimedia.org/69640) (owner: 10Anomie) [14:03:38] PROBLEM - puppetmaster https on palladium is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 8140: HTTP/1.1 500 Internal Server Error [14:04:18] RECOVERY - Puppet freshness on mw1099 is OK: puppet ran at Sat Aug 16 14:04:07 UTC 2014 [14:04:58] RECOVERY - Puppet freshness on mw1009 is OK: puppet ran at Sat Aug 16 14:04:53 UTC 2014 [14:05:09] PROBLEM - puppet last run on tridge is CRITICAL: CRITICAL: Epic puppet fail [14:06:38] RECOVERY - puppetmaster https on palladium is OK: HTTP OK: Status line output matched 400 - 335 bytes in 0.065 second response time [14:08:58] PROBLEM - Puppet freshness on mw1202 is CRITICAL: Last successful Puppet run was Sat 16 Aug 2014 12:08:15 UTC [14:08:58] PROBLEM - Puppet freshness on sodium is CRITICAL: Last successful Puppet run was Sat 16 Aug 2014 12:08:15 UTC [14:09:49] PROBLEM - puppet last run on nfs1 is CRITICAL: CRITICAL: Epic puppet fail [14:10:58] PROBLEM - Puppet freshness on protactinium is CRITICAL: Last successful Puppet run was Sat 16 Aug 2014 12:10:31 UTC [14:11:58] PROBLEM - Puppet freshness on db1030 is CRITICAL: Last successful Puppet run was Sat 16 Aug 2014 12:11:42 UTC [14:12:28] RECOVERY - Puppet freshness on db1030 is OK: puppet ran at Sat Aug 16 14:12:20 UTC 2014 [14:12:28] RECOVERY - Puppet freshness on mw1024 is OK: puppet ran at Sat Aug 16 14:12:20 UTC 2014 [14:13:39] PROBLEM - puppet last run on sanger is CRITICAL: CRITICAL: Epic puppet fail [14:13:58] PROBLEM - Puppet freshness on wtp1010 is CRITICAL: Last successful Puppet run was Sat 16 Aug 2014 12:13:42 UTC [14:14:18] RECOVERY - Puppet freshness on cp1067 is OK: puppet ran at Sat Aug 16 14:14:16 UTC 2014 [14:14:58] PROBLEM - Puppet freshness on mw1071 is CRITICAL: Last successful Puppet run was Sat 16 Aug 2014 12:14:28 UTC [14:14:58] PROBLEM - Puppet freshness on search1004 is CRITICAL: Last successful Puppet run was Sat 16 Aug 2014 12:14:28 UTC [14:16:39] PROBLEM - puppet last run on linne is CRITICAL: CRITICAL: Epic puppet fail [14:18:38] PROBLEM - puppetmaster https on palladium is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 8140: HTTP/1.1 500 Internal Server Error [14:19:38] RECOVERY - puppetmaster https on palladium is OK: HTTP OK: Status line output matched 400 - 335 bytes in 0.157 second response time [14:22:18] RECOVERY - Puppet freshness on mc1016 is OK: puppet ran at Sat Aug 16 14:22:15 UTC 2014 [14:23:09] RECOVERY - puppet last run on tridge is OK: OK: Puppet is currently enabled, last run 8 seconds ago with 0 failures [14:25:58] PROBLEM - Puppet freshness on holmium is CRITICAL: Last successful Puppet run was Sat 16 Aug 2014 12:25:37 UTC [14:26:58] PROBLEM - Puppet freshness on antimony is CRITICAL: Last successful Puppet run was Sat 16 Aug 2014 12:26:47 UTC [14:26:58] PROBLEM - Puppet freshness on mw1002 is CRITICAL: Last successful Puppet run was Sat 16 Aug 2014 12:26:37 UTC [14:26:58] PROBLEM - Puppet freshness on mw1076 is CRITICAL: Last successful Puppet run was Sat 16 Aug 2014 12:26:47 UTC [14:27:18] RECOVERY - Puppet freshness on mw1076 is OK: puppet ran at Sat Aug 16 14:27:12 UTC 2014 [14:27:58] PROBLEM - Puppet freshness on mw1151 is CRITICAL: Last successful Puppet run was Sat 16 Aug 2014 12:27:43 UTC [14:28:18] RECOVERY - Puppet freshness on sodium is OK: puppet ran at Sat Aug 16 14:28:07 UTC 2014 [14:28:18] RECOVERY - Puppet freshness on mw1202 is OK: puppet ran at Sat Aug 16 14:28:12 UTC 2014 [14:28:28] RECOVERY - Puppet freshness on mw1151 is OK: puppet ran at Sat Aug 16 14:28:17 UTC 2014 [14:28:48] RECOVERY - puppet last run on sodium is OK: OK: Puppet is currently enabled, last run 25 seconds ago with 0 failures [14:28:49] RECOVERY - puppet last run on nfs1 is OK: OK: Puppet is currently enabled, last run 32 seconds ago with 0 failures [14:28:58] PROBLEM - Puppet freshness on amssq56 is CRITICAL: Last successful Puppet run was Sat 16 Aug 2014 12:28:08 UTC [14:29:58] PROBLEM - Puppet freshness on ssl3002 is CRITICAL: Last successful Puppet run was Sat 16 Aug 2014 12:29:29 UTC [14:31:18] RECOVERY - Puppet freshness on protactinium is OK: puppet ran at Sat Aug 16 14:31:14 UTC 2014 [14:32:39] RECOVERY - puppet last run on sanger is OK: OK: Puppet is currently enabled, last run 14 seconds ago with 0 failures [14:33:58] RECOVERY - Puppet freshness on search1004 is OK: puppet ran at Sat Aug 16 14:33:50 UTC 2014 [14:34:08] RECOVERY - Puppet freshness on mw1071 is OK: puppet ran at Sat Aug 16 14:34:00 UTC 2014 [14:34:18] RECOVERY - Puppet freshness on wtp1010 is OK: puppet ran at Sat Aug 16 14:34:15 UTC 2014 [14:34:22] (03PS1) 10Jforrester: Increase default thumbnail display size from 220px to 300px [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/154408 (https://bugzilla.wikimedia.org/67709) [14:35:38] RECOVERY - puppet last run on linne is OK: OK: Puppet is currently enabled, last run 39 seconds ago with 0 failures [14:39:38] PROBLEM - puppet last run on nickel is CRITICAL: CRITICAL: Epic puppet fail [14:40:28] RECOVERY - Puppet freshness on db1007 is OK: puppet ran at Sat Aug 16 14:40:22 UTC 2014 [14:40:58] PROBLEM - Puppet freshness on db1010 is CRITICAL: Last successful Puppet run was Sat 16 Aug 2014 12:40:33 UTC [14:41:18] PROBLEM - puppet last run on es4 is CRITICAL: CRITICAL: Epic puppet fail [14:43:08] PROBLEM - puppet last run on ms1001 is CRITICAL: CRITICAL: Epic puppet fail [14:44:58] PROBLEM - Puppet freshness on ms1004 is CRITICAL: Last successful Puppet run was Sat 16 Aug 2014 12:43:55 UTC [14:45:28] RECOVERY - Puppet freshness on holmium is OK: puppet ran at Sat Aug 16 14:45:19 UTC 2014 [14:46:38] RECOVERY - Puppet freshness on mw1002 is OK: puppet ran at Sat Aug 16 14:46:34 UTC 2014 [14:47:07] (03CR) 10Jforrester: [C: 04-1] "Scheduled for the SF-morning SWAT on Monday 25 August; let's not merge this prematurely." [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/154408 (https://bugzilla.wikimedia.org/67709) (owner: 10Jforrester) [14:47:28] RECOVERY - Puppet freshness on antimony is OK: puppet ran at Sat Aug 16 14:47:25 UTC 2014 [14:47:58] PROBLEM - Puppet freshness on db1043 is CRITICAL: Last successful Puppet run was Sat 16 Aug 2014 12:46:51 UTC [14:48:18] RECOVERY - Puppet freshness on search1005 is OK: puppet ran at Sat Aug 16 14:48:10 UTC 2014 [14:48:58] PROBLEM - Puppet freshness on cp4005 is CRITICAL: Last successful Puppet run was Sat 16 Aug 2014 12:47:51 UTC [14:49:49] PROBLEM - puppet last run on nfs1 is CRITICAL: CRITICAL: Epic puppet fail [14:51:58] PROBLEM - Puppet freshness on wtp1011 is CRITICAL: Last successful Puppet run was Sat 16 Aug 2014 12:50:48 UTC [14:52:14] 'Epic puppet fail' :D [14:54:39] PROBLEM - puppet last run on sanger is CRITICAL: CRITICAL: Epic puppet fail [14:54:58] PROBLEM - Puppet freshness on mw1215 is CRITICAL: Last successful Puppet run was Sat 16 Aug 2014 12:54:04 UTC [14:56:58] PROBLEM - Puppet freshness on mw1199 is CRITICAL: Last successful Puppet run was Sat 16 Aug 2014 12:56:30 UTC [14:58:38] RECOVERY - puppet last run on nickel is OK: OK: Puppet is currently enabled, last run 3 seconds ago with 0 failures [14:58:58] PROBLEM - Puppet freshness on es1005 is CRITICAL: Last successful Puppet run was Sat 16 Aug 2014 12:58:11 UTC [14:59:18] RECOVERY - puppet last run on es4 is OK: OK: Puppet is currently enabled, last run 7 seconds ago with 0 failures [14:59:58] PROBLEM - Puppet freshness on mw1216 is CRITICAL: Last successful Puppet run was Sat 16 Aug 2014 12:59:46 UTC [15:00:09] RECOVERY - Puppet freshness on mw1216 is OK: puppet ran at Sat Aug 16 15:00:05 UTC 2014 [15:00:58] PROBLEM - Puppet freshness on cp3006 is CRITICAL: Last successful Puppet run was Sat 16 Aug 2014 13:00:47 UTC [15:01:09] RECOVERY - Puppet freshness on cp3006 is OK: puppet ran at Sat Aug 16 15:01:05 UTC 2014 [15:01:58] PROBLEM - Puppet freshness on mw1005 is CRITICAL: Last successful Puppet run was Sat 16 Aug 2014 13:01:27 UTC [15:01:58] PROBLEM - Puppet freshness on analytics1033 is CRITICAL: Last successful Puppet run was Sat 16 Aug 2014 13:01:42 UTC [15:02:08] RECOVERY - puppet last run on ms1001 is OK: OK: Puppet is currently enabled, last run 55 seconds ago with 0 failures [15:04:58] PROBLEM - Puppet freshness on mw1117 is CRITICAL: Last successful Puppet run was Sat 16 Aug 2014 13:04:18 UTC [15:05:58] PROBLEM - Puppet freshness on ms-be3003 is CRITICAL: Last successful Puppet run was Sat 16 Aug 2014 13:05:43 UTC [15:06:38] RECOVERY - Puppet freshness on amssq55 is OK: puppet ran at Sat Aug 16 15:06:32 UTC 2014 [15:06:58] PROBLEM - Puppet freshness on mw1177 is CRITICAL: Last successful Puppet run was Sat 16 Aug 2014 13:06:14 UTC [15:07:28] RECOVERY - Puppet freshness on db1043 is OK: puppet ran at Sat Aug 16 15:07:23 UTC 2014 [15:07:48] RECOVERY - Puppet freshness on cp4005 is OK: puppet ran at Sat Aug 16 15:07:43 UTC 2014 [15:09:18] RECOVERY - Puppet freshness on ssl3002 is OK: puppet ran at Sat Aug 16 15:09:14 UTC 2014 [15:10:28] RECOVERY - Puppet freshness on amssq62 is OK: puppet ran at Sat Aug 16 15:10:19 UTC 2014 [15:10:38] PROBLEM - puppetmaster https on palladium is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 8140: HTTP/1.1 500 Internal Server Error [15:10:48] PROBLEM - puppet last run on sodium is CRITICAL: CRITICAL: Epic puppet fail [15:11:38] RECOVERY - puppetmaster https on palladium is OK: HTTP OK: Status line output matched 400 - 335 bytes in 0.102 second response time [15:11:58] PROBLEM - Puppet freshness on mw1148 is CRITICAL: Last successful Puppet run was Sat 16 Aug 2014 13:11:31 UTC [15:12:39] RECOVERY - puppet last run on sanger is OK: OK: Puppet is currently enabled, last run 31 seconds ago with 0 failures [15:12:58] PROBLEM - Puppet freshness on mw1167 is CRITICAL: Last successful Puppet run was Sat 16 Aug 2014 13:11:51 UTC [15:13:58] PROBLEM - Puppet freshness on db1033 is CRITICAL: Last successful Puppet run was Sat 16 Aug 2014 13:13:11 UTC [15:13:58] PROBLEM - Puppet freshness on analytics1004 is CRITICAL: Last successful Puppet run was Sat 16 Aug 2014 13:13:16 UTC [15:14:28] RECOVERY - Puppet freshness on mw1215 is OK: puppet ran at Sat Aug 16 15:14:21 UTC 2014 [15:15:58] RECOVERY - Puppet freshness on mw1199 is OK: puppet ran at Sat Aug 16 15:15:57 UTC 2014 [15:17:54] (03PS1) 10Ori.livneh: HHVM: [operations/puppet] - 10https://gerrit.wikimedia.org/r/154414 [15:18:22] (03PS2) 10Ori.livneh: Improve logging for HHVM [operations/puppet] - 10https://gerrit.wikimedia.org/r/154414 [15:18:58] PROBLEM - Puppet freshness on db1019 is CRITICAL: Last successful Puppet run was Sat 16 Aug 2014 13:18:23 UTC [15:20:38] PROBLEM - puppet last run on nickel is CRITICAL: CRITICAL: Epic puppet fail [15:20:38] PROBLEM - puppetmaster https on palladium is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 8140: HTTP/1.1 500 Internal Server Error [15:21:28] RECOVERY - Puppet freshness on analytics1033 is OK: puppet ran at Sat Aug 16 15:21:24 UTC 2014 [15:22:39] RECOVERY - puppetmaster https on palladium is OK: HTTP OK: Status line output matched 400 - 335 bytes in 0.036 second response time [15:23:08] PROBLEM - puppet last run on ms1001 is CRITICAL: CRITICAL: Epic puppet fail [15:23:58] PROBLEM - Puppet freshness on search1016 is CRITICAL: Last successful Puppet run was Sat 16 Aug 2014 13:23:05 UTC [15:24:18] RECOVERY - Puppet freshness on ms1004 is OK: puppet ran at Sat Aug 16 15:24:14 UTC 2014 [15:24:38] RECOVERY - Puppet freshness on mw1117 is OK: puppet ran at Sat Aug 16 15:24:29 UTC 2014 [15:25:09] RECOVERY - puppet last run on ms1004 is OK: OK: Puppet is currently enabled, last run 50 seconds ago with 0 failures [15:26:38] PROBLEM - puppetmaster https on palladium is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 8140: HTTP/1.1 500 Internal Server Error [15:27:48] RECOVERY - puppet last run on sodium is OK: OK: Puppet is currently enabled, last run 0 seconds ago with 0 failures [15:27:58] PROBLEM - Puppet freshness on amssq36 is CRITICAL: Last successful Puppet run was Sat 16 Aug 2014 13:27:47 UTC [15:28:38] RECOVERY - puppetmaster https on palladium is OK: HTTP OK: Status line output matched 400 - 335 bytes in 0.045 second response time [15:28:49] RECOVERY - puppet last run on nfs1 is OK: OK: Puppet is currently enabled, last run 3 seconds ago with 0 failures [15:29:38] RECOVERY - Puppet freshness on hooft is OK: puppet ran at Sat Aug 16 15:29:32 UTC 2014 [15:29:58] PROBLEM - Puppet freshness on stat1002 is CRITICAL: Last successful Puppet run was Sat 16 Aug 2014 13:29:08 UTC [15:32:48] RECOVERY - Puppet freshness on mw1167 is OK: puppet ran at Sat Aug 16 15:32:38 UTC 2014 [15:32:58] RECOVERY - Puppet freshness on analytics1004 is OK: puppet ran at Sat Aug 16 15:32:48 UTC 2014 [15:33:58] PROBLEM - Puppet freshness on db1011 is CRITICAL: Last successful Puppet run was Sat 16 Aug 2014 13:33:35 UTC [15:35:58] PROBLEM - Puppet freshness on cp4010 is CRITICAL: Last successful Puppet run was Sat 16 Aug 2014 13:35:46 UTC [15:36:39] PROBLEM - puppet last run on linne is CRITICAL: CRITICAL: Epic puppet fail [15:37:38] RECOVERY - Puppet freshness on db1019 is OK: puppet ran at Sat Aug 16 15:37:35 UTC 2014 [15:37:48] RECOVERY - Puppet freshness on es1005 is OK: puppet ran at Sat Aug 16 15:37:45 UTC 2014 [15:38:38] RECOVERY - puppet last run on nickel is OK: OK: Puppet is currently enabled, last run 6 seconds ago with 0 failures [15:39:58] PROBLEM - Puppet freshness on db1024 is CRITICAL: Last successful Puppet run was Sat 16 Aug 2014 13:39:07 UTC [15:41:09] RECOVERY - Puppet freshness on mw1005 is OK: puppet ran at Sat Aug 16 15:41:06 UTC 2014 [15:41:18] PROBLEM - puppet last run on es4 is CRITICAL: CRITICAL: Epic puppet fail [15:41:58] PROBLEM - Puppet freshness on carbon is CRITICAL: Last successful Puppet run was Sat 16 Aug 2014 13:41:43 UTC [15:41:58] RECOVERY - Puppet freshness on carbon is OK: puppet ran at Sat Aug 16 15:41:51 UTC 2014 [15:42:08] RECOVERY - puppet last run on ms1001 is OK: OK: Puppet is currently enabled, last run 58 seconds ago with 0 failures [15:44:58] PROBLEM - Puppet freshness on cp3003 is CRITICAL: Last successful Puppet run was Sat 16 Aug 2014 13:43:54 UTC [15:45:38] PROBLEM - puppetmaster https on palladium is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 8140: HTTP/1.1 500 Internal Server Error [15:46:09] PROBLEM - puppet last run on ms1004 is CRITICAL: CRITICAL: Epic puppet fail [15:46:28] RECOVERY - Puppet freshness on mw1177 is OK: puppet ran at Sat Aug 16 15:46:18 UTC 2014 [15:46:38] RECOVERY - puppetmaster https on palladium is OK: HTTP OK: Status line output matched 400 - 335 bytes in 0.061 second response time [15:47:58] PROBLEM - Puppet freshness on mc1012 is CRITICAL: Last successful Puppet run was Sat 16 Aug 2014 13:47:00 UTC [15:47:58] RECOVERY - Puppet freshness on amssq36 is OK: puppet ran at Sat Aug 16 15:47:54 UTC 2014 [15:48:28] RECOVERY - Puppet freshness on db1048 is OK: puppet ran at Sat Aug 16 15:48:19 UTC 2014 [15:49:18] RECOVERY - Puppet freshness on stat1002 is OK: puppet ran at Sat Aug 16 15:49:09 UTC 2014 [15:49:18] PROBLEM - puppet last run on nescio is CRITICAL: CRITICAL: Epic puppet fail [15:49:58] PROBLEM - Puppet freshness on mw1165 is CRITICAL: Last successful Puppet run was Sat 16 Aug 2014 13:49:16 UTC [15:49:58] PROBLEM - Puppet freshness on mw1030 is CRITICAL: Last successful Puppet run was Sat 16 Aug 2014 13:49:47 UTC [15:53:28] RECOVERY - Puppet freshness on db1033 is OK: puppet ran at Sat Aug 16 15:53:21 UTC 2014 [15:53:39] PROBLEM - puppet last run on sanger is CRITICAL: CRITICAL: Epic puppet fail [15:54:58] PROBLEM - Puppet freshness on copper is CRITICAL: Last successful Puppet run was Sat 16 Aug 2014 13:54:03 UTC [15:54:58] PROBLEM - Puppet freshness on mw1203 is CRITICAL: Last successful Puppet run was Sat 16 Aug 2014 13:54:03 UTC [15:56:58] PROBLEM - Puppet freshness on mw1103 is CRITICAL: Last successful Puppet run was Sat 16 Aug 2014 13:56:39 UTC [15:56:58] PROBLEM - Puppet freshness on mw1179 is CRITICAL: Last successful Puppet run was Sat 16 Aug 2014 13:56:39 UTC [15:59:58] PROBLEM - Puppet freshness on search1014 is CRITICAL: Last successful Puppet run was Sat 16 Aug 2014 13:58:55 UTC [16:00:18] RECOVERY - Puppet freshness on db1010 is OK: puppet ran at Sat Aug 16 16:00:09 UTC 2014 [16:00:18] RECOVERY - puppet last run on es4 is OK: OK: Puppet is currently enabled, last run 29 seconds ago with 0 failures [16:01:38] PROBLEM - puppetmaster https on palladium is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 8140: HTTP/1.1 500 Internal Server Error [16:01:58] PROBLEM - Puppet freshness on mw1038 is CRITICAL: Last successful Puppet run was Sat 16 Aug 2014 14:01:36 UTC [16:01:58] PROBLEM - Puppet freshness on ms-be1001 is CRITICAL: Last successful Puppet run was Sat 16 Aug 2014 14:01:16 UTC [16:03:14] (03CR) 10Reza: [C: 031] Add botadmin user group on fa.wikipedia [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/154126 (https://bugzilla.wikimedia.org/69411) (owner: 10Calak) [16:03:28] RECOVERY - Puppet freshness on search1016 is OK: puppet ran at Sat Aug 16 16:03:20 UTC 2014 [16:04:18] RECOVERY - Puppet freshness on cp3003 is OK: puppet ran at Sat Aug 16 16:04:16 UTC 2014 [16:05:09] RECOVERY - puppet last run on ms1004 is OK: OK: Puppet is currently enabled, last run 27 seconds ago with 0 failures [16:05:38] RECOVERY - puppetmaster https on palladium is OK: HTTP OK: Status line output matched 400 - 335 bytes in 0.055 second response time [16:07:18] RECOVERY - Puppet freshness on mc1012 is OK: puppet ran at Sat Aug 16 16:07:07 UTC 2014 [16:07:58] PROBLEM - Puppet freshness on snapshot1001 is CRITICAL: Last successful Puppet run was Sat 16 Aug 2014 14:07:08 UTC [16:09:08] RECOVERY - Puppet freshness on mw1030 is OK: puppet ran at Sat Aug 16 16:09:03 UTC 2014 [16:09:48] RECOVERY - Puppet freshness on mw1165 is OK: puppet ran at Sat Aug 16 16:09:38 UTC 2014 [16:09:48] PROBLEM - puppet last run on nfs1 is CRITICAL: CRITICAL: Epic puppet fail [16:09:58] PROBLEM - Puppet freshness on mw1159 is CRITICAL: Last successful Puppet run was Sat 16 Aug 2014 14:09:09 UTC [16:09:58] PROBLEM - Puppet freshness on mw1210 is CRITICAL: Last successful Puppet run was Sat 16 Aug 2014 14:09:45 UTC [16:10:18] RECOVERY - Puppet freshness on mw1210 is OK: puppet ran at Sat Aug 16 16:10:08 UTC 2014 [16:10:49] PROBLEM - puppet last run on sodium is CRITICAL: CRITICAL: Epic puppet fail [16:10:58] RECOVERY - Puppet freshness on wtp1011 is OK: puppet ran at Sat Aug 16 16:10:54 UTC 2014 [16:12:39] RECOVERY - puppet last run on sanger is OK: OK: Puppet is currently enabled, last run 0 seconds ago with 0 failures [16:14:18] RECOVERY - Puppet freshness on copper is OK: puppet ran at Sat Aug 16 16:14:15 UTC 2014 [16:17:38] RECOVERY - Puppet freshness on mw1179 is OK: puppet ran at Sat Aug 16 16:17:31 UTC 2014 [16:17:58] PROBLEM - Puppet freshness on mw1017 is CRITICAL: Last successful Puppet run was Sat 16 Aug 2014 14:17:12 UTC [16:20:38] PROBLEM - puppet last run on nickel is CRITICAL: CRITICAL: Epic puppet fail [16:20:58] RECOVERY - Puppet freshness on mw1038 is OK: puppet ran at Sat Aug 16 16:20:52 UTC 2014 [16:21:28] RECOVERY - Puppet freshness on ms-be1001 is OK: puppet ran at Sat Aug 16 16:21:23 UTC 2014 [16:21:38] PROBLEM - puppetmaster https on palladium is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 8140: HTTP/1.1 500 Internal Server Error [16:22:58] PROBLEM - Puppet freshness on potassium is CRITICAL: Last successful Puppet run was Sat 16 Aug 2014 14:22:30 UTC [16:22:58] PROBLEM - Puppet freshness on analytics1040 is CRITICAL: Last successful Puppet run was Sat 16 Aug 2014 14:22:35 UTC [16:23:08] PROBLEM - puppet last run on ms1001 is CRITICAL: CRITICAL: Epic puppet fail [16:23:58] PROBLEM - Puppet freshness on analytics1025 is CRITICAL: Last successful Puppet run was Sat 16 Aug 2014 14:23:10 UTC [16:24:38] RECOVERY - puppetmaster https on palladium is OK: HTTP OK: Status line output matched 400 - 335 bytes in 0.901 second response time [16:24:58] PROBLEM - Puppet freshness on mw1189 is CRITICAL: Last successful Puppet run was Sat 16 Aug 2014 14:24:21 UTC [16:25:09] PROBLEM - puppet last run on ms1004 is CRITICAL: CRITICAL: Epic puppet fail [16:26:38] RECOVERY - Puppet freshness on ms-be3003 is OK: puppet ran at Sat Aug 16 16:26:29 UTC 2014 [16:27:58] PROBLEM - Puppet freshness on es1007 is CRITICAL: Last successful Puppet run was Sat 16 Aug 2014 14:27:02 UTC [16:29:48] RECOVERY - puppet last run on sodium is OK: OK: Puppet is currently enabled, last run 51 seconds ago with 0 failures [16:29:48] RECOVERY - puppet last run on nfs1 is OK: OK: Puppet is currently enabled, last run 48 seconds ago with 0 failures [16:29:58] PROBLEM - Puppet freshness on amssq56 is CRITICAL: Last successful Puppet run was Sat 16 Aug 2014 12:28:08 UTC [16:29:58] PROBLEM - Puppet freshness on mw1163 is CRITICAL: Last successful Puppet run was Sat 16 Aug 2014 14:29:38 UTC [16:29:58] PROBLEM - Puppet freshness on rdb1001 is CRITICAL: Last successful Puppet run was Sat 16 Aug 2014 14:29:08 UTC [16:29:58] RECOVERY - Puppet freshness on mw1163 is OK: puppet ran at Sat Aug 16 16:29:56 UTC 2014 [16:30:08] RECOVERY - Puppet freshness on mw1159 is OK: puppet ran at Sat Aug 16 16:30:01 UTC 2014 [16:30:38] PROBLEM - puppetmaster https on palladium is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 8140: HTTP/1.1 500 Internal Server Error [16:31:38] RECOVERY - Puppet freshness on mw1148 is OK: puppet ran at Sat Aug 16 16:31:32 UTC 2014 [16:32:58] PROBLEM - Puppet freshness on analytics1028 is CRITICAL: Last successful Puppet run was Sat 16 Aug 2014 14:32:34 UTC [16:33:58] PROBLEM - Puppet freshness on mw1142 is CRITICAL: Last successful Puppet run was Sat 16 Aug 2014 14:33:09 UTC [16:33:58] PROBLEM - Puppet freshness on mw1193 is CRITICAL: Last successful Puppet run was Sat 16 Aug 2014 14:33:45 UTC [16:34:39] RECOVERY - puppetmaster https on palladium is OK: HTTP OK: Status line output matched 400 - 335 bytes in 4.148 second response time [16:35:39] RECOVERY - puppet last run on linne is OK: OK: Puppet is currently enabled, last run 2 seconds ago with 0 failures [16:37:28] RECOVERY - Puppet freshness on mw1017 is OK: puppet ran at Sat Aug 16 16:37:24 UTC 2014 [16:37:58] PROBLEM - Puppet freshness on elastic1010 is CRITICAL: Last successful Puppet run was Sat 16 Aug 2014 14:37:26 UTC [16:37:58] PROBLEM - Puppet freshness on cp4015 is CRITICAL: Last successful Puppet run was Sat 16 Aug 2014 14:36:56 UTC [16:38:38] RECOVERY - puppet last run on nickel is OK: OK: Puppet is currently enabled, last run 34 seconds ago with 0 failures [16:39:18] RECOVERY - Puppet freshness on db1024 is OK: puppet ran at Sat Aug 16 16:39:10 UTC 2014 [16:39:18] RECOVERY - Puppet freshness on search1014 is OK: puppet ran at Sat Aug 16 16:39:10 UTC 2014 [16:39:39] PROBLEM - puppetmaster https on palladium is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 8140: HTTP/1.1 500 Internal Server Error [16:39:58] PROBLEM - Puppet freshness on amssq58 is CRITICAL: Last successful Puppet run was Sat 16 Aug 2014 14:39:27 UTC [16:41:18] PROBLEM - puppet last run on es4 is CRITICAL: CRITICAL: Epic puppet fail [16:41:38] RECOVERY - puppetmaster https on palladium is OK: HTTP OK: Status line output matched 400 - 335 bytes in 0.045 second response time [16:41:58] PROBLEM - Puppet freshness on mw1031 is CRITICAL: Last successful Puppet run was Sat 16 Aug 2014 14:41:08 UTC [16:43:28] RECOVERY - Puppet freshness on analytics1040 is OK: puppet ran at Sat Aug 16 16:43:22 UTC 2014 [16:44:29] RECOVERY - Puppet freshness on mw1189 is OK: puppet ran at Sat Aug 16 16:44:27 UTC 2014 [16:45:09] PROBLEM - puppet last run on tridge is CRITICAL: CRITICAL: Epic puppet fail [16:46:58] RECOVERY - Puppet freshness on snapshot1001 is OK: puppet ran at Sat Aug 16 16:46:53 UTC 2014 [16:47:38] RECOVERY - Puppet freshness on es1007 is OK: puppet ran at Sat Aug 16 16:47:33 UTC 2014 [16:48:18] RECOVERY - puppet last run on nescio is OK: OK: Puppet is currently enabled, last run 3 seconds ago with 0 failures [16:48:38] RECOVERY - Puppet freshness on amssq56 is OK: puppet ran at Sat Aug 16 16:48:29 UTC 2014 [16:48:58] PROBLEM - Puppet freshness on mw1098 is CRITICAL: Last successful Puppet run was Sat 16 Aug 2014 14:48:15 UTC [16:49:49] PROBLEM - puppet last run on sodium is CRITICAL: CRITICAL: Epic puppet fail [16:50:58] PROBLEM - Puppet freshness on mw1029 is CRITICAL: Last successful Puppet run was Sat 16 Aug 2014 14:50:36 UTC [16:52:58] PROBLEM - Puppet freshness on mw1043 is CRITICAL: Last successful Puppet run was Sat 16 Aug 2014 14:52:02 UTC [16:52:58] PROBLEM - Puppet freshness on mw1201 is CRITICAL: Last successful Puppet run was Sat 16 Aug 2014 14:52:22 UTC [16:53:18] RECOVERY - Puppet freshness on db1011 is OK: puppet ran at Sat Aug 16 16:53:16 UTC 2014 [16:53:39] PROBLEM - puppet last run on sanger is CRITICAL: CRITICAL: Epic puppet fail [16:53:58] RECOVERY - Puppet freshness on mw1142 is OK: puppet ran at Sat Aug 16 16:53:51 UTC 2014 [16:54:18] RECOVERY - Puppet freshness on mw1203 is OK: puppet ran at Sat Aug 16 16:54:11 UTC 2014 [16:56:39] PROBLEM - puppet last run on linne is CRITICAL: CRITICAL: Epic puppet fail [16:56:39] RECOVERY - Puppet freshness on mw1103 is OK: puppet ran at Sat Aug 16 16:56:37 UTC 2014 [16:57:28] RECOVERY - Puppet freshness on cp4015 is OK: puppet ran at Sat Aug 16 16:57:22 UTC 2014 [16:59:08] RECOVERY - Puppet freshness on amssq58 is OK: puppet ran at Sat Aug 16 16:59:03 UTC 2014 [17:00:58] PROBLEM - Puppet freshness on mc1010 is CRITICAL: Last successful Puppet run was Sat 16 Aug 2014 14:59:59 UTC [17:01:08] RECOVERY - puppet last run on ms1001 is OK: OK: Puppet is currently enabled, last run 22 seconds ago with 0 failures [17:01:18] RECOVERY - Puppet freshness on mw1031 is OK: puppet ran at Sat Aug 16 17:01:14 UTC 2014 [17:01:58] PROBLEM - Puppet freshness on logstash1003 is CRITICAL: Last successful Puppet run was Sat 16 Aug 2014 15:01:05 UTC [17:03:08] RECOVERY - Puppet freshness on analytics1025 is OK: puppet ran at Sat Aug 16 17:03:04 UTC 2014 [17:03:29] RECOVERY - Puppet freshness on potassium is OK: puppet ran at Sat Aug 16 17:03:25 UTC 2014 [17:03:58] PROBLEM - Puppet freshness on wtp1006 is CRITICAL: Last successful Puppet run was Sat 16 Aug 2014 15:02:56 UTC [17:03:58] PROBLEM - Puppet freshness on cp1047 is CRITICAL: Last successful Puppet run was Sat 16 Aug 2014 15:03:01 UTC [17:04:58] PROBLEM - Puppet freshness on lead is CRITICAL: Last successful Puppet run was Sat 16 Aug 2014 15:03:51 UTC [17:10:18] RECOVERY - Puppet freshness on mw1029 is OK: puppet ran at Sat Aug 16 17:10:12 UTC 2014 [17:11:38] PROBLEM - puppetmaster https on palladium is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 8140: HTTP/1.1 500 Internal Server Error [17:12:08] RECOVERY - Puppet freshness on mw1201 is OK: puppet ran at Sat Aug 16 17:11:58 UTC 2014 [17:12:38] RECOVERY - puppetmaster https on palladium is OK: HTTP OK: Status line output matched 400 - 335 bytes in 0.069 second response time [17:13:28] RECOVERY - Puppet freshness on analytics1028 is OK: puppet ran at Sat Aug 16 17:13:23 UTC 2014 [17:13:58] PROBLEM - Puppet freshness on es1010 is CRITICAL: Last successful Puppet run was Sat 16 Aug 2014 15:13:26 UTC [17:13:58] PROBLEM - Puppet freshness on ms-be1002 is CRITICAL: Last successful Puppet run was Sat 16 Aug 2014 15:13:46 UTC [17:14:38] RECOVERY - Puppet freshness on mw1193 is OK: puppet ran at Sat Aug 16 17:14:34 UTC 2014 [17:15:39] RECOVERY - puppet last run on linne is OK: OK: Puppet is currently enabled, last run 31 seconds ago with 0 failures [17:16:48] RECOVERY - Puppet freshness on elastic1010 is OK: puppet ran at Sat Aug 16 17:16:40 UTC 2014 [17:16:58] RECOVERY - Puppet freshness on cp4010 is OK: puppet ran at Sat Aug 16 17:16:50 UTC 2014 [17:17:58] PROBLEM - Puppet freshness on lvs3002 is CRITICAL: Last successful Puppet run was Sat 16 Aug 2014 15:17:07 UTC [17:19:18] RECOVERY - puppet last run on es4 is OK: OK: Puppet is currently enabled, last run 17 seconds ago with 0 failures [17:19:38] PROBLEM - puppet last run on nickel is CRITICAL: CRITICAL: Epic puppet fail [17:19:58] RECOVERY - Puppet freshness on mc1010 is OK: puppet ran at Sat Aug 16 17:19:56 UTC 2014 [17:20:58] RECOVERY - Puppet freshness on logstash1003 is OK: puppet ran at Sat Aug 16 17:20:56 UTC 2014 [17:23:08] PROBLEM - puppet last run on ms1001 is CRITICAL: CRITICAL: Epic puppet fail [17:23:09] RECOVERY - Puppet freshness on wtp1006 is OK: puppet ran at Sat Aug 16 17:23:07 UTC 2014 [17:24:09] RECOVERY - puppet last run on tridge is OK: OK: Puppet is currently enabled, last run 41 seconds ago with 0 failures [17:25:58] PROBLEM - Puppet freshness on db1018 is CRITICAL: Last successful Puppet run was Sat 16 Aug 2014 15:24:55 UTC [17:28:28] RECOVERY - Puppet freshness on mw1098 is OK: puppet ran at Sat Aug 16 17:28:19 UTC 2014 [17:28:58] RECOVERY - Puppet freshness on rdb1001 is OK: puppet ran at Sat Aug 16 17:28:54 UTC 2014 [17:29:38] PROBLEM - puppetmaster https on palladium is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 8140: HTTP/1.1 500 Internal Server Error [17:30:18] PROBLEM - Host ms-be1006 is DOWN: PING CRITICAL - Packet loss = 100% [17:30:38] RECOVERY - puppetmaster https on palladium is OK: HTTP OK: Status line output matched 400 - 335 bytes in 1.703 second response time [17:32:58] PROBLEM - Puppet freshness on cp1037 is CRITICAL: Last successful Puppet run was Sat 16 Aug 2014 15:32:43 UTC [17:33:39] RECOVERY - puppet last run on sanger is OK: OK: Puppet is currently enabled, last run 58 seconds ago with 0 failures [17:33:44] wth is going on with puppet? [17:33:58] PROBLEM - Puppet freshness on analytics1004 is CRITICAL: Last successful Puppet run was Sat 16 Aug 2014 15:32:48 UTC [17:36:08] RECOVERY - puppet last run on virt1008 is OK: OK: Puppet is currently enabled, last run 16 seconds ago with 0 failures [17:36:08] RECOVERY - puppet last run on palladium is OK: OK: Puppet is currently enabled, last run 11 seconds ago with 0 failures [17:36:18] RECOVERY - puppetmaster backend https on palladium is OK: HTTP OK: Status line output matched 400 - 335 bytes in 0.027 second response time [17:36:38] RECOVERY - puppet last run on mw1131 is OK: OK: Puppet is currently enabled, last run 17 seconds ago with 0 failures [17:36:38] RECOVERY - puppet last run on ms-be1010 is OK: OK: Puppet is currently enabled, last run 11 seconds ago with 0 failures [17:36:39] RECOVERY - puppet last run on mexia is OK: OK: Puppet is currently enabled, last run 29 seconds ago with 0 failures [17:36:48] RECOVERY - puppet last run on mc1015 is OK: OK: Puppet is currently enabled, last run 36 seconds ago with 0 failures [17:36:48] RECOVERY - puppet last run on ssl1003 is OK: OK: Puppet is currently enabled, last run 21 seconds ago with 0 failures [17:36:48] RECOVERY - puppet last run on dysprosium is OK: OK: Puppet is currently enabled, last run 54 seconds ago with 0 failures [17:36:48] RECOVERY - puppet last run on search1003 is OK: OK: Puppet is currently enabled, last run 10 seconds ago with 0 failures [17:36:49] RECOVERY - puppet last run on lvs4004 is OK: OK: Puppet is currently enabled, last run 44 seconds ago with 0 failures [17:37:08] RECOVERY - puppet last run on db1068 is OK: OK: Puppet is currently enabled, last run 2 seconds ago with 0 failures [17:37:08] RECOVERY - puppet last run on db1027 is OK: OK: Puppet is currently enabled, last run 6 seconds ago with 0 failures [17:37:09] RECOVERY - puppet last run on db1009 is OK: OK: Puppet is currently enabled, last run 1 seconds ago with 0 failures [17:37:09] RECOVERY - puppet last run on labsdb1005 is OK: OK: Puppet is currently enabled, last run 30 seconds ago with 0 failures [17:37:09] RECOVERY - puppet last run on analytics1021 is OK: OK: Puppet is currently enabled, last run 0 seconds ago with 0 failures [17:37:09] RECOVERY - puppet last run on analytics1012 is OK: OK: Puppet is currently enabled, last run 31 seconds ago with 0 failures [17:37:09] RECOVERY - puppet last run on mw1073 is OK: OK: Puppet is currently enabled, last run 2 seconds ago with 0 failures [17:37:09] RECOVERY - puppet last run on mw1047 is OK: OK: Puppet is currently enabled, last run 6 seconds ago with 0 failures [17:37:10] RECOVERY - puppet last run on mw1199 is OK: OK: Puppet is currently enabled, last run 1 seconds ago with 0 failures [17:37:18] RECOVERY - puppet last run on cp3019 is OK: OK: Puppet is currently enabled, last run 15 seconds ago with 0 failures [17:37:18] RECOVERY - puppet last run on cp1064 is OK: OK: Puppet is currently enabled, last run 7 seconds ago with 0 failures [17:37:18] RECOVERY - puppet last run on cp3021 is OK: OK: Puppet is currently enabled, last run 0 seconds ago with 0 failures [17:37:27] !log restarted apache2 on palladium... looks like something went horribly wrong with its puppet of itself that somehow killed off puppetmaster service? [17:37:29] RECOVERY - puppet last run on cp4016 is OK: OK: Puppet is currently enabled, last run 12 seconds ago with 0 failures [17:37:29] RECOVERY - puppet last run on cp1066 is OK: OK: Puppet is currently enabled, last run 26 seconds ago with 0 failures [17:37:33] Logged the message, Master [17:37:38] RECOVERY - puppet last run on virt1005 is OK: OK: Puppet is currently enabled, last run 18 seconds ago with 0 failures [17:37:38] RECOVERY - puppet last run on elastic1010 is OK: OK: Puppet is currently enabled, last run 32 seconds ago with 0 failures [17:37:38] RECOVERY - puppet last run on es1004 is OK: OK: Puppet is currently enabled, last run 57 seconds ago with 0 failures [17:37:38] RECOVERY - Puppet freshness on lvs3002 is OK: puppet ran at Sat Aug 16 17:37:33 UTC 2014 [17:37:38] RECOVERY - puppet last run on labsdb1002 is OK: OK: Puppet is currently enabled, last run 24 seconds ago with 0 failures [17:37:38] RECOVERY - puppet last run on search1019 is OK: OK: Puppet is currently enabled, last run 1 seconds ago with 0 failures [17:37:39] RECOVERY - puppet last run on fenari is OK: OK: Puppet is currently enabled, last run 37 seconds ago with 0 failures [17:37:39] RECOVERY - puppet last run on hydrogen is OK: OK: Puppet is currently enabled, last run 4 seconds ago with 0 failures [17:37:40] RECOVERY - puppet last run on zinc is OK: OK: Puppet is currently enabled, last run 39 seconds ago with 0 failures [17:37:49] RECOVERY - puppet last run on mw1085 is OK: OK: Puppet is currently enabled, last run 29 seconds ago with 0 failures [17:37:49] RECOVERY - puppet last run on ms-be1014 is OK: OK: Puppet is currently enabled, last run 44 seconds ago with 0 failures [17:37:49] RECOVERY - puppet last run on mw1128 is OK: OK: Puppet is currently enabled, last run 33 seconds ago with 0 failures [17:37:49] RECOVERY - puppet last run on search1008 is OK: OK: Puppet is currently enabled, last run 10 seconds ago with 0 failures [17:37:49] RECOVERY - puppet last run on db1049 is OK: OK: Puppet is currently enabled, last run 48 seconds ago with 0 failures [17:37:49] RECOVERY - puppet last run on mw1102 is OK: OK: Puppet is currently enabled, last run 8 seconds ago with 0 failures [17:37:49] RECOVERY - puppet last run on elastic1013 is OK: OK: Puppet is currently enabled, last run 51 seconds ago with 0 failures [17:37:50] RECOVERY - puppet last run on mw1137 is OK: OK: Puppet is currently enabled, last run 25 seconds ago with 0 failures [17:38:09] RECOVERY - puppet last run on mw1101 is OK: OK: Puppet is currently enabled, last run 26 seconds ago with 0 failures [17:38:09] RECOVERY - puppet last run on mw1070 is OK: OK: Puppet is currently enabled, last run 31 seconds ago with 0 failures [17:38:09] RECOVERY - puppet last run on mw1018 is OK: OK: Puppet is currently enabled, last run 50 seconds ago with 0 failures [17:38:09] RECOVERY - puppet last run on mw1095 is OK: OK: Puppet is currently enabled, last run 4 seconds ago with 0 failures [17:38:18] RECOVERY - puppet last run on cp4010 is OK: OK: Puppet is currently enabled, last run 39 seconds ago with 0 failures [17:38:18] RECOVERY - puppet last run on mw1157 is OK: OK: Puppet is currently enabled, last run 37 seconds ago with 0 failures [17:38:18] RECOVERY - puppet last run on cp3017 is OK: OK: Puppet is currently enabled, last run 38 seconds ago with 0 failures [17:38:18] RECOVERY - puppet last run on db1005 is OK: OK: Puppet is currently enabled, last run 37 seconds ago with 0 failures [17:38:18] RECOVERY - puppet last run on amssq50 is OK: OK: Puppet is currently enabled, last run 55 seconds ago with 0 failures [17:38:29] RECOVERY - puppet last run on labsdb1007 is OK: OK: Puppet is currently enabled, last run 2 seconds ago with 0 failures [17:38:38] RECOVERY - puppet last run on db1019 is OK: OK: Puppet is currently enabled, last run 3 seconds ago with 0 failures [17:38:38] RECOVERY - puppet last run on nickel is OK: OK: Puppet is currently enabled, last run 2 seconds ago with 0 failures [17:38:38] RECOVERY - puppet last run on elastic1016 is OK: OK: Puppet is currently enabled, last run 1 seconds ago with 0 failures [17:38:38] RECOVERY - puppet last run on lanthanum is OK: OK: Puppet is currently enabled, last run 54 seconds ago with 0 failures [17:38:38] RECOVERY - puppet last run on elastic1009 is OK: OK: Puppet is currently enabled, last run 55 seconds ago with 0 failures [17:38:38] RECOVERY - puppet last run on ocg1002 is OK: OK: Puppet is currently enabled, last run 8 seconds ago with 0 failures [17:38:39] RECOVERY - puppet last run on mw1015 is OK: OK: Puppet is currently enabled, last run 55 seconds ago with 0 failures [17:38:48] RECOVERY - puppet last run on mw1214 is OK: OK: Puppet is currently enabled, last run 1 seconds ago with 0 failures [17:38:48] RECOVERY - puppet last run on mw1179 is OK: OK: Puppet is currently enabled, last run 60 seconds ago with 0 failures [17:38:48] RECOVERY - puppet last run on analytics1029 is OK: OK: Puppet is currently enabled, last run 12 seconds ago with 0 failures [17:38:48] RECOVERY - puppet last run on mw1058 is OK: OK: Puppet is currently enabled, last run 19 seconds ago with 0 failures [17:38:48] RECOVERY - puppet last run on mw1019 is OK: OK: Puppet is currently enabled, last run 7 seconds ago with 0 failures [17:38:48] RECOVERY - puppet last run on mw1020 is OK: OK: Puppet is currently enabled, last run 11 seconds ago with 0 failures [17:38:48] RECOVERY - puppet last run on erbium is OK: OK: Puppet is currently enabled, last run 9 seconds ago with 0 failures [17:38:49] RECOVERY - puppet last run on search1021 is OK: OK: Puppet is currently enabled, last run 12 seconds ago with 0 failures [17:38:58] RECOVERY - puppet last run on ms-be3004 is OK: OK: Puppet is currently enabled, last run 58 seconds ago with 0 failures [17:38:58] RECOVERY - puppet last run on amssq45 is OK: OK: Puppet is currently enabled, last run 2 seconds ago with 0 failures [17:38:58] RECOVERY - puppet last run on cp3007 is OK: OK: Puppet is currently enabled, last run 15 seconds ago with 0 failures [17:39:08] RECOVERY - puppet last run on mw1182 is OK: OK: Puppet is currently enabled, last run 20 seconds ago with 0 failures [17:39:08] RECOVERY - puppet last run on mw1094 is OK: OK: Puppet is currently enabled, last run 27 seconds ago with 0 failures [17:39:09] RECOVERY - puppet last run on mw1017 is OK: OK: Puppet is currently enabled, last run 34 seconds ago with 0 failures [17:39:09] RECOVERY - puppet last run on cp4013 is OK: OK: Puppet is currently enabled, last run 1 seconds ago with 0 failures [17:39:09] RECOVERY - puppet last run on cp4015 is OK: OK: Puppet is currently enabled, last run 38 seconds ago with 0 failures [17:39:18] RECOVERY - puppet last run on amssq52 is OK: OK: Puppet is currently enabled, last run 36 seconds ago with 0 failures [17:39:18] RECOVERY - puppet last run on lvs3002 is OK: OK: Puppet is currently enabled, last run 37 seconds ago with 0 failures [17:39:18] RECOVERY - puppet last run on analytics1024 is OK: OK: Puppet is currently enabled, last run 44 seconds ago with 0 failures [17:39:18] RECOVERY - puppet last run on mw1075 is OK: OK: Puppet is currently enabled, last run 35 seconds ago with 0 failures [17:39:28] RECOVERY - puppet last run on tmh1002 is OK: OK: Puppet is currently enabled, last run 47 seconds ago with 0 failures [17:39:28] RECOVERY - puppet last run on mw1184 is OK: OK: Puppet is currently enabled, last run 12 seconds ago with 0 failures [17:39:29] RECOVERY - puppet last run on wtp1014 is OK: OK: Puppet is currently enabled, last run 57 seconds ago with 0 failures [17:39:29] RECOVERY - puppet last run on db1058 is OK: OK: Puppet is currently enabled, last run 18 seconds ago with 0 failures [17:39:29] RECOVERY - puppet last run on ssl1004 is OK: OK: Puppet is currently enabled, last run 56 seconds ago with 0 failures [17:39:29] RECOVERY - puppet last run on cp1057 is OK: OK: Puppet is currently enabled, last run 55 seconds ago with 0 failures [17:39:29] RECOVERY - puppet last run on es1005 is OK: OK: Puppet is currently enabled, last run 34 seconds ago with 0 failures [17:39:38] RECOVERY - puppet last run on mw1136 is OK: OK: Puppet is currently enabled, last run 49 seconds ago with 0 failures [17:39:38] RECOVERY - puppet last run on mw1078 is OK: OK: Puppet is currently enabled, last run 57 seconds ago with 0 failures [17:39:38] RECOVERY - puppet last run on mw1127 is OK: OK: Puppet is currently enabled, last run 22 seconds ago with 0 failures [17:39:39] RECOVERY - puppet last run on analytics1019 is OK: OK: Puppet is currently enabled, last run 15 seconds ago with 0 failures [17:39:39] RECOVERY - puppet last run on praseodymium is OK: OK: Puppet is currently enabled, last run 50 seconds ago with 0 failures [17:39:48] RECOVERY - puppet last run on mw1169 is OK: OK: Puppet is currently enabled, last run 59 seconds ago with 0 failures [17:39:48] RECOVERY - puppet last run on virt0 is OK: OK: Puppet is currently enabled, last run 54 seconds ago with 0 failures [17:39:48] RECOVERY - puppet last run on cp1065 is OK: OK: Puppet is currently enabled, last run 6 seconds ago with 0 failures [17:39:48] RECOVERY - puppet last run on cp1069 is OK: OK: Puppet is currently enabled, last run 7 seconds ago with 0 failures [17:39:48] RECOVERY - puppet last run on zirconium is OK: OK: Puppet is currently enabled, last run 38 seconds ago with 0 failures [17:39:48] RECOVERY - puppet last run on db1024 is OK: OK: Puppet is currently enabled, last run 26 seconds ago with 0 failures [17:39:48] RECOVERY - puppet last run on mw1083 is OK: OK: Puppet is currently enabled, last run 19 seconds ago with 0 failures [17:39:49] RECOVERY - puppet last run on db1041 is OK: OK: Puppet is currently enabled, last run 29 seconds ago with 0 failures [17:39:49] RECOVERY - puppet last run on lvs1006 is OK: OK: Puppet is currently enabled, last run 26 seconds ago with 0 failures [17:39:50] RECOVERY - puppet last run on cp1051 is OK: OK: Puppet is currently enabled, last run 36 seconds ago with 0 failures [17:39:50] RECOVERY - puppet last run on lvs4001 is OK: OK: Puppet is currently enabled, last run 37 seconds ago with 0 failures [17:39:51] RECOVERY - puppet last run on cp4017 is OK: OK: Puppet is currently enabled, last run 46 seconds ago with 0 failures [17:40:08] RECOVERY - puppet last run on es1003 is OK: OK: Puppet is currently enabled, last run 53 seconds ago with 0 failures [17:40:09] RECOVERY - puppet last run on mw1013 is OK: OK: Puppet is currently enabled, last run 23 seconds ago with 0 failures [17:40:09] RECOVERY - puppet last run on neon is OK: OK: Puppet is currently enabled, last run 29 seconds ago with 0 failures [17:40:09] RECOVERY - puppet last run on mw1036 is OK: OK: Puppet is currently enabled, last run 6 seconds ago with 0 failures [17:40:09] RECOVERY - puppet last run on search1009 is OK: OK: Puppet is currently enabled, last run 41 seconds ago with 0 failures [17:40:18] RECOVERY - puppet last run on mc1008 is OK: OK: Puppet is currently enabled, last run 46 seconds ago with 0 failures [17:40:28] RECOVERY - puppet last run on analytics1036 is OK: OK: Puppet is currently enabled, last run 18 seconds ago with 0 failures [17:40:29] RECOVERY - puppet last run on analytics1034 is OK: OK: Puppet is currently enabled, last run 24 seconds ago with 0 failures [17:40:38] RECOVERY - puppet last run on mw1096 is OK: OK: Puppet is currently enabled, last run 3 seconds ago with 0 failures [17:40:38] RECOVERY - puppet last run on search1014 is OK: OK: Puppet is currently enabled, last run 30 seconds ago with 0 failures [17:40:39] RECOVERY - puppet last run on es10 is OK: OK: Puppet is currently enabled, last run 27 seconds ago with 0 failures [17:40:39] RECOVERY - puppet last run on es7 is OK: OK: Puppet is currently enabled, last run 12 seconds ago with 0 failures [17:40:39] RECOVERY - puppet last run on wtp1024 is OK: OK: Puppet is currently enabled, last run 30 seconds ago with 0 failures [17:40:48] RECOVERY - puppet last run on ytterbium is OK: OK: Puppet is currently enabled, last run 1 seconds ago with 0 failures [17:40:48] RECOVERY - puppet last run on wtp1021 is OK: OK: Puppet is currently enabled, last run 13 seconds ago with 0 failures [17:40:48] RECOVERY - puppet last run on mw1216 is OK: OK: Puppet is currently enabled, last run 8 seconds ago with 0 failures [17:40:48] RECOVERY - puppet last run on mc1010 is OK: OK: Puppet is currently enabled, last run 17 seconds ago with 0 failures [17:40:48] RECOVERY - puppet last run on analytics1039 is OK: OK: Puppet is currently enabled, last run 4 seconds ago with 0 failures [17:40:49] RECOVERY - puppet last run on mw1035 is OK: OK: Puppet is currently enabled, last run 8 seconds ago with 0 failures [17:40:49] RECOVERY - puppet last run on mw1040 is OK: OK: Puppet is currently enabled, last run 10 seconds ago with 0 failures [17:40:49] RECOVERY - puppet last run on es1006 is OK: OK: Puppet is currently enabled, last run 6 seconds ago with 0 failures [17:40:49] RECOVERY - puppet last run on db1007 is OK: OK: Puppet is currently enabled, last run 12 seconds ago with 0 failures [17:40:58] RECOVERY - puppet last run on amslvs4 is OK: OK: Puppet is currently enabled, last run 45 seconds ago with 0 failures [17:40:58] RECOVERY - puppet last run on amssq58 is OK: OK: Puppet is currently enabled, last run 18 seconds ago with 0 failures [17:40:58] RECOVERY - puppet last run on amssq57 is OK: OK: Puppet is currently enabled, last run 19 seconds ago with 0 failures [17:41:08] RECOVERY - puppet last run on mw1138 is OK: OK: Puppet is currently enabled, last run 26 seconds ago with 0 failures [17:41:08] RECOVERY - puppet last run on ms1001 is OK: OK: Puppet is currently enabled, last run 24 seconds ago with 0 failures [17:41:08] RECOVERY - puppet last run on analytics1015 is OK: OK: Puppet is currently enabled, last run 35 seconds ago with 0 failures [17:41:09] RECOVERY - puppet last run on mw1161 is OK: OK: Puppet is currently enabled, last run 32 seconds ago with 0 failures [17:41:09] RECOVERY - puppet last run on mw1196 is OK: OK: Puppet is currently enabled, last run 31 seconds ago with 0 failures [17:41:09] RECOVERY - puppet last run on ms-fe1003 is OK: OK: Puppet is currently enabled, last run 23 seconds ago with 0 failures [17:41:09] RECOVERY - puppet last run on mw1191 is OK: OK: Puppet is currently enabled, last run 36 seconds ago with 0 failures [17:41:18] RECOVERY - puppet last run on cp1043 is OK: OK: Puppet is currently enabled, last run 51 seconds ago with 0 failures [17:41:29] RECOVERY - puppet last run on virt1002 is OK: OK: Puppet is currently enabled, last run 60 seconds ago with 0 failures [17:41:29] RECOVERY - puppet last run on mw1109 is OK: OK: Puppet is currently enabled, last run 21 seconds ago with 0 failures [17:41:29] RECOVERY - puppet last run on es1009 is OK: OK: Puppet is currently enabled, last run 42 seconds ago with 0 failures [17:41:29] RECOVERY - puppet last run on mw1062 is OK: OK: Puppet is currently enabled, last run 8 seconds ago with 0 failures [17:41:38] RECOVERY - puppet last run on wtp1017 is OK: OK: Puppet is currently enabled, last run 37 seconds ago with 0 failures [17:41:38] RECOVERY - puppet last run on mw1147 is OK: OK: Puppet is currently enabled, last run 44 seconds ago with 0 failures [17:41:38] RECOVERY - puppet last run on rdb1004 is OK: OK: Puppet is currently enabled, last run 16 seconds ago with 0 failures [17:41:38] RECOVERY - puppet last run on mw1132 is OK: OK: Puppet is currently enabled, last run 44 seconds ago with 0 failures [17:41:38] RECOVERY - puppet last run on db1010 is OK: OK: Puppet is currently enabled, last run 50 seconds ago with 0 failures [17:41:38] RECOVERY - puppet last run on ocg1001 is OK: OK: Puppet is currently enabled, last run 50 seconds ago with 0 failures [17:41:39] RECOVERY - puppet last run on tarin is OK: OK: Puppet is currently enabled, last run 57 seconds ago with 0 failures [17:41:48] RECOVERY - puppet last run on mc1009 is OK: OK: Puppet is currently enabled, last run 59 seconds ago with 0 failures [17:41:48] RECOVERY - puppet last run on mw1218 is OK: OK: Puppet is currently enabled, last run 34 seconds ago with 0 failures [17:41:48] RECOVERY - puppet last run on cp4007 is OK: OK: Puppet is currently enabled, last run 40 seconds ago with 0 failures [17:41:48] RECOVERY - puppet last run on dbstore1001 is OK: OK: Puppet is currently enabled, last run 54 seconds ago with 0 failures [17:41:48] RECOVERY - puppet last run on ms-be1001 is OK: OK: Puppet is currently enabled, last run 57 seconds ago with 0 failures [17:41:48] RECOVERY - puppet last run on mw1124 is OK: OK: Puppet is currently enabled, last run 44 seconds ago with 0 failures [17:41:58] RECOVERY - puppet last run on cp3013 is OK: OK: Puppet is currently enabled, last run 58 seconds ago with 0 failures [17:41:58] RECOVERY - puppet last run on cp3022 is OK: OK: Puppet is currently enabled, last run 59 seconds ago with 0 failures [17:41:58] RECOVERY - puppet last run on ssl3003 is OK: OK: Puppet is currently enabled, last run 41 seconds ago with 0 failures [17:42:08] RECOVERY - puppet last run on labsdb1001 is OK: OK: Puppet is currently enabled, last run 47 seconds ago with 0 failures [17:42:08] RECOVERY - puppet last run on pc1001 is OK: OK: Puppet is currently enabled, last run 31 seconds ago with 0 failures [17:42:08] RECOVERY - puppet last run on ms-be1013 is OK: OK: Puppet is currently enabled, last run 2 seconds ago with 0 failures [17:42:08] RECOVERY - puppet last run on mw1072 is OK: OK: Puppet is currently enabled, last run 39 seconds ago with 0 failures [17:42:09] RECOVERY - puppet last run on mw1115 is OK: OK: Puppet is currently enabled, last run 25 seconds ago with 0 failures [17:42:09] RECOVERY - puppet last run on mc1004 is OK: OK: Puppet is currently enabled, last run 57 seconds ago with 0 failures [17:42:09] RECOVERY - puppet last run on wtp1019 is OK: OK: Puppet is currently enabled, last run 54 seconds ago with 0 failures [17:42:18] RECOVERY - puppet last run on dbproxy1002 is OK: OK: Puppet is currently enabled, last run 0 seconds ago with 0 failures [17:42:18] RECOVERY - puppet last run on carbon is OK: OK: Puppet is currently enabled, last run 7 seconds ago with 0 failures [17:42:28] RECOVERY - puppet last run on mw1089 is OK: OK: Puppet is currently enabled, last run 49 seconds ago with 0 failures [17:42:29] RECOVERY - puppet last run on mw1038 is OK: OK: Puppet is currently enabled, last run 15 seconds ago with 0 failures [17:42:29] RECOVERY - puppet last run on search1020 is OK: OK: Puppet is currently enabled, last run 20 seconds ago with 0 failures [17:42:29] RECOVERY - puppet last run on logstash1003 is OK: OK: Puppet is currently enabled, last run 49 seconds ago with 0 failures [17:42:38] RECOVERY - puppet last run on ms-be1004 is OK: OK: Puppet is currently enabled, last run 25 seconds ago with 0 failures [17:42:38] RECOVERY - puppet last run on mw1005 is OK: OK: Puppet is currently enabled, last run 31 seconds ago with 0 failures [17:42:38] RECOVERY - puppet last run on analytics1017 is OK: OK: Puppet is currently enabled, last run 23 seconds ago with 0 failures [17:42:39] RECOVERY - puppet last run on wtp1009 is OK: OK: Puppet is currently enabled, last run 2 seconds ago with 0 failures [17:42:48] RECOVERY - puppet last run on db1053 is OK: OK: Puppet is currently enabled, last run 29 seconds ago with 0 failures [17:42:48] RECOVERY - puppet last run on vanadium is OK: OK: Puppet is currently enabled, last run 39 seconds ago with 0 failures [17:42:48] RECOVERY - puppet last run on cp1059 is OK: OK: Puppet is currently enabled, last run 41 seconds ago with 0 failures [17:42:48] RECOVERY - puppet last run on mw1028 is OK: OK: Puppet is currently enabled, last run 2 seconds ago with 0 failures [17:42:48] RECOVERY - puppet last run on mw1178 is OK: OK: Puppet is currently enabled, last run 31 seconds ago with 0 failures [17:42:48] RECOVERY - puppet last run on cp1070 is OK: OK: Puppet is currently enabled, last run 30 seconds ago with 0 failures [17:42:48] RECOVERY - puppet last run on es1001 is OK: OK: Puppet is currently enabled, last run 10 seconds ago with 0 failures [17:42:49] RECOVERY - puppet last run on mw1031 is OK: OK: Puppet is currently enabled, last run 9 seconds ago with 0 failures [17:42:58] RECOVERY - puppet last run on amssq59 is OK: OK: Puppet is currently enabled, last run 21 seconds ago with 0 failures [17:42:58] RECOVERY - puppet last run on magnesium is OK: OK: Puppet is currently enabled, last run 19 seconds ago with 0 failures [17:42:58] RECOVERY - puppet last run on cp3006 is OK: OK: Puppet is currently enabled, last run 50 seconds ago with 0 failures [17:43:08] RECOVERY - puppet last run on db1065 is OK: OK: Puppet is currently enabled, last run 24 seconds ago with 0 failures [17:43:08] RECOVERY - Puppet freshness on cp1047 is OK: puppet ran at Sat Aug 16 17:43:03 UTC 2014 [17:43:08] RECOVERY - puppet last run on mc1011 is OK: OK: Puppet is currently enabled, last run 58 seconds ago with 0 failures [17:43:09] RECOVERY - puppet last run on mc1016 is OK: OK: Puppet is currently enabled, last run 24 seconds ago with 0 failures [17:43:09] RECOVERY - puppet last run on mw1106 is OK: OK: Puppet is currently enabled, last run 17 seconds ago with 0 failures [17:43:09] RECOVERY - puppet last run on mw1045 is OK: OK: Puppet is currently enabled, last run 15 seconds ago with 0 failures [17:43:09] RECOVERY - puppet last run on mw1067 is OK: OK: Puppet is currently enabled, last run 29 seconds ago with 0 failures [17:43:10] RECOVERY - puppet last run on mw1192 is OK: OK: Puppet is currently enabled, last run 23 seconds ago with 0 failures [17:43:10] RECOVERY - puppet last run on netmon1001 is OK: OK: Puppet is currently enabled, last run 54 seconds ago with 0 failures [17:43:11] RECOVERY - puppet last run on db1017 is OK: OK: Puppet is currently enabled, last run 25 seconds ago with 0 failures [17:43:11] RECOVERY - puppet last run on mw1134 is OK: OK: Puppet is currently enabled, last run 25 seconds ago with 0 failures [17:43:18] RECOVERY - puppet last run on cp3020 is OK: OK: Puppet is currently enabled, last run 13 seconds ago with 0 failures [17:43:18] RECOVERY - puppet last run on amssq44 is OK: OK: Puppet is currently enabled, last run 52 seconds ago with 0 failures [17:43:18] RECOVERY - puppet last run on cp3015 is OK: OK: Puppet is currently enabled, last run 27 seconds ago with 0 failures [17:43:18] RECOVERY - puppet last run on amssq43 is OK: OK: Puppet is currently enabled, last run 6 seconds ago with 0 failures [17:43:18] RECOVERY - puppet last run on db1029 is OK: OK: Puppet is currently enabled, last run 33 seconds ago with 0 failures [17:43:18] RECOVERY - puppet last run on rbf1002 is OK: OK: Puppet is currently enabled, last run 28 seconds ago with 0 failures [17:43:18] RECOVERY - puppet last run on amssq54 is OK: OK: Puppet is currently enabled, last run 3 seconds ago with 0 failures [17:43:19] RECOVERY - puppet last run on cp1049 is OK: OK: Puppet is currently enabled, last run 33 seconds ago with 0 failures [17:43:28] RECOVERY - puppet last run on mw1063 is OK: OK: Puppet is currently enabled, last run 13 seconds ago with 0 failures [17:43:29] RECOVERY - puppet last run on wtp1006 is OK: OK: Puppet is currently enabled, last run 14 seconds ago with 0 failures [17:43:29] RECOVERY - puppet last run on ms-be1003 is OK: OK: Puppet is currently enabled, last run 8 seconds ago with 0 failures [17:43:29] RECOVERY - puppet last run on mw1048 is OK: OK: Puppet is currently enabled, last run 48 seconds ago with 0 failures [17:43:29] RECOVERY - puppet last run on mw1141 is OK: OK: Puppet is currently enabled, last run 38 seconds ago with 0 failures [17:43:29] RECOVERY - puppet last run on elastic1004 is OK: OK: Puppet is currently enabled, last run 21 seconds ago with 0 failures [17:43:38] RECOVERY - puppet last run on mw1006 is OK: OK: Puppet is currently enabled, last run 15 seconds ago with 0 failures [17:43:38] RECOVERY - puppet last run on search1016 is OK: OK: Puppet is currently enabled, last run 17 seconds ago with 0 failures [17:43:38] RECOVERY - puppet last run on mw1197 is OK: OK: Puppet is currently enabled, last run 12 seconds ago with 0 failures [17:43:38] RECOVERY - puppet last run on mw1080 is OK: OK: Puppet is currently enabled, last run 52 seconds ago with 0 failures [17:43:38] RECOVERY - puppet last run on gold is OK: OK: Puppet is currently enabled, last run 18 seconds ago with 0 failures [17:43:38] RECOVERY - puppet last run on platinum is OK: OK: Puppet is currently enabled, last run 23 seconds ago with 0 failures [17:43:39] RECOVERY - puppet last run on db73 is OK: OK: Puppet is currently enabled, last run 29 seconds ago with 0 failures [17:43:39] RECOVERY - puppet last run on potassium is OK: OK: Puppet is currently enabled, last run 18 seconds ago with 0 failures [17:43:40] RECOVERY - puppet last run on logstash1001 is OK: OK: Puppet is currently enabled, last run 48 seconds ago with 0 failures [17:43:48] RECOVERY - puppet last run on db1073 is OK: OK: Puppet is currently enabled, last run 32 seconds ago with 0 failures [17:43:49] RECOVERY - puppet last run on analytics1041 is OK: OK: Puppet is currently enabled, last run 34 seconds ago with 0 failures [17:43:49] RECOVERY - puppet last run on mw1174 is OK: OK: Puppet is currently enabled, last run 23 seconds ago with 0 failures [17:43:49] RECOVERY - puppet last run on cp1047 is OK: OK: Puppet is currently enabled, last run 24 seconds ago with 0 failures [17:43:49] RECOVERY - puppet last run on analytics1040 is OK: OK: Puppet is currently enabled, last run 7 seconds ago with 0 failures [17:43:49] RECOVERY - puppet last run on mw1059 is OK: OK: Puppet is currently enabled, last run 34 seconds ago with 0 failures [17:43:49] RECOVERY - puppet last run on es1008 is OK: OK: Puppet is currently enabled, last run 9 seconds ago with 0 failures [17:43:50] RECOVERY - puppet last run on cp1055 is OK: OK: Puppet is currently enabled, last run 35 seconds ago with 0 failures [17:43:50] RECOVERY - puppet last run on db1031 is OK: OK: Puppet is currently enabled, last run 39 seconds ago with 0 failures [17:43:51] RECOVERY - puppet last run on cp1039 is OK: OK: Puppet is currently enabled, last run 56 seconds ago with 0 failures [17:43:51] RECOVERY - puppet last run on elastic1001 is OK: OK: Puppet is currently enabled, last run 8 seconds ago with 0 failures [17:43:52] RECOVERY - puppet last run on analytics1020 is OK: OK: Puppet is currently enabled, last run 29 seconds ago with 0 failures [17:43:52] RECOVERY - puppet last run on xenon is OK: OK: Puppet is currently enabled, last run 4 seconds ago with 0 failures [17:43:53] RECOVERY - puppet last run on lvs4002 is OK: OK: Puppet is currently enabled, last run 28 seconds ago with 0 failures [17:43:58] RECOVERY - puppet last run on amssq49 is OK: OK: Puppet is currently enabled, last run 38 seconds ago with 0 failures [17:43:58] RECOVERY - puppet last run on amslvs2 is OK: OK: Puppet is currently enabled, last run 10 seconds ago with 0 failures [17:43:58] RECOVERY - puppet last run on cp3008 is OK: OK: Puppet is currently enabled, last run 6 seconds ago with 0 failures [17:43:58] RECOVERY - puppet last run on caesium is OK: OK: Puppet is currently enabled, last run 40 seconds ago with 0 failures [17:44:08] RECOVERY - puppet last run on analytics1025 is OK: OK: Puppet is currently enabled, last run 50 seconds ago with 0 failures [17:44:08] RECOVERY - puppet last run on helium is OK: OK: Puppet is currently enabled, last run 15 seconds ago with 0 failures [17:44:08] RECOVERY - puppet last run on mw1007 is OK: OK: Puppet is currently enabled, last run 48 seconds ago with 0 failures [17:44:09] RECOVERY - puppet last run on analytics1033 is OK: OK: Puppet is currently enabled, last run 57 seconds ago with 0 failures [17:44:09] RECOVERY - puppet last run on elastic1007 is OK: OK: Puppet is currently enabled, last run 48 seconds ago with 0 failures [17:44:09] RECOVERY - puppet last run on elastic1008 is OK: OK: Puppet is currently enabled, last run 16 seconds ago with 0 failures [17:44:09] RECOVERY - puppet last run on mw1082 is OK: OK: Puppet is currently enabled, last run 3 seconds ago with 0 failures [17:44:10] RECOVERY - puppet last run on mw1145 is OK: OK: Puppet is currently enabled, last run 22 seconds ago with 0 failures [17:44:10] RECOVERY - puppet last run on wtp1020 is OK: OK: Puppet is currently enabled, last run 48 seconds ago with 0 failures [17:44:11] RECOVERY - puppet last run on wtp1016 is OK: OK: Puppet is currently enabled, last run 1 seconds ago with 0 failures [17:44:11] RECOVERY - puppet last run on mw1041 is OK: OK: Puppet is currently enabled, last run 55 seconds ago with 0 failures [17:44:12] RECOVERY - puppet last run on ms1004 is OK: OK: Puppet is currently enabled, last run 3 seconds ago with 0 failures [17:44:12] RECOVERY - puppet last run on mw1160 is OK: OK: Puppet is currently enabled, last run 1 seconds ago with 0 failures [17:44:13] RECOVERY - puppet last run on mw1140 is OK: OK: Puppet is currently enabled, last run 54 seconds ago with 0 failures [17:44:13] RECOVERY - puppet last run on mw1026 is OK: OK: Puppet is currently enabled, last run 19 seconds ago with 0 failures [17:44:14] RECOVERY - puppet last run on db74 is OK: OK: Puppet is currently enabled, last run 5 seconds ago with 0 failures [17:44:18] RECOVERY - puppet last run on mc1006 is OK: OK: Puppet is currently enabled, last run 58 seconds ago with 0 failures [17:44:18] RECOVERY - puppet last run on analytics1035 is OK: OK: Puppet is currently enabled, last run 25 seconds ago with 0 failures [17:44:18] RECOVERY - puppet last run on lvs1002 is OK: OK: Puppet is currently enabled, last run 8 seconds ago with 0 failures [17:44:18] RECOVERY - Puppet freshness on lead is OK: puppet ran at Sat Aug 16 17:44:14 UTC 2014 [17:44:18] RECOVERY - puppet last run on cp4006 is OK: OK: Puppet is currently enabled, last run 17 seconds ago with 0 failures [17:44:29] RECOVERY - puppet last run on snapshot1003 is OK: OK: Puppet is currently enabled, last run 18 seconds ago with 0 failures [17:44:38] RECOVERY - puppet last run on mw1012 is OK: OK: Puppet is currently enabled, last run 49 seconds ago with 0 failures [17:44:39] RECOVERY - puppet last run on cerium is OK: OK: Puppet is currently enabled, last run 52 seconds ago with 0 failures [17:44:39] RECOVERY - puppet last run on mw1009 is OK: OK: Puppet is currently enabled, last run 7 seconds ago with 0 failures [17:44:39] RECOVERY - puppet last run on db1066 is OK: OK: Puppet is currently enabled, last run 32 seconds ago with 0 failures [17:44:39] RECOVERY - puppet last run on mw1200 is OK: OK: Puppet is currently enabled, last run 50 seconds ago with 0 failures [17:44:39] RECOVERY - puppet last run on ssl1002 is OK: OK: Puppet is currently enabled, last run 53 seconds ago with 0 failures [17:44:48] RECOVERY - puppet last run on mw1187 is OK: OK: Puppet is currently enabled, last run 59 seconds ago with 0 failures [17:44:48] RECOVERY - puppet last run on mw1046 is OK: OK: Puppet is currently enabled, last run 24 seconds ago with 0 failures [17:44:48] RECOVERY - puppet last run on mw1117 is OK: OK: Puppet is currently enabled, last run 20 seconds ago with 0 failures [17:44:49] RECOVERY - puppet last run on db1050 is OK: OK: Puppet is currently enabled, last run 21 seconds ago with 0 failures [17:44:49] RECOVERY - puppet last run on elastic1012 is OK: OK: Puppet is currently enabled, last run 44 seconds ago with 0 failures [17:44:49] RECOVERY - puppet last run on lead is OK: OK: Puppet is currently enabled, last run 26 seconds ago with 0 failures [17:44:58] RECOVERY - puppet last run on amssq61 is OK: OK: Puppet is currently enabled, last run 33 seconds ago with 0 failures [17:44:58] RECOVERY - puppet last run on amssq32 is OK: OK: Puppet is currently enabled, last run 38 seconds ago with 0 failures [17:45:08] RECOVERY - puppet last run on mw1100 is OK: OK: Puppet is currently enabled, last run 28 seconds ago with 0 failures [17:45:08] RECOVERY - puppet last run on mc1003 is OK: OK: Puppet is currently enabled, last run 39 seconds ago with 0 failures [17:45:09] RECOVERY - puppet last run on mc1002 is OK: OK: Puppet is currently enabled, last run 40 seconds ago with 0 failures [17:45:09] RECOVERY - puppet last run on db1022 is OK: OK: Puppet is currently enabled, last run 53 seconds ago with 0 failures [17:45:09] RECOVERY - puppet last run on searchidx1001 is OK: OK: Puppet is currently enabled, last run 57 seconds ago with 0 failures [17:45:09] RECOVERY - puppet last run on ms-fe1001 is OK: OK: Puppet is currently enabled, last run 50 seconds ago with 0 failures [17:45:09] RECOVERY - puppet last run on mw1060 is OK: OK: Puppet is currently enabled, last run 45 seconds ago with 0 failures [17:45:18] RECOVERY - puppet last run on amssq53 is OK: OK: Puppet is currently enabled, last run 35 seconds ago with 0 failures [17:45:28] RECOVERY - puppet last run on db1059 is OK: OK: Puppet is currently enabled, last run 1 seconds ago with 0 failures [17:45:29] RECOVERY - puppet last run on cp1056 is OK: OK: Puppet is currently enabled, last run 2 seconds ago with 0 failures [17:45:29] RECOVERY - puppet last run on labsdb1003 is OK: OK: Puppet is currently enabled, last run 54 seconds ago with 0 failures [17:45:29] RECOVERY - puppet last run on ms-fe1004 is OK: OK: Puppet is currently enabled, last run 3 seconds ago with 0 failures [17:45:39] RECOVERY - puppet last run on iron is OK: OK: Puppet is currently enabled, last run 13 seconds ago with 0 failures [17:45:39] RECOVERY - puppet last run on mw1173 is OK: OK: Puppet is currently enabled, last run 12 seconds ago with 0 failures [17:45:48] RECOVERY - puppet last run on mw1176 is OK: OK: Puppet is currently enabled, last run 60 seconds ago with 0 failures [17:45:48] RECOVERY - puppet last run on search1010 is OK: OK: Puppet is currently enabled, last run 21 seconds ago with 0 failures [17:45:48] RECOVERY - puppet last run on db1015 is OK: OK: Puppet is currently enabled, last run 20 seconds ago with 0 failures [17:45:48] RECOVERY - puppet last run on mw1003 is OK: OK: Puppet is currently enabled, last run 17 seconds ago with 0 failures [17:45:58] RECOVERY - puppet last run on lvs3001 is OK: OK: Puppet is currently enabled, last run 0 seconds ago with 0 failures [17:45:58] RECOVERY - puppet last run on mw1069 is OK: OK: Puppet is currently enabled, last run 24 seconds ago with 0 failures [17:45:58] RECOVERY - puppet last run on mw1150 is OK: OK: Puppet is currently enabled, last run 35 seconds ago with 0 failures [17:45:58] RECOVERY - Puppet freshness on db1018 is OK: puppet ran at Sat Aug 16 17:45:55 UTC 2014 [17:46:08] RECOVERY - puppet last run on db1051 is OK: OK: Puppet is currently enabled, last run 4 seconds ago with 0 failures [17:46:08] RECOVERY - puppet last run on mw1068 is OK: OK: Puppet is currently enabled, last run 41 seconds ago with 0 failures [17:46:09] RECOVERY - puppet last run on mw1164 is OK: OK: Puppet is currently enabled, last run 8 seconds ago with 0 failures [17:46:09] RECOVERY - puppet last run on mw1217 is OK: OK: Puppet is currently enabled, last run 36 seconds ago with 0 failures [17:46:09] RECOVERY - puppet last run on mw1166 is OK: OK: Puppet is currently enabled, last run 8 seconds ago with 0 failures [17:46:09] RECOVERY - puppet last run on mw1061 is OK: OK: Puppet is currently enabled, last run 4 seconds ago with 0 failures [17:46:09] RECOVERY - puppet last run on db1046 is OK: OK: Puppet is currently enabled, last run 32 seconds ago with 0 failures [17:46:10] RECOVERY - puppet last run on db1023 is OK: OK: Puppet is currently enabled, last run 5 seconds ago with 0 failures [17:46:10] RECOVERY - puppet last run on lvs1005 is OK: OK: Puppet is currently enabled, last run 35 seconds ago with 0 failures [17:46:11] RECOVERY - puppet last run on search1018 is OK: OK: Puppet is currently enabled, last run 37 seconds ago with 0 failures [17:46:18] RECOVERY - puppet last run on amssq35 is OK: OK: Puppet is currently enabled, last run 16 seconds ago with 0 failures [17:46:18] RECOVERY - puppet last run on cp3003 is OK: OK: Puppet is currently enabled, last run 7 seconds ago with 0 failures [17:46:18] RECOVERY - puppet last run on mw1119 is OK: OK: Puppet is currently enabled, last run 11 seconds ago with 0 failures [17:46:18] RECOVERY - puppet last run on mw1120 is OK: OK: Puppet is currently enabled, last run 16 seconds ago with 0 failures [17:46:18] RECOVERY - puppet last run on mw1189 is OK: OK: Puppet is currently enabled, last run 14 seconds ago with 0 failures [17:46:18] RECOVERY - puppet last run on mw1153 is OK: OK: Puppet is currently enabled, last run 43 seconds ago with 0 failures [17:46:18] RECOVERY - puppet last run on mw1099 is OK: OK: Puppet is currently enabled, last run 43 seconds ago with 0 failures [17:46:28] RECOVERY - puppet last run on db1002 is OK: OK: Puppet is currently enabled, last run 57 seconds ago with 0 failures [17:46:28] RECOVERY - puppet last run on mw1123 is OK: OK: Puppet is currently enabled, last run 27 seconds ago with 0 failures [17:46:29] RECOVERY - puppet last run on db1034 is OK: OK: Puppet is currently enabled, last run 28 seconds ago with 0 failures [17:46:29] RECOVERY - puppet last run on mw1092 is OK: OK: Puppet is currently enabled, last run 28 seconds ago with 0 failures [17:46:29] RECOVERY - puppet last run on search1001 is OK: OK: Puppet is currently enabled, last run 28 seconds ago with 0 failures [17:46:38] RECOVERY - puppet last run on mw1088 is OK: OK: Puppet is currently enabled, last run 40 seconds ago with 0 failures [17:46:38] RECOVERY - puppet last run on mw1025 is OK: OK: Puppet is currently enabled, last run 34 seconds ago with 0 failures [17:46:38] RECOVERY - puppet last run on mw1118 is OK: OK: Puppet is currently enabled, last run 34 seconds ago with 0 failures [17:46:38] RECOVERY - puppet last run on mw1008 is OK: OK: Puppet is currently enabled, last run 35 seconds ago with 0 failures [17:46:38] RECOVERY - puppet last run on holmium is OK: OK: Puppet is currently enabled, last run 32 seconds ago with 0 failures [17:46:38] RECOVERY - puppet last run on mw1205 is OK: OK: Puppet is currently enabled, last run 35 seconds ago with 0 failures [17:46:49] RECOVERY - puppet last run on mw1175 is OK: OK: Puppet is currently enabled, last run 8 seconds ago with 0 failures [17:46:49] RECOVERY - puppet last run on virt1006 is OK: OK: Puppet is currently enabled, last run 22 seconds ago with 0 failures [17:46:49] RECOVERY - puppet last run on mw1065 is OK: OK: Puppet is currently enabled, last run 47 seconds ago with 0 failures [17:46:49] RECOVERY - puppet last run on dbproxy1001 is OK: OK: Puppet is currently enabled, last run 17 seconds ago with 0 failures [17:46:49] RECOVERY - puppet last run on mw1042 is OK: OK: Puppet is currently enabled, last run 46 seconds ago with 0 failures [17:46:49] RECOVERY - puppet last run on db1040 is OK: OK: Puppet is currently enabled, last run 47 seconds ago with 0 failures [17:46:50] RECOVERY - puppet last run on analytics1030 is OK: OK: Puppet is currently enabled, last run 50 seconds ago with 0 failures [17:46:50] RECOVERY - puppet last run on cp4008 is OK: OK: Puppet is currently enabled, last run 24 seconds ago with 0 failures [17:46:58] RECOVERY - puppet last run on mw1052 is OK: OK: Puppet is currently enabled, last run 18 seconds ago with 0 failures [17:46:58] RECOVERY - puppet last run on mw1129 is OK: OK: Puppet is currently enabled, last run 3 seconds ago with 0 failures [17:46:58] RECOVERY - puppet last run on amslvs1 is OK: OK: Puppet is currently enabled, last run 16 seconds ago with 0 failures [17:46:58] RECOVERY - puppet last run on cp3016 is OK: OK: Puppet is currently enabled, last run 57 seconds ago with 0 failures [17:47:08] RECOVERY - puppet last run on db1067 is OK: OK: Puppet is currently enabled, last run 39 seconds ago with 0 failures [17:47:08] RECOVERY - puppet last run on analytics1010 is OK: OK: Puppet is currently enabled, last run 38 seconds ago with 0 failures [17:47:08] RECOVERY - puppet last run on mw1211 is OK: OK: Puppet is currently enabled, last run 28 seconds ago with 0 failures [17:47:08] RECOVERY - puppet last run on gallium is OK: OK: Puppet is currently enabled, last run 34 seconds ago with 0 failures [17:47:08] RECOVERY - puppet last run on mw1170 is OK: OK: Puppet is currently enabled, last run 37 seconds ago with 0 failures [17:47:08] RECOVERY - puppet last run on mc1012 is OK: OK: Puppet is currently enabled, last run 4 seconds ago with 0 failures [17:47:08] RECOVERY - puppet last run on db1042 is OK: OK: Puppet is currently enabled, last run 7 seconds ago with 0 failures [17:47:09] RECOVERY - puppet last run on logstash1002 is OK: OK: Puppet is currently enabled, last run 46 seconds ago with 0 failures [17:47:09] RECOVERY - puppet last run on db1018 is OK: OK: Puppet is currently enabled, last run 47 seconds ago with 0 failures [17:47:10] RECOVERY - puppet last run on db1028 is OK: OK: Puppet is currently enabled, last run 47 seconds ago with 0 failures [17:47:10] RECOVERY - puppet last run on cp4003 is OK: OK: Puppet is currently enabled, last run 32 seconds ago with 0 failures [17:47:18] RECOVERY - puppet last run on mw1114 is OK: OK: Puppet is currently enabled, last run 11 seconds ago with 0 failures [17:47:18] RECOVERY - puppet last run on cp1061 is OK: OK: Puppet is currently enabled, last run 53 seconds ago with 0 failures [17:47:18] RECOVERY - puppet last run on amssq48 is OK: OK: Puppet is currently enabled, last run 2 seconds ago with 0 failures [17:47:18] RECOVERY - puppet last run on mw1011 is OK: OK: Puppet is currently enabled, last run 14 seconds ago with 0 failures [17:47:18] RECOVERY - puppet last run on es1007 is OK: OK: Puppet is currently enabled, last run 9 seconds ago with 0 failures [17:47:29] RECOVERY - puppet last run on wtp1012 is OK: OK: Puppet is currently enabled, last run 27 seconds ago with 0 failures [17:47:29] RECOVERY - puppet last run on pc1002 is OK: OK: Puppet is currently enabled, last run 25 seconds ago with 0 failures [17:47:38] RECOVERY - puppet last run on labstore1001 is OK: OK: Puppet is currently enabled, last run 1 seconds ago with 0 failures [17:47:38] RECOVERY - puppet last run on polonium is OK: OK: Puppet is currently enabled, last run 29 seconds ago with 0 failures [17:47:48] RECOVERY - puppet last run on mw1177 is OK: OK: Puppet is currently enabled, last run 34 seconds ago with 0 failures [17:47:48] RECOVERY - puppet last run on mw1213 is OK: OK: Puppet is currently enabled, last run 10 seconds ago with 0 failures [17:47:48] RECOVERY - puppet last run on labnet1001 is OK: OK: Puppet is currently enabled, last run 11 seconds ago with 0 failures [17:47:48] RECOVERY - puppet last run on wtp1005 is OK: OK: Puppet is currently enabled, last run 5 seconds ago with 0 failures [17:47:48] RECOVERY - puppet last run on cp1058 is OK: OK: Puppet is currently enabled, last run 11 seconds ago with 0 failures [17:47:48] RECOVERY - puppet last run on mw1039 is OK: OK: Puppet is currently enabled, last run 10 seconds ago with 0 failures [17:47:49] RECOVERY - puppet last run on mw1054 is OK: OK: Puppet is currently enabled, last run 9 seconds ago with 0 failures [17:47:49] RECOVERY - puppet last run on silver is OK: OK: Puppet is currently enabled, last run 7 seconds ago with 0 failures [17:47:49] RECOVERY - puppet last run on db1043 is OK: OK: Puppet is currently enabled, last run 47 seconds ago with 0 failures [17:47:50] RECOVERY - puppet last run on antimony is OK: OK: Puppet is currently enabled, last run 5 seconds ago with 0 failures [17:47:50] RECOVERY - puppet last run on db1021 is OK: OK: Puppet is currently enabled, last run 50 seconds ago with 0 failures [17:47:51] RECOVERY - puppet last run on mw1144 is OK: OK: Puppet is currently enabled, last run 49 seconds ago with 0 failures [17:47:51] RECOVERY - puppet last run on cp4014 is OK: OK: Puppet is currently enabled, last run 52 seconds ago with 0 failures [17:47:58] RECOVERY - puppet last run on amssq46 is OK: OK: Puppet is currently enabled, last run 44 seconds ago with 0 failures [17:47:58] RECOVERY - puppet last run on cp3014 is OK: OK: Puppet is currently enabled, last run 56 seconds ago with 0 failures [17:48:08] RECOVERY - puppet last run on analytics1038 is OK: OK: Puppet is currently enabled, last run 24 seconds ago with 0 failures [17:48:08] RECOVERY - puppet last run on search1007 is OK: OK: Puppet is currently enabled, last run 58 seconds ago with 0 failures [17:48:08] RECOVERY - puppet last run on mw1172 is OK: OK: Puppet is currently enabled, last run 31 seconds ago with 0 failures [17:48:08] RECOVERY - puppet last run on mw1195 is OK: OK: Puppet is currently enabled, last run 22 seconds ago with 0 failures [17:48:09] RECOVERY - puppet last run on db1003 is OK: OK: Puppet is currently enabled, last run 18 seconds ago with 0 failures [17:48:09] RECOVERY - puppet last run on virt1003 is OK: OK: Puppet is currently enabled, last run 21 seconds ago with 0 failures [17:48:09] RECOVERY - puppet last run on db1016 is OK: OK: Puppet is currently enabled, last run 29 seconds ago with 0 failures [17:48:09] RECOVERY - puppet last run on elastic1006 is OK: OK: Puppet is currently enabled, last run 4 seconds ago with 0 failures [17:48:10] RECOVERY - puppet last run on analytics1016 is OK: OK: Puppet is currently enabled, last run 27 seconds ago with 0 failures [17:48:10] RECOVERY - puppet last run on ruthenium is OK: OK: Puppet is currently enabled, last run 38 seconds ago with 0 failures [17:48:18] RECOVERY - puppet last run on ms-be3002 is OK: OK: Puppet is currently enabled, last run 3 seconds ago with 0 failures [17:48:18] RECOVERY - puppet last run on mw1126 is OK: OK: Puppet is currently enabled, last run 34 seconds ago with 0 failures [17:48:18] RECOVERY - puppet last run on db1026 is OK: OK: Puppet is currently enabled, last run 1 seconds ago with 0 failures [17:48:18] RECOVERY - puppet last run on amssq55 is OK: OK: Puppet is currently enabled, last run 7 seconds ago with 0 failures [17:48:18] RECOVERY - puppet last run on labmon1001 is OK: OK: Puppet is currently enabled, last run 42 seconds ago with 0 failures [17:48:19] RECOVERY - puppet last run on lvs3004 is OK: OK: Puppet is currently enabled, last run 0 seconds ago with 0 failures [17:48:29] RECOVERY - puppet last run on db1060 is OK: OK: Puppet is currently enabled, last run 20 seconds ago with 0 failures [17:48:29] RECOVERY - puppet last run on db1052 is OK: OK: Puppet is currently enabled, last run 50 seconds ago with 0 failures [17:48:29] RECOVERY - puppet last run on mc1005 is OK: OK: Puppet is currently enabled, last run 11 seconds ago with 0 failures [17:48:39] RECOVERY - puppet last run on mw1002 is OK: OK: Puppet is currently enabled, last run 52 seconds ago with 0 failures [17:48:39] RECOVERY - puppet last run on dataset1001 is OK: OK: Puppet is currently enabled, last run 25 seconds ago with 0 failures [17:48:48] RECOVERY - puppet last run on mc1014 is OK: OK: Puppet is currently enabled, last run 34 seconds ago with 0 failures [17:48:48] RECOVERY - puppet last run on mw1180 is OK: OK: Puppet is currently enabled, last run 4 seconds ago with 0 failures [17:48:48] RECOVERY - puppet last run on virt1001 is OK: OK: Puppet is currently enabled, last run 36 seconds ago with 0 failures [17:48:48] RECOVERY - puppet last run on ssl1005 is OK: OK: Puppet is currently enabled, last run 28 seconds ago with 0 failures [17:48:48] RECOVERY - puppet last run on mw1044 is OK: OK: Puppet is currently enabled, last run 31 seconds ago with 0 failures [17:48:48] RECOVERY - puppet last run on search1005 is OK: OK: Puppet is currently enabled, last run 32 seconds ago with 0 failures [17:48:49] RECOVERY - puppet last run on db1048 is OK: OK: Puppet is currently enabled, last run 31 seconds ago with 0 failures [17:48:49] RECOVERY - puppet last run on mw1206 is OK: OK: Puppet is currently enabled, last run 10 seconds ago with 0 failures [17:48:50] RECOVERY - puppet last run on cp4004 is OK: OK: Puppet is currently enabled, last run 47 seconds ago with 0 failures [17:48:58] RECOVERY - puppet last run on amssq47 is OK: OK: Puppet is currently enabled, last run 56 seconds ago with 0 failures [17:48:58] RECOVERY - puppet last run on amssq60 is OK: OK: Puppet is currently enabled, last run 51 seconds ago with 0 failures [17:49:08] RECOVERY - puppet last run on mw1125 is OK: OK: Puppet is currently enabled, last run 2 seconds ago with 0 failures [17:49:08] RECOVERY - puppet last run on db1071 is OK: OK: Puppet is currently enabled, last run 4 seconds ago with 0 failures [17:49:08] RECOVERY - puppet last run on mw1162 is OK: OK: Puppet is currently enabled, last run 25 seconds ago with 0 failures [17:49:09] RECOVERY - puppet last run on mw1076 is OK: OK: Puppet is currently enabled, last run 58 seconds ago with 0 failures [17:49:09] RECOVERY - puppet last run on cp1046 is OK: OK: Puppet is currently enabled, last run 27 seconds ago with 0 failures [17:49:09] RECOVERY - puppet last run on rdb1001 is OK: OK: Puppet is currently enabled, last run 6 seconds ago with 0 failures [17:49:09] RECOVERY - puppet last run on db1036 is OK: OK: Puppet is currently enabled, last run 56 seconds ago with 0 failures [17:49:09] RECOVERY - puppet last run on mw1208 is OK: OK: Puppet is currently enabled, last run 58 seconds ago with 0 failures [17:49:09] RECOVERY - puppet last run on search1002 is OK: OK: Puppet is currently enabled, last run 6 seconds ago with 0 failures [17:49:10] RECOVERY - puppet last run on mw1133 is OK: OK: Puppet is currently enabled, last run 7 seconds ago with 0 failures [17:49:10] RECOVERY - puppet last run on snapshot1001 is OK: OK: Puppet is currently enabled, last run 28 seconds ago with 0 failures [17:49:11] RECOVERY - puppet last run on mw1055 is OK: OK: Puppet is currently enabled, last run 60 seconds ago with 0 failures [17:49:11] RECOVERY - puppet last run on cp4001 is OK: OK: Puppet is currently enabled, last run 11 seconds ago with 0 failures [17:49:12] RECOVERY - puppet last run on cp4019 is OK: OK: Puppet is currently enabled, last run 0 seconds ago with 0 failures [17:49:18] RECOVERY - puppet last run on mw1168 is OK: OK: Puppet is currently enabled, last run 11 seconds ago with 0 failures [17:49:18] RECOVERY - puppet last run on tin is OK: OK: Puppet is currently enabled, last run 17 seconds ago with 0 failures [17:49:29] RECOVERY - puppet last run on wtp1018 is OK: OK: Puppet is currently enabled, last run 27 seconds ago with 0 failures [17:49:29] RECOVERY - puppet last run on gadolinium is OK: OK: Puppet is currently enabled, last run 47 seconds ago with 0 failures [17:49:29] RECOVERY - puppet last run on search1017 is OK: OK: Puppet is currently enabled, last run 25 seconds ago with 0 failures [17:49:38] RECOVERY - puppet last run on mw1202 is OK: OK: Puppet is currently enabled, last run 16 seconds ago with 0 failures [17:49:38] RECOVERY - puppet last run on analytics1013 is OK: OK: Puppet is currently enabled, last run 29 seconds ago with 0 failures [17:49:39] RECOVERY - puppet last run on thallium is OK: OK: Puppet is currently enabled, last run 10 seconds ago with 0 failures [17:49:39] RECOVERY - puppet last run on analytics1022 is OK: OK: Puppet is currently enabled, last run 38 seconds ago with 0 failures [17:49:39] RECOVERY - puppet last run on db1069 is OK: OK: Puppet is currently enabled, last run 43 seconds ago with 0 failures [17:49:48] RECOVERY - puppet last run on mw1190 is OK: OK: Puppet is currently enabled, last run 36 seconds ago with 0 failures [17:49:49] RECOVERY - puppet last run on sodium is OK: OK: Puppet is currently enabled, last run 49 seconds ago with 0 failures [17:49:49] RECOVERY - puppet last run on virt1004 is OK: OK: Puppet is currently enabled, last run 50 seconds ago with 0 failures [17:49:49] RECOVERY - puppet last run on mw1149 is OK: OK: Puppet is currently enabled, last run 49 seconds ago with 0 failures [17:49:49] RECOVERY - puppet last run on elastic1015 is OK: OK: Puppet is currently enabled, last run 37 seconds ago with 0 failures [17:49:49] RECOVERY - puppet last run on mw1051 is OK: OK: Puppet is currently enabled, last run 22 seconds ago with 0 failures [17:49:49] RECOVERY - puppet last run on db1004 is OK: OK: Puppet is currently enabled, last run 42 seconds ago with 0 failures [17:49:49] RECOVERY - puppet last run on cp1062 is OK: OK: Puppet is currently enabled, last run 20 seconds ago with 0 failures [17:49:50] RECOVERY - puppet last run on cp1050 is OK: OK: Puppet is currently enabled, last run 42 seconds ago with 0 failures [17:49:50] RECOVERY - puppet last run on db1039 is OK: OK: Puppet is currently enabled, last run 45 seconds ago with 0 failures [17:49:51] RECOVERY - puppet last run on snapshot1002 is OK: OK: Puppet is currently enabled, last run 50 seconds ago with 0 failures [17:49:51] RECOVERY - puppet last run on lvs4003 is OK: OK: Puppet is currently enabled, last run 49 seconds ago with 0 failures [17:49:52] RECOVERY - puppet last run on bast4001 is OK: OK: Puppet is currently enabled, last run 38 seconds ago with 0 failures [17:49:58] RECOVERY - puppet last run on ms-be3001 is OK: OK: Puppet is currently enabled, last run 53 seconds ago with 0 failures [17:49:58] RECOVERY - puppet last run on amssq40 is OK: OK: Puppet is currently enabled, last run 28 seconds ago with 0 failures [17:49:58] RECOVERY - puppet last run on ms-fe3002 is OK: OK: Puppet is currently enabled, last run 11 seconds ago with 0 failures [17:49:58] RECOVERY - puppet last run on amssq34 is OK: OK: Puppet is currently enabled, last run 54 seconds ago with 0 failures [17:49:58] RECOVERY - puppet last run on mw1084 is OK: OK: Puppet is currently enabled, last run 60 seconds ago with 0 failures [17:49:58] RECOVERY - puppet last run on mw1098 is OK: OK: Puppet is currently enabled, last run 28 seconds ago with 0 failures [17:49:58] RECOVERY - puppet last run on db72 is OK: OK: Puppet is currently enabled, last run 31 seconds ago with 0 failures [17:49:59] RECOVERY - puppet last run on cp3010 is OK: OK: Puppet is currently enabled, last run 56 seconds ago with 0 failures [17:50:08] RECOVERY - puppet last run on labsdb1006 is OK: OK: Puppet is currently enabled, last run 9 seconds ago with 0 failures [17:50:08] RECOVERY - puppet last run on argon is OK: OK: Puppet is currently enabled, last run 37 seconds ago with 0 failures [17:50:08] RECOVERY - puppet last run on mw1165 is OK: OK: Puppet is currently enabled, last run 14 seconds ago with 0 failures [17:50:08] RECOVERY - puppet last run on mw1030 is OK: OK: Puppet is currently enabled, last run 9 seconds ago with 0 failures [17:50:09] RECOVERY - puppet last run on mw1111 is OK: OK: Puppet is currently enabled, last run 58 seconds ago with 0 failures [17:50:09] RECOVERY - puppet last run on mw1151 is OK: OK: Puppet is currently enabled, last run 58 seconds ago with 0 failures [17:50:09] RECOVERY - puppet last run on snapshot1004 is OK: OK: Puppet is currently enabled, last run 3 seconds ago with 0 failures [17:50:10] RECOVERY - puppet last run on mw1057 is OK: OK: Puppet is currently enabled, last run 11 seconds ago with 0 failures [17:50:10] RECOVERY - puppet last run on mw1079 is OK: OK: Puppet is currently enabled, last run 55 seconds ago with 0 failures [17:50:11] RECOVERY - puppet last run on cp4005 is OK: OK: Puppet is currently enabled, last run 13 seconds ago with 0 failures [17:50:11] RECOVERY - puppet last run on mw1056 is OK: OK: Puppet is currently enabled, last run 4 seconds ago with 0 failures [17:50:12] RECOVERY - puppet last run on elastic1011 is OK: OK: Puppet is currently enabled, last run 10 seconds ago with 0 failures [17:50:12] RECOVERY - puppet last run on mw1014 is OK: OK: Puppet is currently enabled, last run 38 seconds ago with 0 failures [17:50:13] RECOVERY - puppet last run on analytics1023 is OK: OK: Puppet is currently enabled, last run 41 seconds ago with 0 failures [17:50:13] RECOVERY - puppet last run on stat1002 is OK: OK: Puppet is currently enabled, last run 58 seconds ago with 0 failures [17:50:14] RECOVERY - puppet last run on ssl1009 is OK: OK: Puppet is currently enabled, last run 8 seconds ago with 0 failures [17:50:14] RECOVERY - puppet last run on hafnium is OK: OK: Puppet is currently enabled, last run 11 seconds ago with 0 failures [17:50:18] RECOVERY - puppet last run on amssq36 is OK: OK: Puppet is currently enabled, last run 14 seconds ago with 0 failures [17:50:18] RECOVERY - puppet last run on amssq51 is OK: OK: Puppet is currently enabled, last run 14 seconds ago with 0 failures [17:50:18] RECOVERY - puppet last run on rhenium is OK: OK: Puppet is currently enabled, last run 17 seconds ago with 0 failures [17:50:18] RECOVERY - puppet last run on cp1063 is OK: OK: Puppet is currently enabled, last run 23 seconds ago with 0 failures [17:50:18] RECOVERY - puppet last run on cp4018 is OK: OK: Puppet is currently enabled, last run 21 seconds ago with 0 failures [17:50:28] RECOVERY - puppet last run on wtp1023 is OK: OK: Puppet is currently enabled, last run 19 seconds ago with 0 failures [17:50:29] RECOVERY - puppet last run on rubidium is OK: OK: Puppet is currently enabled, last run 38 seconds ago with 0 failures [17:50:29] RECOVERY - puppet last run on mw1049 is OK: OK: Puppet is currently enabled, last run 37 seconds ago with 0 failures [17:50:29] RECOVERY - puppet last run on elastic1014 is OK: OK: Puppet is currently enabled, last run 25 seconds ago with 0 failures [17:50:29] RECOVERY - puppet last run on cp1048 is OK: OK: Puppet is currently enabled, last run 37 seconds ago with 0 failures [17:50:29] RECOVERY - puppet last run on ms-be1012 is OK: OK: Puppet is currently enabled, last run 37 seconds ago with 0 failures [17:50:38] RECOVERY - puppet last run on db1062 is OK: OK: Puppet is currently enabled, last run 39 seconds ago with 0 failures [17:50:39] RECOVERY - puppet last run on osmium is OK: OK: Puppet is currently enabled, last run 34 seconds ago with 0 failures [17:50:39] RECOVERY - puppet last run on db69 is OK: OK: Puppet is currently enabled, last run 33 seconds ago with 0 failures [17:50:39] RECOVERY - puppet last run on wtp1004 is OK: OK: Puppet is currently enabled, last run 33 seconds ago with 0 failures [17:50:39] RECOVERY - puppet last run on virt1007 is OK: OK: Puppet is currently enabled, last run 43 seconds ago with 0 failures [17:50:48] RECOVERY - puppet last run on oxygen is OK: OK: Puppet is currently enabled, last run 35 seconds ago with 0 failures [17:50:48] RECOVERY - puppet last run on mw1181 is OK: OK: Puppet is currently enabled, last run 33 seconds ago with 0 failures [17:50:48] RECOVERY - puppet last run on mw1171 is OK: OK: Puppet is currently enabled, last run 5 seconds ago with 0 failures [17:50:48] RECOVERY - puppet last run on mw1212 is OK: OK: Puppet is currently enabled, last run 5 seconds ago with 0 failures [17:50:48] RECOVERY - puppet last run on amssq56 is OK: OK: Puppet is currently enabled, last run 51 seconds ago with 0 failures [17:50:48] RECOVERY - puppet last run on analytics1032 is OK: OK: Puppet is currently enabled, last run 34 seconds ago with 0 failures [17:50:48] RECOVERY - puppet last run on mw1156 is OK: OK: Puppet is currently enabled, last run 9 seconds ago with 0 failures [17:50:49] RECOVERY - puppet last run on mc1001 is OK: OK: Puppet is currently enabled, last run 43 seconds ago with 0 failures [17:50:49] RECOVERY - puppet last run on ms-be1008 is OK: OK: Puppet is currently enabled, last run 50 seconds ago with 0 failures [17:50:50] RECOVERY - puppet last run on mw1050 is OK: OK: Puppet is currently enabled, last run 31 seconds ago with 0 failures [17:50:50] RECOVERY - puppet last run on ms-be1007 is OK: OK: Puppet is currently enabled, last run 37 seconds ago with 0 failures [17:50:58] RECOVERY - puppet last run on ssl3002 is OK: OK: Puppet is currently enabled, last run 14 seconds ago with 0 failures [17:50:58] RECOVERY - puppet last run on ms-be1009 is OK: OK: Puppet is currently enabled, last run 16 seconds ago with 0 failures [17:50:58] RECOVERY - puppet last run on db1020 is OK: OK: Puppet is currently enabled, last run 59 seconds ago with 0 failures [17:50:58] RECOVERY - puppet last run on amssq41 is OK: OK: Puppet is currently enabled, last run 21 seconds ago with 0 failures [17:51:08] RECOVERY - puppet last run on db1054 is OK: OK: Puppet is currently enabled, last run 1 seconds ago with 0 failures [17:51:08] RECOVERY - puppet last run on mw1081 is OK: OK: Puppet is currently enabled, last run 25 seconds ago with 0 failures [17:51:08] RECOVERY - puppet last run on mw1183 is OK: OK: Puppet is currently enabled, last run 55 seconds ago with 0 failures [17:51:09] RECOVERY - puppet last run on mw1034 is OK: OK: Puppet is currently enabled, last run 52 seconds ago with 0 failures [17:51:09] RECOVERY - puppet last run on tungsten is OK: OK: Puppet is currently enabled, last run 60 seconds ago with 0 failures [17:51:09] RECOVERY - puppet last run on mw1146 is OK: OK: Puppet is currently enabled, last run 7 seconds ago with 0 failures [17:51:09] RECOVERY - puppet last run on db1055 is OK: OK: Puppet is currently enabled, last run 12 seconds ago with 0 failures [17:51:09] RECOVERY - puppet last run on search1023 is OK: OK: Puppet is currently enabled, last run 13 seconds ago with 0 failures [17:51:10] RECOVERY - puppet last run on search1024 is OK: OK: Puppet is currently enabled, last run 3 seconds ago with 0 failures [17:51:10] ori: this has something to do with your "killall master" last night on palladium I think, although I can't quite completely make the connection as to how. the timing is too coincidental, though. [17:51:18] RECOVERY - puppet last run on amssq42 is OK: OK: Puppet is currently enabled, last run 24 seconds ago with 0 failures [17:51:18] RECOVERY - puppet last run on ssl1008 is OK: OK: Puppet is currently enabled, last run 5 seconds ago with 0 failures [17:51:18] RECOVERY - puppet last run on mw1029 is OK: OK: Puppet is currently enabled, last run 10 seconds ago with 0 failures [17:51:18] RECOVERY - puppet last run on mw1023 is OK: OK: Puppet is currently enabled, last run 16 seconds ago with 0 failures [17:51:29] RECOVERY - puppet last run on mw1210 is OK: OK: Puppet is currently enabled, last run 24 seconds ago with 0 failures [17:51:29] RECOVERY - puppet last run on mw1163 is OK: OK: Puppet is currently enabled, last run 24 seconds ago with 0 failures [17:51:29] RECOVERY - puppet last run on mw1032 is OK: OK: Puppet is currently enabled, last run 11 seconds ago with 0 failures [17:51:29] RECOVERY - puppet last run on mw1159 is OK: OK: Puppet is currently enabled, last run 46 seconds ago with 0 failures [17:51:29] RECOVERY - puppet last run on db1001 is OK: OK: Puppet is currently enabled, last run 6 seconds ago with 0 failures [17:51:38] please don't do that :p [17:51:38] RECOVERY - puppet last run on mw1097 is OK: OK: Puppet is currently enabled, last run 10 seconds ago with 0 failures [17:51:38] RECOVERY - puppet last run on wtp1022 is OK: OK: Puppet is currently enabled, last run 51 seconds ago with 0 failures [17:51:38] RECOVERY - puppet last run on mw1087 is OK: OK: Puppet is currently enabled, last run 24 seconds ago with 0 failures [17:51:39] RECOVERY - puppet last run on analytics1037 is OK: OK: Puppet is currently enabled, last run 51 seconds ago with 0 failures [17:51:39] RECOVERY - puppet last run on cp1038 is OK: OK: Puppet is currently enabled, last run 35 seconds ago with 0 failures [17:51:39] RECOVERY - puppet last run on mw1004 is OK: OK: Puppet is currently enabled, last run 56 seconds ago with 0 failures [17:51:39] RECOVERY - puppet last run on titanium is OK: OK: Puppet is currently enabled, last run 40 seconds ago with 0 failures [17:51:39] RECOVERY - puppet last run on wtp1002 is OK: OK: Puppet is currently enabled, last run 58 seconds ago with 0 failures [17:51:48] RECOVERY - puppet last run on wtp1013 is OK: OK: Puppet is currently enabled, last run 23 seconds ago with 0 failures [17:51:48] RECOVERY - puppet last run on lvs1001 is OK: OK: Puppet is currently enabled, last run 44 seconds ago with 0 failures [17:51:48] RECOVERY - puppet last run on db1064 is OK: OK: Puppet is currently enabled, last run 4 seconds ago with 0 failures [17:51:48] RECOVERY - puppet last run on mw1188 is OK: OK: Puppet is currently enabled, last run 37 seconds ago with 0 failures [17:51:48] RECOVERY - puppet last run on mc1013 is OK: OK: Puppet is currently enabled, last run 48 seconds ago with 0 failures [17:51:48] RECOVERY - puppet last run on wtp1015 is OK: OK: Puppet is currently enabled, last run 14 seconds ago with 0 failures [17:51:48] RECOVERY - puppet last run on mw1074 is OK: OK: Puppet is currently enabled, last run 39 seconds ago with 0 failures [17:51:49] RECOVERY - puppet last run on mw1116 is OK: OK: Puppet is currently enabled, last run 38 seconds ago with 0 failures [17:51:49] RECOVERY - puppet last run on mc1007 is OK: OK: Puppet is currently enabled, last run 5 seconds ago with 0 failures [17:51:50] RECOVERY - puppet last run on rcs1001 is OK: OK: Puppet is currently enabled, last run 38 seconds ago with 0 failures [17:51:50] RECOVERY - puppet last run on rcs1002 is OK: OK: Puppet is currently enabled, last run 39 seconds ago with 0 failures [17:51:58] RECOVERY - puppet last run on elastic1005 is OK: OK: Puppet is currently enabled, last run 40 seconds ago with 0 failures [17:52:08] RECOVERY - puppet last run on labsdb1004 is OK: OK: Puppet is currently enabled, last run 18 seconds ago with 0 failures [17:52:08] RECOVERY - puppet last run on db1057 is OK: OK: Puppet is currently enabled, last run 27 seconds ago with 0 failures [17:52:08] RECOVERY - puppet last run on wtp1011 is OK: OK: Puppet is currently enabled, last run 57 seconds ago with 0 failures [17:52:08] RECOVERY - puppet last run on rdb1002 is OK: OK: Puppet is currently enabled, last run 45 seconds ago with 0 failures [17:52:08] RECOVERY - puppet last run on protactinium is OK: OK: Puppet is currently enabled, last run 44 seconds ago with 0 failures [17:52:09] RECOVERY - puppet last run on mw1198 is OK: OK: Puppet is currently enabled, last run 49 seconds ago with 0 failures [17:52:09] RECOVERY - puppet last run on elastic1002 is OK: OK: Puppet is currently enabled, last run 42 seconds ago with 0 failures [17:52:09] RECOVERY - puppet last run on cp1060 is OK: OK: Puppet is currently enabled, last run 3 seconds ago with 0 failures [17:52:18] RECOVERY - puppet last run on hooft is OK: OK: Puppet is currently enabled, last run 31 seconds ago with 0 failures [17:52:18] RECOVERY - puppet last run on amssq62 is OK: OK: Puppet is currently enabled, last run 50 seconds ago with 0 failures [17:52:18] RECOVERY - puppet last run on amslvs3 is OK: OK: Puppet is currently enabled, last run 35 seconds ago with 0 failures [17:52:28] RECOVERY - puppet last run on db1063 is OK: OK: Puppet is currently enabled, last run 20 seconds ago with 0 failures [17:52:38] RECOVERY - puppet last run on db1070 is OK: OK: Puppet is currently enabled, last run 3 seconds ago with 0 failures [17:52:38] RECOVERY - puppet last run on analytics1014 is OK: OK: Puppet is currently enabled, last run 52 seconds ago with 0 failures [17:52:39] RECOVERY - puppet last run on analytics1011 is OK: OK: Puppet is currently enabled, last run 27 seconds ago with 0 failures [17:52:39] RECOVERY - puppet last run on ocg1003 is OK: OK: Puppet is currently enabled, last run 2 seconds ago with 0 failures [17:52:39] RECOVERY - puppet last run on db1061 is OK: OK: Puppet is currently enabled, last run 2 seconds ago with 0 failures [17:52:48] RECOVERY - Puppet freshness on mw1043 is OK: puppet ran at Sat Aug 16 17:52:41 UTC 2014 [17:52:48] RECOVERY - puppet last run on wtp1007 is OK: OK: Puppet is currently enabled, last run 38 seconds ago with 0 failures [17:52:48] RECOVERY - puppet last run on mw1001 is OK: OK: Puppet is currently enabled, last run 14 seconds ago with 0 failures [17:52:48] RECOVERY - puppet last run on pc1003 is OK: OK: Puppet is currently enabled, last run 27 seconds ago with 0 failures [17:52:48] RECOVERY - puppet last run on nitrogen is OK: OK: Puppet is currently enabled, last run 28 seconds ago with 0 failures [17:52:48] RECOVERY - puppet last run on elastic1017 is OK: OK: Puppet is currently enabled, last run 29 seconds ago with 0 failures [17:52:48] RECOVERY - puppet last run on es1002 is OK: OK: Puppet is currently enabled, last run 40 seconds ago with 0 failures [17:52:49] RECOVERY - puppet last run on db1030 is OK: OK: Puppet is currently enabled, last run 29 seconds ago with 0 failures [17:52:58] RECOVERY - puppet last run on db1044 is OK: OK: Puppet is currently enabled, last run 9 seconds ago with 0 failures [17:52:58] RECOVERY - puppet last run on amssq38 is OK: OK: Puppet is currently enabled, last run 34 seconds ago with 0 failures [17:52:58] RECOVERY - Puppet freshness on es1010 is OK: puppet ran at Sat Aug 16 17:52:56 UTC 2014 [17:53:08] RECOVERY - puppet last run on mw1185 is OK: OK: Puppet is currently enabled, last run 23 seconds ago with 0 failures [17:53:08] RECOVERY - puppet last run on mw1105 is OK: OK: Puppet is currently enabled, last run 4 seconds ago with 0 failures [17:53:08] RECOVERY - puppet last run on rdb1003 is OK: OK: Puppet is currently enabled, last run 27 seconds ago with 0 failures [17:53:08] RECOVERY - puppet last run on ssl1007 is OK: OK: Puppet is currently enabled, last run 37 seconds ago with 0 failures [17:53:08] RECOVERY - puppet last run on mw1077 is OK: OK: Puppet is currently enabled, last run 12 seconds ago with 0 failures [17:53:09] RECOVERY - puppet last run on virt1000 is OK: OK: Puppet is currently enabled, last run 28 seconds ago with 0 failures [17:53:09] RECOVERY - puppet last run on mw1148 is OK: OK: Puppet is currently enabled, last run 33 seconds ago with 0 failures [17:53:09] RECOVERY - puppet last run on mw1201 is OK: OK: Puppet is currently enabled, last run 7 seconds ago with 0 failures [17:53:09] RECOVERY - puppet last run on search1015 is OK: OK: Puppet is currently enabled, last run 52 seconds ago with 0 failures [17:53:10] RECOVERY - puppet last run on search1013 is OK: OK: Puppet is currently enabled, last run 48 seconds ago with 0 failures [17:53:18] RECOVERY - puppet last run on cp3004 is OK: OK: Puppet is currently enabled, last run 18 seconds ago with 0 failures [17:53:18] RECOVERY - puppet last run on cp3012 is OK: OK: Puppet is currently enabled, last run 47 seconds ago with 0 failures [17:53:18] RECOVERY - puppet last run on cp3009 is OK: OK: Puppet is currently enabled, last run 37 seconds ago with 0 failures [17:53:18] RECOVERY - puppet last run on ms-fe3001 is OK: OK: Puppet is currently enabled, last run 21 seconds ago with 0 failures [17:53:18] RECOVERY - Puppet freshness on cp1037 is OK: puppet ran at Sat Aug 16 17:53:11 UTC 2014 [17:53:18] RECOVERY - puppet last run on mw1108 is OK: OK: Puppet is currently enabled, last run 15 seconds ago with 0 failures [17:53:18] RECOVERY - puppet last run on mercury is OK: OK: Puppet is currently enabled, last run 33 seconds ago with 0 failures [17:53:28] RECOVERY - Puppet freshness on analytics1004 is OK: puppet ran at Sat Aug 16 17:53:21 UTC 2014 [17:53:28] RECOVERY - puppet last run on mw1024 is OK: OK: Puppet is currently enabled, last run 9 seconds ago with 0 failures [17:53:29] RECOVERY - puppet last run on mw1167 is OK: OK: Puppet is currently enabled, last run 43 seconds ago with 0 failures [17:53:38] RECOVERY - puppet last run on db1011 is OK: OK: Puppet is currently enabled, last run 37 seconds ago with 0 failures [17:53:39] RECOVERY - puppet last run on wtp1003 is OK: OK: Puppet is currently enabled, last run 54 seconds ago with 0 failures [17:53:48] RECOVERY - puppet last run on stat1003 is OK: OK: Puppet is currently enabled, last run 46 seconds ago with 0 failures [17:53:48] RECOVERY - puppet last run on dbstore1002 is OK: OK: Puppet is currently enabled, last run 54 seconds ago with 0 failures [17:53:58] PROBLEM - Puppet freshness on db1006 is CRITICAL: Last successful Puppet run was Sat 16 Aug 2014 15:52:56 UTC [17:53:58] RECOVERY - puppet last run on mw1033 is OK: OK: Puppet is currently enabled, last run 6 seconds ago with 0 failures [17:54:08] RECOVERY - puppet last run on mw1209 is OK: OK: Puppet is currently enabled, last run 12 seconds ago with 0 failures [17:54:08] RECOVERY - puppet last run on db1038 is OK: OK: Puppet is currently enabled, last run 10 seconds ago with 0 failures [17:54:08] RECOVERY - puppet last run on mw1219 is OK: OK: Puppet is currently enabled, last run 10 seconds ago with 0 failures [17:54:08] RECOVERY - puppet last run on search1011 is OK: OK: Puppet is currently enabled, last run 12 seconds ago with 0 failures [17:54:08] RECOVERY - puppet last run on calcium is OK: OK: Puppet is currently enabled, last run 48 seconds ago with 0 failures [17:54:08] RECOVERY - puppet last run on db1047 is OK: OK: Puppet is currently enabled, last run 12 seconds ago with 0 failures [17:54:08] RECOVERY - puppet last run on mw1186 is OK: OK: Puppet is currently enabled, last run 16 seconds ago with 0 failures [17:54:09] RECOVERY - puppet last run on cp1037 is OK: OK: Puppet is currently enabled, last run 14 seconds ago with 0 failures [17:54:09] RECOVERY - puppet last run on es1010 is OK: OK: Puppet is currently enabled, last run 49 seconds ago with 0 failures [17:54:10] RECOVERY - puppet last run on analytics1028 is OK: OK: Puppet is currently enabled, last run 39 seconds ago with 0 failures [17:54:10] RECOVERY - puppet last run on ms-be1011 is OK: OK: Puppet is currently enabled, last run 14 seconds ago with 0 failures [17:54:11] RECOVERY - puppet last run on search1006 is OK: OK: Puppet is currently enabled, last run 5 seconds ago with 0 failures [17:54:11] RECOVERY - puppet last run on mw1122 is OK: OK: Puppet is currently enabled, last run 15 seconds ago with 0 failures [17:54:12] RECOVERY - puppet last run on ssl1006 is OK: OK: Puppet is currently enabled, last run 8 seconds ago with 0 failures [17:54:18] RECOVERY - puppet last run on amssq31 is OK: OK: Puppet is currently enabled, last run 2 seconds ago with 0 failures [17:54:18] RECOVERY - puppet last run on mw1022 is OK: OK: Puppet is currently enabled, last run 27 seconds ago with 0 failures [17:54:18] RECOVERY - Puppet freshness on ms-be1002 is OK: puppet ran at Sat Aug 16 17:54:12 UTC 2014 [17:54:18] RECOVERY - puppet last run on ssl1001 is OK: OK: Puppet is currently enabled, last run 60 seconds ago with 0 failures [17:54:18] RECOVERY - puppet last run on mw1043 is OK: OK: Puppet is currently enabled, last run 60 seconds ago with 0 failures [17:54:29] RECOVERY - puppet last run on analytics1027 is OK: OK: Puppet is currently enabled, last run 42 seconds ago with 0 failures [17:54:29] RECOVERY - puppet last run on strontium is OK: OK: Puppet is currently enabled, last run 26 seconds ago with 0 failures [17:54:29] RECOVERY - puppet last run on db1033 is OK: OK: Puppet is currently enabled, last run 38 seconds ago with 0 failures [17:54:29] RECOVERY - puppet last run on search1012 is OK: OK: Puppet is currently enabled, last run 1 seconds ago with 0 failures [17:54:38] RECOVERY - puppet last run on iodine is OK: OK: Puppet is currently enabled, last run 31 seconds ago with 0 failures [17:54:39] RECOVERY - puppet last run on mw1142 is OK: OK: Puppet is currently enabled, last run 1 seconds ago with 0 failures [17:54:39] RECOVERY - puppet last run on mw1121 is OK: OK: Puppet is currently enabled, last run 41 seconds ago with 0 failures [17:54:48] RECOVERY - puppet last run on analytics1004 is OK: OK: Puppet is currently enabled, last run 54 seconds ago with 0 failures [17:54:48] RECOVERY - puppet last run on cp1067 is OK: OK: Puppet is currently enabled, last run 14 seconds ago with 0 failures [17:54:48] RECOVERY - puppet last run on mw1152 is OK: OK: Puppet is currently enabled, last run 17 seconds ago with 0 failures [17:54:48] RECOVERY - puppet last run on cp1052 is OK: OK: Puppet is currently enabled, last run 57 seconds ago with 0 failures [17:54:48] RECOVERY - puppet last run on stat1001 is OK: OK: Puppet is currently enabled, last run 6 seconds ago with 0 failures [17:54:48] RECOVERY - puppet last run on search1022 is OK: OK: Puppet is currently enabled, last run 56 seconds ago with 0 failures [17:54:48] RECOVERY - puppet last run on fluorine is OK: OK: Puppet is currently enabled, last run 38 seconds ago with 0 failures [17:54:49] RECOVERY - puppet last run on mw1091 is OK: OK: Puppet is currently enabled, last run 43 seconds ago with 0 failures [17:54:49] RECOVERY - puppet last run on db1006 is OK: OK: Puppet is currently enabled, last run 56 seconds ago with 0 failures [17:54:50] RECOVERY - puppet last run on mw1066 is OK: OK: Puppet is currently enabled, last run 7 seconds ago with 0 failures [17:54:50] RECOVERY - puppet last run on mw1139 is OK: OK: Puppet is currently enabled, last run 40 seconds ago with 0 failures [17:54:51] RECOVERY - puppet last run on mw1010 is OK: OK: Puppet is currently enabled, last run 9 seconds ago with 0 failures [17:54:58] RECOVERY - puppet last run on mw1016 is OK: OK: Puppet is currently enabled, last run 21 seconds ago with 0 failures [17:54:58] RECOVERY - puppet last run on lvs3003 is OK: OK: Puppet is currently enabled, last run 27 seconds ago with 0 failures [17:55:08] RECOVERY - puppet last run on analytics1003 is OK: OK: Puppet is currently enabled, last run 22 seconds ago with 0 failures [17:55:08] RECOVERY - puppet last run on mw1093 is OK: OK: Puppet is currently enabled, last run 53 seconds ago with 0 failures [17:55:09] RECOVERY - puppet last run on rbf1001 is OK: OK: Puppet is currently enabled, last run 58 seconds ago with 0 failures [17:55:09] RECOVERY - puppet last run on elastic1003 is OK: OK: Puppet is currently enabled, last run 25 seconds ago with 0 failures [17:55:09] RECOVERY - puppet last run on ms-be1002 is OK: OK: Puppet is currently enabled, last run 30 seconds ago with 0 failures [17:55:18] RECOVERY - puppet last run on amssq33 is OK: OK: Puppet is currently enabled, last run 26 seconds ago with 0 failures [17:55:18] RECOVERY - puppet last run on cp4020 is OK: OK: Puppet is currently enabled, last run 25 seconds ago with 0 failures [17:55:18] RECOVERY - puppet last run on analytics1018 is OK: OK: Puppet is currently enabled, last run 37 seconds ago with 0 failures [17:55:18] RECOVERY - puppet last run on wtp1010 is OK: OK: Puppet is currently enabled, last run 9 seconds ago with 0 failures [17:55:21] !log rebooting ms-be1006, ping-dead in icinga for 23m, console was unresponsive [17:55:25] Logged the message, Master [17:55:29] RECOVERY - puppet last run on terbium is OK: OK: Puppet is currently enabled, last run 18 seconds ago with 0 failures [17:55:29] RECOVERY - puppet last run on cp1044 is OK: OK: Puppet is currently enabled, last run 48 seconds ago with 0 failures [17:55:38] RECOVERY - puppet last run on mw1220 is OK: OK: Puppet is currently enabled, last run 24 seconds ago with 0 failures [17:55:38] RECOVERY - puppet last run on mw1107 is OK: OK: Puppet is currently enabled, last run 47 seconds ago with 0 failures [17:55:39] RECOVERY - puppet last run on db60 is OK: OK: Puppet is currently enabled, last run 51 seconds ago with 0 failures [17:55:39] RECOVERY - puppet last run on wtp1001 is OK: OK: Puppet is currently enabled, last run 0 seconds ago with 0 failures [17:55:48] RECOVERY - puppet last run on db1072 is OK: OK: Puppet is currently enabled, last run 60 seconds ago with 0 failures [17:55:49] RECOVERY - puppet last run on db71 is OK: OK: Puppet is currently enabled, last run 56 seconds ago with 0 failures [17:55:49] RECOVERY - puppet last run on db1037 is OK: OK: Puppet is currently enabled, last run 2 seconds ago with 0 failures [17:55:49] RECOVERY - puppet last run on mw1071 is OK: OK: Puppet is currently enabled, last run 35 seconds ago with 0 failures [17:55:49] RECOVERY - puppet last run on cp1040 is OK: OK: Puppet is currently enabled, last run 57 seconds ago with 0 failures [17:55:49] RECOVERY - puppet last run on mw1204 is OK: OK: Puppet is currently enabled, last run 7 seconds ago with 0 failures [17:55:49] RECOVERY - puppet last run on ms-fe1002 is OK: OK: Puppet is currently enabled, last run 8 seconds ago with 0 failures [17:56:08] RECOVERY - puppet last run on bast1001 is OK: OK: Puppet is currently enabled, last run 49 seconds ago with 0 failures [17:56:08] RECOVERY - puppet last run on cp1054 is OK: OK: Puppet is currently enabled, last run 10 seconds ago with 0 failures [17:56:08] RECOVERY - puppet last run on cp1068 is OK: OK: Puppet is currently enabled, last run 19 seconds ago with 0 failures [17:56:08] RECOVERY - puppet last run on analytics1031 is OK: OK: Puppet is currently enabled, last run 17 seconds ago with 0 failures [17:56:08] RECOVERY - puppet last run on wtp1008 is OK: OK: Puppet is currently enabled, last run 20 seconds ago with 0 failures [17:56:09] RECOVERY - puppet last run on chromium is OK: OK: Puppet is currently enabled, last run 53 seconds ago with 0 failures [17:56:09] RECOVERY - puppet last run on ms-be1015 is OK: OK: Puppet is currently enabled, last run 20 seconds ago with 0 failures [17:56:09] RECOVERY - puppet last run on cp4009 is OK: OK: Puppet is currently enabled, last run 18 seconds ago with 0 failures [17:56:10] RECOVERY - puppet last run on db1035 is OK: OK: Puppet is currently enabled, last run 25 seconds ago with 0 failures [17:56:10] RECOVERY - puppet last run on db1056 is OK: OK: Puppet is currently enabled, last run 26 seconds ago with 0 failures [17:56:11] RECOVERY - puppet last run on mw1090 is OK: OK: Puppet is currently enabled, last run 19 seconds ago with 0 failures [17:56:11] RECOVERY - puppet last run on search1004 is OK: OK: Puppet is currently enabled, last run 29 seconds ago with 0 failures [17:56:18] RECOVERY - puppet last run on amssq39 is OK: OK: Puppet is currently enabled, last run 27 seconds ago with 0 failures [17:56:18] RECOVERY - puppet last run on db1045 is OK: OK: Puppet is currently enabled, last run 20 seconds ago with 0 failures [17:56:29] RECOVERY - puppet last run on copper is OK: OK: Puppet is currently enabled, last run 50 seconds ago with 0 failures [17:56:29] RECOVERY - puppet last run on osm-cp1001 is OK: OK: Puppet is currently enabled, last run 50 seconds ago with 0 failures [17:56:29] RECOVERY - puppet last run on cp1045 is OK: OK: Puppet is currently enabled, last run 44 seconds ago with 0 failures [17:56:29] RECOVERY - puppet last run on mw1158 is OK: OK: Puppet is currently enabled, last run 12 seconds ago with 0 failures [17:56:38] RECOVERY - puppet last run on lvs1004 is OK: OK: Puppet is currently enabled, last run 17 seconds ago with 0 failures [17:56:38] RECOVERY - puppet last run on mw1113 is OK: OK: Puppet is currently enabled, last run 44 seconds ago with 0 failures [17:56:38] RECOVERY - puppet last run on mw1143 is OK: OK: Puppet is currently enabled, last run 41 seconds ago with 0 failures [17:56:39] RECOVERY - puppet last run on mw1027 is OK: OK: Puppet is currently enabled, last run 47 seconds ago with 0 failures [17:56:39] RECOVERY - puppet last run on mw1086 is OK: OK: Puppet is currently enabled, last run 43 seconds ago with 0 failures [17:56:48] RECOVERY - puppet last run on mw1104 is OK: OK: Puppet is currently enabled, last run 24 seconds ago with 0 failures [17:56:48] RECOVERY - puppet last run on mw1215 is OK: OK: Puppet is currently enabled, last run 57 seconds ago with 0 failures [17:56:48] RECOVERY - puppet last run on mw1037 is OK: OK: Puppet is currently enabled, last run 26 seconds ago with 0 failures [17:56:48] RECOVERY - puppet last run on mw1064 is OK: OK: Puppet is currently enabled, last run 60 seconds ago with 0 failures [17:56:48] RECOVERY - puppet last run on cp1053 is OK: OK: Puppet is currently enabled, last run 32 seconds ago with 0 failures [17:56:48] RECOVERY - puppet last run on mw1155 is OK: OK: Puppet is currently enabled, last run 28 seconds ago with 0 failures [17:56:48] RECOVERY - puppet last run on cp4002 is OK: OK: Puppet is currently enabled, last run 15 seconds ago with 0 failures [17:56:58] RECOVERY - puppet last run on mw1207 is OK: OK: Puppet is currently enabled, last run 32 seconds ago with 0 failures [17:56:58] RECOVERY - puppet last run on ssl3001 is OK: OK: Puppet is currently enabled, last run 35 seconds ago with 0 failures [17:56:58] RECOVERY - puppet last run on eeden is OK: OK: Puppet is currently enabled, last run 41 seconds ago with 0 failures [17:57:08] RECOVERY - puppet last run on mw1154 is OK: OK: Puppet is currently enabled, last run 44 seconds ago with 0 failures [17:57:08] RECOVERY - puppet last run on mw1110 is OK: OK: Puppet is currently enabled, last run 48 seconds ago with 0 failures [17:57:09] RECOVERY - puppet last run on tmh1001 is OK: OK: Puppet is currently enabled, last run 3 seconds ago with 0 failures [17:57:09] RECOVERY - puppet last run on mw1203 is OK: OK: Puppet is currently enabled, last run 49 seconds ago with 0 failures [17:57:09] RECOVERY - puppet last run on mw1021 is OK: OK: Puppet is currently enabled, last run 51 seconds ago with 0 failures [17:57:09] RECOVERY - puppet last run on mw1135 is OK: OK: Puppet is currently enabled, last run 49 seconds ago with 0 failures [17:57:09] RECOVERY - puppet last run on mw1112 is OK: OK: Puppet is currently enabled, last run 53 seconds ago with 0 failures [17:57:10] RECOVERY - puppet last run on cp4012 is OK: OK: Puppet is currently enabled, last run 51 seconds ago with 0 failures [17:57:18] RECOVERY - puppet last run on cp3011 is OK: OK: Puppet is currently enabled, last run 46 seconds ago with 0 failures [17:57:18] RECOVERY - puppet last run on lvs1003 is OK: OK: Puppet is currently enabled, last run 28 seconds ago with 0 failures [17:57:18] RECOVERY - puppet last run on cp3018 is OK: OK: Puppet is currently enabled, last run 49 seconds ago with 0 failures [17:57:18] RECOVERY - puppet last run on cp3005 is OK: OK: Puppet is currently enabled, last run 34 seconds ago with 0 failures [17:57:29] RECOVERY - puppet last run on mw1193 is OK: OK: Puppet is currently enabled, last run 47 seconds ago with 0 failures [17:57:38] RECOVERY - puppet last run on mw1103 is OK: OK: Puppet is currently enabled, last run 23 seconds ago with 0 failures [17:57:49] RECOVERY - puppet last run on cp4011 is OK: OK: Puppet is currently enabled, last run 22 seconds ago with 0 failures [17:57:58] RECOVERY - puppet last run on mw1194 is OK: OK: Puppet is currently enabled, last run 56 seconds ago with 0 failures [17:57:58] RECOVERY - puppet last run on amssq37 is OK: OK: Puppet is currently enabled, last run 41 seconds ago with 0 failures [17:58:19] RECOVERY - Host ms-be1006 is UP: PING OK - Packet loss = 0%, RTA = 0.69 ms [18:02:52] !log ms-be1006: syslog indicates it started generating repeated "BUG: soft lockup" 10 minutes before dying, in XFS kernel code again... [18:02:58] Logged the message, Master [18:11:47] !log powering off amssq33, it's clipping network traffic at peak times due to bad ethernet connection negotiated down to 100Mbps (see existing RT 7933 in esams queue) [18:11:53] Logged the message, Master [18:12:23] !log (amssq33: and yes, removing from fe/be cache pools) [18:12:29] Logged the message, Master [18:12:58] RECOVERY - Puppet freshness on db1006 is OK: puppet ran at Sat Aug 16 18:12:56 UTC 2014 [18:15:28] (03PS1) 10BBlack: take amssq33 out of backend texts lists [operations/puppet] - 10https://gerrit.wikimedia.org/r/154427 [18:15:45] (03CR) 10BBlack: [C: 032 V: 032] take amssq33 out of backend texts lists [operations/puppet] - 10https://gerrit.wikimedia.org/r/154427 (owner: 10BBlack) [18:30:48] PROBLEM - puppet last run on labnet1001 is CRITICAL: CRITICAL: Puppet has 1 failures [18:46:18] PROBLEM - ElasticSearch health check on elastic1008 is CRITICAL: CRITICAL - elasticsearch (production-search-eqiad) is running. status: red: timed_out: false: number_of_nodes: 17: number_of_data_nodes: 17: active_primary_shards: 2037: active_shards: 6110: relocating_shards: 0: initializing_shards: 1: unassigned_shards: 1 [18:47:18] RECOVERY - ElasticSearch health check on elastic1008 is OK: OK - elasticsearch (production-search-eqiad) is running. status: green: timed_out: false: number_of_nodes: 17: number_of_data_nodes: 17: active_primary_shards: 2038: active_shards: 6113: relocating_shards: 0: initializing_shards: 0: unassigned_shards: 0 [18:47:48] RECOVERY - puppet last run on labnet1001 is OK: OK: Puppet is currently enabled, last run 10 seconds ago with 0 failures [19:00:09] PROBLEM - puppet last run on cp4010 is CRITICAL: CRITICAL: Puppet has 1 failures [19:18:09] RECOVERY - puppet last run on cp4010 is OK: OK: Puppet is currently enabled, last run 42 seconds ago with 0 failures [19:29:28] PROBLEM - Disk space on oxygen is CRITICAL: DISK CRITICAL - free space: / 4253 MB (3% inode=99%): [20:00:58] PROBLEM - Puppet freshness on db1010 is CRITICAL: Last successful Puppet run was Sat 16 Aug 2014 17:59:56 UTC [20:21:32] (03CR) 10Perhelion: [C: 04-1] Fix $wgRestrictionLevels ordering [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/154376 (https://bugzilla.wikimedia.org/69640) (owner: 10Anomie) [20:22:14] by gods [20:22:25] our community really is seriously brain-damanged [20:22:46] MatmaRex: citation needed! :D [20:22:53] hashar: [22:21] (CR) Perhelion: [C: -1] Fix $wgRestrictionLevels ordering [operations/mediawiki-config] - https://gerrit.wikimedia.org/r/154376 (https://bugzilla.wikimedia.org/69640) (owner: Anomie) [20:23:01] there are three more -1s on that patch [20:23:25] this is the stupidest thing i've seen this year [20:23:52] MatmaRex: seems to be just about properly ordering the rights in some form [20:23:56] I guess it is a UI issue [20:23:57] it is [20:24:08] like i said, the people -1ing this are just retarded [20:24:17] why don't they just -1 all the patches in the repo [20:24:20] somehow malfuncitoning on a basic level [20:24:21] would be more useful [20:24:38] unable to understand how life works [20:24:51] i just don't get it [20:25:05] can we introduce a code-based captcha to gerrit login? [20:25:18] like [20:25:22] $a = 5; [20:25:33] $b = $a * $a + $a; [20:25:37] what is the value of b [20:25:40] seriously [20:25:52] MatmaRex: make them figure out what http://benkurtovic.com/2014/06/01/obfuscating-hello-world.html does [20:26:41] pff [20:26:48] i am trying to understand the sort order issue :D [20:29:41] pff [20:33:52] (03CR) 10Hashar: [C: 031 V: 031] "I have removed all unrelated -1 votes which are related to super protect. This should be talked about on wikitech-l or wikimedia-l or wher" [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/154376 (https://bugzilla.wikimedia.org/69640) (owner: 10Anomie) [20:34:06] MatmaRex: good way to trick me into reviewing the change hehe [20:35:55] MatmaRex: legoktm: I have removed all the non sense -1 Patch is good, too late for me to deploy it though [20:38:39] hashar: heh [20:40:08] RECOVERY - Puppet freshness on db1010 is OK: puppet ran at Sat Aug 16 20:40:00 UTC 2014 [20:42:58] (03PS1) 10Legoktm: Enable GlobalCssJs on all CentralAuth wikis minus loginwiki [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/154432 (https://bugzilla.wikimedia.org/57891) [20:44:18] minus loginwiki? :O [20:44:45] does that mean I can't use my deletion scripts there? :/ [20:47:18] the extension wouldn't work there anyways, so we might as well not enable it [20:47:30] MatmaRex: fuck you if you can't understand the reasoning behind the -1 [20:47:39] and stop calling people retards, you fuck [20:48:12] odder: please change your language [20:48:26] hashar: and yet you didn't order MatmaRex to change his language? [20:48:33] what the fuck. [20:48:51] nah, that's normal language on Commons, see also INeverCry's comments on meta [20:49:19] OldManOfAran: true but irrelevant right now :p [20:49:36] odder: he was merely saying that some community is retarded. when you are directly insulting him. That is the difference. [20:49:57] No, he was directly insulting me and other people who -1'd that patch. [20:50:09] which is imho retarded [20:50:12] And your reaction was zero, so just get off my back. [20:50:20] granted I already told you and other that Gerrit was not the place to protest [20:50:35] you would reach a wider and more interested audience by bringing the subject to wikitech-l / meta or whatever [20:50:38] And that's reason to insult people, in your opinion? [20:50:45] but neither Gerrit nor Bugzilla are a good place for such a protest. [20:50:58] that is just making some engineers waste some time [20:51:13] there is never a good reason to insult people [20:51:15] Treat people the same is all I am saying. [20:51:33] He insulted me, and you didn't react, so just stop telling me what to do. [20:51:38] but seeing people -1 ing changes over and over when they pretty well know the vote is not going to change anything. Yeah that is not very clever [20:52:01] and as I said he pointed the community. Not you personally [20:52:04] and yeah [20:52:11] please, learn to read. [20:52:13] -1 ing patches is not smart [20:52:21] 22:24 MatmaRex: like i said, the people -1ing this are just retarded [20:52:39] that's also your timezone, so no need to look far in the backlog. [20:52:47] yeah I have read it [20:52:52] that prompted me to review the patch [20:52:57] (which is good code wise) [20:53:06] and you didn't react, but you did react when I insulted him. [20:53:09] the issue is really to fix the ordering [20:53:12] good riddance. [20:53:19] not adding/removing whatever permission you might complain about [20:53:26] hashar: so that is how we get you to review code? I thought I had been doing it wrong ;) [20:53:53] that is why -1 ing that change in protest is not really nice. That just waste time of Brad and Me reviewing a patch which is unrelated to the super protect permission. [20:54:23] JohnLewis: if the change is simple enough and I have some spare brain time, I try to review on sight :-] [20:54:41] That's not my point, hashar. My point is that you are using double morals towards myself and MatmaRex [20:54:48] JohnLewis: otherwise, I tend to dismiss most review requests unless that is an area where my code review can make a difference / tremendously help a submitter. [20:54:56] Same stuff, totally different reactions. That's 'not really nice.' [20:54:58] JohnLewis: so I mainly review volunteers code :] [20:55:02] odder: fuck me indeed, i should have kept my mouth shut. now can we stop fucking up this serious channel? :/ [20:55:13] hashar: well review my code when I submit some :D [20:55:47] also, fuck fuckity fuck, because this hasn't been said enough times yet [20:56:11] MatmaRex: stop fucking saying that fucking word. :p [20:56:11] It was, at 13:27 today :-) [20:56:28] Fuck you JohnLewis, you arrogant shitbag. [20:56:35] Be nice. [20:57:03] OldManOfAran: If you are going to be like that - Don't steal it off INC. [20:57:19] sorry, forgot the quotation marks [20:57:37] OldManOfAran: That's better :D [20:58:22] MatmaRex: Yes, we can stop fucking up this channel the moment you stop insulting people in here. [20:58:35] No hard feelings though, I know this might be a bit irritating. [20:58:38] okay [20:59:11] RD: You see; it is the weekend. Ops aren't paid to be nice during the weekends just be here to solve broken clusters :D [20:59:44] Apparently [21:00:19] though we can page ops if needed :] [21:21:39] PROBLEM - puppet last run on es10 is CRITICAL: CRITICAL: Epic puppet fail [21:40:39] RECOVERY - puppet last run on es10 is OK: OK: Puppet is currently enabled, last run 44 seconds ago with 0 failures [21:54:42] (03PS1) 10Vogone: Add 'global-renamer' user group to metawiki [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/154440 (https://bugzilla.wikimedia.org/69651) [22:20:54] (03CR) 10Rschen7754: [C: 031] Add 'global-renamer' user group to metawiki [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/154440 (https://bugzilla.wikimedia.org/69651) (owner: 10Vogone) [22:26:11] (03PS3) 10Ori.livneh: HHVM: improvements to logging [operations/puppet] - 10https://gerrit.wikimedia.org/r/154414 [22:57:58] PROBLEM - Puppet freshness on db1009 is CRITICAL: Last successful Puppet run was Sat 16 Aug 2014 20:56:59 UTC [23:03:09] PROBLEM - Packetloss_Average on oxygen is CRITICAL: packet_loss_average CRITICAL: 54.8310445455 [23:16:39] (03CR) 10MF-Warburg: "Why not on loginwiki?" [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/154432 (https://bugzilla.wikimedia.org/57891) (owner: 10Legoktm) [23:19:11] (03CR) 10Legoktm: "loginwiki has $wgAllowUserJs and $wgAllowUserCss set to false, so the extension wouldn't do anything anyways, so there's no point in enabl" [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/154432 (https://bugzilla.wikimedia.org/57891) (owner: 10Legoktm) [23:38:18] RECOVERY - Puppet freshness on db1009 is OK: puppet ran at Sat Aug 16 23:38:11 UTC 2014 [23:49:35] (03CR) 10Ori.livneh: [C: 04-1] "I'd like to move out the PHP packages into a new class, that can be included or excluded in toto. I'll amend the patch." [operations/puppet] - 10https://gerrit.wikimedia.org/r/153772 (owner: 10Giuseppe Lavagetto) [23:52:58] PROBLEM - Puppet freshness on db1006 is CRITICAL: Last successful Puppet run was Sat 16 Aug 2014 21:52:33 UTC [23:59:18] PROBLEM - puppet last run on amssq52 is CRITICAL: CRITICAL: Epic puppet fail