[00:00:53] 3Beta-Cluster: Remove beta specific mediawiki roles - https://phabricator.wikimedia.org/T87210#985990 (10yuvipanda) 3NEW [00:01:32] 3ops-core, operations, Wikimedia-Git-or-Gerrit: Chrome warns about insecure certificate on gerrit.wikimedia.org - https://phabricator.wikimedia.org/T76562#985997 (10mark) a:3RobH [00:02:40] (03PS3) 10Yuvipanda: mediawiki: Move jobrunner config into hiera [puppet] - 10https://gerrit.wikimedia.org/r/185931 (https://phabricator.wikimedia.org/T87210) [00:02:42] (03PS1) 10Yuvipanda: beta: Kill videoscaler role [puppet] - 10https://gerrit.wikimedia.org/r/185932 (https://phabricator.wikimedia.org/T87210) [00:03:29] 3ops-core, operations, Wikimedia-Git-or-Gerrit: Chrome warns about insecure certificate on gerrit.wikimedia.org - https://phabricator.wikimedia.org/T76562#805957 (10mark) Alright, if it's complicated to do with misc-web, let's buy a one-off ticket for it then. [00:03:55] 3ops-core: mailman - replace lighttpd - https://phabricator.wikimedia.org/T84053#986003 (10faidon) [00:04:07] 3ops-core: Replace all instances of lighttpd with nginx - https://phabricator.wikimedia.org/T84041#986010 (10faidon) [00:04:10] 3ops-core: mailman - replace lighttpd - https://phabricator.wikimedia.org/T84053#922050 (10faidon) 5Open>3Resolved a:3faidon Done with b949e96e00e32b5989f4a6f0f79064ab64c45d6f. [00:04:29] 3ops-core: Replace all instances of lighttpd with nginx - https://phabricator.wikimedia.org/T84041#921832 (10faidon) [00:04:31] 3ops-core: download server - use nginx instead of lighttpd - https://phabricator.wikimedia.org/T84021#986011 (10faidon) 5Open>3Resolved Done with 9675b0fe32407533050c0bf467f63aadd5e6fdd0. [00:06:40] (03PS1) 10Giuseppe Lavagetto: lvs: move the hiera file to the correct location [puppet] - 10https://gerrit.wikimedia.org/r/185933 [00:07:25] (03CR) 10Giuseppe Lavagetto: [C: 032 V: 032] lvs: move the hiera file to the correct location [puppet] - 10https://gerrit.wikimedia.org/r/185933 (owner: 10Giuseppe Lavagetto) [00:07:53] 3ops-core: Replace all instances of lighttpd with nginx - https://phabricator.wikimedia.org/T84041#986016 (10faidon) 5Open>3Resolved a:3faidon lighttpd is gone from production now. Tool Labs still uses it but isn't going to go away anytime soon, as we were giving users the option to ship their own lighttpd... [00:09:19] PROBLEM - puppet last run on lvs4001 is CRITICAL: CRITICAL: puppet fail [00:10:29] RECOVERY - puppet last run on lvs4001 is OK: OK: Puppet is currently enabled, last run 30 seconds ago with 0 failures [00:16:44] (03PS3) 10Giuseppe Lavagetto: memcached: use role, hiera [puppet] - 10https://gerrit.wikimedia.org/r/184928 [00:18:46] (03PS2) 10Yuvipanda: beta: Kill videoscaler role [puppet] - 10https://gerrit.wikimedia.org/r/185932 (https://phabricator.wikimedia.org/T87210) [00:18:48] (03PS4) 10Yuvipanda: mediawiki: Move jobrunner config into hiera [puppet] - 10https://gerrit.wikimedia.org/r/185931 (https://phabricator.wikimedia.org/T87210) [00:19:19] RECOVERY - Apache HTTP on mw1176 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 440 bytes in 0.731 second response time [00:19:30] RECOVERY - HHVM rendering on mw1176 is OK: HTTP OK: HTTP/1.1 200 OK - 64897 bytes in 0.146 second response time [00:20:06] _joe_: is https://phabricator.wikimedia.org/T81353 still relevant? I suspect not [00:20:58] <_joe_> I suspect too [00:21:02] <_joe_> :) [00:21:38] (03PS1) 10Springle: deploy dbstore2001 dbstore2002 [puppet] - 10https://gerrit.wikimedia.org/r/185936 [00:23:37] (03CR) 10Springle: [C: 032] deploy dbstore2001 dbstore2002 [puppet] - 10https://gerrit.wikimedia.org/r/185936 (owner: 10Springle) [00:28:35] 3ops-core: cp* boxes, pagecache issues & trying newer kernels - https://phabricator.wikimedia.org/T83809#986109 (10faidon) [00:29:46] (03PS1) 10Alexandros Kosiaris: WIP: Move ulsfo to ganglia_new module [puppet] - 10https://gerrit.wikimedia.org/r/185937 [00:30:08] (03PS4) 10Giuseppe Lavagetto: memcached: use role, hiera [puppet] - 10https://gerrit.wikimedia.org/r/184928 [00:30:26] (03CR) 10jenkins-bot: [V: 04-1] WIP: Move ulsfo to ganglia_new module [puppet] - 10https://gerrit.wikimedia.org/r/185937 (owner: 10Alexandros Kosiaris) [00:30:48] (03PS1) 10Springle: correct mysql ganglia aggregators for eqiad T87209 [puppet] - 10https://gerrit.wikimedia.org/r/185938 [00:31:06] 3ops-core: cp* boxes, pagecache issues & trying newer kernels - https://phabricator.wikimedia.org/T83809#986120 (10faidon) This is about to become more relevant (or fixed :)) with T86648. [00:31:47] _joe_: ^ seem ok? [00:32:25] _joe_: hmm, https://wikitech.wikimedia.org/wiki/Hiera:Deployment-prep. referring to queue_servers in aggr_servers doesn’t actually seem to work... [00:33:33] (03PS2) 10Alexandros Kosiaris: WIP: Move ulsfo to ganglia_new module [puppet] - 10https://gerrit.wikimedia.org/r/185937 [00:37:21] (03CR) 10Giuseppe Lavagetto: [C: 031] correct mysql ganglia aggregators for eqiad T87209 [puppet] - 10https://gerrit.wikimedia.org/r/185938 (owner: 10Springle) [00:43:40] 3ops-core: Upgrade stat1002 to trusty - https://phabricator.wikimedia.org/T84370#986150 (10faidon) [00:44:03] 3ops-core: Upgrade stat1002 to trusty - https://phabricator.wikimedia.org/T84370#986155 (10faidon) 5Open>3Resolved a:3faidon stat1002 seems to be trusty now. [00:53:52] (03PS1) 10Yuvipanda: beta: Kill beta specific jobrunner class [puppet] - 10https://gerrit.wikimedia.org/r/185939 (https://phabricator.wikimedia.org/T87210) [00:55:58] (03PS2) 10Yuvipanda: beta: Kill beta specific jobrunner class [puppet] - 10https://gerrit.wikimedia.org/r/185939 (https://phabricator.wikimedia.org/T87210) [00:56:00] (03PS3) 10Yuvipanda: beta: Kill videoscaler role [puppet] - 10https://gerrit.wikimedia.org/r/185932 (https://phabricator.wikimedia.org/T87210) [00:56:02] (03PS5) 10Yuvipanda: mediawiki: Move jobrunner config into hiera [puppet] - 10https://gerrit.wikimedia.org/r/185931 (https://phabricator.wikimedia.org/T87210) [00:56:08] _joe_: ^ updated for the aggr_servers stuf [00:57:19] (03PS1) 10Giuseppe Lavagetto: hiera: fix up for labs hierarchy [puppet] - 10https://gerrit.wikimedia.org/r/185940 [00:58:44] (03CR) 10Springle: [C: 032] correct mysql ganglia aggregators for eqiad T87209 [puppet] - 10https://gerrit.wikimedia.org/r/185938 (owner: 10Springle) [00:59:58] (03CR) 10Giuseppe Lavagetto: [C: 032] memcached: use role, hiera [puppet] - 10https://gerrit.wikimedia.org/r/184928 (owner: 10Giuseppe Lavagetto) [01:00:09] (03PS5) 10Giuseppe Lavagetto: memcached: use role, hiera [puppet] - 10https://gerrit.wikimedia.org/r/184928 [01:02:06] !log power-cycling rhenium [01:02:09] Logged the message, Master [01:03:55] (03PS2) 10Yuvipanda: Tools: Install valgrind on bastions [puppet] - 10https://gerrit.wikimedia.org/r/185664 (https://phabricator.wikimedia.org/T87117) (owner: 10Tim Landscheidt) [01:04:21] (03CR) 10Yuvipanda: [C: 032] Tools: Install valgrind on bastions [puppet] - 10https://gerrit.wikimedia.org/r/185664 (https://phabricator.wikimedia.org/T87117) (owner: 10Tim Landscheidt) [01:04:48] RECOVERY - RAID on rhenium is OK: OK: Active: 6, Working: 6, Failed: 0, Spare: 0 [01:04:58] RECOVERY - salt-minion processes on rhenium is OK: PROCS OK: 1 process with regex args ^/usr/bin/python /usr/bin/salt-minion [01:05:18] RECOVERY - dhclient process on rhenium is OK: PROCS OK: 0 processes with command name dhclient [01:05:28] RECOVERY - DPKG on rhenium is OK: All packages OK [01:05:28] RECOVERY - puppet last run on rhenium is OK: OK: Puppet is currently enabled, last run 1 second ago with 0 failures [01:05:38] RECOVERY - SSH on rhenium is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1.4 (protocol 2.0) [01:05:38] RECOVERY - Disk space on rhenium is OK: DISK OK [01:05:38] RECOVERY - configured eth on rhenium is OK: NRPE: Unable to read output [01:05:59] !log rebooting netmon1001, upstart stuck(?!) [01:06:04] Logged the message, Master [01:07:30] 3ops-core: Upgrade Exim to >=4.73 - https://phabricator.wikimedia.org/T83541#986188 (10faidon) [01:08:03] 3ops-core: Status of ms1004? - https://phabricator.wikimedia.org/T84435#986190 (10faidon) 5Open>3Resolved Decom tracked with T86933. [01:08:04] 3ops-core: Get rid of all Ubuntu Lucid (10.04) installs - https://phabricator.wikimedia.org/T80945#986194 (10faidon) [01:08:52] 3ops-core: Get rid of all Ubuntu Lucid (10.04) installs - https://phabricator.wikimedia.org/T80945#986196 (10yuvipanda) Note that when all these go, deployment-lucid on deployment-prep can also go :) It currently is used by @apergos to test salt upgrades. [01:09:17] PROBLEM - LibreNMS HTTPS on netmon1001 is CRITICAL: Connection refused [01:09:38] PROBLEM - SSH on netmon1001 is CRITICAL: Connection refused [01:10:07] PROBLEM - salt-minion processes on netmon1001 is CRITICAL: Connection refused by host [01:11:22] (03PS6) 10Giuseppe Lavagetto: memcached: use role, hiera [puppet] - 10https://gerrit.wikimedia.org/r/184928 [01:11:32] (03CR) 10Giuseppe Lavagetto: [C: 032] memcached: use role, hiera [puppet] - 10https://gerrit.wikimedia.org/r/184928 (owner: 10Giuseppe Lavagetto) [01:12:38] PROBLEM - Host netmon1001 is DOWN: PING CRITICAL - Packet loss = 100% [01:13:08] RECOVERY - SSH on netmon1001 is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1.4 (protocol 2.0) [01:13:17] RECOVERY - Host netmon1001 is UP: PING OK - Packet loss = 0%, RTA = 1.46 ms [01:13:38] RECOVERY - salt-minion processes on netmon1001 is OK: PROCS OK: 1 process with regex args ^/usr/bin/python /usr/bin/salt-minion [01:14:07] RECOVERY - LibreNMS HTTPS on netmon1001 is OK: HTTP OK: HTTP/1.1 200 OK - 6604 bytes in 0.235 second response time [01:14:18] RECOVERY - puppet last run on netmon1001 is OK: OK: Puppet is currently enabled, last run 9 seconds ago with 0 failures [01:16:17] (03PS3) 10Giuseppe Lavagetto: redis: user role, hiera [puppet] - 10https://gerrit.wikimedia.org/r/184929 [01:22:48] 3operations, Labs-Team: stray files created in /etc/ssh/userkeys - https://phabricator.wikimedia.org/T85814#986231 (10faidon) a:3faidon [01:24:47] RECOVERY - NTP on rhenium is OK: NTP OK: Offset -0.01095795631 secs [01:26:43] _joe_: mind if I merge https://gerrit.wikimedia.org/r/#/c/185931/? [01:26:46] (03PS4) 10Giuseppe Lavagetto: redis: user role, hiera [puppet] - 10https://gerrit.wikimedia.org/r/184929 [01:30:27] PROBLEM - NTP on netmon1001 is CRITICAL: NTP CRITICAL: Offset unknown [01:32:27] (03PS1) 10Yuvipanda: beta: Kill beta specific mediawiki logging role [puppet] - 10https://gerrit.wikimedia.org/r/185946 (https://phabricator.wikimedia.org/T87210) [01:32:47] RECOVERY - NTP on netmon1001 is OK: NTP OK: Offset -0.002358675003 secs [01:33:16] (03CR) 10Ori.livneh: [C: 031] beta: Kill beta specific mediawiki logging role [puppet] - 10https://gerrit.wikimedia.org/r/185946 (https://phabricator.wikimedia.org/T87210) (owner: 10Yuvipanda) [01:33:32] (03CR) 10Ori.livneh: [C: 031] beta: Kill beta specific jobrunner class [puppet] - 10https://gerrit.wikimedia.org/r/185939 (https://phabricator.wikimedia.org/T87210) (owner: 10Yuvipanda) [01:33:49] (03CR) 10Ori.livneh: [C: 031] beta: Kill videoscaler role [puppet] - 10https://gerrit.wikimedia.org/r/185932 (https://phabricator.wikimedia.org/T87210) (owner: 10Yuvipanda) [01:33:51] (03PS5) 10Giuseppe Lavagetto: redis: user role, hiera [puppet] - 10https://gerrit.wikimedia.org/r/184929 [01:34:18] ori: review of https://gerrit.wikimedia.org/r/#/c/185931/ would be nice too! :D should be a nop. I cant get puppet compiler to work [01:36:00] (03CR) 10Ori.livneh: [C: 031] "Personally I find the layer of indirection that moving this data into Hiera introduces to be a little unpalatable. A good Puppet manifest " [puppet] - 10https://gerrit.wikimedia.org/r/185931 (https://phabricator.wikimedia.org/T87210) (owner: 10Yuvipanda) [01:36:07] YuviPanda: i gave you a cranky +1 [01:36:11] (03CR) 10Giuseppe Lavagetto: [C: 032] redis: user role, hiera [puppet] - 10https://gerrit.wikimedia.org/r/184929 (owner: 10Giuseppe Lavagetto) [01:37:00] ori: hmm. I’ll think about that before forming strong opinions, but right now it’s one puppet code base and two instances of it, so we do have to separate configuration from code [01:38:16] 3ops-core: Create icinga HTTPS check that supports SNI - https://phabricator.wikimedia.org/T84444#986255 (10faidon) a:3faidon [01:39:20] 3ops-core: Create icinga HTTPS check that supports SNI - https://phabricator.wikimedia.org/T84444#986259 (10faidon) 5Open>3Resolved We do check for the SNI certificates for a while now. check_ssl, the new SSL check, also supports that. [01:41:01] (03PS1) 10Yuvipanda: mediawiki: Fix incredibly annnoyyying stray space [puppet] - 10https://gerrit.wikimedia.org/r/185947 [01:41:09] (03PS2) 10Yuvipanda: mediawiki: Fix incredibly annnoyyying stray space [puppet] - 10https://gerrit.wikimedia.org/r/185947 [01:41:18] (03CR) 10Yuvipanda: [C: 032] mediawiki: Fix incredibly annnoyyying stray space [puppet] - 10https://gerrit.wikimedia.org/r/185947 (owner: 10Yuvipanda) [01:51:52] 3operations: Re-image osmium and repurpose into VE testing host - https://phabricator.wikimedia.org/T87215#986278 (10ori) 3NEW [02:03:50] (03PS6) 10Yuvipanda: mediawiki: Move jobrunner config into hiera [puppet] - 10https://gerrit.wikimedia.org/r/185931 (https://phabricator.wikimedia.org/T87210) [02:04:05] (03CR) 10Yuvipanda: [C: 032] mediawiki: Move jobrunner config into hiera [puppet] - 10https://gerrit.wikimedia.org/r/185931 (https://phabricator.wikimedia.org/T87210) (owner: 10Yuvipanda) [02:07:33] (03PS3) 10Yuvipanda: beta: Kill beta specific jobrunner class [puppet] - 10https://gerrit.wikimedia.org/r/185939 (https://phabricator.wikimedia.org/T87210) [02:07:35] (03PS4) 10Yuvipanda: beta: Kill videoscaler role [puppet] - 10https://gerrit.wikimedia.org/r/185932 (https://phabricator.wikimedia.org/T87210) [02:07:37] (03PS2) 10Yuvipanda: beta: Kill beta specific mediawiki logging role [puppet] - 10https://gerrit.wikimedia.org/r/185946 (https://phabricator.wikimedia.org/T87210) [02:07:39] (03PS1) 10Yuvipanda: mediawiki: Explicitly open port 80 on mediawiki webservers [puppet] - 10https://gerrit.wikimedia.org/r/185948 [02:07:59] akosiaris: _joe_ https://gerrit.wikimedia.org/r/#/c/185948/ I’m adding ferm roles when required, even if there’s no base::firewall [02:08:10] I think that’s a good idea, do -1 / -2 if you feel otherwise [02:08:14] paravoid: ^ [02:08:16] err [02:08:17] ferm rules [02:08:18] not roles [02:08:33] (03CR) 10Yuvipanda: [C: 032] beta: Kill videoscaler role [puppet] - 10https://gerrit.wikimedia.org/r/185932 (https://phabricator.wikimedia.org/T87210) (owner: 10Yuvipanda) [02:08:44] (03CR) 10Yuvipanda: [C: 032] beta: Kill beta specific jobrunner class [puppet] - 10https://gerrit.wikimedia.org/r/185939 (https://phabricator.wikimedia.org/T87210) (owner: 10Yuvipanda) [02:08:54] (03CR) 10Yuvipanda: [C: 032] beta: Kill beta specific mediawiki logging role [puppet] - 10https://gerrit.wikimedia.org/r/185946 (https://phabricator.wikimedia.org/T87210) (owner: 10Yuvipanda) [02:10:52] !log l10nupdate Synchronized php-1.25wmf14/cache/l10n: (no message) (duration: 00m 01s) [02:10:56] !log LocalisationUpdate completed (1.25wmf14) at 2015-01-20 02:10:56+00:00 [02:11:04] Logged the message, Master [02:11:07] Logged the message, Master [02:12:45] isn’t https://github.com/wikimedia/operations-puppet/blob/production/modules/beta/templates/pam-access.conf.erb moot now? [02:12:49] mwdeploy is in ldap [02:12:53] and should be able to ssh [02:16:42] (03CR) 10Alexandros Kosiaris: [C: 04-1] "LGTM, minor comment" (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/185948 (owner: 10Yuvipanda) [02:17:34] (03PS2) 10Yuvipanda: mediawiki: Explicitly open port 80 on mediawiki webservers [puppet] - 10https://gerrit.wikimedia.org/r/185948 [02:18:12] (03CR) 10Alexandros Kosiaris: [C: 032] mediawiki: Explicitly open port 80 on mediawiki webservers [puppet] - 10https://gerrit.wikimedia.org/r/185948 (owner: 10Yuvipanda) [02:19:00] !log l10nupdate Synchronized php-1.25wmf15/cache/l10n: (no message) (duration: 00m 03s) [02:19:04] !log LocalisationUpdate completed (1.25wmf15) at 2015-01-20 02:19:04+00:00 [02:19:09] Logged the message, Master [02:19:12] Logged the message, Master [02:24:58] (03PS1) 10Yuvipanda: beta: Remove explict ssh grant for mwdeploy user [puppet] - 10https://gerrit.wikimedia.org/r/185949 [02:25:40] (03CR) 10Yuvipanda: [C: 04-2] "mwdeploy isn't a member of deployment-prep because it doesn't have loginviashell, which is probably a good thing." [puppet] - 10https://gerrit.wikimedia.org/r/185949 (owner: 10Yuvipanda) [02:26:25] (03PS1) 10Alexandros Kosiaris: Update servermon's service_name [puppet] - 10https://gerrit.wikimedia.org/r/185950 [02:27:47] (03PS1) 10Springle: update role::ganglia::config $data_sources [puppet] - 10https://gerrit.wikimedia.org/r/185951 [02:29:07] (03CR) 10Springle: [C: 032] update role::ganglia::config $data_sources [puppet] - 10https://gerrit.wikimedia.org/r/185951 (owner: 10Springle) [02:36:39] (03PS1) 10Yuvipanda: mediawiki: Load prod/beta apache configurabion based on realm [puppet] - 10https://gerrit.wikimedia.org/r/185952 [02:36:48] PROBLEM - puppet last run on lvs4004 is CRITICAL: CRITICAL: puppet fail [02:36:57] PROBLEM - puppet last run on sca1001 is CRITICAL: CRITICAL: puppet fail [02:36:57] PROBLEM - puppet last run on lvs3003 is CRITICAL: CRITICAL: Puppet has 4 failures [02:36:57] PROBLEM - puppet last run on cp3018 is CRITICAL: CRITICAL: puppet fail [02:36:57] PROBLEM - puppet last run on ms-fe1002 is CRITICAL: CRITICAL: puppet fail [02:36:58] PROBLEM - puppet last run on chromium is CRITICAL: CRITICAL: puppet fail [02:36:58] PROBLEM - puppet last run on strontium is CRITICAL: CRITICAL: Puppet has 25 failures [02:37:07] PROBLEM - puppet last run on mw1071 is CRITICAL: CRITICAL: Puppet has 36 failures [02:37:07] PROBLEM - puppet last run on analytics1003 is CRITICAL: CRITICAL: Puppet has 12 failures [02:37:08] PROBLEM - puppet last run on wtp1010 is CRITICAL: CRITICAL: Puppet has 16 failures [02:37:08] PROBLEM - puppet last run on mw1090 is CRITICAL: CRITICAL: puppet fail [02:37:08] PROBLEM - puppetmaster backend https on strontium is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 8141: HTTP/1.1 500 Internal Server Error [02:37:17] PROBLEM - puppet last run on mw1064 is CRITICAL: CRITICAL: Puppet has 34 failures [02:37:17] PROBLEM - puppet last run on mw1220 is CRITICAL: CRITICAL: Puppet has 72 failures [02:37:18] PROBLEM - puppet last run on cp1067 is CRITICAL: CRITICAL: Puppet has 27 failures [02:37:18] PROBLEM - puppet last run on db1045 is CRITICAL: CRITICAL: Puppet has 21 failures [02:37:18] PROBLEM - puppet last run on cp1068 is CRITICAL: CRITICAL: puppet fail [02:37:18] PROBLEM - puppet last run on einsteinium is CRITICAL: CRITICAL: Puppet has 8 failures [02:37:18] PROBLEM - puppet last run on mw1193 is CRITICAL: CRITICAL: Puppet has 62 failures [02:37:18] PROBLEM - puppet last run on mw1107 is CRITICAL: CRITICAL: Puppet has 23 failures [02:37:18] PROBLEM - puppet last run on elastic1003 is CRITICAL: CRITICAL: Puppet has 27 failures [02:37:19] PROBLEM - puppet last run on mw1203 is CRITICAL: CRITICAL: Puppet has 60 failures [02:37:27] PROBLEM - puppet last run on mw1204 is CRITICAL: CRITICAL: Puppet has 68 failures [02:37:27] PROBLEM - puppet last run on mw1112 is CRITICAL: CRITICAL: Puppet has 63 failures [02:37:28] PROBLEM - puppet last run on mw1027 is CRITICAL: CRITICAL: Puppet has 60 failures [02:37:37] PROBLEM - puppet last run on cp4002 is CRITICAL: CRITICAL: Puppet has 16 failures [02:37:37] PROBLEM - puppet last run on cp3005 is CRITICAL: CRITICAL: puppet fail [02:37:37] PROBLEM - puppet last run on mw1155 is CRITICAL: CRITICAL: puppet fail [02:37:37] PROBLEM - puppet last run on mw1154 is CRITICAL: CRITICAL: Puppet has 29 failures [02:37:38] PROBLEM - puppet last run on db1037 is CRITICAL: CRITICAL: Puppet has 22 failures [02:37:38] PROBLEM - puppet last run on terbium is CRITICAL: CRITICAL: Puppet has 7 failures [02:37:38] PROBLEM - puppet last run on cp1053 is CRITICAL: CRITICAL: Puppet has 16 failures [02:37:47] PROBLEM - puppet last run on mw1066 is CRITICAL: CRITICAL: Puppet has 32 failures [02:37:47] PROBLEM - puppet last run on ms-be1015 is CRITICAL: CRITICAL: Puppet has 24 failures [02:37:47] PROBLEM - puppet last run on mw1236 is CRITICAL: CRITICAL: Puppet has 61 failures [02:37:48] PROBLEM - puppet last run on mw1131 is CRITICAL: CRITICAL: puppet fail [02:37:48] PROBLEM - puppet last run on bast1001 is CRITICAL: CRITICAL: Puppet has 8 failures [02:37:48] PROBLEM - puppet last run on lvs1003 is CRITICAL: CRITICAL: puppet fail [02:37:48] PROBLEM - puppet last run on achernar is CRITICAL: CRITICAL: puppet fail [02:37:48] PROBLEM - puppet last run on analytics1018 is CRITICAL: CRITICAL: Puppet has 10 failures [02:37:48] PROBLEM - puppet last run on mw1086 is CRITICAL: CRITICAL: puppet fail [02:37:49] PROBLEM - puppet last run on amssq31 is CRITICAL: CRITICAL: Puppet has 13 failures [02:37:57] PROBLEM - puppet last run on elastic1020 is CRITICAL: CRITICAL: Puppet has 9 failures [02:37:58] PROBLEM - puppet last run on mw1241 is CRITICAL: CRITICAL: Puppet has 76 failures [02:37:58] PROBLEM - puppet last run on mw1110 is CRITICAL: CRITICAL: puppet fail [02:37:58] PROBLEM - puppet last run on dysprosium is CRITICAL: CRITICAL: puppet fail [02:37:58] PROBLEM - puppet last run on wtp1001 is CRITICAL: CRITICAL: Puppet has 14 failures [02:38:07] PROBLEM - puppet last run on cp1040 is CRITICAL: CRITICAL: Puppet has 13 failures [02:38:07] PROBLEM - puppet last run on analytics1021 is CRITICAL: CRITICAL: puppet fail [02:38:07] PROBLEM - puppet last run on cp4009 is CRITICAL: CRITICAL: Puppet has 22 failures [02:38:07] PROBLEM - puppet last run on amssq37 is CRITICAL: CRITICAL: puppet fail [02:38:08] PROBLEM - puppet last run on lvs1004 is CRITICAL: CRITICAL: puppet fail [02:38:08] PROBLEM - puppet last run on db1056 is CRITICAL: CRITICAL: puppet fail [02:38:17] PROBLEM - puppet last run on baham is CRITICAL: CRITICAL: Puppet has 17 failures [02:38:17] PROBLEM - puppet last run on es2010 is CRITICAL: CRITICAL: puppet fail [02:38:17] PROBLEM - puppet last run on cp4016 is CRITICAL: CRITICAL: puppet fail [02:38:17] PROBLEM - puppet last run on cp4012 is CRITICAL: CRITICAL: Puppet has 22 failures [02:38:17] PROBLEM - puppet last run on cp3021 is CRITICAL: CRITICAL: puppet fail [02:38:18] PROBLEM - puppet last run on eeden is CRITICAL: CRITICAL: Puppet has 13 failures [02:38:18] PROBLEM - puppet last run on cp3011 is CRITICAL: CRITICAL: Puppet has 15 failures [02:38:18] PROBLEM - puppet last run on palladium is CRITICAL: CRITICAL: puppet fail [02:38:19] PROBLEM - puppet last run on amssq39 is CRITICAL: CRITICAL: Puppet has 40 failures [02:38:19] PROBLEM - puppet last run on amssq33 is CRITICAL: CRITICAL: Puppet has 19 failures [02:38:19] PROBLEM - puppet last run on mw1113 is CRITICAL: CRITICAL: puppet fail [02:38:20] PROBLEM - puppet last run on mw1158 is CRITICAL: CRITICAL: Puppet has 35 failures [02:38:21] PROBLEM - puppet last run on osm-cp1001 is CRITICAL: CRITICAL: puppet fail [02:38:21] PROBLEM - puppet last run on mw1253 is CRITICAL: CRITICAL: Puppet has 66 failures [02:38:21] PROBLEM - puppet last run on mw1143 is CRITICAL: CRITICAL: Puppet has 67 failures [02:38:22] PROBLEM - puppet last run on mw1255 is CRITICAL: CRITICAL: puppet fail [02:38:22] PROBLEM - puppet last run on cp1054 is CRITICAL: CRITICAL: Puppet has 16 failures [02:38:28] PROBLEM - puppet last run on mw1207 is CRITICAL: CRITICAL: Puppet has 75 failures [02:38:37] PROBLEM - puppet last run on mw1104 is CRITICAL: CRITICAL: puppet fail [02:38:37] PROBLEM - puppet last run on radon is CRITICAL: CRITICAL: Puppet has 11 failures [02:38:37] PROBLEM - puppet last run on cp1045 is CRITICAL: CRITICAL: puppet fail [02:38:37] PROBLEM - puppet last run on db1035 is CRITICAL: CRITICAL: Puppet has 21 failures [02:38:38] PROBLEM - puppet last run on mw1021 is CRITICAL: CRITICAL: puppet fail [02:38:38] PROBLEM - puppet last run on mw1047 is CRITICAL: CRITICAL: puppet fail [02:38:47] PROBLEM - puppet last run on wtp1008 is CRITICAL: CRITICAL: puppet fail [02:38:47] PROBLEM - puppet last run on mw1037 is CRITICAL: CRITICAL: puppet fail [02:38:48] PROBLEM - puppet last run on bast2001 is CRITICAL: CRITICAL: puppet fail [02:38:48] PROBLEM - puppet last run on amssq50 is CRITICAL: CRITICAL: puppet fail [02:38:58] PROBLEM - puppet last run on mw1194 is CRITICAL: CRITICAL: puppet fail [02:39:07] PROBLEM - puppet last run on radium is CRITICAL: CRITICAL: Puppet has 20 failures [02:39:07] PROBLEM - puppet last run on mw1135 is CRITICAL: CRITICAL: Puppet has 33 failures [02:39:07] PROBLEM - puppet last run on mw1137 is CRITICAL: CRITICAL: puppet fail [02:39:07] PROBLEM - puppet last run on db1027 is CRITICAL: CRITICAL: Puppet has 9 failures [02:39:08] PROBLEM - puppet last run on mw1073 is CRITICAL: CRITICAL: Puppet has 72 failures [02:39:08] PROBLEM - puppet last run on tmh1001 is CRITICAL: CRITICAL: Puppet has 15 failures [02:39:08] PROBLEM - puppet last run on elastic1025 is CRITICAL: CRITICAL: Puppet has 26 failures [02:39:08] PROBLEM - puppet last run on es1004 is CRITICAL: CRITICAL: puppet fail [02:39:17] PROBLEM - puppet last run on lvs2002 is CRITICAL: CRITICAL: Puppet has 13 failures [02:39:17] PROBLEM - puppet last run on db1049 is CRITICAL: CRITICAL: puppet fail [02:39:17] PROBLEM - puppet last run on es2006 is CRITICAL: CRITICAL: Puppet has 13 failures [02:39:18] PROBLEM - puppet last run on db2010 is CRITICAL: CRITICAL: Puppet has 24 failures [02:39:18] PROBLEM - puppet last run on db2017 is CRITICAL: CRITICAL: Puppet has 8 failures [02:39:18] PROBLEM - puppet last run on virt1009 is CRITICAL: CRITICAL: Puppet has 15 failures [02:39:27] PROBLEM - puppet last run on mw1128 is CRITICAL: CRITICAL: Puppet has 35 failures [02:39:27] PROBLEM - puppet last run on analytics1012 is CRITICAL: CRITICAL: Puppet has 13 failures [02:39:27] PROBLEM - puppet last run on mw1078 is CRITICAL: CRITICAL: puppet fail [02:39:27] PROBLEM - puppet last run on mc1015 is CRITICAL: CRITICAL: Puppet has 24 failures [02:39:27] PROBLEM - puppet last run on virt1005 is CRITICAL: CRITICAL: Puppet has 12 failures [02:39:28] PROBLEM - puppet last run on cp3007 is CRITICAL: CRITICAL: puppet fail [02:39:28] PROBLEM - puppet last run on mw1085 is CRITICAL: CRITICAL: puppet fail [02:39:38] PROBLEM - puppet last run on cp4011 is CRITICAL: CRITICAL: puppet fail [02:39:39] PROBLEM - puppet last run on cp3019 is CRITICAL: CRITICAL: Puppet has 21 failures [02:39:39] PROBLEM - puppet last run on elastic1013 is CRITICAL: CRITICAL: puppet fail [02:39:47] PROBLEM - puppet last run on db1068 is CRITICAL: CRITICAL: puppet fail [02:39:48] PROBLEM - puppet last run on mw1103 is CRITICAL: CRITICAL: Puppet has 63 failures [02:39:48] RECOVERY - puppetmaster backend https on strontium is OK: HTTP OK: Status line output matched 400 - 335 bytes in 1.008 second response time [02:39:48] PROBLEM - puppet last run on mw1095 is CRITICAL: CRITICAL: puppet fail [02:39:48] PROBLEM - puppet last run on lvs3002 is CRITICAL: CRITICAL: Puppet has 19 failures [02:39:57] PROBLEM - puppet last run on cp4010 is CRITICAL: CRITICAL: Puppet has 16 failures [02:39:57] PROBLEM - puppet last run on ms-be1010 is CRITICAL: CRITICAL: Puppet has 14 failures [02:39:58] PROBLEM - puppet last run on mw1157 is CRITICAL: CRITICAL: puppet fail [02:39:58] PROBLEM - puppet last run on mw1017 is CRITICAL: CRITICAL: puppet fail [02:39:58] PROBLEM - puppet last run on labsdb1005 is CRITICAL: CRITICAL: Puppet has 25 failures [02:39:58] PROBLEM - puppet last run on elastic1029 is CRITICAL: CRITICAL: Puppet has 16 failures [02:39:58] PROBLEM - puppet last run on cp1066 is CRITICAL: CRITICAL: Puppet has 27 failures [02:39:58] PROBLEM - puppet last run on mw1101 is CRITICAL: CRITICAL: Puppet has 68 failures [02:40:07] PROBLEM - puppet last run on mw1075 is CRITICAL: CRITICAL: puppet fail [02:40:08] PROBLEM - puppet last run on mw1199 is CRITICAL: CRITICAL: Puppet has 74 failures [02:40:08] PROBLEM - puppet last run on cp4015 is CRITICAL: CRITICAL: puppet fail [02:40:08] PROBLEM - puppet last run on ms-be3004 is CRITICAL: CRITICAL: puppet fail [02:40:08] PROBLEM - puppet last run on cp3017 is CRITICAL: CRITICAL: Puppet has 19 failures [02:40:08] PROBLEM - puppet last run on db1005 is CRITICAL: CRITICAL: Puppet has 12 failures [02:40:08] PROBLEM - puppet last run on elastic1010 is CRITICAL: CRITICAL: Puppet has 18 failures [02:40:08] PROBLEM - puppet last run on ms-be1014 is CRITICAL: CRITICAL: puppet fail [02:40:17] PROBLEM - puppet last run on mw1019 is CRITICAL: CRITICAL: puppet fail [02:40:17] PROBLEM - puppet last run on labsdb1002 is CRITICAL: CRITICAL: Puppet has 11 failures [02:40:17] PROBLEM - puppet last run on mw1191 is CRITICAL: CRITICAL: puppet fail [02:40:17] PROBLEM - puppet last run on cp1057 is CRITICAL: CRITICAL: puppet fail [02:40:17] PROBLEM - puppet last run on mw1230 is CRITICAL: CRITICAL: Puppet has 66 failures [02:40:18] PROBLEM - puppet last run on db1009 is CRITICAL: CRITICAL: puppet fail [02:40:18] PROBLEM - puppet last run on analytics1031 is CRITICAL: CRITICAL: Puppet has 13 failures [02:40:18] PROBLEM - puppet last run on mw1015 is CRITICAL: CRITICAL: puppet fail [02:40:27] PROBLEM - puppet last run on analytics1029 is CRITICAL: CRITICAL: Puppet has 11 failures [02:40:28] PROBLEM - puppet last run on lvs2005 is CRITICAL: CRITICAL: Puppet has 23 failures [02:40:37] PROBLEM - puppet last run on pollux is CRITICAL: CRITICAL: puppet fail [02:40:37] PROBLEM - puppet last run on hydrogen is CRITICAL: CRITICAL: Puppet has 26 failures [02:40:37] PROBLEM - puppet last run on elastic1009 is CRITICAL: CRITICAL: puppet fail [02:40:37] PROBLEM - puppet last run on ms-be2009 is CRITICAL: CRITICAL: Puppet has 17 failures [02:40:37] PROBLEM - puppet last run on mw1018 is CRITICAL: CRITICAL: Puppet has 70 failures [02:40:37] PROBLEM - puppet last run on mw1058 is CRITICAL: CRITICAL: Puppet has 76 failures [02:40:38] PROBLEM - puppet last run on mw1214 is CRITICAL: CRITICAL: puppet fail [02:40:38] PROBLEM - puppet last run on mw1070 is CRITICAL: CRITICAL: Puppet has 37 failures [02:40:39] PROBLEM - puppet last run on amssq45 is CRITICAL: CRITICAL: Puppet has 13 failures [02:40:39] PROBLEM - puppet last run on amssq52 is CRITICAL: CRITICAL: Puppet has 24 failures [02:40:47] PROBLEM - puppet last run on es1003 is CRITICAL: CRITICAL: Puppet has 16 failures [02:40:47] PROBLEM - puppet last run on wtp1014 is CRITICAL: CRITICAL: Puppet has 10 failures [02:40:47] PROBLEM - puppet last run on lvs4001 is CRITICAL: CRITICAL: Puppet has 17 failures [02:40:57] PROBLEM - puppet last run on erbium is CRITICAL: CRITICAL: puppet fail [02:40:57] PROBLEM - puppet last run on mw1020 is CRITICAL: CRITICAL: puppet fail [02:40:57] PROBLEM - puppet last run on analytics1024 is CRITICAL: CRITICAL: Puppet has 21 failures [02:40:58] PROBLEM - puppet last run on tmh1002 is CRITICAL: CRITICAL: puppet fail [02:40:58] PROBLEM - puppet last run on db1058 is CRITICAL: CRITICAL: Puppet has 11 failures [02:40:58] PROBLEM - puppet last run on ms-be2015 is CRITICAL: CRITICAL: puppet fail [02:41:08] PROBLEM - puppet last run on mw1179 is CRITICAL: CRITICAL: Puppet has 72 failures [02:41:08] PROBLEM - puppet last run on es1005 is CRITICAL: CRITICAL: Puppet has 12 failures [02:41:08] PROBLEM - puppet last run on mw1136 is CRITICAL: CRITICAL: puppet fail [02:41:08] PROBLEM - puppet last run on elastic1016 is CRITICAL: CRITICAL: Puppet has 24 failures [02:41:17] PROBLEM - puppet last run on mw1127 is CRITICAL: CRITICAL: puppet fail [02:41:17] PROBLEM - puppet last run on cp1064 is CRITICAL: CRITICAL: Puppet has 24 failures [02:41:17] PROBLEM - puppet last run on mw1102 is CRITICAL: CRITICAL: Puppet has 78 failures [02:41:18] PROBLEM - puppet last run on lanthanum is CRITICAL: CRITICAL: Puppet has 33 failures [02:41:18] PROBLEM - puppet last run on labsdb1007 is CRITICAL: CRITICAL: puppet fail [02:41:18] PROBLEM - puppet last run on analytics1034 is CRITICAL: CRITICAL: Puppet has 5 failures [02:41:18] PROBLEM - puppet last run on cp4013 is CRITICAL: CRITICAL: puppet fail [02:41:18] PROBLEM - puppet last run on mw1184 is CRITICAL: CRITICAL: puppet fail [02:41:28] PROBLEM - puppet last run on mw1138 is CRITICAL: CRITICAL: puppet fail [02:41:28] PROBLEM - puppet last run on db1041 is CRITICAL: CRITICAL: puppet fail [02:41:28] PROBLEM - puppet last run on praseodymium is CRITICAL: CRITICAL: Puppet has 6 failures [02:41:28] PROBLEM - puppet last run on ms-be1005 is CRITICAL: CRITICAL: Puppet has 24 failures [02:41:28] PROBLEM - puppet last run on db1019 is CRITICAL: CRITICAL: Puppet has 23 failures [02:41:29] PROBLEM - puppet last run on cp1051 is CRITICAL: CRITICAL: puppet fail [02:41:38] PROBLEM - puppet last run on mw1169 is CRITICAL: CRITICAL: Puppet has 73 failures [02:41:38] PROBLEM - puppet last run on cp1065 is CRITICAL: CRITICAL: puppet fail [02:41:47] PROBLEM - puppet last run on ocg1002 is CRITICAL: CRITICAL: Puppet has 27 failures [02:41:47] PROBLEM - puppet last run on stat1001 is CRITICAL: CRITICAL: puppet fail [02:41:47] PROBLEM - puppet last run on lvs1006 is CRITICAL: CRITICAL: Puppet has 10 failures [02:41:47] PROBLEM - puppet last run on db2041 is CRITICAL: CRITICAL: puppet fail [02:41:48] PROBLEM - puppet last run on mw1083 is CRITICAL: CRITICAL: puppet fail [02:41:48] PROBLEM - puppet last run on ms-be2010 is CRITICAL: CRITICAL: puppet fail [02:41:48] PROBLEM - puppet last run on analytics1019 is CRITICAL: CRITICAL: puppet fail [02:41:48] PROBLEM - puppet last run on wtp1021 is CRITICAL: CRITICAL: puppet fail [02:41:58] PROBLEM - puppet last run on amslvs4 is CRITICAL: CRITICAL: puppet fail [02:42:07] PROBLEM - puppet last run on amssq58 is CRITICAL: CRITICAL: puppet fail [02:42:07] PROBLEM - puppet last run on mw1013 is CRITICAL: CRITICAL: Puppet has 23 failures [02:42:07] PROBLEM - puppet last run on mw1232 is CRITICAL: CRITICAL: Puppet has 71 failures [02:42:07] PROBLEM - puppet last run on virt1012 is CRITICAL: CRITICAL: puppet fail [02:42:08] PROBLEM - puppet last run on zirconium is CRITICAL: CRITICAL: puppet fail [02:42:08] PROBLEM - puppet last run on cp1069 is CRITICAL: CRITICAL: Puppet has 13 failures [02:42:08] PROBLEM - puppet last run on mw1035 is CRITICAL: CRITICAL: Puppet has 71 failures [02:42:08] PROBLEM - puppet last run on mw1094 is CRITICAL: CRITICAL: Puppet has 72 failures [02:42:17] PROBLEM - puppet last run on db1024 is CRITICAL: CRITICAL: puppet fail [02:42:17] PROBLEM - puppet last run on mw1182 is CRITICAL: CRITICAL: Puppet has 75 failures [02:42:17] PROBLEM - puppet last run on mw1036 is CRITICAL: CRITICAL: puppet fail [02:42:17] PROBLEM - puppet last run on mw1234 is CRITICAL: CRITICAL: puppet fail [02:42:27] PROBLEM - puppet last run on mc1008 is CRITICAL: CRITICAL: Puppet has 19 failures [02:42:28] PROBLEM - puppet last run on es1009 is CRITICAL: CRITICAL: Puppet has 1 failures [02:42:28] PROBLEM - puppet last run on cp1043 is CRITICAL: CRITICAL: Puppet has 12 failures [02:42:38] PROBLEM - puppet last run on mw1246 is CRITICAL: CRITICAL: Puppet has 72 failures [02:43:07] PROBLEM - puppet last run on cp4017 is CRITICAL: CRITICAL: Puppet has 27 failures [02:43:07] PROBLEM - puppet last run on mw1096 is CRITICAL: CRITICAL: Puppet has 65 failures [02:43:17] PROBLEM - puppet last run on amssq57 is CRITICAL: CRITICAL: Puppet has 4 failures [02:43:18] PROBLEM - puppet last run on mw1196 is CRITICAL: CRITICAL: Puppet has 12 failures [02:43:20] !log restarted puppet on strontium [02:43:27] Logged the message, Master [02:43:27] RECOVERY - puppet last run on db1058 is OK: OK: Puppet is currently enabled, last run 50 seconds ago with 0 failures [02:45:08] PROBLEM - puppet last run on mw1174 is CRITICAL: CRITICAL: puppet fail [02:45:18] PROBLEM - puppet last run on cp1039 is CRITICAL: CRITICAL: puppet fail [02:45:18] PROBLEM - puppet last run on ms-fe2004 is CRITICAL: CRITICAL: puppet fail [02:45:18] PROBLEM - puppet last run on es2002 is CRITICAL: CRITICAL: puppet fail [02:45:18] PROBLEM - puppet last run on db2005 is CRITICAL: CRITICAL: Puppet has 31 failures [02:45:27] PROBLEM - puppet last run on mw1012 is CRITICAL: CRITICAL: Puppet has 100 failures [02:45:27] PROBLEM - puppet last run on mw1063 is CRITICAL: CRITICAL: puppet fail [02:45:27] PROBLEM - puppet last run on ms-be2006 is CRITICAL: CRITICAL: puppet fail [02:45:27] PROBLEM - puppet last run on ms-be2003 is CRITICAL: CRITICAL: Puppet has 4 failures [02:45:28] PROBLEM - puppet last run on amssq61 is CRITICAL: CRITICAL: puppet fail [02:45:28] PROBLEM - puppet last run on mw1187 is CRITICAL: CRITICAL: puppet fail [02:45:28] PROBLEM - puppet last run on xenon is CRITICAL: CRITICAL: puppet fail [02:45:28] PROBLEM - puppet last run on db2034 is CRITICAL: CRITICAL: puppet fail [02:45:37] PROBLEM - puppet last run on amssq53 is CRITICAL: CRITICAL: puppet fail [02:45:37] PROBLEM - puppet last run on cp3008 is CRITICAL: CRITICAL: puppet fail [02:45:37] PROBLEM - puppet last run on amssq32 is CRITICAL: CRITICAL: puppet fail [02:45:38] PROBLEM - puppet last run on mw1067 is CRITICAL: CRITICAL: Puppet has 13 failures [02:45:38] PROBLEM - puppet last run on mw1031 is CRITICAL: CRITICAL: Puppet has 53 failures [02:45:38] PROBLEM - puppet last run on analytics1025 is CRITICAL: CRITICAL: puppet fail [02:45:47] PROBLEM - puppet last run on wtp1020 is CRITICAL: CRITICAL: puppet fail [02:45:47] PROBLEM - puppet last run on mc1006 is CRITICAL: CRITICAL: Puppet has 23 failures [02:45:48] PROBLEM - puppet last run on mw1082 is CRITICAL: CRITICAL: puppet fail [02:45:48] PROBLEM - puppet last run on db2039 is CRITICAL: CRITICAL: puppet fail [02:45:49] PROBLEM - puppet last run on mw1200 is CRITICAL: CRITICAL: Puppet has 76 failures [02:45:57] PROBLEM - puppet last run on mw1041 is CRITICAL: CRITICAL: Puppet has 115 failures [02:45:57] PROBLEM - puppet last run on mw1254 is CRITICAL: CRITICAL: puppet fail [02:45:57] PROBLEM - puppet last run on es1008 is CRITICAL: CRITICAL: Puppet has 21 failures [02:45:57] PROBLEM - puppet last run on helium is CRITICAL: CRITICAL: puppet fail [02:45:57] PROBLEM - puppet last run on snapshot1003 is CRITICAL: CRITICAL: puppet fail [02:45:57] PROBLEM - puppet last run on mw1140 is CRITICAL: CRITICAL: Puppet has 114 failures [02:45:58] PROBLEM - puppet last run on mw1250 is CRITICAL: CRITICAL: puppet fail [02:45:58] PROBLEM - puppet last run on elastic1007 is CRITICAL: CRITICAL: puppet fail [02:45:59] PROBLEM - puppet last run on mw1106 is CRITICAL: CRITICAL: Puppet has 112 failures [02:46:08] PROBLEM - puppet last run on db1031 is CRITICAL: CRITICAL: puppet fail [02:46:08] PROBLEM - puppet last run on cp1055 is CRITICAL: CRITICAL: puppet fail [02:46:17] PROBLEM - puppet last run on mw1006 is CRITICAL: CRITICAL: Puppet has 100 failures [02:46:17] PROBLEM - puppet last run on logstash1001 is CRITICAL: CRITICAL: Puppet has 28 failures [02:46:17] PROBLEM - puppet last run on analytics1020 is CRITICAL: CRITICAL: puppet fail [02:46:18] PROBLEM - puppet last run on mw1007 is CRITICAL: CRITICAL: Puppet has 99 failures [02:46:18] PROBLEM - puppet last run on mw1224 is CRITICAL: CRITICAL: puppet fail [02:46:18] PROBLEM - puppet last run on mw1160 is CRITICAL: CRITICAL: puppet fail [02:46:18] PROBLEM - puppet last run on cp1049 is CRITICAL: CRITICAL: Puppet has 23 failures [02:46:18] PROBLEM - puppet last run on netmon1001 is CRITICAL: CRITICAL: puppet fail [02:46:27] PROBLEM - puppet last run on es2005 is CRITICAL: CRITICAL: Puppet has 21 failures [02:46:27] PROBLEM - puppet last run on amssq43 is CRITICAL: CRITICAL: Puppet has 46 failures [02:46:27] PROBLEM - puppet last run on rbf1002 is CRITICAL: CRITICAL: Puppet has 32 failures [02:46:33] (03PS2) 10Alexandros Kosiaris: Update servermon's service_name [puppet] - 10https://gerrit.wikimedia.org/r/185950 [02:46:35] (03PS1) 10Alexandros Kosiaris: Update servermon settings [puppet] - 10https://gerrit.wikimedia.org/r/185954 [02:46:37] PROBLEM - puppet last run on es2008 is CRITICAL: CRITICAL: Puppet has 19 failures [02:46:38] PROBLEM - puppet last run on amssq54 is CRITICAL: CRITICAL: Puppet has 18 failures [02:46:38] PROBLEM - puppet last run on amssq49 is CRITICAL: CRITICAL: Puppet has 13 failures [02:46:38] PROBLEM - puppet last run on mw1059 is CRITICAL: CRITICAL: Puppet has 75 failures [02:46:47] PROBLEM - puppet last run on amslvs2 is CRITICAL: CRITICAL: Puppet has 17 failures [02:46:48] PROBLEM - puppet last run on amssq59 is CRITICAL: CRITICAL: Puppet has 23 failures [02:46:48] PROBLEM - puppet last run on cp3015 is CRITICAL: CRITICAL: Puppet has 17 failures [02:47:17] PROBLEM - puppet last run on cp4006 is CRITICAL: CRITICAL: Puppet has 10 failures [02:47:17] PROBLEM - puppet last run on cp3020 is CRITICAL: CRITICAL: Puppet has 7 failures [02:47:18] PROBLEM - puppet last run on mw1045 is CRITICAL: CRITICAL: Puppet has 13 failures [02:47:37] PROBLEM - puppet last run on mw1145 is CRITICAL: CRITICAL: Puppet has 23 failures [02:47:38] (03CR) 10Alexandros Kosiaris: [C: 032] Update servermon's service_name [puppet] - 10https://gerrit.wikimedia.org/r/185950 (owner: 10Alexandros Kosiaris) [02:48:08] RECOVERY - puppet last run on db1024 is OK: OK: Puppet is currently enabled, last run 38 seconds ago with 0 failures [02:48:16] (03CR) 10Alexandros Kosiaris: [C: 032] Update servermon settings [puppet] - 10https://gerrit.wikimedia.org/r/185954 (owner: 10Alexandros Kosiaris) [02:52:17] RECOVERY - puppet last run on netmon1001 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [02:53:57] RECOVERY - puppet last run on analytics1003 is OK: OK: Puppet is currently enabled, last run 12 seconds ago with 0 failures [02:53:58] RECOVERY - puppet last run on wtp1010 is OK: OK: Puppet is currently enabled, last run 5 seconds ago with 0 failures [02:54:37] RECOVERY - puppet last run on analytics1018 is OK: OK: Puppet is currently enabled, last run 44 seconds ago with 0 failures [02:54:38] RECOVERY - puppet last run on elastic1020 is OK: OK: Puppet is currently enabled, last run 34 seconds ago with 0 failures [02:54:38] RECOVERY - puppet last run on achernar is OK: OK: Puppet is currently enabled, last run 38 seconds ago with 0 failures [02:54:47] RECOVERY - puppet last run on amssq31 is OK: OK: Puppet is currently enabled, last run 9 seconds ago with 0 failures [02:54:48] RECOVERY - puppet last run on cp1040 is OK: OK: Puppet is currently enabled, last run 42 seconds ago with 0 failures [02:54:57] RECOVERY - puppet last run on ms-fe1002 is OK: OK: Puppet is currently enabled, last run 12 seconds ago with 0 failures [02:54:58] RECOVERY - puppet last run on lvs3003 is OK: OK: Puppet is currently enabled, last run 36 seconds ago with 0 failures [02:54:58] RECOVERY - puppet last run on chromium is OK: OK: Puppet is currently enabled, last run 17 seconds ago with 0 failures [02:54:58] RECOVERY - puppet last run on strontium is OK: OK: Puppet is currently enabled, last run 44 seconds ago with 0 failures [02:54:58] RECOVERY - puppet last run on baham is OK: OK: Puppet is currently enabled, last run 6 seconds ago with 0 failures [02:55:07] RECOVERY - puppet last run on mw1071 is OK: OK: Puppet is currently enabled, last run 39 seconds ago with 0 failures [02:55:07] RECOVERY - puppet last run on amssq39 is OK: OK: Puppet is currently enabled, last run 2 seconds ago with 0 failures [02:55:07] RECOVERY - puppet last run on mw1143 is OK: OK: Puppet is currently enabled, last run 26 seconds ago with 0 failures [02:55:08] RECOVERY - puppet last run on mw1090 is OK: OK: Puppet is currently enabled, last run 44 seconds ago with 0 failures [02:55:17] RECOVERY - puppet last run on mw1064 is OK: OK: Puppet is currently enabled, last run 17 seconds ago with 0 failures [02:55:17] RECOVERY - puppet last run on mw1220 is OK: OK: Puppet is currently enabled, last run 58 seconds ago with 0 failures [02:55:17] RECOVERY - puppet last run on cp1067 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [02:55:18] RECOVERY - puppet last run on cp1068 is OK: OK: Puppet is currently enabled, last run 24 seconds ago with 0 failures [02:55:18] RECOVERY - puppet last run on db1045 is OK: OK: Puppet is currently enabled, last run 41 seconds ago with 0 failures [02:55:18] RECOVERY - puppet last run on einsteinium is OK: OK: Puppet is currently enabled, last run 53 seconds ago with 0 failures [02:55:18] RECOVERY - puppet last run on mw1193 is OK: OK: Puppet is currently enabled, last run 35 seconds ago with 0 failures [02:55:18] RECOVERY - puppet last run on cp1045 is OK: OK: Puppet is currently enabled, last run 15 seconds ago with 0 failures [02:55:19] RECOVERY - puppet last run on mw1107 is OK: OK: Puppet is currently enabled, last run 8 seconds ago with 0 failures [02:55:19] RECOVERY - puppet last run on mw1203 is OK: OK: Puppet is currently enabled, last run 33 seconds ago with 0 failures [02:55:19] RECOVERY - puppet last run on elastic1003 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [02:55:20] RECOVERY - puppet last run on mw1204 is OK: OK: Puppet is currently enabled, last run 21 seconds ago with 0 failures [02:55:21] RECOVERY - puppet last run on db1035 is OK: OK: Puppet is currently enabled, last run 19 seconds ago with 0 failures [02:55:27] RECOVERY - puppet last run on mw1112 is OK: OK: Puppet is currently enabled, last run 48 seconds ago with 0 failures [02:55:28] RECOVERY - puppet last run on mw1027 is OK: OK: Puppet is currently enabled, last run 16 seconds ago with 0 failures [02:55:37] RECOVERY - puppet last run on mw1155 is OK: OK: Puppet is currently enabled, last run 19 seconds ago with 0 failures [02:55:37] RECOVERY - puppet last run on cp4002 is OK: OK: Puppet is currently enabled, last run 1 second ago with 0 failures [02:55:37] RECOVERY - puppet last run on db1037 is OK: OK: Puppet is currently enabled, last run 7 seconds ago with 0 failures [02:55:38] RECOVERY - puppet last run on terbium is OK: OK: Puppet is currently enabled, last run 54 seconds ago with 0 failures [02:55:38] RECOVERY - puppet last run on cp1053 is OK: OK: Puppet is currently enabled, last run 2 seconds ago with 0 failures [02:55:38] RECOVERY - puppet last run on mw1066 is OK: OK: Puppet is currently enabled, last run 52 seconds ago with 0 failures [02:55:47] RECOVERY - puppet last run on ms-be1015 is OK: OK: Puppet is currently enabled, last run 37 seconds ago with 0 failures [02:55:47] RECOVERY - puppet last run on mw1236 is OK: OK: Puppet is currently enabled, last run 57 seconds ago with 0 failures [02:55:47] RECOVERY - puppet last run on radium is OK: OK: Puppet is currently enabled, last run 15 seconds ago with 0 failures [02:55:47] RECOVERY - puppet last run on bast1001 is OK: OK: Puppet is currently enabled, last run 53 seconds ago with 0 failures [02:55:47] RECOVERY - puppet last run on lvs1003 is OK: OK: Puppet is currently enabled, last run 39 seconds ago with 0 failures [02:55:48] RECOVERY - puppet last run on mw1086 is OK: OK: Puppet is currently enabled, last run 34 seconds ago with 0 failures [02:55:57] RECOVERY - puppet last run on elastic1025 is OK: OK: Puppet is currently enabled, last run 16 seconds ago with 0 failures [02:55:57] RECOVERY - puppet last run on mw1241 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [02:55:58] RECOVERY - puppet last run on dysprosium is OK: OK: Puppet is currently enabled, last run 29 seconds ago with 0 failures [02:55:58] RECOVERY - puppet last run on wtp1001 is OK: OK: Puppet is currently enabled, last run 24 seconds ago with 0 failures [02:56:02] (03PS2) 10Yuvipanda: mediawiki: Load prod/beta apache configurabion based on realm [puppet] - 10https://gerrit.wikimedia.org/r/185952 (https://phabricator.wikimedia.org/T87210) [02:56:05] _joe_: ^ [02:56:07] RECOVERY - puppet last run on analytics1012 is OK: OK: Puppet is currently enabled, last run 8 seconds ago with 0 failures [02:56:07] RECOVERY - puppet last run on mc1015 is OK: OK: Puppet is currently enabled, last run 24 seconds ago with 0 failures [02:56:07] RECOVERY - puppet last run on lvs1004 is OK: OK: Puppet is currently enabled, last run 21 seconds ago with 0 failures [02:56:07] RECOVERY - puppet last run on lvs4004 is OK: OK: Puppet is currently enabled, last run 17 seconds ago with 0 failures [02:56:07] RECOVERY - puppet last run on sca1001 is OK: OK: Puppet is currently enabled, last run 47 seconds ago with 0 failures [02:56:08] RECOVERY - puppet last run on cp4009 is OK: OK: Puppet is currently enabled, last run 23 seconds ago with 0 failures [02:56:08] RECOVERY - puppet last run on db1056 is OK: OK: Puppet is currently enabled, last run 38 seconds ago with 0 failures [02:56:08] RECOVERY - puppet last run on cp3018 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [02:56:11] _joe_: that’s not LVS related [02:56:16] _joe_: not sure how else to branch that... [02:56:17] RECOVERY - puppet last run on es2010 is OK: OK: Puppet is currently enabled, last run 59 seconds ago with 0 failures [02:56:17] RECOVERY - puppet last run on cp4012 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [02:56:17] RECOVERY - puppet last run on mw1158 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [02:56:17] RECOVERY - puppet last run on osm-cp1001 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [02:56:17] RECOVERY - puppet last run on mw1253 is OK: OK: Puppet is currently enabled, last run 27 seconds ago with 0 failures [02:56:18] RECOVERY - puppet last run on eeden is OK: OK: Puppet is currently enabled, last run 48 seconds ago with 0 failures [02:56:18] RECOVERY - puppet last run on amssq33 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [02:56:18] RECOVERY - puppet last run on mw1255 is OK: OK: Puppet is currently enabled, last run 28 seconds ago with 0 failures [02:56:18] RECOVERY - puppet last run on cp1054 is OK: OK: Puppet is currently enabled, last run 47 seconds ago with 0 failures [02:56:26] and I don’t want to fix like, a *lot* of apache config [02:56:28] RECOVERY - puppet last run on mw1207 is OK: OK: Puppet is currently enabled, last run 5 seconds ago with 0 failures [02:56:28] RECOVERY - puppet last run on radon is OK: OK: Puppet is currently enabled, last run 42 seconds ago with 0 failures [02:56:28] RECOVERY - puppet last run on mw1104 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [02:56:28] RECOVERY - puppet last run on ms-be1010 is OK: OK: Puppet is currently enabled, last run 41 seconds ago with 0 failures [02:56:29] * YuviPanda hates apache config language [02:56:37] RECOVERY - puppet last run on mw1021 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [02:56:37] RECOVERY - puppet last run on labsdb1005 is OK: OK: Puppet is currently enabled, last run 16 seconds ago with 0 failures [02:56:37] RECOVERY - puppet last run on cp1066 is OK: OK: Puppet is currently enabled, last run 9 seconds ago with 0 failures [02:56:38] RECOVERY - puppet last run on elastic1029 is OK: OK: Puppet is currently enabled, last run 20 seconds ago with 0 failures [02:56:38] RECOVERY - puppet last run on wtp1008 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [02:56:38] RECOVERY - puppet last run on mw1037 is OK: OK: Puppet is currently enabled, last run 29 seconds ago with 0 failures [02:56:47] RECOVERY - puppet last run on bast2001 is OK: OK: Puppet is currently enabled, last run 14 seconds ago with 0 failures [02:56:48] RECOVERY - puppet last run on mw1194 is OK: OK: Puppet is currently enabled, last run 14 seconds ago with 0 failures [02:56:48] RECOVERY - puppet last run on mw1154 is OK: OK: Puppet is currently enabled, last run 45 seconds ago with 0 failures [02:56:48] RECOVERY - puppet last run on cp3005 is OK: OK: Puppet is currently enabled, last run 59 seconds ago with 0 failures [02:56:48] RECOVERY - puppet last run on labsdb1002 is OK: OK: Puppet is currently enabled, last run 4 seconds ago with 0 failures [02:56:52] 3operations: mysql boxes not in ganglia - https://phabricator.wikimedia.org/T87209#986315 (10Springle) 5Open>3Resolved a:3Springle [02:56:57] RECOVERY - puppet last run on analytics1031 is OK: OK: Puppet is currently enabled, last run 20 seconds ago with 0 failures [02:56:57] RECOVERY - puppet last run on mw1131 is OK: OK: Puppet is currently enabled, last run 56 seconds ago with 0 failures [02:56:57] RECOVERY - puppet last run on mw1135 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [02:56:57] RECOVERY - puppet last run on mw1137 is OK: OK: Puppet is currently enabled, last run 3 seconds ago with 0 failures [02:57:07] RECOVERY - puppet last run on db1027 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [02:57:07] RECOVERY - puppet last run on tmh1001 is OK: OK: Puppet is currently enabled, last run 26 seconds ago with 0 failures [02:57:07] RECOVERY - puppet last run on mw1073 is OK: OK: Puppet is currently enabled, last run 31 seconds ago with 0 failures [02:57:07] RECOVERY - puppet last run on es1004 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [02:57:08] RECOVERY - puppet last run on mw1110 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [02:57:08] RECOVERY - puppet last run on lvs2002 is OK: OK: Puppet is currently enabled, last run 34 seconds ago with 0 failures [02:57:17] RECOVERY - puppet last run on es2006 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [02:57:17] RECOVERY - puppet last run on db2017 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [02:57:17] RECOVERY - puppet last run on db2010 is OK: OK: Puppet is currently enabled, last run 34 seconds ago with 0 failures [02:57:17] RECOVERY - puppet last run on analytics1021 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [02:57:17] RECOVERY - puppet last run on mw1128 is OK: OK: Puppet is currently enabled, last run 12 seconds ago with 0 failures [02:57:18] RECOVERY - puppet last run on virt1005 is OK: OK: Puppet is currently enabled, last run 40 seconds ago with 0 failures [02:57:27] RECOVERY - puppet last run on amssq37 is OK: OK: Puppet is currently enabled, last run 59 seconds ago with 0 failures [02:57:27] RECOVERY - puppet last run on palladium is OK: OK: Puppet is currently enabled, last run 17 seconds ago with 0 failures [02:57:28] RECOVERY - puppet last run on mw1113 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [02:57:28] RECOVERY - puppet last run on cp4011 is OK: OK: Puppet is currently enabled, last run 13 seconds ago with 0 failures [02:57:28] RECOVERY - puppet last run on cp3019 is OK: OK: Puppet is currently enabled, last run 6 seconds ago with 0 failures [02:57:28] RECOVERY - puppet last run on cp3011 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [02:57:37] RECOVERY - puppet last run on elastic1013 is OK: OK: Puppet is currently enabled, last run 44 seconds ago with 0 failures [02:57:37] RECOVERY - puppet last run on db1068 is OK: OK: Puppet is currently enabled, last run 19 seconds ago with 0 failures [02:57:38] RECOVERY - puppet last run on mw1103 is OK: OK: Puppet is currently enabled, last run 49 seconds ago with 0 failures [02:57:38] RECOVERY - puppet last run on mw1095 is OK: OK: Puppet is currently enabled, last run 3 seconds ago with 0 failures [02:57:47] RECOVERY - puppet last run on cp4010 is OK: OK: Puppet is currently enabled, last run 10 seconds ago with 0 failures [02:57:48] RECOVERY - puppet last run on mw1017 is OK: OK: Puppet is currently enabled, last run 21 seconds ago with 0 failures [02:57:48] RECOVERY - puppet last run on cp1064 is OK: OK: Puppet is currently enabled, last run 47 seconds ago with 0 failures [02:57:48] RECOVERY - puppet last run on mw1047 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [02:57:57] RECOVERY - puppet last run on mw1101 is OK: OK: Puppet is currently enabled, last run 26 seconds ago with 0 failures [02:57:57] RECOVERY - puppet last run on mw1102 is OK: OK: Puppet is currently enabled, last run 8 seconds ago with 0 failures [02:57:57] RECOVERY - puppet last run on mw1075 is OK: OK: Puppet is currently enabled, last run 4 seconds ago with 0 failures [02:57:57] RECOVERY - puppet last run on lanthanum is OK: OK: Puppet is currently enabled, last run 30 seconds ago with 0 failures [02:57:57] RECOVERY - puppet last run on mw1199 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [02:57:57] RECOVERY - puppet last run on db1005 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [02:57:58] RECOVERY - puppet last run on elastic1010 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [02:57:58] RECOVERY - puppet last run on ms-be3004 is OK: OK: Puppet is currently enabled, last run 44 seconds ago with 0 failures [02:57:58] RECOVERY - puppet last run on amssq50 is OK: OK: Puppet is currently enabled, last run 58 seconds ago with 0 failures [02:57:59] RECOVERY - puppet last run on cp3017 is OK: OK: Puppet is currently enabled, last run 38 seconds ago with 0 failures [02:57:59] RECOVERY - puppet last run on ms-be1014 is OK: OK: Puppet is currently enabled, last run 52 seconds ago with 0 failures [02:58:08] RECOVERY - puppet last run on ms-be1005 is OK: OK: Puppet is currently enabled, last run 21 seconds ago with 0 failures [02:58:08] RECOVERY - puppet last run on cp1057 is OK: OK: Puppet is currently enabled, last run 2 seconds ago with 0 failures [02:58:08] RECOVERY - puppet last run on db1019 is OK: OK: Puppet is currently enabled, last run 17 seconds ago with 0 failures [02:58:08] RECOVERY - puppet last run on mw1230 is OK: OK: Puppet is currently enabled, last run 55 seconds ago with 0 failures [02:58:08] RECOVERY - puppet last run on db1009 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [02:58:09] RECOVERY - puppet last run on mw1015 is OK: OK: Puppet is currently enabled, last run 19 seconds ago with 0 failures [02:58:17] RECOVERY - puppet last run on analytics1029 is OK: OK: Puppet is currently enabled, last run 13 seconds ago with 0 failures [02:58:18] RECOVERY - puppet last run on db1049 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [02:58:27] RECOVERY - puppet last run on ocg1002 is OK: OK: Puppet is currently enabled, last run 37 seconds ago with 0 failures [02:58:27] RECOVERY - puppet last run on hydrogen is OK: OK: Puppet is currently enabled, last run 48 seconds ago with 0 failures [02:58:28] RECOVERY - puppet last run on lvs2005 is OK: OK: Puppet is currently enabled, last run 59 seconds ago with 0 failures [02:58:28] RECOVERY - puppet last run on elastic1009 is OK: OK: Puppet is currently enabled, last run 57 seconds ago with 0 failures [02:58:28] RECOVERY - puppet last run on virt1009 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [02:58:28] RECOVERY - puppet last run on pollux is OK: OK: Puppet is currently enabled, last run 30 seconds ago with 0 failures [02:58:28] RECOVERY - puppet last run on mw1018 is OK: OK: Puppet is currently enabled, last run 59 seconds ago with 0 failures [02:58:29] RECOVERY - puppet last run on ms-be2009 is OK: OK: Puppet is currently enabled, last run 39 seconds ago with 0 failures [02:58:29] RECOVERY - puppet last run on mw1058 is OK: OK: Puppet is currently enabled, last run 59 seconds ago with 0 failures [02:58:29] RECOVERY - puppet last run on mw1083 is OK: OK: Puppet is currently enabled, last run 7 seconds ago with 0 failures [02:58:30] RECOVERY - puppet last run on mw1070 is OK: OK: Puppet is currently enabled, last run 23 seconds ago with 0 failures [02:58:30] RECOVERY - puppet last run on mw1078 is OK: OK: Puppet is currently enabled, last run 18 seconds ago with 0 failures [02:58:37] RECOVERY - puppet last run on es1003 is OK: OK: Puppet is currently enabled, last run 50 seconds ago with 0 failures [02:58:37] RECOVERY - puppet last run on mw1085 is OK: OK: Puppet is currently enabled, last run 48 seconds ago with 0 failures [02:58:38] RECOVERY - puppet last run on cp3007 is OK: OK: Puppet is currently enabled, last run 18 seconds ago with 0 failures [02:58:38] RECOVERY - puppet last run on amssq45 is OK: OK: Puppet is currently enabled, last run 10 seconds ago with 0 failures [02:58:38] RECOVERY - puppet last run on wtp1014 is OK: OK: Puppet is currently enabled, last run 34 seconds ago with 0 failures [02:58:38] RECOVERY - puppet last run on lvs4001 is OK: OK: Puppet is currently enabled, last run 35 seconds ago with 0 failures [02:58:38] RECOVERY - puppet last run on cp4016 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [02:58:47] RECOVERY - puppet last run on cp3021 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [02:58:47] RECOVERY - puppet last run on erbium is OK: OK: Puppet is currently enabled, last run 13 seconds ago with 0 failures [02:58:47] RECOVERY - puppet last run on analytics1024 is OK: OK: Puppet is currently enabled, last run 33 seconds ago with 0 failures [02:58:48] RECOVERY - puppet last run on mw1020 is OK: OK: Puppet is currently enabled, last run 32 seconds ago with 0 failures [02:58:48] RECOVERY - puppet last run on tmh1002 is OK: OK: Puppet is currently enabled, last run 0 seconds ago with 0 failures [02:58:57] RECOVERY - puppet last run on mw1182 is OK: OK: Puppet is currently enabled, last run 44 seconds ago with 0 failures [02:58:58] RECOVERY - puppet last run on mw1179 is OK: OK: Puppet is currently enabled, last run 54 seconds ago with 0 failures [02:58:58] RECOVERY - puppet last run on es1005 is OK: OK: Puppet is currently enabled, last run 38 seconds ago with 0 failures [02:58:58] RECOVERY - puppet last run on ms-be2015 is OK: OK: Puppet is currently enabled, last run 10 seconds ago with 0 failures [02:58:58] RECOVERY - puppet last run on lvs3002 is OK: OK: Puppet is currently enabled, last run 56 seconds ago with 0 failures [02:59:07] RECOVERY - puppet last run on mw1136 is OK: OK: Puppet is currently enabled, last run 39 seconds ago with 0 failures [02:59:07] RECOVERY - puppet last run on mw1157 is OK: OK: Puppet is currently enabled, last run 59 seconds ago with 0 failures [02:59:07] RECOVERY - puppet last run on elastic1016 is OK: OK: Puppet is currently enabled, last run 37 seconds ago with 0 failures [02:59:07] RECOVERY - puppet last run on mw1127 is OK: OK: Puppet is currently enabled, last run 27 seconds ago with 0 failures [02:59:18] RECOVERY - puppet last run on labsdb1007 is OK: OK: Puppet is currently enabled, last run 57 seconds ago with 0 failures [02:59:18] RECOVERY - puppet last run on analytics1034 is OK: OK: Puppet is currently enabled, last run 8 seconds ago with 0 failures [02:59:18] RECOVERY - puppet last run on mw1184 is OK: OK: Puppet is currently enabled, last run 43 seconds ago with 0 failures [02:59:18] RECOVERY - puppet last run on mw1246 is OK: OK: Puppet is currently enabled, last run 7 seconds ago with 0 failures [02:59:18] RECOVERY - puppet last run on cp4015 is OK: OK: Puppet is currently enabled, last run 25 seconds ago with 0 failures [02:59:18] RECOVERY - puppet last run on cp4013 is OK: OK: Puppet is currently enabled, last run 8 seconds ago with 0 failures [02:59:19] RECOVERY - puppet last run on db1041 is OK: OK: Puppet is currently enabled, last run 21 seconds ago with 0 failures [02:59:19] RECOVERY - puppet last run on mw1019 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [02:59:28] RECOVERY - puppet last run on praseodymium is OK: OK: Puppet is currently enabled, last run 42 seconds ago with 0 failures [02:59:28] RECOVERY - puppet last run on mw1191 is OK: OK: Puppet is currently enabled, last run 36 seconds ago with 0 failures [02:59:38] RECOVERY - puppet last run on mw1169 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [02:59:47] RECOVERY - puppet last run on stat1001 is OK: OK: Puppet is currently enabled, last run 42 seconds ago with 0 failures [02:59:47] RECOVERY - puppet last run on lvs1006 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [02:59:47] RECOVERY - puppet last run on db2041 is OK: OK: Puppet is currently enabled, last run 57 seconds ago with 0 failures [02:59:48] RECOVERY - puppet last run on mw1214 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [02:59:48] RECOVERY - puppet last run on analytics1019 is OK: OK: Puppet is currently enabled, last run 19 seconds ago with 0 failures [02:59:48] RECOVERY - puppet last run on ms-be2010 is OK: OK: Puppet is currently enabled, last run 36 seconds ago with 0 failures [02:59:57] RECOVERY - puppet last run on mw1096 is OK: OK: Puppet is currently enabled, last run 31 seconds ago with 0 failures [02:59:57] RECOVERY - puppet last run on cp4017 is OK: OK: Puppet is currently enabled, last run 7 seconds ago with 0 failures [02:59:57] RECOVERY - puppet last run on amssq52 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [02:59:58] RECOVERY - puppet last run on mw1232 is OK: OK: Puppet is currently enabled, last run 39 seconds ago with 0 failures [03:00:07] RECOVERY - puppet last run on amslvs4 is OK: OK: Puppet is currently enabled, last run 33 seconds ago with 0 failures [03:00:07] RECOVERY - puppet last run on virt1012 is OK: OK: Puppet is currently enabled, last run 8 seconds ago with 0 failures [03:00:08] RECOVERY - puppet last run on zirconium is OK: OK: Puppet is currently enabled, last run 44 seconds ago with 0 failures [03:00:08] RECOVERY - puppet last run on cp1069 is OK: OK: Puppet is currently enabled, last run 51 seconds ago with 0 failures [03:00:08] RECOVERY - puppet last run on mw1196 is OK: OK: Puppet is currently enabled, last run 59 seconds ago with 0 failures [03:00:08] RECOVERY - puppet last run on mw1094 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [03:00:08] RECOVERY - puppet last run on mw1035 is OK: OK: Puppet is currently enabled, last run 12 seconds ago with 0 failures [03:00:17] RECOVERY - puppet last run on mw1234 is OK: OK: Puppet is currently enabled, last run 23 seconds ago with 0 failures [03:00:17] RECOVERY - puppet last run on mw1036 is OK: OK: Puppet is currently enabled, last run 56 seconds ago with 0 failures [03:00:27] RECOVERY - puppet last run on es1009 is OK: OK: Puppet is currently enabled, last run 0 seconds ago with 0 failures [03:00:27] RECOVERY - puppet last run on mc1008 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [03:00:27] RECOVERY - puppet last run on cp1043 is OK: OK: Puppet is currently enabled, last run 45 seconds ago with 0 failures [03:00:28] RECOVERY - puppet last run on mw1138 is OK: OK: Puppet is currently enabled, last run 38 seconds ago with 0 failures [03:00:39] RECOVERY - puppet last run on cp1051 is OK: OK: Puppet is currently enabled, last run 52 seconds ago with 0 failures [03:00:48] RECOVERY - puppet last run on cp1065 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [03:00:58] RECOVERY - puppet last run on wtp1021 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [03:01:08] RECOVERY - puppet last run on mw1013 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [03:01:17] RECOVERY - puppet last run on amssq57 is OK: OK: Puppet is currently enabled, last run 59 seconds ago with 0 failures [03:01:17] RECOVERY - puppet last run on amssq58 is OK: OK: Puppet is currently enabled, last run 36 seconds ago with 0 failures [03:02:28] RECOVERY - puppet last run on amssq59 is OK: OK: Puppet is currently enabled, last run 5 seconds ago with 0 failures [03:02:28] RECOVERY - puppet last run on mw1067 is OK: OK: Puppet is currently enabled, last run 50 seconds ago with 0 failures [03:02:28] RECOVERY - puppet last run on cp3015 is OK: OK: Puppet is currently enabled, last run 3 seconds ago with 0 failures [03:02:28] RECOVERY - puppet last run on mw1031 is OK: OK: Puppet is currently enabled, last run 12 seconds ago with 0 failures [03:02:38] RECOVERY - puppet last run on mw1200 is OK: OK: Puppet is currently enabled, last run 3 seconds ago with 0 failures [03:02:47] RECOVERY - puppet last run on mw1140 is OK: OK: Puppet is currently enabled, last run 9 seconds ago with 0 failures [03:02:58] RECOVERY - puppet last run on mw1045 is OK: OK: Puppet is currently enabled, last run 12 seconds ago with 0 failures [03:02:58] RECOVERY - puppet last run on mw1006 is OK: OK: Puppet is currently enabled, last run 27 seconds ago with 0 failures [03:02:58] RECOVERY - puppet last run on logstash1001 is OK: OK: Puppet is currently enabled, last run 21 seconds ago with 0 failures [03:03:07] RECOVERY - puppet last run on mw1007 is OK: OK: Puppet is currently enabled, last run 40 seconds ago with 0 failures [03:03:07] RECOVERY - puppet last run on cp1049 is OK: OK: Puppet is currently enabled, last run 18 seconds ago with 0 failures [03:03:08] RECOVERY - puppet last run on rbf1002 is OK: OK: Puppet is currently enabled, last run 10 seconds ago with 0 failures [03:03:08] RECOVERY - puppet last run on mw1145 is OK: OK: Puppet is currently enabled, last run 4 seconds ago with 0 failures [03:03:17] RECOVERY - puppet last run on es2005 is OK: OK: Puppet is currently enabled, last run 21 seconds ago with 0 failures [03:03:17] RECOVERY - puppet last run on amssq43 is OK: OK: Puppet is currently enabled, last run 36 seconds ago with 0 failures [03:03:18] RECOVERY - puppet last run on mw1012 is OK: OK: Puppet is currently enabled, last run 10 seconds ago with 0 failures [03:03:18] RECOVERY - puppet last run on mw1063 is OK: OK: Puppet is currently enabled, last run 17 seconds ago with 0 failures [03:03:18] RECOVERY - puppet last run on es2008 is OK: OK: Puppet is currently enabled, last run 20 seconds ago with 0 failures [03:03:27] RECOVERY - puppet last run on ms-be2003 is OK: OK: Puppet is currently enabled, last run 49 seconds ago with 0 failures [03:03:27] RECOVERY - puppet last run on mw1059 is OK: OK: Puppet is currently enabled, last run 41 seconds ago with 0 failures [03:03:28] RECOVERY - puppet last run on xenon is OK: OK: Puppet is currently enabled, last run 1 second ago with 0 failures [03:03:37] RECOVERY - puppet last run on amslvs2 is OK: OK: Puppet is currently enabled, last run 41 seconds ago with 0 failures [03:03:38] RECOVERY - puppet last run on analytics1025 is OK: OK: Puppet is currently enabled, last run 0 seconds ago with 0 failures [03:03:38] RECOVERY - puppet last run on wtp1020 is OK: OK: Puppet is currently enabled, last run 39 seconds ago with 0 failures [03:03:48] RECOVERY - puppet last run on mc1006 is OK: OK: Puppet is currently enabled, last run 57 seconds ago with 0 failures [03:03:48] RECOVERY - puppet last run on mw1082 is OK: OK: Puppet is currently enabled, last run 27 seconds ago with 0 failures [03:03:49] RECOVERY - puppet last run on db2039 is OK: OK: Puppet is currently enabled, last run 17 seconds ago with 0 failures [03:03:49] RECOVERY - puppet last run on es1008 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [03:03:49] RECOVERY - puppet last run on mw1254 is OK: OK: Puppet is currently enabled, last run 13 seconds ago with 0 failures [03:03:49] RECOVERY - puppet last run on mw1041 is OK: OK: Puppet is currently enabled, last run 34 seconds ago with 0 failures [03:03:57] RECOVERY - puppet last run on helium is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [03:03:57] RECOVERY - puppet last run on snapshot1003 is OK: OK: Puppet is currently enabled, last run 14 seconds ago with 0 failures [03:03:57] RECOVERY - puppet last run on mw1250 is OK: OK: Puppet is currently enabled, last run 53 seconds ago with 0 failures [03:03:57] RECOVERY - puppet last run on elastic1007 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [03:03:58] RECOVERY - puppet last run on mw1106 is OK: OK: Puppet is currently enabled, last run 54 seconds ago with 0 failures [03:04:07] RECOVERY - puppet last run on cp4006 is OK: OK: Puppet is currently enabled, last run 16 seconds ago with 0 failures [03:04:07] RECOVERY - puppet last run on cp3020 is OK: OK: Puppet is currently enabled, last run 35 seconds ago with 0 failures [03:04:07] RECOVERY - puppet last run on db1031 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [03:04:08] RECOVERY - puppet last run on cp1055 is OK: OK: Puppet is currently enabled, last run 49 seconds ago with 0 failures [03:04:17] RECOVERY - puppet last run on analytics1020 is OK: OK: Puppet is currently enabled, last run 59 seconds ago with 0 failures [03:04:18] RECOVERY - puppet last run on mw1224 is OK: OK: Puppet is currently enabled, last run 22 seconds ago with 0 failures [03:04:18] RECOVERY - puppet last run on mw1160 is OK: OK: Puppet is currently enabled, last run 18 seconds ago with 0 failures [03:04:27] RECOVERY - puppet last run on mw1174 is OK: OK: Puppet is currently enabled, last run 13 seconds ago with 0 failures [03:04:28] RECOVERY - puppet last run on cp1039 is OK: OK: Puppet is currently enabled, last run 58 seconds ago with 0 failures [03:04:28] RECOVERY - puppet last run on ms-fe2004 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [03:04:37] RECOVERY - puppet last run on db2005 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [03:04:37] RECOVERY - puppet last run on es2002 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [03:04:37] RECOVERY - puppet last run on ms-be2006 is OK: OK: Puppet is currently enabled, last run 17 seconds ago with 0 failures [03:04:38] RECOVERY - puppet last run on mw1187 is OK: OK: Puppet is currently enabled, last run 42 seconds ago with 0 failures [03:04:38] RECOVERY - puppet last run on amssq54 is OK: OK: Puppet is currently enabled, last run 39 seconds ago with 0 failures [03:04:38] RECOVERY - puppet last run on amssq49 is OK: OK: Puppet is currently enabled, last run 53 seconds ago with 0 failures [03:04:38] RECOVERY - puppet last run on amssq61 is OK: OK: Puppet is currently enabled, last run 39 seconds ago with 0 failures [03:04:47] RECOVERY - puppet last run on db2034 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [03:04:48] RECOVERY - puppet last run on amssq32 is OK: OK: Puppet is currently enabled, last run 41 seconds ago with 0 failures [03:04:48] RECOVERY - puppet last run on amssq53 is OK: OK: Puppet is currently enabled, last run 36 seconds ago with 0 failures [03:04:48] RECOVERY - puppet last run on cp3008 is OK: OK: Puppet is currently enabled, last run 58 seconds ago with 0 failures [04:00:59] !log LocalisationUpdate ResourceLoader cache refresh completed at Tue Jan 20 04:00:59 UTC 2015 (duration 0m 58s) [04:01:07] Logged the message, Master [06:28:48] PROBLEM - puppet last run on db1059 is CRITICAL: CRITICAL: Puppet has 1 failures [06:29:08] PROBLEM - puppet last run on cp1061 is CRITICAL: CRITICAL: Puppet has 1 failures [06:29:17] PROBLEM - puppet last run on analytics1030 is CRITICAL: CRITICAL: Puppet has 1 failures [06:29:18] PROBLEM - puppet last run on db1040 is CRITICAL: CRITICAL: Puppet has 1 failures [06:29:28] PROBLEM - puppet last run on mw1166 is CRITICAL: CRITICAL: Puppet has 1 failures [06:29:37] PROBLEM - puppet last run on mw1144 is CRITICAL: CRITICAL: Puppet has 1 failures [06:29:38] PROBLEM - puppet last run on ms-fe2003 is CRITICAL: CRITICAL: Puppet has 1 failures [06:29:38] PROBLEM - puppet last run on db2036 is CRITICAL: CRITICAL: Puppet has 2 failures [06:30:07] PROBLEM - puppet last run on mw1061 is CRITICAL: CRITICAL: Puppet has 1 failures [06:30:08] PROBLEM - puppet last run on amssq35 is CRITICAL: CRITICAL: Puppet has 1 failures [06:45:27] RECOVERY - puppet last run on db1059 is OK: OK: Puppet is currently enabled, last run 15 seconds ago with 0 failures [06:45:38] RECOVERY - puppet last run on amssq35 is OK: OK: Puppet is currently enabled, last run 7 seconds ago with 0 failures [06:45:48] RECOVERY - puppet last run on cp1061 is OK: OK: Puppet is currently enabled, last run 25 seconds ago with 0 failures [06:45:58] RECOVERY - puppet last run on analytics1030 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [06:45:58] RECOVERY - puppet last run on db1040 is OK: OK: Puppet is currently enabled, last run 26 seconds ago with 0 failures [06:46:08] RECOVERY - puppet last run on mw1166 is OK: OK: Puppet is currently enabled, last run 56 seconds ago with 0 failures [06:46:18] RECOVERY - puppet last run on mw1144 is OK: OK: Puppet is currently enabled, last run 25 seconds ago with 0 failures [06:46:18] RECOVERY - puppet last run on db2036 is OK: OK: Puppet is currently enabled, last run 13 seconds ago with 0 failures [06:46:47] RECOVERY - puppet last run on mw1061 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [06:47:37] RECOVERY - puppet last run on ms-fe2003 is OK: OK: Puppet is currently enabled, last run 33 seconds ago with 0 failures [07:58:20] (03PS1) 10Yuvipanda: mediawiki: Introduce feature flag to enable/disable lvs [puppet] - 10https://gerrit.wikimedia.org/r/185965 (https://phabricator.wikimedia.org/T87210) [07:59:35] (03PS1) 10Yuvipanda: beta: Remove mediawiki appserver role [puppet] - 10https://gerrit.wikimedia.org/r/185966 (https://phabricator.wikimedia.org/T87210) [07:59:58] (03CR) 10Yuvipanda: [C: 04-2] "Must re-image / migrate all the current mediawiki servers before merging this." [puppet] - 10https://gerrit.wikimedia.org/r/185966 (https://phabricator.wikimedia.org/T87210) (owner: 10Yuvipanda) [08:07:19] Krinkle|detached: uh, cvn-dev seems down? [08:10:51] (03PS1) 10Yuvipanda: contint: Don't include base firewall by default [puppet] - 10https://gerrit.wikimedia.org/r/185967 [08:11:18] (03PS2) 10Yuvipanda: contint: Don't include base firewall by default [puppet] - 10https://gerrit.wikimedia.org/r/185967 [08:11:36] (03CR) 10Yuvipanda: [C: 032] contint: Don't include base firewall by default [puppet] - 10https://gerrit.wikimedia.org/r/185967 (owner: 10Yuvipanda) [08:19:24] 3Beta-Cluster, operations: Minimize differences between beta and production - https://phabricator.wikimedia.org/T87220#986422 (10yuvipanda) 3NEW [08:22:06] 3Beta-Cluster: Remove beta specific mediawiki roles - https://phabricator.wikimedia.org/T87210#986429 (10yuvipanda) a:3yuvipanda [08:22:19] 3Beta-Cluster: Remove beta specific mediawiki roles - https://phabricator.wikimedia.org/T87210#985990 (10yuvipanda) [08:22:21] 3Beta-Cluster, operations: Unify ::production / ::beta roles for *oid - https://phabricator.wikimedia.org/T86633#986433 (10yuvipanda) [08:22:23] 3Beta-Cluster, operations: Minimize differences between beta and production - https://phabricator.wikimedia.org/T87220#986430 (10yuvipanda) [08:23:27] 3Beta-Cluster, operations: Set 'cluster' salt grain appropriately for all instances in beta cluster - https://phabricator.wikimedia.org/T87199#986436 (10yuvipanda) [08:23:29] 3Beta-Cluster, operations: Minimize differences between beta and production - https://phabricator.wikimedia.org/T87220#986422 (10yuvipanda) [08:24:48] PROBLEM - Unmerged changes on repository puppet on palladium is CRITICAL: There is one unmerged change in puppet (dir /var/lib/git/operations/puppet). [08:24:48] PROBLEM - Unmerged changes on repository puppet on strontium is CRITICAL: There is one unmerged change in puppet (dir /var/lib/git/operations/puppet). [08:25:13] 3Beta-Cluster, operations: Unify ::production / ::beta roles for *oid - https://phabricator.wikimedia.org/T86633#986438 (10yuvipanda) [08:25:15] 3Beta-Cluster: Remove all ::beta roles in puppet - https://phabricator.wikimedia.org/T86644#986440 (10yuvipanda) [08:25:18] 3Beta-Cluster, operations: Minimize differences between beta and production - https://phabricator.wikimedia.org/T87220#986437 (10yuvipanda) [08:27:17] RECOVERY - Unmerged changes on repository puppet on palladium is OK: No changes to merge. [08:27:17] RECOVERY - Unmerged changes on repository puppet on strontium is OK: No changes to merge. [08:28:03] 3Beta-Cluster, operations: Move scap puppet code into a module - https://phabricator.wikimedia.org/T87221#986442 (10yuvipanda) 3NEW [08:28:19] 3Beta-Cluster, operations: Move scap puppet code into a module - https://phabricator.wikimedia.org/T87221#986457 (10yuvipanda) [08:28:20] 3Beta-Cluster, operations: Minimize differences between beta and production - https://phabricator.wikimedia.org/T87220#986456 (10yuvipanda) [08:29:45] 3Beta-Cluster, operations: Move deployment-prep hiera data values into ops/puppet.git repo - https://phabricator.wikimedia.org/T87223#986458 (10yuvipanda) 3NEW [08:30:27] 3Beta-Cluster, operations: Move deployment-prep hiera data values into ops/puppet.git repo - https://phabricator.wikimedia.org/T87223#986458 (10yuvipanda) [08:30:28] 3Beta-Cluster, operations: Minimize differences between beta and production - https://phabricator.wikimedia.org/T87220#986464 (10yuvipanda) [08:36:05] 3Beta-Cluster, operations: Minimize differences between beta and production (Tracking) - https://phabricator.wikimedia.org/T87220#986475 (10yuvipanda) [08:36:31] 3Beta-Cluster, operations: Minimize differences between beta and production (Tracking) - https://phabricator.wikimedia.org/T87220#986422 (10yuvipanda) [08:50:27] PROBLEM - Slow CirrusSearch query rate on fluorine is CRITICAL: CirrusSearch-slow.log_line_rate CRITICAL: 0.00333333333333 [08:55:37] RECOVERY - Slow CirrusSearch query rate on fluorine is OK: CirrusSearch-slow.log_line_rate OKAY: 0.0 [09:16:48] PROBLEM - puppetmaster backend https on strontium is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 8141: HTTP/1.1 500 Internal Server Error [09:16:48] PROBLEM - puppet last run on db1072 is CRITICAL: CRITICAL: Puppet has 2 failures [09:16:57] PROBLEM - puppet last run on analytics1003 is CRITICAL: CRITICAL: Puppet has 14 failures [09:16:57] PROBLEM - puppet last run on mw1253 is CRITICAL: CRITICAL: puppet fail [09:16:58] PROBLEM - puppet last run on wtp1010 is CRITICAL: CRITICAL: Puppet has 13 failures [09:16:58] PROBLEM - puppet last run on cp4020 is CRITICAL: CRITICAL: Puppet has 3 failures [09:17:17] PROBLEM - puppet last run on mw1071 is CRITICAL: CRITICAL: Puppet has 38 failures [09:17:17] PROBLEM - puppet last run on mw1139 is CRITICAL: CRITICAL: Puppet has 5 failures [09:17:17] PROBLEM - puppet last run on cp1045 is CRITICAL: CRITICAL: puppet fail [09:17:17] PROBLEM - puppet last run on einsteinium is CRITICAL: CRITICAL: Puppet has 12 failures [09:17:18] PROBLEM - puppet last run on mw1090 is CRITICAL: CRITICAL: Puppet has 37 failures [09:17:27] PROBLEM - puppet last run on wtp1008 is CRITICAL: CRITICAL: Puppet has 10 failures [09:17:27] PROBLEM - puppet last run on mw1064 is CRITICAL: CRITICAL: Puppet has 35 failures [09:17:27] PROBLEM - puppet last run on mw1155 is CRITICAL: CRITICAL: puppet fail [09:17:27] PROBLEM - puppet last run on mw1204 is CRITICAL: CRITICAL: Puppet has 64 failures [09:17:28] PROBLEM - puppet last run on terbium is CRITICAL: CRITICAL: Puppet has 67 failures [09:17:28] PROBLEM - puppet last run on mw1104 is CRITICAL: CRITICAL: puppet fail [09:17:28] PROBLEM - puppet last run on mw1107 is CRITICAL: CRITICAL: Puppet has 33 failures [09:17:28] PROBLEM - puppet last run on cp1053 is CRITICAL: CRITICAL: puppet fail [09:17:28] PROBLEM - puppet last run on elastic1003 is CRITICAL: CRITICAL: Puppet has 24 failures [09:17:29] PROBLEM - puppet last run on mw1112 is CRITICAL: CRITICAL: Puppet has 33 failures [09:17:29] PROBLEM - puppet last run on ms-be1015 is CRITICAL: CRITICAL: puppet fail [09:17:30] PROBLEM - puppet last run on mw1027 is CRITICAL: CRITICAL: Puppet has 66 failures [09:17:37] PROBLEM - puppet last run on mw1037 is CRITICAL: CRITICAL: puppet fail [09:17:38] PROBLEM - puppet last run on mw1236 is CRITICAL: CRITICAL: Puppet has 11 failures [09:17:38] PROBLEM - puppet last run on lvs1003 is CRITICAL: CRITICAL: puppet fail [09:17:38] PROBLEM - puppet last run on db1037 is CRITICAL: CRITICAL: puppet fail [09:17:38] PROBLEM - puppet last run on achernar is CRITICAL: CRITICAL: Puppet has 16 failures [09:17:38] PROBLEM - puppet last run on baham is CRITICAL: CRITICAL: Puppet has 26 failures [09:17:47] PROBLEM - puppet last run on dysprosium is CRITICAL: CRITICAL: puppet fail [09:17:47] PROBLEM - puppet last run on mw1241 is CRITICAL: CRITICAL: Puppet has 32 failures [09:17:47] PROBLEM - puppet last run on cp4009 is CRITICAL: CRITICAL: Puppet has 31 failures [09:17:48] PROBLEM - puppet last run on cp1040 is CRITICAL: CRITICAL: Puppet has 30 failures [09:17:48] PROBLEM - puppet last run on wtp1001 is CRITICAL: CRITICAL: Puppet has 15 failures [09:17:48] PROBLEM - puppet last run on mc1015 is CRITICAL: CRITICAL: puppet fail [09:17:48] PROBLEM - puppet last run on mw1135 is CRITICAL: CRITICAL: puppet fail [09:17:48] PROBLEM - puppet last run on mw1066 is CRITICAL: CRITICAL: Puppet has 69 failures [09:17:48] PROBLEM - puppet last run on radium is CRITICAL: CRITICAL: puppet fail [09:17:57] PROBLEM - puppet last run on analytics1018 is CRITICAL: CRITICAL: Puppet has 19 failures [09:17:57] PROBLEM - puppet last run on chromium is CRITICAL: CRITICAL: Puppet has 13 failures [09:17:57] PROBLEM - puppet last run on lvs3003 is CRITICAL: CRITICAL: Puppet has 14 failures [09:17:57] PROBLEM - puppet last run on mw1086 is CRITICAL: CRITICAL: Puppet has 73 failures [09:17:57] PROBLEM - puppet last run on amssq31 is CRITICAL: CRITICAL: Puppet has 32 failures [09:17:57] PROBLEM - puppet last run on cp3018 is CRITICAL: CRITICAL: Puppet has 11 failures [09:17:58] PROBLEM - puppet last run on virt1008 is CRITICAL: CRITICAL: Puppet has 14 failures [09:17:58] PROBLEM - puppet last run on lvs1004 is CRITICAL: CRITICAL: Puppet has 10 failures [09:17:58] PROBLEM - puppet last run on strontium is CRITICAL: CRITICAL: Puppet has 20 failures [09:17:59] PROBLEM - puppet last run on osm-cp1001 is CRITICAL: CRITICAL: Puppet has 19 failures [09:18:00] PROBLEM - puppet last run on mw1110 is CRITICAL: CRITICAL: puppet fail [09:18:00] PROBLEM - puppet last run on virt1011 is CRITICAL: CRITICAL: Puppet has 31 failures [09:18:01] PROBLEM - puppet last run on sca1001 is CRITICAL: CRITICAL: puppet fail [09:18:01] PROBLEM - puppet last run on db1056 is CRITICAL: CRITICAL: Puppet has 23 failures [09:18:07] PROBLEM - puppet last run on cp1054 is CRITICAL: CRITICAL: Puppet has 14 failures [09:18:08] PROBLEM - puppet last run on mw1255 is CRITICAL: CRITICAL: Puppet has 73 failures [09:18:18] PROBLEM - puppet last run on cp4012 is CRITICAL: CRITICAL: Puppet has 17 failures [09:18:18] PROBLEM - puppet last run on cp4016 is CRITICAL: CRITICAL: puppet fail [09:18:19] PROBLEM - puppet last run on cp4010 is CRITICAL: CRITICAL: puppet fail [09:18:20] PROBLEM - puppet last run on cp4002 is CRITICAL: CRITICAL: Puppet has 33 failures [09:18:20] PROBLEM - puppet last run on mw1113 is CRITICAL: CRITICAL: puppet fail [09:18:20] PROBLEM - puppet last run on amssq33 is CRITICAL: CRITICAL: Puppet has 18 failures [09:18:20] PROBLEM - puppet last run on amssq39 is CRITICAL: CRITICAL: Puppet has 19 failures [09:18:20] PROBLEM - puppet last run on cp3005 is CRITICAL: CRITICAL: puppet fail [09:18:27] PROBLEM - puppet last run on cp1068 is CRITICAL: CRITICAL: puppet fail [09:18:27] PROBLEM - puppet last run on db1045 is CRITICAL: CRITICAL: puppet fail [09:18:27] PROBLEM - puppet last run on mw1143 is CRITICAL: CRITICAL: Puppet has 63 failures [09:18:27] PROBLEM - puppet last run on ms-be1010 is CRITICAL: CRITICAL: puppet fail [09:18:27] PROBLEM - puppet last run on db1035 is CRITICAL: CRITICAL: Puppet has 16 failures [09:18:37] PROBLEM - puppet last run on labsdb1005 is CRITICAL: CRITICAL: puppet fail [09:18:37] PROBLEM - puppet last run on radon is CRITICAL: CRITICAL: puppet fail [09:18:37] PROBLEM - puppet last run on mw1207 is CRITICAL: CRITICAL: puppet fail [09:18:38] PROBLEM - puppet last run on mw1154 is CRITICAL: CRITICAL: Puppet has 35 failures [09:18:38] PROBLEM - puppet last run on mw1203 is CRITICAL: CRITICAL: Puppet has 80 failures [09:18:38] PROBLEM - puppet last run on elastic1029 is CRITICAL: CRITICAL: Puppet has 14 failures [09:18:38] PROBLEM - puppet last run on mw1193 is CRITICAL: CRITICAL: Puppet has 31 failures [09:18:38] PROBLEM - puppet last run on bast1001 is CRITICAL: CRITICAL: Puppet has 9 failures [09:18:47] PROBLEM - puppet last run on mw1047 is CRITICAL: CRITICAL: puppet fail [09:18:47] PROBLEM - puppet last run on mw1021 is CRITICAL: CRITICAL: puppet fail [09:18:47] PROBLEM - puppet last run on mw1199 is CRITICAL: CRITICAL: puppet fail [09:18:57] PROBLEM - puppet last run on lvs2002 is CRITICAL: CRITICAL: puppet fail [09:18:57] PROBLEM - puppet last run on mw1230 is CRITICAL: CRITICAL: puppet fail [09:18:57] PROBLEM - puppet last run on es1004 is CRITICAL: CRITICAL: puppet fail [09:18:58] PROBLEM - puppet last run on es2006 is CRITICAL: CRITICAL: puppet fail [09:19:07] PROBLEM - puppet last run on lvs4004 is CRITICAL: CRITICAL: Puppet has 19 failures [09:19:07] PROBLEM - puppet last run on analytics1031 is CRITICAL: CRITICAL: puppet fail [09:19:07] PROBLEM - puppet last run on elastic1025 is CRITICAL: CRITICAL: puppet fail [09:19:07] PROBLEM - puppet last run on mw1131 is CRITICAL: CRITICAL: Puppet has 32 failures [09:19:08] PROBLEM - puppet last run on db1027 is CRITICAL: CRITICAL: puppet fail [09:19:08] PROBLEM - puppet last run on ms-fe1002 is CRITICAL: CRITICAL: Puppet has 26 failures [09:19:08] PROBLEM - puppet last run on mw1073 is CRITICAL: CRITICAL: Puppet has 78 failures [09:19:08] PROBLEM - puppet last run on mw1015 is CRITICAL: CRITICAL: puppet fail [09:19:17] PROBLEM - puppet last run on virt1005 is CRITICAL: CRITICAL: puppet fail [09:19:17] PROBLEM - puppet last run on amssq45 is CRITICAL: CRITICAL: puppet fail [09:19:17] PROBLEM - puppet last run on amssq37 is CRITICAL: CRITICAL: Puppet has 16 failures [09:19:17] PROBLEM - puppet last run on palladium is CRITICAL: CRITICAL: puppet fail [09:19:18] PROBLEM - puppet last run on mw1158 is CRITICAL: CRITICAL: Puppet has 26 failures [09:19:18] PROBLEM - puppet last run on es2010 is CRITICAL: CRITICAL: Puppet has 18 failures [09:19:18] PROBLEM - puppet last run on analytics1012 is CRITICAL: CRITICAL: Puppet has 29 failures [09:19:18] PROBLEM - puppet last run on analytics1021 is CRITICAL: CRITICAL: puppet fail [09:19:18] PROBLEM - puppet last run on elastic1009 is CRITICAL: CRITICAL: puppet fail [09:19:19] PROBLEM - puppet last run on mw1128 is CRITICAL: CRITICAL: Puppet has 37 failures [09:19:27] PROBLEM - puppet last run on mw1018 is CRITICAL: CRITICAL: Puppet has 60 failures [09:19:27] PROBLEM - puppet last run on mw1078 is CRITICAL: CRITICAL: puppet fail [09:19:28] PROBLEM - puppet last run on bast2001 is CRITICAL: CRITICAL: puppet fail [09:19:39] PROBLEM - puppet last run on cp4015 is CRITICAL: CRITICAL: puppet fail [09:19:39] PROBLEM - puppet last run on mw1085 is CRITICAL: CRITICAL: puppet fail [09:19:39] PROBLEM - puppet last run on eeden is CRITICAL: CRITICAL: Puppet has 25 failures [09:19:39] PROBLEM - puppet last run on cp3011 is CRITICAL: CRITICAL: Puppet has 12 failures [09:19:40] PROBLEM - puppet last run on amssq50 is CRITICAL: CRITICAL: Puppet has 25 failures [09:19:40] PROBLEM - puppet last run on ms-be3004 is CRITICAL: CRITICAL: puppet fail [09:19:40] PROBLEM - puppet last run on cp3017 is CRITICAL: CRITICAL: puppet fail [09:19:47] PROBLEM - puppet last run on mw1020 is CRITICAL: CRITICAL: puppet fail [09:19:48] PROBLEM - puppet last run on mw1103 is CRITICAL: CRITICAL: Puppet has 66 failures [09:19:48] PROBLEM - puppet last run on cp1066 is CRITICAL: CRITICAL: Puppet has 14 failures [09:19:57] PROBLEM - puppet last run on mw1179 is CRITICAL: CRITICAL: puppet fail [09:19:57] PROBLEM - puppet last run on cp1064 is CRITICAL: CRITICAL: Puppet has 17 failures [09:19:57] PROBLEM - puppet last run on mw1095 is CRITICAL: CRITICAL: puppet fail [09:19:58] PROBLEM - puppet last run on db1005 is CRITICAL: CRITICAL: Puppet has 10 failures [09:19:58] PROBLEM - puppet last run on ms-be1014 is CRITICAL: CRITICAL: Puppet has 12 failures [09:20:07] PROBLEM - puppet last run on mw1017 is CRITICAL: CRITICAL: puppet fail [09:20:07] PROBLEM - puppet last run on db1019 is CRITICAL: CRITICAL: puppet fail [09:20:07] PROBLEM - puppet last run on labsdb1002 is CRITICAL: CRITICAL: Puppet has 20 failures [09:20:07] PROBLEM - puppet last run on mw1101 is CRITICAL: CRITICAL: puppet fail [09:20:07] PROBLEM - puppet last run on db1009 is CRITICAL: CRITICAL: puppet fail [09:20:08] PROBLEM - puppet last run on mw1075 is CRITICAL: CRITICAL: puppet fail [09:20:08] PROBLEM - puppet last run on mw1194 is CRITICAL: CRITICAL: Puppet has 32 failures [09:20:08] PROBLEM - puppet last run on lvs2005 is CRITICAL: CRITICAL: puppet fail [09:20:08] PROBLEM - puppet last run on tmh1001 is CRITICAL: CRITICAL: Puppet has 15 failures [09:20:09] PROBLEM - puppet last run on hydrogen is CRITICAL: CRITICAL: Puppet has 25 failures [09:20:09] PROBLEM - puppet last run on pollux is CRITICAL: CRITICAL: puppet fail [09:20:17] PROBLEM - puppet last run on db2010 is CRITICAL: CRITICAL: Puppet has 16 failures [09:20:17] PROBLEM - puppet last run on db2017 is CRITICAL: CRITICAL: Puppet has 15 failures [09:20:17] PROBLEM - puppet last run on ms-be2009 is CRITICAL: CRITICAL: puppet fail [09:20:17] PROBLEM - puppet last run on mw1019 is CRITICAL: CRITICAL: Puppet has 36 failures [09:20:18] PROBLEM - puppet last run on virt1009 is CRITICAL: CRITICAL: Puppet has 14 failures [09:20:27] PROBLEM - puppet last run on elastic1010 is CRITICAL: CRITICAL: Puppet has 10 failures [09:20:27] PROBLEM - puppet last run on mw1137 is CRITICAL: CRITICAL: Puppet has 79 failures [09:20:28] PROBLEM - puppet last run on praseodymium is CRITICAL: CRITICAL: puppet fail [09:20:28] PROBLEM - puppet last run on cp3007 is CRITICAL: CRITICAL: puppet fail [09:20:28] PROBLEM - puppet last run on es1003 is CRITICAL: CRITICAL: Puppet has 24 failures [09:20:28] PROBLEM - puppet last run on db1049 is CRITICAL: CRITICAL: Puppet has 7 failures [09:20:28] PROBLEM - puppet last run on db1068 is CRITICAL: CRITICAL: puppet fail [09:20:28] PROBLEM - puppet last run on wtp1014 is CRITICAL: CRITICAL: puppet fail [09:20:38] PROBLEM - puppet last run on mw1058 is CRITICAL: CRITICAL: Puppet has 34 failures [09:20:38] PROBLEM - puppet last run on mw1083 is CRITICAL: CRITICAL: puppet fail [09:20:38] PROBLEM - puppet last run on db1058 is CRITICAL: CRITICAL: puppet fail [09:20:38] PROBLEM - puppet last run on mw1070 is CRITICAL: CRITICAL: Puppet has 33 failures [09:20:48] PROBLEM - puppet last run on tmh1002 is CRITICAL: CRITICAL: puppet fail [09:20:48] PROBLEM - puppet last run on ms-be2015 is CRITICAL: CRITICAL: puppet fail [09:20:49] PROBLEM - puppet last run on lvs4001 is CRITICAL: CRITICAL: puppet fail [09:20:49] PROBLEM - puppet last run on cp4011 is CRITICAL: CRITICAL: Puppet has 24 failures [09:20:49] PROBLEM - puppet last run on cp4013 is CRITICAL: CRITICAL: puppet fail [09:20:49] PROBLEM - puppet last run on lvs3002 is CRITICAL: CRITICAL: Puppet has 19 failures [09:20:49] PROBLEM - puppet last run on cp3021 is CRITICAL: CRITICAL: Puppet has 20 failures [09:20:50] PROBLEM - puppet last run on cp3019 is CRITICAL: CRITICAL: Puppet has 17 failures [09:20:57] PROBLEM - puppet last run on es1005 is CRITICAL: CRITICAL: puppet fail [09:20:58] PROBLEM - puppet last run on mw1157 is CRITICAL: CRITICAL: Puppet has 35 failures [09:21:07] PROBLEM - puppet last run on elastic1013 is CRITICAL: CRITICAL: Puppet has 22 failures [09:21:07] PROBLEM - puppet last run on labsdb1007 is CRITICAL: CRITICAL: Puppet has 10 failures [09:21:07] PROBLEM - puppet last run on mw1094 is CRITICAL: CRITICAL: puppet fail [09:21:08] PROBLEM - puppet last run on lanthanum is CRITICAL: CRITICAL: Puppet has 36 failures [09:21:08] PROBLEM - puppet last run on mw1136 is CRITICAL: CRITICAL: Puppet has 70 failures [09:21:08] PROBLEM - puppet last run on ms-be1005 is CRITICAL: CRITICAL: Puppet has 9 failures [09:21:17] PROBLEM - puppet last run on elastic1016 is CRITICAL: CRITICAL: puppet fail [09:21:17] PROBLEM - puppet last run on mw1102 is CRITICAL: CRITICAL: Puppet has 72 failures [09:21:17] PROBLEM - puppet last run on mw1127 is CRITICAL: CRITICAL: puppet fail [09:21:17] PROBLEM - puppet last run on cp1057 is CRITICAL: CRITICAL: Puppet has 28 failures [09:21:17] PROBLEM - puppet last run on mw1184 is CRITICAL: CRITICAL: Puppet has 63 failures [09:21:17] PROBLEM - puppet last run on analytics1034 is CRITICAL: CRITICAL: puppet fail [09:21:27] PROBLEM - puppet last run on mw1191 is CRITICAL: CRITICAL: puppet fail [09:21:27] PROBLEM - puppet last run on cp1065 is CRITICAL: CRITICAL: Puppet has 17 failures [09:21:28] PROBLEM - puppet last run on stat1001 is CRITICAL: CRITICAL: puppet fail [09:21:28] PROBLEM - puppet last run on ms-be2010 is CRITICAL: CRITICAL: Puppet has 11 failures [09:21:28] PROBLEM - puppet last run on cp4017 is CRITICAL: CRITICAL: puppet fail [09:21:37] PROBLEM - puppet last run on lvs1006 is CRITICAL: CRITICAL: Puppet has 20 failures [09:21:37] PROBLEM - puppet last run on ocg1002 is CRITICAL: CRITICAL: Puppet has 23 failures [09:21:37] PROBLEM - puppet last run on mw1169 is CRITICAL: CRITICAL: puppet fail [09:21:38] PROBLEM - puppet last run on analytics1029 is CRITICAL: CRITICAL: Puppet has 10 failures [09:21:38] PROBLEM - puppet last run on amssq52 is CRITICAL: CRITICAL: Puppet has 20 failures [09:21:47] PROBLEM - puppet last run on mw1214 is CRITICAL: CRITICAL: Puppet has 43 failures [09:21:47] PROBLEM - puppet last run on erbium is CRITICAL: CRITICAL: Puppet has 20 failures [09:21:48] PROBLEM - puppet last run on cp1069 is CRITICAL: CRITICAL: puppet fail [09:21:48] PROBLEM - puppet last run on analytics1024 is CRITICAL: CRITICAL: Puppet has 23 failures [09:21:48] PROBLEM - puppet last run on zirconium is CRITICAL: CRITICAL: puppet fail [09:21:57] PROBLEM - puppet last run on mw1232 is CRITICAL: CRITICAL: Puppet has 30 failures [09:21:57] PROBLEM - puppet last run on virt1012 is CRITICAL: CRITICAL: puppet fail [09:21:58] PROBLEM - puppet last run on db1024 is CRITICAL: CRITICAL: puppet fail [09:21:58] PROBLEM - puppet last run on mw1013 is CRITICAL: CRITICAL: puppet fail [09:22:07] PROBLEM - puppet last run on analytics1019 is CRITICAL: CRITICAL: Puppet has 12 failures [09:22:08] PROBLEM - puppet last run on mw1244 is CRITICAL: CRITICAL: puppet fail [09:22:08] PROBLEM - puppet last run on mw1234 is CRITICAL: CRITICAL: puppet fail [09:22:17] PROBLEM - puppet last run on mw1196 is CRITICAL: CRITICAL: puppet fail [09:22:17] PROBLEM - puppet last run on mw1182 is CRITICAL: CRITICAL: Puppet has 88 failures [09:22:17] PROBLEM - puppet last run on mw1035 is CRITICAL: CRITICAL: puppet fail [09:22:17] PROBLEM - puppet last run on es1009 is CRITICAL: CRITICAL: puppet fail [09:22:17] PROBLEM - puppet last run on mc1008 is CRITICAL: CRITICAL: puppet fail [09:22:18] PROBLEM - puppet last run on db1007 is CRITICAL: CRITICAL: puppet fail [09:22:18] PROBLEM - puppet last run on cp1043 is CRITICAL: CRITICAL: puppet fail [09:22:18] PROBLEM - puppet last run on es1006 is CRITICAL: CRITICAL: puppet fail [09:22:27] PROBLEM - puppet last run on mw1036 is CRITICAL: CRITICAL: Puppet has 60 failures [09:22:27] PROBLEM - puppet last run on mw1132 is CRITICAL: CRITICAL: puppet fail [09:22:27] PROBLEM - puppet last run on analytics1015 is CRITICAL: CRITICAL: Puppet has 12 failures [09:22:27] PROBLEM - puppet last run on wtp1024 is CRITICAL: CRITICAL: Puppet has 11 failures [09:22:27] PROBLEM - puppet last run on db1041 is CRITICAL: CRITICAL: Puppet has 11 failures [09:22:27] PROBLEM - puppet last run on mw1161 is CRITICAL: CRITICAL: puppet fail [09:22:28] PROBLEM - puppet last run on mw1246 is CRITICAL: CRITICAL: puppet fail [09:22:28] PROBLEM - puppet last run on mc1010 is CRITICAL: CRITICAL: Puppet has 13 failures [09:22:28] PROBLEM - puppet last run on cp1051 is CRITICAL: CRITICAL: Puppet has 16 failures [09:22:37] PROBLEM - puppet last run on mw1138 is CRITICAL: CRITICAL: puppet fail [09:22:38] PROBLEM - puppet last run on db2041 is CRITICAL: CRITICAL: puppet fail [09:22:38] PROBLEM - puppet last run on db2035 is CRITICAL: CRITICAL: puppet fail [09:22:38] PROBLEM - puppet last run on mw1257 is CRITICAL: CRITICAL: puppet fail [09:22:47] PROBLEM - puppet last run on db1010 is CRITICAL: CRITICAL: Puppet has 21 failures [09:22:47] PROBLEM - puppet last run on ytterbium is CRITICAL: CRITICAL: puppet fail [09:22:48] PROBLEM - puppet last run on mw1216 is CRITICAL: CRITICAL: puppet fail [09:22:57] PROBLEM - puppet last run on wtp1021 is CRITICAL: CRITICAL: puppet fail [09:22:57] PROBLEM - puppet last run on mw1245 is CRITICAL: CRITICAL: puppet fail [09:22:57] PROBLEM - puppet last run on cp3013 is CRITICAL: CRITICAL: puppet fail [09:22:58] PROBLEM - puppet last run on mw1040 is CRITICAL: CRITICAL: puppet fail [09:22:58] PROBLEM - puppet last run on dbstore1001 is CRITICAL: CRITICAL: puppet fail [09:22:58] PROBLEM - puppet last run on elastic1026 is CRITICAL: CRITICAL: Puppet has 29 failures [09:22:58] PROBLEM - puppet last run on mw1109 is CRITICAL: CRITICAL: puppet fail [09:22:58] PROBLEM - puppet last run on mw1124 is CRITICAL: CRITICAL: puppet fail [09:22:59] PROBLEM - puppet last run on labsdb1001 is CRITICAL: CRITICAL: Puppet has 19 failures [09:23:07] PROBLEM - puppet last run on virt1002 is CRITICAL: CRITICAL: puppet fail [09:23:17] PROBLEM - puppet last run on mw1096 is CRITICAL: CRITICAL: Puppet has 34 failures [09:23:17] PROBLEM - puppet last run on amslvs4 is CRITICAL: CRITICAL: Puppet has 12 failures [09:23:18] PROBLEM - puppet last run on amssq58 is CRITICAL: CRITICAL: Puppet has 31 failures [09:23:18] PROBLEM - puppet last run on amssq57 is CRITICAL: CRITICAL: Puppet has 17 failures [09:23:18] PROBLEM - puppet last run on amssq44 is CRITICAL: CRITICAL: puppet fail [09:23:18] PROBLEM - puppet last run on pc1001 is CRITICAL: CRITICAL: Puppet has 20 failures [09:23:28] PROBLEM - puppet last run on elastic1028 is CRITICAL: CRITICAL: puppet fail [09:23:28] PROBLEM - puppet last run on vanadium is CRITICAL: CRITICAL: Puppet has 9 failures [09:23:28] PROBLEM - puppet last run on mw1256 is CRITICAL: CRITICAL: puppet fail [09:23:29] PROBLEM - puppet last run on wtp1017 is CRITICAL: CRITICAL: puppet fail [09:23:29] PROBLEM - puppet last run on ms-be1013 is CRITICAL: CRITICAL: puppet fail [09:23:29] PROBLEM - puppet last run on ms-be1001 is CRITICAL: CRITICAL: puppet fail [09:23:37] PROBLEM - puppet last run on mc1009 is CRITICAL: CRITICAL: Puppet has 13 failures [09:23:37] PROBLEM - puppet last run on mw1252 is CRITICAL: CRITICAL: Puppet has 87 failures [09:23:38] PROBLEM - puppet last run on rdb1004 is CRITICAL: CRITICAL: puppet fail [09:23:38] PROBLEM - puppet last run on mw1178 is CRITICAL: CRITICAL: puppet fail [09:23:38] PROBLEM - puppet last run on uranium is CRITICAL: CRITICAL: Puppet has 17 failures [09:23:48] PROBLEM - puppet last run on analytics1039 is CRITICAL: CRITICAL: Puppet has 24 failures [09:23:48] PROBLEM - puppet last run on mw1233 is CRITICAL: CRITICAL: puppet fail [09:23:48] PROBLEM - puppet last run on mc1004 is CRITICAL: CRITICAL: Puppet has 11 failures [09:23:48] PROBLEM - puppet last run on mw1028 is CRITICAL: CRITICAL: puppet fail [09:23:49] PROBLEM - puppet last run on es2003 is CRITICAL: CRITICAL: puppet fail [09:23:49] PROBLEM - puppet last run on elastic1031 is CRITICAL: CRITICAL: Puppet has 14 failures [09:23:49] PROBLEM - puppet last run on ocg1001 is CRITICAL: CRITICAL: puppet fail [09:23:49] PROBLEM - puppet last run on ms-be2002 is CRITICAL: CRITICAL: puppet fail [09:23:57] PROBLEM - puppet last run on logstash1003 is CRITICAL: CRITICAL: puppet fail [09:23:57] PROBLEM - puppet last run on mw1089 is CRITICAL: CRITICAL: puppet fail [09:23:57] PROBLEM - puppet last run on mw1218 is CRITICAL: CRITICAL: Puppet has 31 failures [09:23:57] PROBLEM - puppet last run on mc1011 is CRITICAL: CRITICAL: Puppet has 9 failures [09:23:58] PROBLEM - puppet last run on cp1059 is CRITICAL: CRITICAL: puppet fail [09:23:58] PROBLEM - puppet last run on cerium is CRITICAL: CRITICAL: puppet fail [09:23:58] PROBLEM - puppet last run on analytics1036 is CRITICAL: CRITICAL: Puppet has 9 failures [09:23:58] PROBLEM - puppet last run on mw1134 is CRITICAL: CRITICAL: puppet fail [09:24:07] PROBLEM - puppet last run on mw1007 is CRITICAL: CRITICAL: puppet fail [09:24:07] PROBLEM - puppet last run on amssq43 is CRITICAL: CRITICAL: puppet fail [09:24:08] PROBLEM - puppet last run on mw1145 is CRITICAL: CRITICAL: puppet fail [09:24:08] PROBLEM - puppet last run on mw1240 is CRITICAL: CRITICAL: Puppet has 34 failures [09:24:08] PROBLEM - puppet last run on mw1005 is CRITICAL: CRITICAL: Puppet has 51 failures [09:24:08] PROBLEM - puppet last run on mc1016 is CRITICAL: CRITICAL: Puppet has 22 failures [09:24:08] PROBLEM - puppet last run on ms-be1004 is CRITICAL: CRITICAL: puppet fail [09:24:17] PROBLEM - puppet last run on mw1012 is CRITICAL: CRITICAL: puppet fail [09:24:17] PROBLEM - puppet last run on mw1063 is CRITICAL: CRITICAL: puppet fail [09:24:17] PROBLEM - puppet last run on mw1221 is CRITICAL: CRITICAL: Puppet has 35 failures [09:24:17] PROBLEM - puppet last run on mw1080 is CRITICAL: CRITICAL: puppet fail [09:24:28] PROBLEM - puppet last run on ms-fe1003 is CRITICAL: CRITICAL: puppet fail [09:24:28] PROBLEM - puppet last run on cp4007 is CRITICAL: CRITICAL: Puppet has 22 failures [09:24:29] PROBLEM - puppet last run on mw1130 is CRITICAL: CRITICAL: Puppet has 35 failures [09:24:29] PROBLEM - puppet last run on cp3022 is CRITICAL: CRITICAL: Puppet has 19 failures [09:24:29] PROBLEM - puppet last run on cp3008 is CRITICAL: CRITICAL: puppet fail [09:24:29] PROBLEM - puppet last run on cp3015 is CRITICAL: CRITICAL: Puppet has 26 failures [09:24:29] PROBLEM - puppet last run on cp3006 is CRITICAL: CRITICAL: puppet fail [09:24:29] PROBLEM - puppet last run on magnesium is CRITICAL: CRITICAL: Puppet has 26 failures [09:24:37] PROBLEM - puppet last run on mw1031 is CRITICAL: CRITICAL: Puppet has 28 failures [09:24:37] PROBLEM - puppet last run on mw1147 is CRITICAL: CRITICAL: Puppet has 66 failures [09:24:37] PROBLEM - puppet last run on mw1067 is CRITICAL: CRITICAL: puppet fail [09:24:37] PROBLEM - puppet last run on wtp1019 is CRITICAL: CRITICAL: puppet fail [09:24:37] PROBLEM - puppet last run on wtp1009 is CRITICAL: CRITICAL: puppet fail [09:24:47] PROBLEM - puppet last run on dbproxy1002 is CRITICAL: CRITICAL: Puppet has 27 failures [09:24:47] PROBLEM - puppet last run on mw1115 is CRITICAL: CRITICAL: Puppet has 34 failures [09:24:47] PROBLEM - puppet last run on mw1200 is CRITICAL: CRITICAL: puppet fail [09:24:48] PROBLEM - puppet last run on mw1041 is CRITICAL: CRITICAL: puppet fail [09:24:48] PROBLEM - puppet last run on db1053 is CRITICAL: CRITICAL: Puppet has 26 failures [09:24:57] PROBLEM - puppet last run on logstash1001 is CRITICAL: CRITICAL: Puppet has 31 failures [09:24:57] PROBLEM - puppet last run on mw1160 is CRITICAL: CRITICAL: puppet fail [09:24:58] PROBLEM - puppet last run on carbon is CRITICAL: CRITICAL: puppet fail [09:24:58] PROBLEM - puppet last run on netmon1001 is CRITICAL: CRITICAL: Puppet has 19 failures [09:24:58] PROBLEM - puppet last run on es1001 is CRITICAL: CRITICAL: Puppet has 21 failures [09:24:58] PROBLEM - puppet last run on cp1049 is CRITICAL: CRITICAL: Puppet has 28 failures [09:24:58] PROBLEM - puppet last run on db1065 is CRITICAL: CRITICAL: Puppet has 9 failures [09:24:58] PROBLEM - puppet last run on ms-be2013 is CRITICAL: CRITICAL: Puppet has 13 failures [09:24:58] PROBLEM - puppet last run on es2005 is CRITICAL: CRITICAL: Puppet has 22 failures [09:24:59] PROBLEM - puppet last run on es2002 is CRITICAL: CRITICAL: puppet fail [09:25:00] PROBLEM - puppet last run on labstore2001 is CRITICAL: CRITICAL: puppet fail [09:25:00] PROBLEM - puppet last run on mw1045 is CRITICAL: CRITICAL: Puppet has 70 failures [09:25:07] PROBLEM - puppet last run on db1029 is CRITICAL: CRITICAL: Puppet has 23 failures [09:25:07] PROBLEM - puppet last run on ms-be2003 is CRITICAL: CRITICAL: Puppet has 13 failures [09:25:07] PROBLEM - puppet last run on analytics1033 is CRITICAL: CRITICAL: Puppet has 9 failures [09:25:07] PROBLEM - puppet last run on mw1072 is CRITICAL: CRITICAL: Puppet has 35 failures [09:25:08] PROBLEM - puppet last run on lvs4002 is CRITICAL: CRITICAL: Puppet has 11 failures [09:25:08] PROBLEM - puppet last run on caesium is CRITICAL: CRITICAL: Puppet has 13 failures [09:25:08] PROBLEM - puppet last run on mw1006 is CRITICAL: CRITICAL: puppet fail [09:25:08] PROBLEM - puppet last run on mw1174 is CRITICAL: CRITICAL: puppet fail [09:25:17] PROBLEM - puppet last run on cp1070 is CRITICAL: CRITICAL: Puppet has 29 failures [09:25:17] PROBLEM - puppet last run on analytics1017 is CRITICAL: CRITICAL: Puppet has 13 failures [09:25:18] PROBLEM - puppet last run on mw1141 is CRITICAL: CRITICAL: puppet fail [09:25:18] PROBLEM - puppet last run on mw1048 is CRITICAL: CRITICAL: Puppet has 37 failures [09:25:18] PROBLEM - puppet last run on amssq49 is CRITICAL: CRITICAL: puppet fail [09:25:18] PROBLEM - puppet last run on amssq54 is CRITICAL: CRITICAL: puppet fail [09:25:18] PROBLEM - puppet last run on mw1197 is CRITICAL: CRITICAL: puppet fail [09:25:27] PROBLEM - puppet last run on wtp1006 is CRITICAL: CRITICAL: puppet fail [09:25:27] PROBLEM - puppet last run on mw1187 is CRITICAL: CRITICAL: puppet fail [09:25:27] PROBLEM - puppet last run on neptunium is CRITICAL: CRITICAL: Puppet has 21 failures [09:25:28] PROBLEM - puppet last run on elastic1004 is CRITICAL: CRITICAL: puppet fail [09:25:28] PROBLEM - puppet last run on analytics1025 is CRITICAL: CRITICAL: Puppet has 24 failures [09:25:28] PROBLEM - puppet last run on mw1038 is CRITICAL: CRITICAL: Puppet has 72 failures [09:25:28] PROBLEM - puppet last run on mc1006 is CRITICAL: CRITICAL: Puppet has 20 failures [09:25:29] PROBLEM - puppet last run on mw1059 is CRITICAL: CRITICAL: puppet fail [09:25:38] PROBLEM - puppet last run on db2009 is CRITICAL: CRITICAL: puppet fail [09:25:39] PROBLEM - puppet last run on wtp1020 is CRITICAL: CRITICAL: puppet fail [09:25:39] PROBLEM - puppet last run on analytics1041 is CRITICAL: CRITICAL: puppet fail [09:25:39] PROBLEM - puppet last run on cp4006 is CRITICAL: CRITICAL: puppet fail [09:25:39] PROBLEM - puppet last run on amssq32 is CRITICAL: CRITICAL: puppet fail [09:25:39] PROBLEM - puppet last run on amslvs2 is CRITICAL: CRITICAL: puppet fail [09:25:39] PROBLEM - puppet last run on amssq59 is CRITICAL: CRITICAL: Puppet has 23 failures [09:25:40] PROBLEM - puppet last run on amssq53 is CRITICAL: CRITICAL: puppet fail [09:25:40] PROBLEM - puppet last run on lvs3001 is CRITICAL: CRITICAL: puppet fail [09:25:47] PROBLEM - puppet last run on helium is CRITICAL: CRITICAL: puppet fail [09:25:47] PROBLEM - puppet last run on es1008 is CRITICAL: CRITICAL: Puppet has 22 failures [09:25:48] PROBLEM - puppet last run on snapshot1003 is CRITICAL: CRITICAL: Puppet has 17 failures [09:25:48] PROBLEM - puppet last run on gold is CRITICAL: CRITICAL: puppet fail [09:25:48] PROBLEM - puppet last run on xenon is CRITICAL: CRITICAL: Puppet has 7 failures [09:25:57] PROBLEM - puppet last run on mw1254 is CRITICAL: CRITICAL: Puppet has 74 failures [09:25:57] PROBLEM - puppet last run on lvs1002 is CRITICAL: CRITICAL: puppet fail [09:25:57] PROBLEM - puppet last run on mw1250 is CRITICAL: CRITICAL: Puppet has 63 failures [09:25:57] PROBLEM - puppet last run on mw1082 is CRITICAL: CRITICAL: puppet fail [09:25:58] PROBLEM - puppet last run on mw1026 is CRITICAL: CRITICAL: puppet fail [09:25:58] PROBLEM - puppet last run on platinum is CRITICAL: CRITICAL: puppet fail [09:25:58] PROBLEM - puppet last run on mc1002 is CRITICAL: CRITICAL: puppet fail [09:25:58] PROBLEM - puppet last run on db1031 is CRITICAL: CRITICAL: puppet fail [09:25:58] PROBLEM - puppet last run on analytics1040 is CRITICAL: CRITICAL: Puppet has 21 failures [09:25:59] PROBLEM - puppet last run on db1022 is CRITICAL: CRITICAL: puppet fail [09:26:00] PROBLEM - puppet last run on cp1055 is CRITICAL: CRITICAL: Puppet has 11 failures [09:26:07] PROBLEM - puppet last run on mw1140 is CRITICAL: CRITICAL: Puppet has 36 failures [09:26:07] PROBLEM - puppet last run on elastic1007 is CRITICAL: CRITICAL: Puppet has 11 failures [09:26:07] PROBLEM - puppet last run on mw1106 is CRITICAL: CRITICAL: Puppet has 71 failures [09:26:07] PROBLEM - puppet last run on db1050 is CRITICAL: CRITICAL: Puppet has 11 failures [09:26:08] PROBLEM - puppet last run on lead is CRITICAL: CRITICAL: puppet fail [09:26:08] PROBLEM - puppet last run on db1073 is CRITICAL: CRITICAL: puppet fail [09:26:08] PROBLEM - puppet last run on cp1047 is CRITICAL: CRITICAL: puppet fail [09:26:08] PROBLEM - puppet last run on rbf1002 is CRITICAL: CRITICAL: puppet fail [09:26:08] PROBLEM - puppet last run on ms-fe2004 is CRITICAL: CRITICAL: Puppet has 28 failures [09:26:09] PROBLEM - puppet last run on db2034 is CRITICAL: CRITICAL: Puppet has 13 failures [09:26:09] PROBLEM - puppet last run on sca1002 is CRITICAL: CRITICAL: puppet fail [09:26:10] PROBLEM - puppet last run on capella is CRITICAL: CRITICAL: puppet fail [09:26:11] PROBLEM - puppet last run on mw1173 is CRITICAL: CRITICAL: puppet fail [09:26:11] PROBLEM - puppet last run on heze is CRITICAL: CRITICAL: Puppet has 6 failures [09:26:11] PROBLEM - puppet last run on es2008 is CRITICAL: CRITICAL: Puppet has 9 failures [09:26:12] PROBLEM - puppet last run on db2005 is CRITICAL: CRITICAL: Puppet has 12 failures [09:26:13] PROBLEM - puppet last run on db2019 is CRITICAL: CRITICAL: Puppet has 7 failures [09:26:17] PROBLEM - puppet last run on ms-be1006 is CRITICAL: CRITICAL: Puppet has 22 failures [09:26:17] PROBLEM - puppet last run on mw1224 is CRITICAL: CRITICAL: Puppet has 67 failures [09:26:17] PROBLEM - puppet last run on mw1226 is CRITICAL: CRITICAL: puppet fail [09:26:17] PROBLEM - puppet last run on ms-be2006 is CRITICAL: CRITICAL: puppet fail [09:26:17] PROBLEM - puppet last run on elastic1018 is CRITICAL: CRITICAL: puppet fail [09:26:18] PROBLEM - puppet last run on cp1039 is CRITICAL: CRITICAL: Puppet has 17 failures [09:26:18] PROBLEM - puppet last run on elastic1001 is CRITICAL: CRITICAL: Puppet has 20 failures [09:26:18] PROBLEM - puppet last run on searchidx1001 is CRITICAL: CRITICAL: Puppet has 8 failures [09:26:18] PROBLEM - puppet last run on potassium is CRITICAL: CRITICAL: puppet fail [09:26:27] PROBLEM - puppet last run on mw1189 is CRITICAL: CRITICAL: puppet fail [09:26:27] PROBLEM - puppet last run on ms-be1003 is CRITICAL: CRITICAL: Puppet has 14 failures [09:26:28] PROBLEM - puppet last run on ms-fe1001 is CRITICAL: CRITICAL: Puppet has 18 failures [09:26:28] PROBLEM - puppet last run on amssq61 is CRITICAL: CRITICAL: puppet fail [09:26:28] PROBLEM - puppet last run on cp3016 is CRITICAL: CRITICAL: puppet fail [09:26:28] PROBLEM - puppet last run on labstore1003 is CRITICAL: CRITICAL: Puppet has 9 failures [09:26:28] PROBLEM - puppet last run on analytics1020 is CRITICAL: CRITICAL: Puppet has 20 failures [09:26:38] PROBLEM - puppet last run on db2039 is CRITICAL: CRITICAL: Puppet has 11 failures [09:26:39] PROBLEM - puppet last run on mw1222 is CRITICAL: CRITICAL: puppet fail [09:26:48] PROBLEM - puppet last run on mw1118 is CRITICAL: CRITICAL: puppet fail [09:26:48] PROBLEM - puppet last run on mw1100 is CRITICAL: CRITICAL: Puppet has 39 failures [09:26:49] PROBLEM - puppet last run on cp3020 is CRITICAL: CRITICAL: Puppet has 17 failures [09:26:57] PROBLEM - puppetmaster https on palladium is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 8140: HTTP/1.1 500 Internal Server Error [09:26:58] PROBLEM - puppet last run on elastic1021 is CRITICAL: CRITICAL: puppet fail [09:26:58] PROBLEM - puppet last run on mw1205 is CRITICAL: CRITICAL: Puppet has 42 failures [09:26:58] PROBLEM - puppet last run on mw1166 is CRITICAL: CRITICAL: puppet fail [09:26:58] PROBLEM - puppet last run on mw1099 is CRITICAL: CRITICAL: puppet fail [09:26:58] PROBLEM - puppet last run on mc1003 is CRITICAL: CRITICAL: Puppet has 24 failures [09:26:58] PROBLEM - puppet last run on mw1088 is CRITICAL: CRITICAL: puppet fail [09:26:58] PROBLEM - puppet last run on mw1242 is CRITICAL: CRITICAL: puppet fail [09:26:59] PROBLEM - puppet last run on mw1009 is CRITICAL: CRITICAL: Puppet has 51 failures [09:27:07] PROBLEM - puppet last run on wtp1016 is CRITICAL: CRITICAL: puppet fail [09:27:07] PROBLEM - puppet last run on mw1150 is CRITICAL: CRITICAL: Puppet has 33 failures [09:27:08] PROBLEM - puppet last run on elastic1012 is CRITICAL: CRITICAL: puppet fail [09:27:08] PROBLEM - puppet last run on mw1046 is CRITICAL: CRITICAL: puppet fail [09:27:17] PROBLEM - puppet last run on lvs1005 is CRITICAL: CRITICAL: puppet fail [09:27:17] PROBLEM - puppet last run on mw1164 is CRITICAL: CRITICAL: puppet fail [09:27:17] PROBLEM - puppet last run on db1066 is CRITICAL: CRITICAL: puppet fail [09:27:17] PROBLEM - puppet last run on mw1176 is CRITICAL: CRITICAL: Puppet has 34 failures [09:27:17] PROBLEM - puppet last run on mw1003 is CRITICAL: CRITICAL: Puppet has 62 failures [09:27:18] PROBLEM - puppet last run on db1059 is CRITICAL: CRITICAL: puppet fail [09:27:18] PROBLEM - puppet last run on analytics1035 is CRITICAL: CRITICAL: Puppet has 17 failures [09:27:18] PROBLEM - puppet last run on mw1235 is CRITICAL: CRITICAL: puppet fail [09:27:27] PROBLEM - puppet last run on db2018 is CRITICAL: CRITICAL: puppet fail [09:27:27] PROBLEM - puppet last run on es2001 is CRITICAL: CRITICAL: puppet fail [09:27:27] PROBLEM - puppet last run on ms-fe1004 is CRITICAL: CRITICAL: Puppet has 14 failures [09:27:27] PROBLEM - puppet last run on mw1217 is CRITICAL: CRITICAL: Puppet has 76 failures [09:27:28] PROBLEM - puppet last run on mw1061 is CRITICAL: CRITICAL: puppet fail [09:27:28] PROBLEM - puppet last run on cp1056 is CRITICAL: CRITICAL: puppet fail [09:27:37] PROBLEM - puppet last run on mw1008 is CRITICAL: CRITICAL: Puppet has 61 failures [09:27:37] PROBLEM - puppet last run on mw1120 is CRITICAL: CRITICAL: Puppet has 33 failures [09:27:37] PROBLEM - puppet last run on virt1006 is CRITICAL: CRITICAL: puppet fail [09:27:38] PROBLEM - puppet last run on cp1061 is CRITICAL: CRITICAL: puppet fail [09:27:38] PROBLEM - puppet last run on mw1042 is CRITICAL: CRITICAL: puppet fail [09:27:38] PROBLEM - puppet last run on mw1117 is CRITICAL: CRITICAL: Puppet has 73 failures [09:27:38] PROBLEM - puppet last run on mw1170 is CRITICAL: CRITICAL: puppet fail [09:27:38] PROBLEM - puppet last run on mw1228 is CRITICAL: CRITICAL: puppet fail [09:27:39] PROBLEM - puppet last run on cp3003 is CRITICAL: CRITICAL: Puppet has 15 failures [09:27:39] PROBLEM - puppet last run on mw1060 is CRITICAL: CRITICAL: Puppet has 69 failures [09:27:47] PROBLEM - puppet last run on mw1065 is CRITICAL: CRITICAL: Puppet has 30 failures [09:27:47] PROBLEM - puppet last run on es2009 is CRITICAL: CRITICAL: Puppet has 11 failures [09:27:48] PROBLEM - puppet last run on ms-be2004 is CRITICAL: CRITICAL: Puppet has 13 failures [09:27:48] PROBLEM - puppet last run on db2002 is CRITICAL: CRITICAL: Puppet has 9 failures [09:27:48] PROBLEM - puppet last run on db1040 is CRITICAL: CRITICAL: puppet fail [09:27:48] PROBLEM - puppet last run on db1002 is CRITICAL: CRITICAL: puppet fail [09:27:48] PROBLEM - puppet last run on mw1153 is CRITICAL: CRITICAL: Puppet has 41 failures [09:27:48] PROBLEM - puppet last run on analytics1030 is CRITICAL: CRITICAL: puppet fail [09:27:49] PROBLEM - puppet last run on mw1025 is CRITICAL: CRITICAL: puppet fail [09:27:50] PROBLEM - puppet last run on iron is CRITICAL: CRITICAL: puppet fail [09:27:50] PROBLEM - puppet last run on ms-fe2001 is CRITICAL: CRITICAL: Puppet has 30 failures [09:27:58] PROBLEM - puppet last run on ruthenium is CRITICAL: CRITICAL: puppet fail [09:27:58] PROBLEM - puppet last run on elastic1008 is CRITICAL: CRITICAL: Puppet has 18 failures [09:27:59] PROBLEM - puppet last run on labsdb1003 is CRITICAL: CRITICAL: puppet fail [09:27:59] PROBLEM - puppet last run on cp4003 is CRITICAL: CRITICAL: puppet fail [09:27:59] PROBLEM - puppet last run on amssq35 is CRITICAL: CRITICAL: Puppet has 21 failures [09:28:08] PROBLEM - puppet last run on db1042 is CRITICAL: CRITICAL: puppet fail [09:28:08] PROBLEM - puppet last run on mw1068 is CRITICAL: CRITICAL: Puppet has 63 failures [09:28:17] PROBLEM - puppet last run on mw1092 is CRITICAL: CRITICAL: puppet fail [09:28:18] PROBLEM - puppet last run on mw1123 is CRITICAL: CRITICAL: Puppet has 71 failures [09:28:18] PROBLEM - puppet last run on db1015 is CRITICAL: CRITICAL: puppet fail [09:28:18] PROBLEM - puppet last run on logstash1002 is CRITICAL: CRITICAL: puppet fail [09:28:18] PROBLEM - puppet last run on mw1251 is CRITICAL: CRITICAL: puppet fail [09:28:27] PROBLEM - puppet last run on mw1052 is CRITICAL: CRITICAL: Puppet has 63 failures [09:28:27] PROBLEM - puppet last run on elastic1027 is CRITICAL: CRITICAL: puppet fail [09:28:27] PROBLEM - puppet last run on db1021 is CRITICAL: CRITICAL: Puppet has 10 failures [09:28:27] PROBLEM - puppet last run on mw1069 is CRITICAL: CRITICAL: Puppet has 65 failures [09:28:28] PROBLEM - puppet last run on db1034 is CRITICAL: CRITICAL: Puppet has 24 failures [09:28:28] PROBLEM - puppet last run on lvs2001 is CRITICAL: CRITICAL: puppet fail [09:28:28] PROBLEM - puppet last run on lvs2004 is CRITICAL: CRITICAL: Puppet has 19 failures [09:28:37] PROBLEM - puppet last run on ms-fe2003 is CRITICAL: CRITICAL: puppet fail [09:28:37] PROBLEM - puppet last run on db1018 is CRITICAL: CRITICAL: Puppet has 25 failures [09:28:38] PROBLEM - puppet last run on db1028 is CRITICAL: CRITICAL: puppet fail [09:28:38] PROBLEM - puppet last run on mw1175 is CRITICAL: CRITICAL: puppet fail [09:28:38] PROBLEM - puppet last run on db1046 is CRITICAL: CRITICAL: Puppet has 9 failures [09:28:47] PROBLEM - puppet last run on cp4014 is CRITICAL: CRITICAL: puppet fail [09:28:47] PROBLEM - puppet last run on tin is CRITICAL: CRITICAL: puppet fail [09:28:48] PROBLEM - puppet last run on elastic1022 is CRITICAL: CRITICAL: Puppet has 26 failures [09:28:48] PROBLEM - puppet last run on mw1172 is CRITICAL: CRITICAL: puppet fail [09:28:48] PROBLEM - puppet last run on amssq34 is CRITICAL: CRITICAL: puppet fail [09:28:48] PROBLEM - puppet last run on amslvs1 is CRITICAL: CRITICAL: puppet fail [09:28:57] PROBLEM - puppet last run on mw1114 is CRITICAL: CRITICAL: puppet fail [09:28:57] PROBLEM - puppet last run on mw1119 is CRITICAL: CRITICAL: Puppet has 71 failures [09:28:58] PROBLEM - puppet last run on db1023 is CRITICAL: CRITICAL: puppet fail [09:28:58] PROBLEM - puppet last run on mw1039 is CRITICAL: CRITICAL: puppet fail [09:29:07] PROBLEM - puppet last run on labnet1001 is CRITICAL: CRITICAL: Puppet has 7 failures [09:29:07] PROBLEM - puppet last run on db2040 is CRITICAL: CRITICAL: Puppet has 11 failures [09:29:07] PROBLEM - puppet last run on labcontrol2001 is CRITICAL: CRITICAL: puppet fail [09:29:07] PROBLEM - puppet last run on mw1211 is CRITICAL: CRITICAL: puppet fail [09:29:08] PROBLEM - puppet last run on elastic1030 is CRITICAL: CRITICAL: Puppet has 26 failures [09:29:08] PROBLEM - puppet last run on cp4004 is CRITICAL: CRITICAL: puppet fail [09:29:08] PROBLEM - puppet last run on cp4008 is CRITICAL: CRITICAL: Puppet has 19 failures [09:29:08] PROBLEM - puppet last run on ms-be3002 is CRITICAL: CRITICAL: puppet fail [09:29:09] PROBLEM - puppet last run on amssq55 is CRITICAL: CRITICAL: puppet fail [09:29:09] PROBLEM - puppet last run on amssq48 is CRITICAL: CRITICAL: puppet fail [09:29:09] PROBLEM - puppet last run on amssq60 is CRITICAL: CRITICAL: puppet fail [09:29:17] PROBLEM - puppet last run on cp3014 is CRITICAL: CRITICAL: Puppet has 33 failures [09:29:17] PROBLEM - puppet last run on mw1213 is CRITICAL: CRITICAL: puppet fail [09:29:17] PROBLEM - puppet last run on db1051 is CRITICAL: CRITICAL: Puppet has 22 failures [09:29:27] PROBLEM - puppet last run on mw1208 is CRITICAL: CRITICAL: puppet fail [09:29:27] PROBLEM - puppet last run on mw1129 is CRITICAL: CRITICAL: Puppet has 29 failures [09:29:28] PROBLEM - puppet last run on mw1177 is CRITICAL: CRITICAL: Puppet has 87 failures [09:29:28] PROBLEM - puppet last run on db1016 is CRITICAL: CRITICAL: puppet fail [09:29:28] PROBLEM - puppet last run on analytics1010 is CRITICAL: CRITICAL: puppet fail [09:29:28] PROBLEM - puppet last run on mw1144 is CRITICAL: CRITICAL: Puppet has 45 failures [09:29:28] PROBLEM - puppet last run on db1067 is CRITICAL: CRITICAL: puppet fail [09:29:28] PROBLEM - puppet last run on mw1249 is CRITICAL: CRITICAL: puppet fail [09:29:28] PROBLEM - puppet last run on db1003 is CRITICAL: CRITICAL: puppet fail [09:29:37] PROBLEM - puppet last run on mw1011 is CRITICAL: CRITICAL: Puppet has 20 failures [09:29:37] PROBLEM - puppet last run on virt1003 is CRITICAL: CRITICAL: puppet fail [09:29:37] PROBLEM - puppet last run on mw1076 is CRITICAL: CRITICAL: puppet fail [09:29:38] PROBLEM - puppet last run on mw1002 is CRITICAL: CRITICAL: Puppet has 53 failures [09:29:38] PROBLEM - puppet last run on gallium is CRITICAL: CRITICAL: puppet fail [09:29:38] PROBLEM - puppet last run on mw1149 is CRITICAL: CRITICAL: puppet fail [09:29:38] PROBLEM - puppet last run on db1043 is CRITICAL: CRITICAL: Puppet has 19 failures [09:29:38] PROBLEM - puppet last run on labstore1001 is CRITICAL: CRITICAL: puppet fail [09:29:47] PROBLEM - puppet last run on mc1012 is CRITICAL: CRITICAL: puppet fail [09:29:47] PROBLEM - puppet last run on db2038 is CRITICAL: CRITICAL: puppet fail [09:29:48] PROBLEM - puppet last run on db2036 is CRITICAL: CRITICAL: Puppet has 11 failures [09:29:48] PROBLEM - puppet last run on virt1001 is CRITICAL: CRITICAL: Puppet has 18 failures [09:29:48] PROBLEM - puppet last run on lithium is CRITICAL: CRITICAL: puppet fail [09:29:48] PROBLEM - puppet last run on dbproxy1001 is CRITICAL: CRITICAL: Puppet has 23 failures [09:29:48] PROBLEM - puppet last run on db2029 is CRITICAL: CRITICAL: Puppet has 9 failures [09:29:49] PROBLEM - puppet last run on ms-be2011 is CRITICAL: CRITICAL: Puppet has 12 failures [09:29:49] PROBLEM - puppet last run on db2007 is CRITICAL: CRITICAL: puppet fail [09:29:50] PROBLEM - puppet last run on snapshot1001 is CRITICAL: CRITICAL: puppet fail [09:29:50] PROBLEM - puppet last run on analytics1026 is CRITICAL: CRITICAL: puppet fail [09:29:57] PROBLEM - puppet last run on cp1058 is CRITICAL: CRITICAL: Puppet has 13 failures [09:29:57] PROBLEM - puppet last run on wtp1012 is CRITICAL: CRITICAL: Puppet has 29 failures [09:29:58] PROBLEM - puppet last run on wtp1005 is CRITICAL: CRITICAL: Puppet has 17 failures [09:29:58] PROBLEM - puppet last run on mw1202 is CRITICAL: CRITICAL: puppet fail [09:30:07] PROBLEM - puppet last run on mw1126 is CRITICAL: CRITICAL: Puppet has 79 failures [09:30:07] PROBLEM - puppet last run on mw1044 is CRITICAL: CRITICAL: Puppet has 82 failures [09:30:08] PROBLEM - puppet last run on mw1237 is CRITICAL: CRITICAL: puppet fail [09:30:08] PROBLEM - puppet last run on mw1195 is CRITICAL: CRITICAL: puppet fail [09:30:09] PROBLEM - puppet last run on antimony is CRITICAL: CRITICAL: Puppet has 25 failures [09:30:17] PROBLEM - puppet last run on virt1004 is CRITICAL: CRITICAL: Puppet has 11 failures [09:30:17] PROBLEM - puppet last run on dataset1001 is CRITICAL: CRITICAL: Puppet has 27 failures [09:30:17] PROBLEM - puppet last run on mw1055 is CRITICAL: CRITICAL: Puppet has 74 failures [09:30:17] PROBLEM - puppet last run on db2042 is CRITICAL: CRITICAL: puppet fail [09:30:18] PROBLEM - puppet last run on nembus is CRITICAL: CRITICAL: Puppet has 25 failures [09:30:18] PROBLEM - puppet last run on cp4001 is CRITICAL: CRITICAL: puppet fail [09:30:18] PROBLEM - puppet last run on analytics1038 is CRITICAL: CRITICAL: puppet fail [09:30:18] PROBLEM - puppet last run on db1052 is CRITICAL: CRITICAL: Puppet has 13 failures [09:30:19] PROBLEM - puppet last run on analytics1016 is CRITICAL: CRITICAL: Puppet has 20 failures [09:30:19] PROBLEM - puppet last run on cp4005 is CRITICAL: CRITICAL: puppet fail [09:30:27] PROBLEM - puppet last run on gadolinium is CRITICAL: CRITICAL: puppet fail [09:30:27] PROBLEM - puppet last run on ms-be3001 is CRITICAL: CRITICAL: puppet fail [09:30:27] PROBLEM - puppet last run on amssq46 is CRITICAL: CRITICAL: Puppet has 16 failures [09:30:28] PROBLEM - puppet last run on cp3010 is CRITICAL: CRITICAL: puppet fail [09:30:28] PROBLEM - puppet last run on es1007 is CRITICAL: CRITICAL: puppet fail [09:30:28] PROBLEM - puppet last run on amssq47 is CRITICAL: CRITICAL: Puppet has 27 failures [09:30:28] PROBLEM - puppet last run on snapshot1002 is CRITICAL: CRITICAL: Puppet has 53 failures [09:30:28] PROBLEM - puppet last run on pc1002 is CRITICAL: CRITICAL: Puppet has 17 failures [09:30:37] PROBLEM - puppet last run on db1060 is CRITICAL: CRITICAL: puppet fail [09:30:38] PROBLEM - puppet last run on elastic1019 is CRITICAL: CRITICAL: puppet fail [09:30:38] PROBLEM - puppet last run on mw1162 is CRITICAL: CRITICAL: puppet fail [09:30:38] PROBLEM - puppet last run on mw1049 is CRITICAL: CRITICAL: puppet fail [09:30:38] PROBLEM - puppet last run on analytics1013 is CRITICAL: CRITICAL: Puppet has 20 failures [09:30:47] PROBLEM - puppet last run on db1039 is CRITICAL: CRITICAL: puppet fail [09:30:47] PROBLEM - puppet last run on mw1206 is CRITICAL: CRITICAL: Puppet has 28 failures [09:30:47] PROBLEM - puppet last run on polonium is CRITICAL: CRITICAL: puppet fail [09:30:47] PROBLEM - puppet last run on mw1014 is CRITICAL: CRITICAL: puppet fail [09:30:47] PROBLEM - puppet last run on stat1003 is CRITICAL: CRITICAL: Puppet has 54 failures [09:30:48] PROBLEM - puppet last run on mw1054 is CRITICAL: CRITICAL: Puppet has 70 failures [09:30:48] PROBLEM - puppet last run on mw1111 is CRITICAL: CRITICAL: puppet fail [09:30:48] PROBLEM - puppet last run on elastic1006 is CRITICAL: CRITICAL: puppet fail [09:30:49] PROBLEM - puppet last run on mw1238 is CRITICAL: CRITICAL: Puppet has 74 failures [09:30:58] PROBLEM - puppet last run on lvs2006 is CRITICAL: CRITICAL: puppet fail [09:30:58] PROBLEM - puppet last run on mw1190 is CRITICAL: CRITICAL: puppet fail [09:30:58] PROBLEM - puppet last run on sodium is CRITICAL: CRITICAL: puppet fail [09:30:58] PROBLEM - puppet last run on mc1005 is CRITICAL: CRITICAL: Puppet has 24 failures [09:30:59] PROBLEM - puppet last run on install2001 is CRITICAL: CRITICAL: Puppet has 22 failures [09:31:07] PROBLEM - puppet last run on mw1159 is CRITICAL: CRITICAL: puppet fail [09:31:07] PROBLEM - puppet last run on cp1050 is CRITICAL: CRITICAL: Puppet has 13 failures [09:31:07] PROBLEM - puppet last run on oxygen is CRITICAL: CRITICAL: Puppet has 18 failures [09:31:07] PROBLEM - puppet last run on labmon1001 is CRITICAL: CRITICAL: Puppet has 24 failures [09:31:07] PROBLEM - puppet last run on lvs4003 is CRITICAL: CRITICAL: Puppet has 13 failures [09:31:08] PROBLEM - puppet last run on cp4019 is CRITICAL: CRITICAL: Puppet has 14 failures [09:31:08] PROBLEM - puppet last run on bast4001 is CRITICAL: CRITICAL: Puppet has 26 failures [09:31:09] PROBLEM - puppet last run on db1048 is CRITICAL: CRITICAL: puppet fail [09:31:09] PROBLEM - puppet last run on db1071 is CRITICAL: CRITICAL: Puppet has 23 failures [09:31:17] PROBLEM - puppet last run on mw1098 is CRITICAL: CRITICAL: Puppet has 37 failures [09:31:17] PROBLEM - puppet last run on mw1079 is CRITICAL: CRITICAL: Puppet has 30 failures [09:31:18] PROBLEM - puppet last run on plutonium is CRITICAL: CRITICAL: puppet fail [09:31:18] PROBLEM - puppet last run on snapshot1004 is CRITICAL: CRITICAL: Puppet has 73 failures [09:31:18] PROBLEM - puppet last run on osmium is CRITICAL: CRITICAL: puppet fail [09:31:18] PROBLEM - puppet last run on db2016 is CRITICAL: CRITICAL: Puppet has 11 failures [09:31:18] PROBLEM - puppet last run on mw1227 is CRITICAL: CRITICAL: Puppet has 71 failures [09:31:18] PROBLEM - puppet last run on wtp1018 is CRITICAL: CRITICAL: puppet fail [09:31:18] PROBLEM - puppet last run on mw1151 is CRITICAL: CRITICAL: Puppet has 69 failures [09:31:27] PROBLEM - puppet last run on mw1133 is CRITICAL: CRITICAL: puppet fail [09:31:27] PROBLEM - puppet last run on cp1048 is CRITICAL: CRITICAL: puppet fail [09:31:28] PROBLEM - puppet last run on db1004 is CRITICAL: CRITICAL: puppet fail [09:31:28] PROBLEM - puppet last run on cp1046 is CRITICAL: CRITICAL: Puppet has 14 failures [09:31:28] PROBLEM - puppet last run on db1036 is CRITICAL: CRITICAL: puppet fail [09:31:28] PROBLEM - puppet last run on es2004 is CRITICAL: CRITICAL: puppet fail [09:31:28] PROBLEM - puppet last run on mw1247 is CRITICAL: CRITICAL: Puppet has 80 failures [09:31:28] PROBLEM - puppet last run on ms-be2001 is CRITICAL: CRITICAL: puppet fail [09:31:37] PROBLEM - puppet last run on mw1030 is CRITICAL: CRITICAL: puppet fail [09:31:38] PROBLEM - puppet last run on lvs3004 is CRITICAL: CRITICAL: Puppet has 18 failures [09:31:38] PROBLEM - puppet last run on amssq40 is CRITICAL: CRITICAL: Puppet has 27 failures [09:31:38] PROBLEM - puppet last run on amssq41 is CRITICAL: CRITICAL: puppet fail [09:31:38] PROBLEM - puppet last run on ms-fe3002 is CRITICAL: CRITICAL: Puppet has 15 failures [09:31:38] PROBLEM - puppet last run on mw1183 is CRITICAL: CRITICAL: puppet fail [09:31:38] PROBLEM - puppet last run on mw1258 is CRITICAL: CRITICAL: puppet fail [09:31:39] PROBLEM - puppet last run on argon is CRITICAL: CRITICAL: puppet fail [09:31:39] PROBLEM - puppet last run on mw1165 is CRITICAL: CRITICAL: puppet fail [09:31:47] PROBLEM - puppet last run on analytics1023 is CRITICAL: CRITICAL: puppet fail [09:31:47] PROBLEM - puppet last run on cp1063 is CRITICAL: CRITICAL: puppet fail [09:31:47] PROBLEM - puppet last run on mc1014 is CRITICAL: CRITICAL: Puppet has 11 failures [09:31:48] PROBLEM - puppet last run on db1026 is CRITICAL: CRITICAL: Puppet has 22 failures [09:31:48] PROBLEM - puppet last run on mw1181 is CRITICAL: CRITICAL: Puppet has 30 failures [09:31:48] PROBLEM - puppet last run on rubidium is CRITICAL: CRITICAL: Puppet has 26 failures [09:31:57] PROBLEM - puppet last run on mw1050 is CRITICAL: CRITICAL: puppet fail [09:31:57] PROBLEM - puppet last run on stat1002 is CRITICAL: CRITICAL: Puppet has 15 failures [09:31:57] PROBLEM - puppet last run on mw1084 is CRITICAL: CRITICAL: Puppet has 32 failures [09:31:57] PROBLEM - puppet last run on ms-be1012 is CRITICAL: CRITICAL: Puppet has 13 failures [09:31:57] PROBLEM - puppet last run on tungsten is CRITICAL: CRITICAL: Puppet has 17 failures [09:31:58] PROBLEM - puppet last run on ms-be1008 is CRITICAL: CRITICAL: puppet fail [09:31:58] PROBLEM - puppet last run on db1069 is CRITICAL: CRITICAL: puppet fail [09:31:58] PROBLEM - puppet last run on elastic1024 is CRITICAL: CRITICAL: Puppet has 14 failures [09:31:58] PROBLEM - puppet last run on labsdb1006 is CRITICAL: CRITICAL: Puppet has 18 failures [09:31:59] PROBLEM - puppet last run on cp1062 is CRITICAL: CRITICAL: Puppet has 17 failures [09:31:59] PROBLEM - puppet last run on analytics1022 is CRITICAL: CRITICAL: Puppet has 9 failures [09:32:00] PROBLEM - puppet last run on rdb1001 is CRITICAL: CRITICAL: Puppet has 9 failures [09:32:01] PROBLEM - puppet last run on mw1074 is CRITICAL: CRITICAL: puppet fail [09:32:01] PROBLEM - puppet last run on mw1051 is CRITICAL: CRITICAL: Puppet has 38 failures [09:32:01] PROBLEM - puppet last run on mw1125 is CRITICAL: CRITICAL: Puppet has 37 failures [09:32:07] PROBLEM - puppet last run on mw1180 is CRITICAL: CRITICAL: Puppet has 41 failures [09:32:07] PROBLEM - puppet last run on virt1010 is CRITICAL: CRITICAL: puppet fail [09:32:07] PROBLEM - puppet last run on mw1168 is CRITICAL: CRITICAL: Puppet has 35 failures [09:32:08] PROBLEM - puppet last run on db2037 is CRITICAL: CRITICAL: puppet fail [09:32:08] PROBLEM - puppet last run on db2023 is CRITICAL: CRITICAL: Puppet has 20 failures [09:32:08] PROBLEM - puppet last run on ms-be2012 is CRITICAL: CRITICAL: puppet fail [09:32:08] PROBLEM - puppet last run on db2004 is CRITICAL: CRITICAL: Puppet has 11 failures [09:32:08] PROBLEM - puppet last run on mw1248 is CRITICAL: CRITICAL: puppet fail [09:32:17] PROBLEM - puppet last run on acamar is CRITICAL: CRITICAL: puppet fail [09:32:18] PROBLEM - puppet last run on thallium is CRITICAL: CRITICAL: puppet fail [09:32:18] PROBLEM - puppet last run on mw1210 is CRITICAL: CRITICAL: puppet fail [09:32:18] PROBLEM - puppet last run on cp4018 is CRITICAL: CRITICAL: Puppet has 13 failures [09:32:18] PROBLEM - puppet last run on mw1198 is CRITICAL: CRITICAL: puppet fail [09:32:27] PROBLEM - puppet last run on virt1007 is CRITICAL: CRITICAL: Puppet has 10 failures [09:32:27] PROBLEM - puppet last run on mw1034 is CRITICAL: CRITICAL: Puppet has 64 failures [09:32:28] PROBLEM - puppet last run on amssq36 is CRITICAL: CRITICAL: Puppet has 31 failures [09:32:28] PROBLEM - puppet last run on amssq51 is CRITICAL: CRITICAL: Puppet has 13 failures [09:32:28] PROBLEM - puppet last run on mw1146 is CRITICAL: CRITICAL: Puppet has 30 failures [09:32:28] PROBLEM - puppet last run on db1020 is CRITICAL: CRITICAL: Puppet has 24 failures [09:32:28] PROBLEM - puppet last run on hafnium is CRITICAL: CRITICAL: puppet fail [09:32:38] PROBLEM - puppet last run on cp1038 is CRITICAL: CRITICAL: Puppet has 34 failures [09:32:38] PROBLEM - puppet last run on es2007 is CRITICAL: CRITICAL: Puppet has 13 failures [09:32:38] PROBLEM - puppet last run on mw1004 is CRITICAL: CRITICAL: Puppet has 22 failures [09:32:38] PROBLEM - puppet last run on ms-be1007 is CRITICAL: CRITICAL: Puppet has 13 failures [09:32:38] PROBLEM - puppet last run on mw1056 is CRITICAL: CRITICAL: Puppet has 37 failures [09:32:38] PROBLEM - puppet last run on analytics1037 is CRITICAL: CRITICAL: puppet fail [09:32:39] PROBLEM - puppet last run on wtp1023 is CRITICAL: CRITICAL: Puppet has 17 failures [09:32:39] PROBLEM - puppet last run on lvs1001 is CRITICAL: CRITICAL: puppet fail [09:32:39] PROBLEM - puppet last run on db2001 is CRITICAL: CRITICAL: Puppet has 10 failures [09:32:39] PROBLEM - puppet last run on ms-be2005 is CRITICAL: CRITICAL: puppet fail [09:32:48] PROBLEM - puppet last run on mw1097 is CRITICAL: CRITICAL: puppet fail [09:32:48] PROBLEM - puppet last run on ms-be2007 is CRITICAL: CRITICAL: Puppet has 15 failures [09:32:48] PROBLEM - puppet last run on ms-be2008 is CRITICAL: CRITICAL: Puppet has 13 failures [09:32:49] PROBLEM - puppet last run on amssq56 is CRITICAL: CRITICAL: Puppet has 19 failures [09:32:49] PROBLEM - puppet last run on rhenium is CRITICAL: CRITICAL: Puppet has 11 failures [09:32:57] RECOVERY - puppetmaster https on palladium is OK: HTTP OK: Status line output matched 400 - 335 bytes in 0.021 second response time [09:32:58] PROBLEM - puppet last run on mw1053 is CRITICAL: CRITICAL: puppet fail [09:32:58] PROBLEM - puppet last run on mw1156 is CRITICAL: CRITICAL: Puppet has 39 failures [09:32:58] PROBLEM - puppet last run on db1062 is CRITICAL: CRITICAL: puppet fail [09:32:58] PROBLEM - puppet last run on mw1171 is CRITICAL: CRITICAL: puppet fail [09:32:58] PROBLEM - puppet last run on analytics1032 is CRITICAL: CRITICAL: puppet fail [09:32:58] PROBLEM - puppet last run on elastic1011 is CRITICAL: CRITICAL: puppet fail [09:33:07] PROBLEM - puppet last run on mw1148 is CRITICAL: CRITICAL: puppet fail [09:33:07] PROBLEM - puppet last run on mw1163 is CRITICAL: CRITICAL: Puppet has 68 failures [09:33:07] PROBLEM - puppet last run on wtp1004 is CRITICAL: CRITICAL: Puppet has 15 failures [09:33:08] PROBLEM - puppet last run on mw1081 is CRITICAL: CRITICAL: Puppet has 42 failures [09:33:08] PROBLEM - puppet last run on titanium is CRITICAL: CRITICAL: Puppet has 15 failures [09:33:08] PROBLEM - puppet last run on db1055 is CRITICAL: CRITICAL: Puppet has 9 failures [09:33:08] PROBLEM - puppet last run on elastic1014 is CRITICAL: CRITICAL: Puppet has 24 failures [09:33:08] PROBLEM - puppet last run on mc1001 is CRITICAL: CRITICAL: puppet fail [09:33:08] PROBLEM - puppet last run on es1002 is CRITICAL: CRITICAL: Puppet has 10 failures [09:33:09] PROBLEM - puppet last run on mw1116 is CRITICAL: CRITICAL: puppet fail [09:33:10] PROBLEM - puppet last run on elastic1015 is CRITICAL: CRITICAL: Puppet has 26 failures [09:33:17] PROBLEM - puppet last run on labsdb1004 is CRITICAL: CRITICAL: puppet fail [09:33:17] PROBLEM - puppet last run on mc1013 is CRITICAL: CRITICAL: puppet fail [09:33:18] PROBLEM - puppet last run on mc1007 is CRITICAL: CRITICAL: puppet fail [09:33:18] PROBLEM - puppet last run on mw1057 is CRITICAL: CRITICAL: Puppet has 37 failures [09:33:27] PROBLEM - puppet last run on wtp1022 is CRITICAL: CRITICAL: puppet fail [09:33:27] PROBLEM - puppet last run on haedus is CRITICAL: CRITICAL: Puppet has 27 failures [09:33:28] PROBLEM - puppet last run on wtp1011 is CRITICAL: CRITICAL: puppet fail [09:33:37] PROBLEM - puppet last run on mw1188 is CRITICAL: CRITICAL: Puppet has 70 failures [09:33:38] PROBLEM - puppet last run on cp3012 is CRITICAL: CRITICAL: puppet fail [09:33:38] PROBLEM - puppet last run on amslvs3 is CRITICAL: CRITICAL: puppet fail [09:33:38] PROBLEM - puppet last run on mw1029 is CRITICAL: CRITICAL: puppet fail [09:33:38] PROBLEM - puppet last run on elastic1005 is CRITICAL: CRITICAL: puppet fail [09:33:48] PROBLEM - puppet last run on db1001 is CRITICAL: CRITICAL: Puppet has 13 failures [09:33:48] PROBLEM - puppet last run on elastic1002 is CRITICAL: CRITICAL: puppet fail [09:33:48] PROBLEM - puppet last run on db2003 is CRITICAL: CRITICAL: puppet fail [09:33:49] PROBLEM - puppet last run on mw1212 is CRITICAL: CRITICAL: Puppet has 64 failures [09:33:49] PROBLEM - puppet last run on mw1032 is CRITICAL: CRITICAL: puppet fail [09:33:49] PROBLEM - puppet last run on rcs1002 is CRITICAL: CRITICAL: puppet fail [09:33:49] PROBLEM - puppet last run on wtp1002 is CRITICAL: CRITICAL: Puppet has 15 failures [09:33:49] PROBLEM - puppet last run on db1054 is CRITICAL: CRITICAL: puppet fail [09:33:57] PROBLEM - puppet last run on cp1060 is CRITICAL: CRITICAL: Puppet has 20 failures [09:33:58] PROBLEM - puppet last run on ms-be1009 is CRITICAL: CRITICAL: Puppet has 26 failures [09:33:58] PROBLEM - puppet last run on amssq38 is CRITICAL: CRITICAL: puppet fail [09:33:58] PROBLEM - puppet last run on cp3009 is CRITICAL: CRITICAL: Puppet has 15 failures [09:33:58] PROBLEM - puppet last run on hooft is CRITICAL: CRITICAL: Puppet has 57 failures [09:33:59] PROBLEM - puppet last run on amssq62 is CRITICAL: CRITICAL: Puppet has 30 failures [09:34:07] PROBLEM - puppet last run on rdb1003 is CRITICAL: CRITICAL: puppet fail [09:34:17] PROBLEM - puppet last run on mw1087 is CRITICAL: CRITICAL: Puppet has 74 failures [09:34:17] PROBLEM - puppet last run on elastic1023 is CRITICAL: CRITICAL: puppet fail [09:34:17] PROBLEM - puppet last run on mw1022 is CRITICAL: CRITICAL: puppet fail [09:34:17] PROBLEM - puppet last run on mw1023 is CRITICAL: CRITICAL: Puppet has 32 failures [09:34:17] PROBLEM - puppet last run on wtp1015 is CRITICAL: CRITICAL: Puppet has 14 failures [09:34:27] PROBLEM - puppet last run on mw1186 is CRITICAL: CRITICAL: puppet fail [09:34:27] PROBLEM - puppet last run on virt1000 is CRITICAL: CRITICAL: puppet fail [09:34:28] PROBLEM - puppet last run on db1057 is CRITICAL: CRITICAL: Puppet has 20 failures [09:34:28] PROBLEM - puppet last run on mw1225 is CRITICAL: CRITICAL: puppet fail [09:34:28] PROBLEM - puppet last run on lvs2003 is CRITICAL: CRITICAL: puppet fail [09:34:28] PROBLEM - puppet last run on pc1003 is CRITICAL: CRITICAL: Puppet has 22 failures [09:34:37] PROBLEM - puppet last run on mw1239 is CRITICAL: CRITICAL: puppet fail [09:34:37] PROBLEM - puppet last run on wtp1013 is CRITICAL: CRITICAL: Puppet has 27 failures [09:34:37] PROBLEM - puppet last run on db2011 is CRITICAL: CRITICAL: Puppet has 23 failures [09:34:37] PROBLEM - puppet last run on protactinium is CRITICAL: CRITICAL: Puppet has 15 failures [09:34:38] PROBLEM - puppet last run on rdb1002 is CRITICAL: CRITICAL: Puppet has 20 failures [09:34:38] PROBLEM - puppet last run on nitrogen is CRITICAL: CRITICAL: Puppet has 21 failures [09:34:38] PROBLEM - puppet last run on db1063 is CRITICAL: CRITICAL: Puppet has 21 failures [09:34:38] PROBLEM - puppet last run on db1064 is CRITICAL: CRITICAL: puppet fail [09:34:38] PROBLEM - puppet last run on mw1243 is CRITICAL: CRITICAL: Puppet has 30 failures [09:34:39] PROBLEM - puppet last run on mw1024 is CRITICAL: CRITICAL: puppet fail [09:34:48] PROBLEM - puppet last run on mw1201 is CRITICAL: CRITICAL: puppet fail [09:34:57] PROBLEM - puppet last run on db1070 is CRITICAL: CRITICAL: Puppet has 22 failures [09:34:57] PROBLEM - puppet last run on mw1077 is CRITICAL: CRITICAL: puppet fail [09:34:57] PROBLEM - puppet last run on mw1185 is CRITICAL: CRITICAL: puppet fail [09:34:57] PROBLEM - puppet last run on mw1122 is CRITICAL: CRITICAL: puppet fail [09:34:58] PROBLEM - puppet last run on wtp1003 is CRITICAL: CRITICAL: puppet fail [09:34:58] PROBLEM - puppet last run on rcs1001 is CRITICAL: CRITICAL: Puppet has 26 failures [09:34:58] PROBLEM - puppet last run on ocg1003 is CRITICAL: CRITICAL: puppet fail [09:34:58] PROBLEM - puppet last run on analytics1014 is CRITICAL: CRITICAL: Puppet has 10 failures [09:35:07] PROBLEM - puppet last run on mw1043 is CRITICAL: CRITICAL: puppet fail [09:35:08] PROBLEM - puppet last run on analytics1011 is CRITICAL: CRITICAL: Puppet has 11 failures [09:35:08] PROBLEM - puppet last run on ms-fe3001 is CRITICAL: CRITICAL: Puppet has 24 failures [09:35:17] PROBLEM - puppet last run on db1033 is CRITICAL: CRITICAL: Puppet has 15 failures [09:35:17] PROBLEM - puppet last run on mw1219 is CRITICAL: CRITICAL: puppet fail [09:35:17] PROBLEM - puppet last run on mw1167 is CRITICAL: CRITICAL: Puppet has 68 failures [09:35:18] PROBLEM - puppet last run on mw1142 is CRITICAL: CRITICAL: puppet fail [09:35:27] PROBLEM - puppet last run on db1061 is CRITICAL: CRITICAL: puppet fail [09:35:27] PROBLEM - puppet last run on db1038 is CRITICAL: CRITICAL: Puppet has 22 failures [09:35:27] PROBLEM - puppet last run on db1030 is CRITICAL: CRITICAL: Puppet has 10 failures [09:35:28] PROBLEM - puppet last run on mw1033 is CRITICAL: CRITICAL: puppet fail [09:35:28] PROBLEM - puppet last run on mw1105 is CRITICAL: CRITICAL: Puppet has 76 failures [09:35:28] PROBLEM - puppet last run on elastic1017 is CRITICAL: CRITICAL: puppet fail [09:35:28] PROBLEM - puppet last run on db1044 is CRITICAL: CRITICAL: puppet fail [09:35:28] PROBLEM - puppet last run on analytics1028 is CRITICAL: CRITICAL: Puppet has 11 failures [09:35:28] PROBLEM - puppet last run on mw1229 is CRITICAL: CRITICAL: Puppet has 31 failures [09:35:37] PROBLEM - puppet last run on wtp1007 is CRITICAL: CRITICAL: puppet fail [09:35:37] PROBLEM - puppet last run on db1006 is CRITICAL: CRITICAL: puppet fail [09:35:38] PROBLEM - puppet last run on mercury is CRITICAL: CRITICAL: Puppet has 20 failures [09:35:38] PROBLEM - puppet last run on iodine is CRITICAL: CRITICAL: Puppet has 19 failures [09:35:48] PROBLEM - puppet last run on mw1152 is CRITICAL: CRITICAL: puppet fail [09:35:48] PROBLEM - puppet last run on db2012 is CRITICAL: CRITICAL: puppet fail [09:35:48] PROBLEM - puppet last run on db1047 is CRITICAL: CRITICAL: Puppet has 9 failures [09:35:48] PROBLEM - puppet last run on elastic1020 is CRITICAL: CRITICAL: puppet fail [09:35:48] PROBLEM - puppet last run on d-i-test is CRITICAL: CRITICAL: puppet fail [09:35:48] PROBLEM - puppet last run on cp1052 is CRITICAL: CRITICAL: puppet fail [09:35:57] PROBLEM - puppet last run on mw1001 is CRITICAL: CRITICAL: puppet fail [09:35:57] PROBLEM - puppet last run on mw1231 is CRITICAL: CRITICAL: puppet fail [09:35:57] PROBLEM - puppet last run on ms-be1011 is CRITICAL: CRITICAL: Puppet has 19 failures [09:35:57] PROBLEM - puppet last run on analytics1027 is CRITICAL: CRITICAL: puppet fail [09:35:58] PROBLEM - puppet last run on cp1044 is CRITICAL: CRITICAL: puppet fail [09:35:58] PROBLEM - puppet last run on db1011 is CRITICAL: CRITICAL: Puppet has 16 failures [09:35:58] PROBLEM - puppet last run on mw1108 is CRITICAL: CRITICAL: Puppet has 79 failures [09:36:07] PROBLEM - puppet last run on copper is CRITICAL: CRITICAL: puppet fail [09:36:07] PROBLEM - puppet last run on mw1209 is CRITICAL: CRITICAL: Puppet has 74 failures [09:36:07] PROBLEM - puppet last run on cp1037 is CRITICAL: CRITICAL: Puppet has 22 failures [09:36:08] PROBLEM - puppet last run on iridium is CRITICAL: CRITICAL: puppet fail [09:36:08] PROBLEM - puppet last run on analytics1004 is CRITICAL: CRITICAL: Puppet has 8 failures [09:36:08] PROBLEM - puppet last run on mw1091 is CRITICAL: CRITICAL: puppet fail [09:36:08] PROBLEM - puppet last run on ms-fe2002 is CRITICAL: CRITICAL: puppet fail [09:36:17] PROBLEM - puppet last run on es1010 is CRITICAL: CRITICAL: Puppet has 23 failures [09:36:18] PROBLEM - puppet last run on mw1016 is CRITICAL: CRITICAL: puppet fail [09:36:18] PROBLEM - puppet last run on mw1121 is CRITICAL: CRITICAL: Puppet has 31 failures [09:36:27] PROBLEM - puppet last run on cp3004 is CRITICAL: CRITICAL: Puppet has 29 failures [09:36:27] PROBLEM - puppet last run on cp1067 is CRITICAL: CRITICAL: puppet fail [09:36:27] PROBLEM - puppet last run on mw1093 is CRITICAL: CRITICAL: Puppet has 72 failures [09:36:37] PROBLEM - puppet last run on dbstore1002 is CRITICAL: CRITICAL: puppet fail [09:36:37] PROBLEM - puppet last run on fluorine is CRITICAL: CRITICAL: Puppet has 68 failures [09:36:37] PROBLEM - puppet last run on calcium is CRITICAL: CRITICAL: puppet fail [09:36:38] PROBLEM - puppet last run on mw1223 is CRITICAL: CRITICAL: puppet fail [09:36:58] PROBLEM - puppet last run on db2028 is CRITICAL: CRITICAL: Puppet has 16 failures [09:37:17] PROBLEM - puppet last run on rbf1001 is CRITICAL: CRITICAL: Puppet has 9 failures [09:37:38] PROBLEM - puppetmaster https on palladium is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 8140: HTTP/1.1 500 Internal Server Error [09:37:47] PROBLEM - puppet last run on mw1215 is CRITICAL: CRITICAL: Puppet has 27 failures [09:37:47] PROBLEM - puppet last run on mw1220 is CRITICAL: CRITICAL: Puppet has 33 failures [09:37:47] PROBLEM - puppet last run on mw1010 is CRITICAL: CRITICAL: Puppet has 63 failures [09:37:47] PROBLEM - puppet last run on ms-be1002 is CRITICAL: CRITICAL: Puppet has 22 failures [09:38:57] RECOVERY - puppetmaster https on palladium is OK: HTTP OK: Status line output matched 400 - 335 bytes in 0.030 second response time [09:44:49] PROBLEM - puppetmaster https on palladium is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 8140: HTTP/1.1 500 Internal Server Error [09:45:59] RECOVERY - puppetmaster https on palladium is OK: HTTP OK: Status line output matched 400 - 335 bytes in 0.064 second response time [09:48:48] RECOVERY - puppet last run on sodium is OK: OK: Puppet is currently enabled, last run 57 seconds ago with 0 failures [09:50:38] PROBLEM - puppet last run on nescio is CRITICAL: CRITICAL: puppet fail [10:00:08] PROBLEM - puppetmaster https on palladium is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 8140: HTTP/1.1 500 Internal Server Error [10:01:18] RECOVERY - puppetmaster https on palladium is OK: HTTP OK: Status line output matched 400 - 335 bytes in 0.020 second response time [10:04:48] PROBLEM - puppetmaster https on palladium is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 8140: HTTP/1.1 500 Internal Server Error [10:05:58] RECOVERY - puppetmaster https on palladium is OK: HTTP OK: Status line output matched 400 - 335 bytes in 0.052 second response time [10:06:07] PROBLEM - puppet last run on ms1004 is CRITICAL: CRITICAL: puppet fail [10:11:07] PROBLEM - puppet last run on sodium is CRITICAL: CRITICAL: puppet fail [10:24:58] RECOVERY - puppet last run on ms1004 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [10:28:18] RECOVERY - puppet last run on nescio is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [10:58:48] PROBLEM - puppetmaster https on palladium is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 8140: HTTP/1.1 500 Internal Server Error [11:00:07] RECOVERY - puppetmaster https on palladium is OK: HTTP OK: Status line output matched 400 - 335 bytes in 0.017 second response time [11:06:08] PROBLEM - puppet last run on ms1004 is CRITICAL: CRITICAL: puppet fail [11:23:48] RECOVERY - puppet last run on ms1004 is OK: OK: Puppet is currently enabled, last run 3 seconds ago with 0 failures [11:28:58] RECOVERY - puppet last run on sodium is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [11:29:48] PROBLEM - puppet last run on nescio is CRITICAL: CRITICAL: puppet fail [11:40:08] PROBLEM - puppetmaster https on palladium is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 8140: HTTP/1.1 500 Internal Server Error [11:41:18] RECOVERY - puppetmaster https on palladium is OK: HTTP OK: Status line output matched 400 - 335 bytes in 0.030 second response time [11:50:17] PROBLEM - puppet last run on sodium is CRITICAL: CRITICAL: puppet fail [12:26:18] PROBLEM - puppet last run on ms1004 is CRITICAL: CRITICAL: puppet fail [12:29:17] RECOVERY - puppet last run on sodium is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [12:44:17] RECOVERY - puppet last run on ms1004 is OK: OK: Puppet is currently enabled, last run 14 seconds ago with 0 failures [12:48:08] RECOVERY - puppet last run on nescio is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [12:50:37] PROBLEM - puppet last run on sodium is CRITICAL: CRITICAL: puppet fail [13:09:27] RECOVERY - puppet last run on sodium is OK: OK: Puppet is currently enabled, last run 57 seconds ago with 0 failures [13:09:27] PROBLEM - puppet last run on nescio is CRITICAL: CRITICAL: puppet fail [13:14:37] PROBLEM - puppetmaster https on palladium is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 8140: HTTP/1.1 500 Internal Server Error [13:15:47] RECOVERY - puppetmaster https on palladium is OK: HTTP OK: Status line output matched 400 - 335 bytes in 0.039 second response time [13:26:37] PROBLEM - puppet last run on ms1004 is CRITICAL: CRITICAL: puppet fail [13:28:18] RECOVERY - puppet last run on nescio is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [13:30:28] PROBLEM - puppet last run on sodium is CRITICAL: CRITICAL: puppet fail [13:42:48] PROBLEM - puppetmaster https on palladium is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 8140: HTTP/1.1 500 Internal Server Error [13:44:17] RECOVERY - puppet last run on ms1004 is OK: OK: Puppet is currently enabled, last run 34 seconds ago with 0 failures [13:47:39] RECOVERY - puppetmaster https on palladium is OK: HTTP OK: Status line output matched 400 - 335 bytes in 0.021 second response time [13:48:08] RECOVERY - puppet last run on sodium is OK: OK: Puppet is currently enabled, last run 7 seconds ago with 0 failures [13:51:07] PROBLEM - puppetmaster https on palladium is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 8140: HTTP/1.1 500 Internal Server Error [14:03:58] RECOVERY - puppetmaster https on palladium is OK: HTTP OK: Status line output matched 400 - 335 bytes in 0.037 second response time [14:09:28] PROBLEM - puppet last run on nescio is CRITICAL: CRITICAL: puppet fail [14:10:28] PROBLEM - puppet last run on sodium is CRITICAL: CRITICAL: puppet fail [14:34:48] PROBLEM - puppetmaster https on palladium is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 8140: HTTP/1.1 500 Internal Server Error [14:37:12] RECOVERY - puppetmaster https on palladium is OK: HTTP OK: Status line output matched 400 - 335 bytes in 0.053 second response time [14:41:58] PROBLEM - puppetmaster https on palladium is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 8140: HTTP/1.1 500 Internal Server Error [14:47:58] RECOVERY - puppetmaster https on palladium is OK: HTTP OK: Status line output matched 400 - 335 bytes in 0.042 second response time [14:48:28] RECOVERY - puppet last run on sodium is OK: OK: Puppet is currently enabled, last run 21 seconds ago with 0 failures [14:57:16] ori: YuviPanda: Around? [14:57:17] PROBLEM - puppetmaster https on palladium is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 8140: HTTP/1.1 500 Internal Server Error [14:57:47] <^d> hoo: or.i's probably asleep :p [14:57:57] <^d> He's usually asleep when I wake up, at least. [14:58:27] RECOVERY - puppetmaster https on palladium is OK: HTTP OK: Status line output matched 400 - 335 bytes in 0.044 second response time [14:58:41] meh [15:00:32] Any other root around? [15:01:10] <_joe_> hoo: hi [15:01:13] akosiaris: do you remember if the puppet run was finished with my fix before the script that generates the wikidata json dumps was restarted? [15:01:18] <_joe_> need something? [15:01:36] _joe_: the json dumps are still missing :( [15:01:54] Just start the script, again [15:01:57] PROBLEM - puppetmaster https on palladium is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 8140: HTTP/1.1 500 Internal Server Error [15:02:05] <_joe_> are we sure that is a good idea? [15:02:15] <_joe_> mmmh sorry I need to check palladium first [15:03:12] <_joe_> so, do we know why the dumps are missing? [15:03:17] hoo: did you check if there is anything still running? [15:03:25] <_joe_> sorry I know exactly zero about the dumps [15:03:37] jzerebecki: yep, nothing running [15:03:39] <_joe_> also, did this happen after some puppet change we can revert? [15:03:40] _joe_: last time because the script deleted them after creating a new one [15:03:43] unless it was started as the wrong user [15:04:00] <_joe_> hoo: what's the machine they run on? [15:04:02] nothign running [15:04:05] snapshot1003 [15:04:13] <_joe_> I woke up just now sorry :) [15:04:19] take your time [15:04:30] takes >10h to run anyway [15:05:00] <_joe_> eheh [15:05:28] RECOVERY - puppetmaster https on palladium is OK: HTTP OK: Status line output matched 400 - 335 bytes in 0.033 second response time [15:05:34] <_joe_> so, my point is there is something deleting the scripts right? [15:05:40] <_joe_> the dumps [15:05:50] <_joe_> do we know what that is? [15:05:56] _joe_: nope the script deleted the dump itself, i fixed that by a commmit [15:06:02] or at least thought so [15:06:05] <_joe_> oh ok [15:06:20] there aren't even logs left over [15:06:26] <_joe_> and it probably happened again [15:06:27] so that must have been it [15:06:30] _joe_: akosiaris yesterday started the script in a screen according to the adming log [15:06:49] <_joe_> yeah checking that [15:06:49] PROBLEM - puppet last run on ms1004 is CRITICAL: CRITICAL: puppet fail [15:06:59] <_joe_> as what user does it run? [15:07:25] datasets [15:07:37] RECOVERY - puppet last run on nescio is OK: OK: Puppet is currently enabled, last run 22 seconds ago with 0 failures [15:08:32] <_joe_> ok something funny happened here [15:09:07] <_joe_> there is a leftover screen from jan 19th for a user that is not on the server anymore [15:09:31] what [15:10:28] funky :-O do perhaps puppet logs say anything about removing users? [15:10:37] <_joe_> I am looking at it [15:10:48] <_joe_> but the hotel wifi is really really crappy [15:11:00] jzerebecki: Who started the script yesterday? [15:11:26] hoo: akosiaris [15:11:44] <_joe_> no sign of it btw [15:11:49] <_joe_> so, my proposal is [15:11:58] <_joe_> for now I start the script again [15:12:47] <_joe_> later I'll check with alex what he has done an why do we have that weird screen session opened [15:13:06] _joe_: yes, thank you [15:13:13] <_joe_> I'm not sure starting this script will fix it [15:13:14] * hoo rages about vim [15:13:26] <_joe_> but it's the best I can do right now [15:13:27] how can a syntax highlighting be *so* slow [15:14:00] _joe_: you could start it with debugging in a screen [15:14:55] not sure from head what the equivalent to set -x is when setting it from outside the script [15:15:17] <_joe_> what is the script? [15:15:38] <_joe_> jzerebecki: set -x outside the script will act on the parent bash [15:16:22] _joe_: [15:16:23] hoo@snapshot1003:~$ id -u akosiaris [15:16:23] 3194 [15:16:31] So that's akosiaris's screen [15:16:38] not sure why top doesn't find it [15:16:50] eh ps [15:16:53] tired [15:16:54] <_joe_> ok [15:17:29] ps -O cmd -u akosiaris [15:17:30] looks fine [15:17:34] <_joe_> meh I got the colums wrong [15:17:36] no idea what was going on there [15:17:44] ah ok [15:17:49] <_joe_> yeah i saw that screen [15:18:00] <_joe_> I'm just in need of caffeine [15:18:40] I can relate to that... [15:20:52] <_joe_> ok so [15:21:05] <_joe_> the screen is utterly uninteresting [15:21:20] <_joe_> alex ran the script, which exited without one line of output [15:21:49] <_joe_> with $? being 0 [15:21:55] :P [15:22:02] <_joe_> so, I'll start it again [15:22:12] <_joe_> but I won't expect it to do anything different [15:22:13] But all the debug output should go to the /var/log [15:22:21] well, ok [15:22:31] <_joe_> ? [15:22:52] I'm such a fan of netbook keyboards [15:23:08] Going to tail -f capture the log output to my home, I guess [15:23:38] <_joe_> ok [15:24:28] PROBLEM - puppetmaster https on palladium is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 8140: HTTP/1.1 500 Internal Server Error [15:24:41] <_joe_> hoo: dumpwikidatajson, right? [15:25:08] _joe_: so it is either the puppet run with my fix wasn't finished yet or there is still another bug, so go ahead start it again (there is no way to easily make a bash script output traces without modifying it, so we can start it first) [15:25:10] <_joe_> !log restarted apache on palladium [15:25:16] _joe_: yes that is the script [15:25:17] Logged the message, Master [15:26:08] <_joe_> !log started manually dumpwikidatajson on snapshot1003, in a root-owned screen session [15:26:12] Logged the message, Master [15:26:42] :) [15:27:08] hoo: please test what this outputs :) find /var/log/wikidatadump/ -name 'dumpwikidatajson-*-*.log' -mtime +36 [15:27:58] RECOVERY - puppetmaster https on palladium is OK: HTTP OK: Status line output matched 400 - 335 bytes in 0.035 second response time [15:28:17] <_joe_> guys, I need a shower and breakfast, do you still need me? [15:28:21] <_joe_> else, I'll go [15:28:25] jzerebecki: Nothing [15:28:30] crap [15:28:32] _joe_: No, not now. Thank you [15:28:47] hoo: sorry, reverse that, it is a good thing [15:28:54] hehe, yeah [15:29:09] hoo: is the log there and does it look good? [15:29:14] I'm not capturing the logs, although I doubt they'll be interesting [15:29:15] _joe _: thank you [15:29:19] yep, log is there and good [15:29:41] s/not/now/ [15:31:28] PROBLEM - puppetmaster https on palladium is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 8140: HTTP/1.1 500 Internal Server Error [15:32:38] RECOVERY - puppetmaster https on palladium is OK: HTTP OK: Status line output matched 400 - 335 bytes in 0.048 second response time [15:37:19] (03PS2) 10EBernhardson: Enable wgContentHandlerUseDB on all wikis [mediawiki-config] - 10https://gerrit.wikimedia.org/r/175565 [15:44:48] RECOVERY - puppet last run on ms1004 is OK: OK: Puppet is currently enabled, last run 53 seconds ago with 0 failures [15:49:17] PROBLEM - puppet last run on nescio is CRITICAL: CRITICAL: puppet fail [16:00:04] manybubbles, anomie, ^d, marktraceur: Dear anthropoid, the time has come. Please deploy Morning SWAT(Max 8 patches) (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20150120T1600). [16:07:18] any swatters? should i just do it? [16:07:45] its just two config changes ... sending them ut [16:07:54] Nikerabbit: around for swat? [16:08:07] RECOVERY - puppet last run on nescio is OK: OK: Puppet is currently enabled, last run 24 seconds ago with 0 failures [16:08:56] Nikerabbit: i'll give you a couple more minutes, but you need to be around to deploy your patch [16:09:04] (i can deploy, you test :) [16:09:47] (03CR) 10EBernhardson: [C: 032] Enable wgContentHandlerUseDB on all wikis [mediawiki-config] - 10https://gerrit.wikimedia.org/r/175565 (owner: 10EBernhardson) [16:09:56] (03Merged) 10jenkins-bot: Enable wgContentHandlerUseDB on all wikis [mediawiki-config] - 10https://gerrit.wikimedia.org/r/175565 (owner: 10EBernhardson) [16:12:00] ebernhardson: You can go for config change. [16:12:31] I requested it for SWAT deployment for it. Nikerabbit must be travelling. [16:12:44] (03CR) 10EBernhardson: [C: 032] Re-enable event logging for cx [mediawiki-config] - 10https://gerrit.wikimedia.org/r/185667 (owner: 10Nikerabbit) [16:12:51] kart_: ok [16:12:51] (03Merged) 10jenkins-bot: Re-enable event logging for cx [mediawiki-config] - 10https://gerrit.wikimedia.org/r/185667 (owner: 10Nikerabbit) [16:14:12] !log ebernhardson Synchronized wmf-config/: tues morning swat config changes (duration: 00m 07s) [16:14:16] Logged the message, Master [16:14:23] one error while syncing: 16:14:07 ['/srv/deployment/scap/scap/bin/sync-common', '--no-update-l10n', '--include', 'wmf-config', '--include', 'wmf-config/***', 'mw1033.eqiad.wmnet', 'mw1070.eqiad.wmnet', 'mw1097.eqiad.wmnet', 'mw1216.eqiad.wmnet', 'mw1161.eqiad.wmnet', 'mw1201.eqiad.wmnet'] on mw1162 returned [255]: Error reading response length from authentication socket. [16:14:43] !log ebernhardson Synchronized wmf-config/: tues morning swat config changes (duration: 00m 07s) [16:14:49] re-ran same command, no error [16:14:52] kart_: go ahead and test [16:17:42] ebernhardson: I think that's usually benign [16:18:00] TURN DOWN FOR SWAT. [16:18:09] Thanks for covering us, ebernhardson [16:18:24] I think the usual cabal is in SF and still hasn't stumbled out of CQ [16:18:30] Apart from Chad, who has no excuse. [16:19:17] I suppose to add one more deployment, but right now stressed with lost luggage :D [16:20:21] <^d> marktraceur: s/CQ/home/ [16:20:24] <^d> But otherwise yeah. [16:21:05] marktraceur: np [16:26:29] kart_: good luck then [16:27:53] !log restarting puppetmasters (palladium/strontium) [16:27:58] oh passenger [16:28:00] Logged the message, Master [16:28:47] RECOVERY - puppetmaster backend https on strontium is OK: HTTP OK: Status line output matched 400 - 335 bytes in 0.024 second response time [16:28:48] RECOVERY - puppet last run on analytics1013 is OK: OK: Puppet is currently enabled, last run 17 seconds ago with 0 failures [16:28:58] RECOVERY - puppet last run on es2004 is OK: OK: Puppet is currently enabled, last run 23 seconds ago with 0 failures [16:28:58] RECOVERY - puppet last run on thallium is OK: OK: Puppet is currently enabled, last run 3 seconds ago with 0 failures [16:28:58] RECOVERY - puppet last run on elastic1006 is OK: OK: Puppet is currently enabled, last run 5 seconds ago with 0 failures [16:28:58] RECOVERY - puppet last run on labmon1001 is OK: OK: Puppet is currently enabled, last run 19 seconds ago with 0 failures [16:29:20] RECOVERY - puppet last run on wtp1018 is OK: OK: Puppet is currently enabled, last run 1 second ago with 0 failures [16:29:20] RECOVERY - puppet last run on cp1046 is OK: OK: Puppet is currently enabled, last run 18 seconds ago with 0 failures [16:29:20] RECOVERY - puppet last run on argon is OK: OK: Puppet is currently enabled, last run 17 seconds ago with 0 failures [16:29:20] RECOVERY - puppet last run on osmium is OK: OK: Puppet is currently enabled, last run 14 seconds ago with 0 failures [16:29:20] RECOVERY - puppet last run on db1020 is OK: OK: Puppet is currently enabled, last run 3 seconds ago with 0 failures [16:29:28] RECOVERY - puppet last run on rhenium is OK: OK: Puppet is currently enabled, last run 1 second ago with 0 failures [16:29:28] RECOVERY - puppet last run on mw1247 is OK: OK: Puppet is currently enabled, last run 36 seconds ago with 0 failures [16:29:28] RECOVERY - puppet last run on plutonium is OK: OK: Puppet is currently enabled, last run 36 seconds ago with 0 failures [16:29:28] RECOVERY - puppet last run on snapshot1002 is OK: OK: Puppet is currently enabled, last run 13 seconds ago with 0 failures [16:29:28] RECOVERY - puppet last run on db1060 is OK: OK: Puppet is currently enabled, last run 46 seconds ago with 0 failures [16:29:28] RECOVERY - puppet last run on mw1133 is OK: OK: Puppet is currently enabled, last run 32 seconds ago with 0 failures [16:29:29] RECOVERY - puppet last run on rubidium is OK: OK: Puppet is currently enabled, last run 18 seconds ago with 0 failures [16:29:29] RECOVERY - puppet last run on analytics1023 is OK: OK: Puppet is currently enabled, last run 12 seconds ago with 0 failures [16:29:29] RECOVERY - puppet last run on db1004 is OK: OK: Puppet is currently enabled, last run 16 seconds ago with 0 failures [16:29:30] RECOVERY - puppet last run on mw1004 is OK: OK: Puppet is currently enabled, last run 5 seconds ago with 0 failures [16:29:37] RECOVERY - puppet last run on cp1063 is OK: OK: Puppet is currently enabled, last run 7 seconds ago with 0 failures [16:29:38] PROBLEM - Disk space on antimony is CRITICAL: DISK CRITICAL - free space: / 1422 MB (3% inode=38%): [16:29:38] RECOVERY - puppet last run on elastic1019 is OK: OK: Puppet is currently enabled, last run 52 seconds ago with 0 failures [16:29:38] RECOVERY - puppet last run on db1069 is OK: OK: Puppet is currently enabled, last run 43 seconds ago with 0 failures [16:29:38] RECOVERY - puppet last run on labsdb1006 is OK: OK: Puppet is currently enabled, last run 32 seconds ago with 0 failures [16:29:39] RECOVERY - puppet last run on lvs2006 is OK: OK: Puppet is currently enabled, last run 46 seconds ago with 0 failures [16:29:39] ^d: this looks good? https://gerrit.wikimedia.org/r/#/c/185987/ ? [16:29:47] RECOVERY - puppet last run on ms-be1008 is OK: OK: Puppet is currently enabled, last run 18 seconds ago with 0 failures [16:29:47] RECOVERY - puppet last run on rdb1001 is OK: OK: Puppet is currently enabled, last run 25 seconds ago with 0 failures [16:29:48] RECOVERY - puppet last run on stat1002 is OK: OK: Puppet is currently enabled, last run 6 seconds ago with 0 failures [16:29:57] RECOVERY - puppet last run on virt1010 is OK: OK: Puppet is currently enabled, last run 40 seconds ago with 0 failures [16:29:57] RECOVERY - puppet last run on mw1081 is OK: OK: Puppet is currently enabled, last run 0 seconds ago with 0 failures [16:29:57] RECOVERY - puppet last run on oxygen is OK: OK: Puppet is currently enabled, last run 39 seconds ago with 0 failures [16:29:58] RECOVERY - puppet last run on tungsten is OK: OK: Puppet is currently enabled, last run 5 seconds ago with 0 failures [16:29:58] RECOVERY - puppet last run on db1071 is OK: OK: Puppet is currently enabled, last run 54 seconds ago with 0 failures [16:29:58] RECOVERY - puppet last run on mw1159 is OK: OK: Puppet is currently enabled, last run 3 seconds ago with 0 failures [16:29:58] RECOVERY - puppet last run on mw1111 is OK: OK: Puppet is currently enabled, last run 43 seconds ago with 0 failures [16:29:58] RECOVERY - puppet last run on mw1014 is OK: OK: Puppet is currently enabled, last run 26 seconds ago with 0 failures [16:29:59] RECOVERY - puppet last run on mw1180 is OK: OK: Puppet is currently enabled, last run 19 seconds ago with 0 failures [16:29:59] RECOVERY - puppet last run on virt1007 is OK: OK: Puppet is currently enabled, last run 52 seconds ago with 0 failures [16:30:00] RECOVERY - puppet last run on mw1168 is OK: OK: Puppet is currently enabled, last run 28 seconds ago with 0 failures [16:30:00] RECOVERY - puppet last run on db2037 is OK: OK: Puppet is currently enabled, last run 29 seconds ago with 0 failures [16:30:08] RECOVERY - puppet last run on analytics1022 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [16:30:08] RECOVERY - puppet last run on mw1125 is OK: OK: Puppet is currently enabled, last run 53 seconds ago with 0 failures [16:30:08] RECOVERY - puppet last run on elastic1015 is OK: OK: Puppet is currently enabled, last run 47 seconds ago with 0 failures [16:30:08] RECOVERY - puppet last run on mw1051 is OK: OK: Puppet is currently enabled, last run 29 seconds ago with 0 failures [16:30:08] RECOVERY - puppet last run on db2023 is OK: OK: Puppet is currently enabled, last run 28 seconds ago with 0 failures [16:30:09] RECOVERY - puppet last run on db2004 is OK: OK: Puppet is currently enabled, last run 48 seconds ago with 0 failures [16:30:09] RECOVERY - puppet last run on ms-be2012 is OK: OK: Puppet is currently enabled, last run 47 seconds ago with 0 failures [16:30:09] RECOVERY - puppet last run on es2007 is OK: OK: Puppet is currently enabled, last run 6 seconds ago with 0 failures [16:30:09] RECOVERY - puppet last run on ms-be2005 is OK: OK: Puppet is currently enabled, last run 15 seconds ago with 0 failures [16:30:10] RECOVERY - puppet last run on db2001 is OK: OK: Puppet is currently enabled, last run 58 seconds ago with 0 failures [16:30:11] RECOVERY - puppet last run on ms-be2001 is OK: OK: Puppet is currently enabled, last run 27 seconds ago with 0 failures [16:30:11] RECOVERY - puppet last run on mw1057 is OK: OK: Puppet is currently enabled, last run 14 seconds ago with 0 failures [16:30:12] RECOVERY - puppet last run on mw1190 is OK: OK: Puppet is currently enabled, last run 51 seconds ago with 0 failures [16:30:12] RECOVERY - puppet last run on db1055 is OK: OK: Puppet is currently enabled, last run 21 seconds ago with 0 failures [16:30:12] RECOVERY - puppet last run on bast4001 is OK: OK: Puppet is currently enabled, last run 38 seconds ago with 0 failures [16:30:13] RECOVERY - puppet last run on cp4018 is OK: OK: Puppet is currently enabled, last run 8 seconds ago with 0 failures [16:30:17] RECOVERY - puppet last run on cp4005 is OK: OK: Puppet is currently enabled, last run 2 seconds ago with 0 failures [16:30:17] RECOVERY - puppet last run on hafnium is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [16:30:18] RECOVERY - puppet last run on snapshot1004 is OK: OK: Puppet is currently enabled, last run 49 seconds ago with 0 failures [16:30:18] RECOVERY - puppet last run on cp1038 is OK: OK: Puppet is currently enabled, last run 14 seconds ago with 0 failures [16:30:27] RECOVERY - puppet last run on ms-be1007 is OK: OK: Puppet is currently enabled, last run 17 seconds ago with 0 failures [16:30:28] RECOVERY - puppet last run on cp1048 is OK: OK: Puppet is currently enabled, last run 49 seconds ago with 0 failures [16:30:28] RECOVERY - puppet last run on ms-be3001 is OK: OK: Puppet is currently enabled, last run 59 seconds ago with 0 failures [16:30:28] RECOVERY - puppet last run on amssq40 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [16:30:28] RECOVERY - puppet last run on amssq36 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [16:30:28] RECOVERY - puppet last run on mw1227 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [16:30:28] RECOVERY - puppet last run on amssq51 is OK: OK: Puppet is currently enabled, last run 30 seconds ago with 0 failures [16:30:28] RECOVERY - puppet last run on mw1034 is OK: OK: Puppet is currently enabled, last run 41 seconds ago with 0 failures [16:30:29] RECOVERY - puppet last run on wtp1002 is OK: OK: Puppet is currently enabled, last run 5 seconds ago with 0 failures [16:30:30] RECOVERY - puppet last run on ms-fe3002 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [16:30:37] RECOVERY - puppet last run on ms-be1009 is OK: OK: Puppet is currently enabled, last run 15 seconds ago with 0 failures [16:30:37] RECOVERY - puppet last run on mw1098 is OK: OK: Puppet is currently enabled, last run 56 seconds ago with 0 failures [16:30:37] RECOVERY - puppet last run on db1062 is OK: OK: Puppet is currently enabled, last run 56 seconds ago with 0 failures [16:30:37] RECOVERY - puppet last run on mw1079 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [16:30:38] RECOVERY - puppet last run on mw1212 is OK: OK: Puppet is currently enabled, last run 18 seconds ago with 0 failures [16:30:38] RECOVERY - puppet last run on analytics1037 is OK: OK: Puppet is currently enabled, last run 40 seconds ago with 0 failures [16:30:38] RECOVERY - puppet last run on mw1029 is OK: OK: Puppet is currently enabled, last run 2 seconds ago with 0 failures [16:30:38] RECOVERY - puppet last run on mw1056 is OK: OK: Puppet is currently enabled, last run 35 seconds ago with 0 failures [16:30:39] RECOVERY - puppet last run on mw1146 is OK: OK: Puppet is currently enabled, last run 24 seconds ago with 0 failures [16:30:39] RECOVERY - puppet last run on mw1258 is OK: OK: Puppet is currently enabled, last run 28 seconds ago with 0 failures [16:30:40] RECOVERY - puppet last run on mw1156 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [16:30:40] RECOVERY - puppet last run on mw1183 is OK: OK: Puppet is currently enabled, last run 52 seconds ago with 0 failures [16:30:40] RECOVERY - puppet last run on lvs1001 is OK: OK: Puppet is currently enabled, last run 5 seconds ago with 0 failures [16:30:47] RECOVERY - puppet last run on mw1165 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [16:30:47] RECOVERY - puppet last run on mw1030 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [16:30:47] RECOVERY - puppet last run on mw1171 is OK: OK: Puppet is currently enabled, last run 13 seconds ago with 0 failures [16:30:48] RECOVERY - puppet last run on mw1097 is OK: OK: Puppet is currently enabled, last run 1 second ago with 0 failures [16:30:48] RECOVERY - puppet last run on wtp1004 is OK: OK: Puppet is currently enabled, last run 51 seconds ago with 0 failures [16:30:48] RECOVERY - puppet last run on analytics1032 is OK: OK: Puppet is currently enabled, last run 13 seconds ago with 0 failures [16:30:48] RECOVERY - puppet last run on ms-be1012 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [16:30:48] RECOVERY - puppet last run on mw1181 is OK: OK: Puppet is currently enabled, last run 58 seconds ago with 0 failures [16:30:49] RECOVERY - puppet last run on mc1001 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [16:30:53] <^d> kart_: I guess so? I dunno what sha1s they point to but the commit to core looks fine. [16:30:57] RECOVERY - puppet last run on cp1062 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [16:31:07] RECOVERY - puppet last run on elastic1011 is OK: OK: Puppet is currently enabled, last run 40 seconds ago with 0 failures [16:31:07] RECOVERY - puppet last run on mw1023 is OK: OK: Puppet is currently enabled, last run 8 seconds ago with 0 failures [16:31:07] RECOVERY - puppet last run on mw1050 is OK: OK: Puppet is currently enabled, last run 47 seconds ago with 0 failures [16:31:08] RECOVERY - puppet last run on protactinium is OK: OK: Puppet is currently enabled, last run 16 seconds ago with 0 failures [16:31:08] RECOVERY - puppet last run on wtp1022 is OK: OK: Puppet is currently enabled, last run 41 seconds ago with 0 failures [16:31:08] RECOVERY - puppet last run on rdb1002 is OK: OK: Puppet is currently enabled, last run 34 seconds ago with 0 failures [16:31:08] RECOVERY - puppet last run on mw1116 is OK: OK: Puppet is currently enabled, last run 31 seconds ago with 0 failures [16:31:08] RECOVERY - puppet last run on mw1074 is OK: OK: Puppet is currently enabled, last run 21 seconds ago with 0 failures [16:31:09] RECOVERY - puppet last run on mw1248 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [16:31:09] RECOVERY - puppet last run on elastic1014 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [16:31:19] RECOVERY - puppet last run on db2003 is OK: OK: Puppet is currently enabled, last run 10 seconds ago with 0 failures [16:31:19] RECOVERY - puppet last run on haedus is OK: OK: Puppet is currently enabled, last run 35 seconds ago with 0 failures [16:31:19] RECOVERY - puppet last run on ms-be2008 is OK: OK: Puppet is currently enabled, last run 57 seconds ago with 0 failures [16:31:19] RECOVERY - puppet last run on ms-be2007 is OK: OK: Puppet is currently enabled, last run 23 seconds ago with 0 failures [16:31:23] ^d: Updating extension to master, that's all I need, right? [16:31:27] RECOVERY - puppet last run on mw1210 is OK: OK: Puppet is currently enabled, last run 46 seconds ago with 0 failures [16:31:34] <^d> Yeah [16:31:37] RECOVERY - puppet last run on db1001 is OK: OK: Puppet is currently enabled, last run 50 seconds ago with 0 failures [16:31:37] RECOVERY - puppet last run on mw1188 is OK: OK: Puppet is currently enabled, last run 44 seconds ago with 0 failures [16:31:37] RECOVERY - puppet last run on mw1198 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [16:31:38] RECOVERY - puppet last run on cp1060 is OK: OK: Puppet is currently enabled, last run 15 seconds ago with 0 failures [16:31:38] RECOVERY - puppet last run on wtp1023 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [16:31:38] RECOVERY - puppet last run on ocg1003 is OK: OK: Puppet is currently enabled, last run 1 second ago with 0 failures [16:31:38] RECOVERY - puppet last run on amssq41 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [16:31:38] RECOVERY - puppet last run on amslvs3 is OK: OK: Puppet is currently enabled, last run 3 seconds ago with 0 failures [16:31:39] RECOVERY - puppet last run on hooft is OK: OK: Puppet is currently enabled, last run 26 seconds ago with 0 failures [16:31:39] RECOVERY - puppet last run on db1054 is OK: OK: Puppet is currently enabled, last run 44 seconds ago with 0 failures [16:31:43] ^d: cool. Evening SWAT then. [16:31:48] RECOVERY - puppet last run on elastic1005 is OK: OK: Puppet is currently enabled, last run 56 seconds ago with 0 failures [16:31:48] RECOVERY - puppet last run on rcs1001 is OK: OK: Puppet is currently enabled, last run 6 seconds ago with 0 failures [16:31:48] RECOVERY - puppet last run on titanium is OK: OK: Puppet is currently enabled, last run 30 seconds ago with 0 failures [16:31:57] RECOVERY - puppet last run on elastic1002 is OK: OK: Puppet is currently enabled, last run 34 seconds ago with 0 failures [16:31:57] RECOVERY - puppet last run on rcs1002 is OK: OK: Puppet is currently enabled, last run 12 seconds ago with 0 failures [16:31:58] RECOVERY - puppet last run on wtp1015 is OK: OK: Puppet is currently enabled, last run 26 seconds ago with 0 failures [16:31:58] RECOVERY - puppet last run on db1044 is OK: OK: Puppet is currently enabled, last run 8 seconds ago with 0 failures [16:31:58] RECOVERY - puppet last run on elastic1023 is OK: OK: Puppet is currently enabled, last run 25 seconds ago with 0 failures [16:31:58] RECOVERY - puppet last run on es1002 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [16:31:58] RECOVERY - puppet last run on mw1163 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [16:32:07] RECOVERY - puppet last run on lvs2003 is OK: OK: Puppet is currently enabled, last run 42 seconds ago with 0 failures [16:32:07] RECOVERY - puppet last run on mw1053 is OK: OK: Puppet is currently enabled, last run 56 seconds ago with 0 failures [16:32:07] RECOVERY - puppet last run on analytics1014 is OK: OK: Puppet is currently enabled, last run 25 seconds ago with 0 failures [16:32:07] RECOVERY - puppet last run on mc1013 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [16:32:08] RECOVERY - puppet last run on labsdb1004 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [16:32:08] RECOVERY - puppet last run on wtp1007 is OK: OK: Puppet is currently enabled, last run 24 seconds ago with 0 failures [16:32:08] RECOVERY - puppet last run on analytics1011 is OK: OK: Puppet is currently enabled, last run 27 seconds ago with 0 failures [16:32:08] RECOVERY - puppet last run on mw1087 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [16:32:09] RECOVERY - puppet last run on mw1148 is OK: OK: Puppet is currently enabled, last run 20 seconds ago with 0 failures [16:32:17] RECOVERY - puppet last run on pc1003 is OK: OK: Puppet is currently enabled, last run 7 seconds ago with 0 failures [16:32:17] RECOVERY - puppet last run on db1063 is OK: OK: Puppet is currently enabled, last run 44 seconds ago with 0 failures [16:32:17] RECOVERY - puppet last run on elastic1017 is OK: OK: Puppet is currently enabled, last run 31 seconds ago with 0 failures [16:32:17] RECOVERY - puppet last run on nitrogen is OK: OK: Puppet is currently enabled, last run 36 seconds ago with 0 failures [16:32:17] RECOVERY - puppet last run on mc1007 is OK: OK: Puppet is currently enabled, last run 45 seconds ago with 0 failures [16:32:17] RECOVERY - puppet last run on db1057 is OK: OK: Puppet is currently enabled, last run 52 seconds ago with 0 failures [16:32:18] RECOVERY - puppet last run on db1064 is OK: OK: Puppet is currently enabled, last run 23 seconds ago with 0 failures [16:32:18] RECOVERY - puppet last run on wtp1013 is OK: OK: Puppet is currently enabled, last run 54 seconds ago with 0 failures [16:32:18] RECOVERY - puppet last run on mw1239 is OK: OK: Puppet is currently enabled, last run 25 seconds ago with 0 failures [16:32:27] RECOVERY - puppet last run on db2011 is OK: OK: Puppet is currently enabled, last run 33 seconds ago with 0 failures [16:32:28] RECOVERY - puppet last run on wtp1011 is OK: OK: Puppet is currently enabled, last run 59 seconds ago with 0 failures [16:32:28] RECOVERY - puppet last run on db1070 is OK: OK: Puppet is currently enabled, last run 44 seconds ago with 0 failures [16:32:28] RECOVERY - puppet last run on mw1243 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [16:32:37] RECOVERY - puppet last run on db1011 is OK: OK: Puppet is currently enabled, last run 2 seconds ago with 0 failures [16:32:47] RECOVERY - puppet last run on wtp1003 is OK: OK: Puppet is currently enabled, last run 58 seconds ago with 0 failures [16:32:48] RECOVERY - puppet last run on mw1001 is OK: OK: Puppet is currently enabled, last run 6 seconds ago with 0 failures [16:32:48] RECOVERY - puppet last run on iridium is OK: OK: Puppet is currently enabled, last run 4 seconds ago with 0 failures [16:32:57] RECOVERY - puppet last run on amssq38 is OK: OK: Puppet is currently enabled, last run 28 seconds ago with 0 failures [16:32:58] RECOVERY - puppet last run on cp3012 is OK: OK: Puppet is currently enabled, last run 9 seconds ago with 0 failures [16:32:58] RECOVERY - puppet last run on mw1185 is OK: OK: Puppet is currently enabled, last run 31 seconds ago with 0 failures [16:32:58] RECOVERY - puppet last run on mw1108 is OK: OK: Puppet is currently enabled, last run 13 seconds ago with 0 failures [16:32:58] RECOVERY - puppet last run on ms-fe3001 is OK: OK: Puppet is currently enabled, last run 13 seconds ago with 0 failures [16:32:58] RECOVERY - puppet last run on cp3009 is OK: OK: Puppet is currently enabled, last run 56 seconds ago with 0 failures [16:32:58] RECOVERY - puppet last run on amssq62 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [16:32:58] RECOVERY - puppet last run on mw1201 is OK: OK: Puppet is currently enabled, last run 14 seconds ago with 0 failures [16:32:59] RECOVERY - puppet last run on calcium is OK: OK: Puppet is currently enabled, last run 12 seconds ago with 0 failures [16:32:59] RECOVERY - puppet last run on mw1122 is OK: OK: Puppet is currently enabled, last run 9 seconds ago with 0 failures [16:32:59] RECOVERY - puppet last run on mw1077 is OK: OK: Puppet is currently enabled, last run 29 seconds ago with 0 failures [16:33:00] RECOVERY - puppet last run on rdb1003 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [16:33:07] RECOVERY - puppet last run on db1030 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [16:33:07] RECOVERY - puppet last run on es1010 is OK: OK: Puppet is currently enabled, last run 7 seconds ago with 0 failures [16:33:07] RECOVERY - puppet last run on mw1032 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [16:33:08] RECOVERY - puppet last run on db1038 is OK: OK: Puppet is currently enabled, last run 7 seconds ago with 0 failures [16:33:08] RECOVERY - puppet last run on mw1043 is OK: OK: Puppet is currently enabled, last run 22 seconds ago with 0 failures [16:33:08] RECOVERY - puppet last run on mw1167 is OK: OK: Puppet is currently enabled, last run 58 seconds ago with 0 failures [16:33:08] RECOVERY - puppet last run on dbstore1002 is OK: OK: Puppet is currently enabled, last run 4 seconds ago with 0 failures [16:33:17] RECOVERY - puppet last run on mw1121 is OK: OK: Puppet is currently enabled, last run 3 seconds ago with 0 failures [16:33:19] RECOVERY - puppet last run on virt1000 is OK: OK: Puppet is currently enabled, last run 57 seconds ago with 0 failures [16:33:27] RECOVERY - puppet last run on mw1229 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [16:33:27] RECOVERY - puppet last run on db1006 is OK: OK: Puppet is currently enabled, last run 17 seconds ago with 0 failures [16:33:27] RECOVERY - puppet last run on mw1022 is OK: OK: Puppet is currently enabled, last run 9 seconds ago with 0 failures [16:33:28] RECOVERY - puppet last run on mw1033 is OK: OK: Puppet is currently enabled, last run 25 seconds ago with 0 failures [16:33:28] RECOVERY - puppet last run on mw1105 is OK: OK: Puppet is currently enabled, last run 27 seconds ago with 0 failures [16:33:28] RECOVERY - puppet last run on mw1186 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [16:33:28] RECOVERY - puppet last run on mercury is OK: OK: Puppet is currently enabled, last run 36 seconds ago with 0 failures [16:33:28] RECOVERY - puppet last run on mw1225 is OK: OK: Puppet is currently enabled, last run 31 seconds ago with 0 failures [16:33:29] RECOVERY - puppet last run on iodine is OK: OK: Puppet is currently enabled, last run 14 seconds ago with 0 failures [16:33:29] RECOVERY - puppet last run on db1047 is OK: OK: Puppet is currently enabled, last run 29 seconds ago with 0 failures [16:33:37] RECOVERY - puppet last run on cp1052 is OK: OK: Puppet is currently enabled, last run 9 seconds ago with 0 failures [16:33:38] RECOVERY - puppet last run on ms-be1011 is OK: OK: Puppet is currently enabled, last run 37 seconds ago with 0 failures [16:33:38] RECOVERY - puppet last run on db2028 is OK: OK: Puppet is currently enabled, last run 44 seconds ago with 0 failures [16:33:38] RECOVERY - puppet last run on db2012 is OK: OK: Puppet is currently enabled, last run 39 seconds ago with 0 failures [16:33:38] RECOVERY - puppet last run on ms-fe2002 is OK: OK: Puppet is currently enabled, last run 16 seconds ago with 0 failures [16:33:38] RECOVERY - puppet last run on analytics1027 is OK: OK: Puppet is currently enabled, last run 18 seconds ago with 0 failures [16:33:48] RECOVERY - puppet last run on analytics1004 is OK: OK: Puppet is currently enabled, last run 21 seconds ago with 0 failures [16:33:57] RECOVERY - puppet last run on copper is OK: OK: Puppet is currently enabled, last run 1 second ago with 0 failures [16:33:57] RECOVERY - puppet last run on mw1152 is OK: OK: Puppet is currently enabled, last run 23 seconds ago with 0 failures [16:33:57] RECOVERY - puppet last run on cp1037 is OK: OK: Puppet is currently enabled, last run 15 seconds ago with 0 failures [16:33:58] RECOVERY - puppet last run on mw1024 is OK: OK: Puppet is currently enabled, last run 36 seconds ago with 0 failures [16:34:07] RECOVERY - puppet last run on d-i-test is OK: OK: Puppet is currently enabled, last run 57 seconds ago with 0 failures [16:34:07] RECOVERY - puppet last run on mw1209 is OK: OK: Puppet is currently enabled, last run 52 seconds ago with 0 failures [16:34:07] RECOVERY - puppet last run on cp3004 is OK: OK: Puppet is currently enabled, last run 49 seconds ago with 0 failures [16:34:08] RECOVERY - puppet last run on db1033 is OK: OK: Puppet is currently enabled, last run 41 seconds ago with 0 failures [16:34:08] RECOVERY - puppet last run on wtp1010 is OK: OK: Puppet is currently enabled, last run 3 seconds ago with 0 failures [16:34:17] RECOVERY - puppet last run on db1061 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [16:34:17] RECOVERY - puppet last run on fluorine is OK: OK: Puppet is currently enabled, last run 30 seconds ago with 0 failures [16:34:18] RECOVERY - puppet last run on mw1091 is OK: OK: Puppet is currently enabled, last run 28 seconds ago with 0 failures [16:34:18] RECOVERY - puppet last run on einsteinium is OK: OK: Puppet is currently enabled, last run 48 seconds ago with 0 failures [16:34:27] RECOVERY - puppet last run on mw1215 is OK: OK: Puppet is currently enabled, last run 9 seconds ago with 0 failures [16:34:27] RECOVERY - puppet last run on terbium is OK: OK: Puppet is currently enabled, last run 5 seconds ago with 0 failures [16:34:27] RECOVERY - puppet last run on mw1219 is OK: OK: Puppet is currently enabled, last run 23 seconds ago with 0 failures [16:34:27] RECOVERY - puppet last run on mw1093 is OK: OK: Puppet is currently enabled, last run 55 seconds ago with 0 failures [16:34:27] RECOVERY - puppet last run on mw1220 is OK: OK: Puppet is currently enabled, last run 14 seconds ago with 0 failures [16:34:27] RECOVERY - puppet last run on mw1139 is OK: OK: Puppet is currently enabled, last run 15 seconds ago with 0 failures [16:34:28] RECOVERY - puppet last run on mw1016 is OK: OK: Puppet is currently enabled, last run 52 seconds ago with 0 failures [16:34:28] RECOVERY - puppet last run on mw1223 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [16:34:37] RECOVERY - puppet last run on mw1142 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [16:34:37] RECOVERY - puppet last run on mw1010 is OK: OK: Puppet is currently enabled, last run 43 seconds ago with 0 failures [16:34:37] RECOVERY - puppet last run on analytics1028 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [16:34:38] RECOVERY - puppet last run on mw1064 is OK: OK: Puppet is currently enabled, last run 12 seconds ago with 0 failures [16:34:38] RECOVERY - puppet last run on mw1193 is OK: OK: Puppet is currently enabled, last run 16 seconds ago with 0 failures [16:34:47] RECOVERY - puppet last run on mw1107 is OK: OK: Puppet is currently enabled, last run 9 seconds ago with 0 failures [16:34:47] RECOVERY - puppet last run on mw1236 is OK: OK: Puppet is currently enabled, last run 35 seconds ago with 0 failures [16:34:47] RECOVERY - puppet last run on mw1112 is OK: OK: Puppet is currently enabled, last run 4 seconds ago with 0 failures [16:34:48] RECOVERY - puppet last run on cp1040 is OK: OK: Puppet is currently enabled, last run 26 seconds ago with 0 failures [16:34:48] RECOVERY - puppet last run on elastic1003 is OK: OK: Puppet is currently enabled, last run 24 seconds ago with 0 failures [16:34:48] RECOVERY - puppet last run on achernar is OK: OK: Puppet is currently enabled, last run 3 seconds ago with 0 failures [16:34:48] RECOVERY - puppet last run on mw1241 is OK: OK: Puppet is currently enabled, last run 50 seconds ago with 0 failures [16:34:57] RECOVERY - puppet last run on osm-cp1001 is OK: OK: Puppet is currently enabled, last run 12 seconds ago with 0 failures [16:34:57] RECOVERY - puppet last run on elastic1020 is OK: OK: Puppet is currently enabled, last run 41 seconds ago with 0 failures [16:34:57] RECOVERY - puppet last run on virt1008 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [16:34:57] RECOVERY - puppet last run on mw1231 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [16:34:58] RECOVERY - puppet last run on cp1044 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [16:34:58] RECOVERY - puppet last run on virt1011 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [16:34:58] RECOVERY - puppet last run on strontium is OK: OK: Puppet is currently enabled, last run 18 seconds ago with 0 failures [16:34:58] RECOVERY - puppet last run on lvs1004 is OK: OK: Puppet is currently enabled, last run 3 seconds ago with 0 failures [16:35:08] RECOVERY - puppet last run on cp4020 is OK: OK: Puppet is currently enabled, last run 18 seconds ago with 0 failures [16:35:08] RECOVERY - puppet last run on rbf1001 is OK: OK: Puppet is currently enabled, last run 48 seconds ago with 0 failures [16:35:08] RECOVERY - puppet last run on cp1054 is OK: OK: Puppet is currently enabled, last run 8 seconds ago with 0 failures [16:35:09] RECOVERY - puppet last run on lvs3003 is OK: OK: Puppet is currently enabled, last run 17 seconds ago with 0 failures [16:35:09] RECOVERY - puppet last run on db1072 is OK: OK: Puppet is currently enabled, last run 51 seconds ago with 0 failures [16:35:17] RECOVERY - puppet last run on mw1066 is OK: OK: Puppet is currently enabled, last run 33 seconds ago with 0 failures [16:35:18] RECOVERY - puppet last run on mw1086 is OK: OK: Puppet is currently enabled, last run 24 seconds ago with 0 failures [16:35:18] RECOVERY - puppet last run on analytics1003 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [16:35:18] RECOVERY - puppet last run on analytics1018 is OK: OK: Puppet is currently enabled, last run 30 seconds ago with 0 failures [16:35:18] RECOVERY - puppet last run on mw1253 is OK: OK: Puppet is currently enabled, last run 12 seconds ago with 0 failures [16:35:18] RECOVERY - puppet last run on amssq31 is OK: OK: Puppet is currently enabled, last run 36 seconds ago with 0 failures [16:35:18] RECOVERY - puppet last run on cp1067 is OK: OK: Puppet is currently enabled, last run 32 seconds ago with 0 failures [16:35:19] RECOVERY - puppet last run on cp1068 is OK: OK: Puppet is currently enabled, last run 6 seconds ago with 0 failures [16:35:20] RECOVERY - puppet last run on cp3018 is OK: OK: Puppet is currently enabled, last run 20 seconds ago with 0 failures [16:35:27] RECOVERY - puppet last run on cp1045 is OK: OK: Puppet is currently enabled, last run 1 second ago with 0 failures [16:35:28] RECOVERY - puppet last run on mw1143 is OK: OK: Puppet is currently enabled, last run 56 seconds ago with 0 failures [16:35:28] RECOVERY - puppet last run on db1035 is OK: OK: Puppet is currently enabled, last run 19 seconds ago with 0 failures [16:35:28] RECOVERY - puppet last run on ms-be1002 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [16:35:28] RECOVERY - puppet last run on bast1001 is OK: OK: Puppet is currently enabled, last run 13 seconds ago with 0 failures [16:35:38] RECOVERY - puppet last run on wtp1008 is OK: OK: Puppet is currently enabled, last run 43 seconds ago with 0 failures [16:35:38] RECOVERY - puppet last run on mw1071 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [16:35:38] RECOVERY - puppet last run on ms-be1015 is OK: OK: Puppet is currently enabled, last run 11 seconds ago with 0 failures [16:35:47] RECOVERY - puppet last run on mw1207 is OK: OK: Puppet is currently enabled, last run 15 seconds ago with 0 failures [16:35:47] RECOVERY - puppet last run on mw1090 is OK: OK: Puppet is currently enabled, last run 57 seconds ago with 0 failures [16:35:48] RECOVERY - puppet last run on mw1203 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [16:35:48] RECOVERY - puppet last run on mw1204 is OK: OK: Puppet is currently enabled, last run 47 seconds ago with 0 failures [16:35:48] RECOVERY - puppet last run on dysprosium is OK: OK: Puppet is currently enabled, last run 33 seconds ago with 0 failures [16:35:57] RECOVERY - puppet last run on es1004 is OK: OK: Puppet is currently enabled, last run 4 seconds ago with 0 failures [16:35:57] RECOVERY - puppet last run on mw1104 is OK: OK: Puppet is currently enabled, last run 10 seconds ago with 0 failures [16:35:58] RECOVERY - puppet last run on chromium is OK: OK: Puppet is currently enabled, last run 46 seconds ago with 0 failures [16:35:58] RECOVERY - puppet last run on lvs1003 is OK: OK: Puppet is currently enabled, last run 46 seconds ago with 0 failures [16:35:58] RECOVERY - puppet last run on wtp1001 is OK: OK: Puppet is currently enabled, last run 39 seconds ago with 0 failures [16:35:58] RECOVERY - puppet last run on ms-fe1002 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [16:35:58] RECOVERY - puppet last run on mw1027 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [16:36:07] RECOVERY - puppet last run on baham is OK: OK: Puppet is currently enabled, last run 51 seconds ago with 0 failures [16:36:08] RECOVERY - puppet last run on es2010 is OK: OK: Puppet is currently enabled, last run 41 seconds ago with 0 failures [16:36:08] RECOVERY - puppet last run on virt1005 is OK: OK: Puppet is currently enabled, last run 0 seconds ago with 0 failures [16:36:08] RECOVERY - puppet last run on bast2001 is OK: OK: Puppet is currently enabled, last run 4 seconds ago with 0 failures [16:36:08] RECOVERY - puppet last run on lvs4004 is OK: OK: Puppet is currently enabled, last run 32 seconds ago with 0 failures [16:36:08] RECOVERY - puppet last run on cp4009 is OK: OK: Puppet is currently enabled, last run 59 seconds ago with 0 failures [16:36:08] RECOVERY - puppet last run on mw1021 is OK: OK: Puppet is currently enabled, last run 38 seconds ago with 0 failures [16:36:08] RECOVERY - puppet last run on elastic1025 is OK: OK: Puppet is currently enabled, last run 33 seconds ago with 0 failures [16:36:09] RECOVERY - puppet last run on analytics1031 is OK: OK: Puppet is currently enabled, last run 3 seconds ago with 0 failures [16:36:09] RECOVERY - puppet last run on mw1037 is OK: OK: Puppet is currently enabled, last run 17 seconds ago with 0 failures [16:36:17] RECOVERY - puppet last run on cp4002 is OK: OK: Puppet is currently enabled, last run 41 seconds ago with 0 failures [16:36:17] RECOVERY - puppet last run on cp4012 is OK: OK: Puppet is currently enabled, last run 41 seconds ago with 0 failures [16:36:17] RECOVERY - puppet last run on db1056 is OK: OK: Puppet is currently enabled, last run 43 seconds ago with 0 failures [16:36:17] RECOVERY - puppet last run on db1037 is OK: OK: Puppet is currently enabled, last run 39 seconds ago with 0 failures [16:36:17] RECOVERY - puppet last run on mw1158 is OK: OK: Puppet is currently enabled, last run 16 seconds ago with 0 failures [16:36:18] RECOVERY - puppet last run on sca1001 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [16:36:27] RECOVERY - puppet last run on mw1135 is OK: OK: Puppet is currently enabled, last run 46 seconds ago with 0 failures [16:36:27] RECOVERY - puppet last run on radium is OK: OK: Puppet is currently enabled, last run 52 seconds ago with 0 failures [16:36:27] RECOVERY - puppet last run on db1027 is OK: OK: Puppet is currently enabled, last run 20 seconds ago with 0 failures [16:36:28] RECOVERY - puppet last run on mw1131 is OK: OK: Puppet is currently enabled, last run 49 seconds ago with 0 failures [16:36:28] RECOVERY - puppet last run on mw1073 is OK: OK: Puppet is currently enabled, last run 10 seconds ago with 0 failures [16:36:28] RECOVERY - puppet last run on mw1255 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [16:36:28] RECOVERY - puppet last run on eeden is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [16:36:28] RECOVERY - puppet last run on amssq33 is OK: OK: Puppet is currently enabled, last run 55 seconds ago with 0 failures [16:36:29] RECOVERY - puppet last run on mw1110 is OK: OK: Puppet is currently enabled, last run 58 seconds ago with 0 failures [16:36:29] RECOVERY - puppet last run on amssq37 is OK: OK: Puppet is currently enabled, last run 16 seconds ago with 0 failures [16:36:30] RECOVERY - puppet last run on amssq39 is OK: OK: Puppet is currently enabled, last run 45 seconds ago with 0 failures [16:36:30] RECOVERY - puppet last run on cp3005 is OK: OK: Puppet is currently enabled, last run 10 seconds ago with 0 failures [16:36:38] RECOVERY - puppet last run on db1045 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [16:36:38] RECOVERY - puppet last run on analytics1012 is OK: OK: Puppet is currently enabled, last run 12 seconds ago with 0 failures [16:36:38] RECOVERY - puppet last run on ms-be1010 is OK: OK: Puppet is currently enabled, last run 45 seconds ago with 0 failures [16:36:38] RECOVERY - puppet last run on cp1066 is OK: OK: Puppet is currently enabled, last run 27 seconds ago with 0 failures [16:36:48] RECOVERY - puppet last run on labsdb1005 is OK: OK: Puppet is currently enabled, last run 5 seconds ago with 0 failures [16:36:48] RECOVERY - puppet last run on mw1113 is OK: OK: Puppet is currently enabled, last run 34 seconds ago with 0 failures [16:36:48] RECOVERY - puppet last run on db1005 is OK: OK: Puppet is currently enabled, last run 7 seconds ago with 0 failures [16:36:48] RECOVERY - puppet last run on mw1155 is OK: OK: Puppet is currently enabled, last run 32 seconds ago with 0 failures [16:36:48] RECOVERY - puppet last run on mw1154 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [16:36:48] RECOVERY - puppet last run on radon is OK: OK: Puppet is currently enabled, last run 50 seconds ago with 0 failures [16:36:49] RECOVERY - puppet last run on cp1053 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [16:36:49] RECOVERY - puppet last run on lvs2002 is OK: OK: Puppet is currently enabled, last run 24 seconds ago with 0 failures [16:36:57] RECOVERY - puppet last run on mw1103 is OK: OK: Puppet is currently enabled, last run 33 seconds ago with 0 failures [16:36:57] RECOVERY - puppet last run on elastic1029 is OK: OK: Puppet is currently enabled, last run 57 seconds ago with 0 failures [16:37:07] RECOVERY - puppet last run on elastic1013 is OK: OK: Puppet is currently enabled, last run 12 seconds ago with 0 failures [16:37:07] RECOVERY - puppet last run on tmh1001 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [16:37:07] RECOVERY - puppet last run on mw1194 is OK: OK: Puppet is currently enabled, last run 17 seconds ago with 0 failures [16:37:08] RECOVERY - puppet last run on mc1015 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [16:37:08] RECOVERY - puppet last run on mw1047 is OK: OK: Puppet is currently enabled, last run 14 seconds ago with 0 failures [16:37:08] RECOVERY - puppet last run on mw1199 is OK: OK: Puppet is currently enabled, last run 21 seconds ago with 0 failures [16:37:08] RECOVERY - puppet last run on pollux is OK: OK: Puppet is currently enabled, last run 2 seconds ago with 0 failures [16:37:18] RECOVERY - puppet last run on db2010 is OK: OK: Puppet is currently enabled, last run 46 seconds ago with 0 failures [16:37:18] RECOVERY - puppet last run on es2006 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [16:37:18] RECOVERY - puppet last run on db2017 is OK: OK: Puppet is currently enabled, last run 35 seconds ago with 0 failures [16:37:18] RECOVERY - puppet last run on palladium is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [16:37:27] RECOVERY - puppet last run on cp4016 is OK: OK: Puppet is currently enabled, last run 19 seconds ago with 0 failures [16:37:38] RECOVERY - puppet last run on mw1137 is OK: OK: Puppet is currently enabled, last run 17 seconds ago with 0 failures [16:37:38] RECOVERY - puppet last run on elastic1010 is OK: OK: Puppet is currently enabled, last run 33 seconds ago with 0 failures [16:37:47] RECOVERY - puppet last run on db1049 is OK: OK: Puppet is currently enabled, last run 54 seconds ago with 0 failures [16:37:47] RECOVERY - puppet last run on ms-be3004 is OK: OK: Puppet is currently enabled, last run 32 seconds ago with 0 failures [16:37:47] RECOVERY - puppet last run on cp3011 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [16:37:47] RECOVERY - puppet last run on amssq50 is OK: OK: Puppet is currently enabled, last run 3 seconds ago with 0 failures [16:37:47] RECOVERY - puppet last run on cp3017 is OK: OK: Puppet is currently enabled, last run 1 second ago with 0 failures [16:37:47] RECOVERY - puppet last run on mw1015 is OK: OK: Puppet is currently enabled, last run 30 seconds ago with 0 failures [16:37:48] RECOVERY - puppet last run on mw1070 is OK: OK: Puppet is currently enabled, last run 18 seconds ago with 0 failures [16:37:48] RECOVERY - puppet last run on mw1128 is OK: OK: Puppet is currently enabled, last run 50 seconds ago with 0 failures [16:37:49] RECOVERY - puppet last run on elastic1009 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [16:37:57] RECOVERY - puppet last run on analytics1021 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [16:37:57] RECOVERY - puppet last run on mw1018 is OK: OK: Puppet is currently enabled, last run 53 seconds ago with 0 failures [16:37:57] RECOVERY - puppet last run on mw1078 is OK: OK: Puppet is currently enabled, last run 35 seconds ago with 0 failures [16:37:57] RECOVERY - puppet last run on ms-be1014 is OK: OK: Puppet is currently enabled, last run 48 seconds ago with 0 failures [16:37:58] RECOVERY - puppet last run on lanthanum is OK: OK: Puppet is currently enabled, last run 46 seconds ago with 0 failures [16:37:58] RECOVERY - puppet last run on cp1064 is OK: OK: Puppet is currently enabled, last run 19 seconds ago with 0 failures [16:38:07] RECOVERY - puppet last run on mw1179 is OK: OK: Puppet is currently enabled, last run 30 seconds ago with 0 failures [16:38:08] RECOVERY - puppet last run on lvs2005 is OK: OK: Puppet is currently enabled, last run 47 seconds ago with 0 failures [16:38:09] RECOVERY - puppet last run on db1019 is OK: OK: Puppet is currently enabled, last run 9 seconds ago with 0 failures [16:38:09] RECOVERY - puppet last run on mw1020 is OK: OK: Puppet is currently enabled, last run 2 seconds ago with 0 failures [16:38:09] RECOVERY - puppet last run on hydrogen is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [16:38:17] RECOVERY - puppet last run on db1009 is OK: OK: Puppet is currently enabled, last run 57 seconds ago with 0 failures [16:38:17] RECOVERY - puppet last run on labsdb1002 is OK: OK: Puppet is currently enabled, last run 52 seconds ago with 0 failures [16:38:18] RECOVERY - puppet last run on mw1095 is OK: OK: Puppet is currently enabled, last run 32 seconds ago with 0 failures [16:38:18] RECOVERY - puppet last run on mw1102 is OK: OK: Puppet is currently enabled, last run 49 seconds ago with 0 failures [16:38:18] RECOVERY - puppet last run on virt1009 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [16:38:18] RECOVERY - puppet last run on db1068 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [16:38:18] RECOVERY - puppet last run on mw1230 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [16:38:27] RECOVERY - puppet last run on elastic1016 is OK: OK: Puppet is currently enabled, last run 12 seconds ago with 0 failures [16:38:27] RECOVERY - puppet last run on mw1017 is OK: OK: Puppet is currently enabled, last run 27 seconds ago with 0 failures [16:38:27] RECOVERY - puppet last run on mw1075 is OK: OK: Puppet is currently enabled, last run 36 seconds ago with 0 failures [16:38:27] RECOVERY - puppet last run on mw1101 is OK: OK: Puppet is currently enabled, last run 49 seconds ago with 0 failures [16:38:28] RECOVERY - puppet last run on ms-be2009 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [16:38:28] RECOVERY - puppet last run on lvs1006 is OK: OK: Puppet is currently enabled, last run 29 seconds ago with 0 failures [16:38:28] RECOVERY - puppet last run on ocg1002 is OK: OK: Puppet is currently enabled, last run 46 seconds ago with 0 failures [16:38:28] RECOVERY - puppet last run on mw1191 is OK: OK: Puppet is currently enabled, last run 3 seconds ago with 0 failures [16:38:29] RECOVERY - puppet last run on es1003 is OK: OK: Puppet is currently enabled, last run 24 seconds ago with 0 failures [16:38:37] RECOVERY - puppet last run on db1058 is OK: OK: Puppet is currently enabled, last run 10 seconds ago with 0 failures [16:38:38] RECOVERY - puppet last run on cp4011 is OK: OK: Puppet is currently enabled, last run 39 seconds ago with 0 failures [16:38:38] RECOVERY - puppet last run on mw1169 is OK: OK: Puppet is currently enabled, last run 29 seconds ago with 0 failures [16:38:38] RECOVERY - puppet last run on erbium is OK: OK: Puppet is currently enabled, last run 36 seconds ago with 0 failures [16:38:38] RECOVERY - puppet last run on wtp1014 is OK: OK: Puppet is currently enabled, last run 50 seconds ago with 0 failures [16:38:38] RECOVERY - puppet last run on cp4015 is OK: OK: Puppet is currently enabled, last run 5 seconds ago with 0 failures [16:38:39] RECOVERY - puppet last run on cp4010 is OK: OK: Puppet is currently enabled, last run 46 seconds ago with 0 failures [16:38:47] RECOVERY - puppet last run on mw1019 is OK: OK: Puppet is currently enabled, last run 27 seconds ago with 0 failures [16:38:48] RECOVERY - puppet last run on lvs3002 is OK: OK: Puppet is currently enabled, last run 46 seconds ago with 0 failures [16:38:57] RECOVERY - puppet last run on mw1214 is OK: OK: Puppet is currently enabled, last run 6 seconds ago with 0 failures [16:38:58] RECOVERY - puppet last run on analytics1029 is OK: OK: Puppet is currently enabled, last run 49 seconds ago with 0 failures [16:38:58] RECOVERY - puppet last run on tmh1002 is OK: OK: Puppet is currently enabled, last run 34 seconds ago with 0 failures [16:38:58] RECOVERY - puppet last run on analytics1024 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [16:38:58] RECOVERY - puppet last run on amssq45 is OK: OK: Puppet is currently enabled, last run 48 seconds ago with 0 failures [16:38:58] RECOVERY - puppet last run on cp3019 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [16:38:59] RECOVERY - puppet last run on cp3021 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [16:38:59] RECOVERY - puppet last run on es1005 is OK: OK: Puppet is currently enabled, last run 56 seconds ago with 0 failures [16:39:00] RECOVERY - puppet last run on cp3007 is OK: OK: Puppet is currently enabled, last run 59 seconds ago with 0 failures [16:39:00] RECOVERY - puppet last run on amssq52 is OK: OK: Puppet is currently enabled, last run 6 seconds ago with 0 failures [16:39:07] RECOVERY - puppet last run on mw1058 is OK: OK: Puppet is currently enabled, last run 56 seconds ago with 0 failures [16:39:07] RECOVERY - puppet last run on labsdb1007 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [16:39:08] RECOVERY - puppet last run on mw1157 is OK: OK: Puppet is currently enabled, last run 53 seconds ago with 0 failures [16:39:08] RECOVERY - puppet last run on praseodymium is OK: OK: Puppet is currently enabled, last run 45 seconds ago with 0 failures [16:39:17] RECOVERY - puppet last run on mw1085 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [16:39:18] RECOVERY - puppet last run on db1041 is OK: OK: Puppet is currently enabled, last run 30 seconds ago with 0 failures [16:39:18] RECOVERY - puppet last run on mw1182 is OK: OK: Puppet is currently enabled, last run 56 seconds ago with 0 failures [16:39:18] RECOVERY - puppet last run on ms-be1005 is OK: OK: Puppet is currently enabled, last run 45 seconds ago with 0 failures [16:39:18] RECOVERY - puppet last run on cp1057 is OK: OK: Puppet is currently enabled, last run 35 seconds ago with 0 failures [16:39:38] RECOVERY - puppet last run on analytics1019 is OK: OK: Puppet is currently enabled, last run 3 seconds ago with 0 failures [16:39:38] RECOVERY - puppet last run on mw1094 is OK: OK: Puppet is currently enabled, last run 30 seconds ago with 0 failures [16:39:38] RECOVERY - puppet last run on cp1051 is OK: OK: Puppet is currently enabled, last run 27 seconds ago with 0 failures [16:39:38] RECOVERY - puppet last run on mw1184 is OK: OK: Puppet is currently enabled, last run 49 seconds ago with 0 failures [16:39:38] RECOVERY - puppet last run on cp1065 is OK: OK: Puppet is currently enabled, last run 22 seconds ago with 0 failures [16:39:38] RECOVERY - puppet last run on stat1001 is OK: OK: Puppet is currently enabled, last run 13 seconds ago with 0 failures [16:39:39] RECOVERY - puppet last run on mw1136 is OK: OK: Puppet is currently enabled, last run 30 seconds ago with 0 failures [16:39:47] RECOVERY - puppet last run on analytics1034 is OK: OK: Puppet is currently enabled, last run 17 seconds ago with 0 failures [16:39:47] RECOVERY - puppet last run on mw1127 is OK: OK: Puppet is currently enabled, last run 42 seconds ago with 0 failures [16:39:47] RECOVERY - puppet last run on db2041 is OK: OK: Puppet is currently enabled, last run 6 seconds ago with 0 failures [16:39:48] RECOVERY - puppet last run on ms-be2015 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [16:39:57] RECOVERY - puppet last run on ms-be2010 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [16:39:57] RECOVERY - puppet last run on cp4017 is OK: OK: Puppet is currently enabled, last run 20 seconds ago with 0 failures [16:39:58] RECOVERY - puppet last run on lvs4001 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [16:39:58] RECOVERY - puppet last run on analytics1036 is OK: OK: Puppet is currently enabled, last run 12 seconds ago with 0 failures [16:40:07] RECOVERY - puppet last run on cp4013 is OK: OK: Puppet is currently enabled, last run 53 seconds ago with 0 failures [16:40:07] RECOVERY - puppet last run on db1010 is OK: OK: Puppet is currently enabled, last run 14 seconds ago with 0 failures [16:40:08] RECOVERY - puppet last run on cp1069 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [16:40:08] RECOVERY - puppet last run on zirconium is OK: OK: Puppet is currently enabled, last run 39 seconds ago with 0 failures [16:40:08] RECOVERY - puppet last run on labsdb1001 is OK: OK: Puppet is currently enabled, last run 3 seconds ago with 0 failures [16:40:17] RECOVERY - puppet last run on db1024 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [16:40:18] RECOVERY - puppet last run on mw1232 is OK: OK: Puppet is currently enabled, last run 31 seconds ago with 0 failures [16:40:18] RECOVERY - puppet last run on amssq57 is OK: OK: Puppet is currently enabled, last run 1 second ago with 0 failures [16:40:18] RECOVERY - puppet last run on amslvs4 is OK: OK: Puppet is currently enabled, last run 55 seconds ago with 0 failures [16:40:18] RECOVERY - puppet last run on virt1012 is OK: OK: Puppet is currently enabled, last run 15 seconds ago with 0 failures [16:40:18] RECOVERY - puppet last run on mw1083 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [16:40:27] RECOVERY - puppet last run on mw1234 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [16:40:27] RECOVERY - puppet last run on mw1124 is OK: OK: Puppet is currently enabled, last run 5 seconds ago with 0 failures [16:40:27] RECOVERY - puppet last run on mc1008 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [16:40:27] RECOVERY - puppet last run on elastic1028 is OK: OK: Puppet is currently enabled, last run 8 seconds ago with 0 failures [16:40:27] RECOVERY - puppet last run on mw1244 is OK: OK: Puppet is currently enabled, last run 15 seconds ago with 0 failures [16:40:28] RECOVERY - puppet last run on mw1013 is OK: OK: Puppet is currently enabled, last run 22 seconds ago with 0 failures [16:40:28] RECOVERY - puppet last run on cp1043 is OK: OK: Puppet is currently enabled, last run 44 seconds ago with 0 failures [16:40:28] RECOVERY - puppet last run on mw1096 is OK: OK: Puppet is currently enabled, last run 44 seconds ago with 0 failures [16:40:37] RECOVERY - puppet last run on es1006 is OK: OK: Puppet is currently enabled, last run 7 seconds ago with 0 failures [16:40:37] RECOVERY - puppet last run on mw1196 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [16:40:37] RECOVERY - puppet last run on wtp1024 is OK: OK: Puppet is currently enabled, last run 50 seconds ago with 0 failures [16:40:38] RECOVERY - puppet last run on mc1010 is OK: OK: Puppet is currently enabled, last run 27 seconds ago with 0 failures [16:40:38] RECOVERY - puppet last run on mw1252 is OK: OK: Puppet is currently enabled, last run 15 seconds ago with 0 failures [16:40:47] RECOVERY - puppet last run on mw1035 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [16:40:47] RECOVERY - puppet last run on db1007 is OK: OK: Puppet is currently enabled, last run 53 seconds ago with 0 failures [16:40:47] RECOVERY - puppet last run on mw1246 is OK: OK: Puppet is currently enabled, last run 51 seconds ago with 0 failures [16:40:48] RECOVERY - puppet last run on mw1161 is OK: OK: Puppet is currently enabled, last run 8 seconds ago with 0 failures [16:40:48] RECOVERY - puppet last run on mw1036 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [16:40:48] RECOVERY - puppet last run on mc1011 is OK: OK: Puppet is currently enabled, last run 14 seconds ago with 0 failures [16:40:58] RECOVERY - puppet last run on ytterbium is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [16:40:58] RECOVERY - puppet last run on mw1257 is OK: OK: Puppet is currently enabled, last run 37 seconds ago with 0 failures [16:40:58] RECOVERY - puppet last run on db2035 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [16:40:59] RECOVERY - puppet last run on analytics1039 is OK: OK: Puppet is currently enabled, last run 47 seconds ago with 0 failures [16:40:59] RECOVERY - puppet last run on mw1218 is OK: OK: Puppet is currently enabled, last run 38 seconds ago with 0 failures [16:41:08] RECOVERY - puppet last run on wtp1021 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [16:41:08] RECOVERY - puppet last run on mw1138 is OK: OK: Puppet is currently enabled, last run 56 seconds ago with 0 failures [16:41:17] RECOVERY - puppet last run on analytics1015 is OK: OK: Puppet is currently enabled, last run 40 seconds ago with 0 failures [16:41:17] RECOVERY - puppet last run on mw1245 is OK: OK: Puppet is currently enabled, last run 38 seconds ago with 0 failures [16:41:17] RECOVERY - puppet last run on mw1216 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [16:41:18] RECOVERY - puppet last run on virt1002 is OK: OK: Puppet is currently enabled, last run 42 seconds ago with 0 failures [16:41:27] RECOVERY - puppet last run on ms-fe1003 is OK: OK: Puppet is currently enabled, last run 23 seconds ago with 0 failures [16:41:27] RECOVERY - puppet last run on elastic1026 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [16:41:27] RECOVERY - puppet last run on pc1001 is OK: OK: Puppet is currently enabled, last run 10 seconds ago with 0 failures [16:41:28] RECOVERY - puppet last run on mw1221 is OK: OK: Puppet is currently enabled, last run 55 seconds ago with 0 failures [16:41:28] RECOVERY - puppet last run on cp3022 is OK: OK: Puppet is currently enabled, last run 4 seconds ago with 0 failures [16:41:28] RECOVERY - puppet last run on amssq58 is OK: OK: Puppet is currently enabled, last run 52 seconds ago with 0 failures [16:41:37] RECOVERY - puppet last run on mw1040 is OK: OK: Puppet is currently enabled, last run 7 seconds ago with 0 failures [16:41:37] RECOVERY - puppet last run on wtp1019 is OK: OK: Puppet is currently enabled, last run 52 seconds ago with 0 failures [16:41:37] RECOVERY - puppet last run on mw1109 is OK: OK: Puppet is currently enabled, last run 13 seconds ago with 0 failures [16:41:38] RECOVERY - puppet last run on es1009 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [16:41:38] RECOVERY - puppet last run on ms-be1013 is OK: OK: Puppet is currently enabled, last run 44 seconds ago with 0 failures [16:41:38] RECOVERY - puppet last run on mc1009 is OK: OK: Puppet is currently enabled, last run 16 seconds ago with 0 failures [16:41:38] RECOVERY - puppet last run on wtp1017 is OK: OK: Puppet is currently enabled, last run 18 seconds ago with 0 failures [16:41:38] RECOVERY - puppet last run on ms-be1001 is OK: OK: Puppet is currently enabled, last run 22 seconds ago with 0 failures [16:41:47] RECOVERY - puppet last run on mw1031 is OK: OK: Puppet is currently enabled, last run 7 seconds ago with 0 failures [16:41:47] RECOVERY - puppet last run on mw1130 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [16:41:47] RECOVERY - puppet last run on mw1038 is OK: OK: Puppet is currently enabled, last run 1 second ago with 0 failures [16:41:48] RECOVERY - puppet last run on mw1147 is OK: OK: Puppet is currently enabled, last run 53 seconds ago with 0 failures [16:41:48] RECOVERY - puppet last run on rdb1004 is OK: OK: Puppet is currently enabled, last run 42 seconds ago with 0 failures [16:41:58] RECOVERY - puppet last run on es1001 is OK: OK: Puppet is currently enabled, last run 20 seconds ago with 0 failures [16:41:58] RECOVERY - puppet last run on logstash1003 is OK: OK: Puppet is currently enabled, last run 39 seconds ago with 0 failures [16:41:58] RECOVERY - puppet last run on mc1004 is OK: OK: Puppet is currently enabled, last run 48 seconds ago with 0 failures [16:42:07] RECOVERY - puppet last run on mw1178 is OK: OK: Puppet is currently enabled, last run 27 seconds ago with 0 failures [16:42:07] RECOVERY - puppet last run on ocg1001 is OK: OK: Puppet is currently enabled, last run 33 seconds ago with 0 failures [16:42:07] RECOVERY - puppet last run on mw1115 is OK: OK: Puppet is currently enabled, last run 22 seconds ago with 0 failures [16:42:08] RECOVERY - puppet last run on cp1059 is OK: OK: Puppet is currently enabled, last run 39 seconds ago with 0 failures [16:42:08] RECOVERY - puppet last run on mw1132 is OK: OK: Puppet is currently enabled, last run 47 seconds ago with 0 failures [16:42:08] RECOVERY - puppet last run on elastic1031 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [16:42:17] RECOVERY - puppet last run on es2003 is OK: OK: Puppet is currently enabled, last run 52 seconds ago with 0 failures [16:42:17] RECOVERY - puppet last run on ms-be2013 is OK: OK: Puppet is currently enabled, last run 9 seconds ago with 0 failures [16:42:18] RECOVERY - puppet last run on ms-be2002 is OK: OK: Puppet is currently enabled, last run 10 seconds ago with 0 failures [16:42:18] RECOVERY - puppet last run on mw1028 is OK: OK: Puppet is currently enabled, last run 50 seconds ago with 0 failures [16:42:18] RECOVERY - puppet last run on mc1016 is OK: OK: Puppet is currently enabled, last run 44 seconds ago with 0 failures [16:42:18] RECOVERY - puppet last run on uranium is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [16:42:28] RECOVERY - puppet last run on cp4007 is OK: OK: Puppet is currently enabled, last run 28 seconds ago with 0 failures [16:42:28] RECOVERY - puppet last run on mw1240 is OK: OK: Puppet is currently enabled, last run 18 seconds ago with 0 failures [16:42:28] RECOVERY - puppet last run on mw1089 is OK: OK: Puppet is currently enabled, last run 49 seconds ago with 0 failures [16:42:28] RECOVERY - puppet last run on dbstore1001 is OK: OK: Puppet is currently enabled, last run 58 seconds ago with 0 failures [16:42:37] RECOVERY - puppet last run on magnesium is OK: OK: Puppet is currently enabled, last run 15 seconds ago with 0 failures [16:42:37] RECOVERY - puppet last run on mw1072 is OK: OK: Puppet is currently enabled, last run 27 seconds ago with 0 failures [16:42:37] RECOVERY - puppet last run on mw1134 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [16:42:37] RECOVERY - puppet last run on vanadium is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [16:42:38] RECOVERY - puppet last run on amssq59 is OK: OK: Puppet is currently enabled, last run 22 seconds ago with 0 failures [16:42:38] RECOVERY - puppet last run on cp3006 is OK: OK: Puppet is currently enabled, last run 25 seconds ago with 0 failures [16:42:38] RECOVERY - puppet last run on cp3013 is OK: OK: Puppet is currently enabled, last run 40 seconds ago with 0 failures [16:42:47] RECOVERY - puppet last run on mw1048 is OK: OK: Puppet is currently enabled, last run 46 seconds ago with 0 failures [16:42:47] RECOVERY - puppet last run on mw1005 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [16:42:47] RECOVERY - puppet last run on es1008 is OK: OK: Puppet is currently enabled, last run 3 seconds ago with 0 failures [16:42:47] RECOVERY - puppet last run on wtp1009 is OK: OK: Puppet is currently enabled, last run 40 seconds ago with 0 failures [16:42:48] RECOVERY - puppet last run on mw1256 is OK: OK: Puppet is currently enabled, last run 43 seconds ago with 0 failures [16:42:48] RECOVERY - puppet last run on dbproxy1002 is OK: OK: Puppet is currently enabled, last run 2 seconds ago with 0 failures [16:42:48] RECOVERY - puppet last run on mw1080 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [16:42:57] RECOVERY - puppet last run on analytics1017 is OK: OK: Puppet is currently enabled, last run 7 seconds ago with 0 failures [16:42:57] RECOVERY - puppet last run on db1053 is OK: OK: Puppet is currently enabled, last run 47 seconds ago with 0 failures [16:42:58] RECOVERY - puppet last run on logstash1001 is OK: OK: Puppet is currently enabled, last run 47 seconds ago with 0 failures [16:43:07] RECOVERY - puppet last run on platinum is OK: OK: Puppet is currently enabled, last run 8 seconds ago with 0 failures [16:43:07] RECOVERY - puppet last run on carbon is OK: OK: Puppet is currently enabled, last run 55 seconds ago with 0 failures [16:43:07] RECOVERY - puppet last run on mw1067 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [16:43:07] RECOVERY - puppet last run on db1065 is OK: OK: Puppet is currently enabled, last run 55 seconds ago with 0 failures [16:43:08] RECOVERY - puppet last run on netmon1001 is OK: OK: Puppet is currently enabled, last run 36 seconds ago with 0 failures [16:43:08] RECOVERY - puppet last run on potassium is OK: OK: Puppet is currently enabled, last run 9 seconds ago with 0 failures [16:43:17] RECOVERY - puppet last run on db1029 is OK: OK: Puppet is currently enabled, last run 31 seconds ago with 0 failures [16:43:17] RECOVERY - puppet last run on mw1041 is OK: OK: Puppet is currently enabled, last run 29 seconds ago with 0 failures [16:43:17] RECOVERY - puppet last run on cp1049 is OK: OK: Puppet is currently enabled, last run 8 seconds ago with 0 failures [16:43:17] RECOVERY - puppet last run on xenon is OK: OK: Puppet is currently enabled, last run 4 seconds ago with 0 failures [16:43:17] RECOVERY - puppet last run on mw1233 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [16:43:18] RECOVERY - puppet last run on ms-be1006 is OK: OK: Puppet is currently enabled, last run 25 seconds ago with 0 failures [16:43:18] RECOVERY - puppet last run on mw1140 is OK: OK: Puppet is currently enabled, last run 41 seconds ago with 0 failures [16:43:18] RECOVERY - puppet last run on elastic1007 is OK: OK: Puppet is currently enabled, last run 8 seconds ago with 0 failures [16:43:18] RECOVERY - puppet last run on db2034 is OK: OK: Puppet is currently enabled, last run 28 seconds ago with 0 failures [16:43:19] RECOVERY - puppet last run on cp1070 is OK: OK: Puppet is currently enabled, last run 19 seconds ago with 0 failures [16:43:27] RECOVERY - puppet last run on es2005 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [16:43:27] RECOVERY - puppet last run on labstore2001 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [16:43:27] RECOVERY - puppet last run on db2039 is OK: OK: Puppet is currently enabled, last run 10 seconds ago with 0 failures [16:43:27] RECOVERY - puppet last run on caesium is OK: OK: Puppet is currently enabled, last run 43 seconds ago with 0 failures [16:43:27] RECOVERY - puppet last run on ms-be2003 is OK: OK: Puppet is currently enabled, last run 9 seconds ago with 0 failures [16:43:28] RECOVERY - puppet last run on analytics1033 is OK: OK: Puppet is currently enabled, last run 45 seconds ago with 0 failures [16:43:28] RECOVERY - puppet last run on ms-be1003 is OK: OK: Puppet is currently enabled, last run 42 seconds ago with 0 failures [16:43:28] RECOVERY - puppet last run on lvs4002 is OK: OK: Puppet is currently enabled, last run 42 seconds ago with 0 failures [16:43:37] RECOVERY - puppet last run on mw1045 is OK: OK: Puppet is currently enabled, last run 39 seconds ago with 0 failures [16:43:37] RECOVERY - puppet last run on ms-be1004 is OK: OK: Puppet is currently enabled, last run 52 seconds ago with 0 failures [16:43:38] RECOVERY - puppet last run on wtp1006 is OK: OK: Puppet is currently enabled, last run 26 seconds ago with 0 failures [16:43:47] RECOVERY - puppet last run on elastic1001 is OK: OK: Puppet is currently enabled, last run 29 seconds ago with 0 failures [16:43:47] RECOVERY - puppet last run on mw1006 is OK: OK: Puppet is currently enabled, last run 25 seconds ago with 0 failures [16:43:48] RECOVERY - puppet last run on helium is OK: OK: Puppet is currently enabled, last run 46 seconds ago with 0 failures [16:43:48] RECOVERY - puppet last run on mc1006 is OK: OK: Puppet is currently enabled, last run 53 seconds ago with 0 failures [16:43:48] RECOVERY - puppet last run on analytics1025 is OK: OK: Puppet is currently enabled, last run 37 seconds ago with 0 failures [16:43:57] RECOVERY - puppet last run on mw1187 is OK: OK: Puppet is currently enabled, last run 0 seconds ago with 0 failures [16:43:57] RECOVERY - puppet last run on cp3020 is OK: OK: Puppet is currently enabled, last run 23 seconds ago with 0 failures [16:43:57] RECOVERY - puppet last run on mw1197 is OK: OK: Puppet is currently enabled, last run 44 seconds ago with 0 failures [16:43:57] RECOVERY - puppet last run on amslvs2 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [16:43:57] RECOVERY - puppet last run on amssq44 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [16:43:57] RECOVERY - puppet last run on amssq43 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [16:43:58] RECOVERY - puppet last run on cp3008 is OK: OK: Puppet is currently enabled, last run 37 seconds ago with 0 failures [16:43:58] RECOVERY - puppet last run on amssq54 is OK: OK: Puppet is currently enabled, last run 27 seconds ago with 0 failures [16:43:59] RECOVERY - puppet last run on cp3015 is OK: OK: Puppet is currently enabled, last run 58 seconds ago with 0 failures [16:43:59] RECOVERY - puppet last run on mw1007 is OK: OK: Puppet is currently enabled, last run 57 seconds ago with 0 failures [16:44:00] RECOVERY - puppet last run on mw1141 is OK: OK: Puppet is currently enabled, last run 39 seconds ago with 0 failures [16:44:00] RECOVERY - puppet last run on wtp1020 is OK: OK: Puppet is currently enabled, last run 33 seconds ago with 0 failures [16:44:01] RECOVERY - puppet last run on mw1145 is OK: OK: Puppet is currently enabled, last run 36 seconds ago with 0 failures [16:44:01] RECOVERY - puppet last run on cerium is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [16:44:01] RECOVERY - puppet last run on neptunium is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [16:44:02] RECOVERY - puppet last run on analytics1041 is OK: OK: Puppet is currently enabled, last run 44 seconds ago with 0 failures [16:44:03] RECOVERY - puppet last run on mw1063 is OK: OK: Puppet is currently enabled, last run 45 seconds ago with 0 failures [16:44:03] RECOVERY - puppet last run on snapshot1003 is OK: OK: Puppet is currently enabled, last run 42 seconds ago with 0 failures [16:44:04] RECOVERY - puppet last run on mw1012 is OK: OK: Puppet is currently enabled, last run 52 seconds ago with 0 failures [16:44:04] RECOVERY - puppet last run on analytics1020 is OK: OK: Puppet is currently enabled, last run 39 seconds ago with 0 failures [16:44:07] RECOVERY - puppet last run on mw1059 is OK: OK: Puppet is currently enabled, last run 57 seconds ago with 0 failures [16:44:07] RECOVERY - puppet last run on gold is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [16:44:07] RECOVERY - puppet last run on elastic1004 is OK: OK: Puppet is currently enabled, last run 44 seconds ago with 0 failures [16:44:07] RECOVERY - puppet last run on searchidx1001 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [16:44:08] RECOVERY - puppet last run on wtp1016 is OK: OK: Puppet is currently enabled, last run 15 seconds ago with 0 failures [16:44:08] RECOVERY - puppet last run on mw1254 is OK: OK: Puppet is currently enabled, last run 25 seconds ago with 0 failures [16:44:08] RECOVERY - puppet last run on mw1250 is OK: OK: Puppet is currently enabled, last run 11 seconds ago with 0 failures [16:44:08] RECOVERY - puppet last run on db1031 is OK: OK: Puppet is currently enabled, last run 44 seconds ago with 0 failures [16:44:09] RECOVERY - puppet last run on db1066 is OK: OK: Puppet is currently enabled, last run 18 seconds ago with 0 failures [16:44:09] RECOVERY - puppet last run on mc1002 is OK: OK: Puppet is currently enabled, last run 20 seconds ago with 0 failures [16:44:10] RECOVERY - puppet last run on cp1055 is OK: OK: Puppet is currently enabled, last run 27 seconds ago with 0 failures [16:44:17] RECOVERY - puppet last run on db1022 is OK: OK: Puppet is currently enabled, last run 31 seconds ago with 0 failures [16:44:18] RECOVERY - puppet last run on lvs1005 is OK: OK: Puppet is currently enabled, last run 1 second ago with 0 failures [16:44:18] RECOVERY - puppet last run on lvs1002 is OK: OK: Puppet is currently enabled, last run 33 seconds ago with 0 failures [16:44:18] RECOVERY - puppet last run on cp1047 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [16:44:27] RECOVERY - puppet last run on mw1160 is OK: OK: Puppet is currently enabled, last run 27 seconds ago with 0 failures [16:44:27] RECOVERY - puppet last run on mw1200 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [16:44:27] RECOVERY - puppet last run on rbf1002 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [16:44:27] RECOVERY - puppet last run on mw1082 is OK: OK: Puppet is currently enabled, last run 35 seconds ago with 0 failures [16:44:27] RECOVERY - puppet last run on db1073 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [16:44:27] RECOVERY - puppet last run on analytics1040 is OK: OK: Puppet is currently enabled, last run 57 seconds ago with 0 failures [16:44:28] RECOVERY - puppet last run on mw1026 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [16:44:28] RECOVERY - puppet last run on mw1176 is OK: OK: Puppet is currently enabled, last run 5 seconds ago with 0 failures [16:44:29] RECOVERY - puppet last run on cp1039 is OK: OK: Puppet is currently enabled, last run 58 seconds ago with 0 failures [16:44:29] RECOVERY - puppet last run on sca1002 is OK: OK: Puppet is currently enabled, last run 49 seconds ago with 0 failures [16:44:30] RECOVERY - puppet last run on mw1226 is OK: OK: Puppet is currently enabled, last run 16 seconds ago with 0 failures [16:44:30] RECOVERY - puppet last run on mw1224 is OK: OK: Puppet is currently enabled, last run 47 seconds ago with 0 failures [16:44:37] RECOVERY - puppet last run on ms-fe2004 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [16:44:37] RECOVERY - puppet last run on mw1106 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [16:44:37] RECOVERY - puppet last run on mw1217 is OK: OK: Puppet is currently enabled, last run 23 seconds ago with 0 failures [16:44:37] RECOVERY - puppet last run on capella is OK: OK: Puppet is currently enabled, last run 4 seconds ago with 0 failures [16:44:37] RECOVERY - puppet last run on ms-fe1001 is OK: OK: Puppet is currently enabled, last run 21 seconds ago with 0 failures [16:44:38] RECOVERY - puppet last run on db1050 is OK: OK: Puppet is currently enabled, last run 21 seconds ago with 0 failures [16:44:38] RECOVERY - puppet last run on heze is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [16:44:38] RECOVERY - puppet last run on db2005 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [16:44:39] RECOVERY - puppet last run on db2019 is OK: OK: Puppet is currently enabled, last run 59 seconds ago with 0 failures [16:44:39] RECOVERY - puppet last run on es2001 is OK: OK: Puppet is currently enabled, last run 32 seconds ago with 0 failures [16:44:40] RECOVERY - puppet last run on es2002 is OK: OK: Puppet is currently enabled, last run 43 seconds ago with 0 failures [16:44:40] RECOVERY - puppet last run on analytics1035 is OK: OK: Puppet is currently enabled, last run 16 seconds ago with 0 failures [16:44:41] RECOVERY - puppet last run on es2008 is OK: OK: Puppet is currently enabled, last run 57 seconds ago with 0 failures [16:44:41] RECOVERY - puppet last run on ms-be2004 is OK: OK: Puppet is currently enabled, last run 4 seconds ago with 0 failures [16:44:42] RECOVERY - puppet last run on db2009 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [16:44:42] RECOVERY - puppet last run on db2002 is OK: OK: Puppet is currently enabled, last run 30 seconds ago with 0 failures [16:44:43] RECOVERY - puppet last run on elastic1018 is OK: OK: Puppet is currently enabled, last run 12 seconds ago with 0 failures [16:44:43] RECOVERY - puppet last run on ms-be2006 is OK: OK: Puppet is currently enabled, last run 22 seconds ago with 0 failures [16:44:47] RECOVERY - puppet last run on iron is OK: OK: Puppet is currently enabled, last run 12 seconds ago with 0 failures [16:44:48] RECOVERY - puppet last run on mw1174 is OK: OK: Puppet is currently enabled, last run 50 seconds ago with 0 failures [16:44:48] RECOVERY - puppet last run on cp4006 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [16:44:57] RECOVERY - puppet last run on mw1189 is OK: OK: Puppet is currently enabled, last run 10 seconds ago with 0 failures [16:44:57] RECOVERY - puppet last run on mw1228 is OK: OK: Puppet is currently enabled, last run 33 seconds ago with 0 failures [16:44:57] RECOVERY - puppet last run on mw1153 is OK: OK: Puppet is currently enabled, last run 29 seconds ago with 0 failures [16:44:58] RECOVERY - puppet last run on mw1008 is OK: OK: Puppet is currently enabled, last run 9 seconds ago with 0 failures [16:44:59] EVERYTHING IS GREAT AND NOTHING IS WRONG. [16:45:07] RECOVERY - puppet last run on mw1120 is OK: OK: Puppet is currently enabled, last run 41 seconds ago with 0 failures [16:45:07] RECOVERY - puppet last run on labsdb1003 is OK: OK: Puppet is currently enabled, last run 10 seconds ago with 0 failures [16:45:07] RECOVERY - puppet last run on mw1222 is OK: OK: Puppet is currently enabled, last run 34 seconds ago with 0 failures [16:45:07] RECOVERY - puppet last run on lvs3001 is OK: OK: Puppet is currently enabled, last run 56 seconds ago with 0 failures [16:45:07] RECOVERY - puppet last run on amssq32 is OK: OK: Puppet is currently enabled, last run 8 seconds ago with 0 failures [16:45:08] RECOVERY - puppet last run on amssq61 is OK: OK: Puppet is currently enabled, last run 11 seconds ago with 0 failures [16:45:08] RECOVERY - puppet last run on amssq53 is OK: OK: Puppet is currently enabled, last run 38 seconds ago with 0 failures [16:45:08] RECOVERY - puppet last run on amssq49 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [16:45:08] RECOVERY - puppet last run on mw1060 is OK: OK: Puppet is currently enabled, last run 24 seconds ago with 0 failures [16:45:09] RECOVERY - puppet last run on labstore1003 is OK: OK: Puppet is currently enabled, last run 42 seconds ago with 0 failures [16:45:09] RECOVERY - puppet last run on mc1003 is OK: OK: Puppet is currently enabled, last run 48 seconds ago with 0 failures [16:45:17] RECOVERY - puppet last run on elastic1021 is OK: OK: Puppet is currently enabled, last run 58 seconds ago with 0 failures [16:45:17] RECOVERY - puppet last run on mw1242 is OK: OK: Puppet is currently enabled, last run 28 seconds ago with 0 failures [16:45:18] RECOVERY - puppet last run on mw1100 is OK: OK: Puppet is currently enabled, last run 57 seconds ago with 0 failures [16:45:18] RECOVERY - puppet last run on mw1205 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [16:45:18] RECOVERY - puppet last run on analytics1030 is OK: OK: Puppet is currently enabled, last run 31 seconds ago with 0 failures [16:45:18] RECOVERY - puppet last run on elastic1008 is OK: OK: Puppet is currently enabled, last run 39 seconds ago with 0 failures [16:45:28] RECOVERY - puppet last run on mw1099 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [16:45:29] RECOVERY - puppet last run on mw1068 is OK: OK: Puppet is currently enabled, last run 8 seconds ago with 0 failures [16:45:29] RECOVERY - puppet last run on mw1088 is OK: OK: Puppet is currently enabled, last run 28 seconds ago with 0 failures [16:45:29] RECOVERY - puppet last run on mw1144 is OK: OK: Puppet is currently enabled, last run 13 seconds ago with 0 failures [16:45:37] RECOVERY - puppet last run on db1059 is OK: OK: Puppet is currently enabled, last run 7 seconds ago with 0 failures [16:45:37] RECOVERY - puppet last run on mw1150 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [16:45:37] RECOVERY - puppet last run on db1015 is OK: OK: Puppet is currently enabled, last run 22 seconds ago with 0 failures [16:45:37] RECOVERY - puppet last run on elastic1027 is OK: OK: Puppet is currently enabled, last run 29 seconds ago with 0 failures [16:45:37] RECOVERY - puppet last run on ms-fe1004 is OK: OK: Puppet is currently enabled, last run 18 seconds ago with 0 failures [16:45:38] RECOVERY - puppet last run on mw1009 is OK: OK: Puppet is currently enabled, last run 44 seconds ago with 0 failures [16:45:38] RECOVERY - puppet last run on mw1046 is OK: OK: Puppet is currently enabled, last run 23 seconds ago with 0 failures [16:45:38] RECOVERY - puppet last run on db1018 is OK: OK: Puppet is currently enabled, last run 59 seconds ago with 0 failures [16:45:39] RECOVERY - puppet last run on mw1164 is OK: OK: Puppet is currently enabled, last run 50 seconds ago with 0 failures [16:45:39] RECOVERY - puppet last run on mw1235 is OK: OK: Puppet is currently enabled, last run 2 seconds ago with 0 failures [16:45:39] RECOVERY - puppet last run on lead is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [16:45:40] RECOVERY - puppet last run on cp1056 is OK: OK: Puppet is currently enabled, last run 38 seconds ago with 0 failures [16:45:41] RECOVERY - puppet last run on elastic1012 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [16:45:47] RECOVERY - puppet last run on virt1006 is OK: OK: Puppet is currently enabled, last run 16 seconds ago with 0 failures [16:45:47] RECOVERY - puppet last run on mw1123 is OK: OK: Puppet is currently enabled, last run 21 seconds ago with 0 failures [16:45:47] RECOVERY - puppet last run on mw1069 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [16:45:48] RECOVERY - puppet last run on mw1173 is OK: OK: Puppet is currently enabled, last run 46 seconds ago with 0 failures [16:45:48] RECOVERY - puppet last run on db2018 is OK: OK: Puppet is currently enabled, last run 20 seconds ago with 0 failures [16:45:48] RECOVERY - puppet last run on ms-fe2001 is OK: OK: Puppet is currently enabled, last run 19 seconds ago with 0 failures [16:45:48] RECOVERY - puppet last run on es2009 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [16:45:57] RECOVERY - puppet last run on mw1003 is OK: OK: Puppet is currently enabled, last run 42 seconds ago with 0 failures [16:45:58] RECOVERY - puppet last run on cp1061 is OK: OK: Puppet is currently enabled, last run 44 seconds ago with 0 failures [16:45:58] RECOVERY - puppet last run on elastic1022 is OK: OK: Puppet is currently enabled, last run 14 seconds ago with 0 failures [16:45:58] RECOVERY - puppet last run on db1046 is OK: OK: Puppet is currently enabled, last run 49 seconds ago with 0 failures [16:45:58] RECOVERY - puppet last run on db1040 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [16:46:08] RECOVERY - puppet last run on mw1170 is OK: OK: Puppet is currently enabled, last run 17 seconds ago with 0 failures [16:46:08] RECOVERY - puppet last run on db1002 is OK: OK: Puppet is currently enabled, last run 43 seconds ago with 0 failures [16:46:08] RECOVERY - puppet last run on ruthenium is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [16:46:17] RECOVERY - puppet last run on mw1117 is OK: OK: Puppet is currently enabled, last run 59 seconds ago with 0 failures [16:46:17] RECOVERY - puppet last run on amssq35 is OK: OK: Puppet is currently enabled, last run 54 seconds ago with 0 failures [16:46:18] RECOVERY - puppet last run on amslvs1 is OK: OK: Puppet is currently enabled, last run 10 seconds ago with 0 failures [16:46:18] RECOVERY - puppet last run on cp3003 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [16:46:18] RECOVERY - puppet last run on cp3016 is OK: OK: Puppet is currently enabled, last run 17 seconds ago with 0 failures [16:46:18] RECOVERY - puppet last run on mw1042 is OK: OK: Puppet is currently enabled, last run 16 seconds ago with 0 failures [16:46:18] RECOVERY - puppet last run on db1051 is OK: OK: Puppet is currently enabled, last run 30 seconds ago with 0 failures [16:46:18] RECOVERY - puppet last run on mw1119 is OK: OK: Puppet is currently enabled, last run 51 seconds ago with 0 failures [16:46:19] RECOVERY - puppet last run on db1042 is OK: OK: Puppet is currently enabled, last run 8 seconds ago with 0 failures [16:46:19] RECOVERY - puppet last run on mw1065 is OK: OK: Puppet is currently enabled, last run 6 seconds ago with 0 failures [16:46:27] RECOVERY - puppet last run on analytics1010 is OK: OK: Puppet is currently enabled, last run 28 seconds ago with 0 failures [16:46:27] RECOVERY - puppet last run on db1067 is OK: OK: Puppet is currently enabled, last run 7 seconds ago with 0 failures [16:46:27] RECOVERY - puppet last run on logstash1002 is OK: OK: Puppet is currently enabled, last run 19 seconds ago with 0 failures [16:46:28] RECOVERY - puppet last run on mw1118 is OK: OK: Puppet is currently enabled, last run 55 seconds ago with 0 failures [16:46:28] RECOVERY - puppet last run on mw1166 is OK: OK: Puppet is currently enabled, last run 17 seconds ago with 0 failures [16:46:28] RECOVERY - puppet last run on mw1025 is OK: OK: Puppet is currently enabled, last run 16 seconds ago with 0 failures [16:46:37] RECOVERY - puppet last run on lvs2004 is OK: OK: Puppet is currently enabled, last run 46 seconds ago with 0 failures [16:46:38] RECOVERY - puppet last run on mw1129 is OK: OK: Puppet is currently enabled, last run 2 seconds ago with 0 failures [16:46:47] RECOVERY - puppet last run on db1043 is OK: OK: Puppet is currently enabled, last run 9 seconds ago with 0 failures [16:46:47] RECOVERY - puppet last run on virt1001 is OK: OK: Puppet is currently enabled, last run 2 seconds ago with 0 failures [16:46:47] RECOVERY - puppet last run on db1021 is OK: OK: Puppet is currently enabled, last run 55 seconds ago with 0 failures [16:46:47] RECOVERY - puppet last run on labstore1001 is OK: OK: Puppet is currently enabled, last run 8 seconds ago with 0 failures [16:46:48] RECOVERY - puppet last run on mw1092 is OK: OK: Puppet is currently enabled, last run 50 seconds ago with 0 failures [16:46:48] RECOVERY - puppet last run on mw1054 is OK: OK: Puppet is currently enabled, last run 6 seconds ago with 0 failures [16:46:48] RECOVERY - puppet last run on db1034 is OK: OK: Puppet is currently enabled, last run 23 seconds ago with 0 failures [16:46:48] RECOVERY - puppet last run on dbproxy1001 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [16:46:49] RECOVERY - puppet last run on mw1052 is OK: OK: Puppet is currently enabled, last run 50 seconds ago with 0 failures [16:46:57] RECOVERY - puppet last run on mw1011 is OK: OK: Puppet is currently enabled, last run 8 seconds ago with 0 failures [16:46:58] RECOVERY - puppet last run on db2036 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [16:46:58] RECOVERY - puppet last run on ms-fe2003 is OK: OK: Puppet is currently enabled, last run 51 seconds ago with 0 failures [16:46:58] RECOVERY - puppet last run on db2007 is OK: OK: Puppet is currently enabled, last run 3 seconds ago with 0 failures [16:46:58] RECOVERY - puppet last run on db2040 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [16:46:59] RECOVERY - puppet last run on db2042 is OK: OK: Puppet is currently enabled, last run 12 seconds ago with 0 failures [16:46:59] RECOVERY - puppet last run on labcontrol2001 is OK: OK: Puppet is currently enabled, last run 52 seconds ago with 0 failures [16:46:59] RECOVERY - puppet last run on mw1002 is OK: OK: Puppet is currently enabled, last run 27 seconds ago with 0 failures [16:47:07] RECOVERY - puppet last run on tin is OK: OK: Puppet is currently enabled, last run 20 seconds ago with 0 failures [16:47:07] RECOVERY - puppet last run on mw1175 is OK: OK: Puppet is currently enabled, last run 15 seconds ago with 0 failures [16:47:08] RECOVERY - puppet last run on wtp1005 is OK: OK: Puppet is currently enabled, last run 14 seconds ago with 0 failures [16:47:08] RECOVERY - puppet last run on db1028 is OK: OK: Puppet is currently enabled, last run 55 seconds ago with 0 failures [16:47:08] RECOVERY - puppet last run on cp4003 is OK: OK: Puppet is currently enabled, last run 41 seconds ago with 0 failures [16:47:08] RECOVERY - puppet last run on mw1061 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [16:47:17] RECOVERY - puppet last run on cp4008 is OK: OK: Puppet is currently enabled, last run 36 seconds ago with 0 failures [16:47:18] RECOVERY - puppet last run on db1023 is OK: OK: Puppet is currently enabled, last run 50 seconds ago with 0 failures [16:47:18] RECOVERY - puppet last run on db1052 is OK: OK: Puppet is currently enabled, last run 45 seconds ago with 0 failures [16:47:18] RECOVERY - puppet last run on mw1172 is OK: OK: Puppet is currently enabled, last run 36 seconds ago with 0 failures [16:47:18] RECOVERY - puppet last run on labnet1001 is OK: OK: Puppet is currently enabled, last run 29 seconds ago with 0 failures [16:47:28] RECOVERY - puppet last run on pc1002 is OK: OK: Puppet is currently enabled, last run 29 seconds ago with 0 failures [16:47:28] RECOVERY - puppet last run on es1007 is OK: OK: Puppet is currently enabled, last run 40 seconds ago with 0 failures [16:47:28] RECOVERY - puppet last run on amssq46 is OK: OK: Puppet is currently enabled, last run 10 seconds ago with 0 failures [16:47:28] RECOVERY - puppet last run on amssq48 is OK: OK: Puppet is currently enabled, last run 17 seconds ago with 0 failures [16:47:28] RECOVERY - puppet last run on amssq60 is OK: OK: Puppet is currently enabled, last run 22 seconds ago with 0 failures [16:47:28] RECOVERY - puppet last run on amssq47 is OK: OK: Puppet is currently enabled, last run 18 seconds ago with 0 failures [16:47:28] RECOVERY - puppet last run on cp3014 is OK: OK: Puppet is currently enabled, last run 39 seconds ago with 0 failures [16:47:29] RECOVERY - puppet last run on mw1126 is OK: OK: Puppet is currently enabled, last run 33 seconds ago with 0 failures [16:47:37] RECOVERY - puppet last run on elastic1030 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [16:47:37] RECOVERY - puppet last run on mw1211 is OK: OK: Puppet is currently enabled, last run 46 seconds ago with 0 failures [16:47:37] RECOVERY - puppet last run on analytics1038 is OK: OK: Puppet is currently enabled, last run 5 seconds ago with 0 failures [16:47:37] RECOVERY - puppet last run on mw1114 is OK: OK: Puppet is currently enabled, last run 28 seconds ago with 0 failures [16:47:37] RECOVERY - puppet last run on mw1213 is OK: OK: Puppet is currently enabled, last run 41 seconds ago with 0 failures [16:47:38] RECOVERY - puppet last run on virt1003 is OK: OK: Puppet is currently enabled, last run 21 seconds ago with 0 failures [16:47:38] RECOVERY - puppet last run on mw1039 is OK: OK: Puppet is currently enabled, last run 34 seconds ago with 0 failures [16:47:48] RECOVERY - puppet last run on db1003 is OK: OK: Puppet is currently enabled, last run 40 seconds ago with 0 failures [16:47:48] RECOVERY - puppet last run on mw1177 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [16:47:48] RECOVERY - puppet last run on mw1162 is OK: OK: Puppet is currently enabled, last run 30 seconds ago with 0 failures [16:47:48] RECOVERY - puppet last run on mw1249 is OK: OK: Puppet is currently enabled, last run 12 seconds ago with 0 failures [16:47:48] RECOVERY - puppet last run on lvs2001 is OK: OK: Puppet is currently enabled, last run 47 seconds ago with 0 failures [16:47:49] RECOVERY - puppet last run on mw1251 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [16:47:49] RECOVERY - puppet last run on db1016 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [16:47:49] RECOVERY - puppet last run on gallium is OK: OK: Puppet is currently enabled, last run 42 seconds ago with 0 failures [16:47:50] RECOVERY - puppet last run on analytics1016 is OK: OK: Puppet is currently enabled, last run 10 seconds ago with 0 failures [16:47:57] RECOVERY - puppet last run on stat1003 is OK: OK: Puppet is currently enabled, last run 49 seconds ago with 0 failures [16:47:57] RECOVERY - puppet last run on polonium is OK: OK: Puppet is currently enabled, last run 46 seconds ago with 0 failures [16:47:57] RECOVERY - puppet last run on dataset1001 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [16:47:57] RECOVERY - puppet last run on mc1012 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [16:48:07] RECOVERY - puppet last run on snapshot1001 is OK: OK: Puppet is currently enabled, last run 51 seconds ago with 0 failures [16:48:07] RECOVERY - puppet last run on cp1058 is OK: OK: Puppet is currently enabled, last run 45 seconds ago with 0 failures [16:48:08] RECOVERY - puppet last run on wtp1012 is OK: OK: Puppet is currently enabled, last run 34 seconds ago with 0 failures [16:48:08] RECOVERY - puppet last run on db1048 is OK: OK: Puppet is currently enabled, last run 21 seconds ago with 0 failures [16:48:08] RECOVERY - puppet last run on lithium is OK: OK: Puppet is currently enabled, last run 40 seconds ago with 0 failures [16:48:08] RECOVERY - puppet last run on db2038 is OK: OK: Puppet is currently enabled, last run 33 seconds ago with 0 failures [16:48:08] RECOVERY - puppet last run on db2029 is OK: OK: Puppet is currently enabled, last run 40 seconds ago with 0 failures [16:48:08] RECOVERY - puppet last run on ms-be2011 is OK: OK: Puppet is currently enabled, last run 14 seconds ago with 0 failures [16:48:17] RECOVERY - puppet last run on nembus is OK: OK: Puppet is currently enabled, last run 33 seconds ago with 0 failures [16:48:17] RECOVERY - puppet last run on install2001 is OK: OK: Puppet is currently enabled, last run 20 seconds ago with 0 failures [16:48:17] RECOVERY - puppet last run on cp4014 is OK: OK: Puppet is currently enabled, last run 51 seconds ago with 0 failures [16:48:27] RECOVERY - puppet last run on cp4004 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [16:48:27] RECOVERY - puppet last run on virt1004 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [16:48:37] RECOVERY - puppet last run on mw1237 is OK: OK: Puppet is currently enabled, last run 38 seconds ago with 0 failures [16:48:37] RECOVERY - puppet last run on mw1202 is OK: OK: Puppet is currently enabled, last run 4 seconds ago with 0 failures [16:48:38] RECOVERY - puppet last run on db1036 is OK: OK: Puppet is currently enabled, last run 33 seconds ago with 0 failures [16:48:38] RECOVERY - puppet last run on antimony is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [16:48:38] RECOVERY - puppet last run on mw1195 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [16:48:38] RECOVERY - puppet last run on ms-be3002 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [16:48:38] RECOVERY - puppet last run on cp3010 is OK: OK: Puppet is currently enabled, last run 44 seconds ago with 0 failures [16:48:39] RECOVERY - puppet last run on amssq55 is OK: OK: Puppet is currently enabled, last run 57 seconds ago with 0 failures [16:48:47] RECOVERY - puppet last run on mw1044 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [16:48:47] RECOVERY - puppet last run on mw1151 is OK: OK: Puppet is currently enabled, last run 2 seconds ago with 0 failures [16:48:47] RECOVERY - puppet last run on mc1014 is OK: OK: Puppet is currently enabled, last run 28 seconds ago with 0 failures [16:48:48] RECOVERY - puppet last run on db1026 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [16:48:48] RECOVERY - puppet last run on mw1055 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [16:48:58] RECOVERY - puppet last run on mw1208 is OK: OK: Puppet is currently enabled, last run 59 seconds ago with 0 failures [16:49:07] RECOVERY - puppet last run on mw1049 is OK: OK: Puppet is currently enabled, last run 5 seconds ago with 0 failures [16:49:07] RECOVERY - puppet last run on elastic1024 is OK: OK: Puppet is currently enabled, last run 50 seconds ago with 0 failures [16:49:08] RECOVERY - puppet last run on mw1206 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [16:49:08] RECOVERY - puppet last run on mw1238 is OK: OK: Puppet is currently enabled, last run 27 seconds ago with 0 failures [16:49:17] RECOVERY - puppet last run on db1039 is OK: OK: Puppet is currently enabled, last run 46 seconds ago with 0 failures [16:49:17] RECOVERY - puppet last run on mw1084 is OK: OK: Puppet is currently enabled, last run 28 seconds ago with 0 failures [16:49:17] RECOVERY - puppet last run on mc1005 is OK: OK: Puppet is currently enabled, last run 59 seconds ago with 0 failures [16:49:17] RECOVERY - puppet last run on mw1076 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [16:49:17] RECOVERY - puppet last run on mw1149 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [16:49:18] RECOVERY - puppet last run on analytics1026 is OK: OK: Puppet is currently enabled, last run 40 seconds ago with 0 failures [16:49:27] RECOVERY - puppet last run on db2016 is OK: OK: Puppet is currently enabled, last run 51 seconds ago with 0 failures [16:49:28] RECOVERY - puppet last run on lvs4003 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [16:49:28] RECOVERY - puppet last run on cp4019 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [16:49:37] RECOVERY - puppet last run on cp1050 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [16:49:37] RECOVERY - puppet last run on cp4001 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [16:49:47] RECOVERY - puppet last run on lvs3004 is OK: OK: Puppet is currently enabled, last run 55 seconds ago with 0 failures [16:49:48] RECOVERY - puppet last run on gadolinium is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [16:49:57] RECOVERY - puppet last run on amssq34 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [16:50:38] RECOVERY - puppet last run on acamar is OK: OK: Puppet is currently enabled, last run 30 seconds ago with 0 failures [16:51:07] RECOVERY - puppet last run on amssq56 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [16:52:55] (03PS1) 10Faidon Liambotis: Hardcode geolocation for Level3's DNS clusters [dns] - 10https://gerrit.wikimedia.org/r/185994 [17:32:01] (03PS1) 10Glaisher: Add 'movefile' right to 'autopatrolled' group on frwiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/185997 (https://phabricator.wikimedia.org/T87145) [17:33:54] (03CR) 10Jforrester: "Ping. :-)" [puppet] - 10https://gerrit.wikimedia.org/r/184904 (https://phabricator.wikimedia.org/T76949) (owner: 10Jforrester) [17:41:38] (03PS1) 10Jforrester: Add the Citoid extension to use the citoid service to Beta Labs [mediawiki-config] - 10https://gerrit.wikimedia.org/r/186001 [17:42:18] (03CR) 10Jforrester: [C: 04-1] "Depends on Ib238012e." [mediawiki-config] - 10https://gerrit.wikimedia.org/r/186001 (owner: 10Jforrester) [18:06:24] _joe_: jzerebecki hoo|away everything OK with the dumps? I see a lot of backlog [18:08:10] _joe_: what do you mean with backlog? [18:08:43] (03PS2) 10BBlack: Hardcode geolocation for Level3's DNS clusters [dns] - 10https://gerrit.wikimedia.org/r/185994 (owner: 10Faidon Liambotis) [18:09:10] (03CR) 10CSteipp: [C: 031] "I haven't used it in a while. If we need it again, we can redeploy it." [mediawiki-config] - 10https://gerrit.wikimedia.org/r/185923 (owner: 10Reedy) [18:17:12] (03PS3) 10BBlack: Hardcode geolocation for Level3's DNS clusters [dns] - 10https://gerrit.wikimedia.org/r/185994 (owner: 10Faidon Liambotis) [18:19:32] akosiaris: backlog? [18:22:17] (03PS4) 10BBlack: Hardcode geolocation for Level3's DNS clusters [dns] - 10https://gerrit.wikimedia.org/r/185994 (owner: 10Faidon Liambotis) [18:23:03] (03CR) 10BBlack: "I think the Washington ones are Washingon, DC, so I moved those to eqiad. Also I replaced "eqiad, ulsfo" with just "eqiad"; currently it " [dns] - 10https://gerrit.wikimedia.org/r/185994 (owner: 10Faidon Liambotis) [18:28:58] hoo: IRC backlog. I see _joe_ said the dumpscript finished with a 0 return code so it ran successfully but still dumps are missing ? [18:29:27] akosiaris: I think the old version ran, before puppet put the new one in place [18:29:47] I'm now capturing hte logs [18:31:34] hoo: I am pretty sure puppet ran before I ran the script, but I may be wrong. I was jetlagged [18:31:49] Ok, we'll see [18:32:05] If I had my sudo access to datasets I could follow more closely [18:32:13] right now I can only capture the logs [18:32:20] but not the temporary files it creates [18:45:59] 3ops-core: Evaluation Virtualization management platforms for a private VPS cluster - https://phabricator.wikimedia.org/T87251#986973 (10akosiaris) 3NEW a:3akosiaris [18:57:41] 3Project-Creators, operations: Create #site-incident tag and use it for incident reports - https://phabricator.wikimedia.org/T85889#987000 (10Aklapper) >>! In T85889#966759, @greg wrote: > meta-comment: I want to make sure #Ops is on board with any change given the incident report process came out of their group... [19:01:24] 3ops-core, operations, Wikimedia-Git-or-Gerrit: Chrome warns about insecure certificate on gerrit.wikimedia.org - https://phabricator.wikimedia.org/T76562#987012 (10RobH) Mark: I assigned https://rt.wikimedia.org/Ticket/Display.html?id=9130 to you for your approval. [19:08:19] (03CR) 10Hoo man: "This has been open for almost two months now... any news?" [puppet] - 10https://gerrit.wikimedia.org/r/174896 (owner: 10Hoo man) [19:13:05] !log reedy Started scap: Add magic word for pl [19:13:13] Logged the message, Master [19:15:36] (03CR) 10BBlack: [C: 031] Hardcode geolocation for Level3's DNS clusters [dns] - 10https://gerrit.wikimedia.org/r/185994 (owner: 10Faidon Liambotis) [19:22:53] (03PS5) 10Faidon Liambotis: Hardcode geolocation for Level3's DNS clusters [dns] - 10https://gerrit.wikimedia.org/r/185994 [19:26:50] 3Project-Creators, operations: Create #site-incident tag and use it for incident reports - https://phabricator.wikimedia.org/T85889#987106 (10GWicke) @aklapper, we did bring it up & heard some positive noises, but I think a final decision was not yet made. @greg should know more. [19:27:26] !log reedy Finished scap: Add magic word for pl (duration: 14m 21s) [19:27:33] Logged the message, Master [19:27:45] Wut [19:27:47] That.. worked? [19:28:43] :D [19:28:59] Reedy: it does work, it seems. here's a test page on plwp: https://pl.wikipedia.org/wiki/Wikipedysta:Tufor/brudnopis [19:29:24] woo, reliable deployment process! [19:29:37] 3ops-core: Introduce Virtualization in our infrastructure - https://phabricator.wikimedia.org/T87258#987116 (10akosiaris) 3NEW a:3akosiaris [19:29:53] 3ops-core: Evaluation Virtualization management platforms for a private VPS cluster - https://phabricator.wikimedia.org/T87251#987125 (10akosiaris) [19:29:54] 3ops-core: Introduce Virtualization in our infrastructure - https://phabricator.wikimedia.org/T87258#987124 (10akosiaris) [19:31:27] 3operations: Remove sventura from jimmy@ alias - https://phabricator.wikimedia.org/T87259#987126 (10Chip) 3NEW [19:39:26] 3operations: Remove sventura from jimmy@ alias - https://phabricator.wikimedia.org/T87259#987162 (10yuvipanda) 5Open>3Resolved a:3yuvipanda Done with 53a5a28aa7a5b28bfefa72ea9add3eaa6e434446 in ops/private [19:39:43] (03CR) 10BBlack: [C: 031] Hardcode geolocation for Level3's DNS clusters [dns] - 10https://gerrit.wikimedia.org/r/185994 (owner: 10Faidon Liambotis) [19:40:34] 3operations: Create projects for Ops goals - https://phabricator.wikimedia.org/T87262#987174 (10chasemp) [19:44:29] 3operations, ops-core: Set up the mediawiki application layer in codfw - https://phabricator.wikimedia.org/T86894#987185 (10Joe) [19:48:24] (03CR) 10Ori.livneh: "Although this has +1s from Tim and Daniel, because this involves a change to the security scheme in production, I think either Mark or Fai" [puppet] - 10https://gerrit.wikimedia.org/r/174896 (owner: 10Hoo man) [19:49:21] Reedy, greg-g: Can we grab a deployment window right now? A couple of regressions we want to push out. [19:49:34] (One in VisualEditor, one in TemplateData.) [19:50:07] * hoo tries to resist making a joke about pushing regressions [19:50:13] xD [19:50:28] James_F: Go for it [19:50:42] you want to do this Reedy? [19:50:47] if not I can [19:50:47] I can [19:50:51] Reedy, your gregness? [19:50:51] I don't mind if you want to [19:51:00] I don't mind either :| [19:51:18] Both of you, stop being so British. :-) [19:51:27] Krenair: After you [19:51:35] * YuviPanda pours tea for Krenair, Reedy and James_F [19:51:47] Is it Green? Does it have caffeine? [19:51:53] Tea? Pfft. [19:51:53] APPARENTLY [19:51:58] (03CR) 10Ori.livneh: "and, obviously, CSteipp -- I didn't realize he hand't reviewed this." [puppet] - 10https://gerrit.wikimedia.org/r/174896 (owner: 10Hoo man) [19:52:00] csteipp: ^ [19:52:10] Krenair: https://gerrit.wikimedia.org/r/#/q/owner:Jforrester+status:open+-branch:master+-branch:production,n,z [19:52:53] 3operations, ops-core: Setup redis clusters in codfw - https://phabricator.wikimedia.org/T86887#987211 (10Joe) [19:53:09] Reedy: you gotta be careful, most tea is decaf here for some reason D: [19:53:28] 3ops-codfw: Procure and setup rdb2001-2004 - https://phabricator.wikimedia.org/T86896#987222 (10Joe) [19:53:47] 3operations, ops-core: Check that the redis roles can be applied in codfw, set up puppet. - https://phabricator.wikimedia.org/T86898#987223 (10Joe) [19:55:05] 3operations, ops-core: Setup memcached cluster in codfw - https://phabricator.wikimedia.org/T86888#987234 (10Joe) [19:55:23] 3operations, ops-core: Setup jobrunners cluster in codfw - https://phabricator.wikimedia.org/T86889#987236 (10Joe) [19:55:52] 3operations, ops-core: Setup imagescalers cluster in codfw - https://phabricator.wikimedia.org/T86890#987238 (10Joe) [19:55:56] _joe_, mark: EventLogging in codfw as well? [19:56:10] 3operations, ops-core: Setup videoscalers cluster in codfw - https://phabricator.wikimedia.org/T86891#987240 (10Joe) [19:56:33] 3operations, ops-core: Setup the api appservers cluster in codfw - https://phabricator.wikimedia.org/T86892#987242 (10Joe) [19:56:53] 3operations, ops-core: Setup the main appservers cluster in codfw - https://phabricator.wikimedia.org/T86893#987246 (10Joe) [19:56:55] James_F, urgh, I have to do that thing with the submodules [19:58:16] Krenair: Yeah. :-) [19:58:55] lol. [20:01:37] 3operations: Identify deficiencies in Nimsoft Cloud User Experience Monitor (formerly WatchMouse) - https://phabricator.wikimedia.org/T85829#987252 (10Dzahn) [20:03:04] hoo: we could ask for a hard link to the dump being created, so we have it even if it gets deleted :) [20:03:19] or the tmp files at least [20:03:38] jzerebecki: If I had my access, I could just copy the temp files [20:03:53] * jzerebecki is sad to even suggest such hackish solutions [20:04:17] hoo: they would not be finished yet, so of no use, right? [20:04:42] No, they're not, I guess [20:06:03] hoo: but if we had hard links, then the full copy would be still there even if they will be deleted... [20:07:16] feel free to request that, if you think it makes sense, i'm out now [20:09:37] <_joe_> ori: that was not in the scope for now, (which is this quarter), but we both need your input and your collaboration :) [20:10:20] jzerebecki: mh... I guess it will just run through this time [20:10:29] pretty certain, actually [20:14:28] Reedy, James_F: sigh. git is behaving strangely, my core submodules for the deployment branch are broken somehow [20:14:29] Krenair: You doing OK? Need help? [20:14:34] Ha. Timing. [20:15:29] Krenair: Need me to do the sub-module commits? [20:16:06] yes please [20:19:10] Git has decided that 5 extension submodules, 1 of which is VE, have been made dirty. despite there being no changes [20:21:45] !log deleted localnames/localuser entries in centralauth db that referred to vewikimedia (https://phabricator.wikimedia.org/T87264) [20:21:49] Logged the message, Master [20:26:53] James_F, having the same fun with submodules? [20:32:22] James_F, I need to go for a bit, back soon. [20:43:27] James_F, back [20:50:41] Krenair: Sorry, got dragged away. [20:51:14] I still don't know why git thinks these submodules are dirty [20:53:10] "Versions 1.7.0 and later of git contain an annoying change in the behavior of git submodule. Submodules are now regarded as dirty if they have any modified files or untracked files, whereas previously it would only be the case if HEAD in the submodule pointed to the wrong commit." urgh, this is the problem [20:53:47] --init --recursive [20:53:54] git submodule foreach git reset HEAD --hard [20:54:16] (03CR) 10CSteipp: "I definitely like enabling the ability to drop privileges. Do we run anything else as the generic nobody, that this might give access to?" [puppet] - 10https://gerrit.wikimedia.org/r/174896 (owner: 10Hoo man) [20:54:44] Reedy: Lovely. [20:56:44] James_F, aha, I think I got it [20:57:05] Krenair: Awesome. Running into a meeting now… [20:57:11] ok [21:03:51] (03CR) 10Faidon Liambotis: [C: 04-1] "Yeah, that's why I haven't merged this... We have various things running as nobody, including Ganglia (gmond). Until we move these to syst" [puppet] - 10https://gerrit.wikimedia.org/r/174896 (owner: 10Hoo man) [21:05:08] (03CR) 10Faidon Liambotis: [C: 032] Hardcode geolocation for Level3's DNS clusters [dns] - 10https://gerrit.wikimedia.org/r/185994 (owner: 10Faidon Liambotis) [21:08:29] Reedy, hey. I assume it's still safe to sync-file for those VE changes, right? [21:09:03] (03CR) 10Mattflaschen: Enable 'copyedit' GettingStarted suggestions for ptwiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/185122 (https://phabricator.wikimedia.org/T86590) (owner: 10Mattflaschen) [21:10:03] Krenair: should be [21:10:33] (03PS1) 10Faidon Liambotis: Minor fix for Level3's subnets [dns] - 10https://gerrit.wikimedia.org/r/186082 [21:10:51] (03CR) 10Faidon Liambotis: [C: 032] Minor fix for Level3's subnets [dns] - 10https://gerrit.wikimedia.org/r/186082 (owner: 10Faidon Liambotis) [21:11:03] !log krenair Synchronized php-1.25wmf14/extensions/VisualEditor/modules/ve-mw/ui/pages/ve.ui.MWSettingsPage.js: https://gerrit.wikimedia.org/r/#/c/185992/1 (duration: 00m 07s) [21:11:10] Logged the message, Master [21:12:11] thanks Krenair [21:13:44] waiting for jenkins for wmf15... [21:15:19] (03CR) 10Dzahn: [C: 031] Redirect wikibook(s).(org|com) to www.wikibooks.org [puppet] - 10https://gerrit.wikimedia.org/r/185474 (https://phabricator.wikimedia.org/T87039) (owner: 10Glaisher) [21:16:42] !log krenair Synchronized php-1.25wmf15/extensions/VisualEditor/modules/ve-mw/ui/pages/ve.ui.MWSettingsPage.js: https://gerrit.wikimedia.org/r/#/c/185991/1 (duration: 00m 05s) [21:16:47] Logged the message, Master [21:17:04] that fixed it [21:20:17] PROBLEM - Slow CirrusSearch query rate on fluorine is CRITICAL: CirrusSearch-slow.log_line_rate CRITICAL: 0.00333333333333 [21:20:18] Reedy, greg-g, mooeypoo: ok, now the TemplateData ones he also wanted [21:25:11] (03PS1) 10BBlack: varnish systemd service stuff [puppet] - 10https://gerrit.wikimedia.org/r/186085 [21:30:31] (03CR) 10BBlack: [C: 032] "This is probably broken, but it should only break the jessie boxes, and then I can see it and iterate :)" [puppet] - 10https://gerrit.wikimedia.org/r/186085 (owner: 10BBlack) [21:30:37] RECOVERY - Slow CirrusSearch query rate on fluorine is OK: CirrusSearch-slow.log_line_rate OKAY: 0.0 [21:32:46] (03CR) 10Hoo man: "gmond seems to be run by user ganglia (uid 999). But for the other things... is there a ticket about migrating away from nobody?" [puppet] - 10https://gerrit.wikimedia.org/r/174896 (owner: 10Hoo man) [21:34:44] (03CR) 10Alexandros Kosiaris: "It's gmetad, not gmond. I am unaware of a ticket though" [puppet] - 10https://gerrit.wikimedia.org/r/174896 (owner: 10Hoo man) [21:39:09] !log krenair Synchronized php-1.25wmf15/extensions/TemplateData/modules/ext.templateDataGenerator.ui.tdDialog.js: https://gerrit.wikimedia.org/r/#/c/185998/ (duration: 00m 07s) [21:39:14] Logged the message, Master [21:41:44] hmm [21:44:11] mooeypoo, poke [21:46:07] ahh the fix did work, must've not refreshed enough to see it :) [21:48:26] !log krenair Synchronized php-1.25wmf14/extensions/TemplateData/modules/ext.templateDataGenerator.ui.tdDialog.js: https://gerrit.wikimedia.org/r/#/c/185999/ (duration: 00m 05s) [21:48:29] Logged the message, Master [21:48:32] all done I think James_F|Away [22:02:52] (03PS1) 10QChris: Bump alert thresholds for EventLogging's overall events/s [puppet] - 10https://gerrit.wikimedia.org/r/186091 [22:12:53] greg-g, do we still have a standing Flow deploy window now? If so, can we use it today? [22:16:16] superm401: https://wikitech.wikimedia.org/wiki/Deployments#Week_of_19th aka "no" [22:16:31] superm401: unless there's an Unbreak Now! issue [22:19:24] 3operations: Remove the auto +v for users joining -operations from WMF IP space - https://phabricator.wikimedia.org/T87270#987606 (10greg) 3NEW [22:27:52] 3operations, Wikimedia-Bugzilla: analyze Bugzilla access logs - https://phabricator.wikimedia.org/T86859#987629 (10Dzahn) [22:27:57] 3operations: Remove the auto +v for users joining -operations from WMF IP space - https://phabricator.wikimedia.org/T87270#987632 (10Krenair) It's this line: `-ChanServ- 5 *!*@*.wikimedia.org +V [modified 4y 19w 2d ago]` Maybe corp. should be excluded somehow [22:29:13] 3operations: Remove the auto +v for users joining -operations from WMF IP space - https://phabricator.wikimedia.org/T87270#987635 (10greg) >>! In T87270#987632, @Krenair wrote: > It's this line: > > `-ChanServ- 5 *!*@*.wikimedia.org +V [modified 4y 19w 2d ago]` > Why is that needed anyway? The bots com... [22:30:08] 3Wikimedia-Bugzilla, Phabricator, operations, ops-core: Sanitise a Bugzilla database dump - https://phabricator.wikimedia.org/T85141#987640 (10Dzahn) [22:31:58] greg-g, it's not an unbreak now. [22:32:00] However, I'm wondering if we could get a window for 3 PM to enable Flow on a couple pages on Portugese Wikipedia. [22:32:11] This is the first deployment of Flow to ptwiki, but they are aware and expecting it. [22:34:12] 3operations: Remove the auto +v for users joining -operations from WMF IP space - https://phabricator.wikimedia.org/T87270#987649 (10Krenair) I think a couple more bots used to run on production hosts, but now there's just wmf-insecte (gallium, not in -operations), icinga-wm (neon) and logmsgbot (neon) left. It... [22:34:48] superm401: is there any new code? [22:34:57] superm401: or is it just a config change? [22:35:17] greg-g, no, we would flip the switch to enable Flow, but only on two pages. [22:35:28] Just config [22:36:19] superm401, did I hear ptwiki? [22:36:21] o.O [22:36:27] If you need something, I can test :-) [22:37:59] superm401: ok, sure, 3pm is fine then [22:38:08] greg-g, okay, thanks. [22:38:09] superm401: please add to deploy calendar for record keeping, kthx :) [22:38:16] Yep, will do. [22:39:04] Krenair: Brill, thanks. [22:41:38] Helder, that means both the changes you requested are going out today (the other is GettingStarted copy-editing suggestions on ptwiki, which is in the SWAT window since GettingStarted is already deployed there). [22:41:54] <_joe_> are we doing SWAT today? [22:42:01] <_joe_> so, who should I set the pager to? [22:42:29] <_joe_> ;) [22:42:32] product duty person? /me hides [22:42:36] _joe_, yes, this is the only day this week for SWAT. [22:43:09] 3operations: Remove the auto +v for users joining -operations from WMF IP space - https://phabricator.wikimedia.org/T87270#987665 (10greg) /me nods We should explicitly whitelist them then. [22:43:10] Krenair, sorry, was at lunch and forgot to set myself away -- do you still need me? [22:43:49] _joe_: yeah, 'twas agreed by mar.k last week :) [22:44:03] _joe_: rest of the week is no deploys though [22:50:48] 3operations: Remove the auto +v for users joining -operations from WMF IP space - https://phabricator.wikimedia.org/T87270#987687 (10Dzahn) It would be useful if the channel became moderated because a human starts spamming or trolling. You could more easily block external users but let known employees talk. But... [22:52:54] YuviPanda: Could you please have a look at https://gerrit.wikimedia.org/r/#/c/186091/ ? [22:53:16] That would make the unneeded EventLogging Icinga warnings go away. [22:53:43] we're all in a meeting today [22:53:52] (03CR) 10Yuvipanda: [C: 032] Bump alert thresholds for EventLogging's overall events/s [puppet] - 10https://gerrit.wikimedia.org/r/186091 (owner: 10QChris) [22:54:01] mooeypoo, no, I dealt with it [22:54:07] Thanks YuviPanda! [22:54:08] looks harmless though :) heh [22:54:22] Thanks for mutante for checking too. [22:54:31] 3operations: Remove the auto +v for users joining -operations from WMF IP space - https://phabricator.wikimedia.org/T87270#987690 (10greg) >>! In T87270#987687, @Dzahn wrote: > It would be useful if the channel became moderated because a human starts spamming or trolling. You could more easily block external use... [22:55:23] qchris: done [22:55:36] Thanks YuviPanda. [22:55:54] (03PS1) 10Mattflaschen: Enable Flow on two pages on Portugese Wikipedia [mediawiki-config] - 10https://gerrit.wikimedia.org/r/186096 [22:56:48] 3ops-codfw: es2010 Failed Hard Drive - https://phabricator.wikimedia.org/T86588#987691 (10Papaul) 5Open>3Resolved Complete [22:57:41] greg-g, wmf guests in SF would get +v here as well [22:57:53] Krenair: but WHY?! [22:58:02] oh, I read "should" [22:58:03] :) [22:58:04] yeah [22:58:08] which doesn't make sense! [22:58:38] * greg-g feels like an extremely old man waving a cane at youngins walking by [22:58:51] I think when this rule was set, office might have been on a separate domain? [22:58:59] * greg-g shrugs [22:59:03] WMF guest is a different ISP from WMF proper [22:59:07] So it should be okay? [22:59:24] I remember wiki.static.monkeybrains or something when I visited as a guest [22:59:27] guest-tan1.corp.wikimedia.org ? [22:59:32] Something like that [22:59:40] I can join from xchat to test if you want [22:59:40] if somebody is at office and abuses IRC.. you have other means to telll them [22:59:50] there is already someone logged on from that domain [22:59:56] as opposed to a user from elsewhere [23:00:11] People from the office use IRC? [23:00:17] * YuviPanda runs away [23:00:17] to be honest I brought up guests because greg mentioned employees :) [23:00:45] office traffic actually goes to our DC ? [23:01:03] I'm on the guest network. [23:01:29] as long as you are not a Kaplan student [23:01:32] Mm. [23:01:41] Or a random person hanging out at our office [23:01:46] That's what the guest network is "for" [23:01:50] i think they have to sign in [23:02:47] ABUSE [23:02:54] WMF-Guest should definitely not have any special privilege by dint of IP address. [23:03:15] Okay, enabling Flow on ptwiki per window now. [23:03:28] does this mean I have to assume good faith until you actually abuse it? :P [23:04:55] (03CR) 10Mattflaschen: [C: 032] Enable Flow on two pages on Portugese Wikipedia [mediawiki-config] - 10https://gerrit.wikimedia.org/r/186096 (owner: 10Mattflaschen) [23:05:01] (03Merged) 10jenkins-bot: Enable Flow on two pages on Portugese Wikipedia [mediawiki-config] - 10https://gerrit.wikimedia.org/r/186096 (owner: 10Mattflaschen) [23:05:52] greg-g: [23:05:54] "The authenticity of host '[gerrit.wikimedia.org]:29418 ([2620:0:861:3:208:80:154:81]:29418)' can't be established." [23:05:59] Someone change the IP address or something? [23:06:11] superm401, I think it's domain rather than IP [23:06:14] 3operations: Remove the auto +v for users joining -operations from WMF IP space - https://phabricator.wikimedia.org/T87270#987708 (10greg) Just to keep the bug updated with IRC conversations: That rule also voices people on the WMF-Guest network: ``` 18:00 >>> AbusingIRCAtWMF!~mholmquis@guest-tan1... [23:06:16] ^ robh [23:06:46] ^d: did you do anything with gerrit in your gerrit->github investigations? [23:06:56] (03PS1) 10Kaldari: Turning WikiGrok off on en.wiki (test finished) [mediawiki-config] - 10https://gerrit.wikimedia.org/r/186097 [23:07:04] <^d> greg-g: Not yet, no. [23:07:38] ^d: see ssh auth issue from superm401 [23:07:53] <^d> I saw that reported like a week ago [23:07:57] <^d> I have nfc clue why [23:07:57] That's when connecting from tin. [23:09:05] superm401: you're still updating OK right? [23:09:22] That was when I tried to git pull into mediawiki-config. [23:09:31] I'm checking against known_hosts on my laptop now. [23:09:51] I had that as well [23:09:56] and a few other people [23:13:46] Are the ssh public keys in puppet? [23:14:32] For shell user accounts? yeah [23:15:40] No, for Gerrit. But still yes. [23:16:51] PROBLEM - Unmerged changes on repository mediawiki_config on tin is CRITICAL: There is one unmerged change in mediawiki_config (dir /srv/mediawiki-staging/). [23:20:17] (03PS1) 10BBlack: varnish systemd unit file: iterate on fixups [puppet] - 10https://gerrit.wikimedia.org/r/186098 [23:20:24] Doesn't match. [23:20:30] The one in Puppet has fingerprint 02:e5:90:a2:db:61:dd:d1:d4:e7:b6:51:0a:89:1f:78 [23:20:47] That might be paravoids fault [23:21:19] 3Analytics, operations: Upgrade Analytics Cluster to Trusty, and then to CDH 5.3 - https://phabricator.wikimedia.org/T1200#987766 (10Ottomata) [23:21:21] IIRC he was changing something to do with this [23:21:43] https://git.wikimedia.org/blob/operations%2Fpuppet/943bdc862ad9f7f7cd973c78da3b08636ee29b1f/manifests%2Frole%2Fgerrit.pp#L70 [23:21:44] superm401: git pull doesn't work for me on tin, it requires ssh agent forwarding(afaik). I usually just do `git pull readonly` followed by `git merge readonly/master`. Its the same repository but doesn't need auth [23:21:50] (03CR) 10BBlack: [C: 032] varnish systemd unit file: iterate on fixups [puppet] - 10https://gerrit.wikimedia.org/r/186098 (owner: 10BBlack) [23:22:43] 3operations: db1021 %iowait up - https://phabricator.wikimedia.org/T87277#987769 (10Springle) 3NEW a:3Springle [23:22:43] Okay, I'll try that. [23:24:51] RECOVERY - Unmerged changes on repository mediawiki_config on tin is OK: No changes to merge. [23:28:44] !log mattflaschen Synchronized wmf-config/InitialiseSettings.php: Enable Flow on ptwiki (duration: 00m 05s) [23:28:50] Logged the message, Master [23:29:37] Helder, would you care to make the first post? :) [23:30:28] (03PS1) 10BBlack: varnish systemd unit file: iterate on fixups #2 [puppet] - 10https://gerrit.wikimedia.org/r/186102 [23:30:38] jjj~.~. [23:30:48] ? [23:31:01] well, ~. means kill ssh, so :) [23:31:20] (03CR) 10BBlack: [C: 032 V: 032] varnish systemd unit file: iterate on fixups #2 [puppet] - 10https://gerrit.wikimedia.org/r/186102 (owner: 10BBlack) [23:32:23] yeah I was having connection problems from the wifi here to the host I run my irc screen session from [23:32:32] I guess my ~. that I never saw here made it through :) [23:32:39] andrewbogott: https://phabricator.wikimedia.org/T87220 [23:36:07] I got a weird network error the first time, but seems to be working fine now. [23:36:18] Done the 3 PM deployment window. [23:42:44] who's Guest38? :) [23:43:14] my bad [23:51:09] (03PS1) 10BBlack: varnish systemd unit file: iterate on fixups #3 [puppet] - 10https://gerrit.wikimedia.org/r/186106 [23:51:25] (03CR) 10BBlack: [C: 032 V: 032] varnish systemd unit file: iterate on fixups #3 [puppet] - 10https://gerrit.wikimedia.org/r/186106 (owner: 10BBlack) [23:54:11] (03PS1) 10BBlack: varnish systemd unit file: iterate on fixups #4 [puppet] - 10https://gerrit.wikimedia.org/r/186107 [23:54:27] (03CR) 10BBlack: [C: 032 V: 032] varnish systemd unit file: iterate on fixups #4 [puppet] - 10https://gerrit.wikimedia.org/r/186107 (owner: 10BBlack)