[00:00:04] RoanKattouw, ^d: Respected human, time to deploy Evening SWAT (Max 8 patches) (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20150121T0000). Please do the needful. [00:05:32] RECOVERY - puppet last run on amssq42 is OK: OK: Puppet is currently enabled, last run 14 seconds ago with 0 failures [00:13:01] PROBLEM - puppet last run on haedus is CRITICAL: CRITICAL: Puppet has 1 failures [00:13:20] ^d: go ahead with, https://gerrit.wikimedia.org/r/#/c/185987/ please :) [00:16:01] RoanKattouw: are you available? [00:16:16] I could also do it, if not. [00:16:20] kart_: I'm around [00:16:34] But I don't have time to do SWAT today, I have a meeting in 14 mins [00:16:52] yes. SWAT :) [00:17:12] I can do it. [00:17:21] superm401: nice. Thanks! [00:18:06] kart_, you only want to update CX on one branch? [00:18:50] superm401: CX is deployed in that branch only. [00:18:57] We did it last week, so... [00:19:08] Or am I missing something? [00:20:37] oh. we've 1.25wmf15 too. [00:21:54] lol [00:22:02] kart_, yeah, if you don't do both branches, at least part will disappear when wmf15 rotates on. [00:22:20] kart_: https://noc.wikimedia.org/conf/ [00:22:25] Currently active MediaWiki versions: 1.25wmf14, 1.25wmf15 [00:22:40] Reedy: confusing it is. [00:23:04] Yeah, but I wasn't sure if CX was newly deployed (though in that case, I guess it would be only 15) [00:23:23] superm401: Reedy ^ [00:23:32] Hm? [00:23:33] It's only both, though, like you said. [00:23:36] It's been branched for a while [00:23:39] we did after it deployment of CX? [00:24:00] Need to fix then. [00:24:01] did what? [00:24:45] kart_, also, you're supposed to update your extension deployment branches as well. [00:25:04] See https://wikitech.wikimedia.org/wiki/How_to_deploy_code#Case_1b:_extension_changes . [00:25:26] You have nothing on your extension deployment branches: https://git.wikimedia.org/summary/mediawiki%2Fextensions%2FContentTranslation.git [00:25:44] I could do it for you this time. [00:25:53] kart_, do you really need everything up to including that patch, or just a particular change or couple of changes? [00:25:56] Reedy: can you make me a helper or admin in #wikipedia-userscripts please when you have a moment? /cs access #wikipedia-userscripts add Technical_13 helper (or admin) [00:26:03] superm401: please go ahead. [00:26:04] Do I have right there? [00:26:10] Yes [00:26:11] superm401: update to master [00:26:29] superm401: ie upto that patch [00:26:52] (03PS1) 10Andrew Bogott: Scrap the /var/log lvm partition; make the / 20G by default. [puppet] - 10https://gerrit.wikimedia.org/r/186113 [00:26:55] Is it a lot of patches? [00:27:12] https://github.com/wikimedia/mediawiki-extensions-ContentTranslation/branches [00:27:14] Both 12 behind [00:27:16] There's no clean way to just pull onto the deployment branch, since it's not a FF (since there is the "Created deployment branch" commit). [00:27:32] superm401: Delete the branch in gerrit and recreate [00:27:36] Causes a force update [00:27:39] Yeah, I was just thinking that. [00:27:53] I did that for wmf14 [00:28:27] YuviPanda, Coren: https://gerrit.wikimedia.org/r/#/c/186113/ [00:28:34] Reedy, do you mind doing wmf15 too? [00:28:45] Then I can do the rest. [00:29:01] you want me to rebranch both of the deployment branches in the extension? [00:29:04] andrewbogott: does it scrap /var as a separate partition too? [00:29:52] YuviPanda: that was https://gerrit.wikimedia.org/r/#/c/185195/ [00:29:57] Reedy, yeah. [00:30:02] np [00:30:03] Cloning now [00:30:06] andrewbogott: aha, nice. [00:30:12] andrewbogott: can you do this for trusty also? :D [00:30:52] RECOVERY - puppet last run on haedus is OK: OK: Puppet is currently enabled, last run 22 seconds ago with 0 failures [00:31:08] (03PS2) 10Yuvipanda: Scrap the /var/log lvm partition; make the / 20G by default. [puppet] - 10https://gerrit.wikimedia.org/r/186113 (https://phabricator.wikimedia.org/T87003) (owner: 10Andrew Bogott) [00:31:41] (03CR) 10coren: [C: 04-1] "Can't remove LVM halfway; all needs to be removed. Comments inline." [puppet] - 10https://gerrit.wikimedia.org/r/186113 (https://phabricator.wikimedia.org/T87003) (owner: 10Andrew Bogott) [00:32:33] kart_, you'll probably notice this is a bit of a pain, which is why it's better to cherry-pick just the patches you need if you can't ride the train. [00:33:14] Coren: comments inline? [00:33:15] Coren: can’t see inline comments [00:33:25] superm401: Should be done [00:33:40] Great. Thanks, Reedy. I'll do the rest. [00:33:44] Sweet [00:33:48] It'll need a scap, obviously [00:33:52] Reedy, yep. [00:33:58] Hm? Ah, my comments ended up on CS 1 [00:34:13] Coren: because I edited commit message to add phab task in the meantime [00:34:16] mid air collission [00:34:20] * Coren nods. [00:34:33] (03CR) 10coren: Scrap the /var/log lvm partition; make the / 20G by default. (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/186113 (https://phabricator.wikimedia.org/T87003) (owner: 10Andrew Bogott) [00:34:42] kart_, okay, so we're going to deploy both. [00:34:47] I had to publish them by hand [00:34:58] who’s doing the scap deploy? [00:35:05] Coren, don't we still want lvm installed, though, in order to e.g. recapture unpartitioned space for /srv? [00:35:11] Or do the puppet classes do that anyway? [00:35:13] kaldari, me. [00:35:46] superm401: cool. I have one config change on the list. Let me know if you want any help. [00:36:00] andrewbogott: Well, you can leave the vg there if you want - but the swap must not be. [00:36:45] (03PS3) 10Andrew Bogott: Scrap the /var/log lvm partition; make the / 20G by default. [puppet] - 10https://gerrit.wikimedia.org/r/186113 [00:36:46] better? [00:37:09] superm401: nice [00:37:48] andrewbogott: add Bug: T87003 to the commit message? [00:38:12] (03PS4) 10Andrew Bogott: Scrap the /var/log lvm partition; make the / 20G by default. [puppet] - 10https://gerrit.wikimedia.org/r/186113 (https://phabricator.wikimedia.org/T87003) [00:39:48] kart_, it's okay that there's an additional "Localization update patch" after bb65d, right (i.e. 9486567)? [00:40:48] if it isn't, we have bigger problems :) [00:41:10] (03CR) 10coren: [C: 031] "With the caveat that this provides for no swap space by default (which may not be an issue in practice)" [puppet] - 10https://gerrit.wikimedia.org/r/186113 (https://phabricator.wikimedia.org/T87003) (owner: 10Andrew Bogott) [00:41:16] Reedy: heh [00:41:19] superm401: yes [00:41:31] Reedy, never know with the translation team. ;) Thanks for confirming, kart_. [00:41:46] Reedy: need Deployment 101 lesson when you're free. [00:42:07] kart_: Buy me stuff on my amazon wishlist, and I'll do them [00:42:08] ;) [00:42:20] Reedy: link please. [00:42:20] * YuviPanda buys people Reedy buckfast [00:42:49] Right now, someone need to buy tshirt for me. No luggage yet. [00:42:51] :D [00:43:08] Yikes [00:43:17] (03PS5) 10Andrew Bogott: Scrap the /var/log lvm partition; make the / 20G by default. [puppet] - 10https://gerrit.wikimedia.org/r/186113 (https://phabricator.wikimedia.org/T87003) [00:43:18] kart_, I can explain it some time as well. [00:43:25] If it sounds like a pain, that's cause it is. [00:43:38] andrewbogott: is that the only thing that is blocking debian? [00:43:48] YuviPanda: yes, as far as I know [00:43:52] wheeeee [00:44:05] (03CR) 10Andrew Bogott: [C: 032] Scrap the /var/log lvm partition; make the / 20G by default. [puppet] - 10https://gerrit.wikimedia.org/r/186113 (https://phabricator.wikimedia.org/T87003) (owner: 10Andrew Bogott) [00:44:10] YuviPanda, we're switching to Debian? [00:44:10] superm401: some kind of diagram helps for new persons, but that's wishlist. [00:45:26] kart_, start with http://wikitech.wikimedia.org/view/How_to_deploy_code . That will get you most of the way, but you'll likely have questions, which Reedy or I can answer. [00:48:18] (03PS1) 10Yuvipanda: labs_vmbuilder: Remove separate /var and /var/log [puppet] - 10https://gerrit.wikimedia.org/r/186121 (https://phabricator.wikimedia.org/T87003) [00:48:22] andrewbogott: ^ is this right? [00:48:35] superm401: most it is clear, but I had question related to branches. [00:48:55] I think it is clear now. Update extension is all deployed branches :) [00:49:00] dang, our swap on ubuntu is super huge [00:49:09] YuviPanda: the total needs to add up to less than 20. [00:49:17] should we kill swap? [00:49:28] kart_, generally, yeah (if you need to cherry-pick, you generally need to do so for all deployed branches). [00:49:30] I did 19G/512 on debian. [00:49:32] oh [00:49:34] let me mirror [00:49:37] Not always, but often. [00:49:41] I have no intuition if those are reasonable numbers, but let's make 'em the same [00:49:42] superm401: more when I regularly involve into deployment. [00:49:57] andrewbogott, are we switching to Debian in general, or just for one thing? [00:50:03] in general. [00:50:06] Sweet. [00:50:11] gradually, though [00:50:14] but it'll be pretty gradual [00:50:20] heh :) [00:50:43] Are you sure? I have a Debian CD if you want to re-image all the machines tonight. [00:50:44] i.e. it will be years before we are ubuntu-free [00:50:46] ;) [00:50:55] <_joe_> years? [00:51:01] <_joe_> well, probably [00:51:04] (03PS2) 10Yuvipanda: labs_vmbuilder: Remove separate /var* and increase / to 19G [puppet] - 10https://gerrit.wikimedia.org/r/186121 (https://phabricator.wikimedia.org/T87003) [00:51:06] andrewbogott: ^ [00:51:20] _joe_: we still have lucid [00:51:45] <_joe_> YuviPanda: yeah but removing ubuntu has that kind of sweet taste that can motivate us [00:52:04] YuviPanda: I think you need to remove the swap partition as well, in the firstboot script [00:52:12] oh [00:52:19] andrewbogott, is there an announcement about the Debian switch somewhere? [00:52:21] I'm just curious, since I've run Debian for a while on my personal machine for various reasons (after using Ubuntu variants earlier) [00:52:26] I'm not sure what it does now since swap is clearly defined twice [00:52:31] hahaha [00:52:36] let me do that as a separte patch [00:52:40] superm401: I'm not sure if there's an announcement :) [00:52:59] andrewbogott, what's the reason for the switch? [00:53:04] Or at least a Phabricator? [00:53:11] (03PS1) 10Yuvipanda: labs_vmbuilder: Don't setup lvm based swap [puppet] - 10https://gerrit.wikimedia.org/r/186125 [00:53:12] andrewbogott: ^ [00:53:19] superm401: there’s an ops@ thread [00:53:20] superm401: to debian, you mean? Or the thing yuvi and I are doing? [00:53:31] andrewbogott, to Debian. I can check the ops thread if it's not public. [00:54:04] superm401: Various reasons -- you can read the thread for the full discussion. [00:54:10] Okay, kart_, your stuff is merged. [00:54:15] Not secret, just, it wasn't one big thing. [00:54:16] I'll do the others, then scap. [00:54:35] superm401: hopefully, scap should be quicker since we added the extra scap proxies [00:55:01] Yeah, I know we're behind. [00:55:34] I didn't realize I was doing it until 17 minutes in. [00:55:47] I was meaning generally [00:55:56] Wasn't making a comment about it running late :) [00:56:14] Oh, okay. [00:56:47] Someone still needs to look into that Gerrit issue. [00:56:49] I'll file it. [00:57:12] Please [00:57:20] Then we can find a big stick to poke paravoid with [01:00:02] superm401: Reedy thanks! [01:00:22] 3operations, Wikimedia-Git-or-Gerrit: Gerrit ssh key changed and does not match puppet - https://phabricator.wikimedia.org/T87287#988039 (10Mattflaschen) [01:01:08] 3operations, Wikimedia-Git-or-Gerrit: Gerrit ssh key changed and does not match puppet - https://phabricator.wikimedia.org/T87287#988040 (10Reedy) @faidon is this from your key changes? [01:02:04] ? [01:02:12] which key changed ? [01:02:27] superm401: ^ [01:02:40] and how does puppet come into gerrit's key ? [01:02:40] akosiaris, Gerrit ssh [01:02:46] port 29418 right ? [01:02:51] Yeah. [01:02:52] aka NOT ytterbium's key [01:02:53] akosiaris, the public key is specified in Gerrit. [01:02:57] Err, in puppet [01:03:33] Gah, now I get a rebase conflict in ContentTranslation cause of .gitreview. [01:03:57] I'll just force it. [01:03:59] yeah [01:05:57] (03CR) 10Mattflaschen: [C: 032] Enable 'copyedit' GettingStarted suggestions for ptwiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/185122 (https://phabricator.wikimedia.org/T86590) (owner: 10Mattflaschen) [01:06:20] (03CR) 10Mattflaschen: [C: 032] Turning WikiGrok off on en.wiki (test finished) [mediawiki-config] - 10https://gerrit.wikimedia.org/r/186097 (owner: 10Kaldari) [01:06:22] (03Merged) 10jenkins-bot: Enable 'copyedit' GettingStarted suggestions for ptwiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/185122 (https://phabricator.wikimedia.org/T86590) (owner: 10Mattflaschen) [01:06:29] (03Merged) 10jenkins-bot: Turning WikiGrok off on en.wiki (test finished) [mediawiki-config] - 10https://gerrit.wikimedia.org/r/186097 (owner: 10Kaldari) [01:07:57] superm401: mind pointing me to gerrit's key in puppet ? [01:09:01] akosiaris, https://git.wikimedia.org/blob/operations%2Fpuppet/943bdc862ad9f7f7cd973c78da3b08636ee29b1f/manifests%2Frole%2Fgerrit.pp#L70 . [01:10:03] 3Wikimedia-General-or-Unknown, operations: Varnish cluster 503's should be logged to Graphite/Gdash/Ganglia/etc - https://phabricator.wikimedia.org/T38131#988044 (10Joe) 5Open>3Resolved [01:13:04] superm401: Special:Version doesn't show updated CX yet though... [01:13:46] 3operations, Wikimedia-Git-or-Gerrit: Gerrit ssh key changed and does not match puppet - https://phabricator.wikimedia.org/T87287#988050 (10Joe) I see tin is using ipv6 to connect to gerrit... maybe this is one of the reasons? [01:14:20] kart_, I haven't swatted yet. [01:14:24] Err, scapped. [01:14:26] ah. [01:14:29] :) [01:14:29] 3operations: Remove old Tampa servers from racktables - https://phabricator.wikimedia.org/T87288#988052 (10Cmjohnson) 3NEW a:3Cmjohnson [01:15:12] !log mattflaschen Started scap: Turning off WikiGrok, enabling GettingStarted copyediting suggestions on ptwiki, and upgrading ContentTranslation [01:15:24] Logged the message, Master [01:29:13] 3operations, Wikimedia-Git-or-Gerrit: Gerrit ssh key changed and does not match puppet - https://phabricator.wikimedia.org/T87287#988093 (10Mattflaschen) Yeah, but there's also a ssh key mismatch. If you copy the key from https://git.wikimedia.org/raw/operations%2Fpuppet/production/manifests%2Frole%2Fgerrit.pp... [01:30:57] (03PS1) 10Hoo man: Remove the Bug54847.php script [mediawiki-config] - 10https://gerrit.wikimedia.org/r/186132 [01:31:04] andrewbogott: think you can look at my ubuntu image patches? [01:31:19] andrewbogott: ah, nah, let’s fix jessie first. [01:32:14] greg-g: around? [01:32:14] hoo: You sent me a contentless ping. This is a contentless pong. Please provide a bit of information about what you want and I will respond when I am around. [01:32:18] meh [01:32:24] Did I just miss a SWAT? [01:33:05] hoo: yeah, it's 1 hour 30 minutes after SWAT started :) [01:33:09] hoo, yep. Scap is running now, though... [01:33:16] greg-g: see above [01:33:20] Had to help finalize the CX patches. [01:33:22] (03CR) 10Andrew Bogott: [C: 031] labs_vmbuilder: Don't setup lvm based swap [puppet] - 10https://gerrit.wikimedia.org/r/186125 (owner: 10Yuvipanda) [01:33:24] that is keeping at least two users from logging in [01:33:39] 3operations: merge tickets in project "ops-core" into project "operations" - https://phabricator.wikimedia.org/T87291#988109 (10Krenair) [01:33:42] Possible more, no idea... we still seem to have problems with logging [01:33:46] * possibly [01:33:56] (03CR) 10Andrew Bogott: [C: 032] labs_vmbuilder: Remove separate /var* and increase / to 19G [puppet] - 10https://gerrit.wikimedia.org/r/186121 (https://phabricator.wikimedia.org/T87003) (owner: 10Yuvipanda) [01:34:03] (03PS3) 10Andrew Bogott: labs_vmbuilder: Remove separate /var* and increase / to 19G [puppet] - 10https://gerrit.wikimedia.org/r/186121 (https://phabricator.wikimedia.org/T87003) (owner: 10Yuvipanda) [01:35:20] Maybe 6 users or so affected today [01:35:24] just an estimate [01:35:44] It's too late for SWAT, I think. Has csteipp reviewed? [01:35:59] He told me to just remove it (as it kept doing problems) [01:36:08] But no, he hasn't yet seen my actual patch [01:36:37] s/doing/making/ [01:36:56] hoo: jfdi [01:37:08] yay [01:37:29] Though, is scap still running? [01:37:39] Reedy: looks like it [01:38:00] or at least it never logged completion [01:38:10] I'll be around for a bit longer anyway... yay for doing simulations on a netbook [01:38:46] that was a bad idea... intel atom :P [01:40:02] Reedy: you going to be around for a bit? [01:40:09] Yeah [01:40:21] It's only 01:40 [01:40:23] still early ;) [01:40:51] Unless you have to get up at 6am... [01:41:25] :) [01:41:36] when I came over to SF this time I barely slept, up at 05:30 and went to bed about 1am [01:42:27] I need to get up at 5 or 6 on Friday... that's going to be *fun* [01:43:38] !log mattflaschen Finished scap: Turning off WikiGrok, enabling GettingStarted copyediting suggestions on ptwiki, and upgrading ContentTranslation (duration: 28m 26s) [01:43:48] Logged the message, Master [01:43:53] Yay! [01:44:03] yay! [01:44:38] that's not bad, 28.5 mins [01:45:06] Am I good to go? [01:45:24] I think so, if superm401 says he's done [01:45:26] I think that was it for him [01:45:38] Reedy, I need to run a quick script. [01:47:43] (03PS1) 10Mattflaschen: Add ptwiki to gettingstarted-with-category-suggestions [mediawiki-config] - 10https://gerrit.wikimedia.org/r/186133 [01:47:49] ^ I forgot one minor step before. [01:48:08] (03CR) 10Mattflaschen: [C: 032] Add ptwiki to gettingstarted-with-category-suggestions [mediawiki-config] - 10https://gerrit.wikimedia.org/r/186133 (owner: 10Mattflaschen) [01:48:14] (03Merged) 10jenkins-bot: Add ptwiki to gettingstarted-with-category-suggestions [mediawiki-config] - 10https://gerrit.wikimedia.org/r/186133 (owner: 10Mattflaschen) [01:48:16] Then after that, I need to run the script. [01:49:36] heh, easy enough [01:53:28] !log Re-ran GettingStarted populate_categories.php, also populating ptwiki for the first time. [01:53:30] Done [01:53:43] Logged the message, Master [01:54:14] That's the end of SWAT. kart_, kaldari, please test. [01:55:33] Ok [01:55:45] If everything's fine, I'll go for it [02:02:31] Ok [02:02:38] * hoo goes ahead [02:02:58] (03CR) 10Hoo man: [C: 032] Remove the Bug54847.php script [mediawiki-config] - 10https://gerrit.wikimedia.org/r/186132 (owner: 10Hoo man) [02:03:04] (03Merged) 10jenkins-bot: Remove the Bug54847.php script [mediawiki-config] - 10https://gerrit.wikimedia.org/r/186132 (owner: 10Hoo man) [02:04:00] !log hoo Synchronized wmf-config/: Remove the Bug54847.php script (duration: 00m 07s) [02:04:16] Logged the message, Master [02:05:42] superm401: Did you also have sync failures? :/ [02:06:15] 3operations: Re-image osmium and repurpose into VE testing host - https://phabricator.wikimedia.org/T87215#988172 (10ori) [02:08:09] !log hoo Synchronized wmf-config/: Remove the Bug54847.php script - 2nd (duration: 00m 06s) [02:08:18] mh, worked now [02:08:19] Logged the message, Master [02:08:53] "directory has vanished: "/wmf-config/.~tmp~" (in common)" [02:10:24] hoo, nope. [02:10:37] https://phabricator.wikimedia.org/P227 [02:11:11] mh, weird [02:12:31] https://phabricator.wikimedia.org/P228 [02:12:32] that's mine [02:13:17] hoo, looks like a tmp directory got created and deleted somehow: directory has vanished: "/wmf-config/.~tmp~" (in common)" [02:13:21] Maybe an editor. [02:13:45] I think that may have been the only problem. [02:16:23] Helder, both ptwiki updates are done. [02:16:32] Looks good, except some translations are missing. [02:16:34] !log l10nupdate Synchronized php-1.25wmf14/cache/l10n: (no message) (duration: 00m 01s) [02:16:38] !log LocalisationUpdate completed (1.25wmf14) at 2015-01-21 02:16:38+00:00 [02:16:42] Logged the message, Master [02:16:48] Logged the message, Master [02:19:14] (03CR) 10Mattflaschen: "I forgot about gettingstarted-with-category-suggestions.dblist . No big deal, though. Done in https://gerrit.wikimedia.org/r/#/c/186133/" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/185122 (https://phabricator.wikimedia.org/T86590) (owner: 10Mattflaschen) [02:28:47] !log l10nupdate Synchronized php-1.25wmf15/cache/l10n: (no message) (duration: 00m 01s) [02:28:51] !log LocalisationUpdate completed (1.25wmf15) at 2015-01-21 02:28:50+00:00 [02:28:56] Logged the message, Master [02:29:01] Logged the message, Master [02:32:40] New dump is there: https://dumps.wikimedia.org/other/wikidata/ went through fine [02:34:51] (03PS1) 10Ori.livneh: keyholder: ensure ssh-agent-proxy transmits whole messages [puppet] - 10https://gerrit.wikimedia.org/r/186137 (https://phabricator.wikimedia.org/T86545) [02:36:08] (03CR) 10Ori.livneh: [C: 032 V: 032] keyholder: ensure ssh-agent-proxy transmits whole messages [puppet] - 10https://gerrit.wikimedia.org/r/186137 (https://phabricator.wikimedia.org/T86545) (owner: 10Ori.livneh) [02:38:44] !log ori Synchronized README: Testing I68b5e1c2f (duration: 00m 02s) [02:38:52] Logged the message, Master [02:38:54] lol [02:39:34] !log ori Synchronized README: Testing I68b5e1c2f (duration: 00m 07s) [02:39:41] Logged the message, Master [02:41:20] 3operations: Scap error on mw1111: "Error reading response length from authentication socket." - https://phabricator.wikimedia.org/T86545#988292 (10ori) I hope (but am not certain) that the patch above fixes the problem. Please update this task if you continue seeing this error. [02:41:31] Reedy: ^ [02:44:47] 3operations: merge tickets in project "ops-core" into project "operations" - https://phabricator.wikimedia.org/T87291#988307 (10Aklapper) [02:48:25] 3Phabricator, operations: merge tickets in project "ops-core" into project "operations" - https://phabricator.wikimedia.org/T87291#988309 (10Aklapper) That's currently 2562 tickets and will trigger quite some notification spam (plus T76152 will bite us). Similar to 76942 I wonder whether this could be done in a... [03:38:40] Is there an OTRS queue for users' issues using the servers? They get a 503 somewhere and gave relevant detail. [03:45:00] svetlana, info-en::Tech-issues possibly [03:45:39] but that's quite obscure and managed by very few people, it won't catch ops attention. [03:45:58] for things like 503s you'd want phabricator [03:49:42] Krenair: the problem is that the user can't log in from their location for whatever reason. They'd likely have the same issue with registering at wikitech/LDAP. [03:50:01] interesting [03:50:07] Krenair: may I /msg you details for you to investigate, or do you have no access to any sort of webserver? [03:50:21] 503 when they try to reset password. [03:50:31] when they try to do things related to their account? [03:51:15] or any account they try? [03:51:49] I think I've heard of an issue like this before, needs to go in phabricator [03:51:55] you can /msg me if you like [03:54:11] Thank you. [03:58:37] YuviPanda, around? [04:20:44] 3Phabricator, operations: merge tickets in project "ops-core" into project "operations" - https://phabricator.wikimedia.org/T87291#988324 (10Dzahn) The plan would be to do it on the mysql level, which should get around the issues mentioned above, afaict. [04:39:39] !log LocalisationUpdate ResourceLoader cache refresh completed at Wed Jan 21 04:39:39 UTC 2015 (duration 39m 38s) [04:39:51] Logged the message, Master [05:54:18] 3operations, Continuous-Integration: Acquire old production API servers for use in CI - https://phabricator.wikimedia.org/T84940#988448 (10hashar) [05:55:02] 3Wikimedia-Logstash, operations, ops-core: Production hardware for Logstash service - https://phabricator.wikimedia.org/T84958#988453 (10bd808) Here's my proposal: * Add 3 hosts matching the machine spec used for the MediaWiki Elasticsearch cluster * Move the Logstash Elasticsearch backend to those hosts * Keep... [06:28:01] 3Wikimedia-Logstash, operations, ops-core: Production hardware for Logstash service - https://phabricator.wikimedia.org/T84958#988487 (10Manybubbles) >>! In T84958#988453, @bd808 wrote: > Here's my proposal: > * Add 3 hosts matching the machine spec used for the MediaWiki Elasticsearch cluster Does it make sens... [06:28:12] PROBLEM - puppet last run on mw1114 is CRITICAL: CRITICAL: puppet fail [06:28:32] PROBLEM - puppet last run on labsdb1003 is CRITICAL: CRITICAL: Puppet has 1 failures [06:28:42] PROBLEM - puppet last run on cp1061 is CRITICAL: CRITICAL: Puppet has 1 failures [06:29:01] PROBLEM - puppet last run on mw1175 is CRITICAL: CRITICAL: puppet fail [06:29:01] PROBLEM - puppet last run on mw1092 is CRITICAL: CRITICAL: Puppet has 1 failures [06:29:22] PROBLEM - puppet last run on db1046 is CRITICAL: CRITICAL: Puppet has 1 failures [06:31:32] PROBLEM - puppet last run on cp4003 is CRITICAL: CRITICAL: Puppet has 1 failures [06:31:42] PROBLEM - puppet last run on mw1118 is CRITICAL: CRITICAL: Puppet has 1 failures [06:32:21] PROBLEM - puppet last run on mw1052 is CRITICAL: CRITICAL: Puppet has 1 failures [06:32:41] PROBLEM - puppet last run on cp4014 is CRITICAL: CRITICAL: Puppet has 1 failures [06:45:21] RECOVERY - puppet last run on labsdb1003 is OK: OK: Puppet is currently enabled, last run 35 seconds ago with 0 failures [06:45:21] RECOVERY - puppet last run on cp1061 is OK: OK: Puppet is currently enabled, last run 18 seconds ago with 0 failures [06:45:31] RECOVERY - puppet last run on mw1052 is OK: OK: Puppet is currently enabled, last run 8 seconds ago with 0 failures [06:45:42] RECOVERY - puppet last run on mw1092 is OK: OK: Puppet is currently enabled, last run 18 seconds ago with 0 failures [06:46:01] RECOVERY - puppet last run on db1046 is OK: OK: Puppet is currently enabled, last run 36 seconds ago with 0 failures [06:46:02] RECOVERY - puppet last run on mw1118 is OK: OK: Puppet is currently enabled, last run 11 seconds ago with 0 failures [06:46:51] RECOVERY - puppet last run on mw1175 is OK: OK: Puppet is currently enabled, last run 17 seconds ago with 0 failures [06:47:02] RECOVERY - puppet last run on cp4003 is OK: OK: Puppet is currently enabled, last run 58 seconds ago with 0 failures [06:47:21] RECOVERY - puppet last run on mw1114 is OK: OK: Puppet is currently enabled, last run 30 seconds ago with 0 failures [06:48:12] RECOVERY - puppet last run on cp4014 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [08:16:32] PROBLEM - graphite.wikimedia.org on tungsten is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 525 bytes in 0.008 second response time [08:28:31] RECOVERY - graphite.wikimedia.org on tungsten is OK: HTTP OK: HTTP/1.1 200 OK - 1607 bytes in 0.007 second response time [08:38:02] (03CR) 10Yuvipanda: [C: 031] hiera: fix up for labs hierarchy [puppet] - 10https://gerrit.wikimedia.org/r/185940 (owner: 10Giuseppe Lavagetto) [09:19:32] PROBLEM - Apache HTTP on mw1192 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:46:12] PROBLEM - puppet last run on lvs3001 is CRITICAL: CRITICAL: puppet fail [11:05:21] RECOVERY - puppet last run on lvs3001 is OK: OK: Puppet is currently enabled, last run 48 seconds ago with 0 failures [12:37:02] (03PS1) 10Se4598: Decrease '$wgAbuseFilterAnonBlockDuration' to '7 days' on fawiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/186156 (https://phabricator.wikimedia.org/T87317) [12:38:47] (03CR) 10Calak: [C: 031] Decrease '$wgAbuseFilterAnonBlockDuration' to '7 days' on fawiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/186156 (https://phabricator.wikimedia.org/T87317) (owner: 10Se4598) [14:46:51] PROBLEM - puppet last run on ms-be2004 is CRITICAL: CRITICAL: puppet fail [15:05:52] RECOVERY - puppet last run on ms-be2004 is OK: OK: Puppet is currently enabled, last run 56 seconds ago with 0 failures [16:00:04] Respected human, time to deploy WMF All Hands (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20150121T1600). Please do the needful. [16:05:52] hah [16:10:22] * ^d starts scap for all hands [16:10:25] noooo [16:10:29] dont start it so we dont have to go [16:10:37] thats how it works right/ [16:10:38] ? [16:10:48] <^d> If it's not on the calendar it doesn't exist! [16:14:15] (03CR) 10JanZerebecki: [C: 04-1] "Blocked on discussion in linked bug." [puppet] - 10https://gerrit.wikimedia.org/r/181992 (https://phabricator.wikimedia.org/T58258) (owner: 10Glaisher) [16:17:17] (03Abandoned) 10Glaisher: Redirect Main_[Pp]age to Wikidata:Main_Page [puppet] - 10https://gerrit.wikimedia.org/r/181992 (https://phabricator.wikimedia.org/T58258) (owner: 10Glaisher) [16:17:34] 3operations, Wikimedia-Git-or-Gerrit: Gerrit ssh key changed and does not match puppet - https://phabricator.wikimedia.org/T87287#989130 (10akosiaris) The key mentioned in the puppet repo is used for gerrit replication and is not gerrit's key, so it would not match anyway. The dc:e9:68:7b:99:1b:27:d0:f9:fd:ce:6a... [16:42:12] PROBLEM - Router interfaces on mr1-esams is CRITICAL: CRITICAL: host 91.198.174.247, interfaces up: 36, down: 1, dormant: 0, excluded: 1, unused: 0BRge-0/0/0: down - Core: msw-oe12-esamsBR [17:04:11] PROBLEM - Varnishkafka Delivery Errors per minute on cp3015 is CRITICAL: CRITICAL: 12.50% of data above the critical threshold [20000.0] [17:09:53] 3Beta-Cluster, operations: Mails through deployment-mx SPF & DKIM fails - https://phabricator.wikimedia.org/T87338#989166 (1001tonythomas) 3NEW [17:11:11] RECOVERY - Varnishkafka Delivery Errors per minute on cp3015 is OK: OK: Less than 1.00% above the threshold [0.0] [17:38:42] PROBLEM - git.wikimedia.org on antimony is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:58:12] PROBLEM - puppet last run on cp3017 is CRITICAL: CRITICAL: puppet fail [17:58:21] PROBLEM - puppet last run on cp3018 is CRITICAL: CRITICAL: Puppet has 7 failures [17:58:21] PROBLEM - puppet last run on labsdb1005 is CRITICAL: CRITICAL: Puppet has 11 failures [17:58:21] PROBLEM - puppet last run on ms-be1010 is CRITICAL: CRITICAL: puppet fail [17:58:31] PROBLEM - puppet last run on mw1113 is CRITICAL: CRITICAL: Puppet has 40 failures [17:58:31] PROBLEM - puppet last run on db2010 is CRITICAL: CRITICAL: Puppet has 7 failures [17:58:32] PROBLEM - puppet last run on lvs4004 is CRITICAL: CRITICAL: Puppet has 7 failures [17:58:41] PROBLEM - puppet last run on eeden is CRITICAL: CRITICAL: Puppet has 2 failures [17:58:42] PROBLEM - puppet last run on cp3019 is CRITICAL: CRITICAL: puppet fail [17:58:42] PROBLEM - puppetmaster backend https on palladium is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 8141: HTTP/1.1 500 Internal Server Error [17:58:42] PROBLEM - puppet last run on mc1015 is CRITICAL: CRITICAL: Puppet has 21 failures [17:58:51] PROBLEM - puppet last run on palladium is CRITICAL: CRITICAL: Puppet has 27 failures [17:58:51] PROBLEM - puppet last run on elastic1025 is CRITICAL: CRITICAL: puppet fail [17:58:51] PROBLEM - puppet last run on tmh1001 is CRITICAL: CRITICAL: Puppet has 61 failures [17:58:52] PROBLEM - puppet last run on es1004 is CRITICAL: CRITICAL: puppet fail [17:58:52] PROBLEM - puppet last run on virt1005 is CRITICAL: CRITICAL: puppet fail [17:58:52] PROBLEM - puppet last run on mw1137 is CRITICAL: CRITICAL: puppet fail [17:58:52] PROBLEM - puppet last run on analytics1031 is CRITICAL: CRITICAL: puppet fail [17:58:52] PROBLEM - puppet last run on cp1066 is CRITICAL: CRITICAL: Puppet has 15 failures [17:59:01] PROBLEM - puppet last run on virt1009 is CRITICAL: CRITICAL: puppet fail [17:59:02] PROBLEM - puppet last run on db1027 is CRITICAL: CRITICAL: Puppet has 19 failures [17:59:02] PROBLEM - puppet last run on radon is CRITICAL: CRITICAL: puppet fail [17:59:02] PROBLEM - puppet last run on mw1179 is CRITICAL: CRITICAL: puppet fail [17:59:02] PROBLEM - puppet last run on mw1058 is CRITICAL: CRITICAL: puppet fail [17:59:11] PROBLEM - puppet last run on mw1070 is CRITICAL: CRITICAL: puppet fail [17:59:11] PROBLEM - puppet last run on analytics1012 is CRITICAL: CRITICAL: puppet fail [17:59:11] PROBLEM - puppet last run on mw1073 is CRITICAL: CRITICAL: puppet fail [17:59:11] PROBLEM - puppet last run on mw1131 is CRITICAL: CRITICAL: Puppet has 57 failures [17:59:11] PROBLEM - puppet last run on elastic1029 is CRITICAL: CRITICAL: Puppet has 24 failures [17:59:12] PROBLEM - puppetmaster https on palladium is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 8140: HTTP/1.1 500 Internal Server Error [17:59:12] PROBLEM - puppet last run on mw1199 is CRITICAL: CRITICAL: puppet fail [17:59:12] PROBLEM - puppet last run on mw1037 is CRITICAL: CRITICAL: Puppet has 43 failures [17:59:13] PROBLEM - puppet last run on hydrogen is CRITICAL: CRITICAL: puppet fail [17:59:21] PROBLEM - puppet last run on mw1128 is CRITICAL: CRITICAL: Puppet has 41 failures [17:59:21] PROBLEM - puppet last run on labsdb1002 is CRITICAL: CRITICAL: puppet fail [17:59:21] PROBLEM - puppet last run on pollux is CRITICAL: CRITICAL: puppet fail [17:59:22] PROBLEM - puppet last run on bast2001 is CRITICAL: CRITICAL: puppet fail [17:59:22] PROBLEM - puppet last run on cp4009 is CRITICAL: CRITICAL: Puppet has 15 failures [17:59:22] PROBLEM - puppet last run on cp4010 is CRITICAL: CRITICAL: Puppet has 14 failures [17:59:22] PROBLEM - puppet last run on cp4016 is CRITICAL: CRITICAL: Puppet has 17 failures [17:59:27] PROBLEM - puppet last run on db1005 is CRITICAL: CRITICAL: puppet fail [17:59:28] PROBLEM - puppet last run on analytics1021 is CRITICAL: CRITICAL: Puppet has 9 failures [17:59:28] PROBLEM - puppet last run on ms-be3004 is CRITICAL: CRITICAL: puppet fail [17:59:28] PROBLEM - puppet last run on cp3011 is CRITICAL: CRITICAL: Puppet has 20 failures [17:59:28] PROBLEM - puppet last run on db1068 is CRITICAL: CRITICAL: Puppet has 22 failures [17:59:31] PROBLEM - puppet last run on lanthanum is CRITICAL: CRITICAL: Puppet has 16 failures [17:59:31] PROBLEM - puppet last run on cp3005 is CRITICAL: CRITICAL: Puppet has 17 failures [17:59:31] PROBLEM - puppet last run on mw1155 is CRITICAL: CRITICAL: Puppet has 29 failures [17:59:31] PROBLEM - puppet last run on mw1018 is CRITICAL: CRITICAL: Puppet has 68 failures [17:59:32] PROBLEM - puppet last run on mw1207 is CRITICAL: CRITICAL: Puppet has 61 failures [17:59:32] PROBLEM - puppet last run on elastic1013 is CRITICAL: CRITICAL: puppet fail [17:59:32] PROBLEM - puppet last run on mw1047 is CRITICAL: CRITICAL: Puppet has 32 failures [17:59:41] PROBLEM - puppet last run on mw1194 is CRITICAL: CRITICAL: Puppet has 60 failures [17:59:41] PROBLEM - puppet last run on lvs2002 is CRITICAL: CRITICAL: puppet fail [17:59:41] PROBLEM - puppet last run on mw1103 is CRITICAL: CRITICAL: Puppet has 73 failures [17:59:41] PROBLEM - puppet last run on mw1230 is CRITICAL: CRITICAL: puppet fail [17:59:41] PROBLEM - puppet last run on es2006 is CRITICAL: CRITICAL: Puppet has 16 failures [17:59:42] PROBLEM - puppet last run on db2017 is CRITICAL: CRITICAL: Puppet has 18 failures [17:59:42] PROBLEM - puppet last run on ms-be2009 is CRITICAL: CRITICAL: puppet fail [17:59:51] PROBLEM - puppet last run on mw1078 is CRITICAL: CRITICAL: puppet fail [17:59:52] PROBLEM - puppet last run on cp3021 is CRITICAL: CRITICAL: Puppet has 24 failures [17:59:52] PROBLEM - puppet last run on amssq45 is CRITICAL: CRITICAL: puppet fail [17:59:52] PROBLEM - puppet last run on amssq37 is CRITICAL: CRITICAL: Puppet has 27 failures [17:59:52] PROBLEM - puppet last run on mw1136 is CRITICAL: CRITICAL: puppet fail [18:00:01] PROBLEM - puppet last run on db1009 is CRITICAL: CRITICAL: puppet fail [18:00:01] PROBLEM - puppet last run on mw1102 is CRITICAL: CRITICAL: puppet fail [18:00:01] PROBLEM - puppet last run on mw1095 is CRITICAL: CRITICAL: puppet fail [18:00:02] PROBLEM - puppet last run on mw1020 is CRITICAL: CRITICAL: puppet fail [18:00:02] PROBLEM - puppet last run on cp1064 is CRITICAL: CRITICAL: puppet fail [18:00:02] PROBLEM - puppet last run on mw1019 is CRITICAL: CRITICAL: puppet fail [18:00:11] PROBLEM - puppet last run on ms-be1014 is CRITICAL: CRITICAL: Puppet has 18 failures [18:00:11] PROBLEM - puppet last run on mw1094 is CRITICAL: CRITICAL: puppet fail [18:00:12] PROBLEM - puppet last run on wtp1014 is CRITICAL: CRITICAL: puppet fail [18:00:21] PROBLEM - puppet last run on mw1075 is CRITICAL: CRITICAL: Puppet has 65 failures [18:00:22] PROBLEM - puppet last run on analytics1029 is CRITICAL: CRITICAL: puppet fail [18:00:22] PROBLEM - puppet last run on praseodymium is CRITICAL: CRITICAL: puppet fail [18:00:22] PROBLEM - puppet last run on lvs2005 is CRITICAL: CRITICAL: Puppet has 20 failures [18:00:22] PROBLEM - puppet last run on elastic1010 is CRITICAL: CRITICAL: puppet fail [18:00:22] PROBLEM - puppet last run on db1049 is CRITICAL: CRITICAL: puppet fail [18:00:22] PROBLEM - puppet last run on es1003 is CRITICAL: CRITICAL: puppet fail [18:00:23] RECOVERY - puppetmaster https on palladium is OK: HTTP OK: Status line output matched 400 - 335 bytes in 0.046 second response time [18:00:23] PROBLEM - puppet last run on mw1157 is CRITICAL: CRITICAL: Puppet has 26 failures [18:00:31] PROBLEM - puppet last run on labsdb1007 is CRITICAL: CRITICAL: Puppet has 13 failures [18:00:31] PROBLEM - puppet last run on es1005 is CRITICAL: CRITICAL: Puppet has 13 failures [18:00:31] PROBLEM - puppet last run on mw1017 is CRITICAL: CRITICAL: puppet fail [18:00:32] PROBLEM - puppet last run on lvs4001 is CRITICAL: CRITICAL: puppet fail [18:00:32] PROBLEM - puppet last run on cp4017 is CRITICAL: CRITICAL: puppet fail [18:00:32] PROBLEM - puppet last run on cp4015 is CRITICAL: CRITICAL: Puppet has 29 failures [18:00:32] PROBLEM - puppet last run on cp4011 is CRITICAL: CRITICAL: Puppet has 55 failures [18:00:32] PROBLEM - puppet last run on elastic1016 is CRITICAL: CRITICAL: Puppet has 23 failures [18:00:33] PROBLEM - puppet last run on amssq50 is CRITICAL: CRITICAL: Puppet has 19 failures [18:00:33] PROBLEM - puppet last run on mw1015 is CRITICAL: CRITICAL: Puppet has 61 failures [18:00:42] PROBLEM - puppet last run on cp3007 is CRITICAL: CRITICAL: Puppet has 20 failures [18:00:43] PROBLEM - puppet last run on mw1214 is CRITICAL: CRITICAL: puppet fail [18:00:43] PROBLEM - puppet last run on mw1169 is CRITICAL: CRITICAL: puppet fail [18:00:52] PROBLEM - puppet last run on ms-be1005 is CRITICAL: CRITICAL: Puppet has 18 failures [18:00:52] PROBLEM - puppet last run on ms-be2010 is CRITICAL: CRITICAL: Puppet has 16 failures [18:01:01] PROBLEM - puppet last run on db1019 is CRITICAL: CRITICAL: puppet fail [18:01:02] PROBLEM - puppet last run on mw1101 is CRITICAL: CRITICAL: Puppet has 80 failures [18:01:02] PROBLEM - puppet last run on elastic1009 is CRITICAL: CRITICAL: Puppet has 25 failures [18:01:02] PROBLEM - puppet last run on cp4013 is CRITICAL: CRITICAL: puppet fail [18:01:02] PROBLEM - puppet last run on lvs3002 is CRITICAL: CRITICAL: Puppet has 23 failures [18:01:02] PROBLEM - puppet last run on mw1085 is CRITICAL: CRITICAL: Puppet has 41 failures [18:01:02] PROBLEM - puppet last run on amssq52 is CRITICAL: CRITICAL: Puppet has 19 failures [18:01:03] PROBLEM - puppet last run on amssq58 is CRITICAL: CRITICAL: puppet fail [18:01:03] PROBLEM - puppet last run on mw1127 is CRITICAL: CRITICAL: puppet fail [18:01:12] PROBLEM - puppet last run on db1058 is CRITICAL: CRITICAL: Puppet has 22 failures [18:01:17] why all the puppet fail? [18:01:22] PROBLEM - puppet last run on mw1191 is CRITICAL: CRITICAL: puppet fail [18:01:22] PROBLEM - puppet last run on stat1001 is CRITICAL: CRITICAL: puppet fail [18:01:22] PROBLEM - puppet last run on analytics1024 is CRITICAL: CRITICAL: Puppet has 11 failures [18:01:31] PROBLEM - puppet last run on mw1083 is CRITICAL: CRITICAL: puppet fail [18:01:31] PROBLEM - puppet last run on lvs1006 is CRITICAL: CRITICAL: puppet fail [18:01:31] PROBLEM - puppet last run on mw1246 is CRITICAL: CRITICAL: puppet fail [18:01:32] PROBLEM - puppet last run on analytics1019 is CRITICAL: CRITICAL: puppet fail [18:01:32] PROBLEM - puppet last run on cp1051 is CRITICAL: CRITICAL: puppet fail [18:01:32] PROBLEM - puppet last run on mw1232 is CRITICAL: CRITICAL: Puppet has 69 failures [18:01:32] PROBLEM - puppet last run on erbium is CRITICAL: CRITICAL: Puppet has 49 failures [18:01:41] PROBLEM - puppet last run on mw1182 is CRITICAL: CRITICAL: puppet fail [18:01:41] PROBLEM - puppet last run on ocg1002 is CRITICAL: CRITICAL: puppet fail [18:01:42] PROBLEM - puppet last run on mw1196 is CRITICAL: CRITICAL: puppet fail [18:01:42] PROBLEM - puppet last run on cp1057 is CRITICAL: CRITICAL: Puppet has 14 failures [18:01:42] PROBLEM - puppet last run on amslvs4 is CRITICAL: CRITICAL: puppet fail [18:01:52] PROBLEM - puppet last run on db1041 is CRITICAL: CRITICAL: Puppet has 12 failures [18:01:52] PROBLEM - puppet last run on db1024 is CRITICAL: CRITICAL: Puppet has 14 failures [18:01:53] PROBLEM - puppet last run on mw1184 is CRITICAL: CRITICAL: Puppet has 37 failures [18:01:53] PROBLEM - puppet last run on mw1013 is CRITICAL: CRITICAL: Puppet has 27 failures [18:02:01] PROBLEM - puppet last run on mc1008 is CRITICAL: CRITICAL: Puppet has 22 failures [18:02:02] PROBLEM - puppet last run on ms-be2015 is CRITICAL: CRITICAL: Puppet has 16 failures [18:02:02] PROBLEM - puppet last run on tmh1002 is CRITICAL: CRITICAL: Puppet has 73 failures [18:02:02] PROBLEM - puppet last run on mw1234 is CRITICAL: CRITICAL: Puppet has 33 failures [18:02:11] PROBLEM - puppet last run on analytics1034 is CRITICAL: CRITICAL: Puppet has 13 failures [18:02:11] PROBLEM - puppet last run on mw1244 is CRITICAL: CRITICAL: puppet fail [18:02:11] PROBLEM - puppet last run on wtp1024 is CRITICAL: CRITICAL: Puppet has 14 failures [18:02:12] PROBLEM - puppet last run on cp3013 is CRITICAL: CRITICAL: puppet fail [18:02:21] PROBLEM - puppet last run on mw1035 is CRITICAL: CRITICAL: puppet fail [18:02:21] PROBLEM - puppet last run on mw1245 is CRITICAL: CRITICAL: Puppet has 75 failures [18:02:21] PROBLEM - puppet last run on es1009 is CRITICAL: CRITICAL: Puppet has 12 failures [18:02:21] PROBLEM - puppet last run on elastic1028 is CRITICAL: CRITICAL: puppet fail [18:02:22] PROBLEM - puppet last run on analytics1036 is CRITICAL: CRITICAL: Puppet has 27 failures [18:02:22] PROBLEM - puppet last run on cp1065 is CRITICAL: CRITICAL: Puppet has 30 failures [18:02:22] PROBLEM - puppet last run on cp1069 is CRITICAL: CRITICAL: Puppet has 19 failures [18:02:22] PROBLEM - puppet last run on mw1036 is CRITICAL: CRITICAL: Puppet has 32 failures [18:02:22] PROBLEM - puppet last run on mw1138 is CRITICAL: CRITICAL: Puppet has 34 failures [18:02:23] PROBLEM - puppet last run on mw1216 is CRITICAL: CRITICAL: puppet fail [18:02:24] PROBLEM - puppet last run on ytterbium is CRITICAL: CRITICAL: Puppet has 23 failures [18:02:31] PROBLEM - puppet last run on mw1252 is CRITICAL: CRITICAL: puppet fail [18:02:31] PROBLEM - puppet last run on virt1002 is CRITICAL: CRITICAL: puppet fail [18:02:32] PROBLEM - puppet last run on mw1040 is CRITICAL: CRITICAL: puppet fail [18:02:42] PROBLEM - puppet last run on labsdb1001 is CRITICAL: CRITICAL: puppet fail [18:02:42] PROBLEM - puppet last run on zirconium is CRITICAL: CRITICAL: Puppet has 35 failures [18:02:52] PROBLEM - puppet last run on cp1043 is CRITICAL: CRITICAL: Puppet has 23 failures [18:02:52] PROBLEM - puppet last run on elastic1026 is CRITICAL: CRITICAL: Puppet has 20 failures [18:02:52] PROBLEM - puppet last run on mw1048 is CRITICAL: CRITICAL: puppet fail [18:02:52] PROBLEM - puppet last run on db2041 is CRITICAL: CRITICAL: Puppet has 13 failures [18:02:52] PROBLEM - puppet last run on wtp1021 is CRITICAL: CRITICAL: puppet fail [18:03:01] PROBLEM - puppet last run on mw1257 is CRITICAL: CRITICAL: puppet fail [18:03:01] PROBLEM - puppet last run on virt1012 is CRITICAL: CRITICAL: Puppet has 10 failures [18:03:02] PROBLEM - puppet last run on mw1005 is CRITICAL: CRITICAL: puppet fail [18:03:02] PROBLEM - puppet last run on mw1132 is CRITICAL: CRITICAL: Puppet has 37 failures [18:03:02] PROBLEM - puppet last run on analytics1015 is CRITICAL: CRITICAL: Puppet has 18 failures [18:03:02] PROBLEM - puppet last run on cp3022 is CRITICAL: CRITICAL: puppet fail [18:03:02] PROBLEM - puppet last run on mc1010 is CRITICAL: CRITICAL: puppet fail [18:03:04] PROBLEM - puppet last run on es1006 is CRITICAL: CRITICAL: Puppet has 18 failures [18:03:04] PROBLEM - puppet last run on wtp1017 is CRITICAL: CRITICAL: puppet fail [18:03:04] PROBLEM - puppet last run on db1007 is CRITICAL: CRITICAL: Puppet has 20 failures [18:03:11] PROBLEM - puppet last run on mw1161 is CRITICAL: CRITICAL: puppet fail [18:03:11] PROBLEM - puppet last run on mw1218 is CRITICAL: CRITICAL: Puppet has 40 failures [18:03:11] PROBLEM - puppet last run on db1010 is CRITICAL: CRITICAL: puppet fail [18:03:12] PROBLEM - puppet last run on db2035 is CRITICAL: CRITICAL: Puppet has 11 failures [18:03:12] PROBLEM - puppet last run on mw1109 is CRITICAL: CRITICAL: Puppet has 39 failures [18:03:12] PROBLEM - puppet last run on ms-be2013 is CRITICAL: CRITICAL: puppet fail [18:03:12] PROBLEM - puppet last run on es2003 is CRITICAL: CRITICAL: puppet fail [18:03:21] PROBLEM - puppet last run on mw1124 is CRITICAL: CRITICAL: puppet fail [18:03:21] PROBLEM - puppet last run on dbstore1001 is CRITICAL: CRITICAL: puppet fail [18:03:22] PROBLEM - puppet last run on mw1147 is CRITICAL: CRITICAL: Puppet has 37 failures [18:03:22] PROBLEM - puppet last run on cp4007 is CRITICAL: CRITICAL: puppet fail [18:03:22] PROBLEM - puppet last run on mw1096 is CRITICAL: CRITICAL: Puppet has 65 failures [18:03:31] PROBLEM - puppet last run on amssq44 is CRITICAL: CRITICAL: puppet fail [18:03:31] PROBLEM - puppet last run on mw1130 is CRITICAL: CRITICAL: Puppet has 65 failures [18:03:31] PROBLEM - puppet last run on amssq57 is CRITICAL: CRITICAL: Puppet has 20 failures [18:03:31] PROBLEM - puppet last run on mc1011 is CRITICAL: CRITICAL: Puppet has 22 failures [18:03:31] PROBLEM - puppet last run on mw1221 is CRITICAL: CRITICAL: Puppet has 30 failures [18:03:31] PROBLEM - puppet last run on mw1240 is CRITICAL: CRITICAL: puppet fail [18:03:32] PROBLEM - puppet last run on analytics1039 is CRITICAL: CRITICAL: Puppet has 23 failures [18:03:32] PROBLEM - puppet last run on mw1067 is CRITICAL: CRITICAL: puppet fail [18:03:33] PROBLEM - puppet last run on vanadium is CRITICAL: CRITICAL: Puppet has 14 failures [18:03:41] PROBLEM - puppet last run on cp1059 is CRITICAL: CRITICAL: Puppet has 16 failures [18:03:42] PROBLEM - puppet last run on mc1004 is CRITICAL: CRITICAL: puppet fail [18:03:42] PROBLEM - puppet last run on cerium is CRITICAL: CRITICAL: puppet fail [18:03:52] PROBLEM - puppet last run on pc1001 is CRITICAL: CRITICAL: puppet fail [18:03:52] PROBLEM - puppet last run on mc1009 is CRITICAL: CRITICAL: puppet fail [18:03:52] PROBLEM - puppetmaster https on palladium is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 8140: HTTP/1.1 500 Internal Server Error [18:03:53] PROBLEM - puppet last run on carbon is CRITICAL: CRITICAL: puppet fail [18:04:01] PROBLEM - puppet last run on mw1115 is CRITICAL: CRITICAL: Puppet has 44 failures [18:04:01] PROBLEM - puppet last run on mw1233 is CRITICAL: CRITICAL: puppet fail [18:04:01] PROBLEM - puppet last run on ms-be1013 is CRITICAL: CRITICAL: Puppet has 11 failures [18:04:01] PROBLEM - puppet last run on mw1031 is CRITICAL: CRITICAL: Puppet has 69 failures [18:04:02] PROBLEM - puppet last run on ms-be2002 is CRITICAL: CRITICAL: puppet fail [18:04:11] PROBLEM - puppet last run on logstash1001 is CRITICAL: CRITICAL: Puppet has 35 failures [18:04:11] PROBLEM - puppet last run on uranium is CRITICAL: CRITICAL: puppet fail [18:04:11] PROBLEM - puppet last run on neptunium is CRITICAL: CRITICAL: Puppet has 32 failures [18:04:11] PROBLEM - puppet last run on ocg1001 is CRITICAL: CRITICAL: Puppet has 25 failures [18:04:11] PROBLEM - puppet last run on cp3006 is CRITICAL: CRITICAL: puppet fail [18:04:12] PROBLEM - puppet last run on db1065 is CRITICAL: CRITICAL: Puppet has 20 failures [18:04:12] PROBLEM - puppet last run on mw1145 is CRITICAL: CRITICAL: puppet fail [18:04:21] PROBLEM - puppet last run on mw1059 is CRITICAL: CRITICAL: puppet fail [18:04:21] PROBLEM - puppet last run on mw1178 is CRITICAL: CRITICAL: Puppet has 36 failures [18:04:22] PROBLEM - puppet last run on rdb1004 is CRITICAL: CRITICAL: puppet fail [18:04:22] PROBLEM - puppet last run on mw1089 is CRITICAL: CRITICAL: Puppet has 30 failures [18:04:22] PROBLEM - puppet last run on mw1256 is CRITICAL: CRITICAL: Puppet has 76 failures [18:04:22] PROBLEM - puppet last run on es2005 is CRITICAL: CRITICAL: Puppet has 11 failures [18:04:31] PROBLEM - puppet last run on mw1012 is CRITICAL: CRITICAL: puppet fail [18:04:31] PROBLEM - puppet last run on wtp1019 is CRITICAL: CRITICAL: Puppet has 13 failures [18:04:31] PROBLEM - puppet last run on ms-be2003 is CRITICAL: CRITICAL: puppet fail [18:04:31] PROBLEM - puppet last run on mw1080 is CRITICAL: CRITICAL: Puppet has 35 failures [18:04:31] PROBLEM - puppet last run on lvs4002 is CRITICAL: CRITICAL: puppet fail [18:04:32] PROBLEM - puppet last run on mw1200 is CRITICAL: CRITICAL: puppet fail [18:04:32] PROBLEM - puppet last run on mw1041 is CRITICAL: CRITICAL: Puppet has 31 failures [18:04:32] PROBLEM - puppet last run on caesium is CRITICAL: CRITICAL: puppet fail [18:04:33] PROBLEM - puppet last run on analytics1033 is CRITICAL: CRITICAL: puppet fail [18:04:33] PROBLEM - puppet last run on mw1038 is CRITICAL: CRITICAL: Puppet has 60 failures [18:04:34] PROBLEM - puppet last run on ms-be1001 is CRITICAL: CRITICAL: Puppet has 12 failures [18:04:41] PROBLEM - puppet last run on cp3020 is CRITICAL: CRITICAL: puppet fail [18:04:41] PROBLEM - puppet last run on amssq43 is CRITICAL: CRITICAL: Puppet has 15 failures [18:04:41] PROBLEM - puppet last run on cp3015 is CRITICAL: CRITICAL: Puppet has 14 failures [18:04:41] PROBLEM - puppet last run on mw1134 is CRITICAL: CRITICAL: Puppet has 34 failures [18:04:42] PROBLEM - puppet last run on elastic1031 is CRITICAL: CRITICAL: Puppet has 11 failures [18:04:42] PROBLEM - puppet last run on analytics1041 is CRITICAL: CRITICAL: puppet fail [18:04:42] PROBLEM - puppet last run on es1001 is CRITICAL: CRITICAL: Puppet has 21 failures [18:04:42] PROBLEM - puppet last run on mw1072 is CRITICAL: CRITICAL: Puppet has 31 failures [18:04:43] PROBLEM - puppet last run on netmon1001 is CRITICAL: CRITICAL: puppet fail [18:04:43] PROBLEM - puppet last run on mw1197 is CRITICAL: CRITICAL: puppet fail [18:04:44] PROBLEM - puppet last run on magnesium is CRITICAL: CRITICAL: puppet fail [18:04:44] PROBLEM - puppet last run on mw1254 is CRITICAL: CRITICAL: puppet fail [18:04:52] PROBLEM - puppet last run on db1029 is CRITICAL: CRITICAL: puppet fail [18:04:52] PROBLEM - puppet last run on db1053 is CRITICAL: CRITICAL: Puppet has 21 failures [18:04:52] PROBLEM - puppet last run on dbproxy1002 is CRITICAL: CRITICAL: Puppet has 11 failures [18:05:01] PROBLEM - puppet last run on logstash1003 is CRITICAL: CRITICAL: Puppet has 18 failures [18:05:01] PROBLEM - puppet last run on mw1045 is CRITICAL: CRITICAL: puppet fail [18:05:01] PROBLEM - puppet last run on cp1070 is CRITICAL: CRITICAL: puppet fail [18:05:02] PROBLEM - puppet last run on db1073 is CRITICAL: CRITICAL: puppet fail [18:05:02] PROBLEM - puppet last run on mw1106 is CRITICAL: CRITICAL: puppet fail [18:05:02] PROBLEM - puppet last run on ms-fe1003 is CRITICAL: CRITICAL: Puppet has 23 failures [18:05:02] PROBLEM - puppet last run on cp1049 is CRITICAL: CRITICAL: Puppet has 29 failures [18:05:02] PROBLEM - puppet last run on analytics1017 is CRITICAL: CRITICAL: puppet fail [18:05:02] PROBLEM - puppet last run on wtp1006 is CRITICAL: CRITICAL: Puppet has 31 failures [18:05:03] PROBLEM - puppet last run on mc1016 is CRITICAL: CRITICAL: Puppet has 13 failures [18:05:04] PROBLEM - puppet last run on mw1006 is CRITICAL: CRITICAL: puppet fail [18:05:11] RECOVERY - puppetmaster https on palladium is OK: HTTP OK: Status line output matched 400 - 335 bytes in 0.028 second response time [18:05:21] PROBLEM - puppet last run on db2034 is CRITICAL: CRITICAL: puppet fail [18:05:21] PROBLEM - puppet last run on cp4006 is CRITICAL: CRITICAL: puppet fail [18:05:21] PROBLEM - puppet last run on mw1141 is CRITICAL: CRITICAL: Puppet has 36 failures [18:05:21] PROBLEM - puppet last run on amssq32 is CRITICAL: CRITICAL: puppet fail [18:05:22] PROBLEM - puppet last run on amssq54 is CRITICAL: CRITICAL: puppet fail [18:05:22] PROBLEM - puppet last run on wtp1009 is CRITICAL: CRITICAL: Puppet has 17 failures [18:05:22] PROBLEM - puppet last run on mw1028 is CRITICAL: CRITICAL: Puppet has 28 failures [18:05:22] PROBLEM - puppet last run on mw1063 is CRITICAL: CRITICAL: Puppet has 75 failures [18:05:22] PROBLEM - puppet last run on amssq59 is CRITICAL: CRITICAL: Puppet has 20 failures [18:05:23] PROBLEM - puppet last run on amslvs2 is CRITICAL: CRITICAL: Puppet has 11 failures [18:05:23] PROBLEM - puppet last run on amssq49 is CRITICAL: CRITICAL: Puppet has 30 failures [18:05:24] PROBLEM - puppet last run on ms-be1004 is CRITICAL: CRITICAL: Puppet has 29 failures [18:05:31] PROBLEM - puppet last run on db1031 is CRITICAL: CRITICAL: puppet fail [18:05:31] PROBLEM - puppet last run on mw1007 is CRITICAL: CRITICAL: Puppet has 27 failures [18:05:32] PROBLEM - puppet last run on mw1026 is CRITICAL: CRITICAL: puppet fail [18:05:32] PROBLEM - puppet last run on mw1140 is CRITICAL: CRITICAL: Puppet has 64 failures [18:05:32] PROBLEM - puppet last run on elastic1007 is CRITICAL: CRITICAL: puppet fail [18:05:33] PROBLEM - puppet last run on analytics1040 is CRITICAL: CRITICAL: puppet fail [18:05:33] PROBLEM - puppet last run on cp1055 is CRITICAL: CRITICAL: puppet fail [18:05:41] PROBLEM - puppet last run on es2008 is CRITICAL: CRITICAL: Puppet has 17 failures [18:05:41] PROBLEM - puppet last run on heze is CRITICAL: CRITICAL: puppet fail [18:05:41] PROBLEM - puppet last run on labstore2001 is CRITICAL: CRITICAL: Puppet has 22 failures [18:05:41] PROBLEM - puppet last run on db2009 is CRITICAL: CRITICAL: Puppet has 9 failures [18:05:41] PROBLEM - puppet last run on snapshot1003 is CRITICAL: CRITICAL: Puppet has 21 failures [18:05:41] PROBLEM - puppet last run on db2019 is CRITICAL: CRITICAL: puppet fail [18:05:41] PROBLEM - puppet last run on mw1187 is CRITICAL: CRITICAL: Puppet has 37 failures [18:05:42] PROBLEM - puppet last run on es1008 is CRITICAL: CRITICAL: Puppet has 12 failures [18:05:42] PROBLEM - puppet last run on cp1047 is CRITICAL: CRITICAL: puppet fail [18:05:55] PROBLEM - puppet last run on amssq53 is CRITICAL: CRITICAL: puppet fail [18:05:55] PROBLEM - puppet last run on cp3008 is CRITICAL: CRITICAL: Puppet has 14 failures [18:05:55] PROBLEM - puppet last run on cp3003 is CRITICAL: CRITICAL: puppet fail [18:05:55] PROBLEM - puppet last run on wtp1020 is CRITICAL: CRITICAL: Puppet has 18 failures [18:05:55] PROBLEM - puppet last run on elastic1004 is CRITICAL: CRITICAL: Puppet has 8 failures [18:05:55] PROBLEM - puppet last run on gold is CRITICAL: CRITICAL: puppet fail [18:05:55] PROBLEM - puppet last run on ms-be1003 is CRITICAL: CRITICAL: Puppet has 10 failures [18:06:01] PROBLEM - puppet last run on sca1002 is CRITICAL: CRITICAL: puppet fail [18:06:01] PROBLEM - puppet last run on rbf1002 is CRITICAL: CRITICAL: Puppet has 13 failures [18:06:01] PROBLEM - puppet last run on lvs1002 is CRITICAL: CRITICAL: puppet fail [18:06:01] PROBLEM - puppet last run on mw1160 is CRITICAL: CRITICAL: Puppet has 38 failures [18:06:02] PROBLEM - puppet last run on mw1250 is CRITICAL: CRITICAL: puppet fail [18:06:02] PROBLEM - puppet last run on xenon is CRITICAL: CRITICAL: puppet fail [18:06:02] PROBLEM - puppet last run on analytics1025 is CRITICAL: CRITICAL: puppet fail [18:06:03] PROBLEM - puppet last run on cp1039 is CRITICAL: CRITICAL: puppet fail [18:06:03] PROBLEM - puppet last run on searchidx1001 is CRITICAL: CRITICAL: Puppet has 22 failures [18:06:03] PROBLEM - puppet last run on analytics1020 is CRITICAL: CRITICAL: Puppet has 24 failures [18:06:11] https://git.wikimedia.org/summary/mediawiki%2Fextensions%2FWikibase :( [18:06:12] PROBLEM - puppet last run on elastic1001 is CRITICAL: CRITICAL: Puppet has 18 failures [18:06:12] PROBLEM - puppet last run on mw1224 is CRITICAL: CRITICAL: puppet fail [18:06:21] PROBLEM - puppet last run on platinum is CRITICAL: CRITICAL: Puppet has 21 failures [18:06:21] PROBLEM - puppet last run on ms-be1006 is CRITICAL: CRITICAL: Puppet has 25 failures [18:06:22] PROBLEM - puppet last run on helium is CRITICAL: CRITICAL: puppet fail [18:06:22] PROBLEM - puppet last run on mw1174 is CRITICAL: CRITICAL: puppet fail [18:06:22] PROBLEM - puppet last run on mc1006 is CRITICAL: CRITICAL: Puppet has 10 failures [18:06:31] PROBLEM - puppet last run on ms-fe1001 is CRITICAL: CRITICAL: puppet fail [18:06:31] PROBLEM - puppet last run on ms-fe2004 is CRITICAL: CRITICAL: Puppet has 29 failures [18:06:31] PROBLEM - puppet last run on db2039 is CRITICAL: CRITICAL: Puppet has 13 failures [18:06:31] PROBLEM - puppet last run on ms-be2004 is CRITICAL: CRITICAL: puppet fail [18:06:32] PROBLEM - puppet last run on amssq61 is CRITICAL: CRITICAL: Puppet has 17 failures [18:06:32] PROBLEM - puppet last run on mw1226 is CRITICAL: CRITICAL: Puppet has 29 failures [18:06:32] PROBLEM - puppet last run on potassium is CRITICAL: CRITICAL: Puppet has 21 failures [18:06:41] PROBLEM - puppet last run on lead is CRITICAL: CRITICAL: Puppet has 9 failures [18:06:41] PROBLEM - puppet last run on mw1082 is CRITICAL: CRITICAL: Puppet has 31 failures [18:06:41] PROBLEM - puppet last run on iron is CRITICAL: CRITICAL: puppet fail [18:06:42] PROBLEM - puppet last run on mw1099 is CRITICAL: CRITICAL: Puppet has 34 failures [18:06:51] PROBLEM - puppet last run on mw1222 is CRITICAL: CRITICAL: puppet fail [18:06:51] PROBLEM - puppet last run on capella is CRITICAL: CRITICAL: Puppet has 12 failures [18:06:51] PROBLEM - puppet last run on mw1228 is CRITICAL: CRITICAL: Puppet has 68 failures [18:06:51] PROBLEM - puppet last run on db1022 is CRITICAL: CRITICAL: puppet fail [18:06:52] PROBLEM - puppet last run on mw1176 is CRITICAL: CRITICAL: Puppet has 30 failures [18:06:52] PROBLEM - puppet last run on es2002 is CRITICAL: CRITICAL: Puppet has 9 failures [18:06:52] PROBLEM - puppet last run on db2002 is CRITICAL: CRITICAL: puppet fail [18:06:52] PROBLEM - puppet last run on db2005 is CRITICAL: CRITICAL: Puppet has 21 failures [18:06:52] PROBLEM - puppet last run on mw1164 is CRITICAL: CRITICAL: puppet fail [18:06:53] PROBLEM - puppet last run on mw1003 is CRITICAL: CRITICAL: puppet fail [18:06:53] PROBLEM - puppet last run on mc1002 is CRITICAL: CRITICAL: Puppet has 12 failures [18:06:54] PROBLEM - puppet last run on ms-be2006 is CRITICAL: CRITICAL: Puppet has 28 failures [18:06:55] PROBLEM - puppet last run on mw1100 is CRITICAL: CRITICAL: puppet fail [18:07:01] PROBLEM - puppet last run on lvs1005 is CRITICAL: CRITICAL: puppet fail [18:07:01] PROBLEM - puppet last run on mw1068 is CRITICAL: CRITICAL: puppet fail [18:07:01] PROBLEM - puppet last run on db1050 is CRITICAL: CRITICAL: puppet fail [18:07:01] PROBLEM - puppet last run on cp4008 is CRITICAL: CRITICAL: puppet fail [18:07:01] PROBLEM - puppet last run on mw1242 is CRITICAL: CRITICAL: puppet fail [18:07:02] PROBLEM - puppet last run on mw1088 is CRITICAL: CRITICAL: puppet fail [18:07:02] PROBLEM - puppet last run on mw1173 is CRITICAL: CRITICAL: Puppet has 36 failures [18:07:02] PROBLEM - puppet last run on elastic1008 is CRITICAL: CRITICAL: puppet fail [18:07:02] PROBLEM - puppet last run on analytics1035 is CRITICAL: CRITICAL: Puppet has 12 failures [18:07:11] PROBLEM - puppet last run on mw1189 is CRITICAL: CRITICAL: puppet fail [18:07:11] PROBLEM - puppet last run on mw1150 is CRITICAL: CRITICAL: Puppet has 80 failures [18:07:11] PROBLEM - puppet last run on elastic1021 is CRITICAL: CRITICAL: Puppet has 22 failures [18:07:12] PROBLEM - puppet last run on mc1003 is CRITICAL: CRITICAL: Puppet has 14 failures [18:07:12] PROBLEM - puppet last run on virt1006 is CRITICAL: CRITICAL: puppet fail [18:07:12] PROBLEM - puppet last run on mw1060 is CRITICAL: CRITICAL: puppet fail [18:07:12] PROBLEM - puppet last run on wtp1016 is CRITICAL: CRITICAL: Puppet has 26 failures [18:07:12] PROBLEM - puppet last run on elastic1012 is CRITICAL: CRITICAL: puppet fail [18:07:21] PROBLEM - puppet last run on mw1117 is CRITICAL: CRITICAL: puppet fail [18:07:21] PROBLEM - puppet last run on db1002 is CRITICAL: CRITICAL: Puppet has 13 failures [18:07:21] PROBLEM - puppet last run on labstore1003 is CRITICAL: CRITICAL: puppet fail [18:07:22] PROBLEM - puppet last run on mw1166 is CRITICAL: CRITICAL: puppet fail [18:07:31] PROBLEM - puppet last run on mw1008 is CRITICAL: CRITICAL: Puppet has 60 failures [18:07:31] PROBLEM - puppet last run on mw1092 is CRITICAL: CRITICAL: puppet fail [18:07:31] PROBLEM - puppet last run on mw1120 is CRITICAL: CRITICAL: Puppet has 79 failures [18:07:31] PROBLEM - puppet last run on elastic1018 is CRITICAL: CRITICAL: Puppet has 17 failures [18:07:32] PROBLEM - puppet last run on mw1217 is CRITICAL: CRITICAL: Puppet has 78 failures [18:07:32] PROBLEM - puppet last run on db1066 is CRITICAL: CRITICAL: Puppet has 15 failures [18:07:32] PROBLEM - puppet last run on mw1205 is CRITICAL: CRITICAL: Puppet has 68 failures [18:07:32] PROBLEM - puppet last run on mw1069 is CRITICAL: CRITICAL: Puppet has 76 failures [18:07:32] PROBLEM - puppet last run on db1046 is CRITICAL: CRITICAL: puppet fail [18:07:41] PROBLEM - puppet last run on db1040 is CRITICAL: CRITICAL: puppet fail [18:07:41] PROBLEM - puppet last run on cp4003 is CRITICAL: CRITICAL: puppet fail [18:07:42] PROBLEM - puppet last run on mw1235 is CRITICAL: CRITICAL: puppet fail [18:07:42] PROBLEM - puppet last run on elastic1027 is CRITICAL: CRITICAL: Puppet has 15 failures [18:07:51] PROBLEM - puppet last run on amssq35 is CRITICAL: CRITICAL: Puppet has 15 failures [18:07:51] PROBLEM - puppet last run on cp3016 is CRITICAL: CRITICAL: Puppet has 28 failures [18:07:52] PROBLEM - puppet last run on labsdb1003 is CRITICAL: CRITICAL: puppet fail [18:07:52] PROBLEM - puppet last run on mw1009 is CRITICAL: CRITICAL: Puppet has 56 failures [18:07:52] PROBLEM - puppet last run on mw1046 is CRITICAL: CRITICAL: Puppet has 71 failures [18:08:01] PROBLEM - puppet last run on lvs2004 is CRITICAL: CRITICAL: Puppet has 9 failures [18:08:01] PROBLEM - puppet last run on gallium is CRITICAL: CRITICAL: puppet fail [18:08:01] PROBLEM - puppet last run on labcontrol2001 is CRITICAL: CRITICAL: puppet fail [18:08:01] PROBLEM - puppet last run on es2009 is CRITICAL: CRITICAL: Puppet has 21 failures [18:08:02] PROBLEM - puppet last run on es2001 is CRITICAL: CRITICAL: Puppet has 22 failures [18:08:02] PROBLEM - puppet last run on db2018 is CRITICAL: CRITICAL: puppet fail [18:08:02] PROBLEM - puppet last run on mw1039 is CRITICAL: CRITICAL: puppet fail [18:08:02] PROBLEM - puppet last run on cp1061 is CRITICAL: CRITICAL: Puppet has 26 failures [18:08:02] PROBLEM - puppet last run on mw1172 is CRITICAL: CRITICAL: puppet fail [18:08:11] PROBLEM - puppet last run on db1042 is CRITICAL: CRITICAL: puppet fail [18:08:11] PROBLEM - puppet last run on ruthenium is CRITICAL: CRITICAL: Puppet has 12 failures [18:08:11] PROBLEM - puppet last run on ms-fe1004 is CRITICAL: CRITICAL: Puppet has 20 failures [18:08:11] PROBLEM - puppet last run on mw1153 is CRITICAL: CRITICAL: Puppet has 42 failures [18:08:11] PROBLEM - puppet last run on cp4004 is CRITICAL: CRITICAL: puppet fail [18:08:12] PROBLEM - puppet last run on mw1025 is CRITICAL: CRITICAL: Puppet has 61 failures [18:08:12] PROBLEM - puppet last run on lvs3001 is CRITICAL: CRITICAL: Puppet has 12 failures [18:08:12] PROBLEM - puppet last run on amssq46 is CRITICAL: CRITICAL: puppet fail [18:08:21] PROBLEM - puppet last run on mw1144 is CRITICAL: CRITICAL: Puppet has 76 failures [18:08:21] PROBLEM - puppet last run on mw1211 is CRITICAL: CRITICAL: puppet fail [18:08:22] PROBLEM - puppet last run on mw1052 is CRITICAL: CRITICAL: Puppet has 43 failures [18:08:22] PROBLEM - puppet last run on mw1123 is CRITICAL: CRITICAL: Puppet has 46 failures [18:08:22] PROBLEM - puppet last run on cp1056 is CRITICAL: CRITICAL: Puppet has 15 failures [18:08:22] PROBLEM - puppet last run on analytics1030 is CRITICAL: CRITICAL: Puppet has 20 failures [18:08:22] PROBLEM - puppet last run on dbproxy1001 is CRITICAL: CRITICAL: puppet fail [18:08:23] PROBLEM - puppet last run on db1028 is CRITICAL: CRITICAL: puppet fail [18:08:31] PROBLEM - puppet last run on db1059 is CRITICAL: CRITICAL: puppet fail [18:08:31] PROBLEM - puppet last run on db1023 is CRITICAL: CRITICAL: puppet fail [18:08:32] PROBLEM - puppet last run on db1018 is CRITICAL: CRITICAL: Puppet has 13 failures [18:08:32] PROBLEM - puppet last run on analytics1010 is CRITICAL: CRITICAL: Puppet has 9 failures [18:08:32] PROBLEM - puppet last run on db1067 is CRITICAL: CRITICAL: puppet fail [18:08:32] PROBLEM - puppet last run on db1051 is CRITICAL: CRITICAL: puppet fail [18:08:32] PROBLEM - puppet last run on mw1042 is CRITICAL: CRITICAL: Puppet has 37 failures [18:08:41] PROBLEM - puppet last run on db1015 is CRITICAL: CRITICAL: puppet fail [18:08:41] PROBLEM - puppet last run on mw1177 is CRITICAL: CRITICAL: puppet fail [18:08:41] PROBLEM - puppet last run on mw1061 is CRITICAL: CRITICAL: puppet fail [18:08:42] PROBLEM - puppet last run on elastic1022 is CRITICAL: CRITICAL: puppet fail [18:08:42] PROBLEM - puppet last run on lvs2001 is CRITICAL: CRITICAL: puppet fail [18:08:42] PROBLEM - puppet last run on mw1126 is CRITICAL: CRITICAL: puppet fail [18:08:42] PROBLEM - puppet last run on logstash1002 is CRITICAL: CRITICAL: puppet fail [18:08:51] PROBLEM - puppet last run on ms-fe2001 is CRITICAL: CRITICAL: Puppet has 26 failures [18:08:51] PROBLEM - puppet last run on ms-fe2003 is CRITICAL: CRITICAL: puppet fail [18:08:52] PROBLEM - puppet last run on db2036 is CRITICAL: CRITICAL: puppet fail [18:08:52] PROBLEM - puppet last run on mw1119 is CRITICAL: CRITICAL: Puppet has 72 failures [18:08:52] PROBLEM - puppet last run on mw1065 is CRITICAL: CRITICAL: Puppet has 70 failures [18:09:02] PROBLEM - puppet last run on mw1118 is CRITICAL: CRITICAL: Puppet has 74 failures [18:09:02] PROBLEM - puppet last run on amslvs1 is CRITICAL: CRITICAL: Puppet has 11 failures [18:09:02] PROBLEM - puppet last run on cp3010 is CRITICAL: CRITICAL: puppet fail [18:09:02] PROBLEM - puppet last run on mw1002 is CRITICAL: CRITICAL: puppet fail [18:09:02] PROBLEM - puppet last run on mw1011 is CRITICAL: CRITICAL: Puppet has 27 failures [18:09:02] PROBLEM - puppet last run on mw1170 is CRITICAL: CRITICAL: Puppet has 65 failures [18:09:03] PROBLEM - puppet last run on db1003 is CRITICAL: CRITICAL: puppet fail [18:09:11] PROBLEM - puppet last run on mw1114 is CRITICAL: CRITICAL: Puppet has 35 failures [18:09:11] PROBLEM - puppet last run on mc1012 is CRITICAL: CRITICAL: puppet fail [18:09:11] PROBLEM - puppet last run on db2040 is CRITICAL: CRITICAL: Puppet has 11 failures [18:09:12] PROBLEM - puppet last run on elastic1030 is CRITICAL: CRITICAL: Puppet has 24 failures [18:09:22] PROBLEM - puppet last run on antimony is CRITICAL: CRITICAL: puppet fail [18:09:23] PROBLEM - puppet last run on mw1129 is CRITICAL: CRITICAL: Puppet has 30 failures [18:09:23] PROBLEM - puppet last run on mw1054 is CRITICAL: CRITICAL: puppet fail [18:09:23] PROBLEM - puppet last run on ms-be3002 is CRITICAL: CRITICAL: puppet fail [18:09:31] PROBLEM - puppet last run on cp3014 is CRITICAL: CRITICAL: Puppet has 28 failures [18:09:31] PROBLEM - puppet last run on mw1149 is CRITICAL: CRITICAL: puppet fail [18:09:31] PROBLEM - puppet last run on db1043 is CRITICAL: CRITICAL: puppet fail [18:09:32] PROBLEM - puppet last run on labnet1001 is CRITICAL: CRITICAL: puppet fail [18:09:32] PROBLEM - puppet last run on mw1195 is CRITICAL: CRITICAL: puppet fail [18:09:32] PROBLEM - puppet last run on cp1058 is CRITICAL: CRITICAL: Puppet has 16 failures [18:09:32] PROBLEM - puppet last run on mw1175 is CRITICAL: CRITICAL: Puppet has 70 failures [18:09:32] PROBLEM - puppet last run on virt1004 is CRITICAL: CRITICAL: puppet fail [18:09:41] PROBLEM - puppet last run on mw1249 is CRITICAL: CRITICAL: Puppet has 64 failures [18:09:41] PROBLEM - puppet last run on db1021 is CRITICAL: CRITICAL: Puppet has 20 failures [18:09:41] PROBLEM - puppet last run on mw1251 is CRITICAL: CRITICAL: Puppet has 78 failures [18:09:41] PROBLEM - puppet last run on mw1213 is CRITICAL: CRITICAL: puppet fail [18:09:42] PROBLEM - puppet last run on mw1237 is CRITICAL: CRITICAL: puppet fail [18:09:42] PROBLEM - puppet last run on db1034 is CRITICAL: CRITICAL: Puppet has 24 failures [18:09:42] PROBLEM - puppet last run on db1052 is CRITICAL: CRITICAL: puppet fail [18:09:42] PROBLEM - puppet last run on labstore1001 is CRITICAL: CRITICAL: puppet fail [18:09:51] PROBLEM - puppet last run on tin is CRITICAL: CRITICAL: Puppet has 33 failures [18:09:51] PROBLEM - puppet last run on mw1051 is CRITICAL: CRITICAL: puppet fail [18:09:51] PROBLEM - puppet last run on stat1003 is CRITICAL: CRITICAL: puppet fail [18:09:52] PROBLEM - puppet last run on mw1084 is CRITICAL: CRITICAL: puppet fail [18:09:52] PROBLEM - puppet last run on polonium is CRITICAL: CRITICAL: puppet fail [18:09:52] PROBLEM - puppet last run on mw1162 is CRITICAL: CRITICAL: puppet fail [18:10:01] PROBLEM - puppet last run on lithium is CRITICAL: CRITICAL: puppet fail [18:10:02] PROBLEM - puppet last run on db2042 is CRITICAL: CRITICAL: Puppet has 10 failures [18:10:02] PROBLEM - puppet last run on wtp1005 is CRITICAL: CRITICAL: Puppet has 29 failures [18:10:02] PROBLEM - puppet last run on db2038 is CRITICAL: CRITICAL: puppet fail [18:10:02] PROBLEM - puppet last run on ms-be2011 is CRITICAL: CRITICAL: puppet fail [18:10:11] PROBLEM - puppet last run on mw1044 is CRITICAL: CRITICAL: puppet fail [18:10:11] PROBLEM - puppet last run on cp4001 is CRITICAL: CRITICAL: puppet fail [18:10:11] PROBLEM - puppet last run on cp4014 is CRITICAL: CRITICAL: Puppet has 14 failures [18:10:11] PROBLEM - puppet last run on pc1002 is CRITICAL: CRITICAL: Puppet has 17 failures [18:10:12] PROBLEM - puppet last run on virt1003 is CRITICAL: CRITICAL: Puppet has 27 failures [18:10:12] PROBLEM - puppet last run on nescio is CRITICAL: CRITICAL: puppet fail [18:10:12] PROBLEM - puppet last run on amssq34 is CRITICAL: CRITICAL: puppet fail [18:10:12] PROBLEM - puppet last run on amssq40 is CRITICAL: CRITICAL: puppet fail [18:10:12] PROBLEM - puppet last run on amssq51 is CRITICAL: CRITICAL: puppet fail [18:10:13] PROBLEM - puppet last run on amssq55 is CRITICAL: CRITICAL: Puppet has 28 failures [18:10:14] PROBLEM - puppet last run on amssq60 is CRITICAL: CRITICAL: Puppet has 17 failures [18:10:14] PROBLEM - puppet last run on amssq48 is CRITICAL: CRITICAL: Puppet has 15 failures [18:10:15] PROBLEM - puppet last run on virt1001 is CRITICAL: CRITICAL: puppet fail [18:10:15] PROBLEM - puppet last run on mw1208 is CRITICAL: CRITICAL: Puppet has 66 failures [18:10:21] PROBLEM - puppet last run on db1016 is CRITICAL: CRITICAL: Puppet has 9 failures [18:10:21] PROBLEM - puppet last run on sodium is CRITICAL: CRITICAL: puppet fail [18:10:21] PROBLEM - puppet last run on db1060 is CRITICAL: CRITICAL: puppet fail [18:10:21] PROBLEM - puppet last run on analytics1022 is CRITICAL: CRITICAL: puppet fail [18:10:21] PROBLEM - puppet last run on dataset1001 is CRITICAL: CRITICAL: puppet fail [18:10:22] PROBLEM - puppet last run on db1039 is CRITICAL: CRITICAL: puppet fail [18:10:22] PROBLEM - puppet last run on es2004 is CRITICAL: CRITICAL: puppet fail [18:10:22] PROBLEM - puppet last run on db2029 is CRITICAL: CRITICAL: Puppet has 7 failures [18:10:22] PROBLEM - puppet last run on db2007 is CRITICAL: CRITICAL: Puppet has 17 failures [18:10:31] PROBLEM - puppet last run on nembus is CRITICAL: CRITICAL: puppet fail [18:10:31] PROBLEM - puppet last run on mw1206 is CRITICAL: CRITICAL: Puppet has 75 failures [18:10:31] PROBLEM - puppet last run on wtp1012 is CRITICAL: CRITICAL: Puppet has 17 failures [18:10:31] PROBLEM - puppet last run on install2001 is CRITICAL: CRITICAL: Puppet has 23 failures [18:10:31] PROBLEM - puppet last run on analytics1016 is CRITICAL: CRITICAL: Puppet has 23 failures [18:10:32] PROBLEM - puppet last run on snapshot1002 is CRITICAL: CRITICAL: puppet fail [18:10:32] PROBLEM - puppet last run on analytics1038 is CRITICAL: CRITICAL: Puppet has 26 failures [18:10:32] PROBLEM - puppet last run on bast4001 is CRITICAL: CRITICAL: puppet fail [18:10:33] PROBLEM - puppet last run on cp4019 is CRITICAL: CRITICAL: Puppet has 32 failures [18:10:33] PROBLEM - puppet last run on cp4005 is CRITICAL: CRITICAL: puppet fail [18:10:41] PROBLEM - puppet last run on ms-be3001 is CRITICAL: CRITICAL: puppet fail [18:10:41] PROBLEM - puppet last run on amssq56 is CRITICAL: CRITICAL: puppet fail [18:10:41] PROBLEM - puppet last run on amssq47 is CRITICAL: CRITICAL: Puppet has 16 failures [18:10:41] PROBLEM - puppet last run on analytics1026 is CRITICAL: CRITICAL: puppet fail [18:10:41] PROBLEM - puppet last run on labmon1001 is CRITICAL: CRITICAL: puppet fail [18:10:41] PROBLEM - puppet last run on mw1049 is CRITICAL: CRITICAL: puppet fail [18:10:42] PROBLEM - puppet last run on elastic1019 is CRITICAL: CRITICAL: puppet fail [18:10:52] PROBLEM - puppet last run on plutonium is CRITICAL: CRITICAL: puppet fail [18:10:52] PROBLEM - puppet last run on gadolinium is CRITICAL: CRITICAL: puppet fail [18:10:52] PROBLEM - puppet last run on mw1165 is CRITICAL: CRITICAL: puppet fail [18:10:52] PROBLEM - puppet last run on es1007 is CRITICAL: CRITICAL: Puppet has 12 failures [18:10:52] PROBLEM - puppet last run on mc1005 is CRITICAL: CRITICAL: puppet fail [18:11:01] PROBLEM - puppet last run on snapshot1001 is CRITICAL: CRITICAL: Puppet has 48 failures [18:11:01] PROBLEM - puppet last run on mw1076 is CRITICAL: CRITICAL: Puppet has 81 failures [18:11:01] PROBLEM - puppet last run on mw1247 is CRITICAL: CRITICAL: puppet fail [18:11:01] PROBLEM - puppet last run on cp1050 is CRITICAL: CRITICAL: Puppet has 30 failures [18:11:01] PROBLEM - puppet last run on mw1098 is CRITICAL: CRITICAL: puppet fail [18:11:02] PROBLEM - puppet last run on mw1055 is CRITICAL: CRITICAL: Puppet has 72 failures [18:11:02] PROBLEM - puppet last run on cp1046 is CRITICAL: CRITICAL: puppet fail [18:11:02] PROBLEM - puppet last run on db1048 is CRITICAL: CRITICAL: Puppet has 11 failures [18:11:11] PROBLEM - puppet last run on lvs2006 is CRITICAL: CRITICAL: Puppet has 14 failures [18:11:11] PROBLEM - puppet last run on mw1190 is CRITICAL: CRITICAL: puppet fail [18:11:11] PROBLEM - puppet last run on db1026 is CRITICAL: CRITICAL: Puppet has 21 failures [18:11:11] PROBLEM - puppet last run on mw1238 is CRITICAL: CRITICAL: Puppet has 68 failures [18:11:12] PROBLEM - puppet last run on elastic1024 is CRITICAL: CRITICAL: Puppet has 22 failures [18:11:21] PROBLEM - puppet last run on lvs4003 is CRITICAL: CRITICAL: puppet fail [18:11:22] PROBLEM - puppet last run on mw1111 is CRITICAL: CRITICAL: Puppet has 73 failures [18:11:22] PROBLEM - puppet last run on mw1133 is CRITICAL: CRITICAL: puppet fail [18:11:22] PROBLEM - puppet last run on mw1079 is CRITICAL: CRITICAL: Puppet has 31 failures [18:11:22] PROBLEM - puppet last run on cp4018 is CRITICAL: CRITICAL: puppet fail [18:11:22] PROBLEM - puppet last run on rubidium is CRITICAL: CRITICAL: puppet fail [18:11:23] PROBLEM - puppet last run on mw1014 is CRITICAL: CRITICAL: Puppet has 56 failures [18:11:23] PROBLEM - puppet last run on db1036 is CRITICAL: CRITICAL: Puppet has 9 failures [18:11:24] PROBLEM - puppet last run on analytics1023 is CRITICAL: CRITICAL: puppet fail [18:11:24] PROBLEM - puppet last run on ms-fe3002 is CRITICAL: CRITICAL: Puppet has 26 failures [18:11:31] PROBLEM - puppet last run on osmium is CRITICAL: CRITICAL: puppet fail [18:11:31] PROBLEM - puppet last run on mw1156 is CRITICAL: CRITICAL: puppet fail [18:11:31] PROBLEM - puppet last run on cp1063 is CRITICAL: CRITICAL: puppet fail [18:11:32] PROBLEM - puppet last run on mw1202 is CRITICAL: CRITICAL: puppet fail [18:11:32] PROBLEM - puppet last run on oxygen is CRITICAL: CRITICAL: Puppet has 17 failures [18:11:32] PROBLEM - puppet last run on rhenium is CRITICAL: CRITICAL: Puppet has 10 failures [18:11:32] PROBLEM - puppet last run on elastic1015 is CRITICAL: CRITICAL: puppet fail [18:11:32] PROBLEM - puppet last run on mw1227 is CRITICAL: CRITICAL: puppet fail [18:11:32] PROBLEM - puppet last run on mw1180 is CRITICAL: CRITICAL: Puppet has 34 failures [18:11:33] PROBLEM - puppet last run on mw1146 is CRITICAL: CRITICAL: puppet fail [18:11:41] PROBLEM - puppet last run on analytics1013 is CRITICAL: CRITICAL: Puppet has 25 failures [18:11:41] PROBLEM - puppet last run on cp1048 is CRITICAL: CRITICAL: Puppet has 25 failures [18:11:42] PROBLEM - puppet last run on ms-be2012 is CRITICAL: CRITICAL: Puppet has 27 failures [18:11:42] PROBLEM - puppet last run on db2016 is CRITICAL: CRITICAL: puppet fail [18:11:42] PROBLEM - puppet last run on rdb1001 is CRITICAL: CRITICAL: puppet fail [18:11:42] PROBLEM - puppet last run on ms-be2008 is CRITICAL: CRITICAL: puppet fail [18:11:42] PROBLEM - puppet last run on mc1014 is CRITICAL: CRITICAL: Puppet has 11 failures [18:11:42] PROBLEM - puppet last run on ms-be1012 is CRITICAL: CRITICAL: puppet fail [18:11:51] PROBLEM - puppet last run on mw1004 is CRITICAL: CRITICAL: puppet fail [18:11:51] PROBLEM - puppet last run on mw1258 is CRITICAL: CRITICAL: Puppet has 69 failures [18:11:51] PROBLEM - puppet last run on amssq36 is CRITICAL: CRITICAL: Puppet has 34 failures [18:11:51] PROBLEM - puppet last run on virt1007 is CRITICAL: CRITICAL: Puppet has 23 failures [18:11:52] PROBLEM - puppet last run on elastic1006 is CRITICAL: CRITICAL: Puppet has 13 failures [18:11:52] PROBLEM - puppet last run on ms-be1009 is CRITICAL: CRITICAL: puppet fail [18:11:52] PROBLEM - puppet last run on mw1183 is CRITICAL: CRITICAL: puppet fail [18:11:52] PROBLEM - puppet last run on tungsten is CRITICAL: CRITICAL: puppet fail [18:11:52] PROBLEM - puppet last run on mw1081 is CRITICAL: CRITICAL: puppet fail [18:11:53] PROBLEM - puppet last run on mw1168 is CRITICAL: CRITICAL: Puppet has 37 failures [18:12:01] PROBLEM - puppet last run on mw1210 is CRITICAL: CRITICAL: puppet fail [18:12:01] PROBLEM - puppet last run on mw1151 is CRITICAL: CRITICAL: Puppet has 34 failures [18:12:01] PROBLEM - puppet last run on labsdb1006 is CRITICAL: CRITICAL: puppet fail [18:12:01] PROBLEM - puppet last run on hafnium is CRITICAL: CRITICAL: Puppet has 25 failures [18:12:02] PROBLEM - puppet last run on wtp1018 is CRITICAL: CRITICAL: puppet fail [18:12:02] PROBLEM - puppet last run on virt1010 is CRITICAL: CRITICAL: Puppet has 19 failures [18:12:02] PROBLEM - puppet last run on mw1125 is CRITICAL: CRITICAL: Puppet has 74 failures [18:12:02] PROBLEM - puppet last run on mw1159 is CRITICAL: CRITICAL: puppet fail [18:12:02] PROBLEM - puppet last run on db1055 is CRITICAL: CRITICAL: puppet fail [18:12:03] PROBLEM - puppet last run on cp1062 is CRITICAL: CRITICAL: Puppet has 29 failures [18:12:11] PROBLEM - puppet last run on lvs3004 is CRITICAL: CRITICAL: Puppet has 14 failures [18:12:11] PROBLEM - puppet last run on db1069 is CRITICAL: CRITICAL: puppet fail [18:12:11] PROBLEM - puppet last run on snapshot1004 is CRITICAL: CRITICAL: Puppet has 23 failures [18:12:12] PROBLEM - puppet last run on db1004 is CRITICAL: CRITICAL: puppet fail [18:12:12] PROBLEM - puppet last run on mw1023 is CRITICAL: CRITICAL: puppet fail [18:12:12] PROBLEM - puppet last run on lvs1001 is CRITICAL: CRITICAL: puppet fail [18:12:21] PROBLEM - puppet last run on db1071 is CRITICAL: CRITICAL: puppet fail [18:12:21] PROBLEM - puppet last run on mw1056 is CRITICAL: CRITICAL: Puppet has 72 failures [18:12:21] PROBLEM - puppet last run on mw1181 is CRITICAL: CRITICAL: puppet fail [18:12:22] PROBLEM - puppet last run on mw1034 is CRITICAL: CRITICAL: Puppet has 39 failures [18:12:31] PROBLEM - puppet last run on db2037 is CRITICAL: CRITICAL: Puppet has 18 failures [18:12:31] PROBLEM - puppet last run on ms-be2001 is CRITICAL: CRITICAL: Puppet has 26 failures [18:12:31] PROBLEM - puppet last run on cp1038 is CRITICAL: CRITICAL: puppet fail [18:12:31] PROBLEM - puppet last run on argon is CRITICAL: CRITICAL: Puppet has 11 failures [18:12:32] PROBLEM - puppet last run on mw1030 is CRITICAL: CRITICAL: Puppet has 74 failures [18:12:32] PROBLEM - puppet last run on stat1002 is CRITICAL: CRITICAL: Puppet has 42 failures [18:12:32] PROBLEM - puppet last run on hooft is CRITICAL: CRITICAL: puppet fail [18:12:32] PROBLEM - puppet last run on mw1248 is CRITICAL: CRITICAL: puppet fail [18:12:33] PROBLEM - puppet last run on mw1029 is CRITICAL: CRITICAL: puppet fail [18:12:41] PROBLEM - puppet last run on thallium is CRITICAL: CRITICAL: Puppet has 12 failures [18:12:51] PROBLEM - puppet last run on ms-be1007 is CRITICAL: CRITICAL: puppet fail [18:12:51] PROBLEM - puppet last run on mc1001 is CRITICAL: CRITICAL: puppet fail [18:12:51] PROBLEM - puppet last run on ms-be1008 is CRITICAL: CRITICAL: Puppet has 32 failures [18:12:51] PROBLEM - puppet last run on wtp1002 is CRITICAL: CRITICAL: Puppet has 13 failures [18:12:51] PROBLEM - puppet last run on es2007 is CRITICAL: CRITICAL: Puppet has 21 failures [18:12:51] PROBLEM - puppet last run on db2004 is CRITICAL: CRITICAL: Puppet has 15 failures [18:12:52] PROBLEM - puppet last run on ms-be2005 is CRITICAL: CRITICAL: Puppet has 25 failures [18:12:52] PROBLEM - puppet last run on db2023 is CRITICAL: CRITICAL: Puppet has 11 failures [18:12:52] PROBLEM - puppet last run on db2001 is CRITICAL: CRITICAL: Puppet has 15 failures [18:12:53] PROBLEM - puppet last run on wtp1004 is CRITICAL: CRITICAL: puppet fail [18:12:54] PROBLEM - puppet last run on mw1188 is CRITICAL: CRITICAL: puppet fail [18:12:54] PROBLEM - puppet last run on mw1057 is CRITICAL: CRITICAL: Puppet has 37 failures [18:12:55] PROBLEM - puppet last run on acamar is CRITICAL: CRITICAL: puppet fail [18:12:55] PROBLEM - puppet last run on ms-be2007 is CRITICAL: CRITICAL: puppet fail [18:13:01] PROBLEM - puppet last run on mw1053 is CRITICAL: CRITICAL: Puppet has 36 failures [18:13:01] PROBLEM - puppet last run on amssq41 is CRITICAL: CRITICAL: Puppet has 26 failures [18:13:02] PROBLEM - puppet last run on mw1212 is CRITICAL: CRITICAL: Puppet has 66 failures [18:13:02] PROBLEM - puppet last run on db1062 is CRITICAL: CRITICAL: Puppet has 9 failures [18:13:02] PROBLEM - puppet last run on mw1050 is CRITICAL: CRITICAL: Puppet has 70 failures [18:13:02] PROBLEM - puppet last run on analytics1037 is CRITICAL: CRITICAL: puppet fail [18:13:11] PROBLEM - puppet last run on mw1198 is CRITICAL: CRITICAL: puppet fail [18:13:11] PROBLEM - puppet last run on db1020 is CRITICAL: CRITICAL: Puppet has 9 failures [18:13:11] PROBLEM - puppet last run on mw1087 is CRITICAL: CRITICAL: Puppet has 37 failures [18:13:11] PROBLEM - puppet last run on mw1116 is CRITICAL: CRITICAL: puppet fail [18:13:11] PROBLEM - puppet last run on db1054 is CRITICAL: CRITICAL: Puppet has 10 failures [18:13:12] PROBLEM - puppet last run on titanium is CRITICAL: CRITICAL: puppet fail [18:13:21] PROBLEM - puppet last run on mw1171 is CRITICAL: CRITICAL: puppet fail [18:13:21] PROBLEM - puppet last run on elastic1014 is CRITICAL: CRITICAL: puppet fail [18:13:21] PROBLEM - puppet last run on wtp1013 is CRITICAL: CRITICAL: Puppet has 21 failures [18:13:22] PROBLEM - puppet last run on analytics1032 is CRITICAL: CRITICAL: puppet fail [18:13:22] PROBLEM - puppet last run on mw1243 is CRITICAL: CRITICAL: puppet fail [18:13:31] PROBLEM - puppet last run on wtp1022 is CRITICAL: CRITICAL: Puppet has 33 failures [18:13:32] PROBLEM - puppet last run on analytics1014 is CRITICAL: CRITICAL: Puppet has 17 failures [18:13:32] PROBLEM - puppet last run on elastic1011 is CRITICAL: CRITICAL: puppet fail [18:13:33] PROBLEM - puppet last run on mw1163 is CRITICAL: CRITICAL: Puppet has 31 failures [18:13:33] PROBLEM - puppet last run on rcs1001 is CRITICAL: CRITICAL: Puppet has 22 failures [18:13:41] PROBLEM - puppet last run on elastic1005 is CRITICAL: CRITICAL: Puppet has 24 failures [18:13:41] PROBLEM - puppet last run on rcs1002 is CRITICAL: CRITICAL: puppet fail [18:13:42] PROBLEM - puppet last run on wtp1023 is CRITICAL: CRITICAL: puppet fail [18:13:42] PROBLEM - puppet last run on amslvs3 is CRITICAL: CRITICAL: puppet fail [18:13:51] PROBLEM - puppet last run on db1001 is CRITICAL: CRITICAL: Puppet has 25 failures [18:13:51] PROBLEM - puppet last run on mw1074 is CRITICAL: CRITICAL: Puppet has 73 failures [18:13:52] PROBLEM - puppet last run on mc1013 is CRITICAL: CRITICAL: Puppet has 13 failures [18:13:52] PROBLEM - puppet last run on es1002 is CRITICAL: CRITICAL: Puppet has 17 failures [18:14:01] PROBLEM - puppet last run on lvs2003 is CRITICAL: CRITICAL: Puppet has 23 failures [18:14:02] PROBLEM - puppet last run on db2003 is CRITICAL: CRITICAL: Puppet has 11 failures [18:14:02] PROBLEM - puppet last run on haedus is CRITICAL: CRITICAL: Puppet has 8 failures [18:14:02] PROBLEM - puppet last run on analytics1011 is CRITICAL: CRITICAL: puppet fail [18:14:02] PROBLEM - puppet last run on labsdb1004 is CRITICAL: CRITICAL: Puppet has 13 failures [18:14:02] PROBLEM - puppet last run on mw1032 is CRITICAL: CRITICAL: Puppet has 39 failures [18:14:11] PROBLEM - puppet last run on mw1097 is CRITICAL: CRITICAL: Puppet has 76 failures [18:14:12] PROBLEM - puppet last run on wtp1015 is CRITICAL: CRITICAL: puppet fail [18:14:12] PROBLEM - puppet last run on elastic1002 is CRITICAL: CRITICAL: Puppet has 11 failures [18:14:12] PROBLEM - puppet last run on amssq38 is CRITICAL: CRITICAL: puppet fail [18:14:12] PROBLEM - puppet last run on cp3009 is CRITICAL: CRITICAL: Puppet has 15 failures [18:14:12] PROBLEM - puppet last run on amssq62 is CRITICAL: CRITICAL: Puppet has 15 failures [18:14:21] PROBLEM - puppet last run on mw1148 is CRITICAL: CRITICAL: puppet fail [18:14:21] PROBLEM - puppet last run on mc1007 is CRITICAL: CRITICAL: puppet fail [18:14:21] PROBLEM - puppet last run on cp1060 is CRITICAL: CRITICAL: puppet fail [18:14:21] PROBLEM - puppet last run on mw1108 is CRITICAL: CRITICAL: puppet fail [18:14:22] PROBLEM - puppet last run on protactinium is CRITICAL: CRITICAL: Puppet has 22 failures [18:14:22] PROBLEM - puppet last run on rdb1002 is CRITICAL: CRITICAL: puppet fail [18:14:22] PROBLEM - puppet last run on mw1121 is CRITICAL: CRITICAL: puppet fail [18:14:31] PROBLEM - puppet last run on wtp1011 is CRITICAL: CRITICAL: puppet fail [18:14:32] PROBLEM - puppet last run on db1030 is CRITICAL: CRITICAL: puppet fail [18:14:32] PROBLEM - puppet last run on db1057 is CRITICAL: CRITICAL: Puppet has 12 failures [18:14:42] PROBLEM - puppet last run on mw1033 is CRITICAL: CRITICAL: Puppet has 28 failures [18:14:42] PROBLEM - puppet last run on db1063 is CRITICAL: CRITICAL: puppet fail [18:14:42] PROBLEM - puppet last run on mw1229 is CRITICAL: CRITICAL: puppet fail [18:14:42] PROBLEM - puppet last run on wtp1003 is CRITICAL: CRITICAL: puppet fail [18:14:42] PROBLEM - puppet last run on ocg1003 is CRITICAL: CRITICAL: Puppet has 12 failures [18:14:51] PROBLEM - puppet last run on nitrogen is CRITICAL: CRITICAL: Puppet has 11 failures [18:14:51] PROBLEM - puppet last run on mw1239 is CRITICAL: CRITICAL: Puppet has 36 failures [18:14:52] PROBLEM - puppet last run on mw1225 is CRITICAL: CRITICAL: puppet fail [18:14:52] PROBLEM - puppet last run on elastic1017 is CRITICAL: CRITICAL: puppet fail [18:15:02] PROBLEM - puppet last run on mercury is CRITICAL: CRITICAL: Puppet has 21 failures [18:15:02] PROBLEM - puppet last run on rdb1003 is CRITICAL: CRITICAL: puppet fail [18:15:02] PROBLEM - puppet last run on cp3004 is CRITICAL: CRITICAL: puppet fail [18:15:02] PROBLEM - puppet last run on mw1186 is CRITICAL: CRITICAL: Puppet has 39 failures [18:15:11] PROBLEM - puppet last run on calcium is CRITICAL: CRITICAL: Puppet has 19 failures [18:15:11] PROBLEM - puppet last run on db1070 is CRITICAL: CRITICAL: puppet fail [18:15:12] PROBLEM - puppet last run on pc1003 is CRITICAL: CRITICAL: Puppet has 20 failures [18:15:12] PROBLEM - puppet last run on db2011 is CRITICAL: CRITICAL: Puppet has 8 failures [18:15:12] PROBLEM - puppet last run on mw1201 is CRITICAL: CRITICAL: Puppet has 69 failures [18:15:12] PROBLEM - puppet last run on mw1185 is CRITICAL: CRITICAL: Puppet has 79 failures [18:15:21] PROBLEM - puppet last run on db1044 is CRITICAL: CRITICAL: Puppet has 11 failures [18:15:21] PROBLEM - puppet last run on mw1093 is CRITICAL: CRITICAL: puppet fail [18:15:21] PROBLEM - puppet last run on db1064 is CRITICAL: CRITICAL: Puppet has 8 failures [18:15:21] PROBLEM - puppet last run on db1006 is CRITICAL: CRITICAL: Puppet has 6 failures [18:15:22] PROBLEM - puppet last run on mw1043 is CRITICAL: CRITICAL: puppet fail [18:15:22] PROBLEM - puppet last run on cp4020 is CRITICAL: CRITICAL: puppet fail [18:15:22] PROBLEM - puppet last run on cp3012 is CRITICAL: CRITICAL: Puppet has 18 failures [18:15:22] PROBLEM - puppet last run on amssq31 is CRITICAL: CRITICAL: puppet fail [18:15:23] PROBLEM - puppet last run on ms-fe3001 is CRITICAL: CRITICAL: puppet fail [18:15:31] PROBLEM - puppet last run on virt1000 is CRITICAL: CRITICAL: Puppet has 47 failures [18:15:32] PROBLEM - puppet last run on mw1152 is CRITICAL: CRITICAL: Puppet has 40 failures [18:15:32] PROBLEM - puppet last run on mw1223 is CRITICAL: CRITICAL: puppet fail [18:15:32] PROBLEM - puppet last run on mw1142 is CRITICAL: CRITICAL: puppet fail [18:15:32] PROBLEM - puppet last run on dbstore1002 is CRITICAL: CRITICAL: Puppet has 7 failures [18:15:32] PROBLEM - puppet last run on mw1010 is CRITICAL: CRITICAL: puppet fail [18:15:32] PROBLEM - puppet last run on db1061 is CRITICAL: CRITICAL: puppet fail [18:15:32] PROBLEM - puppet last run on wtp1007 is CRITICAL: CRITICAL: puppet fail [18:15:33] PROBLEM - puppet last run on mw1001 is CRITICAL: CRITICAL: Puppet has 27 failures [18:15:41] PROBLEM - puppet last run on mw1022 is CRITICAL: CRITICAL: Puppet has 29 failures [18:15:41] PROBLEM - puppet last run on cp1052 is CRITICAL: CRITICAL: puppet fail [18:15:41] PROBLEM - puppet last run on db1047 is CRITICAL: CRITICAL: Puppet has 8 failures [18:15:41] PROBLEM - puppet last run on analytics1004 is CRITICAL: CRITICAL: puppet fail [18:15:41] PROBLEM - puppet last run on iridium is CRITICAL: CRITICAL: puppet fail [18:15:42] PROBLEM - puppet last run on mw1167 is CRITICAL: CRITICAL: Puppet has 76 failures [18:15:42] PROBLEM - puppet last run on mw1231 is CRITICAL: CRITICAL: puppet fail [18:15:42] PROBLEM - puppet last run on elastic1023 is CRITICAL: CRITICAL: Puppet has 21 failures [18:15:43] PROBLEM - puppet last run on mw1219 is CRITICAL: CRITICAL: puppet fail [18:15:43] PROBLEM - puppet last run on d-i-test is CRITICAL: CRITICAL: puppet fail [18:15:51] PROBLEM - puppet last run on es1010 is CRITICAL: CRITICAL: puppet fail [18:15:51] PROBLEM - puppet last run on mw1077 is CRITICAL: CRITICAL: Puppet has 72 failures [18:15:52] PROBLEM - puppet last run on ms-be1011 is CRITICAL: CRITICAL: puppet fail [18:15:52] PROBLEM - puppet last run on iodine is CRITICAL: CRITICAL: puppet fail [18:15:52] PROBLEM - puppet last run on analytics1027 is CRITICAL: CRITICAL: puppet fail [18:15:52] PROBLEM - puppet last run on mw1122 is CRITICAL: CRITICAL: Puppet has 73 failures [18:16:01] PROBLEM - puppet last run on db1011 is CRITICAL: CRITICAL: Puppet has 9 failures [18:16:01] PROBLEM - puppet last run on mw1024 is CRITICAL: CRITICAL: Puppet has 37 failures [18:16:01] PROBLEM - puppet last run on db1038 is CRITICAL: CRITICAL: puppet fail [18:16:01] PROBLEM - puppet last run on ms-fe2002 is CRITICAL: CRITICAL: Puppet has 27 failures [18:16:11] PROBLEM - puppet last run on mw1091 is CRITICAL: CRITICAL: puppet fail [18:16:12] PROBLEM - puppet last run on analytics1028 is CRITICAL: CRITICAL: Puppet has 21 failures [18:16:12] PROBLEM - puppet last run on mw1105 is CRITICAL: CRITICAL: Puppet has 78 failures [18:16:21] PROBLEM - puppet last run on fluorine is CRITICAL: CRITICAL: puppet fail [18:16:21] PROBLEM - puppet last run on mw1107 is CRITICAL: CRITICAL: puppet fail [18:16:21] PROBLEM - puppet last run on mw1071 is CRITICAL: CRITICAL: puppet fail [18:16:22] PROBLEM - puppet last run on db2028 is CRITICAL: CRITICAL: Puppet has 9 failures [18:16:22] PROBLEM - puppet last run on baham is CRITICAL: CRITICAL: puppet fail [18:16:22] PROBLEM - puppet last run on db2012 is CRITICAL: CRITICAL: puppet fail [18:16:22] PROBLEM - puppet last run on achernar is CRITICAL: CRITICAL: Puppet has 9 failures [18:16:31] PROBLEM - puppet last run on cp1037 is CRITICAL: CRITICAL: Puppet has 27 failures [18:16:31] PROBLEM - puppet last run on mw1139 is CRITICAL: CRITICAL: Puppet has 34 failures [18:16:31] PROBLEM - puppet last run on mw1209 is CRITICAL: CRITICAL: puppet fail [18:16:31] PROBLEM - puppet last run on mw1203 is CRITICAL: CRITICAL: puppet fail [18:16:31] PROBLEM - puppet last run on einsteinium is CRITICAL: CRITICAL: Puppet has 18 failures [18:16:32] PROBLEM - puppet last run on cp4002 is CRITICAL: CRITICAL: puppet fail [18:16:32] PROBLEM - puppet last run on db1033 is CRITICAL: CRITICAL: Puppet has 26 failures [18:16:41] PROBLEM - puppet last run on mw1241 is CRITICAL: CRITICAL: puppet fail [18:16:41] PROBLEM - puppet last run on mw1193 is CRITICAL: CRITICAL: puppet fail [18:16:42] PROBLEM - puppet last run on analytics1003 is CRITICAL: CRITICAL: Puppet has 13 failures [18:16:42] PROBLEM - puppet last run on analytics1018 is CRITICAL: CRITICAL: puppet fail [18:16:42] PROBLEM - puppet last run on mw1016 is CRITICAL: CRITICAL: Puppet has 62 failures [18:16:42] PROBLEM - puppet last run on mw1086 is CRITICAL: CRITICAL: puppet fail [18:16:52] PROBLEM - puppet last run on mw1220 is CRITICAL: CRITICAL: Puppet has 41 failures [18:16:52] PROBLEM - puppet last run on cp1040 is CRITICAL: CRITICAL: puppet fail [18:16:52] PROBLEM - puppet last run on mw1215 is CRITICAL: CRITICAL: puppet fail [18:16:52] PROBLEM - puppet last run on mw1066 is CRITICAL: CRITICAL: puppet fail [18:16:52] PROBLEM - puppet last run on lvs3003 is CRITICAL: CRITICAL: Puppet has 12 failures [18:16:52] PROBLEM - puppet last run on virt1011 is CRITICAL: CRITICAL: Puppet has 21 failures [18:17:01] PROBLEM - puppet last run on strontium is CRITICAL: CRITICAL: Puppet has 32 failures [18:17:01] PROBLEM - puppet last run on terbium is CRITICAL: CRITICAL: Puppet has 88 failures [18:17:02] PROBLEM - puppet last run on rbf1001 is CRITICAL: CRITICAL: Puppet has 20 failures [18:17:02] PROBLEM - puppet last run on cp1044 is CRITICAL: CRITICAL: Puppet has 24 failures [18:17:02] PROBLEM - puppet last run on cp1067 is CRITICAL: CRITICAL: puppet fail [18:17:02] PROBLEM - puppet last run on copper is CRITICAL: CRITICAL: Puppet has 23 failures [18:17:12] PROBLEM - puppet last run on ms-be1015 is CRITICAL: CRITICAL: puppet fail [18:17:21] PROBLEM - puppet last run on db1035 is CRITICAL: CRITICAL: puppet fail [18:17:21] PROBLEM - puppet last run on ms-be1002 is CRITICAL: CRITICAL: puppet fail [18:17:22] PROBLEM - puppet last run on cp1053 is CRITICAL: CRITICAL: puppet fail [18:17:22] PROBLEM - puppet last run on mw1236 is CRITICAL: CRITICAL: puppet fail [18:17:22] PROBLEM - puppet last run on elastic1003 is CRITICAL: CRITICAL: Puppet has 16 failures [18:17:22] PROBLEM - puppet last run on lvs1003 is CRITICAL: CRITICAL: puppet fail [18:17:22] PROBLEM - puppet last run on wtp1008 is CRITICAL: CRITICAL: puppet fail [18:17:22] PROBLEM - puppet last run on mw1253 is CRITICAL: CRITICAL: puppet fail [18:17:31] PROBLEM - puppet last run on db1072 is CRITICAL: CRITICAL: Puppet has 25 failures [18:17:32] PROBLEM - puppet last run on elastic1020 is CRITICAL: CRITICAL: Puppet has 22 failures [18:17:32] PROBLEM - puppet last run on mw1204 is CRITICAL: CRITICAL: Puppet has 33 failures [18:17:41] PROBLEM - puppet last run on chromium is CRITICAL: CRITICAL: Puppet has 23 failures [18:17:41] PROBLEM - puppet last run on mw1112 is CRITICAL: CRITICAL: Puppet has 33 failures [18:17:51] PROBLEM - puppet last run on dysprosium is CRITICAL: CRITICAL: puppet fail [18:17:51] PROBLEM - puppet last run on amssq33 is CRITICAL: CRITICAL: Puppet has 30 failures [18:17:52] PROBLEM - puppet last run on mw1090 is CRITICAL: CRITICAL: Puppet has 30 failures [18:18:01] PROBLEM - puppet last run on cp1045 is CRITICAL: CRITICAL: puppet fail [18:18:01] PROBLEM - puppet last run on ms-fe1002 is CRITICAL: CRITICAL: Puppet has 25 failures [18:18:01] PROBLEM - puppet last run on virt1008 is CRITICAL: CRITICAL: Puppet has 20 failures [18:18:02] PROBLEM - puppet last run on osm-cp1001 is CRITICAL: CRITICAL: puppet fail [18:18:02] PROBLEM - puppet last run on mw1021 is CRITICAL: CRITICAL: Puppet has 81 failures [18:18:02] PROBLEM - puppet last run on bast1001 is CRITICAL: CRITICAL: Puppet has 88 failures [18:18:02] PROBLEM - puppet last run on mw1110 is CRITICAL: CRITICAL: puppet fail [18:18:11] PROBLEM - puppet last run on wtp1010 is CRITICAL: CRITICAL: Puppet has 13 failures [18:18:11] PROBLEM - puppet last run on lvs1004 is CRITICAL: CRITICAL: puppet fail [18:18:11] PROBLEM - puppet last run on db1037 is CRITICAL: CRITICAL: puppet fail [18:18:12] PROBLEM - puppet last run on mw1027 is CRITICAL: CRITICAL: Puppet has 36 failures [18:18:22] PROBLEM - puppet last run on cp1068 is CRITICAL: CRITICAL: puppet fail [18:18:22] PROBLEM - puppet last run on mw1143 is CRITICAL: CRITICAL: Puppet has 39 failures [18:18:22] PROBLEM - puppet last run on sca1001 is CRITICAL: CRITICAL: Puppet has 15 failures [18:18:22] PROBLEM - puppet last run on wtp1001 is CRITICAL: CRITICAL: puppet fail [18:18:23] PROBLEM - puppet last run on cp1054 is CRITICAL: CRITICAL: Puppet has 30 failures [18:18:31] PROBLEM - puppet last run on mw1154 is CRITICAL: CRITICAL: Puppet has 25 failures [18:18:31] PROBLEM - puppet last run on db1045 is CRITICAL: CRITICAL: Puppet has 10 failures [18:18:31] PROBLEM - puppet last run on cp4012 is CRITICAL: CRITICAL: Puppet has 22 failures [18:18:32] PROBLEM - puppet last run on amssq39 is CRITICAL: CRITICAL: Puppet has 26 failures [18:18:32] PROBLEM - puppet last run on mw1064 is CRITICAL: CRITICAL: Puppet has 33 failures [18:18:41] PROBLEM - puppet last run on db1056 is CRITICAL: CRITICAL: Puppet has 14 failures [18:18:41] PROBLEM - puppet last run on mw1158 is CRITICAL: CRITICAL: Puppet has 43 failures [18:18:42] PROBLEM - puppet last run on mw1135 is CRITICAL: CRITICAL: puppet fail [18:18:52] PROBLEM - puppet last run on es2010 is CRITICAL: CRITICAL: Puppet has 9 failures [18:19:01] PROBLEM - puppet last run on mw1255 is CRITICAL: CRITICAL: Puppet has 36 failures [18:19:02] PROBLEM - puppet last run on radium is CRITICAL: CRITICAL: Puppet has 17 failures [18:19:12] PROBLEM - puppet last run on mw1104 is CRITICAL: CRITICAL: Puppet has 29 failures [18:20:42] PROBLEM - puppetmaster https on palladium is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 8140: HTTP/1.1 500 Internal Server Error [18:21:52] RECOVERY - puppetmaster https on palladium is OK: HTTP OK: Status line output matched 400 - 335 bytes in 0.038 second response time [18:28:03] RECOVERY - puppet last run on nescio is OK: OK: Puppet is currently enabled, last run 20 seconds ago with 0 failures [18:43:01] PROBLEM - puppetmaster https on palladium is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 8140: HTTP/1.1 500 Internal Server Error [18:44:11] RECOVERY - puppetmaster https on palladium is OK: HTTP OK: Status line output matched 400 - 335 bytes in 0.027 second response time [18:49:12] RECOVERY - puppet last run on sodium is OK: OK: Puppet is currently enabled, last run 48 seconds ago with 0 failures [18:57:01] PROBLEM - puppetmaster https on palladium is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 8140: HTTP/1.1 500 Internal Server Error [18:59:31] RECOVERY - puppetmaster https on palladium is OK: HTTP OK: Status line output matched 400 - 335 bytes in 0.040 second response time [19:07:29] PROBLEM - puppet last run on ms1004 is CRITICAL: CRITICAL: puppet fail [19:09:31] PROBLEM - puppet last run on nescio is CRITICAL: CRITICAL: puppet fail [19:13:17] !log restarting puppetmasters [19:13:28] Logged the message, Master [19:14:21] RECOVERY - puppetmaster backend https on palladium is OK: HTTP OK: Status line output matched 400 - 335 bytes in 1.883 second response time [19:15:22] PROBLEM - Slow CirrusSearch query rate on fluorine is CRITICAL: CirrusSearch-slow.log_line_rate CRITICAL: 0.00333333333333 [19:15:32] RECOVERY - puppet last run on analytics1003 is OK: OK: Puppet is currently enabled, last run 11 seconds ago with 0 failures [19:15:41] RECOVERY - puppet last run on virt1008 is OK: OK: Puppet is currently enabled, last run 1 second ago with 0 failures [19:15:42] RECOVERY - puppet last run on analytics1004 is OK: OK: Puppet is currently enabled, last run 20 seconds ago with 0 failures [19:15:51] RECOVERY - puppet last run on wtp1010 is OK: OK: Puppet is currently enabled, last run 6 seconds ago with 0 failures [19:15:51] RECOVERY - puppet last run on virt1011 is OK: OK: Puppet is currently enabled, last run 4 seconds ago with 0 failures [19:15:52] RECOVERY - puppet last run on strontium is OK: OK: Puppet is currently enabled, last run 2 seconds ago with 0 failures [19:16:01] RECOVERY - puppet last run on rbf1001 is OK: OK: Puppet is currently enabled, last run 17 seconds ago with 0 failures [19:16:01] RECOVERY - puppet last run on cp1044 is OK: OK: Puppet is currently enabled, last run 14 seconds ago with 0 failures [19:16:01] RECOVERY - puppet last run on sca1001 is OK: OK: Puppet is currently enabled, last run 14 seconds ago with 0 failures [19:16:01] RECOVERY - puppet last run on copper is OK: OK: Puppet is currently enabled, last run 35 seconds ago with 0 failures [19:16:02] RECOVERY - puppet last run on ms-be1015 is OK: OK: Puppet is currently enabled, last run 3 seconds ago with 0 failures [19:16:02] RECOVERY - puppet last run on db1035 is OK: OK: Puppet is currently enabled, last run 20 seconds ago with 0 failures [19:16:02] RECOVERY - puppet last run on db1045 is OK: OK: Puppet is currently enabled, last run 20 seconds ago with 0 failures [19:16:11] RECOVERY - puppet last run on ms-be1002 is OK: OK: Puppet is currently enabled, last run 19 seconds ago with 0 failures [19:16:12] RECOVERY - puppet last run on ms-fe2002 is OK: OK: Puppet is currently enabled, last run 12 seconds ago with 0 failures [19:16:12] RECOVERY - puppet last run on mw1091 is OK: OK: Puppet is currently enabled, last run 1 second ago with 0 failures [19:16:12] RECOVERY - puppet last run on wtp1008 is OK: OK: Puppet is currently enabled, last run 15 seconds ago with 0 failures [19:16:12] RECOVERY - puppet last run on elastic1003 is OK: OK: Puppet is currently enabled, last run 29 seconds ago with 0 failures [19:16:21] RECOVERY - puppet last run on db1056 is OK: OK: Puppet is currently enabled, last run 20 seconds ago with 0 failures [19:16:21] RECOVERY - puppet last run on db1072 is OK: OK: Puppet is currently enabled, last run 55 seconds ago with 0 failures [19:16:22] RECOVERY - puppet last run on elastic1020 is OK: OK: Puppet is currently enabled, last run 38 seconds ago with 0 failures [19:16:22] RECOVERY - puppet last run on chromium is OK: OK: Puppet is currently enabled, last run 39 seconds ago with 0 failures [19:16:31] RECOVERY - puppet last run on einsteinium is OK: OK: Puppet is currently enabled, last run 49 seconds ago with 0 failures [19:16:32] RECOVERY - puppet last run on dysprosium is OK: OK: Puppet is currently enabled, last run 50 seconds ago with 0 failures [19:16:32] RECOVERY - puppet last run on baham is OK: OK: Puppet is currently enabled, last run 3 seconds ago with 0 failures [19:16:32] RECOVERY - puppet last run on achernar is OK: OK: Puppet is currently enabled, last run 32 seconds ago with 0 failures [19:16:32] RECOVERY - puppet last run on mw1241 is OK: OK: Puppet is currently enabled, last run 7 seconds ago with 0 failures [19:16:41] RECOVERY - puppet last run on es2010 is OK: OK: Puppet is currently enabled, last run 37 seconds ago with 0 failures [19:16:41] RECOVERY - puppet last run on mw1193 is OK: OK: Puppet is currently enabled, last run 12 seconds ago with 0 failures [19:16:41] RECOVERY - puppet last run on lvs4004 is OK: OK: Puppet is currently enabled, last run 11 seconds ago with 0 failures [19:16:42] RECOVERY - puppet last run on mw1090 is OK: OK: Puppet is currently enabled, last run 2 seconds ago with 0 failures [19:16:42] RECOVERY - puppet last run on radium is OK: OK: Puppet is currently enabled, last run 43 seconds ago with 0 failures [19:16:51] RECOVERY - puppet last run on analytics1018 is OK: OK: Puppet is currently enabled, last run 51 seconds ago with 0 failures [19:16:51] RECOVERY - puppet last run on es1004 is OK: OK: Puppet is currently enabled, last run 21 seconds ago with 0 failures [19:16:51] RECOVERY - puppet last run on mw1223 is OK: OK: Puppet is currently enabled, last run 48 seconds ago with 0 failures [19:16:51] RECOVERY - puppet last run on virt1005 is OK: OK: Puppet is currently enabled, last run 15 seconds ago with 0 failures [19:16:51] RECOVERY - puppet last run on ms-fe1002 is OK: OK: Puppet is currently enabled, last run 50 seconds ago with 0 failures [19:16:51] RECOVERY - puppet last run on cp1045 is OK: OK: Puppet is currently enabled, last run 50 seconds ago with 0 failures [19:16:52] RECOVERY - puppet last run on mw1086 is OK: OK: Puppet is currently enabled, last run 12 seconds ago with 0 failures [19:16:52] RECOVERY - puppet last run on mw1142 is OK: OK: Puppet is currently enabled, last run 13 seconds ago with 0 failures [19:16:53] RECOVERY - puppet last run on mw1016 is OK: OK: Puppet is currently enabled, last run 26 seconds ago with 0 failures [19:16:53] RECOVERY - puppet last run on osm-cp1001 is OK: OK: Puppet is currently enabled, last run 52 seconds ago with 0 failures [19:16:54] RECOVERY - puppet last run on eeden is OK: OK: Puppet is currently enabled, last run 7 seconds ago with 0 failures [19:17:01] RECOVERY - puppet last run on cp1040 is OK: OK: Puppet is currently enabled, last run 47 seconds ago with 0 failures [19:17:01] RECOVERY - puppet last run on mw1104 is OK: OK: Puppet is currently enabled, last run 0 seconds ago with 0 failures [19:17:02] RECOVERY - puppet last run on radon is OK: OK: Puppet is currently enabled, last run 20 seconds ago with 0 failures [19:17:02] RECOVERY - puppet last run on mw1220 is OK: OK: Puppet is currently enabled, last run 19 seconds ago with 0 failures [19:17:02] RECOVERY - puppet last run on db1027 is OK: OK: Puppet is currently enabled, last run 6 seconds ago with 0 failures [19:17:02] RECOVERY - puppet last run on lvs1004 is OK: OK: Puppet is currently enabled, last run 39 seconds ago with 0 failures [19:17:02] RECOVERY - puppet last run on mw1110 is OK: OK: Puppet is currently enabled, last run 12 seconds ago with 0 failures [19:17:02] RECOVERY - puppet last run on mw1215 is OK: OK: Puppet is currently enabled, last run 30 seconds ago with 0 failures [19:17:03] RECOVERY - puppet last run on mw1066 is OK: OK: Puppet is currently enabled, last run 22 seconds ago with 0 failures [19:17:03] RECOVERY - puppet last run on db1037 is OK: OK: Puppet is currently enabled, last run 57 seconds ago with 0 failures [19:17:04] RECOVERY - puppet last run on terbium is OK: OK: Puppet is currently enabled, last run 46 seconds ago with 0 failures [19:17:05] RECOVERY - puppet last run on mw1027 is OK: OK: Puppet is currently enabled, last run 5 seconds ago with 0 failures [19:17:05] RECOVERY - puppet last run on elastic1029 is OK: OK: Puppet is currently enabled, last run 10 seconds ago with 0 failures [19:17:05] RECOVERY - puppet last run on analytics1012 is OK: OK: Puppet is currently enabled, last run 11 seconds ago with 0 failures [19:17:06] RECOVERY - puppet last run on cp1068 is OK: OK: Puppet is currently enabled, last run 14 seconds ago with 0 failures [19:17:07] RECOVERY - puppet last run on lvs3003 is OK: OK: Puppet is currently enabled, last run 56 seconds ago with 0 failures [19:17:11] RECOVERY - puppet last run on mw1143 is OK: OK: Puppet is currently enabled, last run 31 seconds ago with 0 failures [19:17:11] RECOVERY - puppet last run on cp1067 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [19:17:11] RECOVERY - puppet last run on wtp1001 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [19:17:12] RECOVERY - puppet last run on mw1037 is OK: OK: Puppet is currently enabled, last run 5 seconds ago with 0 failures [19:17:12] RECOVERY - puppet last run on cp1054 is OK: OK: Puppet is currently enabled, last run 28 seconds ago with 0 failures [19:17:12] RECOVERY - puppet last run on labsdb1002 is OK: OK: Puppet is currently enabled, last run 16 seconds ago with 0 failures [19:17:12] RECOVERY - puppet last run on mw1154 is OK: OK: Puppet is currently enabled, last run 46 seconds ago with 0 failures [19:17:12] RECOVERY - puppet last run on db1005 is OK: OK: Puppet is currently enabled, last run 16 seconds ago with 0 failures [19:17:21] RECOVERY - puppet last run on analytics1021 is OK: OK: Puppet is currently enabled, last run 32 seconds ago with 0 failures [19:17:21] RECOVERY - puppet last run on cp1053 is OK: OK: Puppet is currently enabled, last run 25 seconds ago with 0 failures [19:17:22] RECOVERY - puppet last run on mw1155 is OK: OK: Puppet is currently enabled, last run 35 seconds ago with 0 failures [19:17:22] RECOVERY - puppet last run on labsdb1005 is OK: OK: Puppet is currently enabled, last run 36 seconds ago with 0 failures [19:17:22] RECOVERY - puppet last run on mw1236 is OK: OK: Puppet is currently enabled, last run 43 seconds ago with 0 failures [19:17:22] RECOVERY - puppet last run on lvs1003 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [19:17:22] RECOVERY - puppet last run on mw1064 is OK: OK: Puppet is currently enabled, last run 43 seconds ago with 0 failures [19:17:31] RECOVERY - puppet last run on ms-be1010 is OK: OK: Puppet is currently enabled, last run 20 seconds ago with 0 failures [19:17:31] RECOVERY - puppet last run on mw1253 is OK: OK: Puppet is currently enabled, last run 21 seconds ago with 0 failures [19:17:31] RECOVERY - puppet last run on mw1158 is OK: OK: Puppet is currently enabled, last run 42 seconds ago with 0 failures [19:17:32] RECOVERY - puppet last run on mw1207 is OK: OK: Puppet is currently enabled, last run 18 seconds ago with 0 failures [19:17:32] RECOVERY - puppet last run on bast2001 is OK: OK: Puppet is currently enabled, last run 39 seconds ago with 0 failures [19:17:33] RECOVERY - puppet last run on cp4009 is OK: OK: Puppet is currently enabled, last run 10 seconds ago with 0 failures [19:17:33] RECOVERY - puppet last run on cp4012 is OK: OK: Puppet is currently enabled, last run 11 seconds ago with 0 failures [19:17:33] RECOVERY - puppet last run on mw1103 is OK: OK: Puppet is currently enabled, last run 7 seconds ago with 0 failures [19:17:33] RECOVERY - puppet last run on mw1107 is OK: OK: Puppet is currently enabled, last run 34 seconds ago with 0 failures [19:17:33] RECOVERY - puppet last run on mw1135 is OK: OK: Puppet is currently enabled, last run 34 seconds ago with 0 failures [19:17:34] RECOVERY - puppet last run on mw1113 is OK: OK: Puppet is currently enabled, last run 41 seconds ago with 0 failures [19:17:34] RECOVERY - puppet last run on mw1071 is OK: OK: Puppet is currently enabled, last run 54 seconds ago with 0 failures [19:17:41] RECOVERY - puppet last run on mw1204 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [19:17:41] RECOVERY - puppet last run on amssq39 is OK: OK: Puppet is currently enabled, last run 20 seconds ago with 0 failures [19:17:41] RECOVERY - puppet last run on cp3011 is OK: OK: Puppet is currently enabled, last run 3 seconds ago with 0 failures [19:17:41] RECOVERY - puppet last run on mw1203 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [19:17:41] RECOVERY - puppet last run on cp3018 is OK: OK: Puppet is currently enabled, last run 28 seconds ago with 0 failures [19:17:42] RECOVERY - puppet last run on cp3005 is OK: OK: Puppet is currently enabled, last run 4 seconds ago with 0 failures [19:17:42] RECOVERY - puppet last run on lvs2002 is OK: OK: Puppet is currently enabled, last run 20 seconds ago with 0 failures [19:17:42] RECOVERY - puppet last run on mw1112 is OK: OK: Puppet is currently enabled, last run 58 seconds ago with 0 failures [19:17:43] RECOVERY - puppet last run on es2006 is OK: OK: Puppet is currently enabled, last run 58 seconds ago with 0 failures [19:17:51] RECOVERY - puppet last run on db2010 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [19:17:51] RECOVERY - puppet last run on db1009 is OK: OK: Puppet is currently enabled, last run 28 seconds ago with 0 failures [19:17:51] RECOVERY - puppet last run on mw1255 is OK: OK: Puppet is currently enabled, last run 57 seconds ago with 0 failures [19:17:51] RECOVERY - puppet last run on elastic1009 is OK: OK: Puppet is currently enabled, last run 23 seconds ago with 0 failures [19:17:51] RECOVERY - puppet last run on db2017 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [19:17:52] RECOVERY - puppet last run on ms-be2009 is OK: OK: Puppet is currently enabled, last run 11 seconds ago with 0 failures [19:17:52] RECOVERY - puppet last run on mc1015 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [19:17:52] RECOVERY - puppet last run on palladium is OK: OK: Puppet is currently enabled, last run 45 seconds ago with 0 failures [19:17:53] RECOVERY - puppet last run on mw1102 is OK: OK: Puppet is currently enabled, last run 10 seconds ago with 0 failures [19:18:01] RECOVERY - puppet last run on elastic1025 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [19:18:01] RECOVERY - puppet last run on db1058 is OK: OK: Puppet is currently enabled, last run 0 seconds ago with 0 failures [19:18:01] RECOVERY - puppet last run on tmh1001 is OK: OK: Puppet is currently enabled, last run 25 seconds ago with 0 failures [19:18:02] RECOVERY - puppet last run on cp4002 is OK: OK: Puppet is currently enabled, last run 47 seconds ago with 0 failures [19:18:02] RECOVERY - puppet last run on mw1020 is OK: OK: Puppet is currently enabled, last run 3 seconds ago with 0 failures [19:18:02] RECOVERY - puppet last run on ms-be1014 is OK: OK: Puppet is currently enabled, last run 49 seconds ago with 0 failures [19:18:02] RECOVERY - puppet last run on cp1066 is OK: OK: Puppet is currently enabled, last run 51 seconds ago with 0 failures [19:18:02] RECOVERY - puppet last run on analytics1031 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [19:18:11] RECOVERY - puppet last run on cp3021 is OK: OK: Puppet is currently enabled, last run 18 seconds ago with 0 failures [19:18:11] RECOVERY - puppet last run on lvs3002 is OK: OK: Puppet is currently enabled, last run 37 seconds ago with 0 failures [19:18:11] RECOVERY - puppet last run on amssq37 is OK: OK: Puppet is currently enabled, last run 28 seconds ago with 0 failures [19:18:11] RECOVERY - puppet last run on cp3019 is OK: OK: Puppet is currently enabled, last run 20 seconds ago with 0 failures [19:18:11] RECOVERY - puppet last run on amssq33 is OK: OK: Puppet is currently enabled, last run 50 seconds ago with 0 failures [19:18:12] RECOVERY - puppet last run on virt1009 is OK: OK: Puppet is currently enabled, last run 32 seconds ago with 0 failures [19:18:12] RECOVERY - puppet last run on mw1137 is OK: OK: Puppet is currently enabled, last run 29 seconds ago with 0 failures [19:18:12] RECOVERY - puppet last run on bast1001 is OK: OK: Puppet is currently enabled, last run 22 seconds ago with 0 failures [19:18:12] RECOVERY - puppet last run on mw1021 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [19:18:13] RECOVERY - puppet last run on analytics1024 is OK: OK: Puppet is currently enabled, last run 23 seconds ago with 0 failures [19:18:14] RECOVERY - puppet last run on mw1179 is OK: OK: Puppet is currently enabled, last run 2 seconds ago with 0 failures [19:18:14] RECOVERY - puppet last run on lvs1006 is OK: OK: Puppet is currently enabled, last run 6 seconds ago with 0 failures [19:18:15] RECOVERY - puppet last run on mw1058 is OK: OK: Puppet is currently enabled, last run 23 seconds ago with 0 failures [19:18:21] RECOVERY - puppet last run on es1003 is OK: OK: Puppet is currently enabled, last run 3 seconds ago with 0 failures [19:18:21] RECOVERY - puppet last run on mw1073 is OK: OK: Puppet is currently enabled, last run 44 seconds ago with 0 failures [19:18:21] RECOVERY - puppet last run on mw1157 is OK: OK: Puppet is currently enabled, last run 36 seconds ago with 0 failures [19:18:22] RECOVERY - puppet last run on mw1131 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [19:18:22] RECOVERY - puppet last run on hydrogen is OK: OK: Puppet is currently enabled, last run 42 seconds ago with 0 failures [19:18:22] RECOVERY - puppet last run on db1049 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [19:18:22] RECOVERY - puppet last run on labsdb1007 is OK: OK: Puppet is currently enabled, last run 38 seconds ago with 0 failures [19:18:22] RECOVERY - puppet last run on elastic1010 is OK: OK: Puppet is currently enabled, last run 56 seconds ago with 0 failures [19:18:23] RECOVERY - puppet last run on praseodymium is OK: OK: Puppet is currently enabled, last run 17 seconds ago with 0 failures [19:18:23] RECOVERY - puppet last run on mw1199 is OK: OK: Puppet is currently enabled, last run 37 seconds ago with 0 failures [19:18:24] RECOVERY - puppet last run on es1005 is OK: OK: Puppet is currently enabled, last run 4 seconds ago with 0 failures [19:18:24] RECOVERY - puppet last run on ocg1002 is OK: OK: Puppet is currently enabled, last run 11 seconds ago with 0 failures [19:18:31] RECOVERY - puppet last run on mw1128 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [19:18:31] RECOVERY - puppet last run on mw1017 is OK: OK: Puppet is currently enabled, last run 8 seconds ago with 0 failures [19:18:31] RECOVERY - puppet last run on db1068 is OK: OK: Puppet is currently enabled, last run 59 seconds ago with 0 failures [19:18:32] RECOVERY - puppet last run on mw1015 is OK: OK: Puppet is currently enabled, last run 29 seconds ago with 0 failures [19:18:32] RECOVERY - puppet last run on lvs2005 is OK: OK: Puppet is currently enabled, last run 41 seconds ago with 0 failures [19:18:32] RECOVERY - puppet last run on lanthanum is OK: OK: Puppet is currently enabled, last run 23 seconds ago with 0 failures [19:18:32] RECOVERY - puppet last run on pollux is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [19:18:41] RECOVERY - puppet last run on mw1018 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [19:18:41] RECOVERY - puppet last run on mw1184 is OK: OK: Puppet is currently enabled, last run 23 seconds ago with 0 failures [19:18:41] RECOVERY - puppet last run on elastic1013 is OK: OK: Puppet is currently enabled, last run 59 seconds ago with 0 failures [19:18:41] RECOVERY - puppet last run on mw1194 is OK: OK: Puppet is currently enabled, last run 56 seconds ago with 0 failures [19:18:42] RECOVERY - puppet last run on mw1047 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [19:18:42] RECOVERY - puppet last run on mw1230 is OK: OK: Puppet is currently enabled, last run 50 seconds ago with 0 failures [19:18:42] RECOVERY - puppet last run on lvs4001 is OK: OK: Puppet is currently enabled, last run 29 seconds ago with 0 failures [19:18:42] RECOVERY - puppet last run on cp4010 is OK: OK: Puppet is currently enabled, last run 35 seconds ago with 0 failures [19:18:42] RECOVERY - puppet last run on cp4016 is OK: OK: Puppet is currently enabled, last run 26 seconds ago with 0 failures [19:18:43] RECOVERY - puppet last run on cp4011 is OK: OK: Puppet is currently enabled, last run 13 seconds ago with 0 failures [19:18:51] RECOVERY - puppet last run on tmh1002 is OK: OK: Puppet is currently enabled, last run 21 seconds ago with 0 failures [19:18:51] RECOVERY - puppet last run on ms-be1005 is OK: OK: Puppet is currently enabled, last run 50 seconds ago with 0 failures [19:18:51] RECOVERY - puppet last run on ms-be3004 is OK: OK: Puppet is currently enabled, last run 50 seconds ago with 0 failures [19:18:52] RECOVERY - puppet last run on amssq50 is OK: OK: Puppet is currently enabled, last run 28 seconds ago with 0 failures [19:18:52] RECOVERY - puppet last run on db1019 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [19:18:52] RECOVERY - puppet last run on cp3017 is OK: OK: Puppet is currently enabled, last run 56 seconds ago with 0 failures [19:18:52] RECOVERY - puppet last run on cp3007 is OK: OK: Puppet is currently enabled, last run 19 seconds ago with 0 failures [19:19:01] RECOVERY - puppet last run on mw1078 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [19:19:01] RECOVERY - puppet last run on mw1101 is OK: OK: Puppet is currently enabled, last run 48 seconds ago with 0 failures [19:19:01] RECOVERY - puppet last run on ms-be2015 is OK: OK: Puppet is currently enabled, last run 41 seconds ago with 0 failures [19:19:02] RECOVERY - puppet last run on mw1136 is OK: OK: Puppet is currently enabled, last run 22 seconds ago with 0 failures [19:19:02] RECOVERY - puppet last run on mw1085 is OK: OK: Puppet is currently enabled, last run 54 seconds ago with 0 failures [19:19:02] RECOVERY - puppet last run on mw1127 is OK: OK: Puppet is currently enabled, last run 23 seconds ago with 0 failures [19:19:11] RECOVERY - puppet last run on ms-be2010 is OK: OK: Puppet is currently enabled, last run 7 seconds ago with 0 failures [19:19:12] RECOVERY - puppet last run on cp1069 is OK: OK: Puppet is currently enabled, last run 0 seconds ago with 0 failures [19:19:12] RECOVERY - puppet last run on mw1095 is OK: OK: Puppet is currently enabled, last run 51 seconds ago with 0 failures [19:19:12] RECOVERY - puppet last run on cp1064 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [19:19:22] RECOVERY - puppet last run on wtp1014 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [19:19:22] RECOVERY - puppet last run on amssq45 is OK: OK: Puppet is currently enabled, last run 41 seconds ago with 0 failures [19:19:22] RECOVERY - puppet last run on mw1019 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [19:19:22] RECOVERY - puppet last run on stat1001 is OK: OK: Puppet is currently enabled, last run 23 seconds ago with 0 failures [19:19:22] RECOVERY - puppet last run on amssq52 is OK: OK: Puppet is currently enabled, last run 15 seconds ago with 0 failures [19:19:32] RECOVERY - puppet last run on mw1094 is OK: OK: Puppet is currently enabled, last run 36 seconds ago with 0 failures [19:19:32] RECOVERY - puppet last run on mw1191 is OK: OK: Puppet is currently enabled, last run 42 seconds ago with 0 failures [19:19:32] RECOVERY - puppet last run on mw1083 is OK: OK: Puppet is currently enabled, last run 44 seconds ago with 0 failures [19:19:32] RECOVERY - puppet last run on mw1075 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [19:19:33] RECOVERY - puppet last run on mw1070 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [19:19:33] RECOVERY - puppet last run on analytics1029 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [19:19:33] RECOVERY - puppet last run on cp1051 is OK: OK: Puppet is currently enabled, last run 5 seconds ago with 0 failures [19:19:33] RECOVERY - puppet last run on erbium is OK: OK: Puppet is currently enabled, last run 50 seconds ago with 0 failures [19:19:41] RECOVERY - puppet last run on mw1232 is OK: OK: Puppet is currently enabled, last run 13 seconds ago with 0 failures [19:19:41] RECOVERY - puppet last run on labsdb1001 is OK: OK: Puppet is currently enabled, last run 5 seconds ago with 0 failures [19:19:41] RECOVERY - puppet last run on analytics1019 is OK: OK: Puppet is currently enabled, last run 44 seconds ago with 0 failures [19:19:42] RECOVERY - puppet last run on zirconium is OK: OK: Puppet is currently enabled, last run 40 seconds ago with 0 failures [19:19:42] RECOVERY - puppet last run on mw1182 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [19:19:51] RECOVERY - puppet last run on cp1057 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [19:19:51] RECOVERY - puppet last run on mw1196 is OK: OK: Puppet is currently enabled, last run 25 seconds ago with 0 failures [19:19:51] RECOVERY - puppet last run on wtp1021 is OK: OK: Puppet is currently enabled, last run 11 seconds ago with 0 failures [19:19:52] RECOVERY - puppet last run on virt1012 is OK: OK: Puppet is currently enabled, last run 3 seconds ago with 0 failures [19:19:52] RECOVERY - puppet last run on elastic1016 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [19:19:52] RECOVERY - puppet last run on db2041 is OK: OK: Puppet is currently enabled, last run 19 seconds ago with 0 failures [19:19:52] RECOVERY - puppet last run on db1041 is OK: OK: Puppet is currently enabled, last run 46 seconds ago with 0 failures [19:20:02] RECOVERY - puppet last run on db1024 is OK: OK: Puppet is currently enabled, last run 54 seconds ago with 0 failures [19:20:02] RECOVERY - puppet last run on mw1169 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [19:20:02] RECOVERY - puppet last run on mw1214 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [19:20:02] RECOVERY - puppet last run on mw1013 is OK: OK: Puppet is currently enabled, last run 8 seconds ago with 0 failures [19:20:02] RECOVERY - puppet last run on mc1008 is OK: OK: Puppet is currently enabled, last run 59 seconds ago with 0 failures [19:20:03] RECOVERY - puppet last run on cp4015 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [19:20:03] RECOVERY - puppet last run on cp4017 is OK: OK: Puppet is currently enabled, last run 43 seconds ago with 0 failures [19:20:11] RECOVERY - puppet last run on mw1234 is OK: OK: Puppet is currently enabled, last run 46 seconds ago with 0 failures [19:20:12] RECOVERY - puppet last run on amslvs4 is OK: OK: Puppet is currently enabled, last run 51 seconds ago with 0 failures [19:20:12] RECOVERY - puppet last run on analytics1034 is OK: OK: Puppet is currently enabled, last run 53 seconds ago with 0 failures [19:20:12] RECOVERY - puppet last run on mw1244 is OK: OK: Puppet is currently enabled, last run 12 seconds ago with 0 failures [19:20:12] RECOVERY - puppet last run on wtp1024 is OK: OK: Puppet is currently enabled, last run 20 seconds ago with 0 failures [19:20:27] RECOVERY - puppet last run on db2035 is OK: OK: Puppet is currently enabled, last run 22 seconds ago with 0 failures [19:20:28] RECOVERY - puppet last run on mw1096 is OK: OK: Puppet is currently enabled, last run 53 seconds ago with 0 failures [19:20:28] RECOVERY - puppet last run on mw1035 is OK: OK: Puppet is currently enabled, last run 53 seconds ago with 0 failures [19:20:28] RECOVERY - puppet last run on es1009 is OK: OK: Puppet is currently enabled, last run 2 seconds ago with 0 failures [19:20:28] RECOVERY - puppet last run on mw1245 is OK: OK: Puppet is currently enabled, last run 14 seconds ago with 0 failures [19:20:28] RECOVERY - puppet last run on mw1221 is OK: OK: Puppet is currently enabled, last run 8 seconds ago with 0 failures [19:20:28] RECOVERY - puppet last run on cp1065 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [19:20:28] RECOVERY - puppet last run on analytics1039 is OK: OK: Puppet is currently enabled, last run 23 seconds ago with 0 failures [19:20:28] RECOVERY - puppet last run on elastic1028 is OK: OK: Puppet is currently enabled, last run 11 seconds ago with 0 failures [19:20:31] RECOVERY - puppet last run on analytics1036 is OK: OK: Puppet is currently enabled, last run 38 seconds ago with 0 failures [19:20:31] RECOVERY - puppet last run on cp4013 is OK: OK: Puppet is currently enabled, last run 43 seconds ago with 0 failures [19:20:32] RECOVERY - puppet last run on mw1036 is OK: OK: Puppet is currently enabled, last run 54 seconds ago with 0 failures [19:20:32] RECOVERY - puppet last run on ytterbium is OK: OK: Puppet is currently enabled, last run 35 seconds ago with 0 failures [19:20:32] RECOVERY - puppet last run on mw1138 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [19:20:41] RECOVERY - puppet last run on virt1002 is OK: OK: Puppet is currently enabled, last run 11 seconds ago with 0 failures [19:20:41] RECOVERY - puppet last run on mw1252 is OK: OK: Puppet is currently enabled, last run 21 seconds ago with 0 failures [19:20:42] RECOVERY - puppet last run on amssq57 is OK: OK: Puppet is currently enabled, last run 13 seconds ago with 0 failures [19:20:42] RECOVERY - puppet last run on amssq58 is OK: OK: Puppet is currently enabled, last run 16 seconds ago with 0 failures [19:20:42] RECOVERY - Slow CirrusSearch query rate on fluorine is OK: CirrusSearch-slow.log_line_rate OKAY: 0.0 [19:20:42] RECOVERY - puppet last run on pc1001 is OK: OK: Puppet is currently enabled, last run 4 seconds ago with 0 failures [19:20:51] RECOVERY - puppet last run on mw1246 is OK: OK: Puppet is currently enabled, last run 48 seconds ago with 0 failures [19:21:01] RECOVERY - puppet last run on cp1043 is OK: OK: Puppet is currently enabled, last run 42 seconds ago with 0 failures [19:21:01] RECOVERY - puppet last run on elastic1026 is OK: OK: Puppet is currently enabled, last run 42 seconds ago with 0 failures [19:21:02] RECOVERY - puppet last run on mc1010 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [19:21:02] RECOVERY - puppet last run on ocg1001 is OK: OK: Puppet is currently enabled, last run 15 seconds ago with 0 failures [19:21:11] RECOVERY - puppet last run on es1006 is OK: OK: Puppet is currently enabled, last run 36 seconds ago with 0 failures [19:21:11] RECOVERY - puppet last run on analytics1015 is OK: OK: Puppet is currently enabled, last run 38 seconds ago with 0 failures [19:21:12] RECOVERY - puppet last run on mw1161 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [19:21:12] RECOVERY - puppet last run on db1007 is OK: OK: Puppet is currently enabled, last run 55 seconds ago with 0 failures [19:21:12] RECOVERY - puppet last run on mw1218 is OK: OK: Puppet is currently enabled, last run 30 seconds ago with 0 failures [19:21:21] RECOVERY - puppet last run on rdb1004 is OK: OK: Puppet is currently enabled, last run 14 seconds ago with 0 failures [19:21:21] RECOVERY - puppet last run on db1010 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [19:21:22] RECOVERY - puppet last run on dbstore1001 is OK: OK: Puppet is currently enabled, last run 40 seconds ago with 0 failures [19:21:22] RECOVERY - puppet last run on wtp1019 is OK: OK: Puppet is currently enabled, last run 40 seconds ago with 0 failures [19:21:22] RECOVERY - puppet last run on mw1109 is OK: OK: Puppet is currently enabled, last run 34 seconds ago with 0 failures [19:21:22] RECOVERY - puppet last run on mw1124 is OK: OK: Puppet is currently enabled, last run 10 seconds ago with 0 failures [19:21:31] RECOVERY - puppet last run on ms-be2013 is OK: OK: Puppet is currently enabled, last run 8 seconds ago with 0 failures [19:21:31] RECOVERY - puppet last run on mw1147 is OK: OK: Puppet is currently enabled, last run 34 seconds ago with 0 failures [19:21:31] RECOVERY - puppet last run on mc1011 is OK: OK: Puppet is currently enabled, last run 55 seconds ago with 0 failures [19:21:31] RECOVERY - puppet last run on mw1130 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [19:21:32] RECOVERY - puppet last run on mw1240 is OK: OK: Puppet is currently enabled, last run 35 seconds ago with 0 failures [19:21:32] RECOVERY - puppet last run on elastic1031 is OK: OK: Puppet is currently enabled, last run 53 seconds ago with 0 failures [19:21:41] RECOVERY - puppet last run on vanadium is OK: OK: Puppet is currently enabled, last run 35 seconds ago with 0 failures [19:21:42] RECOVERY - puppet last run on mw1067 is OK: OK: Puppet is currently enabled, last run 20 seconds ago with 0 failures [19:21:42] RECOVERY - puppet last run on mw1072 is OK: OK: Puppet is currently enabled, last run 27 seconds ago with 0 failures [19:21:42] RECOVERY - puppet last run on cp4007 is OK: OK: Puppet is currently enabled, last run 29 seconds ago with 0 failures [19:21:42] RECOVERY - puppet last run on cp1059 is OK: OK: Puppet is currently enabled, last run 48 seconds ago with 0 failures [19:21:51] RECOVERY - puppet last run on mw1216 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [19:21:51] RECOVERY - puppet last run on mc1004 is OK: OK: Puppet is currently enabled, last run 53 seconds ago with 0 failures [19:21:51] RECOVERY - puppet last run on cp3013 is OK: OK: Puppet is currently enabled, last run 30 seconds ago with 0 failures [19:21:52] RECOVERY - puppet last run on mw1040 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [19:21:52] RECOVERY - puppet last run on logstash1003 is OK: OK: Puppet is currently enabled, last run 58 seconds ago with 0 failures [19:21:52] RECOVERY - puppet last run on ms-fe1003 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [19:22:01] RECOVERY - puppet last run on mc1009 is OK: OK: Puppet is currently enabled, last run 34 seconds ago with 0 failures [19:22:02] RECOVERY - puppet last run on ms-be1013 is OK: OK: Puppet is currently enabled, last run 42 seconds ago with 0 failures [19:22:02] RECOVERY - puppet last run on mw1233 is OK: OK: Puppet is currently enabled, last run 2 seconds ago with 0 failures [19:22:11] RECOVERY - puppet last run on logstash1001 is OK: OK: Puppet is currently enabled, last run 6 seconds ago with 0 failures [19:22:11] RECOVERY - puppet last run on mw1115 is OK: OK: Puppet is currently enabled, last run 37 seconds ago with 0 failures [19:22:11] RECOVERY - puppet last run on mw1048 is OK: OK: Puppet is currently enabled, last run 54 seconds ago with 0 failures [19:22:11] RECOVERY - puppet last run on mw1257 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [19:22:12] RECOVERY - puppet last run on uranium is OK: OK: Puppet is currently enabled, last run 49 seconds ago with 0 failures [19:22:21] RECOVERY - puppet last run on ms-be1004 is OK: OK: Puppet is currently enabled, last run 4 seconds ago with 0 failures [19:22:21] RECOVERY - puppet last run on mw1028 is OK: OK: Puppet is currently enabled, last run 59 seconds ago with 0 failures [19:22:21] RECOVERY - puppet last run on mw1132 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [19:22:21] RECOVERY - puppet last run on db1065 is OK: OK: Puppet is currently enabled, last run 11 seconds ago with 0 failures [19:22:21] RECOVERY - puppet last run on wtp1017 is OK: OK: Puppet is currently enabled, last run 52 seconds ago with 0 failures [19:22:31] RECOVERY - puppet last run on ms-be2002 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [19:22:31] RECOVERY - puppet last run on mw1178 is OK: OK: Puppet is currently enabled, last run 11 seconds ago with 0 failures [19:22:31] RECOVERY - puppet last run on mw1256 is OK: OK: Puppet is currently enabled, last run 15 seconds ago with 0 failures [19:22:32] RECOVERY - puppet last run on mw1089 is OK: OK: Puppet is currently enabled, last run 1 second ago with 0 failures [19:22:32] RECOVERY - puppet last run on cp3022 is OK: OK: Puppet is currently enabled, last run 27 seconds ago with 0 failures [19:22:32] RECOVERY - puppet last run on mw1080 is OK: OK: Puppet is currently enabled, last run 16 seconds ago with 0 failures [19:22:32] RECOVERY - puppet last run on mw1200 is OK: OK: Puppet is currently enabled, last run 2 seconds ago with 0 failures [19:22:32] RECOVERY - puppet last run on analytics1033 is OK: OK: Puppet is currently enabled, last run 22 seconds ago with 0 failures [19:22:41] RECOVERY - puppet last run on ms-be1001 is OK: OK: Puppet is currently enabled, last run 33 seconds ago with 0 failures [19:22:41] RECOVERY - puppet last run on mw1038 is OK: OK: Puppet is currently enabled, last run 7 seconds ago with 0 failures [19:22:42] RECOVERY - puppet last run on es2005 is OK: OK: Puppet is currently enabled, last run 23 seconds ago with 0 failures [19:22:42] RECOVERY - puppet last run on mw1134 is OK: OK: Puppet is currently enabled, last run 24 seconds ago with 0 failures [19:22:42] RECOVERY - puppet last run on labstore2001 is OK: OK: Puppet is currently enabled, last run 3 seconds ago with 0 failures [19:22:42] RECOVERY - puppet last run on es2003 is OK: OK: Puppet is currently enabled, last run 40 seconds ago with 0 failures [19:22:55] RECOVERY - puppet last run on ms-be1003 is OK: OK: Puppet is currently enabled, last run 5 seconds ago with 0 failures [19:22:55] RECOVERY - puppet last run on es1001 is OK: OK: Puppet is currently enabled, last run 34 seconds ago with 0 failures [19:22:55] RECOVERY - puppet last run on netmon1001 is OK: OK: Puppet is currently enabled, last run 19 seconds ago with 0 failures [19:22:55] RECOVERY - puppet last run on analytics1041 is OK: OK: Puppet is currently enabled, last run 1 second ago with 0 failures [19:22:55] RECOVERY - puppet last run on magnesium is OK: OK: Puppet is currently enabled, last run 27 seconds ago with 0 failures [19:22:55] RECOVERY - puppet last run on mw1197 is OK: OK: Puppet is currently enabled, last run 19 seconds ago with 0 failures [19:23:01] RECOVERY - puppet last run on db1029 is OK: OK: Puppet is currently enabled, last run 48 seconds ago with 0 failures [19:23:01] RECOVERY - puppet last run on db1053 is OK: OK: Puppet is currently enabled, last run 32 seconds ago with 0 failures [19:23:01] RECOVERY - puppet last run on searchidx1001 is OK: OK: Puppet is currently enabled, last run 29 seconds ago with 0 failures [19:23:01] RECOVERY - puppet last run on dbproxy1002 is OK: OK: Puppet is currently enabled, last run 49 seconds ago with 0 failures [19:23:02] RECOVERY - puppet last run on cp1070 is OK: OK: Puppet is currently enabled, last run 11 seconds ago with 0 failures [19:23:11] RECOVERY - puppet last run on db1073 is OK: OK: Puppet is currently enabled, last run 27 seconds ago with 0 failures [19:23:11] RECOVERY - puppet last run on cerium is OK: OK: Puppet is currently enabled, last run 20 seconds ago with 0 failures [19:23:11] RECOVERY - puppet last run on elastic1001 is OK: OK: Puppet is currently enabled, last run 26 seconds ago with 0 failures [19:23:11] RECOVERY - puppet last run on cp1049 is OK: OK: Puppet is currently enabled, last run 42 seconds ago with 0 failures [19:23:11] RECOVERY - puppet last run on mc1016 is OK: OK: Puppet is currently enabled, last run 58 seconds ago with 0 failures [19:23:11] RECOVERY - puppet last run on platinum is OK: OK: Puppet is currently enabled, last run 15 seconds ago with 0 failures [19:23:11] RECOVERY - puppet last run on wtp1006 is OK: OK: Puppet is currently enabled, last run 10 seconds ago with 0 failures [19:23:13] RECOVERY - puppet last run on carbon is OK: OK: Puppet is currently enabled, last run 52 seconds ago with 0 failures [19:23:13] RECOVERY - puppet last run on mw1006 is OK: OK: Puppet is currently enabled, last run 2 seconds ago with 0 failures [19:23:21] RECOVERY - puppet last run on analytics1017 is OK: OK: Puppet is currently enabled, last run 39 seconds ago with 0 failures [19:23:22] RECOVERY - puppet last run on mc1006 is OK: OK: Puppet is currently enabled, last run 43 seconds ago with 0 failures [19:23:22] RECOVERY - puppet last run on mw1031 is OK: OK: Puppet is currently enabled, last run 53 seconds ago with 0 failures [19:23:22] RECOVERY - puppet last run on neptunium is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [19:23:22] RECOVERY - puppet last run on wtp1009 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [19:23:22] RECOVERY - puppet last run on potassium is OK: OK: Puppet is currently enabled, last run 7 seconds ago with 0 failures [19:23:31] RECOVERY - puppet last run on mw1005 is OK: OK: Puppet is currently enabled, last run 58 seconds ago with 0 failures [19:23:32] RECOVERY - puppet last run on ms-fe2004 is OK: OK: Puppet is currently enabled, last run 24 seconds ago with 0 failures [19:23:32] RECOVERY - puppet last run on mw1141 is OK: OK: Puppet is currently enabled, last run 24 seconds ago with 0 failures [19:23:32] RECOVERY - puppet last run on mw1063 is OK: OK: Puppet is currently enabled, last run 14 seconds ago with 0 failures [19:23:32] RECOVERY - puppet last run on db2034 is OK: OK: Puppet is currently enabled, last run 7 seconds ago with 0 failures [19:23:32] RECOVERY - puppet last run on db1031 is OK: OK: Puppet is currently enabled, last run 9 seconds ago with 0 failures [19:23:32] RECOVERY - puppet last run on db2039 is OK: OK: Puppet is currently enabled, last run 28 seconds ago with 0 failures [19:23:33] RECOVERY - puppet last run on mw1145 is OK: OK: Puppet is currently enabled, last run 34 seconds ago with 0 failures [19:23:33] RECOVERY - puppet last run on analytics1040 is OK: OK: Puppet is currently enabled, last run 40 seconds ago with 0 failures [19:23:34] RECOVERY - puppet last run on mw1140 is OK: OK: Puppet is currently enabled, last run 34 seconds ago with 0 failures [19:23:34] RECOVERY - puppet last run on mw1082 is OK: OK: Puppet is currently enabled, last run 9 seconds ago with 0 failures [19:23:34] RECOVERY - puppet last run on mw1007 is OK: OK: Puppet is currently enabled, last run 55 seconds ago with 0 failures [19:23:35] RECOVERY - puppet last run on mw1059 is OK: OK: Puppet is currently enabled, last run 16 seconds ago with 0 failures [19:23:41] RECOVERY - puppet last run on elastic1007 is OK: OK: Puppet is currently enabled, last run 29 seconds ago with 0 failures [19:23:41] RECOVERY - puppet last run on cp4006 is OK: OK: Puppet is currently enabled, last run 2 seconds ago with 0 failures [19:23:41] RECOVERY - puppet last run on snapshot1003 is OK: OK: Puppet is currently enabled, last run 36 seconds ago with 0 failures [19:23:42] RECOVERY - puppet last run on mw1012 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [19:23:42] RECOVERY - puppet last run on caesium is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [19:23:42] RECOVERY - puppet last run on mw1041 is OK: OK: Puppet is currently enabled, last run 36 seconds ago with 0 failures [19:23:42] RECOVERY - puppet last run on es1008 is OK: OK: Puppet is currently enabled, last run 24 seconds ago with 0 failures [19:23:51] RECOVERY - puppet last run on amslvs2 is OK: OK: Puppet is currently enabled, last run 59 seconds ago with 0 failures [19:23:51] RECOVERY - puppet last run on amssq49 is OK: OK: Puppet is currently enabled, last run 31 seconds ago with 0 failures [19:23:51] RECOVERY - puppet last run on amssq59 is OK: OK: Puppet is currently enabled, last run 46 seconds ago with 0 failures [19:23:51] RECOVERY - puppet last run on cp3006 is OK: OK: Puppet is currently enabled, last run 46 seconds ago with 0 failures [19:23:52] RECOVERY - puppet last run on wtp1020 is OK: OK: Puppet is currently enabled, last run 40 seconds ago with 0 failures [19:23:52] RECOVERY - puppet last run on ms1004 is OK: OK: Puppet is currently enabled, last run 2 seconds ago with 0 failures [19:23:52] RECOVERY - puppet last run on heze is OK: OK: Puppet is currently enabled, last run 26 seconds ago with 0 failures [19:23:52] RECOVERY - puppet last run on es2008 is OK: OK: Puppet is currently enabled, last run 14 seconds ago with 0 failures [19:23:52] RECOVERY - puppet last run on es2002 is OK: OK: Puppet is currently enabled, last run 26 seconds ago with 0 failures [19:23:53] RECOVERY - puppet last run on db2009 is OK: OK: Puppet is currently enabled, last run 14 seconds ago with 0 failures [19:23:53] RECOVERY - puppet last run on db2005 is OK: OK: Puppet is currently enabled, last run 20 seconds ago with 0 failures [19:23:54] RECOVERY - puppet last run on db2019 is OK: OK: Puppet is currently enabled, last run 31 seconds ago with 0 failures [19:24:07] RECOVERY - puppet last run on ms-be2006 is OK: OK: Puppet is currently enabled, last run 1 second ago with 0 failures [19:24:07] RECOVERY - puppet last run on ms-be2003 is OK: OK: Puppet is currently enabled, last run 46 seconds ago with 0 failures [19:24:07] RECOVERY - puppet last run on elastic1004 is OK: OK: Puppet is currently enabled, last run 31 seconds ago with 0 failures [19:24:07] RECOVERY - puppet last run on lvs4002 is OK: OK: Puppet is currently enabled, last run 55 seconds ago with 0 failures [19:24:08] RECOVERY - puppet last run on gold is OK: OK: Puppet is currently enabled, last run 44 seconds ago with 0 failures [19:24:08] RECOVERY - puppet last run on rbf1002 is OK: OK: Puppet is currently enabled, last run 44 seconds ago with 0 failures [19:24:08] RECOVERY - puppet last run on mw1254 is OK: OK: Puppet is currently enabled, last run 23 seconds ago with 0 failures [19:24:11] RECOVERY - puppet last run on mw1250 is OK: OK: Puppet is currently enabled, last run 52 seconds ago with 0 failures [19:24:11] RECOVERY - puppet last run on analytics1025 is OK: OK: Puppet is currently enabled, last run 37 seconds ago with 0 failures [19:24:11] RECOVERY - puppet last run on mc1003 is OK: OK: Puppet is currently enabled, last run 8 seconds ago with 0 failures [19:24:11] RECOVERY - puppet last run on cp1039 is OK: OK: Puppet is currently enabled, last run 16 seconds ago with 0 failures [19:24:11] RECOVERY - puppet last run on lvs1002 is OK: OK: Puppet is currently enabled, last run 10 seconds ago with 0 failures [19:24:11] RECOVERY - puppet last run on cp3020 is OK: OK: Puppet is currently enabled, last run 44 seconds ago with 0 failures [19:24:12] RECOVERY - puppet last run on amssq43 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [19:24:12] RECOVERY - puppet last run on amssq44 is OK: OK: Puppet is currently enabled, last run 58 seconds ago with 0 failures [19:24:12] RECOVERY - puppet last run on cp3015 is OK: OK: Puppet is currently enabled, last run 54 seconds ago with 0 failures [19:24:13] RECOVERY - puppet last run on cp3008 is OK: OK: Puppet is currently enabled, last run 30 seconds ago with 0 failures [19:24:14] RECOVERY - puppet last run on xenon is OK: OK: Puppet is currently enabled, last run 38 seconds ago with 0 failures [19:24:14] RECOVERY - puppet last run on analytics1020 is OK: OK: Puppet is currently enabled, last run 36 seconds ago with 0 failures [19:24:21] RECOVERY - puppet last run on mw1045 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [19:24:21] RECOVERY - puppet last run on mw1224 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [19:24:22] RECOVERY - puppet last run on mw1106 is OK: OK: Puppet is currently enabled, last run 55 seconds ago with 0 failures [19:24:22] RECOVERY - puppet last run on ms-be1006 is OK: OK: Puppet is currently enabled, last run 55 seconds ago with 0 failures [19:24:22] RECOVERY - puppet last run on elastic1018 is OK: OK: Puppet is currently enabled, last run 8 seconds ago with 0 failures [19:24:22] RECOVERY - puppet last run on db1066 is OK: OK: Puppet is currently enabled, last run 30 seconds ago with 0 failures [19:24:22] RECOVERY - puppet last run on helium is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [19:24:31] RECOVERY - puppet last run on mw1205 is OK: OK: Puppet is currently enabled, last run 4 seconds ago with 0 failures [19:24:31] RECOVERY - puppet last run on mw1174 is OK: OK: Puppet is currently enabled, last run 39 seconds ago with 0 failures [19:24:31] RECOVERY - puppet last run on mw1069 is OK: OK: Puppet is currently enabled, last run 13 seconds ago with 0 failures [19:24:42] RECOVERY - puppet last run on lead is OK: OK: Puppet is currently enabled, last run 42 seconds ago with 0 failures [19:24:42] RECOVERY - puppet last run on cp1055 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [19:24:42] RECOVERY - puppet last run on mw1026 is OK: OK: Puppet is currently enabled, last run 53 seconds ago with 0 failures [19:24:51] RECOVERY - puppet last run on mw1046 is OK: OK: Puppet is currently enabled, last run 19 seconds ago with 0 failures [19:24:51] RECOVERY - puppet last run on mw1222 is OK: OK: Puppet is currently enabled, last run 1 second ago with 0 failures [19:24:51] RECOVERY - puppet last run on mw1228 is OK: OK: Puppet is currently enabled, last run 28 seconds ago with 0 failures [19:24:51] RECOVERY - puppet last run on db1022 is OK: OK: Puppet is currently enabled, last run 30 seconds ago with 0 failures [19:24:52] RECOVERY - puppet last run on mc1002 is OK: OK: Puppet is currently enabled, last run 38 seconds ago with 0 failures [19:24:52] RECOVERY - puppet last run on mw1176 is OK: OK: Puppet is currently enabled, last run 38 seconds ago with 0 failures [19:24:52] RECOVERY - puppet last run on mw1164 is OK: OK: Puppet is currently enabled, last run 35 seconds ago with 0 failures [19:24:52] RECOVERY - puppet last run on mw1187 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [19:24:52] RECOVERY - puppet last run on amssq61 is OK: OK: Puppet is currently enabled, last run 7 seconds ago with 0 failures [19:24:53] RECOVERY - puppet last run on amssq54 is OK: OK: Puppet is currently enabled, last run 53 seconds ago with 0 failures [19:24:53] RECOVERY - puppet last run on amssq32 is OK: OK: Puppet is currently enabled, last run 32 seconds ago with 0 failures [19:24:54] RECOVERY - puppet last run on mw1003 is OK: OK: Puppet is currently enabled, last run 9 seconds ago with 0 failures [19:25:01] RECOVERY - puppet last run on cp1047 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [19:25:01] RECOVERY - puppet last run on lvs1005 is OK: OK: Puppet is currently enabled, last run 59 seconds ago with 0 failures [19:25:01] RECOVERY - puppet last run on ms-fe1004 is OK: OK: Puppet is currently enabled, last run 12 seconds ago with 0 failures [19:25:01] RECOVERY - puppet last run on mw1153 is OK: OK: Puppet is currently enabled, last run 51 seconds ago with 0 failures [19:25:01] RECOVERY - puppet last run on mw1100 is OK: OK: Puppet is currently enabled, last run 7 seconds ago with 0 failures [19:25:01] RECOVERY - puppet last run on mw1068 is OK: OK: Puppet is currently enabled, last run 44 seconds ago with 0 failures [19:25:02] RECOVERY - puppet last run on mw1242 is OK: OK: Puppet is currently enabled, last run 40 seconds ago with 0 failures [19:25:02] RECOVERY - puppet last run on capella is OK: OK: Puppet is currently enabled, last run 26 seconds ago with 0 failures [19:25:02] RECOVERY - puppet last run on mw1173 is OK: OK: Puppet is currently enabled, last run 14 seconds ago with 0 failures [19:25:03] RECOVERY - puppet last run on db1050 is OK: OK: Puppet is currently enabled, last run 47 seconds ago with 0 failures [19:25:03] RECOVERY - puppet last run on db2002 is OK: OK: Puppet is currently enabled, last run 23 seconds ago with 0 failures [19:25:04] RECOVERY - puppet last run on es2001 is OK: OK: Puppet is currently enabled, last run 15 seconds ago with 0 failures [19:25:05] RECOVERY - puppet last run on es2009 is OK: OK: Puppet is currently enabled, last run 32 seconds ago with 0 failures [19:25:13] RECOVERY - puppet last run on analytics1035 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [19:25:13] RECOVERY - puppet last run on elastic1008 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [19:25:13] RECOVERY - puppet last run on mw1189 is OK: OK: Puppet is currently enabled, last run 7 seconds ago with 0 failures [19:25:13] RECOVERY - puppet last run on mw1160 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [19:25:13] RECOVERY - puppet last run on sca1002 is OK: OK: Puppet is currently enabled, last run 37 seconds ago with 0 failures [19:25:13] RECOVERY - puppet last run on cp1056 is OK: OK: Puppet is currently enabled, last run 14 seconds ago with 0 failures [19:25:13] RECOVERY - puppet last run on mw1150 is OK: OK: Puppet is currently enabled, last run 33 seconds ago with 0 failures [19:25:13] RECOVERY - puppet last run on elastic1021 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [19:25:21] RECOVERY - puppet last run on virt1006 is OK: OK: Puppet is currently enabled, last run 21 seconds ago with 0 failures [19:25:21] RECOVERY - puppet last run on mw1123 is OK: OK: Puppet is currently enabled, last run 3 seconds ago with 0 failures [19:25:21] RECOVERY - puppet last run on db1059 is OK: OK: Puppet is currently enabled, last run 27 seconds ago with 0 failures [19:25:21] RECOVERY - puppet last run on wtp1016 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [19:25:21] RECOVERY - puppet last run on lvs3001 is OK: OK: Puppet is currently enabled, last run 19 seconds ago with 0 failures [19:25:22] RECOVERY - puppet last run on amssq53 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [19:25:22] RECOVERY - puppet last run on db1018 is OK: OK: Puppet is currently enabled, last run 34 seconds ago with 0 failures [19:25:22] RECOVERY - puppet last run on mw1060 is OK: OK: Puppet is currently enabled, last run 50 seconds ago with 0 failures [19:25:23] RECOVERY - puppet last run on cp3003 is OK: OK: Puppet is currently enabled, last run 17 seconds ago with 0 failures [19:25:23] RECOVERY - puppet last run on db1002 is OK: OK: Puppet is currently enabled, last run 9 seconds ago with 0 failures [19:25:24] RECOVERY - puppet last run on elastic1012 is OK: OK: Puppet is currently enabled, last run 45 seconds ago with 0 failures [19:25:24] RECOVERY - puppet last run on mw1117 is OK: OK: Puppet is currently enabled, last run 31 seconds ago with 0 failures [19:25:24] RECOVERY - puppet last run on labstore1003 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [19:25:31] RECOVERY - puppet last run on db1015 is OK: OK: Puppet is currently enabled, last run 34 seconds ago with 0 failures [19:25:31] RECOVERY - puppet last run on mw1008 is OK: OK: Puppet is currently enabled, last run 43 seconds ago with 0 failures [19:25:32] RECOVERY - puppet last run on mw1217 is OK: OK: Puppet is currently enabled, last run 28 seconds ago with 0 failures [19:25:41] RECOVERY - puppet last run on mw1120 is OK: OK: Puppet is currently enabled, last run 43 seconds ago with 0 failures [19:25:41] RECOVERY - puppet last run on ms-fe1001 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [19:25:41] RECOVERY - puppet last run on db1046 is OK: OK: Puppet is currently enabled, last run 38 seconds ago with 0 failures [19:25:42] RECOVERY - puppet last run on db1040 is OK: OK: Puppet is currently enabled, last run 26 seconds ago with 0 failures [19:25:42] RECOVERY - puppet last run on mw1119 is OK: OK: Puppet is currently enabled, last run 19 seconds ago with 0 failures [19:25:51] RECOVERY - puppet last run on mw1226 is OK: OK: Puppet is currently enabled, last run 52 seconds ago with 0 failures [19:25:51] RECOVERY - puppet last run on elastic1027 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [19:25:51] RECOVERY - puppet last run on mw1065 is OK: OK: Puppet is currently enabled, last run 26 seconds ago with 0 failures [19:25:52] RECOVERY - puppet last run on mw1118 is OK: OK: Puppet is currently enabled, last run 32 seconds ago with 0 failures [19:25:52] RECOVERY - puppet last run on ms-fe2001 is OK: OK: Puppet is currently enabled, last run 15 seconds ago with 0 failures [19:25:52] RECOVERY - puppet last run on labsdb1003 is OK: OK: Puppet is currently enabled, last run 31 seconds ago with 0 failures [19:25:52] RECOVERY - puppet last run on iron is OK: OK: Puppet is currently enabled, last run 53 seconds ago with 0 failures [19:25:52] RECOVERY - git.wikimedia.org on antimony is OK: HTTP OK: HTTP/1.1 200 OK - 59709 bytes in 0.519 second response time [19:25:52] RECOVERY - puppet last run on mw1170 is OK: OK: Puppet is currently enabled, last run 37 seconds ago with 0 failures [19:25:53] RECOVERY - puppet last run on ms-be2004 is OK: OK: Puppet is currently enabled, last run 56 seconds ago with 0 failures [19:26:01] RECOVERY - puppet last run on mw1009 is OK: OK: Puppet is currently enabled, last run 56 seconds ago with 0 failures [19:26:02] RECOVERY - puppet last run on mw1099 is OK: OK: Puppet is currently enabled, last run 48 seconds ago with 0 failures [19:26:02] RECOVERY - puppet last run on elastic1030 is OK: OK: Puppet is currently enabled, last run 26 seconds ago with 0 failures [19:26:03] RECOVERY - puppet last run on cp1061 is OK: OK: Puppet is currently enabled, last run 54 seconds ago with 0 failures [19:26:11] RECOVERY - puppet last run on ruthenium is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [19:26:11] RECOVERY - puppet last run on db1042 is OK: OK: Puppet is currently enabled, last run 6 seconds ago with 0 failures [19:26:11] RECOVERY - puppet last run on cp3016 is OK: OK: Puppet is currently enabled, last run 46 seconds ago with 0 failures [19:26:11] RECOVERY - puppet last run on amssq35 is OK: OK: Puppet is currently enabled, last run 22 seconds ago with 0 failures [19:26:11] RECOVERY - puppet last run on lvs2004 is OK: OK: Puppet is currently enabled, last run 26 seconds ago with 0 failures [19:26:12] RECOVERY - puppet last run on mw1088 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [19:26:12] RECOVERY - puppet last run on mw1025 is OK: OK: Puppet is currently enabled, last run 55 seconds ago with 0 failures [19:26:21] RECOVERY - puppet last run on db2018 is OK: OK: Puppet is currently enabled, last run 44 seconds ago with 0 failures [19:26:31] RECOVERY - puppet last run on dbproxy1001 is OK: OK: Puppet is currently enabled, last run 45 seconds ago with 0 failures [19:26:31] RECOVERY - puppet last run on mw1144 is OK: OK: Puppet is currently enabled, last run 52 seconds ago with 0 failures [19:26:31] RECOVERY - puppet last run on cp4008 is OK: OK: Puppet is currently enabled, last run 13 seconds ago with 0 failures [19:26:31] RECOVERY - puppet last run on mw1052 is OK: OK: Puppet is currently enabled, last run 36 seconds ago with 0 failures [19:26:31] RECOVERY - puppet last run on mw1175 is OK: OK: Puppet is currently enabled, last run 7 seconds ago with 0 failures [19:26:32] RECOVERY - puppet last run on analytics1030 is OK: OK: Puppet is currently enabled, last run 54 seconds ago with 0 failures [19:26:32] RECOVERY - puppet last run on db1023 is OK: OK: Puppet is currently enabled, last run 11 seconds ago with 0 failures [19:26:32] RECOVERY - puppet last run on db1021 is OK: OK: Puppet is currently enabled, last run 43 seconds ago with 0 failures [19:26:33] RECOVERY - puppet last run on mw1251 is OK: OK: Puppet is currently enabled, last run 20 seconds ago with 0 failures [19:26:33] RECOVERY - puppet last run on es1007 is OK: OK: Puppet is currently enabled, last run 2 seconds ago with 0 failures [19:26:33] RECOVERY - puppet last run on db1028 is OK: OK: Puppet is currently enabled, last run 15 seconds ago with 0 failures [19:26:34] RECOVERY - puppet last run on db1067 is OK: OK: Puppet is currently enabled, last run 34 seconds ago with 0 failures [19:26:34] RECOVERY - puppet last run on analytics1010 is OK: OK: Puppet is currently enabled, last run 2 seconds ago with 0 failures [19:26:35] RECOVERY - puppet last run on mw1213 is OK: OK: Puppet is currently enabled, last run 10 seconds ago with 0 failures [19:26:36] RECOVERY - puppet last run on db1034 is OK: OK: Puppet is currently enabled, last run 37 seconds ago with 0 failures [19:26:36] RECOVERY - puppet last run on cp3014 is OK: OK: Puppet is currently enabled, last run 6 seconds ago with 0 failures [19:26:41] RECOVERY - puppet last run on db1051 is OK: OK: Puppet is currently enabled, last run 24 seconds ago with 0 failures [19:26:41] RECOVERY - puppet last run on mw1166 is OK: OK: Puppet is currently enabled, last run 40 seconds ago with 0 failures [19:26:42] RECOVERY - puppet last run on mw1042 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [19:26:42] RECOVERY - puppet last run on mw1177 is OK: OK: Puppet is currently enabled, last run 21 seconds ago with 0 failures [19:26:42] RECOVERY - puppet last run on mw1061 is OK: OK: Puppet is currently enabled, last run 39 seconds ago with 0 failures [19:26:42] RECOVERY - puppet last run on elastic1022 is OK: OK: Puppet is currently enabled, last run 59 seconds ago with 0 failures [19:26:42] RECOVERY - puppet last run on mw1092 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [19:26:51] RECOVERY - puppet last run on logstash1002 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [19:26:51] RECOVERY - puppet last run on lithium is OK: OK: Puppet is currently enabled, last run 0 seconds ago with 0 failures [19:26:52] RECOVERY - puppet last run on mw1126 is OK: OK: Puppet is currently enabled, last run 4 seconds ago with 0 failures [19:26:52] RECOVERY - puppet last run on mw1235 is OK: OK: Puppet is currently enabled, last run 58 seconds ago with 0 failures [19:26:52] RECOVERY - puppet last run on pc1002 is OK: OK: Puppet is currently enabled, last run 24 seconds ago with 0 failures [19:27:01] RECOVERY - puppet last run on lvs2001 is OK: OK: Puppet is currently enabled, last run 4 seconds ago with 0 failures [19:27:02] RECOVERY - puppet last run on db2042 is OK: OK: Puppet is currently enabled, last run 16 seconds ago with 0 failures [19:27:02] RECOVERY - puppet last run on ms-fe2003 is OK: OK: Puppet is currently enabled, last run 16 seconds ago with 0 failures [19:27:11] RECOVERY - puppet last run on mw1002 is OK: OK: Puppet is currently enabled, last run 9 seconds ago with 0 failures [19:27:11] RECOVERY - puppet last run on mw1011 is OK: OK: Puppet is currently enabled, last run 36 seconds ago with 0 failures [19:27:11] RECOVERY - puppet last run on db2036 is OK: OK: Puppet is currently enabled, last run 29 seconds ago with 0 failures [19:27:11] RECOVERY - puppet last run on db1016 is OK: OK: Puppet is currently enabled, last run 9 seconds ago with 0 failures [19:27:11] RECOVERY - puppet last run on gallium is OK: OK: Puppet is currently enabled, last run 24 seconds ago with 0 failures [19:27:11] RECOVERY - puppet last run on cp4003 is OK: OK: Puppet is currently enabled, last run 21 seconds ago with 0 failures [19:27:12] RECOVERY - puppet last run on wtp1012 is OK: OK: Puppet is currently enabled, last run 30 seconds ago with 0 failures [19:27:12] RECOVERY - puppet last run on antimony is OK: OK: Puppet is currently enabled, last run 23 seconds ago with 0 failures [19:27:21] RECOVERY - puppet last run on mw1172 is OK: OK: Puppet is currently enabled, last run 17 seconds ago with 0 failures [19:27:21] RECOVERY - puppet last run on mw1039 is OK: OK: Puppet is currently enabled, last run 29 seconds ago with 0 failures [19:27:22] RECOVERY - puppet last run on amslvs1 is OK: OK: Puppet is currently enabled, last run 49 seconds ago with 0 failures [19:27:22] RECOVERY - puppet last run on amssq48 is OK: OK: Puppet is currently enabled, last run 2 seconds ago with 0 failures [19:27:22] RECOVERY - puppet last run on db2040 is OK: OK: Puppet is currently enabled, last run 54 seconds ago with 0 failures [19:27:22] RECOVERY - puppet last run on analytics1038 is OK: OK: Puppet is currently enabled, last run 41 seconds ago with 0 failures [19:27:22] RECOVERY - puppet last run on labcontrol2001 is OK: OK: Puppet is currently enabled, last run 48 seconds ago with 0 failures [19:27:22] RECOVERY - puppet last run on mw1054 is OK: OK: Puppet is currently enabled, last run 51 seconds ago with 0 failures [19:27:31] RECOVERY - puppet last run on mw1129 is OK: OK: Puppet is currently enabled, last run 22 seconds ago with 0 failures [19:27:31] RECOVERY - puppet last run on nembus is OK: OK: Puppet is currently enabled, last run 6 seconds ago with 0 failures [19:27:32] RECOVERY - puppet last run on labnet1001 is OK: OK: Puppet is currently enabled, last run 24 seconds ago with 0 failures [19:27:41] RECOVERY - puppet last run on mw1211 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [19:27:41] RECOVERY - puppet last run on cp4004 is OK: OK: Puppet is currently enabled, last run 7 seconds ago with 0 failures [19:27:41] RECOVERY - puppet last run on virt1004 is OK: OK: Puppet is currently enabled, last run 14 seconds ago with 0 failures [19:27:41] RECOVERY - puppet last run on mw1195 is OK: OK: Puppet is currently enabled, last run 20 seconds ago with 0 failures [19:27:41] RECOVERY - puppet last run on cp1058 is OK: OK: Puppet is currently enabled, last run 8 seconds ago with 0 failures [19:27:41] RECOVERY - puppet last run on mw1249 is OK: OK: Puppet is currently enabled, last run 24 seconds ago with 0 failures [19:27:51] RECOVERY - puppet last run on mw1237 is OK: OK: Puppet is currently enabled, last run 30 seconds ago with 0 failures [19:27:51] RECOVERY - puppet last run on amssq47 is OK: OK: Puppet is currently enabled, last run 38 seconds ago with 0 failures [19:27:52] RECOVERY - puppet last run on db1052 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [19:27:52] RECOVERY - puppet last run on labstore1001 is OK: OK: Puppet is currently enabled, last run 32 seconds ago with 0 failures [19:27:52] RECOVERY - puppet last run on snapshot1001 is OK: OK: Puppet is currently enabled, last run 40 seconds ago with 0 failures [19:27:52] RECOVERY - puppet last run on mw1076 is OK: OK: Puppet is currently enabled, last run 24 seconds ago with 0 failures [19:28:01] RECOVERY - puppet last run on polonium is OK: OK: Puppet is currently enabled, last run 52 seconds ago with 0 failures [19:28:01] RECOVERY - puppet last run on mw1162 is OK: OK: Puppet is currently enabled, last run 2 seconds ago with 0 failures [19:28:02] RECOVERY - puppet last run on wtp1005 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [19:28:11] RECOVERY - puppet last run on mw1044 is OK: OK: Puppet is currently enabled, last run 46 seconds ago with 0 failures [19:28:12] RECOVERY - puppet last run on db1036 is OK: OK: Puppet is currently enabled, last run 11 seconds ago with 0 failures [19:28:12] RECOVERY - puppet last run on virt1003 is OK: OK: Puppet is currently enabled, last run 37 seconds ago with 0 failures [19:28:12] RECOVERY - puppet last run on db2038 is OK: OK: Puppet is currently enabled, last run 38 seconds ago with 0 failures [19:28:12] RECOVERY - puppet last run on virt1001 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [19:28:21] RECOVERY - puppet last run on db1003 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [19:28:21] RECOVERY - puppet last run on mw1208 is OK: OK: Puppet is currently enabled, last run 46 seconds ago with 0 failures [19:28:21] RECOVERY - puppet last run on mc1012 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [19:28:22] RECOVERY - puppet last run on db1060 is OK: OK: Puppet is currently enabled, last run 24 seconds ago with 0 failures [19:28:22] RECOVERY - puppet last run on ms-be2011 is OK: OK: Puppet is currently enabled, last run 4 seconds ago with 0 failures [19:28:22] RECOVERY - puppet last run on mw1227 is OK: OK: Puppet is currently enabled, last run 3 seconds ago with 0 failures [19:28:22] RECOVERY - puppet last run on lvs4003 is OK: OK: Puppet is currently enabled, last run 5 seconds ago with 0 failures [19:28:22] RECOVERY - puppet last run on mw1180 is OK: OK: Puppet is currently enabled, last run 1 second ago with 0 failures [19:28:22] RECOVERY - puppet last run on mw1114 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [19:28:23] RECOVERY - puppet last run on dataset1001 is OK: OK: Puppet is currently enabled, last run 55 seconds ago with 0 failures [19:28:24] RECOVERY - puppet last run on cp4014 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [19:28:31] RECOVERY - puppet last run on mc1014 is OK: OK: Puppet is currently enabled, last run 32 seconds ago with 0 failures [19:28:32] RECOVERY - puppet last run on mw1206 is OK: OK: Puppet is currently enabled, last run 20 seconds ago with 0 failures [19:28:32] RECOVERY - puppet last run on analytics1013 is OK: OK: Puppet is currently enabled, last run 7 seconds ago with 0 failures [19:28:32] RECOVERY - puppet last run on snapshot1002 is OK: OK: Puppet is currently enabled, last run 6 seconds ago with 0 failures [19:28:32] RECOVERY - puppet last run on amssq34 is OK: OK: Puppet is currently enabled, last run 14 seconds ago with 0 failures [19:28:32] RECOVERY - puppet last run on amssq55 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [19:28:33] RECOVERY - puppet last run on nescio is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [19:28:33] RECOVERY - puppet last run on amssq60 is OK: OK: Puppet is currently enabled, last run 57 seconds ago with 0 failures [19:28:34] RECOVERY - puppet last run on ms-fe3002 is OK: OK: Puppet is currently enabled, last run 3 seconds ago with 0 failures [19:28:34] RECOVERY - puppet last run on analytics1016 is OK: OK: Puppet is currently enabled, last run 57 seconds ago with 0 failures [19:28:34] RECOVERY - puppet last run on db2007 is OK: OK: Puppet is currently enabled, last run 45 seconds ago with 0 failures [19:28:35] RECOVERY - puppet last run on es2004 is OK: OK: Puppet is currently enabled, last run 25 seconds ago with 0 failures [19:28:35] RECOVERY - puppet last run on db2029 is OK: OK: Puppet is currently enabled, last run 42 seconds ago with 0 failures [19:28:41] RECOVERY - puppet last run on analytics1026 is OK: OK: Puppet is currently enabled, last run 19 seconds ago with 0 failures [19:28:41] RECOVERY - puppet last run on mw1149 is OK: OK: Puppet is currently enabled, last run 27 seconds ago with 0 failures [19:28:41] RECOVERY - puppet last run on db2016 is OK: OK: Puppet is currently enabled, last run 20 seconds ago with 0 failures [19:28:41] RECOVERY - puppet last run on elastic1006 is OK: OK: Puppet is currently enabled, last run 1 second ago with 0 failures [19:28:41] RECOVERY - puppet last run on db1043 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [19:28:42] RECOVERY - puppet last run on install2001 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [19:28:51] RECOVERY - puppet last run on elastic1019 is OK: OK: Puppet is currently enabled, last run 53 seconds ago with 0 failures [19:28:51] RECOVERY - puppet last run on mw1168 is OK: OK: Puppet is currently enabled, last run 1 second ago with 0 failures [19:28:51] RECOVERY - puppet last run on bast4001 is OK: OK: Puppet is currently enabled, last run 30 seconds ago with 0 failures [19:28:52] RECOVERY - puppet last run on hafnium is OK: OK: Puppet is currently enabled, last run 7 seconds ago with 0 failures [19:28:52] RECOVERY - puppet last run on gadolinium is OK: OK: Puppet is currently enabled, last run 7 seconds ago with 0 failures [19:28:52] RECOVERY - puppet last run on mc1005 is OK: OK: Puppet is currently enabled, last run 23 seconds ago with 0 failures [19:28:52] RECOVERY - puppet last run on ms-be3001 is OK: OK: Puppet is currently enabled, last run 30 seconds ago with 0 failures [19:29:01] RECOVERY - puppet last run on plutonium is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [19:29:01] RECOVERY - puppet last run on mw1125 is OK: OK: Puppet is currently enabled, last run 10 seconds ago with 0 failures [19:29:01] RECOVERY - puppet last run on amssq46 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [19:29:01] RECOVERY - puppet last run on ms-be3002 is OK: OK: Puppet is currently enabled, last run 57 seconds ago with 0 failures [19:29:01] RECOVERY - puppet last run on cp1062 is OK: OK: Puppet is currently enabled, last run 2 seconds ago with 0 failures [19:29:02] RECOVERY - puppet last run on tin is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [19:29:02] RECOVERY - puppet last run on mw1247 is OK: OK: Puppet is currently enabled, last run 44 seconds ago with 0 failures [19:29:02] RECOVERY - puppet last run on cp1050 is OK: OK: Puppet is currently enabled, last run 23 seconds ago with 0 failures [19:29:03] RECOVERY - puppet last run on cp1046 is OK: OK: Puppet is currently enabled, last run 46 seconds ago with 0 failures [19:29:03] RECOVERY - puppet last run on db1004 is OK: OK: Puppet is currently enabled, last run 6 seconds ago with 0 failures [19:29:04] RECOVERY - puppet last run on stat1003 is OK: OK: Puppet is currently enabled, last run 59 seconds ago with 0 failures [19:29:04] RECOVERY - puppet last run on db1048 is OK: OK: Puppet is currently enabled, last run 38 seconds ago with 0 failures [19:29:05] RECOVERY - puppet last run on mw1051 is OK: OK: Puppet is currently enabled, last run 35 seconds ago with 0 failures [19:29:05] RECOVERY - puppet last run on db1071 is OK: OK: Puppet is currently enabled, last run 15 seconds ago with 0 failures [19:29:11] RECOVERY - puppet last run on lvs3004 is OK: OK: Puppet is currently enabled, last run 23 seconds ago with 0 failures [19:29:11] RECOVERY - puppet last run on mw1055 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [19:29:11] RECOVERY - puppet last run on mw1098 is OK: OK: Puppet is currently enabled, last run 10 seconds ago with 0 failures [19:29:11] RECOVERY - puppet last run on db1026 is OK: OK: Puppet is currently enabled, last run 46 seconds ago with 0 failures [19:29:12] RECOVERY - puppet last run on mw1084 is OK: OK: Puppet is currently enabled, last run 44 seconds ago with 0 failures [19:29:12] RECOVERY - puppet last run on mw1190 is OK: OK: Puppet is currently enabled, last run 18 seconds ago with 0 failures [19:29:12] RECOVERY - puppet last run on mw1238 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [19:29:12] RECOVERY - puppet last run on elastic1024 is OK: OK: Puppet is currently enabled, last run 31 seconds ago with 0 failures [19:29:13] RECOVERY - puppet last run on argon is OK: OK: Puppet is currently enabled, last run 49 seconds ago with 0 failures [19:29:21] RECOVERY - puppet last run on mw1133 is OK: OK: Puppet is currently enabled, last run 16 seconds ago with 0 failures [19:29:21] RECOVERY - puppet last run on mw1111 is OK: OK: Puppet is currently enabled, last run 52 seconds ago with 0 failures [19:29:21] RECOVERY - puppet last run on lvs2006 is OK: OK: Puppet is currently enabled, last run 56 seconds ago with 0 failures [19:29:22] RECOVERY - puppet last run on analytics1023 is OK: OK: Puppet is currently enabled, last run 28 seconds ago with 0 failures [19:29:22] RECOVERY - puppet last run on mw1079 is OK: OK: Puppet is currently enabled, last run 46 seconds ago with 0 failures [19:29:22] RECOVERY - puppet last run on mw1014 is OK: OK: Puppet is currently enabled, last run 54 seconds ago with 0 failures [19:29:31] RECOVERY - puppet last run on thallium is OK: OK: Puppet is currently enabled, last run 29 seconds ago with 0 failures [19:29:31] RECOVERY - puppet last run on osmium is OK: OK: Puppet is currently enabled, last run 40 seconds ago with 0 failures [19:29:31] RECOVERY - puppet last run on oxygen is OK: OK: Puppet is currently enabled, last run 14 seconds ago with 0 failures [19:29:31] RECOVERY - puppet last run on mw1202 is OK: OK: Puppet is currently enabled, last run 19 seconds ago with 0 failures [19:29:31] RECOVERY - puppet last run on rhenium is OK: OK: Puppet is currently enabled, last run 5 seconds ago with 0 failures [19:29:32] RECOVERY - puppet last run on analytics1022 is OK: OK: Puppet is currently enabled, last run 43 seconds ago with 0 failures [19:29:32] RECOVERY - puppet last run on cp4001 is OK: OK: Puppet is currently enabled, last run 54 seconds ago with 0 failures [19:29:32] RECOVERY - puppet last run on cp4018 is OK: OK: Puppet is currently enabled, last run 20 seconds ago with 0 failures [19:29:41] RECOVERY - puppet last run on db1039 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [19:29:41] RECOVERY - puppet last run on rdb1001 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [19:29:41] RECOVERY - puppet last run on mw1146 is OK: OK: Puppet is currently enabled, last run 14 seconds ago with 0 failures [19:29:41] RECOVERY - puppet last run on elastic1015 is OK: OK: Puppet is currently enabled, last run 53 seconds ago with 0 failures [19:29:41] RECOVERY - puppet last run on wtp1002 is OK: OK: Puppet is currently enabled, last run 2 seconds ago with 0 failures [19:29:42] RECOVERY - puppet last run on ms-be1012 is OK: OK: Puppet is currently enabled, last run 49 seconds ago with 0 failures [19:29:42] RECOVERY - puppet last run on amssq40 is OK: OK: Puppet is currently enabled, last run 35 seconds ago with 0 failures [19:29:42] RECOVERY - puppet last run on amssq51 is OK: OK: Puppet is currently enabled, last run 4 seconds ago with 0 failures [19:29:42] RECOVERY - puppet last run on cp3010 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [19:29:51] RECOVERY - puppet last run on virt1007 is OK: OK: Puppet is currently enabled, last run 46 seconds ago with 0 failures [19:29:51] RECOVERY - puppet last run on mw1258 is OK: OK: Puppet is currently enabled, last run 36 seconds ago with 0 failures [19:29:52] RECOVERY - puppet last run on db2004 is OK: OK: Puppet is currently enabled, last run 37 seconds ago with 0 failures [19:29:52] RECOVERY - puppet last run on db2001 is OK: OK: Puppet is currently enabled, last run 32 seconds ago with 0 failures [19:29:52] RECOVERY - puppet last run on ms-be2012 is OK: OK: Puppet is currently enabled, last run 16 seconds ago with 0 failures [19:29:52] RECOVERY - puppet last run on db2023 is OK: OK: Puppet is currently enabled, last run 36 seconds ago with 0 failures [19:29:52] RECOVERY - puppet last run on labmon1001 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [19:29:53] RECOVERY - puppet last run on mw1049 is OK: OK: Puppet is currently enabled, last run 44 seconds ago with 0 failures [19:29:53] RECOVERY - puppet last run on mw1050 is OK: OK: Puppet is currently enabled, last run 28 seconds ago with 0 failures [19:29:53] RECOVERY - puppet last run on ms-be2008 is OK: OK: Puppet is currently enabled, last run 23 seconds ago with 0 failures [19:29:54] RECOVERY - puppet last run on mw1183 is OK: OK: Puppet is currently enabled, last run 37 seconds ago with 0 failures [19:30:01] RECOVERY - puppet last run on db1020 is OK: OK: Puppet is currently enabled, last run 56 seconds ago with 0 failures [19:30:01] RECOVERY - puppet last run on mw1081 is OK: OK: Puppet is currently enabled, last run 14 seconds ago with 0 failures [19:30:02] RECOVERY - puppet last run on cp4005 is OK: OK: Puppet is currently enabled, last run 3 seconds ago with 0 failures [19:30:02] RECOVERY - puppet last run on cp4019 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [19:30:02] RECOVERY - puppet last run on wtp1018 is OK: OK: Puppet is currently enabled, last run 32 seconds ago with 0 failures [19:30:02] RECOVERY - puppet last run on virt1010 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [19:30:02] RECOVERY - puppet last run on mw1151 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [19:30:11] RECOVERY - puppet last run on mw1159 is OK: OK: Puppet is currently enabled, last run 7 seconds ago with 0 failures [19:30:11] RECOVERY - puppet last run on mw1165 is OK: OK: Puppet is currently enabled, last run 15 seconds ago with 0 failures [19:30:11] RECOVERY - puppet last run on amssq36 is OK: OK: Puppet is currently enabled, last run 20 seconds ago with 0 failures [19:30:11] RECOVERY - puppet last run on amssq56 is OK: OK: Puppet is currently enabled, last run 38 seconds ago with 0 failures [19:30:11] RECOVERY - puppet last run on amssq41 is OK: OK: Puppet is currently enabled, last run 2 seconds ago with 0 failures [19:30:12] RECOVERY - puppet last run on db1069 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [19:30:12] RECOVERY - puppet last run on labsdb1006 is OK: OK: Puppet is currently enabled, last run 34 seconds ago with 0 failures [19:30:12] RECOVERY - puppet last run on snapshot1004 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [19:30:12] RECOVERY - puppet last run on analytics1032 is OK: OK: Puppet is currently enabled, last run 20 seconds ago with 0 failures [19:30:22] RECOVERY - puppet last run on mw1056 is OK: OK: Puppet is currently enabled, last run 15 seconds ago with 0 failures [19:30:22] RECOVERY - puppet last run on mw1181 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [19:30:22] RECOVERY - puppet last run on elastic1011 is OK: OK: Puppet is currently enabled, last run 32 seconds ago with 0 failures [19:30:31] RECOVERY - puppet last run on mw1034 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [19:30:31] RECOVERY - puppet last run on rubidium is OK: OK: Puppet is currently enabled, last run 57 seconds ago with 0 failures [19:30:31] RECOVERY - puppet last run on mw1030 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [19:30:31] RECOVERY - puppet last run on wtp1023 is OK: OK: Puppet is currently enabled, last run 50 seconds ago with 0 failures [19:30:32] RECOVERY - puppet last run on stat1002 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [19:30:32] RECOVERY - puppet last run on mw1248 is OK: OK: Puppet is currently enabled, last run 9 seconds ago with 0 failures [19:30:32] RECOVERY - puppet last run on db2037 is OK: OK: Puppet is currently enabled, last run 56 seconds ago with 0 failures [19:30:41] RECOVERY - puppet last run on mw1156 is OK: OK: Puppet is currently enabled, last run 55 seconds ago with 0 failures [19:30:41] RECOVERY - puppet last run on es1002 is OK: OK: Puppet is currently enabled, last run 2 seconds ago with 0 failures [19:30:41] RECOVERY - puppet last run on cp1063 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [19:30:42] RECOVERY - puppet last run on ms-be2001 is OK: OK: Puppet is currently enabled, last run 43 seconds ago with 0 failures [19:30:42] RECOVERY - puppet last run on ms-be1007 is OK: OK: Puppet is currently enabled, last run 46 seconds ago with 0 failures [19:30:42] RECOVERY - puppet last run on mc1001 is OK: OK: Puppet is currently enabled, last run 57 seconds ago with 0 failures [19:30:42] RECOVERY - puppet last run on ms-be1008 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [19:30:51] RECOVERY - puppet last run on cp1048 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [19:30:51] RECOVERY - puppet last run on wtp1004 is OK: OK: Puppet is currently enabled, last run 29 seconds ago with 0 failures [19:30:52] RECOVERY - puppet last run on hooft is OK: OK: Puppet is currently enabled, last run 1 second ago with 0 failures [19:30:52] RECOVERY - puppet last run on mw1188 is OK: OK: Puppet is currently enabled, last run 35 seconds ago with 0 failures [19:30:52] RECOVERY - puppet last run on mw1057 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [19:30:52] RECOVERY - puppet last run on analytics1011 is OK: OK: Puppet is currently enabled, last run 1 second ago with 0 failures [19:31:01] RECOVERY - puppet last run on mw1097 is OK: OK: Puppet is currently enabled, last run 16 seconds ago with 0 failures [19:31:01] RECOVERY - puppet last run on mw1053 is OK: OK: Puppet is currently enabled, last run 1 second ago with 0 failures [19:31:01] RECOVERY - puppet last run on es2007 is OK: OK: Puppet is currently enabled, last run 17 seconds ago with 0 failures [19:31:01] RECOVERY - puppet last run on ms-be2005 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [19:31:01] RECOVERY - puppet last run on db1062 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [19:31:02] RECOVERY - puppet last run on mw1212 is OK: OK: Puppet is currently enabled, last run 21 seconds ago with 0 failures [19:31:02] RECOVERY - puppet last run on mw1004 is OK: OK: Puppet is currently enabled, last run 54 seconds ago with 0 failures [19:31:02] RECOVERY - puppet last run on ms-be1009 is OK: OK: Puppet is currently enabled, last run 43 seconds ago with 0 failures [19:31:03] RECOVERY - puppet last run on analytics1037 is OK: OK: Puppet is currently enabled, last run 40 seconds ago with 0 failures [19:31:07] RECOVERY - puppet last run on haedus is OK: OK: Puppet is currently enabled, last run 42 seconds ago with 0 failures [19:31:07] RECOVERY - puppet last run on acamar is OK: OK: Puppet is currently enabled, last run 26 seconds ago with 0 failures [19:31:07] RECOVERY - puppet last run on ms-be2007 is OK: OK: Puppet is currently enabled, last run 9 seconds ago with 0 failures [19:31:11] RECOVERY - puppet last run on tungsten is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [19:31:11] RECOVERY - puppet last run on mw1198 is OK: OK: Puppet is currently enabled, last run 14 seconds ago with 0 failures [19:31:11] RECOVERY - puppet last run on db1054 is OK: OK: Puppet is currently enabled, last run 19 seconds ago with 0 failures [19:31:12] RECOVERY - puppet last run on protactinium is OK: OK: Puppet is currently enabled, last run 2 seconds ago with 0 failures [19:31:12] RECOVERY - puppet last run on mw1116 is OK: OK: Puppet is currently enabled, last run 6 seconds ago with 0 failures [19:31:12] RECOVERY - puppet last run on mw1087 is OK: OK: Puppet is currently enabled, last run 21 seconds ago with 0 failures [19:31:12] RECOVERY - puppet last run on rdb1002 is OK: OK: Puppet is currently enabled, last run 33 seconds ago with 0 failures [19:31:12] RECOVERY - puppet last run on titanium is OK: OK: Puppet is currently enabled, last run 10 seconds ago with 0 failures [19:31:13] RECOVERY - puppet last run on mw1210 is OK: OK: Puppet is currently enabled, last run 30 seconds ago with 0 failures [19:31:21] RECOVERY - puppet last run on wtp1011 is OK: OK: Puppet is currently enabled, last run 41 seconds ago with 0 failures [19:31:21] RECOVERY - puppet last run on amssq62 is OK: OK: Puppet is currently enabled, last run 9 seconds ago with 0 failures [19:31:22] RECOVERY - puppet last run on mw1171 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [19:31:22] RECOVERY - puppet last run on wtp1013 is OK: OK: Puppet is currently enabled, last run 49 seconds ago with 0 failures [19:31:22] RECOVERY - puppet last run on db1057 is OK: OK: Puppet is currently enabled, last run 44 seconds ago with 0 failures [19:31:22] RECOVERY - puppet last run on elastic1014 is OK: OK: Puppet is currently enabled, last run 57 seconds ago with 0 failures [19:31:22] RECOVERY - puppet last run on db1055 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [19:31:32] RECOVERY - puppet last run on wtp1022 is OK: OK: Puppet is currently enabled, last run 58 seconds ago with 0 failures [19:31:32] RECOVERY - puppet last run on mw1243 is OK: OK: Puppet is currently enabled, last run 20 seconds ago with 0 failures [19:31:32] RECOVERY - puppet last run on mw1023 is OK: OK: Puppet is currently enabled, last run 21 seconds ago with 0 failures [19:31:32] RECOVERY - puppet last run on lvs1001 is OK: OK: Puppet is currently enabled, last run 58 seconds ago with 0 failures [19:31:33] RECOVERY - puppet last run on analytics1014 is OK: OK: Puppet is currently enabled, last run 32 seconds ago with 0 failures [19:31:41] RECOVERY - puppet last run on mw1163 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [19:31:41] RECOVERY - puppet last run on rcs1001 is OK: OK: Puppet is currently enabled, last run 55 seconds ago with 0 failures [19:31:41] RECOVERY - puppet last run on cp1038 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [19:31:42] RECOVERY - puppet last run on elastic1005 is OK: OK: Puppet is currently enabled, last run 16 seconds ago with 0 failures [19:31:51] RECOVERY - puppet last run on rcs1002 is OK: OK: Puppet is currently enabled, last run 29 seconds ago with 0 failures [19:31:51] RECOVERY - puppet last run on db1001 is OK: OK: Puppet is currently enabled, last run 54 seconds ago with 0 failures [19:31:52] RECOVERY - puppet last run on mw1029 is OK: OK: Puppet is currently enabled, last run 53 seconds ago with 0 failures [19:31:52] RECOVERY - puppet last run on mc1013 is OK: OK: Puppet is currently enabled, last run 28 seconds ago with 0 failures [19:31:52] RECOVERY - puppet last run on mw1074 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [19:31:53] RECOVERY - puppet last run on db1070 is OK: OK: Puppet is currently enabled, last run 29 seconds ago with 0 failures [19:32:02] RECOVERY - puppet last run on labsdb1004 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [19:32:02] RECOVERY - puppet last run on db1064 is OK: OK: Puppet is currently enabled, last run 29 seconds ago with 0 failures [19:32:02] RECOVERY - puppet last run on amslvs3 is OK: OK: Puppet is currently enabled, last run 6 seconds ago with 0 failures [19:32:02] RECOVERY - puppet last run on db1044 is OK: OK: Puppet is currently enabled, last run 13 seconds ago with 0 failures [19:32:02] RECOVERY - puppet last run on mw1032 is OK: OK: Puppet is currently enabled, last run 23 seconds ago with 0 failures [19:32:11] RECOVERY - puppet last run on lvs2003 is OK: OK: Puppet is currently enabled, last run 59 seconds ago with 0 failures [19:32:11] RECOVERY - puppet last run on wtp1015 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [19:32:11] RECOVERY - puppet last run on db2003 is OK: OK: Puppet is currently enabled, last run 57 seconds ago with 0 failures [19:32:12] RECOVERY - puppet last run on elastic1002 is OK: OK: Puppet is currently enabled, last run 43 seconds ago with 0 failures [19:32:12] RECOVERY - puppet last run on mw1148 is OK: OK: Puppet is currently enabled, last run 17 seconds ago with 0 failures [19:32:21] RECOVERY - puppet last run on mc1007 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [19:32:21] RECOVERY - puppet last run on cp1060 is OK: OK: Puppet is currently enabled, last run 52 seconds ago with 0 failures [19:32:31] RECOVERY - puppet last run on mw1001 is OK: OK: Puppet is currently enabled, last run 10 seconds ago with 0 failures [19:32:31] RECOVERY - puppet last run on db1030 is OK: OK: Puppet is currently enabled, last run 22 seconds ago with 0 failures [19:32:32] RECOVERY - puppet last run on mw1121 is OK: OK: Puppet is currently enabled, last run 8 seconds ago with 0 failures [19:32:32] RECOVERY - puppet last run on elastic1023 is OK: OK: Puppet is currently enabled, last run 51 seconds ago with 0 failures [19:32:41] RECOVERY - puppet last run on db1063 is OK: OK: Puppet is currently enabled, last run 18 seconds ago with 0 failures [19:32:42] RECOVERY - puppet last run on wtp1003 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [19:32:42] RECOVERY - puppet last run on mw1229 is OK: OK: Puppet is currently enabled, last run 21 seconds ago with 0 failures [19:32:42] RECOVERY - puppet last run on ocg1003 is OK: OK: Puppet is currently enabled, last run 51 seconds ago with 0 failures [19:32:42] RECOVERY - puppet last run on nitrogen is OK: OK: Puppet is currently enabled, last run 49 seconds ago with 0 failures [19:32:51] RECOVERY - puppet last run on db1038 is OK: OK: Puppet is currently enabled, last run 1 second ago with 0 failures [19:32:51] RECOVERY - puppet last run on mw1122 is OK: OK: Puppet is currently enabled, last run 28 seconds ago with 0 failures [19:32:51] RECOVERY - puppet last run on mw1024 is OK: OK: Puppet is currently enabled, last run 11 seconds ago with 0 failures [19:32:51] RECOVERY - puppet last run on mw1239 is OK: OK: Puppet is currently enabled, last run 52 seconds ago with 0 failures [19:32:51] RECOVERY - puppet last run on mw1225 is OK: OK: Puppet is currently enabled, last run 42 seconds ago with 0 failures [19:32:52] RECOVERY - puppet last run on elastic1017 is OK: OK: Puppet is currently enabled, last run 54 seconds ago with 0 failures [19:32:52] RECOVERY - puppet last run on mercury is OK: OK: Puppet is currently enabled, last run 9 seconds ago with 0 failures [19:33:02] RECOVERY - puppet last run on rdb1003 is OK: OK: Puppet is currently enabled, last run 58 seconds ago with 0 failures [19:33:05] RECOVERY - puppet last run on mw1186 is OK: OK: Puppet is currently enabled, last run 16 seconds ago with 0 failures [19:33:05] RECOVERY - puppet last run on mw1105 is OK: OK: Puppet is currently enabled, last run 33 seconds ago with 0 failures [19:33:11] RECOVERY - puppet last run on pc1003 is OK: OK: Puppet is currently enabled, last run 49 seconds ago with 0 failures [19:33:12] RECOVERY - puppet last run on mw1201 is OK: OK: Puppet is currently enabled, last run 39 seconds ago with 0 failures [19:33:12] RECOVERY - puppet last run on mw1185 is OK: OK: Puppet is currently enabled, last run 44 seconds ago with 0 failures [19:33:12] RECOVERY - puppet last run on db1006 is OK: OK: Puppet is currently enabled, last run 17 seconds ago with 0 failures [19:33:22] RECOVERY - puppet last run on db1033 is OK: OK: Puppet is currently enabled, last run 16 seconds ago with 0 failures [19:33:22] RECOVERY - puppet last run on cp3004 is OK: OK: Puppet is currently enabled, last run 23 seconds ago with 0 failures [19:33:22] RECOVERY - puppet last run on mw1043 is OK: OK: Puppet is currently enabled, last run 11 seconds ago with 0 failures [19:33:22] RECOVERY - puppet last run on db2028 is OK: OK: Puppet is currently enabled, last run 19 seconds ago with 0 failures [19:33:22] RECOVERY - puppet last run on db2011 is OK: OK: Puppet is currently enabled, last run 35 seconds ago with 0 failures [19:33:22] RECOVERY - puppet last run on virt1000 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [19:33:32] RECOVERY - puppet last run on mw1108 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [19:33:32] RECOVERY - puppet last run on db1061 is OK: OK: Puppet is currently enabled, last run 48 seconds ago with 0 failures [19:33:32] RECOVERY - puppet last run on mw1152 is OK: OK: Puppet is currently enabled, last run 33 seconds ago with 0 failures [19:33:32] RECOVERY - puppet last run on dbstore1002 is OK: OK: Puppet is currently enabled, last run 40 seconds ago with 0 failures [19:33:32] RECOVERY - puppet last run on wtp1007 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [19:33:41] RECOVERY - puppet last run on cp1052 is OK: OK: Puppet is currently enabled, last run 31 seconds ago with 0 failures [19:33:41] RECOVERY - puppet last run on db1047 is OK: OK: Puppet is currently enabled, last run 22 seconds ago with 0 failures [19:33:41] RECOVERY - puppet last run on cp3012 is OK: OK: Puppet is currently enabled, last run 58 seconds ago with 0 failures [19:33:42] RECOVERY - puppet last run on amssq38 is OK: OK: Puppet is currently enabled, last run 50 seconds ago with 0 failures [19:33:42] RECOVERY - puppet last run on mw1022 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [19:33:42] RECOVERY - puppet last run on ms-fe3001 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [19:33:42] RECOVERY - puppet last run on cp3009 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [19:33:48] RECOVERY - puppet last run on iridium is OK: OK: Puppet is currently enabled, last run 44 seconds ago with 0 failures [19:33:48] RECOVERY - puppet last run on mw1231 is OK: OK: Puppet is currently enabled, last run 30 seconds ago with 0 failures [19:33:48] RECOVERY - puppet last run on mw1167 is OK: OK: Puppet is currently enabled, last run 57 seconds ago with 0 failures [19:33:48] RECOVERY - puppet last run on mw1219 is OK: OK: Puppet is currently enabled, last run 31 seconds ago with 0 failures [19:33:51] RECOVERY - puppet last run on d-i-test is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [19:33:51] RECOVERY - puppet last run on es1010 is OK: OK: Puppet is currently enabled, last run 14 seconds ago with 0 failures [19:33:51] RECOVERY - puppet last run on mw1077 is OK: OK: Puppet is currently enabled, last run 40 seconds ago with 0 failures [19:33:52] RECOVERY - puppet last run on ms-be1011 is OK: OK: Puppet is currently enabled, last run 18 seconds ago with 0 failures [19:33:52] RECOVERY - puppet last run on mw1033 is OK: OK: Puppet is currently enabled, last run 41 seconds ago with 0 failures [19:33:52] RECOVERY - puppet last run on iodine is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [19:33:52] RECOVERY - puppet last run on analytics1027 is OK: OK: Puppet is currently enabled, last run 54 seconds ago with 0 failures [19:34:01] RECOVERY - puppet last run on db1011 is OK: OK: Puppet is currently enabled, last run 47 seconds ago with 0 failures [19:34:11] RECOVERY - puppet last run on analytics1028 is OK: OK: Puppet is currently enabled, last run 41 seconds ago with 0 failures [19:34:12] RECOVERY - puppet last run on calcium is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [19:34:12] RECOVERY - puppet last run on fluorine is OK: OK: Puppet is currently enabled, last run 58 seconds ago with 0 failures [19:34:22] RECOVERY - puppet last run on cp1037 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [19:34:22] RECOVERY - puppet last run on mw1209 is OK: OK: Puppet is currently enabled, last run 33 seconds ago with 0 failures [19:34:22] RECOVERY - puppet last run on mw1139 is OK: OK: Puppet is currently enabled, last run 54 seconds ago with 0 failures [19:34:31] RECOVERY - puppet last run on mw1093 is OK: OK: Puppet is currently enabled, last run 59 seconds ago with 0 failures [19:34:32] RECOVERY - puppet last run on db2012 is OK: OK: Puppet is currently enabled, last run 44 seconds ago with 0 failures [19:34:42] RECOVERY - puppet last run on cp4020 is OK: OK: Puppet is currently enabled, last run 6 seconds ago with 0 failures [19:34:51] RECOVERY - puppet last run on mw1010 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [19:34:52] RECOVERY - puppet last run on amssq31 is OK: OK: Puppet is currently enabled, last run 33 seconds ago with 0 failures [20:59:16] <_joe_> !log restarting hhvm on mw1192, stuck in a deadlock inside HPHP::f_ini_set () [20:59:21] Logged the message, Master [21:00:02] RECOVERY - Apache HTTP on mw1192 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 440 bytes in 0.078 second response time [22:31:12] PROBLEM - puppet last run on tungsten is CRITICAL: CRITICAL: puppet fail [22:51:22] RECOVERY - puppet last run on tungsten is OK: OK: Puppet is currently enabled, last run 24 seconds ago with 0 failures [23:07:17] !log xtrabackup clone db1051 to db2016 [23:07:29] Logged the message, Master [23:50:32] PROBLEM - puppet last run on lvs3004 is CRITICAL: CRITICAL: puppet fail [23:56:50] (03CR) 10Alexandros Kosiaris: [C: 032 V: 032] let bastion hosts have base::firewall [puppet] - 10https://gerrit.wikimedia.org/r/96424 (owner: 10Dzahn) [23:57:10] <_joe_> akosiaris: you fool [23:57:12] <_joe_> :) [23:57:37] <_joe_> they're going to move us out in 5 mins :) [23:59:14] _joe_: :P