[00:35:21] <wikibugs>	 06Operations, 10Analytics: Can't log into https://piwik.wikimedia.org/ - https://phabricator.wikimedia.org/T144326#2607183 (10Nuria) I have been able to log in with the old credentials for user piwik-wmf-admin (the ones  on /home/milimetric/piwik.credentials )   Could you try  again @milimetric?
[02:25:13] <logmsgbot>	 !log mwdeploy@tin scap sync-l10n completed (1.28.0-wmf.17) (duration: 09m 57s)
[02:25:19] <morebots>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master
[02:31:20] <logmsgbot>	 !log l10nupdate@tin ResourceLoader cache refresh completed at Sun Sep  4 02:31:20 UTC 2016 (duration 6m 7s)
[02:31:26] <morebots>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master
[04:19:28] <wikibugs>	 10Blocked-on-Operations, 06Operations, 10Kartographer, 06Maps, and 5 others: Enable Interactive Maps (Kartographer) on Macedonian Wikipedia - https://phabricator.wikimedia.org/T139946#2607248 (10Yurik)
[05:43:00] <icinga-wm>	 PROBLEM - MediaWiki exceptions and fatals per minute on graphite1001 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [50.0]
[05:45:29] <icinga-wm>	 RECOVERY - MediaWiki exceptions and fatals per minute on graphite1001 is OK: OK: Less than 1.00% above the threshold [25.0]
[07:00:30] <icinga-wm>	 PROBLEM - tools homepage -admin tool- on tools.wmflabs.org is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 531 bytes in 0.039 second response time
[07:12:40] <icinga-wm>	 RECOVERY - tools homepage -admin tool- on tools.wmflabs.org is OK: HTTP OK: HTTP/1.1 200 OK - 3670 bytes in 0.029 second response time
[08:18:02] <icinga-wm>	 PROBLEM - MD RAID on relforge1001 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds.
[08:22:50] <icinga-wm>	 RECOVERY - MD RAID on relforge1001 is OK: OK: Active: 8, Working: 8, Failed: 0, Spare: 0
[08:35:25] <icinga-wm>	 PROBLEM - MD RAID on relforge1001 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds.
[08:40:15] <icinga-wm>	 RECOVERY - MD RAID on relforge1001 is OK: OK: Active: 8, Working: 8, Failed: 0, Spare: 0
[08:48:31] <icinga-wm>	 PROBLEM - puppet last run on pc2004 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 6 minutes ago with 1 failures. Failed resources (up to 3 shown): File[/usr/lib/nagios/plugins/check-fresh-files-in-dir.py]
[09:01:44] <icinga-wm>	 PROBLEM - MariaDB Slave Lag: s1 on db1047 is CRITICAL: CRITICAL slave_sql_lag Replication lag: 9682.93 seconds
[09:02:33] <icinga-wm>	 PROBLEM - MD RAID on relforge1001 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds.
[09:10:49] <icinga-wm>	 ACKNOWLEDGEMENT - MariaDB Slave Lag: s1 on db1047 is CRITICAL: CRITICAL slave_sql_lag Replication lag: 9649.85 seconds Jcrespo long running update collation script - The acknowledgement expires at: 2016-09-05 09:00:00.
[09:10:58] <icinga-wm>	 RECOVERY - puppet last run on pc2004 is OK: OK: Puppet is currently enabled, last run 10 seconds ago with 0 failures
[09:12:27] <icinga-wm>	 RECOVERY - MD RAID on relforge1001 is OK: OK: Active: 8, Working: 8, Failed: 0, Spare: 0
[09:27:12] <icinga-wm>	 PROBLEM - MD RAID on relforge1001 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds.
[09:29:52] <icinga-wm>	 RECOVERY - MD RAID on relforge1001 is OK: OK: Active: 8, Working: 8, Failed: 0, Spare: 0
[09:36:05] <icinga-wm>	 PROBLEM - puppet last run on ms-be2024 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[10:03:34] <icinga-wm>	 RECOVERY - puppet last run on ms-be2024 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[10:19:46] <grrrit-wm>	 (03PS2) 10MarcoAurelio: Rename 'technican' and 'technician' to 'editinterface' [mediawiki-config] - 10https://gerrit.wikimedia.org/r/308281 (https://phabricator.wikimedia.org/T144638) 
[10:20:29] <grrrit-wm>	 (03CR) 10MarcoAurelio: "> Let's wait a little bit, some discussions offer to distinguish two" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/308281 (https://phabricator.wikimedia.org/T144638) (owner: 10MarcoAurelio)
[10:22:42] <grrrit-wm>	 (03PS3) 10MarcoAurelio: Rename 'technican' and 'technician' to 'editinterface' [mediawiki-config] - 10https://gerrit.wikimedia.org/r/308281 (https://phabricator.wikimedia.org/T144638) 
[10:24:46] <grrrit-wm>	 (03PS4) 10MarcoAurelio: Rename 'technican' and 'technician' to 'editinterface' [mediawiki-config] - 10https://gerrit.wikimedia.org/r/308281 (https://phabricator.wikimedia.org/T144638) 
[10:34:12] <grrrit-wm>	 (03PS4) 10MarcoAurelio: Remove upload rights on wikis where local uploads are disabled [mediawiki-config] - 10https://gerrit.wikimedia.org/r/306443 (https://phabricator.wikimedia.org/T143789) 
[10:34:24] <grrrit-wm>	 (03CR) 10jenkins-bot: [V: 04-1] Remove upload rights on wikis where local uploads are disabled [mediawiki-config] - 10https://gerrit.wikimedia.org/r/306443 (https://phabricator.wikimedia.org/T143789) (owner: 10MarcoAurelio)
[10:40:42] <grrrit-wm>	 (03PS5) 10MarcoAurelio: Remove upload rights on wikis where local uploads are disabled [mediawiki-config] - 10https://gerrit.wikimedia.org/r/306443 (https://phabricator.wikimedia.org/T143789) 
[10:44:40] <grrrit-wm>	 (03PS6) 10MarcoAurelio: Remove upload rights on wikis where local uploads are disabled [mediawiki-config] - 10https://gerrit.wikimedia.org/r/306443 (https://phabricator.wikimedia.org/T143789) 
[10:47:49] <grrrit-wm>	 (03PS5) 10MarcoAurelio: Rename 'technican' and 'technician' to 'interface-editor' [mediawiki-config] - 10https://gerrit.wikimedia.org/r/308281 (https://phabricator.wikimedia.org/T144638) 
[10:49:31] <grrrit-wm>	 (03PS7) 10MarcoAurelio: Remove upload rights on wikis where local uploads are disabled [mediawiki-config] - 10https://gerrit.wikimedia.org/r/306443 (https://phabricator.wikimedia.org/T143789) 
[10:50:43] <grrrit-wm>	 (03CR) 10MarcoAurelio: Remove upload rights on wikis where local uploads are disabled (031 comment) [mediawiki-config] - 10https://gerrit.wikimedia.org/r/306443 (https://phabricator.wikimedia.org/T143789) (owner: 10MarcoAurelio)
[10:54:04] <grrrit-wm>	 (03PS8) 10MarcoAurelio: Remove upload rights on wikis where local uploads are disabled [mediawiki-config] - 10https://gerrit.wikimedia.org/r/306443 (https://phabricator.wikimedia.org/T143789) 
[11:27:48] <grrrit-wm>	 (03PS1) 10Urbanecm: Add source wikis in import page in sawiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/308429 (https://phabricator.wikimedia.org/T133483) 
[11:37:08] <grrrit-wm>	 (03PS1) 10Urbanecm: Change $wgArticleCountMethod in Wikidata from default ('link') to 'any' [mediawiki-config] - 10https://gerrit.wikimedia.org/r/308430 (https://phabricator.wikimedia.org/T144687) 
[11:50:08] <grrrit-wm>	 (03PS2) 10Urbanecm: Add source wikis in import page in sawiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/308429 (https://phabricator.wikimedia.org/T133483) 
[11:58:54] <grrrit-wm>	 (03PS3) 10Urbanecm: Add source wikis in import page in sawiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/308429 (https://phabricator.wikimedia.org/T133483) 
[12:05:47] <grrrit-wm>	 (03CR) 10NehalDaveND: [C: 031] "this addition fro sa.wiki is very well done by Urbanecm" (031 comment) [mediawiki-config] - 10https://gerrit.wikimedia.org/r/308429 (https://phabricator.wikimedia.org/T133483) (owner: 10Urbanecm)
[12:20:10] <grrrit-wm>	 (03CR) 10Sbgujarat: [C: 031] "I also seen this work. Thank you. Please proceed for merge." [mediawiki-config] - 10https://gerrit.wikimedia.org/r/308429 (https://phabricator.wikimedia.org/T133483) (owner: 10Urbanecm)
[12:22:25] <grrrit-wm>	 (03CR) 10Urbanecm: "Will be processed during next Morning SWAT window (there are three SWAT windows in each work week, except Fridays). So this change will be" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/308429 (https://phabricator.wikimedia.org/T133483) (owner: 10Urbanecm)
[12:23:01] <grrrit-wm>	 (03CR) 10Urbanecm: "But you're welcome :)." [mediawiki-config] - 10https://gerrit.wikimedia.org/r/308429 (https://phabricator.wikimedia.org/T133483) (owner: 10Urbanecm)
[12:59:27] <icinga-wm>	 PROBLEM - puppet last run on db2050 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[13:24:34] <icinga-wm>	 RECOVERY - puppet last run on db2050 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures
[13:25:26] <grrrit-wm>	 (03PS1) 10Giuseppe Lavagetto: role::rcstream: move redis settings from legacy config file [puppet] - 10https://gerrit.wikimedia.org/r/308432 (https://phabricator.wikimedia.org/T134400) 
[13:39:43] <wikibugs>	 06Operations, 10Traffic, 07Browser-Support-Internet-Explorer, 07HTTPS: Internet Explorer 6 can not reach https://*.wikipedia.org - https://phabricator.wikimedia.org/T143539#2607566 (10BBlack) Yes, we're not going backwards here and re-enabling any kind of fallback to SSLv3 or SHA-1 certs.  The redirect bei...
[14:22:39] <grrrit-wm>	 (03PS2) 10Alexandros Kosiaris: Add SRV records for puppet (all pointing to palladium.eqiad.wmnet) [dns] - 10https://gerrit.wikimedia.org/r/306642 (https://phabricator.wikimedia.org/T143869) (owner: 10Giuseppe Lavagetto)
[14:23:32] <grrrit-wm>	 (03PS1) 10BBlack: ssl_ciphersuite: remove DHE+chapoly [puppet] - 10https://gerrit.wikimedia.org/r/308433 
[14:23:34] <grrrit-wm>	 (03PS1) 10BBlack: ssl_ciphersuite: remove SHA256 "mid" options [puppet] - 10https://gerrit.wikimedia.org/r/308434 
[14:27:46] <grrrit-wm>	 (03PS3) 10Alexandros Kosiaris: Add SRV records for puppet (all pointing to palladium.eqiad.wmnet) [dns] - 10https://gerrit.wikimedia.org/r/306642 (https://phabricator.wikimedia.org/T143869) (owner: 10Giuseppe Lavagetto)
[14:29:21] <grrrit-wm>	 (03CR) 10Alexandros Kosiaris: [C: 032 V: 032] "Added wikimedia.org as well and merging" [dns] - 10https://gerrit.wikimedia.org/r/306642 (https://phabricator.wikimedia.org/T143869) (owner: 10Giuseppe Lavagetto)
[14:35:07] <grrrit-wm>	 (03PS2) 10BBlack: ssl_ciphersuite: remove SHA256 "mid" options [puppet] - 10https://gerrit.wikimedia.org/r/308434 
[14:40:36] <grrrit-wm>	 (03PS3) 10BBlack: ssl_ciphersuite: remove SHA256 "mid" options [puppet] - 10https://gerrit.wikimedia.org/r/308434 
[15:02:46] <grrrit-wm>	 (03PS3) 10Alexandros Kosiaris: site.pp: Use role keyword as a function [puppet] - 10https://gerrit.wikimedia.org/r/308288 
[15:04:49] <grrrit-wm>	 (03CR) 10Alexandros Kosiaris: [C: 032 V: 032] "PCC is happy at https://puppet-compiler.wmflabs.org/3929/ for 309 hosts. The non-compiling ones is due to labs/private missing data. I am " [puppet] - 10https://gerrit.wikimedia.org/r/308288 (owner: 10Alexandros Kosiaris)
[15:05:06] <grrrit-wm>	 (03PS1) 10Merlijn van Deen: toolserver: Redirect ~mzmcbride/yanker/ to tool labs [puppet] - 10https://gerrit.wikimedia.org/r/308435 (https://phabricator.wikimedia.org/T136924) 
[15:13:07] <grrrit-wm>	 (03CR) 10Alexandros Kosiaris: [C: 032 V: 032] "PCC is pretty happy at https://puppet-compiler.wmflabs.org/3927/, so merging" [puppet] - 10https://gerrit.wikimedia.org/r/307236 (https://phabricator.wikimedia.org/T143869) (owner: 10Giuseppe Lavagetto)
[15:13:13] <grrrit-wm>	 (03PS5) 10Alexandros Kosiaris: puppetmasters: support multiple git masters [puppet] - 10https://gerrit.wikimedia.org/r/307236 (https://phabricator.wikimedia.org/T143869) (owner: 10Giuseppe Lavagetto)
[15:13:16] <grrrit-wm>	 (03CR) 10Alexandros Kosiaris: [V: 032] puppetmasters: support multiple git masters [puppet] - 10https://gerrit.wikimedia.org/r/307236 (https://phabricator.wikimedia.org/T143869) (owner: 10Giuseppe Lavagetto)
[15:20:29] <grrrit-wm>	 (03PS1) 10Alexandros Kosiaris: Fix wrong dependency in puppetmaster::gitclone [puppet] - 10https://gerrit.wikimedia.org/r/308436 
[15:20:53] <grrrit-wm>	 (03CR) 10Alexandros Kosiaris: [C: 032 V: 032] Fix wrong dependency in puppetmaster::gitclone [puppet] - 10https://gerrit.wikimedia.org/r/308436 (owner: 10Alexandros Kosiaris)
[15:22:56] <icinga-wm>	 PROBLEM - puppet last run on palladium is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[15:25:18] <icinga-wm>	 RECOVERY - puppet last run on palladium is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[15:33:47] <grrrit-wm>	 (03PS1) 10Alexandros Kosiaris: puppetmaster: Pedantic whitespace fixes [puppet] - 10https://gerrit.wikimedia.org/r/308437 
[15:38:59] <grrrit-wm>	 (03PS2) 10Alexandros Kosiaris: puppetmaster: Pedantic whitespace fixes [puppet] - 10https://gerrit.wikimedia.org/r/308437 
[15:41:17] <grrrit-wm>	 (03PS1) 10Merlijn van Deen: toollabs: install korean locales [puppet] - 10https://gerrit.wikimedia.org/r/308439 (https://phabricator.wikimedia.org/T130532) 
[15:46:16] <icinga-wm>	 PROBLEM - DPKG on puppetmaster2001 is CRITICAL: DPKG CRITICAL dpkg reports broken packages
[15:46:44] <grrrit-wm>	 (03Draft1) 10Paladox: manifests/role: Fix: class not documented [puppet] - 10https://gerrit.wikimedia.org/r/308440 
[15:47:14] <grrrit-wm>	 (03PS2) 10Paladox: manifests/role: Fix: class not documented [puppet] - 10https://gerrit.wikimedia.org/r/308440 (https://phabricator.wikimedia.org/T93645) 
[15:48:23] <grrrit-wm>	 (03PS3) 10Paladox: manifests/role: Fix: class not documented [puppet] - 10https://gerrit.wikimedia.org/r/308440 (https://phabricator.wikimedia.org/T93645) 
[15:49:15] <icinga-wm>	 PROBLEM - HP RAID on ms-be1024 is CRITICAL: CHECK_NRPE: Socket timeout after 40 seconds.
[15:49:23] <grrrit-wm>	 (03CR) 10Merlijn van Deen: [C: 04-1] "This documentation just repeats what is already stated in the code." [puppet] - 10https://gerrit.wikimedia.org/r/308440 (https://phabricator.wikimedia.org/T93645) (owner: 10Paladox)
[15:50:03] <grrrit-wm>	 (03CR) 10Paladox: "Yes but it fails with" [puppet] - 10https://gerrit.wikimedia.org/r/308440 (https://phabricator.wikimedia.org/T93645) (owner: 10Paladox)
[15:50:50] <grrrit-wm>	 (03CR) 10Merlijn van Deen: "So we either add useful documentation, or we tell Jenkins to ignore the warning. Adding useless documentation is not helpful." [puppet] - 10https://gerrit.wikimedia.org/r/308440 (https://phabricator.wikimedia.org/T93645) (owner: 10Paladox)
[15:51:55] <grrrit-wm>	 (03CR) 10Paladox: "Jenkins all ready ignores it, but we are trying to fix puppet-lint, see https://phabricator.wikimedia.org/T93645 please" [puppet] - 10https://gerrit.wikimedia.org/r/308440 (https://phabricator.wikimedia.org/T93645) (owner: 10Paladox)
[15:53:43] <icinga-wm>	 RECOVERY - DPKG on puppetmaster2001 is OK: All packages OK
[15:54:36] <valhallasw`cloud>	 paladox: my statement stands.
[15:55:00] <paladox>	 oh
[15:55:17] <paladox>	 I have no idea what the description will be
[15:55:29] <valhallasw`cloud>	 then don't create the patchsets
[15:55:42] <paladox>	 Why? I doint see anyone else fixing the problem?
[15:55:56] <valhallasw`cloud>	 because the problem is not that there is a warning
[15:56:02] <valhallasw`cloud>	 the problem is that useful documentation is missing
[15:56:07] <paladox>	 Yep
[15:56:12] <valhallasw`cloud>	 so you're not fixing the problem, you are merely hiding it
[15:56:24] <paladox>	 https://github.com/wikimedia/operations-puppet/blob/production/.puppet-lint.rc#L8
[15:56:39] <paladox>	 valhallasw`cloud ^^ it shows you only need one comment
[15:56:47] <paladox>	 # T127797 - needs at least one comment line for each class
[15:56:48] <stashbot>	 T127797: document all puppet classes / defined types!? - https://phabricator.wikimedia.org/T127797
[15:58:26] <grrrit-wm>	 (03PS3) 10Alexandros Kosiaris: puppetmaster: Pedantic whitespace fixes [puppet] - 10https://gerrit.wikimedia.org/r/308437 
[15:58:39] <valhallasw`cloud>	 paladox: it needs a *useful descriptive comment* that is at least one line
[15:58:55] <paladox>	 oh but it dosent say that on ^^
[15:59:01] <valhallasw`cloud>	 "needs at least one comment line for each class" just describes what --no-documentation-check does
[15:59:02] <paladox>	 it only says needs one comment
[15:59:06] <valhallasw`cloud>	 yes, it does
[15:59:08] <valhallasw`cloud>	 "please insert at least one comment line right before every class to describe what it does"
[15:59:16] <icinga-wm>	 RECOVERY - HP RAID on ms-be1024 is OK: OK: Slot 3: OK: 2I:4:1, 2I:4:2, 1I:1:5, 1I:1:6, 1I:1:7, 1I:1:8, 1I:1:1, 1I:1:2, 1I:1:3, 1I:1:4, 2I:2:1, 2I:2:2, 2I:2:3, 2I:2:4, Controller, Battery/Capacitor
[15:59:21] <paladox>	 Yeh that does
[15:59:23] <paladox>	 but not needs at least one comment line for each class
[15:59:30] <paladox>	 https://github.com/wikimedia/operations-puppet/blob/production/.puppet-lint.rc#L8
[15:59:32] <valhallasw`cloud>	 and even if it didn't say so literally, this is what is implied
[15:59:42] <paladox>	 ok
[16:07:23] <grrrit-wm>	 (03PS4) 10Alexandros Kosiaris: puppetmaster: Pedantic whitespace fixes [puppet] - 10https://gerrit.wikimedia.org/r/308437 
[16:08:35] <icinga-wm>	 PROBLEM - puppetmaster backend https on puppetmaster2001 is CRITICAL: Connection refused
[16:08:39] <grrrit-wm>	 (03CR) 10Giuseppe Lavagetto: [C: 04-1] "I don't see any value in fake documentation just for the sake of linting. See what Merlijn said." [puppet] - 10https://gerrit.wikimedia.org/r/308440 (https://phabricator.wikimedia.org/T93645) (owner: 10Paladox)
[16:08:54] <icinga-wm>	 PROBLEM - puppetmaster https on puppetmaster2001 is CRITICAL: Connection refused
[16:09:02] <grrrit-wm>	 (03Abandoned) 10Paladox: manifests/role: Fix: class not documented [puppet] - 10https://gerrit.wikimedia.org/r/308440 (https://phabricator.wikimedia.org/T93645) (owner: 10Paladox)
[16:15:00] <grrrit-wm>	 (03Draft2) 10Paladox: role: Fix not in autoload module layout [puppet] - 10https://gerrit.wikimedia.org/r/308441 (https://phabricator.wikimedia.org/T93645) 
[16:15:05] <grrrit-wm>	 (03Draft1) 10Paladox: role: Fix not in autoload module layout [puppet] - 10https://gerrit.wikimedia.org/r/308441 (https://phabricator.wikimedia.org/T93645) 
[16:16:13] <grrrit-wm>	 (03PS3) 10Paladox: role: Fix not in autoload module layout [puppet] - 10https://gerrit.wikimedia.org/r/308441 (https://phabricator.wikimedia.org/T93645) 
[16:17:41] <grrrit-wm>	 (03PS5) 10Alexandros Kosiaris: puppetmaster: Pedantic whitespace fixes [puppet] - 10https://gerrit.wikimedia.org/r/308437 
[16:19:05] <icinga-wm>	 PROBLEM - HP RAID on ms-be1024 is CRITICAL: CHECK_NRPE: Socket timeout after 40 seconds.
[16:19:56] <grrrit-wm>	 (03PS4) 10Paladox: role: Fix not in autoload module layout [puppet] - 10https://gerrit.wikimedia.org/r/308441 (https://phabricator.wikimedia.org/T93645) 
[16:21:20] <grrrit-wm>	 (03CR) 10Paladox: "One of the fixes was done here https://gerrit.wikimedia.org/r/#/c/308348/" [puppet] - 10https://gerrit.wikimedia.org/r/308437 (owner: 10Alexandros Kosiaris)
[16:21:32] <grrrit-wm>	 (03PS3) 10Paladox: puppetmaster: Fix trailing whitespace found [puppet] - 10https://gerrit.wikimedia.org/r/308348 (https://phabricator.wikimedia.org/T93645) 
[16:21:47] <grrrit-wm>	 (03PS4) 10Paladox: puppetmaster: Fix indentation of => [puppet] - 10https://gerrit.wikimedia.org/r/308347 (https://phabricator.wikimedia.org/T93645) 
[16:23:44] <grrrit-wm>	 (03PS6) 10Alexandros Kosiaris: puppetmaster: Pedantic whitespace fixes [puppet] - 10https://gerrit.wikimedia.org/r/308437 
[16:23:46] <grrrit-wm>	 (03PS1) 10Alexandros Kosiaris: puppetmaster: Vary ssldir in frontend on masterness [puppet] - 10https://gerrit.wikimedia.org/r/308442 
[16:24:43] <grrrit-wm>	 (03PS5) 10Paladox: role: Fix not in autoload module layout [puppet] - 10https://gerrit.wikimedia.org/r/308441 (https://phabricator.wikimedia.org/T93645) 
[16:26:18] <icinga-wm>	 RECOVERY - HP RAID on ms-be1024 is OK: OK: Slot 3: OK: 2I:4:1, 2I:4:2, 1I:1:5, 1I:1:6, 1I:1:7, 1I:1:8, 1I:1:1, 1I:1:2, 1I:1:3, 1I:1:4, 2I:2:1, 2I:2:2, 2I:2:3, 2I:2:4, Controller, Battery/Capacitor
[16:27:05] <grrrit-wm>	 (03PS2) 10Alexandros Kosiaris: puppetmaster: Vary ssldir in frontend on masterness [puppet] - 10https://gerrit.wikimedia.org/r/308442 
[16:27:52] <grrrit-wm>	 (03PS6) 10Paladox: role: Fix not in autoload module layout [puppet] - 10https://gerrit.wikimedia.org/r/308441 (https://phabricator.wikimedia.org/T93645) 
[16:28:55] <grrrit-wm>	 (03CR) 10jenkins-bot: [V: 04-1] role: Fix not in autoload module layout [puppet] - 10https://gerrit.wikimedia.org/r/308441 (https://phabricator.wikimedia.org/T93645) (owner: 10Paladox)
[16:31:45] <grrrit-wm>	 (03PS3) 10Alexandros Kosiaris: puppetmaster: Vary ssldir in frontend on masterness [puppet] - 10https://gerrit.wikimedia.org/r/308442 
[16:34:20] <grrrit-wm>	 (03CR) 10Alexandros Kosiaris: [C: 032] puppetmaster: Pedantic whitespace fixes [puppet] - 10https://gerrit.wikimedia.org/r/308437 (owner: 10Alexandros Kosiaris)
[16:34:23] <grrrit-wm>	 (03PS17) 10Alex Monk: Add python version of maintain-replicas script [software] - 10https://gerrit.wikimedia.org/r/295607 (https://phabricator.wikimedia.org/T138450) 
[16:34:27] <grrrit-wm>	 (03CR) 10Alexandros Kosiaris: [C: 032] puppetmaster: Vary ssldir in frontend on masterness [puppet] - 10https://gerrit.wikimedia.org/r/308442 (owner: 10Alexandros Kosiaris)
[16:35:12] <grrrit-wm>	 (03PS7) 10Paladox: role: Fix not in autoload module layout [puppet] - 10https://gerrit.wikimedia.org/r/308441 (https://phabricator.wikimedia.org/T93645) 
[16:36:05] <grrrit-wm>	 (03PS4) 10Paladox: puppetmaster: Fix trailing whitespace found [puppet] - 10https://gerrit.wikimedia.org/r/308348 (https://phabricator.wikimedia.org/T93645) 
[16:36:17] <grrrit-wm>	 (03CR) 10jenkins-bot: [V: 04-1] role: Fix not in autoload module layout [puppet] - 10https://gerrit.wikimedia.org/r/308441 (https://phabricator.wikimedia.org/T93645) (owner: 10Paladox)
[16:37:12] <grrrit-wm>	 (03PS8) 10Paladox: role: Fix not in autoload module layout [puppet] - 10https://gerrit.wikimedia.org/r/308441 (https://phabricator.wikimedia.org/T93645) 
[16:37:31] <grrrit-wm>	 (03PS9) 10Paladox: role: Fix not in autoload module layout [puppet] - 10https://gerrit.wikimedia.org/r/308441 (https://phabricator.wikimedia.org/T93645) 
[16:39:00] <grrrit-wm>	 (03PS18) 10Alex Monk: Add python version of maintain-replicas script [software] - 10https://gerrit.wikimedia.org/r/295607 (https://phabricator.wikimedia.org/T138450) 
[16:42:02] <grrrit-wm>	 (03CR) 10Paladox: "Please note that I updated https://gerrit.wikimedia.org/r/#/c/308441/9/modules/role/manifests/coredb/common.pp to include :: on $::topolog" [puppet] - 10https://gerrit.wikimedia.org/r/308441 (https://phabricator.wikimedia.org/T93645) (owner: 10Paladox)
[16:57:36] <grrrit-wm>	 (03PS19) 10Alex Monk: Add python version of maintain-replicas script [software] - 10https://gerrit.wikimedia.org/r/295607 (https://phabricator.wikimedia.org/T138450) 
[16:59:30] <grrrit-wm>	 (03CR) 10Alex Monk: "over to you again Volans" [software] - 10https://gerrit.wikimedia.org/r/295607 (https://phabricator.wikimedia.org/T138450) (owner: 10Alex Monk)
[17:16:05] <lslsksksisjusdj>	 ALLALALLALALALALALLA
[17:16:11] <lslsksksisjusdj>	 VETE ALA MIERFA
[17:16:13] <lslsksksisjusdj>	 ALVARO 
[17:17:42] <icinga-wm>	 PROBLEM - HP RAID on ms-be1024 is CRITICAL: CHECK_NRPE: Socket timeout after 40 seconds.
[17:18:58] <icinga-wm>	 PROBLEM - HP RAID on ms-be1026 is CRITICAL: CHECK_NRPE: Socket timeout after 40 seconds.
[17:20:08] <icinga-wm>	 RECOVERY - HP RAID on ms-be1024 is OK: OK: Slot 3: OK: 2I:4:1, 2I:4:2, 1I:1:5, 1I:1:6, 1I:1:7, 1I:1:8, 1I:1:1, 1I:1:2, 1I:1:3, 1I:1:4, 2I:2:1, 2I:2:2, 2I:2:3, 2I:2:4, Controller, Battery/Capacitor
[17:21:22] <icinga-wm>	 RECOVERY - HP RAID on ms-be1026 is OK: OK: Slot 3: OK: 2I:4:1, 2I:4:2, 1I:1:5, 1I:1:6, 1I:1:7, 1I:1:8, 1I:1:1, 1I:1:2, 1I:1:3, 1I:1:4, 2I:2:1, 2I:2:2, 2I:2:3, 2I:2:4, Controller, Battery/Capacitor
[17:52:49] <icinga-wm>	 PROBLEM - HP RAID on ms-be1025 is CRITICAL: CHECK_NRPE: Socket timeout after 40 seconds.
[18:02:49] <icinga-wm>	 RECOVERY - HP RAID on ms-be1025 is OK: OK: Slot 3: OK: 2I:4:1, 2I:4:2, 1I:1:5, 1I:1:6, 1I:1:7, 1I:1:8, 1I:1:1, 1I:1:2, 1I:1:3, 1I:1:4, 2I:2:1, 2I:2:2, 2I:2:3, 2I:2:4, Controller, Battery/Capacitor
[18:09:00] <icinga-wm>	 PROBLEM - puppet last run on mw2121 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 5 minutes ago with 1 failures. Failed resources (up to 3 shown): File[/etc/update-motd.d/97-last-puppet-run]
[18:16:28] <icinga-wm>	 PROBLEM - Check size of conntrack table on rhodium is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds.
[18:16:53] <icinga-wm>	 PROBLEM - MD RAID on rhodium is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds.
[18:17:47] <icinga-wm>	 PROBLEM - puppet last run on elastic2022 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 4 minutes ago with 1 failures. Failed resources (up to 3 shown): File[/usr/local/bin/es-tool]
[18:18:19] <icinga-wm>	 PROBLEM - puppet last run on rdb1003 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 5 minutes ago with 1 failures. Failed resources (up to 3 shown): File[/usr/lib/nagios/plugins/check-fresh-files-in-dir.py]
[18:18:38] <icinga-wm>	 PROBLEM - DPKG on rhodium is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds.
[18:18:40] <icinga-wm>	 PROBLEM - puppet last run on titanium is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 5 minutes ago with 1 failures. Failed resources (up to 3 shown): File[/usr/lib/nagios/plugins/check-fresh-files-in-dir.py]
[18:18:48] <icinga-wm>	 PROBLEM - puppet last run on rdb2002 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 6 minutes ago with 1 failures. Failed resources (up to 3 shown): File[/usr/local/bin/phaste]
[18:18:48] <icinga-wm>	 PROBLEM - puppet last run on mw1280 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 5 minutes ago with 1 failures. Failed resources (up to 3 shown): File[/etc/apache2/mods-available/setenvif.conf]
[18:19:57] <icinga-wm>	 PROBLEM - puppet last run on mw2225 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 6 minutes ago with 1 failures. Failed resources (up to 3 shown): File[/etc/apache2/mods-available/setenvif.conf]
[18:20:05] <icinga-wm>	 PROBLEM - puppetmaster https on palladium is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[18:20:27] <icinga-wm>	 PROBLEM - puppet last run on rhodium is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds.
[18:21:27] <icinga-wm>	 PROBLEM - Disk space on rhodium is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds.
[18:21:40] <icinga-wm>	 PROBLEM - puppet last run on analytics1003 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:23:09] <icinga-wm>	 PROBLEM - puppet last run on mw2117 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:23:19] <icinga-wm>	 PROBLEM - puppet last run on mw2100 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:23:47] <icinga-wm>	 RECOVERY - Disk space on rhodium is OK: DISK OK
[18:23:54] <grrrit-wm>	 (03PS1) 10MarcoAurelio: Rename 'autopatrol' to 'autopatrolled' on fawiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/308446 (https://phabricator.wikimedia.org/T144699) 
[18:24:38] <icinga-wm>	 PROBLEM - configured eth on rhodium is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds.
[18:24:55] <icinga-wm>	 PROBLEM - puppet last run on mw1267 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:26:07] <icinga-wm>	 PROBLEM - puppet last run on mw2090 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:26:59] <icinga-wm>	 RECOVERY - configured eth on rhodium is OK: OK - interfaces up
[18:27:47] <icinga-wm>	 PROBLEM - SSH on rhodium is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[18:27:57] <icinga-wm>	 PROBLEM - puppetmaster backend https on rhodium is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[18:29:18] <icinga-wm>	 PROBLEM - puppet last run on cp4005 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:29:57] <icinga-wm>	 PROBLEM - puppet last run on mw1235 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:30:18] <icinga-wm>	 PROBLEM - puppet last run on conf2002 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:30:40] <icinga-wm>	 RECOVERY - DPKG on rhodium is OK: All packages OK
[18:30:57] <icinga-wm>	 RECOVERY - Check size of conntrack table on rhodium is OK: OK: nf_conntrack is 0 % full
[18:31:59] <icinga-wm>	 PROBLEM - puppet last run on mw2203 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:31:59] <icinga-wm>	 PROBLEM - Unmerged changes on repository puppet on rhodium is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds.
[18:32:12] <icinga-wm>	 PROBLEM - puppet last run on mw1206 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:32:47] <icinga-wm>	 PROBLEM - puppet last run on labvirt1009 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:33:18] <icinga-wm>	 PROBLEM - puppet last run on cp2019 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:33:37] <icinga-wm>	 PROBLEM - puppet last run on mw1211 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:34:28] <icinga-wm>	 PROBLEM - configured eth on rhodium is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds.
[18:35:07] <icinga-wm>	 PROBLEM - puppet last run on wtp1013 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:35:17] <icinga-wm>	 PROBLEM - puppet last run on ms-be2008 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:35:37] <icinga-wm>	 PROBLEM - puppet last run on cp3045 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:35:40] <icinga-wm>	 PROBLEM - puppet last run on mw2167 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:35:47] <icinga-wm>	 PROBLEM - puppet last run on aluminium is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:35:47] <icinga-wm>	 PROBLEM - puppet last run on es2016 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:35:48] <icinga-wm>	 PROBLEM - puppet last run on ms-fe3001 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:35:58] <icinga-wm>	 PROBLEM - puppet last run on ms-be1004 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:35:58] <icinga-wm>	 PROBLEM - puppet last run on ms-be1003 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:35:59] <icinga-wm>	 PROBLEM - puppet last run on db1093 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:35:59] <icinga-wm>	 PROBLEM - puppet last run on db1081 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:36:00] <icinga-wm>	 PROBLEM - puppet last run on lvs2003 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:36:11] <icinga-wm>	 PROBLEM - puppet last run on mc1011 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:36:16] <icinga-wm>	 PROBLEM - puppet last run on mc1016 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:36:17] <icinga-wm>	 RECOVERY - MD RAID on rhodium is OK: OK: Active: 4, Working: 4, Failed: 0, Spare: 0
[18:36:27] <icinga-wm>	 PROBLEM - puppet last run on cp2017 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:36:37] <icinga-wm>	 PROBLEM - puppet last run on lvs4002 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:36:47] <icinga-wm>	 PROBLEM - puppet last run on db1070 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:36:58] <icinga-wm>	 PROBLEM - puppet last run on mw2152 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:37:01] <icinga-wm>	 PROBLEM - puppet last run on pc1004 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:37:02] <icinga-wm>	 PROBLEM - puppet last run on zosma is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:37:02] <valhallasw`cloud>	 akosiaris: ^ 
[18:37:04] <icinga-wm>	 PROBLEM - puppet last run on cp4009 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:37:06] <icinga-wm>	 PROBLEM - puppet last run on krypton is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:37:07] <icinga-wm>	 PROBLEM - puppet last run on mw1270 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:37:08] <icinga-wm>	 PROBLEM - dhclient process on rhodium is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds.
[18:37:22] <icinga-wm>	 PROBLEM - puppet last run on neon is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:37:22] <icinga-wm>	 PROBLEM - puppet last run on maps-test2003 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:37:27] <icinga-wm>	 PROBLEM - puppet last run on maerlant is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:37:38] <icinga-wm>	 PROBLEM - puppet last run on pc2006 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:37:38] <icinga-wm>	 PROBLEM - puppet last run on mc1006 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:37:38] <icinga-wm>	 PROBLEM - puppet last run on conf2001 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:37:38] <icinga-wm>	 PROBLEM - puppet last run on es2012 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:37:39] <icinga-wm>	 PROBLEM - puppet last run on mw2231 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:37:39] <icinga-wm>	 PROBLEM - puppet last run on mw2091 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:37:39] <icinga-wm>	 PROBLEM - puppet last run on labstore2001 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:37:51] <icinga-wm>	 PROBLEM - puppet last run on mc2004 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:37:51] <icinga-wm>	 PROBLEM - puppet last run on kafka1018 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:37:51] <icinga-wm>	 PROBLEM - puppet last run on wdqs1002 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:37:53] <icinga-wm>	 PROBLEM - puppet last run on restbase1013 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:37:57] <icinga-wm>	 PROBLEM - puppet last run on ms-be1001 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:38:07] <icinga-wm>	 PROBLEM - puppet last run on db2037 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:38:09] <icinga-wm>	 PROBLEM - puppet last run on ms-be2014 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:38:17] <icinga-wm>	 PROBLEM - puppet last run on mw1262 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:38:18] <icinga-wm>	 PROBLEM - puppet last run on aqs1005 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:38:27] <icinga-wm>	 PROBLEM - puppet last run on mc1013 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:38:27] <icinga-wm>	 PROBLEM - puppet last run on db2041 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:38:29] <icinga-wm>	 PROBLEM - Check size of conntrack table on rhodium is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds.
[18:38:29] <icinga-wm>	 PROBLEM - puppet last run on stat1002 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:38:30] <icinga-wm>	 PROBLEM - puppet last run on ms-be1002 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:38:38] <icinga-wm>	 PROBLEM - puppet last run on analytics1032 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:38:46] <icinga-wm>	 PROBLEM - puppet last run on cp3039 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:38:58] <icinga-wm>	 PROBLEM - puppet last run on mw2068 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:38:58] <icinga-wm>	 PROBLEM - puppet last run on mw2174 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:39:07] <icinga-wm>	 PROBLEM - puppet last run on cp4006 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:39:07] <icinga-wm>	 PROBLEM - puppet last run on cp3030 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:39:18] <icinga-wm>	 PROBLEM - puppet last run on cp1047 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:39:18] <icinga-wm>	 RECOVERY - configured eth on rhodium is OK: OK - interfaces up
[18:39:28] <icinga-wm>	 PROBLEM - puppet last run on mw1181 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:39:33] <icinga-wm>	 RECOVERY - dhclient process on rhodium is OK: PROCS OK: 0 processes with command name dhclient
[18:39:47] <icinga-wm>	 PROBLEM - puppet last run on elastic1035 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:39:50] <icinga-wm>	 PROBLEM - puppet last run on cp3049 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:39:57] <icinga-wm>	 PROBLEM - puppet last run on mw2234 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:39:57] <icinga-wm>	 PROBLEM - puppet last run on prometheus2002 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:39:57] <icinga-wm>	 PROBLEM - puppet last run on cp4018 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:40:08] <icinga-wm>	 PROBLEM - puppet last run on db2058 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:40:21] <icinga-wm>	 PROBLEM - puppet last run on analytics1044 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:40:48] <icinga-wm>	 PROBLEM - puppet last run on mw2186 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:40:48] <icinga-wm>	 PROBLEM - puppet last run on ms-be1014 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:40:57] <icinga-wm>	 PROBLEM - puppet last run on cp1052 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:40:58] <icinga-wm>	 PROBLEM - puppet last run on ms-be2018 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:41:12] <icinga-wm>	 PROBLEM - puppet last run on cp2013 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:41:27] <icinga-wm>	 PROBLEM - puppet last run on ms-be2011 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:41:27] <icinga-wm>	 PROBLEM - puppet last run on mc2005 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:41:28] <icinga-wm>	 PROBLEM - puppet last run on mw2069 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:41:28] <icinga-wm>	 PROBLEM - puppet last run on ms-be1017 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:41:28] <icinga-wm>	 PROBLEM - puppet last run on ms-be2016 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:41:29] <icinga-wm>	 PROBLEM - puppet last run on ms-fe2002 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:41:38] <icinga-wm>	 PROBLEM - puppet last run on analytics1051 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:41:38] <icinga-wm>	 PROBLEM - puppet last run on hydrogen is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:41:39] <icinga-wm>	 PROBLEM - puppet last run on mw1289 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:41:48] <icinga-wm>	 PROBLEM - puppet last run on cp4020 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:41:50] <Luke081515>	 o.O
[18:42:18] <icinga-wm>	 PROBLEM - puppet last run on ms-be2017 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:42:18] <icinga-wm>	 PROBLEM - puppet last run on mw1231 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:42:19] <icinga-wm>	 PROBLEM - puppet last run on heze is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:42:37] <icinga-wm>	 PROBLEM - puppet last run on cp1061 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:42:38] <icinga-wm>	 PROBLEM - puppet last run on nihal is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:42:38] <icinga-wm>	 PROBLEM - puppet last run on mw1203 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:42:38] <icinga-wm>	 PROBLEM - puppet last run on mw2144 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:42:39] <icinga-wm>	 PROBLEM - puppet last run on mw2125 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:42:49] <icinga-wm>	 PROBLEM - puppet last run on mw1257 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:42:49] <icinga-wm>	 PROBLEM - puppet last run on ms-fe1003 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:42:49] <icinga-wm>	 PROBLEM - puppet last run on ms-be1005 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:42:59] <icinga-wm>	 PROBLEM - puppet last run on aqs1002 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:43:08] <icinga-wm>	 PROBLEM - puppet last run on cp1048 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:43:09] <icinga-wm>	 PROBLEM - puppet last run on analytics1047 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:43:17] <icinga-wm>	 PROBLEM - puppet last run on ms-be2023 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:43:18] <icinga-wm>	 PROBLEM - puppet last run on mw1185 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:43:18] <icinga-wm>	 PROBLEM - puppet last run on ms-be1019 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:43:18] <icinga-wm>	 RECOVERY - Check size of conntrack table on rhodium is OK: OK: nf_conntrack is 0 % full
[18:43:19] <icinga-wm>	 PROBLEM - puppet last run on ms-be1011 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:43:29] <icinga-wm>	 PROBLEM - puppet last run on aqs1001 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:43:29] <icinga-wm>	 PROBLEM - puppet last run on cp4003 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:43:37] <icinga-wm>	 PROBLEM - puppet last run on cp3037 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:43:42] <icinga-wm>	 PROBLEM - puppet last run on db1011 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:43:47] <icinga-wm>	 PROBLEM - MD RAID on rhodium is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds.
[18:43:47] <icinga-wm>	 PROBLEM - puppet last run on mw1261 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:43:47] <icinga-wm>	 PROBLEM - puppet last run on cp2006 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:43:47] <icinga-wm>	 PROBLEM - puppet last run on ms-be2015 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:43:47] <icinga-wm>	 PROBLEM - puppet last run on mw2208 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:43:47] <icinga-wm>	 PROBLEM - puppet last run on es2002 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:43:48] <icinga-wm>	 PROBLEM - puppet last run on mw1200 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:43:48] <icinga-wm>	 PROBLEM - puppet last run on logstash1005 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:43:57] <icinga-wm>	 PROBLEM - puppet last run on db2050 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:43:57] <icinga-wm>	 PROBLEM - puppet last run on mc2010 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:43:58] <icinga-wm>	 PROBLEM - puppet last run on elastic2021 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:43:58] <icinga-wm>	 PROBLEM - puppet last run on cp4008 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:43:58] <icinga-wm>	 PROBLEM - puppet last run on cp3038 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:43:58] <icinga-wm>	 PROBLEM - puppet last run on ms-be3004 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:43:59] <icinga-wm>	 PROBLEM - puppet last run on mw1233 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:44:07] <icinga-wm>	 PROBLEM - puppet last run on analytics1037 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:44:08] <icinga-wm>	 PROBLEM - puppet last run on francium is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:44:09] <icinga-wm>	 PROBLEM - puppet last run on ms-be1013 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:44:17] <icinga-wm>	 PROBLEM - puppet last run on db1057 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:44:17] <icinga-wm>	 PROBLEM - puppet last run on mw2179 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:44:18] <icinga-wm>	 PROBLEM - puppet last run on mc1003 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:44:33] <icinga-wm>	 PROBLEM - puppet last run on analytics1050 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:44:33] <icinga-wm>	 PROBLEM - puppet last run on mw1244 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:44:38] <icinga-wm>	 PROBLEM - puppet last run on db1056 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:44:48] <icinga-wm>	 PROBLEM - puppet last run on cp4004 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:44:48] <icinga-wm>	 PROBLEM - puppet last run on mw2122 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:44:48] <icinga-wm>	 PROBLEM - puppet last run on mw2209 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:44:48] <icinga-wm>	 PROBLEM - puppet last run on db1053 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:44:48] <icinga-wm>	 PROBLEM - puppet last run on mw1256 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:44:48] <icinga-wm>	 PROBLEM - puppet last run on mw1163 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:44:58] <icinga-wm>	 PROBLEM - puppet last run on relforge1001 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:44:59] <icinga-wm>	 PROBLEM - puppet last run on ms-fe2001 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:44:59] <icinga-wm>	 PROBLEM - puppet last run on mw1210 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:45:07] <icinga-wm>	 PROBLEM - puppet last run on elastic2020 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:45:07] <icinga-wm>	 PROBLEM - puppet last run on cp1050 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:45:07] <icinga-wm>	 PROBLEM - puppet last run on cp4011 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:45:08] <icinga-wm>	 PROBLEM - puppet last run on mw1178 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:45:08] <icinga-wm>	 PROBLEM - puppet last run on kafka1020 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:45:09] <icinga-wm>	 PROBLEM - puppet last run on mw1172 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:45:09] <icinga-wm>	 PROBLEM - puppet last run on mw2201 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:45:17] <icinga-wm>	 PROBLEM - puppet last run on analytics1001 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:45:18] <icinga-wm>	 PROBLEM - puppet last run on mw2145 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:45:18] <icinga-wm>	 PROBLEM - puppet last run on mw2157 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:45:19] <icinga-wm>	 PROBLEM - puppet last run on mw1215 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:45:28] <icinga-wm>	 PROBLEM - puppet last run on elastic2005 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:45:28] <icinga-wm>	 PROBLEM - puppet last run on mw2162 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:45:29] <icinga-wm>	 PROBLEM - puppet last run on mc2011 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:45:29] <icinga-wm>	 PROBLEM - puppet last run on cp1073 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:45:37] <icinga-wm>	 PROBLEM - puppet last run on mw2139 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:45:47] <icinga-wm>	 PROBLEM - puppet last run on mw2189 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:45:48] <icinga-wm>	 PROBLEM - puppet last run on mc1001 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:45:48] <icinga-wm>	 PROBLEM - puppet last run on mw1170 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:45:48] <icinga-wm>	 PROBLEM - puppet last run on mw1202 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:45:48] <icinga-wm>	 PROBLEM - puppet last run on db1030 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:45:48] <icinga-wm>	 PROBLEM - puppet last run on ms-be2022 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:45:49] <icinga-wm>	 PROBLEM - puppet last run on analytics1052 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:45:49] <icinga-wm>	 PROBLEM - puppet last run on mw2210 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:45:49] <icinga-wm>	 PROBLEM - puppet last run on analytics1040 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:45:58] <icinga-wm>	 PROBLEM - puppet last run on analytics1028 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:45:59] <icinga-wm>	 PROBLEM - puppet last run on elastic1020 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:46:01] <icinga-wm>	 PROBLEM - puppet last run on ms-be1020 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:46:07] <icinga-wm>	 RECOVERY - MD RAID on rhodium is OK: OK: Active: 4, Working: 4, Failed: 0, Spare: 0
[18:46:09] <icinga-wm>	 PROBLEM - puppet last run on ocg1003 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:46:09] <icinga-wm>	 PROBLEM - puppet last run on labsdb1007 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:46:09] <icinga-wm>	 PROBLEM - puppet last run on elastic2007 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:46:10] <icinga-wm>	 PROBLEM - puppet last run on mw1271 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:46:10] <icinga-wm>	 PROBLEM - puppet last run on ms-be2012 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:46:17] <icinga-wm>	 PROBLEM - puppet last run on cp2025 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:46:17] <icinga-wm>	 PROBLEM - puppet last run on mw2061 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:46:17] <icinga-wm>	 PROBLEM - puppet last run on ms-be2009 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:46:17] <icinga-wm>	 PROBLEM - puppet last run on wtp1022 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:46:17] <icinga-wm>	 PROBLEM - puppet last run on mw2161 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:46:18] <icinga-wm>	 PROBLEM - puppet last run on mw2120 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:46:18] <icinga-wm>	 PROBLEM - puppet last run on mw2133 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:46:19] <icinga-wm>	 PROBLEM - puppet last run on mw1249 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:46:19] <icinga-wm>	 PROBLEM - puppet last run on mw1174 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:46:20] <icinga-wm>	 PROBLEM - puppet last run on mw2227 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:46:20] <icinga-wm>	 PROBLEM - puppet last run on cp1067 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:46:21] <icinga-wm>	 PROBLEM - puppet last run on mw2065 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:46:21] <icinga-wm>	 PROBLEM - puppet last run on analytics1041 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:46:22] <icinga-wm>	 PROBLEM - puppet last run on labvirt1008 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:46:37] <icinga-wm>	 PROBLEM - puppet last run on restbase1011 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:46:37] <icinga-wm>	 PROBLEM - puppet last run on analytics1038 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:46:37] <icinga-wm>	 PROBLEM - puppet last run on db1044 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:46:37] <icinga-wm>	 PROBLEM - puppet last run on labvirt1013 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:46:38] <icinga-wm>	 PROBLEM - puppet last run on mw1228 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:46:38] <icinga-wm>	 PROBLEM - puppet last run on mw1171 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:46:38] <icinga-wm>	 PROBLEM - puppet last run on mw2160 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:46:39] <icinga-wm>	 PROBLEM - puppet last run on analytics1026 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:46:39] <icinga-wm>	 PROBLEM - puppet last run on elastic1017 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:46:40] <icinga-wm>	 PROBLEM - puppet last run on mw2177 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:46:40] <icinga-wm>	 PROBLEM - puppet last run on mw1176 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:46:47] <icinga-wm>	 PROBLEM - puppet last run on mw2243 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:46:51] <icinga-wm>	 PROBLEM - puppet last run on mw1246 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:46:51] <icinga-wm>	 PROBLEM - puppet last run on labvirt1004 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:46:56] <icinga-wm>	 PROBLEM - puppet last run on mw1201 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:46:58] <icinga-wm>	 PROBLEM - puppet last run on analytics1043 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:46:58] <icinga-wm>	 PROBLEM - dhclient process on rhodium is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds.
[18:47:08] <icinga-wm>	 PROBLEM - puppet last run on analytics1033 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:47:08] <icinga-wm>	 PROBLEM - puppet last run on db2049 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:47:09] <icinga-wm>	 PROBLEM - puppet last run on elastic2006 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:47:10] <icinga-wm>	 PROBLEM - puppet last run on mw2164 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:47:10] <icinga-wm>	 PROBLEM - puppet last run on mw2153 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:47:10] <icinga-wm>	 PROBLEM - puppet last run on mw2138 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:47:10] <icinga-wm>	 PROBLEM - puppet last run on db1045 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:47:17] <icinga-wm>	 PROBLEM - puppet last run on mw1222 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:47:17] <icinga-wm>	 PROBLEM - puppet last run on analytics1057 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:47:18] <icinga-wm>	 PROBLEM - puppet last run on cp2020 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:47:18] <icinga-wm>	 PROBLEM - puppet last run on suhail is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:47:18] <icinga-wm>	 PROBLEM - puppet last run on mw2240 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:47:18] <icinga-wm>	 PROBLEM - puppet last run on sca1002 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:47:18] <icinga-wm>	 PROBLEM - puppet last run on relforge1002 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:47:19] <icinga-wm>	 PROBLEM - puppet last run on mw1282 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:47:19] <icinga-wm>	 PROBLEM - puppet last run on mw1197 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:47:27] <icinga-wm>	 PROBLEM - puppet last run on mw2140 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:47:28] <icinga-wm>	 PROBLEM - puppet last run on ms-be2007 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:47:28] <icinga-wm>	 PROBLEM - puppet last run on elastic2015 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:47:28] <icinga-wm>	 PROBLEM - puppet last run on mw2080 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:47:28] <icinga-wm>	 PROBLEM - puppet last run on mw2221 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:47:28] <icinga-wm>	 PROBLEM - puppet last run on mw2170 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:47:28] <icinga-wm>	 PROBLEM - puppet last run on ms-be2003 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:47:29] <icinga-wm>	 PROBLEM - puppet last run on mc1007 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:47:29] <icinga-wm>	 PROBLEM - puppet last run on mw2071 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:47:30] <icinga-wm>	 PROBLEM - puppet last run on mw2105 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:47:37] <icinga-wm>	 PROBLEM - puppet last run on mw2235 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:47:37] <icinga-wm>	 PROBLEM - puppet last run on elastic2017 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:47:37] <icinga-wm>	 PROBLEM - puppet last run on dbstore2001 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:47:37] <icinga-wm>	 PROBLEM - puppet last run on mw1225 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:47:37] <icinga-wm>	 PROBLEM - puppet last run on mw1243 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:47:38] <icinga-wm>	 PROBLEM - puppet last run on analytics1039 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:47:38] <icinga-wm>	 PROBLEM - puppet last run on cp3021 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 6 minutes ago with 1 failures. Failed resources (up to 3 shown): File[/usr/lib/nagios/plugins/check-fresh-files-in-dir.py]
[18:47:47] <icinga-wm>	 PROBLEM - puppet last run on mw1183 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:47:47] <icinga-wm>	 PROBLEM - puppet last run on mw1186 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:47:47] <icinga-wm>	 PROBLEM - puppet last run on db1015 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:47:47] <icinga-wm>	 PROBLEM - puppet last run on mw2204 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:47:48] <icinga-wm>	 PROBLEM - puppet last run on helium is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:47:48] <icinga-wm>	 PROBLEM - puppet last run on analytics1048 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:47:48] <icinga-wm>	 PROBLEM - puppet last run on elastic2002 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:47:59] <icinga-wm>	 PROBLEM - puppet last run on mw2064 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:47:59] <icinga-wm>	 PROBLEM - puppet last run on mw1207 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:47:59] <icinga-wm>	 PROBLEM - puppet last run on labvirt1006 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:47:59] <icinga-wm>	 PROBLEM - puppet last run on elastic1034 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:48:00] <icinga-wm>	 PROBLEM - puppet last run on mw2229 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:48:00] <icinga-wm>	 PROBLEM - puppet last run on cp1049 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:48:07] <icinga-wm>	 PROBLEM - puppet last run on mw1299 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:48:07] <icinga-wm>	 PROBLEM - puppet last run on mw2109 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:48:07] <icinga-wm>	 PROBLEM - puppet last run on db2039 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:48:07] <icinga-wm>	 PROBLEM - puppet last run on elastic2014 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:48:08] <icinga-wm>	 PROBLEM - puppet last run on elastic1018 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:48:08] <icinga-wm>	 PROBLEM - puppet last run on ms-be2006 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:48:08] <icinga-wm>	 PROBLEM - puppet last run on ms-be2002 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:48:09] <icinga-wm>	 PROBLEM - puppet last run on notebook1002 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:48:09] <icinga-wm>	 PROBLEM - puppet last run on cp4002 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:48:17] <icinga-wm>	 PROBLEM - puppet last run on mw1219 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:48:17] <icinga-wm>	 PROBLEM - puppet last run on mw1175 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:48:17] <icinga-wm>	 PROBLEM - puppet last run on fluorine is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:48:17] <icinga-wm>	 PROBLEM - puppet last run on mw2126 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:48:17] <icinga-wm>	 PROBLEM - puppet last run on mw1205 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:48:18] <icinga-wm>	 PROBLEM - puppet last run on ms-be1018 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:48:18] <icinga-wm>	 PROBLEM - puppet last run on db1022 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:48:19] <icinga-wm>	 PROBLEM - puppet last run on mw1241 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:48:19] <icinga-wm>	 PROBLEM - puppet last run on potassium is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:48:20] <icinga-wm>	 PROBLEM - puppet last run on ms-be2024 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:48:20] <icinga-wm>	 PROBLEM - puppet last run on mw1263 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:48:21] <icinga-wm>	 PROBLEM - puppet last run on cp2023 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:48:21] <icinga-wm>	 PROBLEM - puppet last run on restbase2004 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:48:22] <icinga-wm>	 PROBLEM - puppet last run on db1031 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:48:30] <icinga-wm>	 PROBLEM - puppet last run on analytics1035 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:48:31] <icinga-wm>	 PROBLEM - puppet last run on achernar is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:48:34] <icinga-wm>	 PROBLEM - puppet last run on db1064 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:48:36] <icinga-wm>	 PROBLEM - puppet last run on cp3040 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:48:36] <icinga-wm>	 PROBLEM - puppet last run on mw1198 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:48:36] <icinga-wm>	 PROBLEM - puppet last run on meitnerium is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:48:47] <icinga-wm>	 PROBLEM - puppet last run on db1050 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:48:47] <icinga-wm>	 PROBLEM - puppet last run on db2051 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:48:48] <icinga-wm>	 PROBLEM - puppet last run on mw2207 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:48:48] <icinga-wm>	 PROBLEM - puppet last run on db2045 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:48:48] <icinga-wm>	 PROBLEM - puppet last run on mw2063 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:48:48] <icinga-wm>	 PROBLEM - puppet last run on mw2214 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:48:48] <icinga-wm>	 PROBLEM - puppet last run on mw2075 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:48:49] <icinga-wm>	 PROBLEM - puppet last run on db2019 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:48:49] <icinga-wm>	 PROBLEM - puppet last run on db2016 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:48:50] <icinga-wm>	 PROBLEM - puppet last run on mw2083 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:48:50] <icinga-wm>	 PROBLEM - puppet last run on mw2158 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:48:51] <icinga-wm>	 PROBLEM - puppet last run on mw2077 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:48:51] <icinga-wm>	 PROBLEM - puppet last run on ms-be2004 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:48:52] <icinga-wm>	 PROBLEM - puppet last run on mw2188 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:49:07] <icinga-wm>	 PROBLEM - puppet last run on ms-fe2004 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:49:07] <icinga-wm>	 PROBLEM - puppet last run on maps2002 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:49:07] <icinga-wm>	 PROBLEM - puppet last run on db1046 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:49:08] <icinga-wm>	 PROBLEM - puppet last run on db1066 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:49:08] <icinga-wm>	 PROBLEM - puppet last run on cp2016 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:49:08] <icinga-wm>	 PROBLEM - puppet last run on mw2187 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:49:08] <icinga-wm>	 PROBLEM - puppet last run on elastic1027 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:49:09] <icinga-wm>	 PROBLEM - puppet last run on mw1298 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:49:09] <icinga-wm>	 PROBLEM - puppet last run on es1014 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:49:10] <icinga-wm>	 PROBLEM - puppet last run on mw2079 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:49:10] <icinga-wm>	 PROBLEM - puppet last run on mw2136 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:49:11] <icinga-wm>	 PROBLEM - puppet last run on lvs1005 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:49:11] <icinga-wm>	 PROBLEM - puppet last run on mw2072 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:49:12] <icinga-wm>	 PROBLEM - puppet last run on mw2163 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:49:18] <icinga-wm>	 PROBLEM - puppet last run on lvs1007 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:49:18] <icinga-wm>	 PROBLEM - puppet last run on ganeti1002 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:49:19] <icinga-wm>	 PROBLEM - puppet last run on kafka1001 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:49:24] <icinga-wm>	 PROBLEM - puppet last run on mw1254 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:49:24] <icinga-wm>	 PROBLEM - puppet last run on mw1300 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:49:25] <icinga-wm>	 PROBLEM - puppet last run on mw1285 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:49:26] <icinga-wm>	 PROBLEM - puppet last run on wtp1001 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:49:26] <icinga-wm>	 PROBLEM - puppet last run on mw1294 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:49:27] <icinga-wm>	 PROBLEM - puppet last run on elastic1046 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:49:37] <grrrit-wm>	 (03PS1) 10MarcoAurelio: New 'engineer' group for ruwiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/308448 (https://phabricator.wikimedia.org/T144599) 
[18:49:37] <icinga-wm>	 PROBLEM - puppet last run on cp2011 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:49:38] <icinga-wm>	 PROBLEM - puppet last run on db2035 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:49:38] <icinga-wm>	 PROBLEM - puppet last run on ms-be1006 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:49:38] <icinga-wm>	 PROBLEM - puppet last run on rdb2005 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:49:39] <icinga-wm>	 PROBLEM - puppet last run on elastic1028 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:49:39] <icinga-wm>	 PROBLEM - puppet last run on wtp1017 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:49:40] <icinga-wm>	 PROBLEM - puppet last run on mw2087 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:49:40] <icinga-wm>	 PROBLEM - puppet last run on mw2067 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:49:47] <icinga-wm>	 PROBLEM - puppet last run on ms-be1016 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:49:47] <icinga-wm>	 PROBLEM - puppet last run on mw2247 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:49:48] <icinga-wm>	 PROBLEM - puppet last run on cp3018 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:49:48] <icinga-wm>	 PROBLEM - puppet last run on db1054 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:49:48] <icinga-wm>	 PROBLEM - puppet last run on lvs2004 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:49:48] <icinga-wm>	 PROBLEM - puppet last run on mw1251 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:49:57] <icinga-wm>	 PROBLEM - puppet last run on mw1209 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:49:57] <icinga-wm>	 PROBLEM - puppet last run on mw1204 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:49:57] <icinga-wm>	 PROBLEM - puppet last run on snapshot1001 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:49:57] <icinga-wm>	 PROBLEM - puppet last run on restbase2002 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:49:57] <icinga-wm>	 PROBLEM - puppet last run on lvs1004 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:49:58] <icinga-wm>	 PROBLEM - puppet last run on ms-fe1001 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:49:58] <icinga-wm>	 PROBLEM - puppet last run on elastic1038 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:49:58] <icinga-wm>	 PROBLEM - puppet last run on mw2127 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:49:58] <icinga-wm>	 PROBLEM - puppet last run on rcs1001 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:49:59] <icinga-wm>	 PROBLEM - puppet last run on mw2096 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:50:08] <icinga-wm>	 PROBLEM - puppet last run on mw1189 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:50:09] <icinga-wm>	 PROBLEM - puppet last run on cp3047 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:50:09] <icinga-wm>	 PROBLEM - puppet last run on cp2018 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:50:11] <icinga-wm>	 PROBLEM - puppet last run on mw1229 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:50:12] <icinga-wm>	 PROBLEM - puppet last run on mw1173 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:50:12] <icinga-wm>	 PROBLEM - puppet last run on labcontrol1001 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:50:16] <grrrit-wm>	 (03CR) 10jenkins-bot: [V: 04-1] New 'engineer' group for ruwiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/308448 (https://phabricator.wikimedia.org/T144599) (owner: 10MarcoAurelio)
[18:50:20] <icinga-wm>	 PROBLEM - puppet last run on graphite2001 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:50:28] <icinga-wm>	 PROBLEM - puppet last run on ms-be1015 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:50:29] <icinga-wm>	 PROBLEM - puppet last run on mw1253 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:50:29] <icinga-wm>	 PROBLEM - puppet last run on db2052 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:50:29] <icinga-wm>	 PROBLEM - puppet last run on mw2244 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:50:29] <icinga-wm>	 PROBLEM - puppet last run on mw2082 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:50:29] <icinga-wm>	 PROBLEM - puppet last run on mendelevium is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:50:29] <icinga-wm>	 PROBLEM - puppet last run on wtp2006 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:50:30] <icinga-wm>	 PROBLEM - puppet last run on mw2191 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:50:30] <icinga-wm>	 PROBLEM - puppet last run on bast1001 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:50:31] <icinga-wm>	 PROBLEM - puppet last run on ganeti2004 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:50:37] <icinga-wm>	 PROBLEM - puppet last run on db1079 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:50:37] <icinga-wm>	 PROBLEM - puppet last run on mw2137 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:50:37] <icinga-wm>	 PROBLEM - puppet last run on wtp2001 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:50:37] <icinga-wm>	 PROBLEM - puppet last run on mw2113 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:50:37] <icinga-wm>	 PROBLEM - puppet last run on mw2093 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:50:38] <icinga-wm>	 PROBLEM - puppet last run on labvirt1003 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:50:38] <icinga-wm>	 PROBLEM - puppet last run on ruthenium is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:50:39] <icinga-wm>	 PROBLEM - puppet last run on labvirt1010 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:50:39] <icinga-wm>	 PROBLEM - puppet last run on mw2142 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:50:40] <icinga-wm>	 PROBLEM - puppet last run on mw1213 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:50:40] <icinga-wm>	 PROBLEM - puppet last run on mw1227 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:50:41] <icinga-wm>	 PROBLEM - puppet last run on mw2128 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:50:41] <icinga-wm>	 PROBLEM - puppet last run on db2036 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:50:47] <icinga-wm>	 PROBLEM - puppet last run on db1051 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:50:47] <icinga-wm>	 PROBLEM - puppet last run on mw1194 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:50:47] <icinga-wm>	 PROBLEM - puppet last run on db1037 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:50:48] <icinga-wm>	 PROBLEM - puppet last run on db1035 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:50:48] <icinga-wm>	 PROBLEM - puppet last run on ocg1001 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:50:48] <icinga-wm>	 PROBLEM - puppet last run on mw2190 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:50:59] <icinga-wm>	 PROBLEM - puppet last run on ms-fe1002 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:51:01] <icinga-wm>	 PROBLEM - puppet last run on mw1252 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:51:03] <icinga-wm>	 PROBLEM - puppet last run on cp4001 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:51:03] <icinga-wm>	 PROBLEM - puppet last run on mw2143 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:51:04] <icinga-wm>	 PROBLEM - puppet last run on mw1182 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:51:04] <icinga-wm>	 PROBLEM - puppet last run on db1021 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:51:07] <icinga-wm>	 PROBLEM - puppet last run on ms-fe2003 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:51:07] <icinga-wm>	 PROBLEM - puppet last run on elastic2024 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:51:07] <icinga-wm>	 PROBLEM - puppet last run on wezen is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:51:07] <icinga-wm>	 PROBLEM - puppet last run on mw2212 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:51:07] <icinga-wm>	 PROBLEM - puppet last run on cp2014 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:51:08] <icinga-wm>	 PROBLEM - puppet last run on mw2134 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:51:08] <icinga-wm>	 PROBLEM - puppet last run on mw2176 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:51:09] <icinga-wm>	 PROBLEM - puppet last run on sca2001 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:51:09] <icinga-wm>	 PROBLEM - puppet last run on mw2146 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:51:17] <icinga-wm>	 PROBLEM - puppet last run on mw2211 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:51:17] <icinga-wm>	 PROBLEM - puppet last run on mw2084 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:51:17] <icinga-wm>	 PROBLEM - puppet last run on mw2132 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:51:17] <icinga-wm>	 PROBLEM - puppet last run on labvirt1001 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:51:17] <icinga-wm>	 PROBLEM - puppet last run on db2028 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:51:18] <icinga-wm>	 PROBLEM - puppet last run on db2029 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:51:18] <icinga-wm>	 PROBLEM - puppet last run on restbase2001 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:51:19] <icinga-wm>	 PROBLEM - puppet last run on mw1166 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:51:19] <icinga-wm>	 PROBLEM - puppet last run on cp1058 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:52:39] <icinga-wm>	 PROBLEM - puppet last run on mw1191 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:52:40] <icinga-wm>	 PROBLEM - puppet last run on cp1071 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:52:40] <icinga-wm>	 PROBLEM - puppet last run on mw2097 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:52:41] <icinga-wm>	 PROBLEM - puppet last run on db1026 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:52:48] <icinga-wm>	 PROBLEM - puppet last run on cp4017 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:52:48] <icinga-wm>	 PROBLEM - puppet last run on cp3006 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:52:49] <icinga-wm>	 PROBLEM - puppet last run on mw2159 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:52:57] <icinga-wm>	 PROBLEM - puppet last run on es1016 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:52:58] <icinga-wm>	 PROBLEM - puppet last run on es2011 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:52:59] <icinga-wm>	 PROBLEM - puppet last run on db1036 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:52:59] <icinga-wm>	 PROBLEM - puppet last run on mw2130 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:52:59] <icinga-wm>	 PROBLEM - puppet last run on ms-be2010 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:52:59] <icinga-wm>	 PROBLEM - puppet last run on mw2111 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:52:59] <icinga-wm>	 PROBLEM - puppet last run on mw2101 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:53:07] <icinga-wm>	 PROBLEM - puppet last run on lvs3004 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:53:08] <icinga-wm>	 PROBLEM - puppet last run on mw2172 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:53:08] <icinga-wm>	 PROBLEM - puppet last run on mw2213 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:53:08] <icinga-wm>	 PROBLEM - puppet last run on mw1193 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:53:09] <icinga-wm>	 PROBLEM - puppet last run on ocg1002 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:53:09] <icinga-wm>	 PROBLEM - puppet last run on mw1295 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:53:09] <icinga-wm>	 PROBLEM - puppet last run on cp1062 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:53:09] <icinga-wm>	 PROBLEM - puppet last run on mw1278 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:53:09] <icinga-wm>	 PROBLEM - puppet last run on analytics1056 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:53:18] <icinga-wm>	 PROBLEM - puppet last run on elastic2012 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:53:18] <icinga-wm>	 PROBLEM - puppet last run on mw1304 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:53:19] <icinga-wm>	 PROBLEM - puppet last run on elastic1040 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:53:19] <icinga-wm>	 PROBLEM - puppet last run on stat1001 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:53:19] <icinga-wm>	 PROBLEM - puppet last run on elastic1045 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:53:19] <icinga-wm>	 PROBLEM - puppet last run on kafka1014 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:53:19] <icinga-wm>	 PROBLEM - puppet last run on druid1001 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:53:19] <icinga-wm>	 PROBLEM - puppet last run on db1068 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:53:20] <icinga-wm>	 PROBLEM - puppet last run on mira is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:53:20] <icinga-wm>	 PROBLEM - puppet last run on cp4019 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:53:21] <icinga-wm>	 PROBLEM - puppet last run on cp3009 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:53:30] <icinga-wm>	 PROBLEM - puppet last run on db1082 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:53:31] <icinga-wm>	 PROBLEM - puppet last run on elastic1024 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:53:34] <icinga-wm>	 PROBLEM - puppet last run on mw2178 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:53:34] <icinga-wm>	 PROBLEM - puppet last run on iron is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:53:34] <icinga-wm>	 PROBLEM - puppet last run on cp2022 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:53:34] <icinga-wm>	 PROBLEM - puppet last run on analytics1002 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:53:37] <icinga-wm>	 PROBLEM - puppet last run on mw1180 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:53:37] <icinga-wm>	 PROBLEM - puppet last run on db2023 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:53:37] <icinga-wm>	 PROBLEM - puppet last run on wasat is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:53:38] <icinga-wm>	 PROBLEM - puppet last run on mc2014 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:53:38] <icinga-wm>	 PROBLEM - puppet last run on serpens is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:53:38] <icinga-wm>	 PROBLEM - puppet last run on db1071 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:53:38] <icinga-wm>	 PROBLEM - puppet last run on mw2098 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:53:39] <icinga-wm>	 PROBLEM - puppet last run on lvs1011 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:53:48] <icinga-wm>	 PROBLEM - puppet last run on elastic2018 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:53:48] <icinga-wm>	 PROBLEM - puppet last run on db2057 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:53:49] <icinga-wm>	 PROBLEM - puppet last run on ms-be3001 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:53:49] <icinga-wm>	 PROBLEM - puppet last run on lvs4001 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:53:49] <icinga-wm>	 PROBLEM - puppet last run on ms-fe3002 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:53:59] <icinga-wm>	 PROBLEM - puppet last run on tin is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:53:59] <icinga-wm>	 PROBLEM - puppet last run on maps2001 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:54:00] <icinga-wm>	 PROBLEM - puppet last run on elastic2013 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:54:08] <icinga-wm>	 PROBLEM - puppet last run on mw2200 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:54:08] <icinga-wm>	 PROBLEM - puppet last run on mw2223 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:54:08] <icinga-wm>	 PROBLEM - puppet last run on mw1238 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:54:08] <icinga-wm>	 PROBLEM - puppet last run on mw1152 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:54:08] <icinga-wm>	 PROBLEM - puppet last run on kafka1013 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:54:08] <icinga-wm>	 PROBLEM - puppet last run on mc1008 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:54:08] <icinga-wm>	 PROBLEM - puppet last run on mw1236 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:54:09] <icinga-wm>	 PROBLEM - puppet last run on mw1218 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:54:09] <icinga-wm>	 PROBLEM - puppet last run on mw1274 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:54:10] <icinga-wm>	 PROBLEM - puppet last run on mw1296 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:54:19] <icinga-wm>	 PROBLEM - puppet last run on elastic1030 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:54:20] <icinga-wm>	 PROBLEM - puppet last run on db2040 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:54:21] <icinga-wm>	 PROBLEM - puppet last run on cp1099 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:54:21] <icinga-wm>	 PROBLEM - puppet last run on wtp1002 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:54:21] <icinga-wm>	 PROBLEM - puppet last run on analytics1054 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:54:21] <icinga-wm>	 PROBLEM - puppet last run on db1055 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:54:22] <icinga-wm>	 PROBLEM - puppet last run on mw1169 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:54:22] <icinga-wm>	 PROBLEM - puppet last run on seaborgium is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:54:28] <icinga-wm>	 PROBLEM - puppet last run on pc2005 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:54:28] <icinga-wm>	 PROBLEM - puppet last run on bromine is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:54:28] <icinga-wm>	 PROBLEM - puppet last run on tegmen is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:54:38] <icinga-wm>	 PROBLEM - puppet last run on mw2202 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:54:38] <icinga-wm>	 PROBLEM - puppet last run on db2061 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:54:39] <icinga-wm>	 PROBLEM - puppet last run on mw1245 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:54:39] <icinga-wm>	 PROBLEM - puppet last run on pybal-test2003 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:54:39] <icinga-wm>	 PROBLEM - puppet last run on mw2150 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:54:40] <icinga-wm>	 PROBLEM - puppet last run on bohrium is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:54:47] <icinga-wm>	 PROBLEM - puppet last run on mw1214 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:54:48] <icinga-wm>	 PROBLEM - puppet last run on db1010 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:54:48] <icinga-wm>	 PROBLEM - puppet last run on wdqs1001 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:54:50] <icinga-wm>	 PROBLEM - puppet last run on mw2078 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:54:50] <icinga-wm>	 PROBLEM - puppet last run on acamar is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:54:50] <icinga-wm>	 PROBLEM - puppet last run on rdb1006 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:54:50] <icinga-wm>	 PROBLEM - puppet last run on mw2236 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:54:50] <icinga-wm>	 PROBLEM - puppet last run on mw2242 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:54:51] <icinga-wm>	 PROBLEM - puppet last run on elastic1032 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:54:51] <icinga-wm>	 PROBLEM - puppet last run on mw2241 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:54:51] <icinga-wm>	 PROBLEM - puppet last run on wtp2009 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:54:57] <icinga-wm>	 PROBLEM - puppet last run on mw2094 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:54:58] <icinga-wm>	 PROBLEM - puppet last run on db2048 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:54:58] <icinga-wm>	 PROBLEM - puppet last run on mw1165 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:54:58] <icinga-wm>	 PROBLEM - puppet last run on labvirt1005 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:54:58] <icinga-wm>	 PROBLEM - puppet last run on db1024 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:54:58] <icinga-wm>	 PROBLEM - puppet last run on tungsten is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:55:09] <icinga-wm>	 PROBLEM - puppet last run on cp3013 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:55:09] <icinga-wm>	 PROBLEM - puppet last run on lvs1001 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:55:10] <icinga-wm>	 PROBLEM - puppet last run on mw2106 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:55:10] <icinga-wm>	 PROBLEM - puppet last run on analytics1036 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:55:19] <icinga-wm>	 PROBLEM - puppet last run on db1062 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:55:28] <icinga-wm>	 PROBLEM - puppet last run on mw2237 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:55:28] <icinga-wm>	 PROBLEM - puppet last run on ms-be1024 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:55:28] <icinga-wm>	 PROBLEM - puppet last run on cp1065 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:55:29] <icinga-wm>	 PROBLEM - puppet last run on mw2107 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:55:38] <icinga-wm>	 PROBLEM - puppet last run on labsdb1001 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:55:40] <icinga-wm>	 PROBLEM - puppet last run on dbproxy1006 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:55:40] <icinga-wm>	 PROBLEM - puppet last run on db1052 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:55:40] <icinga-wm>	 PROBLEM - puppet last run on ms-be1009 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:55:40] <icinga-wm>	 PROBLEM - puppet last run on snapshot1005 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:55:40] <icinga-wm>	 PROBLEM - puppet last run on elastic2003 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:55:41] <icinga-wm>	 PROBLEM - puppet last run on mw1297 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:55:48] <icinga-wm>	 PROBLEM - puppet last run on maps-test2001 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:55:49] <icinga-wm>	 PROBLEM - puppet last run on mw1240 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:55:49] <grrrit-wm>	 (03PS2) 10MarcoAurelio: New 'engineer' group for ruwiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/308448 (https://phabricator.wikimedia.org/T144599) 
[18:55:58] <icinga-wm>	 PROBLEM - puppet last run on db1076 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:56:01] <icinga-wm>	 PROBLEM - puppet last run on labvirt1007 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:56:01] <icinga-wm>	 PROBLEM - puppet last run on mw1275 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:56:01] <icinga-wm>	 PROBLEM - puppet last run on cp2015 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:56:07] <icinga-wm>	 PROBLEM - puppet last run on ms-be2013 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:56:08] <icinga-wm>	 PROBLEM - puppet last run on es2003 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:56:08] <icinga-wm>	 PROBLEM - puppet last run on db2055 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:56:09] <icinga-wm>	 PROBLEM - puppet last run on mw1247 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:56:18] <icinga-wm>	 PROBLEM - puppet last run on cp3046 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:56:18] <icinga-wm>	 PROBLEM - puppet last run on cp3031 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:56:18] <icinga-wm>	 PROBLEM - puppet last run on hassium is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:56:18] <icinga-wm>	 PROBLEM - puppet last run on ms-be1007 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:56:19] <icinga-wm>	 PROBLEM - puppet last run on wtp1021 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:56:19] <Luke081515>	 argh...
[18:56:28] <icinga-wm>	 PROBLEM - puppet last run on aqs1004 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:56:38] <icinga-wm>	 PROBLEM - puppet last run on bast4001 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:56:39] <icinga-wm>	 PROBLEM - puppet last run on dubnium is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 5 minutes ago with 1 failures. Failed resources (up to 3 shown): File[/usr/lib/nagios/plugins/check-fresh-files-in-dir.py]
[18:56:41] <icinga-wm>	 PROBLEM - puppet last run on uranium is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:56:57] <Luke081515>	 looks like puppetmaster has issues
[18:56:58] <icinga-wm>	 PROBLEM - puppet last run on mx1001 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:56:58] <icinga-wm>	 PROBLEM - puppet last run on elastic1023 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:56:59] <icinga-wm>	 PROBLEM - puppet last run on labcontrol1002 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:57:09] <icinga-wm>	 RECOVERY - SSH on rhodium is OK: SSH OK - OpenSSH_6.7p1 Debian-5+deb8u3 (protocol 2.0)
[18:57:18] <icinga-wm>	 PROBLEM - puppet last run on netmon1001 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:57:19] <icinga-wm>	 PROBLEM - puppet last run on ganeti2003 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:57:19] <icinga-wm>	 PROBLEM - puppet last run on mw2248 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:57:19] <icinga-wm>	 PROBLEM - puppet last run on mw2118 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:57:27] <icinga-wm>	 PROBLEM - puppet last run on bast2001 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:57:27] <icinga-wm>	 PROBLEM - puppet last run on wtp1006 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:57:27] <icinga-wm>	 PROBLEM - puppet last run on labstore2004 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:57:27] <icinga-wm>	 PROBLEM - puppet last run on mw2086 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:57:28] <icinga-wm>	 PROBLEM - puppet last run on db2011 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:57:28] <icinga-wm>	 PROBLEM - puppet last run on db2043 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:57:28] <icinga-wm>	 PROBLEM - puppet last run on db2034 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:57:29] <icinga-wm>	 PROBLEM - puppet last run on stat1004 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:57:39] <icinga-wm>	 PROBLEM - puppet last run on ganeti2002 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:57:39] <icinga-wm>	 PROBLEM - puppet last run on mw2219 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:57:39] <icinga-wm>	 PROBLEM - puppet last run on wtp1008 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 6 minutes ago with 1 failures. Failed resources (up to 3 shown): File[/usr/lib/nagios/plugins/check-fresh-files-in-dir.py]
[18:57:48] <icinga-wm>	 PROBLEM - puppet last run on dbproxy1009 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:57:58] <icinga-wm>	 PROBLEM - puppet last run on iridium is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:58:00] <icinga-wm>	 PROBLEM - puppet last run on db1074 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:58:08] <icinga-wm>	 PROBLEM - puppet last run on mw1305 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:58:08] <icinga-wm>	 PROBLEM - puppet last run on db1063 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:58:19] <icinga-wm>	 PROBLEM - Disk space on rhodium is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds.
[18:58:19] <icinga-wm>	 PROBLEM - Check size of conntrack table on rhodium is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds.
[18:58:23] <icinga-wm>	 PROBLEM - puppet last run on ms-be2020 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:58:27] <icinga-wm>	 PROBLEM - puppet last run on maps2003 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:58:27] <icinga-wm>	 PROBLEM - puppet last run on elastic2011 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:58:28] <icinga-wm>	 PROBLEM - MD RAID on rhodium is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds.
[18:58:37] <icinga-wm>	 PROBLEM - puppet last run on db1084 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:58:38] <icinga-wm>	 PROBLEM - puppet last run on ms-be2005 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:58:38] <icinga-wm>	 PROBLEM - puppet last run on mw2141 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:58:38] <icinga-wm>	 PROBLEM - puppet last run on mw2081 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:58:38] <icinga-wm>	 PROBLEM - puppet last run on mw2076 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:58:38] <icinga-wm>	 PROBLEM - puppet last run on analytics1053 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:58:47] <icinga-wm>	 PROBLEM - puppet last run on cp3016 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:58:47] <icinga-wm>	 PROBLEM - puppet last run on lvs3001 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:58:58] <icinga-wm>	 PROBLEM - puppet last run on ms-be1025 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:58:58] <icinga-wm>	 PROBLEM - puppet last run on mc2001 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:58:58] <icinga-wm>	 PROBLEM - puppet last run on dbstore1002 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:58:58] <icinga-wm>	 PROBLEM - puppet last run on elastic1021 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:58:59] <icinga-wm>	 PROBLEM - puppet last run on db1072 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:58:59] <icinga-wm>	 PROBLEM - puppet last run on db1073 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:58:59] <icinga-wm>	 PROBLEM - puppet last run on db2067 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:58:59] <icinga-wm>	 PROBLEM - puppet last run on prometheus2001 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:59:16] <icinga-wm>	 PROBLEM - puppet last run on mc1018 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:59:16] <icinga-wm>	 PROBLEM - puppet last run on graphite1002 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:59:17] <icinga-wm>	 PROBLEM - puppet last run on cp1055 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:59:29] <icinga-wm>	 PROBLEM - puppet last run on ms-be1022 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:59:30] <icinga-wm>	 PROBLEM - puppet last run on mc1017 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:59:30] <icinga-wm>	 PROBLEM - puppet last run on labservices1001 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:59:31] <icinga-wm>	 PROBLEM - puppet last run on cp3012 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:59:37] <icinga-wm>	 PROBLEM - puppet last run on cp2002 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:59:38] <icinga-wm>	 PROBLEM - puppet last run on cp1068 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:59:38] <icinga-wm>	 PROBLEM - puppet last run on wtp1018 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:59:38] <icinga-wm>	 PROBLEM - puppet last run on lvs1003 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:59:47] <icinga-wm>	 PROBLEM - puppet last run on fermium is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:59:47] <icinga-wm>	 PROBLEM - puppet last run on terbium is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:59:48] <icinga-wm>	 PROBLEM - puppet last run on db1077 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:59:48] <icinga-wm>	 PROBLEM - puppet last run on mw1284 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:59:48] <icinga-wm>	 PROBLEM - puppet last run on mw2074 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:59:48] <icinga-wm>	 PROBLEM - puppet last run on dbproxy1008 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:59:49] <icinga-wm>	 PROBLEM - puppet last run on pybal-test2002 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:59:49] <icinga-wm>	 PROBLEM - puppet last run on sarin is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:59:49] <icinga-wm>	 PROBLEM - puppet last run on db1086 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:59:50] <icinga-wm>	 PROBLEM - puppet last run on mw2217 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:59:50] <icinga-wm>	 PROBLEM - puppet last run on mw1226 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[19:00:01] <icinga-wm>	 PROBLEM - puppet last run on mw2129 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[19:00:01] <icinga-wm>	 PROBLEM - puppet last run on mw1276 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[19:00:01] <icinga-wm>	 PROBLEM - puppet last run on mc1002 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[19:00:02] <icinga-wm>	 PROBLEM - puppet last run on db1018 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[19:00:08] <icinga-wm>	 PROBLEM - puppet last run on labnet1002 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[19:00:08] <icinga-wm>	 PROBLEM - puppet last run on rdb2003 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[19:00:08] <icinga-wm>	 PROBLEM - puppet last run on cp1053 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[19:00:19] <icinga-wm>	 PROBLEM - puppet last run on mc2007 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[19:00:20] <icinga-wm>	 PROBLEM - puppet last run on cp1054 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[19:00:20] <icinga-wm>	 PROBLEM - puppet last run on es2014 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[19:00:20] <icinga-wm>	 PROBLEM - puppet last run on graphite1003 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[19:00:21] <icinga-wm>	 PROBLEM - puppet last run on mw1265 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[19:00:27] <icinga-wm>	 PROBLEM - puppet last run on wtp2008 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[19:00:28] <icinga-wm>	 PROBLEM - puppet last run on ms-be2025 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[19:00:37] <icinga-wm>	 PROBLEM - puppet last run on gallium is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[19:00:38] <icinga-wm>	 PROBLEM - puppet last run on mw1255 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[19:00:39] <icinga-wm>	 RECOVERY - Disk space on rhodium is OK: DISK OK
[19:00:40] <icinga-wm>	 PROBLEM - puppet last run on mw1260 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[19:00:40] <icinga-wm>	 PROBLEM - puppet last run on db2060 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[19:00:40] <icinga-wm>	 PROBLEM - puppet last run on ms-be1010 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[19:00:40] <icinga-wm>	 PROBLEM - puppet last run on ms-be2026 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[19:00:45] <icinga-wm>	 PROBLEM - puppet last run on chromium is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[19:00:45] <icinga-wm>	 RECOVERY - Check size of conntrack table on rhodium is OK: OK: nf_conntrack is 0 % full
[19:00:55] <icinga-wm>	 PROBLEM - puppet last run on cp3007 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[19:00:55] <icinga-wm>	 PROBLEM - puppet last run on cp3008 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[19:00:55] <icinga-wm>	 RECOVERY - MD RAID on rhodium is OK: OK: Active: 4, Working: 4, Failed: 0, Spare: 0
[19:00:59] <icinga-wm>	 PROBLEM - puppet last run on restbase1007 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[19:00:59] <icinga-wm>	 PROBLEM - puppet last run on db2044 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[19:01:08] <icinga-wm>	 PROBLEM - puppet last run on cp2001 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[19:01:08] <icinga-wm>	 PROBLEM - puppet last run on db1067 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[19:01:08] <icinga-wm>	 PROBLEM - puppet last run on ms-fe1004 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[19:01:08] <icinga-wm>	 PROBLEM - puppet last run on mw2073 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[19:01:08] <icinga-wm>	 PROBLEM - puppet last run on eventlog2001 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[19:01:08] <icinga-wm>	 PROBLEM - puppet last run on druid1002 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[19:01:09] <icinga-wm>	 PROBLEM - puppet last run on mw2095 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[19:01:09] <icinga-wm>	 PROBLEM - puppet last run on db2064 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[19:01:10] <icinga-wm>	 PROBLEM - puppet last run on mc2015 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[19:01:10] <icinga-wm>	 PROBLEM - puppet last run on wtp2017 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[19:01:11] <icinga-wm>	 PROBLEM - puppet last run on cp4016 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[19:01:11] <icinga-wm>	 PROBLEM - puppet last run on cp4010 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[19:01:12] <icinga-wm>	 PROBLEM - puppet last run on cp3048 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[19:01:12] <icinga-wm>	 PROBLEM - puppet last run on cp3036 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[19:01:18] <icinga-wm>	 PROBLEM - puppet last run on conf1001 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[19:01:19] <icinga-wm>	 PROBLEM - puppet last run on wtp1019 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[19:01:19] <icinga-wm>	 PROBLEM - puppet last run on db1028 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[19:01:20] <icinga-wm>	 PROBLEM - puppet last run on db1089 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[19:01:21] <icinga-wm>	 PROBLEM - puppet last run on mw1177 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[19:01:28] <icinga-wm>	 PROBLEM - puppet last run on labsdb1008 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[19:01:32] <icinga-wm>	 PROBLEM - puppet last run on db1059 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[19:01:32] <icinga-wm>	 PROBLEM - puppet last run on wtp2019 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[19:01:44] <icinga-wm>	 PROBLEM - puppet last run on scb1001 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[19:01:44] <icinga-wm>	 PROBLEM - puppet last run on db1049 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[19:01:47] <icinga-wm>	 PROBLEM - puppet last run on mw2135 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[19:01:48] <icinga-wm>	 PROBLEM - puppet last run on silver is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[19:01:58] <icinga-wm>	 PROBLEM - puppet last run on pc1006 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[19:01:58] <icinga-wm>	 PROBLEM - puppet last run on es2018 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[19:02:00] <icinga-wm>	 PROBLEM - puppet last run on cp3017 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[19:02:00] <icinga-wm>	 PROBLEM - puppet last run on mw2166 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[19:02:08] <icinga-wm>	 PROBLEM - puppet last run on labmon1001 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[19:02:08] <icinga-wm>	 PROBLEM - puppet last run on mw1199 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[19:02:08] <icinga-wm>	 PROBLEM - puppet last run on elastic1042 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[19:02:08] <icinga-wm>	 PROBLEM - puppet last run on snapshot1007 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[19:02:08] <icinga-wm>	 PROBLEM - puppet last run on db1043 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[19:02:18] <icinga-wm>	 PROBLEM - puppet last run on restbase2006 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[19:02:18] <icinga-wm>	 PROBLEM - puppet last run on scb2001 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[19:02:18] <icinga-wm>	 PROBLEM - puppet last run on kafka1002 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[19:02:18] <icinga-wm>	 PROBLEM - puppet last run on db2068 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[19:02:18] <icinga-wm>	 PROBLEM - puppet last run on es2013 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[19:02:18] <icinga-wm>	 PROBLEM - puppet last run on mw2250 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[19:02:18] <icinga-wm>	 PROBLEM - puppet last run on lvs2002 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[19:02:19] <icinga-wm>	 PROBLEM - puppet last run on mw2228 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[19:02:19] <icinga-wm>	 PROBLEM - puppet last run on wtp2004 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[19:02:20] <icinga-wm>	 PROBLEM - puppet last run on mw2088 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[19:02:20] <icinga-wm>	 PROBLEM - puppet last run on wtp2015 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[19:02:27] <icinga-wm>	 PROBLEM - puppet last run on mw1184 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[19:02:28] <icinga-wm>	 PROBLEM - puppet last run on db2056 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[19:02:38] <icinga-wm>	 PROBLEM - puppet last run on neodymium is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[19:02:38] <icinga-wm>	 PROBLEM - puppet last run on restbase2008 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[19:02:39] <icinga-wm>	 PROBLEM - puppet last run on ms-be2021 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[19:02:51] <icinga-wm>	 PROBLEM - puppet last run on mw1302 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[19:02:57] <icinga-wm>	 PROBLEM - puppet last run on wtp1014 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[19:02:57] <icinga-wm>	 PROBLEM - puppet last run on lvs1009 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[19:02:58] <icinga-wm>	 PROBLEM - puppet last run on elastic2010 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[19:03:10] <icinga-wm>	 PROBLEM - puppet last run on planet2001 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[19:03:10] <icinga-wm>	 PROBLEM - puppet last run on restbase1012 is CRITICAL: CRITICAL: Puppet has 2 failures. Last run 7 minutes ago with 2 failures. Failed resources (up to 3 shown): File[/home/midom],File[/home/eevans]
[19:03:10] <icinga-wm>	 PROBLEM - puppet last run on cp1064 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[19:03:10] <icinga-wm>	 PROBLEM - puppet last run on praseodymium is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[19:03:20] <icinga-wm>	 PROBLEM - puppet last run on mw1168 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[19:03:38] <icinga-wm>	 PROBLEM - puppet last run on restbase1008 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[19:03:38] <icinga-wm>	 PROBLEM - puppet last run on lvs2006 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[19:03:38] <icinga-wm>	 PROBLEM - puppet last run on mc1014 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[19:03:38] <icinga-wm>	 PROBLEM - puppet last run on mw1195 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[19:03:39] <icinga-wm>	 PROBLEM - puppet last run on mc2006 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[19:03:39] <icinga-wm>	 PROBLEM - puppet last run on db2062 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[19:03:39] <icinga-wm>	 PROBLEM - puppet last run on ms-be2019 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[19:03:47] <icinga-wm>	 PROBLEM - puppet last run on cp3032 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[19:03:47] <icinga-wm>	 PROBLEM - puppet last run on cp3003 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[19:03:47] <icinga-wm>	 PROBLEM - puppet last run on cp4013 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[19:03:48] <icinga-wm>	 PROBLEM - puppet last run on analytics1034 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[19:03:59] <icinga-wm>	 PROBLEM - puppet last run on maps2004 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[19:04:10] <icinga-wm>	 PROBLEM - puppet last run on mw1221 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[19:04:13] <icinga-wm>	 PROBLEM - puppet last run on mw1259 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[19:04:27] <icinga-wm>	 PROBLEM - puppet last run on dbproxy1003 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[19:04:29] <icinga-wm>	 PROBLEM - puppet last run on elastic2019 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[19:04:29] <icinga-wm>	 PROBLEM - puppet last run on mw1303 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[19:04:29] <icinga-wm>	 PROBLEM - puppet last run on mw1287 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[19:04:29] <icinga-wm>	 PROBLEM - puppet last run on analytics1045 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[19:04:29] <icinga-wm>	 PROBLEM - puppet last run on mc2013 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[19:04:30] <icinga-wm>	 PROBLEM - puppet last run on rhenium is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[19:04:30] <icinga-wm>	 PROBLEM - puppet last run on kafka2002 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[19:04:37] <icinga-wm>	 PROBLEM - puppet last run on mw1288 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[19:04:38] <icinga-wm>	 PROBLEM - puppet last run on kafka1012 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[19:04:38] <icinga-wm>	 PROBLEM - SSH on rhodium is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[19:04:48] <icinga-wm>	 PROBLEM - puppet last run on mw2156 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[19:04:48] <icinga-wm>	 RECOVERY - puppet last run on conf2002 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[19:04:48] <icinga-wm>	 RECOVERY - puppet last run on conf2001 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[19:04:48] <icinga-wm>	 PROBLEM - puppet last run on es2015 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[19:04:49] <icinga-wm>	 PROBLEM - puppet last run on db1091 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[19:05:02] <icinga-wm>	 PROBLEM - puppet last run on osmium is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[19:05:02] <icinga-wm>	 PROBLEM - puppet last run on rdb1001 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[19:05:08] <icinga-wm>	 PROBLEM - salt-minion processes on rhodium is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds.
[19:05:17] <icinga-wm>	 PROBLEM - puppet last run on mw1258 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[19:05:18] <icinga-wm>	 PROBLEM - puppet last run on hafnium is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[19:05:19] <icinga-wm>	 PROBLEM - puppet last run on mw2181 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[19:05:19] <icinga-wm>	 PROBLEM - puppet last run on mw2169 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[19:05:19] <icinga-wm>	 PROBLEM - puppet last run on mw2226 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[19:05:19] <icinga-wm>	 PROBLEM - puppet last run on db1094 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[19:05:19] <icinga-wm>	 PROBLEM - puppet last run on mw2232 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[19:05:28] <icinga-wm>	 PROBLEM - puppet last run on rdb2004 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[19:05:29] <icinga-wm>	 PROBLEM - puppet last run on mw1234 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[19:05:29] <icinga-wm>	 PROBLEM - puppet last run on db1090 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[19:05:37] <icinga-wm>	 PROBLEM - puppet last run on ms-be1008 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[19:05:37] <icinga-wm>	 PROBLEM - puppet last run on mw1301 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[19:05:38] <icinga-wm>	 PROBLEM - puppet last run on mw1212 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[19:05:38] <icinga-wm>	 PROBLEM - puppet last run on rcs1002 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[19:05:38] <icinga-wm>	 PROBLEM - puppet last run on mw2155 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[19:05:38] <icinga-wm>	 PROBLEM - puppet last run on mc1010 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[19:05:38] <icinga-wm>	 PROBLEM - puppet last run on cp1059 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[19:05:39] <icinga-wm>	 PROBLEM - puppet last run on wtp2003 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[19:05:47] <icinga-wm>	 PROBLEM - puppet last run on es2019 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[19:05:48] <icinga-wm>	 PROBLEM - puppet last run on mw1239 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[19:05:48] <icinga-wm>	 PROBLEM - puppet last run on labservices1002 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[19:05:48] <icinga-wm>	 PROBLEM - puppet last run on cp3015 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[19:05:49] <icinga-wm>	 PROBLEM - puppet last run on ganeti1003 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 5 minutes ago with 1 failures. Failed resources (up to 3 shown): File[/usr/lib/nagios/plugins/check-fresh-files-in-dir.py]
[19:06:00] <icinga-wm>	 PROBLEM - puppet last run on wtp1011 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[19:06:08] <icinga-wm>	 PROBLEM - puppet last run on maps-test2004 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[19:06:18] <icinga-wm>	 PROBLEM - puppet last run on eeden is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[19:06:18] <icinga-wm>	 PROBLEM - puppet last run on mw1192 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[19:06:19] <icinga-wm>	 PROBLEM - puppet last run on cp1060 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[19:06:19] <icinga-wm>	 PROBLEM - puppet last run on mw1286 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[19:06:28] <icinga-wm>	 PROBLEM - puppet last run on mc2008 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[19:06:29] <icinga-wm>	 PROBLEM - puppet last run on cp2009 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[19:06:29] <icinga-wm>	 PROBLEM - puppet last run on restbase2003 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[19:06:29] <icinga-wm>	 PROBLEM - puppet last run on rdb1004 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[19:06:29] <icinga-wm>	 PROBLEM - puppet last run on labnodepool1001 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[19:06:40] <icinga-wm>	 PROBLEM - puppet last run on mw1216 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[19:06:41] <icinga-wm>	 PROBLEM - puppet last run on mw1264 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[19:06:43] <icinga-wm>	 PROBLEM - puppet last run on wtp1023 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[19:06:43] <icinga-wm>	 PROBLEM - puppet last run on cp2024 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[19:06:44] <icinga-wm>	 PROBLEM - puppet last run on elastic2001 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[19:06:48] <icinga-wm>	 PROBLEM - puppet last run on mw1248 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[19:06:57] <icinga-wm>	 PROBLEM - puppet last run on radium is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 5 minutes ago with 1 failures. Failed resources (up to 3 shown): File[/usr/local/bin/phaste]
[19:06:59] <icinga-wm>	 PROBLEM - puppet last run on elastic1031 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[19:07:00] <icinga-wm>	 PROBLEM - puppet last run on analytics1042 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[19:07:01] <icinga-wm>	 PROBLEM - puppet last run on nescio is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[19:07:08] <icinga-wm>	 PROBLEM - puppet last run on logstash1003 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[19:07:08] <icinga-wm>	 PROBLEM - puppet last run on graphite1001 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[19:07:18] <icinga-wm>	 PROBLEM - puppet last run on mw1306 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[19:07:18] <icinga-wm>	 PROBLEM - puppet last run on db2063 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[19:07:19] <icinga-wm>	 PROBLEM - puppet last run on auth2001 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[19:07:19] <icinga-wm>	 PROBLEM - puppet last run on mw2215 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[19:07:19] <icinga-wm>	 PROBLEM - puppet last run on ganeti2006 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[19:07:19] <icinga-wm>	 PROBLEM - puppet last run on es1019 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[19:07:19] <icinga-wm>	 PROBLEM - puppet last run on es2017 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[19:07:20] <icinga-wm>	 PROBLEM - puppet last run on cerium is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[19:07:20] <icinga-wm>	 PROBLEM - puppet last run on mw2108 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[19:07:27] <icinga-wm>	 RECOVERY - puppetmaster backend https on rhodium is OK: HTTP OK: Status line output matched 400 - 333 bytes in 7.128 second response time
[19:07:28] <icinga-wm>	 RECOVERY - salt-minion processes on rhodium is OK: PROCS OK: 1 process with regex args ^/usr/bin/python /usr/bin/salt-minion
[19:07:28] <icinga-wm>	 PROBLEM - puppet last run on carbon is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[19:07:29] <icinga-wm>	 PROBLEM - puppet last run on mw1167 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[19:07:38] <icinga-wm>	 PROBLEM - puppet last run on cp2004 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[19:07:50] <icinga-wm>	 PROBLEM - puppet last run on mw2148 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[19:07:51] <icinga-wm>	 PROBLEM - puppet last run on kraz is CRITICAL: CRITICAL: Puppet has 2 failures. Last run 6 minutes ago with 2 failures. Failed resources (up to 3 shown): File[/usr/lib/nagios/plugins/check-fresh-files-in-dir.py],File[/usr/local/bin/puppet-enabled]
[19:07:51] <icinga-wm>	 PROBLEM - puppet last run on mw2180 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[19:07:58] <icinga-wm>	 PROBLEM - puppet last run on wtp2013 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[19:07:59] <icinga-wm>	 PROBLEM - puppet last run on mw2154 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[19:08:08] <icinga-wm>	 PROBLEM - puppet last run on mc1005 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 5 minutes ago with 1 failures. Failed resources (up to 3 shown): File[/usr/lib/nagios/plugins/check-fresh-files-in-dir.py]
[19:08:08] <icinga-wm>	 PROBLEM - puppet last run on mw2124 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[19:08:09] <icinga-wm>	 PROBLEM - puppet last run on analytics1027 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[19:08:17] <icinga-wm>	 PROBLEM - puppet last run on xenon is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[19:08:20] <icinga-wm>	 PROBLEM - puppet last run on mw1164 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[19:08:30] <icinga-wm>	 PROBLEM - puppet last run on lvs1010 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[19:08:32] <icinga-wm>	 PROBLEM - puppet last run on ganeti1001 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[19:08:32] <icinga-wm>	 PROBLEM - puppet last run on mw2205 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[19:08:37] <icinga-wm>	 PROBLEM - puppet last run on cp2005 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[19:08:37] <icinga-wm>	 PROBLEM - puppet last run on kafka1022 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[19:08:38] <icinga-wm>	 PROBLEM - puppet last run on es2001 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[19:08:38] <icinga-wm>	 PROBLEM - puppet last run on mw2216 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[19:08:38] <icinga-wm>	 PROBLEM - puppet last run on mw2239 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[19:08:38] <icinga-wm>	 PROBLEM - puppet last run on restbase-test2001 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[19:08:38] <icinga-wm>	 PROBLEM - puppet last run on cp1074 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 5 minutes ago with 1 failures. Failed resources (up to 3 shown): File[/etc/logrotate.d/varnishkafka]
[19:08:39] <icinga-wm>	 PROBLEM - puppet last run on ms-be1026 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[19:08:39] <icinga-wm>	 PROBLEM - puppet last run on db2053 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[19:09:00] <icinga-wm>	 PROBLEM - puppet last run on mw1223 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[19:09:00] <icinga-wm>	 PROBLEM - puppet last run on db1061 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[19:09:00] <icinga-wm>	 PROBLEM - puppet last run on db1038 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[19:09:07] <icinga-wm>	 PROBLEM - puppet last run on labvirt1002 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[19:09:07] <icinga-wm>	 PROBLEM - puppet last run on mw2230 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[19:09:08] <icinga-wm>	 PROBLEM - puppet last run on mw1250 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[19:09:22] <icinga-wm>	 PROBLEM - puppet last run on oresrdb1002 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 6 minutes ago with 1 failures. Failed resources (up to 3 shown): File[/usr/lib/nagios/plugins/check-fresh-files-in-dir.py]
[19:09:23] <icinga-wm>	 PROBLEM - puppet last run on mw1099 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[19:09:27] <icinga-wm>	 PROBLEM - puppet last run on mw1277 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[19:09:28] <icinga-wm>	 PROBLEM - puppet last run on mw1272 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[19:09:28] <icinga-wm>	 PROBLEM - puppet last run on db2012 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[19:09:28] <icinga-wm>	 PROBLEM - puppet last run on mw1242 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[19:09:29] <icinga-wm>	 PROBLEM - puppet last run on lvs1008 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[19:09:47] <icinga-wm>	 PROBLEM - puppet last run on elastic1033 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[19:09:48] <icinga-wm>	 PROBLEM - puppet last run on wtp1015 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 6 minutes ago with 1 failures. Failed resources (up to 3 shown): File[/usr/lib/nagios/plugins/check-fresh-files-in-dir.py]
[19:09:48] <icinga-wm>	 PROBLEM - puppet last run on labstore1003 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[19:09:49] <icinga-wm>	 PROBLEM - puppet last run on scandium is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[19:09:50] <icinga-wm>	 PROBLEM - puppet last run on mw2173 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[19:09:50] <icinga-wm>	 PROBLEM - puppet last run on radon is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[19:09:50] <icinga-wm>	 PROBLEM - puppet last run on pollux is CRITICAL: CRITICAL: Puppet has 3 failures. Last run 6 minutes ago with 3 failures. Failed resources (up to 3 shown): File[/usr/lib/nagios/plugins/check-fresh-files-in-dir.py],File[/usr/local/bin/puppet-enabled],File[/usr/lib/nagios/plugins/check_sysctl]
[19:09:51] <icinga-wm>	 PROBLEM - puppet last run on baham is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[19:09:57] <icinga-wm>	 PROBLEM - puppet last run on mw2066 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[19:09:57] <icinga-wm>	 PROBLEM - puppet last run on db1080 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 7 minutes ago with 1 failures. Failed resources (up to 3 shown): File[/usr/lib/nagios/plugins/check-fresh-files-in-dir.py]
[19:09:58] <icinga-wm>	 PROBLEM - puppet last run on labsdb1003 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[19:10:08] <icinga-wm>	 PROBLEM - puppet last run on mw2206 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[19:10:09] <icinga-wm>	 PROBLEM - puppet last run on db1078 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[19:10:17] <icinga-wm>	 PROBLEM - puppet last run on cp4012 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[19:10:17] <icinga-wm>	 PROBLEM - puppet last run on lvs1012 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[19:10:19] <icinga-wm>	 PROBLEM - puppet last run on logstash1004 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[19:10:20] <icinga-wm>	 PROBLEM - DPKG on rhodium is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds.
[19:10:29] <icinga-wm>	 PROBLEM - puppet last run on nitrogen is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[19:10:30] <icinga-wm>	 PROBLEM - puppet last run on db2010 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[19:10:38] <icinga-wm>	 PROBLEM - puppet last run on mw1268 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[19:10:38] <icinga-wm>	 PROBLEM - puppet last run on elastic2023 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[19:10:47] <icinga-wm>	 PROBLEM - puppet last run on db2018 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[19:11:07] <icinga-wm>	 PROBLEM - puppet last run on lvs2001 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[19:11:08] <icinga-wm>	 PROBLEM - puppet last run on mw2218 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[19:11:18] <icinga-wm>	 PROBLEM - puppet last run on lvs4004 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[19:11:18] <icinga-wm>	 PROBLEM - puppet last run on cp3041 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[19:11:18] <icinga-wm>	 PROBLEM - puppet last run on cp3014 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[19:11:18] <icinga-wm>	 PROBLEM - puppet last run on cp3004 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[19:11:18] <icinga-wm>	 PROBLEM - puppet last run on lvs3002 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[19:11:19] <icinga-wm>	 PROBLEM - puppet last run on db1085 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[19:11:29] <icinga-wm>	 RECOVERY - configured eth on rhodium is OK: OK - interfaces up
[19:11:38] <icinga-wm>	 PROBLEM - puppet last run on restbase-test2003 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[19:11:48] <icinga-wm>	 PROBLEM - puppet last run on mw1162 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[19:11:58] <icinga-wm>	 PROBLEM - puppet last run on db2017 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[19:11:59] <icinga-wm>	 PROBLEM - puppet last run on elastic1022 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[19:12:00] <icinga-wm>	 PROBLEM - puppet last run on mw1237 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[19:12:19] <icinga-wm>	 PROBLEM - puppet last run on cp1046 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[19:12:19] <icinga-wm>	 PROBLEM - puppet last run on mw2224 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[19:12:19] <icinga-wm>	 PROBLEM - puppet last run on mw2119 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[19:12:27] <icinga-wm>	 PROBLEM - puppet last run on mw1017 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[19:12:29] <icinga-wm>	 PROBLEM - puppet last run on lvs2005 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[19:12:29] <icinga-wm>	 PROBLEM - puppet last run on db1092 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[19:12:29] <icinga-wm>	 PROBLEM - puppet last run on analytics1049 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[19:12:29] <icinga-wm>	 PROBLEM - puppet last run on conf1002 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[19:12:29] <icinga-wm>	 PROBLEM - puppet last run on es2004 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[19:12:48] <icinga-wm>	 RECOVERY - DPKG on rhodium is OK: All packages OK
[19:12:48] <icinga-wm>	 PROBLEM - puppet last run on lvs1006 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[19:12:58] <icinga-wm>	 PROBLEM - puppet last run on cp3022 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[19:12:58] <icinga-wm>	 PROBLEM - puppet last run on cp3044 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[19:12:59] <icinga-wm>	 PROBLEM - puppet last run on es1015 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[19:12:59] <icinga-wm>	 PROBLEM - puppet last run on db1048 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[19:13:08] <icinga-wm>	 PROBLEM - puppet last run on elastic2016 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[19:13:09] <icinga-wm>	 PROBLEM - puppet last run on mw2165 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[19:13:38] <icinga-wm>	 PROBLEM - puppet last run on mw2222 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[19:13:38] <icinga-wm>	 PROBLEM - puppet last run on mw1273 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[19:13:39] <icinga-wm>	 PROBLEM - puppet last run on puppetmaster2002 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[19:13:39] <icinga-wm>	 PROBLEM - puppet last run on mw2149 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[19:13:47] <icinga-wm>	 PROBLEM - puppet last run on lvs4003 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[19:13:47] <icinga-wm>	 PROBLEM - puppet last run on cp3033 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[19:13:47] <icinga-wm>	 PROBLEM - puppet last run on cp3043 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[19:13:47] <icinga-wm>	 PROBLEM - puppet last run on cp3005 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[19:13:48] <icinga-wm>	 PROBLEM - puppet last run on db1019 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[19:13:49] <icinga-wm>	 PROBLEM - puppet last run on mc1009 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 6 minutes ago with 1 failures. Failed resources (up to 3 shown): File[/usr/lib/nagios/plugins/check-fresh-files-in-dir.py]
[19:13:59] <icinga-wm>	 PROBLEM - puppet last run on elastic1044 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[19:14:28] <icinga-wm>	 PROBLEM - puppet last run on cp1051 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[19:14:29] <icinga-wm>	 PROBLEM - puppet last run on mw1232 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[19:14:29] <icinga-wm>	 PROBLEM - puppet last run on wtp2007 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[19:14:30] <icinga-wm>	 RECOVERY - SSH on rhodium is OK: SSH OK - OpenSSH_6.7p1 Debian-5+deb8u3 (protocol 2.0)
[19:14:37] <icinga-wm>	 PROBLEM - puppet last run on es1012 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[19:14:37] <icinga-wm>	 PROBLEM - puppet last run on ganeti2005 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[19:14:38] <icinga-wm>	 PROBLEM - puppet last run on aqs1006 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[19:14:38] <icinga-wm>	 PROBLEM - puppet last run on restbase1010 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[19:14:38] <icinga-wm>	 PROBLEM - puppet last run on conf1003 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[19:14:38] <icinga-wm>	 PROBLEM - puppet last run on elastic1036 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[19:14:39] <icinga-wm>	 PROBLEM - puppet last run on es1017 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[19:14:39] <icinga-wm>	 PROBLEM - puppet last run on db1083 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[19:14:47] <icinga-wm>	 PROBLEM - puppet last run on dbproxy1004 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[19:14:47] <icinga-wm>	 PROBLEM - puppet last run on db1069 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[19:14:47] <icinga-wm>	 PROBLEM - puppet last run on es1013 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[19:14:48] <icinga-wm>	 PROBLEM - puppet last run on ms-be2001 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[19:14:48] <icinga-wm>	 PROBLEM - puppet last run on mw2238 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[19:14:48] <hashar>	 !log Puppet is falling since ~ 18:05 UTC.  At least a couple european ops are looking at it
[19:14:48] <icinga-wm>	 PROBLEM - puppet last run on mw2089 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[19:14:49] <icinga-wm>	 PROBLEM - puppet last run on mx2001 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[19:14:49] <icinga-wm>	 PROBLEM - puppet last run on wtp2014 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[19:14:49] <icinga-wm>	 PROBLEM - puppet last run on mw2099 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[19:14:50] <icinga-wm>	 PROBLEM - puppet last run on mw2102 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[19:14:50] <icinga-wm>	 PROBLEM - puppet last run on mw2151 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[19:14:51] <icinga-wm>	 PROBLEM - puppet last run on mw1188 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[19:14:53] <morebots>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master
[19:14:58] <icinga-wm>	 PROBLEM - puppetmaster backend https on rhodium is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[19:14:58] <icinga-wm>	 PROBLEM - puppet last run on ms-be1012 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[19:14:58] <icinga-wm>	 PROBLEM - puppet last run on mw1161 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[19:14:58] <icinga-wm>	 PROBLEM - puppet last run on db1020 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[19:15:08] <icinga-wm>	 PROBLEM - puppet last run on labsdb1006 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[19:15:19] <icinga-wm>	 PROBLEM - puppet last run on elastic1026 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[19:15:19] <icinga-wm>	 PROBLEM - puppet last run on dbstore1001 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[19:15:19] <icinga-wm>	 PROBLEM - puppet last run on es1018 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[19:15:28] <icinga-wm>	 PROBLEM - puppet last run on mw2183 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[19:15:28] <icinga-wm>	 PROBLEM - puppet last run on mw2116 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[19:15:29] <icinga-wm>	 PROBLEM - puppet last run on mw1283 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[19:15:38] <icinga-wm>	 PROBLEM - puppet last run on db2069 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[19:15:39] <icinga-wm>	 PROBLEM - puppet last run on rdb1002 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[19:15:39] <icinga-wm>	 PROBLEM - puppet last run on mw2112 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[19:15:39] <icinga-wm>	 PROBLEM - puppet last run on mw2103 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[19:15:56] <icinga-wm>	 PROBLEM - puppet last run on cp1072 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[19:15:57] <icinga-wm>	 PROBLEM - MD RAID on rhodium is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds.
[19:16:08] <icinga-wm>	 PROBLEM - puppet last run on db1001 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[19:16:08] <icinga-wm>	 PROBLEM - puppet last run on druid1003 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[19:16:08] <icinga-wm>	 PROBLEM - puppet last run on cp2021 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[19:16:08] <icinga-wm>	 PROBLEM - puppet last run on elastic1037 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[19:16:08] <icinga-wm>	 PROBLEM - puppet last run on californium is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[19:16:09] <icinga-wm>	 PROBLEM - puppet last run on labsdb1004 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[19:16:09] <icinga-wm>	 PROBLEM - puppet last run on maps-test2002 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[19:16:09] <icinga-wm>	 PROBLEM - puppet last run on restbase2007 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[19:16:10] <icinga-wm>	 PROBLEM - puppet last run on mw1269 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[19:16:10] <icinga-wm>	 PROBLEM - puppet last run on elastic2009 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[19:16:11] <icinga-wm>	 PROBLEM - puppet last run on cp3034 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[19:16:11] <icinga-wm>	 PROBLEM - puppet last run on cp3035 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[19:16:18] <icinga-wm>	 PROBLEM - puppet last run on mc1004 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[19:16:18] <icinga-wm>	 PROBLEM - puppet last run on labvirt1011 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[19:16:28] <icinga-wm>	 PROBLEM - puppet last run on eventlog1001 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[19:16:29] <icinga-wm>	 PROBLEM - puppet last run on mc2009 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[19:16:29] <icinga-wm>	 PROBLEM - puppet last run on restbase2005 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[19:16:41] <icinga-wm>	 PROBLEM - puppet last run on analytics1015 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[19:16:41] <icinga-wm>	 PROBLEM - puppet last run on cp2007 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[19:16:41] <icinga-wm>	 PROBLEM - puppet last run on dbproxy1010 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[19:16:49] <icinga-wm>	 PROBLEM - puppet last run on mw1290 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[19:16:50] <icinga-wm>	 PROBLEM - puppet last run on mc2002 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[19:17:08] <icinga-wm>	 PROBLEM - puppet last run on elastic1047 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[19:17:17] <icinga-wm>	 PROBLEM - puppet last run on db1029 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[19:17:18] <icinga-wm>	 PROBLEM - puppet last run on ganeti2001 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[19:17:18] <icinga-wm>	 PROBLEM - puppet last run on mw2246 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[19:17:18] <icinga-wm>	 PROBLEM - puppet last run on mw2220 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[19:17:18] <icinga-wm>	 PROBLEM - puppet last run on wtp2020 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[19:17:18] <icinga-wm>	 PROBLEM - puppet last run on wtp2005 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[19:17:19] <icinga-wm>	 PROBLEM - puppet last run on planet1001 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[19:17:19] <icinga-wm>	 PROBLEM - puppet last run on db2046 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[19:17:19] <icinga-wm>	 PROBLEM - puppet last run on ganeti1004 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[19:17:48] <icinga-wm>	 PROBLEM - puppet last run on db1088 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[19:17:57] <icinga-wm>	 PROBLEM - puppet last run on wtp2002 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[19:17:58] <icinga-wm>	 PROBLEM - puppet last run on logstash1001 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[19:18:23] <icinga-wm>	 PROBLEM - puppet last run on dbproxy1002 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[19:18:24] <icinga-wm>	 PROBLEM - puppet last run on db1033 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[19:18:27] <icinga-wm>	 RECOVERY - MD RAID on rhodium is OK: OK: Active: 4, Working: 4, Failed: 0, Spare: 0
[19:18:38] <icinga-wm>	 PROBLEM - puppet last run on db2070 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[19:18:38] <icinga-wm>	 PROBLEM - puppet last run on restbase-test2002 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[19:18:38] <icinga-wm>	 PROBLEM - puppet last run on rdb2006 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[19:20:09] <icinga-wm>	 RECOVERY - puppet last run on aluminium is OK: OK: Puppet is currently enabled, last run 3 seconds ago with 0 failures
[19:21:09] <icinga-wm>	 PROBLEM - puppet last run on etherpad1001 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[19:21:28] <icinga-wm>	 PROBLEM - configured eth on rhodium is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds.
[19:24:38] <icinga-wm>	 PROBLEM - puppet last run on scb1002 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[19:25:57] <icinga-wm>	 PROBLEM - MD RAID on rhodium is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds.
[19:28:27] <icinga-wm>	 PROBLEM - Disk space on rhodium is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds.
[19:28:27] <icinga-wm>	 PROBLEM - Check size of conntrack table on rhodium is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds.
[19:28:27] <icinga-wm>	 PROBLEM - puppetmaster backend https on palladium is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[19:28:40] <icinga-wm>	 PROBLEM - puppet last run on cp3019 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[19:28:59] <icinga-wm>	 RECOVERY - configured eth on rhodium is OK: OK - interfaces up
[19:29:38] <icinga-wm>	 PROBLEM - puppet last run on wtp1010 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[19:30:37] <icinga-wm>	 PROBLEM - puppet last run on wtp1016 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[19:30:38] <icinga-wm>	 RECOVERY - puppetmaster backend https on palladium is OK: HTTP OK: Status line output matched 400 - 378 bytes in 0.327 second response time
[19:30:39] <icinga-wm>	 RECOVERY - Disk space on rhodium is OK: DISK OK
[19:30:40] <grrrit-wm>	 (03PS3) 10MarcoAurelio: New 'engineer' group for ruwiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/308448 (https://phabricator.wikimedia.org/T144599) 
[19:30:51] <icinga-wm>	 RECOVERY - Check size of conntrack table on rhodium is OK: OK: nf_conntrack is 0 % full
[19:31:18] <icinga-wm>	 PROBLEM - puppet last run on rdb1005 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[19:31:29] <icinga-wm>	 PROBLEM - puppet last run on einsteinium is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[19:31:50] <icinga-wm>	 PROBLEM - dhclient process on rhodium is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds.
[19:32:18] <icinga-wm>	 PROBLEM - puppet last run on wtp1012 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[19:32:28] <icinga-wm>	 PROBLEM - puppet last run on wtp1005 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[19:32:47] <elukey>	 !log restarting apache2 on rhodium (attempt to fix it)
[19:32:49] <icinga-wm>	 PROBLEM - puppet last run on ununpentium is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[19:32:52] <morebots>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master
[19:33:07] <icinga-wm>	 PROBLEM - puppet last run on alsafi is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[19:33:27] <icinga-wm>	 RECOVERY - MD RAID on rhodium is OK: OK: Active: 4, Working: 4, Failed: 0, Spare: 0
[19:33:59] <icinga-wm>	 RECOVERY - Unmerged changes on repository puppet on rhodium is OK: No changes to merge.
[19:34:11] <icinga-wm>	 RECOVERY - dhclient process on rhodium is OK: PROCS OK: 0 processes with command name dhclient
[19:34:48] <icinga-wm>	 RECOVERY - puppetmaster backend https on rhodium is OK: HTTP OK: Status line output matched 400 - 332 bytes in 0.179 second response time
[19:35:15] <grrrit-wm>	 (03CR) 10MarcoAurelio: "> > Code looks OK to me, but why do we need an autopatrolled group" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/303183 (owner: 10BryanDavis)
[19:35:18] <icinga-wm>	 RECOVERY - puppet last run on es2014 is OK: OK: Puppet is currently enabled, last run 7 seconds ago with 0 failures
[19:35:18] <icinga-wm>	 RECOVERY - puppet last run on wtp1014 is OK: OK: Puppet is currently enabled, last run 15 seconds ago with 0 failures
[19:35:29] <icinga-wm>	 RECOVERY - puppet last run on mc1010 is OK: OK: Puppet is currently enabled, last run 15 seconds ago with 0 failures
[19:35:33] <Luke081515>	 elukey, nice, seems like that fixed it
[19:35:37] <icinga-wm>	 RECOVERY - puppet last run on db2060 is OK: OK: Puppet is currently enabled, last run 28 seconds ago with 0 failures
[19:35:38] <icinga-wm>	 RECOVERY - puppet last run on cp1064 is OK: OK: Puppet is currently enabled, last run 14 seconds ago with 0 failures
[19:35:38] <icinga-wm>	 RECOVERY - puppet last run on elastic1043 is OK: OK: Puppet is currently enabled, last run 4 seconds ago with 0 failures
[19:35:38] <icinga-wm>	 RECOVERY - puppet last run on praseodymium is OK: OK: Puppet is currently enabled, last run 26 seconds ago with 0 failures
[19:35:39] <icinga-wm>	 RECOVERY - puppet last run on restbase1012 is OK: OK: Puppet is currently enabled, last run 0 seconds ago with 0 failures
[19:35:46] <elukey>	 Luke081515: [Sun Sep 04 19:33:37.136665 2016] [mpm_prefork:error] [pid 10776] AH00161: server reached MaxRequestWorkers setting, consider raising the MaxRequestWorkers setting
[19:35:50] <elukey>	 this is what I found
[19:35:51] <icinga-wm>	 RECOVERY - puppet last run on maps2003 is OK: OK: Puppet is currently enabled, last run 29 seconds ago with 0 failures
[19:35:53] <icinga-wm>	 RECOVERY - puppet last run on iron is OK: OK: Puppet is currently enabled, last run 28 seconds ago with 0 failures
[19:35:56] <icinga-wm>	 RECOVERY - puppet last run on cp3007 is OK: OK: Puppet is currently enabled, last run 19 seconds ago with 0 failures
[19:35:56] <icinga-wm>	 RECOVERY - puppet last run on db2044 is OK: OK: Puppet is currently enabled, last run 6 seconds ago with 0 failures
[19:36:07] <icinga-wm>	 RECOVERY - puppet last run on kafka1022 is OK: OK: Puppet is currently enabled, last run 36 seconds ago with 0 failures
[19:36:07] <icinga-wm>	 RECOVERY - puppet last run on cp2001 is OK: OK: Puppet is currently enabled, last run 46 seconds ago with 0 failures
[19:36:07] <icinga-wm>	 RECOVERY - puppet last run on cp2005 is OK: OK: Puppet is currently enabled, last run 39 seconds ago with 0 failures
[19:36:08] <icinga-wm>	 RECOVERY - puppet last run on dbstore2002 is OK: OK: Puppet is currently enabled, last run 25 seconds ago with 0 failures
[19:36:08] <icinga-wm>	 RECOVERY - puppet last run on mc2015 is OK: OK: Puppet is currently enabled, last run 59 seconds ago with 0 failures
[19:36:08] <icinga-wm>	 RECOVERY - puppet last run on wtp2017 is OK: OK: Puppet is currently enabled, last run 57 seconds ago with 0 failures
[19:36:09] <Luke081515>	 ah
[19:36:09] <icinga-wm>	 RECOVERY - puppet last run on lvs4004 is OK: OK: Puppet is currently enabled, last run 24 seconds ago with 0 failures
[19:36:09] <icinga-wm>	 RECOVERY - puppet last run on cp4016 is OK: OK: Puppet is currently enabled, last run 33 seconds ago with 0 failures
[19:36:09] <icinga-wm>	 RECOVERY - puppet last run on cp3014 is OK: OK: Puppet is currently enabled, last run 22 seconds ago with 0 failures
[19:36:10] <icinga-wm>	 RECOVERY - puppet last run on cp3036 is OK: OK: Puppet is currently enabled, last run 38 seconds ago with 0 failures
[19:36:10] <icinga-wm>	 RECOVERY - puppet last run on aqs1003 is OK: OK: Puppet is currently enabled, last run 31 seconds ago with 0 failures
[19:36:17] <icinga-wm>	 RECOVERY - puppet last run on cp1060 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[19:36:18] <icinga-wm>	 RECOVERY - puppet last run on mw1289 is OK: OK: Puppet is currently enabled, last run 54 seconds ago with 0 failures
[19:36:18] <icinga-wm>	 RECOVERY - puppet last run on aqs1004 is OK: OK: Puppet is currently enabled, last run 54 seconds ago with 0 failures
[19:36:18] <icinga-wm>	 RECOVERY - puppet last run on db1085 is OK: OK: Puppet is currently enabled, last run 21 seconds ago with 0 failures
[19:36:18] <icinga-wm>	 RECOVERY - puppet last run on mw1177 is OK: OK: Puppet is currently enabled, last run 4 seconds ago with 0 failures
[19:36:19] <icinga-wm>	 RECOVERY - puppet last run on dbstore1002 is OK: OK: Puppet is currently enabled, last run 56 seconds ago with 0 failures
[19:36:19] <icinga-wm>	 RECOVERY - puppet last run on elastic1021 is OK: OK: Puppet is currently enabled, last run 52 seconds ago with 0 failures
[19:36:19] <icinga-wm>	 RECOVERY - puppet last run on db1073 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[19:36:19] <icinga-wm>	 RECOVERY - puppet last run on logstash1006 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[19:36:24] <Luke081515>	 elukey, can you quiet icinga, till all recovery is posted here?
[19:36:28] <icinga-wm>	 RECOVERY - puppet last run on db2067 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[19:36:28] <icinga-wm>	 RECOVERY - puppet last run on restbase-test2003 is OK: OK: Puppet is currently enabled, last run 31 seconds ago with 0 failures
[19:36:29] <icinga-wm>	 RECOVERY - puppet last run on mw1218 is OK: OK: Puppet is currently enabled, last run 14 seconds ago with 0 failures
[19:36:29] <icinga-wm>	 PROBLEM - puppet last run on wtp1009 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[19:36:38] <icinga-wm>	 RECOVERY - puppet last run on graphite1002 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[19:36:40] <icinga-wm>	 RECOVERY - puppet last run on restbase1014 is OK: OK: Puppet is currently enabled, last run 57 seconds ago with 0 failures
[19:36:41] <icinga-wm>	 RECOVERY - puppet last run on mw1169 is OK: OK: Puppet is currently enabled, last run 15 seconds ago with 0 failures
[19:36:41] <icinga-wm>	 RECOVERY - puppetmaster https on palladium is OK: HTTP OK: Status line output matched 400 - 353 bytes in 0.081 second response time
[19:36:41] <icinga-wm>	 RECOVERY - puppet last run on mw1099 is OK: OK: Puppet is currently enabled, last run 19 seconds ago with 0 failures
[19:36:41] <icinga-wm>	 RECOVERY - puppet last run on silver is OK: OK: Puppet is currently enabled, last run 34 seconds ago with 0 failures
[19:36:48] <icinga-wm>	 RECOVERY - puppet last run on mw1277 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[19:36:48] <icinga-wm>	 RECOVERY - puppet last run on analytics1045 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[19:36:48] <icinga-wm>	 RECOVERY - puppet last run on mw1272 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[19:36:48] <icinga-wm>	 RECOVERY - puppet last run on elastic1022 is OK: OK: Puppet is currently enabled, last run 47 seconds ago with 0 failures
[19:36:49] <icinga-wm>	 RECOVERY - puppet last run on pc1006 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[19:36:49] <icinga-wm>	 RECOVERY - puppet last run on mw1242 is OK: OK: Puppet is currently enabled, last run 35 seconds ago with 0 failures
[19:36:49] <icinga-wm>	 RECOVERY - puppet last run on db2017 is OK: OK: Puppet is currently enabled, last run 26 seconds ago with 0 failures
[19:36:50] <icinga-wm>	 RECOVERY - puppet last run on mw1222 is OK: OK: Puppet is currently enabled, last run 35 seconds ago with 0 failures
[19:36:50] <icinga-wm>	 RECOVERY - puppet last run on labcontrol1002 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[19:36:50] <icinga-wm>	 RECOVERY - puppet last run on es2018 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[19:36:50] <icinga-wm>	 RECOVERY - puppet last run on mw1163 is OK: OK: Puppet is currently enabled, last run 22 seconds ago with 0 failures
[19:36:51] <icinga-wm>	 RECOVERY - puppet last run on mw2202 is OK: OK: Puppet is currently enabled, last run 18 seconds ago with 0 failures
[19:37:00] <grrrit-wm>	 (03CR) 10MarcoAurelio: "> > > Code looks OK to me, but why do we need an autopatrolled group" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/303183 (owner: 10BryanDavis)
[19:37:01] <icinga-wm>	 RECOVERY - puppet last run on cp2002 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[19:37:01] <icinga-wm>	 RECOVERY - puppet last run on cp4004 is OK: OK: Puppet is currently enabled, last run 20 seconds ago with 0 failures
[19:37:01] <icinga-wm>	 RECOVERY - puppet last run on kafka1012 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[19:37:01] <icinga-wm>	 RECOVERY - puppet last run on rhodium is OK: OK: Puppet is currently enabled, last run 52 seconds ago with 0 failures
[19:37:07] <icinga-wm>	 RECOVERY - puppet last run on elastic1033 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[19:37:07] <icinga-wm>	 RECOVERY - puppet last run on elastic1042 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[19:37:07] <icinga-wm>	 RECOVERY - puppet last run on snapshot1007 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[19:37:08] <icinga-wm>	 RECOVERY - puppet last run on lvs1003 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[19:37:08] <icinga-wm>	 RECOVERY - puppet last run on mw1208 is OK: OK: Puppet is currently enabled, last run 50 seconds ago with 0 failures
[19:37:08] <icinga-wm>	 RECOVERY - puppet last run on labstore1003 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[19:37:09] <elukey>	 Luke081515: I can check, never done it.. going to see on neon
[19:37:09] <icinga-wm>	 RECOVERY - puppet last run on mw2156 is OK: OK: Puppet is currently enabled, last run 38 seconds ago with 0 failures
[19:37:09] <icinga-wm>	 RECOVERY - puppet last run on mw2173 is OK: OK: Puppet is currently enabled, last run 38 seconds ago with 0 failures
[19:37:09] <icinga-wm>	 RECOVERY - puppet last run on radon is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[19:37:10] <icinga-wm>	 RECOVERY - puppet last run on mw2078 is OK: OK: Puppet is currently enabled, last run 22 seconds ago with 0 failures
[19:37:15] <Luke081515>	 thx :)
[19:37:18] <icinga-wm>	 RECOVERY - puppet last run on restbase2006 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[19:37:18] <icinga-wm>	 RECOVERY - puppet last run on ms-fe2001 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[19:37:18] <icinga-wm>	 PROBLEM - puppet last run on conf2002 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[19:37:18] <icinga-wm>	 RECOVERY - puppet last run on mw2080 is OK: OK: Puppet is currently enabled, last run 41 seconds ago with 0 failures
[19:37:18] <icinga-wm>	 RECOVERY - puppet last run on ganeti2006 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures
[19:37:18] <icinga-wm>	 RECOVERY - puppet last run on ganeti2003 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures
[19:37:18] <icinga-wm>	 RECOVERY - puppet last run on mw2118 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[19:37:19] <icinga-wm>	 RECOVERY - puppet last run on mw2250 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[19:37:19] <icinga-wm>	 RECOVERY - puppet last run on mw2231 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[19:37:20] <icinga-wm>	 RECOVERY - puppet last run on mw2066 is OK: OK: Puppet is currently enabled, last run 30 seconds ago with 0 failures
[19:37:20] <icinga-wm>	 RECOVERY - puppet last run on mw1184 is OK: OK: Puppet is currently enabled, last run 56 seconds ago with 0 failures
[19:37:21] <icinga-wm>	 RECOVERY - puppet last run on mw2088 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[19:37:21] <icinga-wm>	 RECOVERY - puppet last run on db2038 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[19:37:22] <icinga-wm>	 RECOVERY - puppet last run on pollux is OK: OK: Puppet is currently enabled, last run 14 seconds ago with 0 failures
[19:37:37] <icinga-wm>	 RECOVERY - puppet last run on labvirt1006 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[19:37:38] <icinga-wm>	 RECOVERY - puppet last run on cp4012 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures
[19:37:39] <icinga-wm>	 RECOVERY - puppet last run on logstash1004 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures
[19:37:39] <icinga-wm>	 RECOVERY - puppet last run on db2052 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[19:37:47] <icinga-wm>	 RECOVERY - puppet last run on mw2082 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[19:37:48] <icinga-wm>	 RECOVERY - puppet last run on mc2011 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[19:37:49] <icinga-wm>	 RECOVERY - puppet last run on mw2137 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[19:37:49] <icinga-wm>	 RECOVERY - puppet last run on mw2130 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[19:37:49] <icinga-wm>	 RECOVERY - puppet last run on nitrogen is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[19:37:49] <icinga-wm>	 RECOVERY - puppet last run on mw1175 is OK: OK: Puppet is currently enabled, last run 35 seconds ago with 0 failures
[19:37:50] <icinga-wm>	 RECOVERY - puppet last run on db2010 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[19:37:50] <icinga-wm>	 RECOVERY - puppet last run on es1015 is OK: OK: Puppet is currently enabled, last run 52 seconds ago with 0 failures
[19:37:57] <icinga-wm>	 RECOVERY - puppet last run on gallium is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures
[19:37:58] <icinga-wm>	 RECOVERY - puppet last run on mw1255 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[19:37:58] <icinga-wm>	 RECOVERY - puppet last run on cp3022 is OK: OK: Puppet is currently enabled, last run 8 seconds ago with 0 failures
[19:37:58] <icinga-wm>	 RECOVERY - puppet last run on db1052 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[19:37:59] <icinga-wm>	 RECOVERY - puppet last run on db2036 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[19:37:59] <icinga-wm>	 RECOVERY - puppet last run on mw2128 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[19:37:59] <icinga-wm>	 RECOVERY - puppet last run on db1037 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures
[19:37:59] <icinga-wm>	 RECOVERY - puppet last run on mw1268 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[19:38:00] <icinga-wm>	 RECOVERY - puppet last run on snapshot1005 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[19:38:00] <icinga-wm>	 RECOVERY - puppet last run on elastic2023 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures
[19:38:07] <icinga-wm>	 RECOVERY - puppet last run on mw2124 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[19:38:07] <icinga-wm>	 RECOVERY - puppet last run on elastic2003 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[19:38:08] <icinga-wm>	 PROBLEM - puppet last run on oxygen is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[19:38:08] <icinga-wm>	 RECOVERY - puppet last run on elastic2012 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures
[19:38:08] <icinga-wm>	 RECOVERY - puppet last run on elastic2016 is OK: OK: Puppet is currently enabled, last run 34 seconds ago with 0 failures
[19:38:08] <icinga-wm>	 RECOVERY - puppet last run on db2018 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures
[19:38:09] <icinga-wm>	 RECOVERY - puppet last run on mw1164 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[19:38:21] <icinga-wm>	 RECOVERY - puppet last run on elastic2011 is OK: OK: Puppet is currently enabled, last run 40 seconds ago with 0 failures
[19:38:24] <icinga-wm>	 RECOVERY - puppet last run on cp4001 is OK: OK: Puppet is currently enabled, last run 35 seconds ago with 0 failures
[19:38:25] <icinga-wm>	 RECOVERY - puppet last run on aqs1001 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures
[19:38:25] <icinga-wm>	 RECOVERY - puppet last run on labvirt1001 is OK: OK: Puppet is currently enabled, last run 22 seconds ago with 0 failures
[19:38:37] <icinga-wm>	 RECOVERY - puppet last run on mw2211 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[19:38:37] <icinga-wm>	 RECOVERY - puppet last run on lvs2001 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[19:38:37] <icinga-wm>	 RECOVERY - puppet last run on mw2076 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[19:38:37] <icinga-wm>	 RECOVERY - puppet last run on mw2062 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[19:38:37] <icinga-wm>	 RECOVERY - puppet last run on mw1273 is OK: OK: Puppet is currently enabled, last run 28 seconds ago with 0 failures
[19:38:38] <icinga-wm>	 RECOVERY - puppet last run on mw2017 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[19:38:38] <icinga-wm>	 RECOVERY - puppet last run on restbase2001 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures
[19:38:39] <icinga-wm>	 RECOVERY - puppet last run on mw2222 is OK: OK: Puppet is currently enabled, last run 22 seconds ago with 0 failures
[19:38:39] <icinga-wm>	 RECOVERY - puppet last run on mw2218 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures
[19:38:40] <icinga-wm>	 PROBLEM - puppet last run on mc2003 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[19:38:40] <icinga-wm>	 RECOVERY - puppet last run on puppetmaster2002 is OK: OK: Puppet is currently enabled, last run 12 seconds ago with 0 failures
[19:38:41] <icinga-wm>	 PROBLEM - puppet last run on hassaleh is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[19:38:41] <icinga-wm>	 RECOVERY - puppet last run on mw2149 is OK: OK: Puppet is currently enabled, last run 9 seconds ago with 0 failures
[19:38:42] <icinga-wm>	 RECOVERY - puppet last run on lvs4003 is OK: OK: Puppet is currently enabled, last run 44 seconds ago with 0 failures
[19:38:58] <icinga-wm>	 PROBLEM - puppet last run on conf2003 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[19:38:58] <icinga-wm>	 PROBLEM - puppet last run on pybal-test2001 is CRITICAL: CRITICAL: Puppet has 12 failures. Last run 5 minutes ago with 12 failures. Failed resources (up to 3 shown): File[/usr/lib/nagios/plugins/check-fresh-files-in-dir.py],File[/usr/local/bin/puppet-enabled],File[/usr/lib/nagios/plugins/check_sysctl],File[/etc/sysctl.d]
[19:39:10] <icinga-wm>	 PROBLEM - puppet last run on prometheus1001 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[19:39:12] <icinga-wm>	 RECOVERY - puppet last run on mw1162 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[19:39:12] <icinga-wm>	 RECOVERY - puppet last run on db2040 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures
[19:39:12] <icinga-wm>	 RECOVERY - puppet last run on oresrdb1002 is OK: OK: Puppet is currently enabled, last run 6 seconds ago with 0 failures
[19:39:12] <icinga-wm>	 RECOVERY - puppet last run on mw2135 is OK: OK: Puppet is currently enabled, last run 9 seconds ago with 0 failures
[19:39:19] <Luke081515>	 elukey: some problems again :-/ ^
[19:39:20] <icinga-wm>	 RECOVERY - puppet last run on cp1051 is OK: OK: Puppet is currently enabled, last run 0 seconds ago with 0 failures
[19:39:21] <icinga-wm>	 RECOVERY - puppet last run on mw1237 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[19:39:30] <icinga-wm>	 RECOVERY - puppet last run on db2042 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures
[19:39:30] <icinga-wm>	 RECOVERY - puppet last run on es1012 is OK: OK: Puppet is currently enabled, last run 58 seconds ago with 0 failures
[19:39:31] <icinga-wm>	 RECOVERY - puppet last run on cp2020 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[19:39:31] <icinga-wm>	 RECOVERY - puppet last run on wtp1018 is OK: OK: Puppet is currently enabled, last run 51 seconds ago with 0 failures
[19:39:37] <icinga-wm>	 RECOVERY - puppet last run on aqs1006 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[19:39:38] <icinga-wm>	 RECOVERY - puppet last run on conf1003 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures
[19:39:38] <icinga-wm>	 RECOVERY - puppet last run on elastic1036 is OK: OK: Puppet is currently enabled, last run 42 seconds ago with 0 failures
[19:39:38] <icinga-wm>	 RECOVERY - puppet last run on es1017 is OK: OK: Puppet is currently enabled, last run 56 seconds ago with 0 failures
[19:39:38] <icinga-wm>	 RECOVERY - puppet last run on dbproxy1004 is OK: OK: Puppet is currently enabled, last run 40 seconds ago with 0 failures
[19:39:38] <icinga-wm>	 RECOVERY - puppet last run on scb1002 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures
[19:39:38] <icinga-wm>	 RECOVERY - puppet last run on es1013 is OK: OK: Puppet is currently enabled, last run 1 second ago with 0 failures
[19:39:39] <icinga-wm>	 RECOVERY - puppet last run on elastic1032 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[19:39:39] <icinga-wm>	 RECOVERY - puppet last run on cp1046 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[19:39:47] <icinga-wm>	 RECOVERY - puppet last run on cp1050 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[19:39:48] <icinga-wm>	 RECOVERY - puppet last run on ms-be2001 is OK: OK: Puppet is currently enabled, last run 8 seconds ago with 0 failures
[19:39:48] <icinga-wm>	 PROBLEM - puppet last run on sinistra is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[19:39:48] <icinga-wm>	 PROBLEM - puppet last run on conf2001 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[19:39:48] <icinga-wm>	 PROBLEM - puppet last run on scb2002 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[19:39:48] <icinga-wm>	 RECOVERY - puppet last run on mw2236 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[19:39:48] <icinga-wm>	 RECOVERY - puppet last run on mw2089 is OK: OK: Puppet is currently enabled, last run 6 seconds ago with 0 failures
[19:39:49] <icinga-wm>	 PROBLEM - puppet last run on rdb2001 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[19:39:49] <icinga-wm>	 PROBLEM - puppet last run on wtp2011 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[19:39:50] <icinga-wm>	 RECOVERY - puppet last run on mw2224 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[19:39:50] <icinga-wm>	 RECOVERY - puppet last run on mw2119 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[19:39:51] <icinga-wm>	 RECOVERY - puppet last run on wtp2014 is OK: OK: Puppet is currently enabled, last run 6 seconds ago with 0 failures
[19:39:51] <icinga-wm>	 RECOVERY - puppet last run on mw2241 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[19:39:52] <icinga-wm>	 RECOVERY - puppet last run on mw2151 is OK: OK: Puppet is currently enabled, last run 32 seconds ago with 0 failures
[19:40:09] <icinga-wm>	 RECOVERY - puppet last run on aqs1002 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures
[19:40:09] <icinga-wm>	 RECOVERY - puppet last run on lvs1006 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures
[19:40:11] <elukey>	 Luke081515: I am checking on the host, something is weird with apache and the ruby backend (via Passenger)
[19:40:18] <icinga-wm>	 RECOVERY - puppet last run on elastic1026 is OK: OK: Puppet is currently enabled, last run 24 seconds ago with 0 failures
[19:40:18] <icinga-wm>	 RECOVERY - puppet last run on db1036 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[19:40:18] <icinga-wm>	 RECOVERY - puppet last run on es1018 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[19:40:19] <icinga-wm>	 RECOVERY - puppet last run on mw1227 is OK: OK: Puppet is currently enabled, last run 10 seconds ago with 0 failures
[19:40:19] <icinga-wm>	 PROBLEM - puppet last run on wtp2018 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[19:40:28] <icinga-wm>	 RECOVERY - puppet last run on cp3044 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures
[19:40:28] <icinga-wm>	 RECOVERY - puppet last run on db1048 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures
[19:40:29] <icinga-wm>	 RECOVERY - puppet last run on elastic1019 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[19:40:29] <icinga-wm>	 PROBLEM - puppet last run on pc1005 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[19:40:37] <icinga-wm>	 RECOVERY - puppet last run on mw2190 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures
[19:40:38] <icinga-wm>	 RECOVERY - puppet last run on mc1005 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures
[19:40:38] <icinga-wm>	 RECOVERY - puppet last run on mw2165 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures
[19:40:38] <icinga-wm>	 RECOVERY - puppet last run on mw2103 is OK: OK: Puppet is currently enabled, last run 4 seconds ago with 0 failures
[19:40:39] <icinga-wm>	 RECOVERY - puppet last run on mw1182 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[19:40:39] <icinga-wm>	 RECOVERY - puppet last run on elastic1024 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[19:40:48] <elukey>	 Luke081515: will keep investigating on the host
[19:40:57] <icinga-wm>	 RECOVERY - puppet last run on mw2178 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures
[19:40:57] <icinga-wm>	 RECOVERY - puppet last run on elastic1037 is OK: OK: Puppet is currently enabled, last run 59 seconds ago with 0 failures
[19:40:57] <icinga-wm>	 RECOVERY - puppet last run on druid1003 is OK: OK: Puppet is currently enabled, last run 11 seconds ago with 0 failures
[19:40:57] <icinga-wm>	 RECOVERY - puppet last run on analytics1046 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures
[19:40:57] <Luke081515>	 ok :)
[19:41:07] <icinga-wm>	 RECOVERY - puppet last run on ms-be2012 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[19:41:07] <icinga-wm>	 RECOVERY - puppet last run on cp1063 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures
[19:41:07] <icinga-wm>	 RECOVERY - puppet last run on ms-be2005 is OK: OK: Puppet is currently enabled, last run 57 seconds ago with 0 failures
[19:41:08] <icinga-wm>	 RECOVERY - puppet last run on mw1247 is OK: OK: Puppet is currently enabled, last run 53 seconds ago with 0 failures
[19:41:08] <icinga-wm>	 RECOVERY - puppet last run on mw1269 is OK: OK: Puppet is currently enabled, last run 32 seconds ago with 0 failures
[19:41:08] <icinga-wm>	 RECOVERY - puppet last run on maps-test2002 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[19:41:08] <icinga-wm>	 RECOVERY - puppet last run on restbase2007 is OK: OK: Puppet is currently enabled, last run 5 seconds ago with 0 failures
[19:41:09] <icinga-wm>	 RECOVERY - puppet last run on ms-be2019 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[19:41:09] <icinga-wm>	 RECOVERY - puppet last run on elastic2009 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[19:41:10] <icinga-wm>	 RECOVERY - puppet last run on cp1074 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures
[19:41:10] <icinga-wm>	 RECOVERY - puppet last run on cp3035 is OK: OK: Puppet is currently enabled, last run 18 seconds ago with 0 failures
[19:41:11] <icinga-wm>	 RECOVERY - puppet last run on cp3005 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures
[19:41:11] <icinga-wm>	 RECOVERY - puppet last run on cp4007 is OK: OK: Puppet is currently enabled, last run 46 seconds ago with 0 failures
[19:41:17] <icinga-wm>	 RECOVERY - puppet last run on mc1004 is OK: OK: Puppet is currently enabled, last run 33 seconds ago with 0 failures
[19:41:19] <icinga-wm>	 RECOVERY - puppet last run on elastic1044 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[19:41:27] <icinga-wm>	 PROBLEM - puppet last run on wtp1007 is CRITICAL: CRITICAL: Puppet has 12 failures. Last run 6 minutes ago with 12 failures. Failed resources (up to 3 shown): File[/usr/lib/nagios/plugins/check_sysctl],File[/etc/sysctl.d],File[/usr/local/sbin/puppet-run],File[/root/.screenrc]
[19:41:28] <icinga-wm>	 RECOVERY - puppet last run on mw2179 is OK: OK: Puppet is currently enabled, last run 19 seconds ago with 0 failures
[19:41:28] <icinga-wm>	 RECOVERY - puppet last run on restbase2005 is OK: OK: Puppet is currently enabled, last run 54 seconds ago with 0 failures
[19:41:28] <icinga-wm>	 RECOVERY - puppet last run on mw1221 is OK: OK: Puppet is currently enabled, last run 45 seconds ago with 0 failures
[19:41:28] <icinga-wm>	 RECOVERY - puppet last run on mw1190 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[19:41:39] <icinga-wm>	 RECOVERY - puppet last run on analytics1015 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[19:41:40] <icinga-wm>	 PROBLEM - puppet last run on prometheus1002 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[19:41:40] <icinga-wm>	 RECOVERY - puppet last run on cp2007 is OK: OK: Puppet is currently enabled, last run 40 seconds ago with 0 failures
[19:41:41] <icinga-wm>	 RECOVERY - puppet last run on mw1290 is OK: OK: Puppet is currently enabled, last run 21 seconds ago with 0 failures
[19:41:49] <icinga-wm>	 RECOVERY - puppet last run on elastic2019 is OK: OK: Puppet is currently enabled, last run 8 seconds ago with 0 failures
[19:41:50] <icinga-wm>	 RECOVERY - puppet last run on mw1196 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[19:41:50] <icinga-wm>	 RECOVERY - puppet last run on mw1232 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[19:41:50] <icinga-wm>	 RECOVERY - puppet last run on ms-be1016 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures
[19:42:01] <icinga-wm>	 RECOVERY - puppet last run on wtp2007 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[19:42:01] <icinga-wm>	 RECOVERY - puppet last run on elastic2022 is OK: OK: Puppet is currently enabled, last run 5 seconds ago with 0 failures
[19:42:01] <icinga-wm>	 RECOVERY - puppet last run on logstash1003 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures
[19:42:02] <icinga-wm>	 RECOVERY - puppet last run on ganeti2005 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[19:42:02] <icinga-wm>	 RECOVERY - puppet last run on mw1214 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures
[19:42:07] <icinga-wm>	 RECOVERY - puppet last run on restbase1010 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[19:42:08] <icinga-wm>	 RECOVERY - puppet last run on elastic1047 is OK: OK: Puppet is currently enabled, last run 57 seconds ago with 0 failures
[19:42:08] <icinga-wm>	 RECOVERY - puppet last run on db1083 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures
[19:42:08] <icinga-wm>	 RECOVERY - puppet last run on db1069 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures
[19:42:09] <icinga-wm>	 RECOVERY - puppet last run on mw1210 is OK: OK: Puppet is currently enabled, last run 42 seconds ago with 0 failures
[19:42:09] <icinga-wm>	 RECOVERY - puppet last run on mw2140 is OK: OK: Puppet is currently enabled, last run 17 seconds ago with 0 failures
[19:42:17] <icinga-wm>	 RECOVERY - puppet last run on mc1007 is OK: OK: Puppet is currently enabled, last run 39 seconds ago with 0 failures
[19:42:18] <icinga-wm>	 RECOVERY - puppet last run on wtp1015 is OK: OK: Puppet is currently enabled, last run 54 seconds ago with 0 failures
[19:42:19] <icinga-wm>	 RECOVERY - puppet last run on acamar is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[19:42:19] <icinga-wm>	 RECOVERY - puppet last run on mw2238 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[19:42:19] <icinga-wm>	 RECOVERY - puppet last run on ms-be2007 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[19:42:19] <icinga-wm>	 RECOVERY - puppet last run on mw2246 is OK: OK: Puppet is currently enabled, last run 56 seconds ago with 0 failures
[19:42:19] <icinga-wm>	 RECOVERY - puppet last run on mw1188 is OK: OK: Puppet is currently enabled, last run 24 seconds ago with 0 failures
[19:42:19] <icinga-wm>	 RECOVERY - puppet last run on mx2001 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[19:42:19] <icinga-wm>	 RECOVERY - puppet last run on db2046 is OK: OK: Puppet is currently enabled, last run 59 seconds ago with 0 failures
[19:42:20] <icinga-wm>	 RECOVERY - puppet last run on mw2099 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[19:42:20] <icinga-wm>	 RECOVERY - puppet last run on mw2102 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[19:42:21] <icinga-wm>	 RECOVERY - puppet last run on ganeti1004 is OK: OK: Puppet is currently enabled, last run 41 seconds ago with 0 failures
[19:42:21] <icinga-wm>	 RECOVERY - puppet last run on pc2004 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[19:42:22] <icinga-wm>	 RECOVERY - puppet last run on cp2018 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[19:42:27] <icinga-wm>	 RECOVERY - puppet last run on mw2201 is OK: OK: Puppet is currently enabled, last run 26 seconds ago with 0 failures
[19:42:28] <icinga-wm>	 RECOVERY - puppet last run on ms-be1012 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures
[19:42:28] <icinga-wm>	 RECOVERY - puppet last run on mw1161 is OK: OK: Puppet is currently enabled, last run 39 seconds ago with 0 failures
[19:42:28] <icinga-wm>	 RECOVERY - puppet last run on rdb1003 is OK: OK: Puppet is currently enabled, last run 26 seconds ago with 0 failures
[19:42:28] <icinga-wm>	 RECOVERY - puppet last run on mw2100 is OK: OK: Puppet is currently enabled, last run 8 seconds ago with 0 failures
[19:42:40] <icinga-wm>	 RECOVERY - puppet last run on cp1049 is OK: OK: Puppet is currently enabled, last run 7 seconds ago with 0 failures
[19:42:40] <icinga-wm>	 RECOVERY - puppet last run on titanium is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[19:42:47] <icinga-wm>	 RECOVERY - puppet last run on mw1265 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[19:42:47] <icinga-wm>	 RECOVERY - puppet last run on mw2139 is OK: OK: Puppet is currently enabled, last run 24 seconds ago with 0 failures
[19:42:48] <icinga-wm>	 RECOVERY - puppet last run on mw1299 is OK: OK: Puppet is currently enabled, last run 38 seconds ago with 0 failures
[19:42:48] <icinga-wm>	 RECOVERY - puppet last run on dbstore1001 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[19:42:49] <icinga-wm>	 RECOVERY - puppet last run on db1088 is OK: OK: Puppet is currently enabled, last run 46 seconds ago with 0 failures
[19:42:58] <icinga-wm>	 RECOVERY - puppet last run on mw1280 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[19:42:58] <icinga-wm>	 RECOVERY - puppet last run on ms-be1018 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[19:42:58] <icinga-wm>	 RECOVERY - puppet last run on logstash1001 is OK: OK: Puppet is currently enabled, last run 43 seconds ago with 0 failures
[19:42:59] <icinga-wm>	 RECOVERY - puppet last run on ms-be2002 is OK: OK: Puppet is currently enabled, last run 24 seconds ago with 0 failures
[19:42:59] <icinga-wm>	 RECOVERY - puppet last run on mw2183 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[19:42:59] <icinga-wm>	 RECOVERY - puppet last run on mw2116 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[19:42:59] <icinga-wm>	 RECOVERY - puppet last run on wtp2002 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[19:43:00] <icinga-wm>	 RECOVERY - puppet last run on rdb2002 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[19:43:00] <icinga-wm>	 RECOVERY - puppet last run on mw1283 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[19:43:01] <icinga-wm>	 RECOVERY - puppet last run on mw2189 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[19:43:01] <icinga-wm>	 RECOVERY - puppet last run on mc1001 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures
[19:43:02] <icinga-wm>	 RECOVERY - puppet last run on mw1263 is OK: OK: Puppet is currently enabled, last run 2 seconds ago with 0 failures
[19:43:08] <icinga-wm>	 RECOVERY - puppet last run on db2069 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures
[19:43:08] <icinga-wm>	 RECOVERY - puppet last run on db1064 is OK: OK: Puppet is currently enabled, last run 56 seconds ago with 0 failures
[19:43:08] <icinga-wm>	 RECOVERY - puppet last run on rdb1002 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[19:43:09] <icinga-wm>	 RECOVERY - puppet last run on mw2112 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[19:43:23] <icinga-wm>	 RECOVERY - puppet last run on dbproxy1002 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[19:43:24] <icinga-wm>	 RECOVERY - puppet last run on cp1072 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[19:43:24] <icinga-wm>	 RECOVERY - puppet last run on db1033 is OK: OK: Puppet is currently enabled, last run 11 seconds ago with 0 failures
[19:43:27] <icinga-wm>	 RECOVERY - puppet last run on db1001 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[19:43:27] <icinga-wm>	 RECOVERY - puppet last run on mw1271 is OK: OK: Puppet is currently enabled, last run 39 seconds ago with 0 failures
[19:43:37] <icinga-wm>	 RECOVERY - puppet last run on californium is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[19:43:37] <icinga-wm>	 RECOVERY - puppet last run on labsdb1004 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[19:43:38] <icinga-wm>	 RECOVERY - puppet last run on cp2021 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures
[19:43:38] <icinga-wm>	 RECOVERY - puppet last run on mw2161 is OK: OK: Puppet is currently enabled, last run 30 seconds ago with 0 failures
[19:43:38] <icinga-wm>	 RECOVERY - puppet last run on cp2025 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[19:43:38] <icinga-wm>	 RECOVERY - puppet last run on es2003 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[19:43:38] <icinga-wm>	 RECOVERY - puppet last run on mw1266 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[19:43:39] <icinga-wm>	 RECOVERY - puppet last run on db2070 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[19:43:39] <icinga-wm>	 RECOVERY - puppet last run on mw2065 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[19:43:40] <icinga-wm>	 RECOVERY - puppet last run on mw1281 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[19:43:40] <icinga-wm>	 RECOVERY - puppet last run on rdb2006 is OK: OK: Puppet is currently enabled, last run 34 seconds ago with 0 failures
[19:43:41] <icinga-wm>	 RECOVERY - puppet last run on mc2012 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[19:43:41] <icinga-wm>	 RECOVERY - puppet last run on mw2227 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[19:43:42] <icinga-wm>	 RECOVERY - puppet last run on cp3034 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures
[19:43:48] <icinga-wm>	 RECOVERY - puppet last run on db1044 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[19:43:48] <icinga-wm>	 RECOVERY - puppet last run on restbase1011 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[19:43:48] <icinga-wm>	 RECOVERY - puppet last run on labvirt1011 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures
[19:43:48] <icinga-wm>	 RECOVERY - puppet last run on labvirt1013 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[19:43:48] <icinga-wm>	 RECOVERY - puppet last run on mw1171 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[19:43:48] <icinga-wm>	 RECOVERY - puppet last run on auth1001 is OK: OK: Puppet is currently enabled, last run 5 seconds ago with 0 failures
[19:43:49] <icinga-wm>	 RECOVERY - puppet last run on elastic1017 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[19:43:49] <icinga-wm>	 RECOVERY - puppet last run on eventlog1001 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures
[19:44:09] <icinga-wm>	 RECOVERY - puppet last run on analytics1043 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[19:44:11] <icinga-wm>	 RECOVERY - puppet last run on analytics1033 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[19:44:12] <icinga-wm>	 RECOVERY - puppet last run on dbproxy1010 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[19:44:12] <icinga-wm>	 RECOVERY - puppet last run on ms-be1006 is OK: OK: Puppet is currently enabled, last run 35 seconds ago with 0 failures
[19:44:22] <icinga-wm>	 RECOVERY - puppet last run on mc2002 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures
[19:44:22] <icinga-wm>	 RECOVERY - puppet last run on analytics1057 is OK: OK: Puppet is currently enabled, last run 35 seconds ago with 0 failures
[19:44:28] <icinga-wm>	 RECOVERY - puppet last run on relforge1002 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[19:44:28] <icinga-wm>	 RECOVERY - puppet last run on mw2153 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[19:44:28] <icinga-wm>	 RECOVERY - puppet last run on mw1197 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[19:44:29] <icinga-wm>	 RECOVERY - puppet last run on mw2164 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[19:44:37] <icinga-wm>	 RECOVERY - puppet last run on mw2247 is OK: OK: Puppet is currently enabled, last run 11 seconds ago with 0 failures
[19:44:37] <icinga-wm>	 RECOVERY - puppet last run on mw2240 is OK: OK: Puppet is currently enabled, last run 45 seconds ago with 0 failures
[19:44:39] <icinga-wm>	 RECOVERY - puppet last run on db1029 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures
[19:44:48] <icinga-wm>	 RECOVERY - puppet last run on planet1001 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[19:44:48] <icinga-wm>	 RECOVERY - puppet last run on mw2170 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures
[19:44:49] <icinga-wm>	 RECOVERY - puppet last run on ganeti2001 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[19:44:49] <icinga-wm>	 RECOVERY - puppet last run on mw1225 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[19:44:49] <icinga-wm>	 RECOVERY - puppet last run on mw1243 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures
[19:44:49] <icinga-wm>	 RECOVERY - puppet last run on mw1189 is OK: OK: Puppet is currently enabled, last run 13 seconds ago with 0 failures
[19:44:49] <icinga-wm>	 RECOVERY - puppet last run on mw2220 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures
[19:44:50] <icinga-wm>	 RECOVERY - puppet last run on wtp2020 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[19:44:50] <icinga-wm>	 RECOVERY - puppet last run on wtp2005 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[19:44:51] <icinga-wm>	 RECOVERY - puppet last run on ms-be2003 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[19:44:51] <icinga-wm>	 RECOVERY - puppet last run on elastic2017 is OK: OK: Puppet is currently enabled, last run 48 seconds ago with 0 failures
[19:44:52] <icinga-wm>	 RECOVERY - puppet last run on mw2221 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures
[19:44:52] <icinga-wm>	 RECOVERY - puppet last run on mw2105 is OK: OK: Puppet is currently enabled, last run 39 seconds ago with 0 failures
[19:44:53] <icinga-wm>	 RECOVERY - puppet last run on analytics1001 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures
[19:44:57] <icinga-wm>	 RECOVERY - puppet last run on helium is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[19:44:57] <icinga-wm>	 RECOVERY - puppet last run on analytics1048 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[19:44:58] <icinga-wm>	 RECOVERY - puppet last run on cp3047 is OK: OK: Puppet is currently enabled, last run 7 seconds ago with 0 failures
[19:44:58] <icinga-wm>	 RECOVERY - puppet last run on cp3021 is OK: OK: Puppet is currently enabled, last run 14 seconds ago with 0 failures
[19:44:58] <icinga-wm>	 RECOVERY - puppet last run on elastic2002 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures
[19:45:01] <elukey>	 Luke081515: rhodium seems stable now but I'd like to see if problems re-occur.. some garbage of criticals is expected
[19:45:07] <elukey>	 but the trend looks good
[19:45:09] <icinga-wm>	 RECOVERY - puppet last run on elastic1034 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures
[19:45:09] <icinga-wm>	 RECOVERY - puppet last run on bast1001 is OK: OK: Puppet is currently enabled, last run 38 seconds ago with 0 failures
[19:45:10] <icinga-wm>	 RECOVERY - puppet last run on mw2064 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[19:45:18] <icinga-wm>	 RECOVERY - puppet last run on mw2244 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[19:45:18] <icinga-wm>	 RECOVERY - puppet last run on mw2229 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[19:45:18] <icinga-wm>	 RECOVERY - puppet last run on elastic1018 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[19:45:18] <icinga-wm>	 RECOVERY - puppet last run on ganeti2004 is OK: OK: Puppet is currently enabled, last run 58 seconds ago with 0 failures
[19:45:19] <icinga-wm>	 RECOVERY - puppet last run on db2039 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures
[19:45:19] <icinga-wm>	 RECOVERY - puppet last run on ruthenium is OK: OK: Puppet is currently enabled, last run 30 seconds ago with 0 failures
[19:45:19] <icinga-wm>	 RECOVERY - puppet last run on mw2109 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[19:45:28] <icinga-wm>	 RECOVERY - puppet last run on mw1205 is OK: OK: Puppet is currently enabled, last run 27 seconds ago with 0 failures
[19:45:28] <icinga-wm>	 RECOVERY - puppet last run on fluorine is OK: OK: Puppet is currently enabled, last run 55 seconds ago with 0 failures
[19:45:28] <icinga-wm>	 RECOVERY - puppet last run on ms-be2006 is OK: OK: Puppet is currently enabled, last run 44 seconds ago with 0 failures
[19:45:29] <icinga-wm>	 RECOVERY - puppet last run on potassium is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures
[19:45:29] <icinga-wm>	 RECOVERY - puppet last run on mw1241 is OK: OK: Puppet is currently enabled, last run 44 seconds ago with 0 failures
[19:45:29] <icinga-wm>	 RECOVERY - puppet last run on wtp2001 is OK: OK: Puppet is currently enabled, last run 46 seconds ago with 0 failures
[19:45:30] <icinga-wm>	 RECOVERY - puppet last run on db1022 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[19:45:30] <icinga-wm>	 RECOVERY - puppet last run on cp4002 is OK: OK: Puppet is currently enabled, last run 38 seconds ago with 0 failures
[19:45:37] <Luke081515>	 elukey: ok, thx for fixing :)
[19:45:37] <icinga-wm>	 RECOVERY - puppet last run on db1031 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures
[19:45:38] <icinga-wm>	 RECOVERY - puppet last run on analytics1003 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[19:45:38] <icinga-wm>	 RECOVERY - puppet last run on analytics1035 is OK: OK: Puppet is currently enabled, last run 19 seconds ago with 0 failures
[19:45:38] <icinga-wm>	 RECOVERY - puppet last run on cp2023 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures
[19:45:38] <icinga-wm>	 RECOVERY - puppet last run on ms-fe1002 is OK: OK: Puppet is currently enabled, last run 6 seconds ago with 0 failures
[19:45:38] <icinga-wm>	 RECOVERY - puppet last run on achernar is OK: OK: Puppet is currently enabled, last run 49 seconds ago with 0 failures
[19:45:54] <icinga-wm>	 RECOVERY - puppet last run on db1050 is OK: OK: Puppet is currently enabled, last run 48 seconds ago with 0 failures
[19:45:56] <icinga-wm>	 RECOVERY - puppet last run on cp3040 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[19:45:57] <icinga-wm>	 RECOVERY - puppet last run on mw1166 is OK: OK: Puppet is currently enabled, last run 7 seconds ago with 0 failures
[19:46:07] <icinga-wm>	 RECOVERY - puppet last run on db2045 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[19:46:07] <icinga-wm>	 RECOVERY - puppet last run on mw2176 is OK: OK: Puppet is currently enabled, last run 6 seconds ago with 0 failures
[19:46:07] <icinga-wm>	 RECOVERY - puppet last run on mw2075 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[19:46:07] <icinga-wm>	 RECOVERY - puppet last run on mw2188 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures
[19:46:08] <icinga-wm>	 RECOVERY - puppet last run on logstash1002 is OK: OK: Puppet is currently enabled, last run 20 seconds ago with 0 failures
[19:46:08] <icinga-wm>	 RECOVERY - puppet last run on wezen is OK: OK: Puppet is currently enabled, last run 32 seconds ago with 0 failures
[19:46:08] <icinga-wm>	 RECOVERY - puppet last run on mw2083 is OK: OK: Puppet is currently enabled, last run 29 seconds ago with 0 failures
[19:46:09] <icinga-wm>	 RECOVERY - puppet last run on ms-be2004 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[19:46:09] <icinga-wm>	 RECOVERY - puppet last run on lvs1002 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures
[19:46:10] <icinga-wm>	 RECOVERY - puppet last run on db2047 is OK: OK: Puppet is currently enabled, last run 14 seconds ago with 0 failures
[19:46:10] <icinga-wm>	 RECOVERY - puppet last run on restbase-test2002 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures
[19:46:11] <icinga-wm>	 RECOVERY - puppet last run on cp2026 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[19:46:11] <icinga-wm>	 RECOVERY - puppet last run on db2059 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures
[19:46:12] <icinga-wm>	 RECOVERY - puppet last run on mc2016 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[19:46:27] <icinga-wm>	 RECOVERY - puppet last run on lvs1005 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures
[19:46:28] <icinga-wm>	 RECOVERY - puppet last run on elastic2004 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[19:46:29] <icinga-wm>	 RECOVERY - puppet last run on es1014 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[19:46:29] <icinga-wm>	 RECOVERY - puppet last run on mw1298 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[19:46:29] <icinga-wm>	 RECOVERY - puppet last run on mw2079 is OK: OK: Puppet is currently enabled, last run 19 seconds ago with 0 failures
[19:46:29] <icinga-wm>	 RECOVERY - puppet last run on ganeti1002 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures
[19:46:29] <icinga-wm>	 RECOVERY - puppet last run on cp2016 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[19:46:29] <icinga-wm>	 RECOVERY - puppet last run on mw2163 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[19:46:30] <icinga-wm>	 RECOVERY - puppet last run on mw1267 is OK: OK: Puppet is currently enabled, last run 56 seconds ago with 0 failures
[19:46:30] <icinga-wm>	 RECOVERY - puppet last run on mw1294 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures
[19:46:31] <icinga-wm>	 RECOVERY - puppet last run on wtp1001 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[19:46:31] <icinga-wm>	 RECOVERY - puppet last run on mw1285 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[19:46:32] <icinga-wm>	 RECOVERY - puppet last run on mw1300 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures
[19:46:32] <icinga-wm>	 RECOVERY - puppet last run on elastic1041 is OK: OK: Puppet is currently enabled, last run 27 seconds ago with 0 failures
[19:46:38] <icinga-wm>	 RECOVERY - puppet last run on elastic1046 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures
[19:46:38] <icinga-wm>	 RECOVERY - puppet last run on mw1235 is OK: OK: Puppet is currently enabled, last run 36 seconds ago with 0 failures
[19:46:42] <icinga-wm>	 RECOVERY - puppet last run on cp2011 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[19:46:52] <icinga-wm>	 RECOVERY - puppet last run on oresrdb1001 is OK: OK: Puppet is currently enabled, last run 59 seconds ago with 0 failures
[19:46:52] <icinga-wm>	 RECOVERY - puppet last run on rdb2005 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures
[19:46:52] <icinga-wm>	 RECOVERY - puppet last run on mc1012 is OK: OK: Puppet is currently enabled, last run 15 seconds ago with 0 failures
[19:46:59] <icinga-wm>	 RECOVERY - puppet last run on mw2087 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[19:46:59] <icinga-wm>	 RECOVERY - puppet last run on mw2067 is OK: OK: Puppet is currently enabled, last run 13 seconds ago with 0 failures
[19:47:00] <icinga-wm>	 RECOVERY - puppet last run on graphite2002 is OK: OK: Puppet is currently enabled, last run 32 seconds ago with 0 failures
[19:47:01] <icinga-wm>	 RECOVERY - puppet last run on prometheus2002 is OK: OK: Puppet is currently enabled, last run 7 seconds ago with 0 failures
[19:47:01] <icinga-wm>	 RECOVERY - puppet last run on dbproxy1001 is OK: OK: Puppet is currently enabled, last run 40 seconds ago with 0 failures
[19:47:01] <icinga-wm>	 RECOVERY - puppet last run on labstore2003 is OK: OK: Puppet is currently enabled, last run 8 seconds ago with 0 failures
[19:47:07] <icinga-wm>	 RECOVERY - puppet last run on cp3018 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures
[19:47:07] <icinga-wm>	 RECOVERY - puppet last run on labvirt1009 is OK: OK: Puppet is currently enabled, last run 18 seconds ago with 0 failures
[19:47:08] <icinga-wm>	 RECOVERY - puppet last run on mw1204 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures
[19:47:09] <icinga-wm>	 RECOVERY - puppet last run on ms-fe1001 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures
[19:47:09] <icinga-wm>	 RECOVERY - puppet last run on lvs1004 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[19:47:09] <icinga-wm>	 RECOVERY - puppet last run on lvs2004 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures
[19:47:18] <icinga-wm>	 RECOVERY - puppet last run on db1034 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[19:47:19] <icinga-wm>	 RECOVERY - puppet last run on restbase2002 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[19:47:19] <icinga-wm>	 RECOVERY - puppet last run on cp2003 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[19:47:19] <icinga-wm>	 RECOVERY - puppet last run on sinistra is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[19:47:19] <icinga-wm>	 RECOVERY - puppet last run on cp2010 is OK: OK: Puppet is currently enabled, last run 20 seconds ago with 0 failures
[19:47:19] <icinga-wm>	 RECOVERY - puppet last run on mw2127 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[19:47:20] <icinga-wm>	 RECOVERY - puppet last run on wtp2010 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[19:47:20] <icinga-wm>	 RECOVERY - puppet last run on mw2117 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures
[19:47:20] <icinga-wm>	 RECOVERY - puppet last run on kafka2001 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[19:47:21] <icinga-wm>	 RECOVERY - puppet last run on mw2110 is OK: OK: Puppet is currently enabled, last run 27 seconds ago with 0 failures
[19:47:21] <icinga-wm>	 RECOVERY - puppet last run on mw2233 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[19:47:22] <icinga-wm>	 RECOVERY - puppet last run on wtp2016 is OK: OK: Puppet is currently enabled, last run 58 seconds ago with 0 failures
[19:47:22] <icinga-wm>	 RECOVERY - puppet last run on mw2182 is OK: OK: Puppet is currently enabled, last run 17 seconds ago with 0 failures
[19:47:23] <icinga-wm>	 RECOVERY - puppet last run on mw2096 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[19:47:27] <icinga-wm>	 RECOVERY - puppet last run on mw1173 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures
[19:47:27] <icinga-wm>	 RECOVERY - puppet last run on labcontrol1001 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[19:47:38] <icinga-wm>	 RECOVERY - puppet last run on mw1253 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[19:47:38] <icinga-wm>	 RECOVERY - puppet last run on ms-be1015 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures
[19:47:47] <icinga-wm>	 RECOVERY - puppet last run on mendelevium is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[19:47:48] <icinga-wm>	 RECOVERY - puppet last run on db1079 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures
[19:47:49] <icinga-wm>	 RECOVERY - puppet last run on labvirt1010 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures
[19:47:57] <icinga-wm>	 RECOVERY - puppet last run on mw1213 is OK: OK: Puppet is currently enabled, last run 23 seconds ago with 0 failures
[19:47:58] <icinga-wm>	 RECOVERY - puppet last run on db1051 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[19:47:58] <icinga-wm>	 RECOVERY - puppet last run on mw2101 is OK: OK: Puppet is currently enabled, last run 3 seconds ago with 0 failures
[19:47:59] <icinga-wm>	 RECOVERY - puppet last run on mw2093 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures
[19:47:59] <icinga-wm>	 RECOVERY - puppet last run on mw2090 is OK: OK: Puppet is currently enabled, last run 49 seconds ago with 0 failures
[19:48:00] <icinga-wm>	 RECOVERY - puppet last run on mw2142 is OK: OK: Puppet is currently enabled, last run 14 seconds ago with 0 failures
[19:48:00] <icinga-wm>	 RECOVERY - puppet last run on mw1211 is OK: OK: Puppet is currently enabled, last run 23 seconds ago with 0 failures
[19:48:00] <icinga-wm>	 RECOVERY - puppet last run on mw1194 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[19:48:00] <icinga-wm>	 RECOVERY - puppet last run on mw2111 is OK: OK: Puppet is currently enabled, last run 23 seconds ago with 0 failures
[19:48:00] <icinga-wm>	 RECOVERY - puppet last run on mw2113 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures
[19:48:07] <icinga-wm>	 RECOVERY - puppet last run on db2054 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures
[19:48:08] <icinga-wm>	 RECOVERY - puppet last run on elastic1040 is OK: OK: Puppet is currently enabled, last run 23 seconds ago with 0 failures
[19:48:08] <icinga-wm>	 RECOVERY - puppet last run on elastic1045 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[19:48:08] <icinga-wm>	 RECOVERY - puppet last run on mw2184 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures
[19:48:09] <icinga-wm>	 RECOVERY - puppet last run on druid1001 is OK: OK: Puppet is currently enabled, last run 2 seconds ago with 0 failures
[19:48:09] <icinga-wm>	 RECOVERY - puppet last run on db1068 is OK: OK: Puppet is currently enabled, last run 47 seconds ago with 0 failures
[19:48:25] <icinga-wm>	 RECOVERY - puppet last run on db1082 is OK: OK: Puppet is currently enabled, last run 49 seconds ago with 0 failures
[19:48:26] <icinga-wm>	 RECOVERY - puppet last run on mw2143 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures
[19:48:27] <icinga-wm>	 RECOVERY - puppet last run on db1021 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[19:48:27] <icinga-wm>	 RECOVERY - puppet last run on cp4019 is OK: OK: Puppet is currently enabled, last run 36 seconds ago with 0 failures
[19:48:27] <icinga-wm>	 RECOVERY - puppet last run on ms-fe2003 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[19:48:27] <icinga-wm>	 RECOVERY - puppet last run on cp1058 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[19:48:37] <icinga-wm>	 RECOVERY - puppet last run on mw2212 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[19:48:37] <icinga-wm>	 RECOVERY - puppet last run on analytics1031 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[19:48:38] <icinga-wm>	 RECOVERY - puppet last run on palladium is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[19:48:38] <icinga-wm>	 RECOVERY - puppet last run on mw2134 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures
[19:48:38] <icinga-wm>	 RECOVERY - puppet last run on ms-be2009 is OK: OK: Puppet is currently enabled, last run 7 seconds ago with 0 failures
[19:48:38] <icinga-wm>	 RECOVERY - puppet last run on mw2084 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[19:48:38] <icinga-wm>	 RECOVERY - puppet last run on lvs1011 is OK: OK: Puppet is currently enabled, last run 7 seconds ago with 0 failures
[19:48:39] <icinga-wm>	 RECOVERY - puppet last run on db2029 is OK: OK: Puppet is currently enabled, last run 30 seconds ago with 0 failures
[19:48:39] <icinga-wm>	 RECOVERY - puppet last run on mw2123 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[19:48:40] <icinga-wm>	 RECOVERY - puppet last run on cp2022 is OK: OK: Puppet is currently enabled, last run 42 seconds ago with 0 failures
[19:48:40] <icinga-wm>	 RECOVERY - puppet last run on cp2014 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures
[19:48:41] <icinga-wm>	 RECOVERY - puppet last run on mw2070 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[19:48:41] <icinga-wm>	 RECOVERY - puppet last run on mc2014 is OK: OK: Puppet is currently enabled, last run 38 seconds ago with 0 failures
[19:48:42] <icinga-wm>	 RECOVERY - puppet last run on serpens is OK: OK: Puppet is currently enabled, last run 25 seconds ago with 0 failures
[19:48:57] <icinga-wm>	 RECOVERY - puppet last run on maps2001 is OK: OK: Puppet is currently enabled, last run 36 seconds ago with 0 failures
[19:48:58] <icinga-wm>	 RECOVERY - puppet last run on mw1238 is OK: OK: Puppet is currently enabled, last run 20 seconds ago with 0 failures
[19:48:58] <icinga-wm>	 RECOVERY - puppet last run on mw1206 is OK: OK: Puppet is currently enabled, last run 38 seconds ago with 0 failures
[19:48:59] <icinga-wm>	 RECOVERY - puppet last run on elastic2013 is OK: OK: Puppet is currently enabled, last run 18 seconds ago with 0 failures
[19:48:59] <icinga-wm>	 RECOVERY - puppet last run on mw2203 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[19:48:59] <icinga-wm>	 RECOVERY - puppet last run on mw2223 is OK: OK: Puppet is currently enabled, last run 30 seconds ago with 0 failures
[19:48:59] <icinga-wm>	 RECOVERY - puppet last run on labstore1001 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[19:49:12] <icinga-wm>	 RECOVERY - puppet last run on mw1230 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[19:49:13] <icinga-wm>	 RECOVERY - puppet last run on mw1179 is OK: OK: Puppet is currently enabled, last run 49 seconds ago with 0 failures
[19:49:14] <icinga-wm>	 RECOVERY - puppet last run on seaborgium is OK: OK: Puppet is currently enabled, last run 13 seconds ago with 0 failures
[19:49:14] <icinga-wm>	 RECOVERY - puppet last run on elastic2008 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[19:49:14] <icinga-wm>	 RECOVERY - puppet last run on cp1099 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[19:49:18] <icinga-wm>	 RECOVERY - puppet last run on mw2131 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[19:49:30] <icinga-wm>	 RECOVERY - puppet last run on bromine is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[19:49:30] <icinga-wm>	 RECOVERY - puppet last run on pc2005 is OK: OK: Puppet is currently enabled, last run 0 seconds ago with 0 failures
[19:49:32] <icinga-wm>	 RECOVERY - puppet last run on db2061 is OK: OK: Puppet is currently enabled, last run 2 seconds ago with 0 failures
[19:49:32] <icinga-wm>	 RECOVERY - puppet last run on ms-be2027 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[19:49:33] <icinga-wm>	 RECOVERY - puppet last run on mw2150 is OK: OK: Puppet is currently enabled, last run 11 seconds ago with 0 failures
[19:49:33] <icinga-wm>	 RECOVERY - puppet last run on mw2249 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures
[19:49:37] <icinga-wm>	 RECOVERY - puppet last run on labvirt1012 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[19:49:38] <icinga-wm>	 RECOVERY - puppet last run on dataset1001 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures
[19:49:38] <icinga-wm>	 RECOVERY - puppet last run on analytics1029 is OK: OK: Puppet is currently enabled, last run 44 seconds ago with 0 failures
[19:49:38] <icinga-wm>	 RECOVERY - puppet last run on rdb1006 is OK: OK: Puppet is currently enabled, last run 46 seconds ago with 0 failures
[19:49:48] <icinga-wm>	 RECOVERY - puppet last run on mw2085 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[19:49:48] <icinga-wm>	 RECOVERY - puppet last run on sca2002 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[19:49:49] <icinga-wm>	 RECOVERY - puppet last run on conf2002 is OK: OK: Puppet is currently enabled, last run 29 seconds ago with 0 failures
[19:49:49] <icinga-wm>	 RECOVERY - puppet last run on mw2242 is OK: OK: Puppet is currently enabled, last run 28 seconds ago with 0 failures
[19:49:50] <icinga-wm>	 RECOVERY - puppet last run on ms-be2008 is OK: OK: Puppet is currently enabled, last run 4 seconds ago with 0 failures
[19:49:50] <icinga-wm>	 RECOVERY - puppet last run on mw2094 is OK: OK: Puppet is currently enabled, last run 20 seconds ago with 0 failures
[19:49:50] <icinga-wm>	 RECOVERY - puppet last run on db1024 is OK: OK: Puppet is currently enabled, last run 17 seconds ago with 0 failures
[19:49:50] <icinga-wm>	 RECOVERY - puppet last run on mw2168 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[19:49:58] <icinga-wm>	 RECOVERY - puppet last run on db1026 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[19:49:58] <icinga-wm>	 RECOVERY - puppet last run on cp1071 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures
[19:50:07] <icinga-wm>	 RECOVERY - puppet last run on db2037 is OK: OK: Puppet is currently enabled, last run 46 seconds ago with 0 failures
[19:50:10] <icinga-wm>	 RECOVERY - puppet last run on cp1073 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[19:50:11] <icinga-wm>	 RECOVERY - puppet last run on db1062 is OK: OK: Puppet is currently enabled, last run 23 seconds ago with 0 failures
[19:50:11] <icinga-wm>	 RECOVERY - puppet last run on ms-be2014 is OK: OK: Puppet is currently enabled, last run 22 seconds ago with 0 failures
[19:50:11] <icinga-wm>	 RECOVERY - puppet last run on cp4017 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[19:50:11] <icinga-wm>	 RECOVERY - puppet last run on cp3006 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures
[19:50:11] <icinga-wm>	 RECOVERY - puppet last run on es1016 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures
[19:50:18] <icinga-wm>	 RECOVERY - puppet last run on cp1065 is OK: OK: Puppet is currently enabled, last run 31 seconds ago with 0 failures
[19:50:29] <icinga-wm>	 RECOVERY - puppet last run on ms-be2010 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[19:50:30] <icinga-wm>	 RECOVERY - puppet last run on mw2107 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[19:50:30] <icinga-wm>	 RECOVERY - puppet last run on ocg1002 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[19:50:30] <icinga-wm>	 RECOVERY - puppet last run on mw2172 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[19:50:30] <icinga-wm>	 RECOVERY - puppet last run on lvs3004 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[19:50:31] <icinga-wm>	 RECOVERY - puppet last run on analytics1052 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[19:50:31] <icinga-wm>	 RECOVERY - puppet last run on cp1062 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[19:50:31] <icinga-wm>	 RECOVERY - puppet last run on mw1295 is OK: OK: Puppet is currently enabled, last run 35 seconds ago with 0 failures
[19:50:31] <icinga-wm>	 RECOVERY - puppet last run on mw1278 is OK: OK: Puppet is currently enabled, last run 53 seconds ago with 0 failures
[19:50:32] <icinga-wm>	 RECOVERY - puppet last run on db2041 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[19:50:32] <icinga-wm>	 RECOVERY - puppet last run on analytics1056 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[19:50:38] <icinga-wm>	 RECOVERY - puppet last run on mw1304 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[19:50:38] <icinga-wm>	 RECOVERY - puppet last run on stat1001 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[19:50:38] <icinga-wm>	 RECOVERY - puppet last run on kafka1014 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[19:50:50] <icinga-wm>	 RECOVERY - puppet last run on db1076 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[19:50:53] <icinga-wm>	 RECOVERY - puppet last run on labvirt1007 is OK: OK: Puppet is currently enabled, last run 3 seconds ago with 0 failures
[19:50:53] <icinga-wm>	 RECOVERY - puppet last run on mira is OK: OK: Puppet is currently enabled, last run 22 seconds ago with 0 failures
[19:50:54] <icinga-wm>	 RECOVERY - puppet last run on mw1275 is OK: OK: Puppet is currently enabled, last run 54 seconds ago with 0 failures
[19:50:55] <icinga-wm>	 RECOVERY - puppet last run on analytics1002 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures
[19:50:55] <icinga-wm>	 RECOVERY - puppet last run on cp3009 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[19:50:55] <icinga-wm>	 RECOVERY - puppet last run on mw1180 is OK: OK: Puppet is currently enabled, last run 47 seconds ago with 0 failures
[19:50:55] <icinga-wm>	 RECOVERY - puppet last run on cp2015 is OK: OK: Puppet is currently enabled, last run 39 seconds ago with 0 failures
[19:51:07] <icinga-wm>	 RECOVERY - puppet last run on db1071 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures
[19:51:08] <icinga-wm>	 RECOVERY - puppet last run on mw2174 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[19:51:08] <icinga-wm>	 RECOVERY - puppet last run on mw2133 is OK: OK: Puppet is currently enabled, last run 30 seconds ago with 0 failures
[19:51:08] <icinga-wm>	 RECOVERY - puppet last run on wasat is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[19:51:08] <icinga-wm>	 RECOVERY - puppet last run on db2023 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[19:51:08] <icinga-wm>	 RECOVERY - puppet last run on db2057 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures
[19:51:08] <icinga-wm>	 RECOVERY - puppet last run on hassium is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[19:51:09] <icinga-wm>	 RECOVERY - puppet last run on ms-be1007 is OK: OK: Puppet is currently enabled, last run 5 seconds ago with 0 failures
[19:51:09] <icinga-wm>	 RECOVERY - puppet last run on cp3031 is OK: OK: Puppet is currently enabled, last run 55 seconds ago with 0 failures
[19:51:10] <icinga-wm>	 RECOVERY - puppet last run on analytics1051 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures
[19:51:17] <icinga-wm>	 RECOVERY - puppet last run on wtp1021 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[19:51:19] <icinga-wm>	 RECOVERY - puppet last run on mw1181 is OK: OK: Puppet is currently enabled, last run 57 seconds ago with 0 failures
[19:51:28] <icinga-wm>	 RECOVERY - puppet last run on kafka1013 is OK: OK: Puppet is currently enabled, last run 42 seconds ago with 0 failures
[19:51:29] <icinga-wm>	 RECOVERY - puppet last run on mc1008 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[19:51:29] <icinga-wm>	 RECOVERY - puppet last run on krypton is OK: OK: Puppet is currently enabled, last run 5 seconds ago with 0 failures
[19:51:29] <icinga-wm>	 RECOVERY - puppet last run on mw2152 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[19:51:29] <icinga-wm>	 RECOVERY - puppet last run on mw2200 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures
[19:51:29] <icinga-wm>	 RECOVERY - puppet last run on conf2003 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[19:51:29] <icinga-wm>	 RECOVERY - puppet last run on zosma is OK: OK: Puppet is currently enabled, last run 15 seconds ago with 0 failures
[19:51:30] <icinga-wm>	 RECOVERY - puppet last run on mw1274 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures
[19:51:30] <icinga-wm>	 RECOVERY - puppet last run on mw1296 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures
[19:51:31] <icinga-wm>	 RECOVERY - puppet last run on elastic1035 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[19:51:39] <icinga-wm>	 RECOVERY - puppet last run on wtp1002 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[19:51:39] <icinga-wm>	 RECOVERY - puppet last run on uranium is OK: OK: Puppet is currently enabled, last run 6 seconds ago with 0 failures
[19:51:39] <icinga-wm>	 RECOVERY - puppet last run on db1055 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[19:51:48] <icinga-wm>	 RECOVERY - puppet last run on ms-be2017 is OK: OK: Puppet is currently enabled, last run 29 seconds ago with 0 failures
[19:51:49] <icinga-wm>	 RECOVERY - puppet last run on mx1001 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[19:51:51] <icinga-wm>	 RECOVERY - puppet last run on mw1245 is OK: OK: Puppet is currently enabled, last run 34 seconds ago with 0 failures
[19:51:52] <icinga-wm>	 RECOVERY - puppet last run on tegmen is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures
[19:51:57] <icinga-wm>	 RECOVERY - puppet last run on bohrium is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures
[19:52:07] <icinga-wm>	 RECOVERY - puppet last run on wtp1013 is OK: OK: Puppet is currently enabled, last run 36 seconds ago with 0 failures
[19:52:07] <icinga-wm>	 RECOVERY - puppet last run on db1010 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[19:52:08] <icinga-wm>	 RECOVERY - puppet last run on relforge1001 is OK: OK: Puppet is currently enabled, last run 18 seconds ago with 0 failures
[19:52:08] <icinga-wm>	 RECOVERY - puppet last run on wdqs1001 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures
[19:52:18] <icinga-wm>	 RECOVERY - puppet last run on mw1178 is OK: OK: Puppet is currently enabled, last run 16 seconds ago with 0 failures
[19:52:18] <icinga-wm>	 RECOVERY - puppet last run on conf2001 is OK: OK: Puppet is currently enabled, last run 57 seconds ago with 0 failures
[19:52:19] <icinga-wm>	 RECOVERY - puppet last run on pc2006 is OK: OK: Puppet is currently enabled, last run 8 seconds ago with 0 failures
[19:52:19] <icinga-wm>	 RECOVERY - puppet last run on labvirt1005 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[19:52:19] <icinga-wm>	 RECOVERY - puppet last run on mw1165 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[19:52:19] <icinga-wm>	 RECOVERY - puppet last run on es2012 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[19:52:19] <icinga-wm>	 RECOVERY - puppet last run on wtp2009 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures
[19:52:20] <icinga-wm>	 RECOVERY - puppet last run on kafka1018 is OK: OK: Puppet is currently enabled, last run 24 seconds ago with 0 failures
[19:52:20] <icinga-wm>	 RECOVERY - puppet last run on labstore2001 is OK: OK: Puppet is currently enabled, last run 26 seconds ago with 0 failures
[19:52:21] <icinga-wm>	 RECOVERY - puppet last run on wdqs1002 is OK: OK: Puppet is currently enabled, last run 29 seconds ago with 0 failures
[19:52:22] <icinga-wm>	 RECOVERY - puppet last run on analytics1044 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[19:52:27] <icinga-wm>	 RECOVERY - puppet last run on ms-fe1003 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[19:52:28] <icinga-wm>	 RECOVERY - puppet last run on lvs1001 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[19:52:28] <icinga-wm>	 RECOVERY - puppet last run on analytics1036 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[19:52:38] <icinga-wm>	 RECOVERY - puppet last run on mw2106 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[19:52:39] <icinga-wm>	 RECOVERY - puppet last run on cp3013 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[19:52:41] <icinga-wm>	 RECOVERY - puppet last run on dbproxy1009 is OK: OK: Puppet is currently enabled, last run 24 seconds ago with 0 failures
[19:52:41] <icinga-wm>	 RECOVERY - puppet last run on cp3045 is OK: OK: Puppet is currently enabled, last run 14 seconds ago with 0 failures
[19:52:48] <icinga-wm>	 RECOVERY - puppet last run on cp2019 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures
[19:52:48] <icinga-wm>	 RECOVERY - puppet last run on mw2237 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures
[19:52:49] <icinga-wm>	 RECOVERY - puppet last run on mw2167 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[19:52:49] <icinga-wm>	 RECOVERY - puppet last run on db1074 is OK: OK: Puppet is currently enabled, last run 54 seconds ago with 0 failures
[19:52:49] <icinga-wm>	 RECOVERY - puppet last run on es2016 is OK: OK: Puppet is currently enabled, last run 25 seconds ago with 0 failures
[19:52:49] <icinga-wm>	 RECOVERY - puppet last run on ms-be1024 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[19:52:50] <icinga-wm>	 RECOVERY - puppet last run on aqs1005 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[19:52:57] <icinga-wm>	 RECOVERY - puppet last run on ms-be1004 is OK: OK: Puppet is currently enabled, last run 30 seconds ago with 0 failures
[19:52:58] <icinga-wm>	 RECOVERY - puppet last run on labsdb1001 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures
[19:52:59] <icinga-wm>	 RECOVERY - puppet last run on dbproxy1006 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[19:52:59] <icinga-wm>	 RECOVERY - puppet last run on mw2186 is OK: OK: Puppet is currently enabled, last run 18 seconds ago with 0 failures
[19:53:00] <icinga-wm>	 RECOVERY - puppet last run on db1063 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[19:53:00] <icinga-wm>	 RECOVERY - puppet last run on ocg1001 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[19:53:00] <icinga-wm>	 RECOVERY - puppet last run on db1093 is OK: OK: Puppet is currently enabled, last run 56 seconds ago with 0 failures
[19:53:00] <icinga-wm>	 RECOVERY - puppet last run on ms-fe3001 is OK: OK: Puppet is currently enabled, last run 20 seconds ago with 0 failures
[19:53:00] <icinga-wm>	 RECOVERY - puppet last run on ms-be1009 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures
[19:53:01] <icinga-wm>	 RECOVERY - puppet last run on db1081 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[19:53:01] <icinga-wm>	 RECOVERY - puppet last run on stat1002 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures
[19:53:03] <elukey>	 ok it looks that rhodium is stable
[19:53:07] <icinga-wm>	 RECOVERY - puppet last run on mw1297 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[19:53:08] <icinga-wm>	 RECOVERY - puppet last run on mc1011 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures
[19:53:08] <icinga-wm>	 RECOVERY - puppet last run on analytics1032 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures
[19:53:08] <icinga-wm>	 RECOVERY - puppet last run on mw1240 is OK: OK: Puppet is currently enabled, last run 52 seconds ago with 0 failures
[19:53:08] <icinga-wm>	 RECOVERY - puppet last run on mc1016 is OK: OK: Puppet is currently enabled, last run 49 seconds ago with 0 failures
[19:53:08] <icinga-wm>	 RECOVERY - puppet last run on lvs2003 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures
[19:53:09] <icinga-wm>	 RECOVERY - puppet last run on maps-test2001 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures
[19:53:09] <icinga-wm>	 RECOVERY - puppet last run on mw1198 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[19:53:23] <icinga-wm>	 RECOVERY - puppet last run on db1011 is OK: OK: Puppet is currently enabled, last run 1 second ago with 0 failures
[19:53:27] <icinga-wm>	 RECOVERY - puppet last run on ms-be1017 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[19:53:32] <icinga-wm>	 RECOVERY - puppet last run on logstash1005 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[19:53:37] <icinga-wm>	 RECOVERY - puppet last run on cp2017 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[19:53:37] <icinga-wm>	 RECOVERY - puppet last run on ms-be2013 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[19:53:38] <icinga-wm>	 RECOVERY - puppet last run on mw2068 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[19:53:38] <icinga-wm>	 RECOVERY - puppet last run on mw1233 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[19:53:38] <icinga-wm>	 RECOVERY - puppet last run on ms-be2016 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[19:53:38] <icinga-wm>	 RECOVERY - puppet last run on elastic2021 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[19:53:38] <icinga-wm>	 RECOVERY - puppet last run on db2050 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[19:53:47] <icinga-wm>	 RECOVERY - puppet last run on lvs4002 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[19:53:47] <icinga-wm>	 RECOVERY - puppet last run on cp3046 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[19:53:47] <icinga-wm>	 RECOVERY - puppet last run on cp3019 is OK: OK: Puppet is currently enabled, last run 53 seconds ago with 0 failures
[19:53:48] <icinga-wm>	 RECOVERY - puppet last run on db1070 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[19:53:48] <icinga-wm>	 RECOVERY - puppet last run on ms-be1025 is OK: OK: Puppet is currently enabled, last run 10 seconds ago with 0 failures
[19:53:58] <icinga-wm>	 RECOVERY - puppet last run on pc1004 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[19:54:00] <icinga-wm>	 RECOVERY - puppet last run on prometheus2001 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[19:54:07] <icinga-wm>	 RECOVERY - puppet last run on mw1270 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures
[19:54:07] <icinga-wm>	 RECOVERY - puppet last run on cp1055 is OK: OK: Puppet is currently enabled, last run 28 seconds ago with 0 failures
[19:54:07] <icinga-wm>	 RECOVERY - puppet last run on mc1018 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[19:54:08] <icinga-wm>	 RECOVERY - puppet last run on mw1231 is OK: OK: Puppet is currently enabled, last run 6 seconds ago with 0 failures
[19:54:18] <icinga-wm>	 RECOVERY - puppet last run on ms-be1022 is OK: OK: Puppet is currently enabled, last run 2 seconds ago with 0 failures
[19:54:18] <icinga-wm>	 RECOVERY - puppet last run on elastic1023 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[19:54:19] <icinga-wm>	 RECOVERY - puppet last run on db1053 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[19:54:19] <icinga-wm>	 RECOVERY - puppet last run on mw1256 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures
[19:54:28] <icinga-wm>	 RECOVERY - puppet last run on heze is OK: OK: Puppet is currently enabled, last run 59 seconds ago with 0 failures
[19:54:31] <icinga-wm>	 RECOVERY - puppet last run on sca1002 is OK: OK: Puppet is currently enabled, last run 7 seconds ago with 0 failures
[19:54:35] <icinga-wm>	 RECOVERY - puppet last run on maps-test2003 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[19:54:35] <icinga-wm>	 RECOVERY - puppet last run on mw2209 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[19:54:35] <icinga-wm>	 RECOVERY - puppet last run on cp3012 is OK: OK: Puppet is currently enabled, last run 54 seconds ago with 0 failures
[19:54:35] <icinga-wm>	 RECOVERY - puppet last run on maerlant is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures
[19:54:37] <icinga-wm>	 RECOVERY - puppet last run on wtp1010 is OK: OK: Puppet is currently enabled, last run 43 seconds ago with 0 failures
[19:54:37] <icinga-wm>	 RECOVERY - puppet last run on netmon1001 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[19:54:47] <icinga-wm>	 RECOVERY - puppet last run on terbium is OK: OK: Puppet is currently enabled, last run 1 second ago with 0 failures
[19:54:47] <icinga-wm>	 RECOVERY - puppet last run on db1077 is OK: OK: Puppet is currently enabled, last run 27 seconds ago with 0 failures
[19:54:47] <icinga-wm>	 RECOVERY - puppet last run on fermium is OK: OK: Puppet is currently enabled, last run 43 seconds ago with 0 failures
[19:54:48] <icinga-wm>	 RECOVERY - puppet last run on mc1006 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[19:54:48] <icinga-wm>	 RECOVERY - puppet last run on dbproxy1008 is OK: OK: Puppet is currently enabled, last run 49 seconds ago with 0 failures
[19:54:48] <icinga-wm>	 RECOVERY - puppet last run on wtp1006 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures
[19:54:49] <icinga-wm>	 RECOVERY - puppet last run on mw1226 is OK: OK: Puppet is currently enabled, last run 12 seconds ago with 0 failures
[19:54:49] <icinga-wm>	 RECOVERY - puppet last run on mw2248 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[19:54:49] <icinga-wm>	 RECOVERY - puppet last run on elastic2020 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures
[19:54:50] <icinga-wm>	 RECOVERY - puppet last run on pybal-test2002 is OK: OK: Puppet is currently enabled, last run 50 seconds ago with 0 failures
[19:54:50] <icinga-wm>	 RECOVERY - puppet last run on sarin is OK: OK: Puppet is currently enabled, last run 37 seconds ago with 0 failures
[19:54:51] <icinga-wm>	 RECOVERY - puppet last run on labstore2004 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[19:54:51] <icinga-wm>	 RECOVERY - puppet last run on mw2091 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures
[19:54:52] <icinga-wm>	 RECOVERY - puppet last run on stat1004 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[19:55:08] <icinga-wm>	 RECOVERY - puppet last run on mw2157 is OK: OK: Puppet is currently enabled, last run 58 seconds ago with 0 failures
[19:55:10] <icinga-wm>	 RECOVERY - puppet last run on ganeti2002 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[19:55:11] <icinga-wm>	 RECOVERY - puppet last run on elastic2005 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures
[19:55:11] <icinga-wm>	 RECOVERY - puppet last run on rdb2003 is OK: OK: Puppet is currently enabled, last run 45 seconds ago with 0 failures
[19:55:14] <icinga-wm>	 RECOVERY - puppet last run on graphite1003 is OK: OK: Puppet is currently enabled, last run 52 seconds ago with 0 failures
[19:55:17] <icinga-wm>	 RECOVERY - puppet last run on mw1262 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures
[19:55:18] <icinga-wm>	 RECOVERY - puppet last run on iridium is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[19:55:27] <icinga-wm>	 RECOVERY - puppet last run on elastic2014 is OK: OK: Puppet is currently enabled, last run 47 seconds ago with 0 failures
[19:55:28] <icinga-wm>	 RECOVERY - puppet last run on mw1219 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[19:55:28] <icinga-wm>	 RECOVERY - puppet last run on mw1185 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[19:55:28] <icinga-wm>	 RECOVERY - puppet last run on ms-be1003 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[19:55:28] <icinga-wm>	 RECOVERY - puppet last run on ms-be1019 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures
[19:55:28] <icinga-wm>	 RECOVERY - puppet last run on mw1305 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures
[19:55:28] <icinga-wm>	 RECOVERY - puppet last run on ms-be2025 is OK: OK: Puppet is currently enabled, last run 10 seconds ago with 0 failures
[19:55:38] <icinga-wm>	 RECOVERY - puppet last run on mw1260 is OK: OK: Puppet is currently enabled, last run 23 seconds ago with 0 failures
[19:55:38] <icinga-wm>	 RECOVERY - puppet last run on wtp1016 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[19:55:38] <icinga-wm>	 RECOVERY - puppet last run on analytics1040 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[19:55:38] <icinga-wm>	 RECOVERY - puppet last run on ms-be1002 is OK: OK: Puppet is currently enabled, last run 57 seconds ago with 0 failures
[19:55:39] <icinga-wm>	 RECOVERY - puppet last run on analytics1028 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[19:55:39] <icinga-wm>	 RECOVERY - puppet last run on elastic1020 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[19:55:47] <icinga-wm>	 RECOVERY - puppet last run on chromium is OK: OK: Puppet is currently enabled, last run 56 seconds ago with 0 failures
[19:55:47] <icinga-wm>	 RECOVERY - puppet last run on subra is OK: OK: Puppet is currently enabled, last run 5 seconds ago with 0 failures
[19:55:48] <icinga-wm>	 RECOVERY - puppet last run on ms-be2020 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[19:55:49] <icinga-wm>	 RECOVERY - puppet last run on db1084 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[19:55:59] <icinga-wm>	 RECOVERY - puppet last run on ms-fe1004 is OK: OK: Puppet is currently enabled, last run 12 seconds ago with 0 failures
[19:56:00] <icinga-wm>	 RECOVERY - puppet last run on cp3039 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures
[19:56:00] <icinga-wm>	 RECOVERY - puppet last run on mw1200 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures
[19:56:01] <icinga-wm>	 RECOVERY - puppet last run on mw1174 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[19:56:04] <icinga-wm>	 RECOVERY - puppet last run on analytics1053 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[19:56:04] <icinga-wm>	 RECOVERY - puppet last run on analytics1041 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures
[19:56:07] <icinga-wm>	 RECOVERY - puppet last run on mw2141 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[19:56:07] <icinga-wm>	 RECOVERY - puppet last run on mw2208 is OK: OK: Puppet is currently enabled, last run 37 seconds ago with 0 failures
[19:56:08] <icinga-wm>	 RECOVERY - puppet last run on mw2081 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[19:56:17] <icinga-wm>	 RECOVERY - puppet last run on es2002 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures
[19:56:17] <icinga-wm>	 RECOVERY - puppet last run on mc2005 is OK: OK: Puppet is currently enabled, last run 57 seconds ago with 0 failures
[19:56:17] <icinga-wm>	 RECOVERY - puppet last run on mw2069 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[19:56:18] <icinga-wm>	 RECOVERY - puppet last run on db2064 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[19:56:18] <icinga-wm>	 RECOVERY - puppet last run on sca1001 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[19:56:19] <icinga-wm>	 RECOVERY - puppet last run on cp3016 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures
[19:56:19] <icinga-wm>	 RECOVERY - puppet last run on lvs3001 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[19:56:19] <icinga-wm>	 RECOVERY - puppet last run on mw1228 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[19:56:19] <icinga-wm>	 RECOVERY - puppet last run on db1046 is OK: OK: Puppet is currently enabled, last run 55 seconds ago with 0 failures
[19:56:29] <icinga-wm>	 RECOVERY - puppet last run on labsdb1008 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[19:56:29] <icinga-wm>	 RECOVERY - puppet last run on db1072 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures
[19:56:30] <icinga-wm>	 RECOVERY - puppet last run on labvirt1004 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures
[19:56:31] <icinga-wm>	 RECOVERY - puppet last run on db1059 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[19:56:31] <icinga-wm>	 RECOVERY - puppet last run on mc2001 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures
[19:56:37] <icinga-wm>	 RECOVERY - puppet last run on dubnium is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures
[19:56:38] <icinga-wm>	 RECOVERY - puppet last run on cp4009 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[19:56:38] <icinga-wm>	 RECOVERY - puppet last run on scb1001 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[19:56:49] <icinga-wm>	 RECOVERY - puppet last run on db1056 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[19:56:49] <icinga-wm>	 RECOVERY - puppet last run on labservices1001 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures
[19:56:49] <icinga-wm>	 RECOVERY - puppet last run on mc1017 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[19:56:49] <icinga-wm>	 RECOVERY - puppet last run on db1045 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[19:56:58] <icinga-wm>	 RECOVERY - puppet last run on elastic2006 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[19:57:00] <icinga-wm>	 RECOVERY - puppet last run on cp1068 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[19:57:07] <icinga-wm>	 RECOVERY - puppet last run on mw2234 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures
[19:57:07] <icinga-wm>	 RECOVERY - puppet last run on cp3017 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[19:57:09] <icinga-wm>	 RECOVERY - puppet last run on mw1251 is OK: OK: Puppet is currently enabled, last run 37 seconds ago with 0 failures
[19:57:10] <icinga-wm>	 RECOVERY - puppet last run on mw1220 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[19:57:18] <icinga-wm>	 RECOVERY - puppet last run on mw1284 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures
[19:57:18] <icinga-wm>	 RECOVERY - puppet last run on kafka1002 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[19:57:19] <icinga-wm>	 RECOVERY - puppet last run on wtp1012 is OK: OK: Puppet is currently enabled, last run 18 seconds ago with 0 failures
[19:57:19] <icinga-wm>	 RECOVERY - puppet last run on db1086 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[19:57:19] <icinga-wm>	 RECOVERY - puppet last run on db2058 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[19:57:27] <icinga-wm>	 RECOVERY - puppet last run on mw1203 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures
[19:57:28] <icinga-wm>	 RECOVERY - puppet last run on lvs2002 is OK: OK: Puppet is currently enabled, last run 30 seconds ago with 0 failures
[19:57:28] <icinga-wm>	 RECOVERY - puppet last run on scb2001 is OK: OK: Puppet is currently enabled, last run 31 seconds ago with 0 failures
[19:57:28] <icinga-wm>	 RECOVERY - puppet last run on es2013 is OK: OK: Puppet is currently enabled, last run 58 seconds ago with 0 failures
[19:57:28] <icinga-wm>	 RECOVERY - puppet last run on mw2228 is OK: OK: Puppet is currently enabled, last run 30 seconds ago with 0 failures
[19:57:28] <icinga-wm>	 RECOVERY - puppet last run on mw1276 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures
[19:57:28] <icinga-wm>	 RECOVERY - puppet last run on wtp2004 is OK: OK: Puppet is currently enabled, last run 33 seconds ago with 0 failures
[19:57:29] <icinga-wm>	 RECOVERY - puppet last run on wtp2015 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[19:57:29] <icinga-wm>	 RECOVERY - puppet last run on mw2217 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures
[19:57:30] <icinga-wm>	 RECOVERY - puppet last run on mw1172 is OK: OK: Puppet is currently enabled, last run 21 seconds ago with 0 failures
[19:57:30] <icinga-wm>	 RECOVERY - puppet last run on db2056 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[19:57:31] <icinga-wm>	 RECOVERY - puppet last run on db1015 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[19:57:31] <icinga-wm>	 RECOVERY - puppet last run on labnet1002 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures
[19:57:32] <icinga-wm>	 RECOVERY - puppet last run on mw1215 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures
[19:57:50] <icinga-wm>	 RECOVERY - puppet last run on mc2007 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[19:57:58] <icinga-wm>	 RECOVERY - puppet last run on analytics1047 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[19:57:58] <icinga-wm>	 RECOVERY - puppet last run on ununpentium is OK: OK: Puppet is currently enabled, last run 16 seconds ago with 0 failures
[19:57:59] <icinga-wm>	 RECOVERY - puppet last run on ms-be1014 is OK: OK: Puppet is currently enabled, last run 16 seconds ago with 0 failures
[19:58:07] <icinga-wm>	 RECOVERY - puppet last run on wtp2008 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures
[19:58:08] <icinga-wm>	 RECOVERY - puppet last run on mw1170 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures
[19:58:09] <icinga-wm>	 RECOVERY - puppet last run on stat1003 is OK: OK: Puppet is currently enabled, last run 34 seconds ago with 0 failures
[19:58:09] <icinga-wm>	 RECOVERY - puppet last run on ms-be1010 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[19:58:09] <icinga-wm>	 RECOVERY - puppet last run on mw2126 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[19:58:10] <icinga-wm>	 RECOVERY - puppet last run on alsafi is OK: OK: Puppet is currently enabled, last run 5 seconds ago with 0 failures
[19:58:18] <icinga-wm>	 RECOVERY - puppet last run on ms-be2022 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures
[19:58:18] <icinga-wm>	 RECOVERY - puppet last run on ms-be2026 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures
[19:58:19] <icinga-wm>	 RECOVERY - puppet last run on cp2013 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[19:58:19] <icinga-wm>	 RECOVERY - puppet last run on restbase1007 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures
[19:58:19] <icinga-wm>	 RECOVERY - puppet last run on restbase1008 is OK: OK: Puppet is currently enabled, last run 28 seconds ago with 0 failures
[19:58:19] <icinga-wm>	 RECOVERY - puppet last run on mw1249 is OK: OK: Puppet is currently enabled, last run 46 seconds ago with 0 failures
[19:58:28] <icinga-wm>	 RECOVERY - puppet last run on db1067 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures
[19:58:29] <icinga-wm>	 RECOVERY - puppet last run on elastic2007 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures
[19:58:34] <icinga-wm>	 RECOVERY - puppet last run on elastic2024 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[19:58:37] <icinga-wm>	 RECOVERY - puppet last run on mw1195 is OK: OK: Puppet is currently enabled, last run 2 seconds ago with 0 failures
[19:58:37] <icinga-wm>	 RECOVERY - puppet last run on cp3008 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures
[19:58:38] <icinga-wm>	 RECOVERY - puppet last run on mc1014 is OK: OK: Puppet is currently enabled, last run 16 seconds ago with 0 failures
[19:58:39] <icinga-wm>	 RECOVERY - puppet last run on druid1002 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures
[19:58:49] <icinga-wm>	 RECOVERY - puppet last run on elastic1039 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[19:58:49] <icinga-wm>	 RECOVERY - puppet last run on db1039 is OK: OK: Puppet is currently enabled, last run 34 seconds ago with 0 failures
[19:58:49] <icinga-wm>	 RECOVERY - puppet last run on mw2207 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures
[19:58:50] <icinga-wm>	 RECOVERY - puppet last run on mw2061 is OK: OK: Puppet is currently enabled, last run 38 seconds ago with 0 failures
[19:58:50] <icinga-wm>	 RECOVERY - puppet last run on mw2146 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[19:58:50] <icinga-wm>	 RECOVERY - puppet last run on mw2158 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures
[19:58:50] <icinga-wm>	 RECOVERY - puppet last run on mw2120 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[19:58:51] <icinga-wm>	 RECOVERY - puppet last run on ms-be2011 is OK: OK: Puppet is currently enabled, last run 39 seconds ago with 0 failures
[19:58:51] <icinga-wm>	 RECOVERY - puppet last run on mw2073 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[19:58:52] <icinga-wm>	 RECOVERY - puppet last run on db2016 is OK: OK: Puppet is currently enabled, last run 27 seconds ago with 0 failures
[19:58:52] <icinga-wm>	 RECOVERY - puppet last run on eventlog2001 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures
[19:58:53] <icinga-wm>	 RECOVERY - puppet last run on mw2077 is OK: OK: Puppet is currently enabled, last run 45 seconds ago with 0 failures
[19:58:53] <icinga-wm>	 RECOVERY - puppet last run on mc2006 is OK: OK: Puppet is currently enabled, last run 7 seconds ago with 0 failures
[19:58:54] <icinga-wm>	 RECOVERY - puppet last run on db2062 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[19:59:12] <icinga-wm>	 RECOVERY - puppet last run on einsteinium is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures
[19:59:13] <icinga-wm>	 RECOVERY - puppet last run on mw2177 is OK: OK: Puppet is currently enabled, last run 9 seconds ago with 0 failures
[19:59:13] <icinga-wm>	 RECOVERY - puppet last run on mw2136 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[19:59:13] <icinga-wm>	 RECOVERY - puppet last run on maps2004 is OK: OK: Puppet is currently enabled, last run 52 seconds ago with 0 failures
[19:59:13] <icinga-wm>	 RECOVERY - puppet last run on wtp2019 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[19:59:19] <icinga-wm>	 RECOVERY - puppet last run on bast4001 is OK: OK: Puppet is currently enabled, last run 22 seconds ago with 0 failures
[19:59:20] <icinga-wm>	 RECOVERY - puppet last run on db1049 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[19:59:20] <icinga-wm>	 RECOVERY - puppet last run on dbproxy1003 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[19:59:30] <icinga-wm>	 RECOVERY - puppet last run on mw1303 is OK: OK: Puppet is currently enabled, last run 12 seconds ago with 0 failures
[19:59:32] <icinga-wm>	 RECOVERY - puppet last run on rhenium is OK: OK: Puppet is currently enabled, last run 49 seconds ago with 0 failures
[19:59:39] <icinga-wm>	 RECOVERY - puppet last run on db2049 is OK: OK: Puppet is currently enabled, last run 19 seconds ago with 0 failures
[19:59:42] <icinga-wm>	 RECOVERY - puppet last run on mc2013 is OK: OK: Puppet is currently enabled, last run 20 seconds ago with 0 failures
[19:59:43] <icinga-wm>	 RECOVERY - puppet last run on mw2138 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[19:59:43] <icinga-wm>	 RECOVERY - puppet last run on mw1288 is OK: OK: Puppet is currently enabled, last run 11 seconds ago with 0 failures
[19:59:43] <icinga-wm>	 RECOVERY - puppet last run on mw1282 is OK: OK: Puppet is currently enabled, last run 17 seconds ago with 0 failures
[19:59:43] <icinga-wm>	 RECOVERY - puppet last run on kafka2002 is OK: OK: Puppet is currently enabled, last run 38 seconds ago with 0 failures
[19:59:44] <icinga-wm>	 RECOVERY - puppet last run on suhail is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[19:59:44] <icinga-wm>	 RECOVERY - puppet last run on mw2166 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[19:59:50] <icinga-wm>	 RECOVERY - puppet last run on cp4018 is OK: OK: Puppet is currently enabled, last run 43 seconds ago with 0 failures
[19:59:52] <icinga-wm>	 RECOVERY - puppet last run on mw1199 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures
[19:59:52] <icinga-wm>	 RECOVERY - puppet last run on labmon1001 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures
[19:59:59] <icinga-wm>	 RECOVERY - puppet last run on db1043 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures
[20:00:00] <icinga-wm>	 RECOVERY - puppet last run on snapshot1001 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures
[20:00:00] <icinga-wm>	 RECOVERY - puppet last run on elastic1038 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[20:00:01] <icinga-wm>	 RECOVERY - puppet last run on db1091 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[20:00:01] <icinga-wm>	 RECOVERY - puppet last run on db2068 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures
[20:00:11] <icinga-wm>	 RECOVERY - puppet last run on mw2235 is OK: OK: Puppet is currently enabled, last run 58 seconds ago with 0 failures
[20:00:11] <icinga-wm>	 RECOVERY - puppet last run on nihal is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures
[20:00:12] <icinga-wm>	 RECOVERY - puppet last run on mw1191 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[20:00:19] <icinga-wm>	 RECOVERY - puppet last run on ms-be1005 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[20:00:20] <icinga-wm>	 RECOVERY - puppet last run on ms-be2021 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures
[20:00:21] <icinga-wm>	 RECOVERY - puppet last run on mw1258 is OK: OK: Puppet is currently enabled, last run 24 seconds ago with 0 failures
[20:00:30] <icinga-wm>	 RECOVERY - puppet last run on graphite2001 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[20:00:32] <icinga-wm>	 RECOVERY - puppet last run on hafnium is OK: OK: Puppet is currently enabled, last run 7 seconds ago with 0 failures
[20:00:33] <icinga-wm>	 RECOVERY - puppet last run on mw2162 is OK: OK: Puppet is currently enabled, last run 11 seconds ago with 0 failures
[20:00:33] <icinga-wm>	 RECOVERY - puppet last run on mw1302 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures
[20:00:33] <icinga-wm>	 RECOVERY - puppet last run on db1094 is OK: OK: Puppet is currently enabled, last run 55 seconds ago with 0 failures
[20:00:40] <icinga-wm>	 RECOVERY - puppet last run on cp1048 is OK: OK: Puppet is currently enabled, last run 56 seconds ago with 0 failures
[20:00:40] <icinga-wm>	 RECOVERY - puppet last run on lvs1009 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures
[20:00:41] <icinga-wm>	 RECOVERY - puppet last run on mw2181 is OK: OK: Puppet is currently enabled, last run 48 seconds ago with 0 failures
[20:00:41] <icinga-wm>	 RECOVERY - puppet last run on mw2232 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[20:00:41] <icinga-wm>	 RECOVERY - puppet last run on mw1234 is OK: OK: Puppet is currently enabled, last run 21 seconds ago with 0 failures
[20:00:41] <icinga-wm>	 RECOVERY - puppet last run on notebook1002 is OK: OK: Puppet is currently enabled, last run 58 seconds ago with 0 failures
[20:00:50] <icinga-wm>	 RECOVERY - puppet last run on ms-be1008 is OK: OK: Puppet is currently enabled, last run 59 seconds ago with 0 failures
[20:00:50] <icinga-wm>	 RECOVERY - puppet last run on elastic2010 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures
[20:00:52] <icinga-wm>	 RECOVERY - puppet last run on mw1202 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[20:01:00] <icinga-wm>	 RECOVERY - puppet last run on ms-be1020 is OK: OK: Puppet is currently enabled, last run 40 seconds ago with 0 failures
[20:01:00] <icinga-wm>	 RECOVERY - puppet last run on mw1168 is OK: OK: Puppet is currently enabled, last run 37 seconds ago with 0 failures
[20:01:00] <icinga-wm>	 RECOVERY - puppet last run on labsdb1007 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures
[20:01:00] <icinga-wm>	 RECOVERY - puppet last run on planet2001 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures
[20:01:01] <icinga-wm>	 RECOVERY - puppet last run on es2019 is OK: OK: Puppet is currently enabled, last run 30 seconds ago with 0 failures
[20:01:01] <icinga-wm>	 RECOVERY - puppet last run on wtp2003 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures
[20:01:13] <icinga-wm>	 RECOVERY - puppet last run on meitnerium is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[20:01:15] <icinga-wm>	 RECOVERY - puppet last run on mw1261 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[20:01:15] <icinga-wm>	 RECOVERY - puppet last run on wtp1022 is OK: OK: Puppet is currently enabled, last run 53 seconds ago with 0 failures
[20:01:16] <icinga-wm>	 RECOVERY - puppet last run on db2051 is OK: OK: Puppet is currently enabled, last run 39 seconds ago with 0 failures
[20:01:16] <icinga-wm>	 RECOVERY - puppet last run on cp3015 is OK: OK: Puppet is currently enabled, last run 14 seconds ago with 0 failures
[20:01:16] <icinga-wm>	 RECOVERY - puppet last run on cp2006 is OK: OK: Puppet is currently enabled, last run 19 seconds ago with 0 failures
[20:01:20] <icinga-wm>	 RECOVERY - puppet last run on db1041 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures
[20:01:20] <icinga-wm>	 RECOVERY - puppet last run on lvs2006 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures
[20:01:31] <icinga-wm>	 RECOVERY - puppet last run on ms-be2015 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures
[20:01:32] <icinga-wm>	 RECOVERY - puppet last run on mw2132 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[20:01:32] <icinga-wm>	 RECOVERY - puppet last run on sca2001 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[20:01:32] <icinga-wm>	 RECOVERY - puppet last run on maps-test2004 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[20:01:32] <icinga-wm>	 RECOVERY - puppet last run on db2066 is OK: OK: Puppet is currently enabled, last run 58 seconds ago with 0 failures
[20:01:32] <icinga-wm>	 RECOVERY - puppet last run on db2033 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[20:01:33] <icinga-wm>	 RECOVERY - puppet last run on mw1192 is OK: OK: Puppet is currently enabled, last run 48 seconds ago with 0 failures
[20:01:33] <icinga-wm>	 RECOVERY - puppet last run on wtp1019 is OK: OK: Puppet is currently enabled, last run 20 seconds ago with 0 failures
[20:01:34] <icinga-wm>	 RECOVERY - puppet last run on cp3032 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures
[20:01:34] <icinga-wm>	 RECOVERY - puppet last run on analytics1037 is OK: OK: Puppet is currently enabled, last run 13 seconds ago with 0 failures
[20:01:35] <icinga-wm>	 RECOVERY - puppet last run on eeden is OK: OK: Puppet is currently enabled, last run 44 seconds ago with 0 failures
[20:01:39] <icinga-wm>	 RECOVERY - puppet last run on mw1286 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[20:01:39] <icinga-wm>	 RECOVERY - puppet last run on analytics1034 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[20:01:40] <icinga-wm>	 RECOVERY - puppet last run on rdb1007 is OK: OK: Puppet is currently enabled, last run 58 seconds ago with 0 failures
[20:01:40] <icinga-wm>	 RECOVERY - puppet last run on ms-be1013 is OK: OK: Puppet is currently enabled, last run 18 seconds ago with 0 failures
[20:01:41] <icinga-wm>	 RECOVERY - puppet last run on mc2008 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[20:01:41] <icinga-wm>	 RECOVERY - puppet last run on mw1246 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[20:01:41] <icinga-wm>	 RECOVERY - puppet last run on rdb1004 is OK: OK: Puppet is currently enabled, last run 29 seconds ago with 0 failures
[20:01:50] <icinga-wm>	 RECOVERY - puppet last run on mw2072 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[20:01:50] <icinga-wm>	 RECOVERY - puppet last run on cp2009 is OK: OK: Puppet is currently enabled, last run 11 seconds ago with 0 failures
[20:01:50] <icinga-wm>	 RECOVERY - puppet last run on mw1216 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[20:01:51] <icinga-wm>	 RECOVERY - puppet last run on restbase2003 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[20:01:51] <icinga-wm>	 RECOVERY - puppet last run on mw1259 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures
[20:01:51] <icinga-wm>	 RECOVERY - puppet last run on mw1264 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[20:01:52] <icinga-wm>	 RECOVERY - puppet last run on analytics1050 is OK: OK: Puppet is currently enabled, last run 13 seconds ago with 0 failures
[20:01:52] <icinga-wm>	 RECOVERY - puppet last run on mw1244 is OK: OK: Puppet is currently enabled, last run 55 seconds ago with 0 failures
[20:02:00] <icinga-wm>	 RECOVERY - puppet last run on wtp1023 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[20:02:02] <icinga-wm>	 RECOVERY - puppet last run on elastic2001 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[20:02:03] <icinga-wm>	 RECOVERY - puppet last run on mw1287 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[20:02:04] <icinga-wm>	 RECOVERY - puppet last run on elastic1028 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[20:02:04] <icinga-wm>	 RECOVERY - puppet last run on elastic1031 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[20:02:04] <icinga-wm>	 RECOVERY - puppet last run on neon is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[20:02:10] <icinga-wm>	 RECOVERY - puppet last run on wtp1017 is OK: OK: Puppet is currently enabled, last run 47 seconds ago with 0 failures
[20:02:10] <icinga-wm>	 RECOVERY - puppet last run on db2035 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[20:02:13] <icinga-wm>	 RECOVERY - puppet last run on db1054 is OK: OK: Puppet is currently enabled, last run 50 seconds ago with 0 failures
[20:02:21] <icinga-wm>	 RECOVERY - puppet last run on nescio is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[20:02:31] <icinga-wm>	 RECOVERY - puppet last run on rcs1001 is OK: OK: Puppet is currently enabled, last run 40 seconds ago with 0 failures
[20:02:40] <icinga-wm>	 RECOVERY - puppet last run on db2063 is OK: OK: Puppet is currently enabled, last run 40 seconds ago with 0 failures
[20:02:40] <icinga-wm>	 RECOVERY - puppet last run on auth2001 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[20:02:40] <icinga-wm>	 RECOVERY - puppet last run on wdqs2002 is OK: OK: Puppet is currently enabled, last run 14 seconds ago with 0 failures
[20:02:40] <icinga-wm>	 RECOVERY - puppet last run on es2015 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures
[20:02:40] <icinga-wm>	 RECOVERY - puppet last run on analytics1039 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[20:02:40] <icinga-wm>	 RECOVERY - puppet last run on kafka1020 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures
[20:02:41] <icinga-wm>	 RECOVERY - puppet last run on es2017 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[20:02:41] <icinga-wm>	 RECOVERY - puppet last run on mw1229 is OK: OK: Puppet is currently enabled, last run 13 seconds ago with 0 failures
[20:02:41] <icinga-wm>	 RECOVERY - puppet last run on mw2108 is OK: OK: Puppet is currently enabled, last run 24 seconds ago with 0 failures
[20:02:42] <icinga-wm>	 RECOVERY - puppet last run on mw1257 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[20:02:42] <icinga-wm>	 RECOVERY - puppet last run on mw2125 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[20:02:43] <icinga-wm>	 RECOVERY - puppet last run on mw1183 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[20:02:43] <icinga-wm>	 RECOVERY - puppet last run on ms-be1001 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[20:02:44] <icinga-wm>	 RECOVERY - puppet last run on wtp1003 is OK: OK: Puppet is currently enabled, last run 37 seconds ago with 0 failures
[20:02:50] <icinga-wm>	 RECOVERY - puppet last run on cp2004 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[20:03:02] <elukey>	 !log stopped ircecho on neon temporarily
[20:03:07] <morebots>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master
[20:03:27] <elukey>	 Luke081515: I think this should stop the notifications
[20:08:43] <Luke081515>	 seems so :)
[20:11:47] <elukey>	 !log re-enabled ircecho on neon
[20:11:51] <morebots>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master
[20:28:01] <icinga-wm>	 PROBLEM - Redis status tcp_6479 on rdb2006 is CRITICAL: CRITICAL: replication_delay is 644 600 - REDIS on 10.192.48.44:6479 has 1 databases (db0) with 5239259 keys - replication_delay is 644
[20:30:53] <icinga-wm>	 RECOVERY - Redis status tcp_6479 on rdb2006 is OK: OK: REDIS on 10.192.48.44:6479 has 1 databases (db0) with 5190434 keys - replication_delay is 0
[20:39:38] <elukey>	 all right going afk, just sent an email to ops
[20:39:44] <elukey>	 will check later 
[21:21:16] <grrrit-wm>	 (03CR) 10Alex Monk: "Error: Could not retrieve catalog from remote server: Error 400 on SERVER: Could not find class role::elasticsearch::server for deployment" [puppet] - 10https://gerrit.wikimedia.org/r/304067 (owner: 10Gehel)
[21:25:53] <gehel>	 Krenair: ^ thanks!
[21:26:04] <gehel>	 Having a look...
[21:26:26] <Krenair>	 gehel, shinken has been alerting about this for 3 days...
[21:28:00] <wikibugs>	 07Puppet, 10Beta-Cluster-Infrastructure: Puppet failing on deployment-conf03 due to missing files - https://phabricator.wikimedia.org/T144703#2607811 (10AlexMonk-WMF)
[21:29:42] <grrrit-wm>	 (03Draft1) 10Paladox: role/cxserver: Fix role::cxserver not in autoload module layout [puppet] - 10https://gerrit.wikimedia.org/r/308498 
[21:29:47] <grrrit-wm>	 (03Draft2) 10Paladox: role/cxserver: Fix role::cxserver not in autoload module layout [puppet] - 10https://gerrit.wikimedia.org/r/308498 
[21:30:52] <grrrit-wm>	 (03PS3) 10Paladox: role/cxserver: Fix role::cxserver not in autoload module layout [puppet] - 10https://gerrit.wikimedia.org/r/308498 (https://phabricator.wikimedia.org/T93645) 
[22:01:18] <icinga-wm>	 PROBLEM - Redis status tcp_6479 on rdb2006 is CRITICAL: CRITICAL: replication_delay is 661 600 - REDIS on 10.192.48.44:6479 has 1 databases (db0) with 5197016 keys - replication_delay is 661
[22:06:08] <icinga-wm>	 RECOVERY - Redis status tcp_6479 on rdb2006 is OK: OK: REDIS on 10.192.48.44:6479 has 1 databases (db0) with 5191019 keys - replication_delay is 0
[23:04:26] <grrrit-wm>	 (03PS1) 10Alexandros Kosiaris: Revert "puppetmaster: Vary ssldir in frontend on masterness" [puppet] - 10https://gerrit.wikimedia.org/r/308500 
[23:10:48] <icinga-wm>	 PROBLEM - Text HTTP 5xx reqs/min on graphite1001 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [1000.0]
[23:11:20] <grrrit-wm>	 (03CR) 10Alexandros Kosiaris: [C: 032] Revert "puppetmaster: Vary ssldir in frontend on masterness" [puppet] - 10https://gerrit.wikimedia.org/r/308500 (owner: 10Alexandros Kosiaris)
[23:12:58] <icinga-wm>	 PROBLEM - Esams HTTP 5xx reqs/min on graphite1001 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [1000.0]
[23:14:19] <icinga-wm>	 PROBLEM - puppet last run on cp3004 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 6 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[generate_varnishkafka_webrequest_gmond_pyconf]
[23:15:19] <icinga-wm>	 RECOVERY - Esams HTTP 5xx reqs/min on graphite1001 is OK: OK: Less than 1.00% above the threshold [250.0]
[23:15:39] <icinga-wm>	 RECOVERY - Text HTTP 5xx reqs/min on graphite1001 is OK: OK: Less than 1.00% above the threshold [250.0]
[23:22:47] <grrrit-wm>	 (03PS1) 10Alexandros Kosiaris: puppet: CRL is now a file [puppet] - 10https://gerrit.wikimedia.org/r/308501 
[23:25:35] <grrrit-wm>	 (03CR) 10Alexandros Kosiaris: [C: 032] puppet: CRL is now a file [puppet] - 10https://gerrit.wikimedia.org/r/308501 (owner: 10Alexandros Kosiaris)
[23:31:06] <grrrit-wm>	 (03PS1) 10Alexandros Kosiaris: puppetmasters: Allow ssh puppet-merge from all frontends [puppet] - 10https://gerrit.wikimedia.org/r/308502 
[23:38:10] <grrrit-wm>	 (03CR) 10Alexandros Kosiaris: [C: 032] puppetmasters: Allow ssh puppet-merge from all frontends [puppet] - 10https://gerrit.wikimedia.org/r/308502 (owner: 10Alexandros Kosiaris)
[23:39:09] <icinga-wm>	 RECOVERY - puppet last run on cp3004 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[23:43:19] <icinga-wm>	 RECOVERY - puppetmaster backend https on puppetmaster2001 is OK: HTTP OK: Status line output matched 400 - 333 bytes in 1.328 second response time
[23:43:39] <icinga-wm>	 RECOVERY - puppetmaster https on puppetmaster2001 is OK: HTTP OK: Status line output matched 400 - 331 bytes in 0.204 second response time
[23:47:19] <grrrit-wm>	 (03CR) 10Alexandros Kosiaris: [C: 04-1] "file should be modules/role/manifests/cxserver/server.pp and the class should be renamed to modules::cxserver::server" [puppet] - 10https://gerrit.wikimedia.org/r/308498 (https://phabricator.wikimedia.org/T93645) (owner: 10Paladox)
[23:57:09] <icinga-wm>	 PROBLEM - puppet last run on scb2001 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[23:59:40] <icinga-wm>	 RECOVERY - puppet last run on scb2001 is OK: OK: Puppet is currently enabled, last run 20 seconds ago with 0 failures