[00:07:24] <icinga-wm>	 RECOVERY - puppet last run on db2042 is OK: OK: Puppet is currently enabled, last run 39 seconds ago with 0 failures
[00:11:24] <icinga-wm>	 PROBLEM - puppet last run on ms-be2012 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[00:17:24] <icinga-wm>	 RECOVERY - puppet last run on mw2168 is OK: OK: Puppet is currently enabled, last run 17 seconds ago with 0 failures
[00:17:24] <icinga-wm>	 RECOVERY - puppet last run on mc2023 is OK: OK: Puppet is currently enabled, last run 56 seconds ago with 0 failures
[00:17:24] <icinga-wm>	 PROBLEM - puppet last run on mw2127 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[00:18:24] <icinga-wm>	 RECOVERY - puppet last run on cp4015 is OK: OK: Puppet is currently enabled, last run 40 seconds ago with 0 failures
[00:26:24] <icinga-wm>	 PROBLEM - puppet last run on elastic2006 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[00:26:24] <icinga-wm>	 PROBLEM - puppet last run on elastic2014 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[00:26:24] <icinga-wm>	 RECOVERY - puppet last run on mc2026 is OK: OK: Puppet is currently enabled, last run 3 seconds ago with 0 failures
[00:38:24] <icinga-wm>	 PROBLEM - puppet last run on mw2211 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[00:40:24] <icinga-wm>	 RECOVERY - puppet last run on ms-be2012 is OK: OK: Puppet is currently enabled, last run 39 seconds ago with 0 failures
[00:45:24] <icinga-wm>	 RECOVERY - puppet last run on mw2127 is OK: OK: Puppet is currently enabled, last run 27 seconds ago with 0 failures
[00:47:24] <icinga-wm>	 PROBLEM - puppet last run on mw2176 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[00:48:24] <icinga-wm>	 PROBLEM - puppet last run on db2047 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[00:48:24] <icinga-wm>	 PROBLEM - puppet last run on graphite2002 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[00:48:24] <icinga-wm>	 PROBLEM - puppet last run on mw2182 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[00:50:24] <icinga-wm>	 PROBLEM - puppet last run on analytics1051 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[00:54:24] <icinga-wm>	 RECOVERY - puppet last run on elastic2014 is OK: OK: Puppet is currently enabled, last run 45 seconds ago with 0 failures
[00:55:24] <icinga-wm>	 RECOVERY - puppet last run on elastic2006 is OK: OK: Puppet is currently enabled, last run 44 seconds ago with 0 failures
[00:58:24] <icinga-wm>	 PROBLEM - puppet last run on cp4010 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[00:58:34] <icinga-wm>	 PROBLEM - puppet last run on es2018 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[01:07:24] <icinga-wm>	 PROBLEM - puppet last run on mw2195 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[01:07:24] <icinga-wm>	 RECOVERY - puppet last run on mw2211 is OK: OK: Puppet is currently enabled, last run 51 seconds ago with 0 failures
[01:16:24] <icinga-wm>	 RECOVERY - puppet last run on db2047 is OK: OK: Puppet is currently enabled, last run 21 seconds ago with 0 failures
[01:16:24] <icinga-wm>	 RECOVERY - puppet last run on mw2176 is OK: OK: Puppet is currently enabled, last run 50 seconds ago with 0 failures
[01:16:24] <icinga-wm>	 RECOVERY - puppet last run on mw2182 is OK: OK: Puppet is currently enabled, last run 37 seconds ago with 0 failures
[01:17:24] <icinga-wm>	 PROBLEM - puppet last run on mw2093 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[01:17:24] <icinga-wm>	 RECOVERY - puppet last run on graphite2002 is OK: OK: Puppet is currently enabled, last run 51 seconds ago with 0 failures
[01:17:24] <icinga-wm>	 PROBLEM - puppet last run on cp2016 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[01:17:25] <icinga-wm>	 PROBLEM - puppet last run on mw2196 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[01:19:24] <icinga-wm>	 RECOVERY - puppet last run on analytics1051 is OK: OK: Puppet is currently enabled, last run 13 seconds ago with 0 failures
[01:26:24] <icinga-wm>	 RECOVERY - puppet last run on cp4010 is OK: OK: Puppet is currently enabled, last run 20 seconds ago with 0 failures
[01:27:24] <icinga-wm>	 PROBLEM - puppet last run on mw2129 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[01:27:34] <icinga-wm>	 RECOVERY - puppet last run on es2018 is OK: OK: Puppet is currently enabled, last run 39 seconds ago with 0 failures
[01:29:54] <icinga-wm>	 PROBLEM - Postgres Replication Lag on maps1003 is CRITICAL: CRITICAL - Rep Delay is: 82511.324548 Seconds
[01:29:54] <icinga-wm>	 PROBLEM - Postgres Replication Lag on maps1004 is CRITICAL: CRITICAL - Rep Delay is: 82511.33063 Seconds
[01:30:14] <icinga-wm>	 PROBLEM - Postgres Replication Lag on maps1002 is CRITICAL: CRITICAL - Rep Delay is: 82535.018635 Seconds
[01:30:24] <icinga-wm>	 PROBLEM - Postgres Replication Lag on maps2004 is CRITICAL: CRITICAL - Rep Delay is: 83859.649195 Seconds
[01:30:24] <icinga-wm>	 PROBLEM - Postgres Replication Lag on maps2003 is CRITICAL: CRITICAL - Rep Delay is: 83859.662721 Seconds
[01:32:34] <icinga-wm>	 PROBLEM - Postgres Replication Lag on maps2002 is CRITICAL: CRITICAL - Rep Delay is: 83987.743129 Seconds
[01:35:24] <icinga-wm>	 RECOVERY - puppet last run on mw2195 is OK: OK: Puppet is currently enabled, last run 24 seconds ago with 0 failures
[01:36:24] <icinga-wm>	 PROBLEM - puppet last run on mw2230 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[01:40:54] <icinga-wm>	 RECOVERY - Postgres Replication Lag on maps1004 is OK: OK - Rep Delay is: 0.0 Seconds
[01:43:54] <icinga-wm>	 PROBLEM - Postgres Replication Lag on maps1004 is CRITICAL: CRITICAL - Rep Delay is: 83351.389158 Seconds
[01:45:24] <icinga-wm>	 PROBLEM - puppet last run on elastic2017 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[01:45:24] <icinga-wm>	 RECOVERY - puppet last run on mw2093 is OK: OK: Puppet is currently enabled, last run 19 seconds ago with 0 failures
[01:45:25] <icinga-wm>	 RECOVERY - puppet last run on mw2196 is OK: OK: Puppet is currently enabled, last run 43 seconds ago with 0 failures
[01:51:14] <icinga-wm>	 RECOVERY - Postgres Replication Lag on maps1002 is OK: OK - Rep Delay is: 0.0 Seconds
[01:52:24] <icinga-wm>	 RECOVERY - Postgres Replication Lag on maps2004 is OK: OK - Rep Delay is: 36.86128 Seconds
[01:52:24] <icinga-wm>	 RECOVERY - Postgres Replication Lag on maps2003 is OK: OK - Rep Delay is: 37.806492 Seconds
[01:52:34] <icinga-wm>	 RECOVERY - Postgres Replication Lag on maps2002 is OK: OK - Rep Delay is: 44.857081 Seconds
[01:52:54] <icinga-wm>	 RECOVERY - Postgres Replication Lag on maps1003 is OK: OK - Rep Delay is: 46.932535 Seconds
[01:52:54] <icinga-wm>	 RECOVERY - Postgres Replication Lag on maps1004 is OK: OK - Rep Delay is: 46.935732 Seconds
[01:55:24] <icinga-wm>	 RECOVERY - puppet last run on mw2129 is OK: OK: Puppet is currently enabled, last run 11 seconds ago with 0 failures
[01:56:24] <icinga-wm>	 PROBLEM - puppet last run on rdb2003 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[01:56:24] <icinga-wm>	 PROBLEM - puppet last run on mw2208 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[02:04:24] <icinga-wm>	 RECOVERY - puppet last run on mw2230 is OK: OK: Puppet is currently enabled, last run 43 seconds ago with 0 failures
[02:14:24] <icinga-wm>	 RECOVERY - puppet last run on elastic2017 is OK: OK: Puppet is currently enabled, last run 50 seconds ago with 0 failures
[02:14:24] <icinga-wm>	 PROBLEM - puppet last run on mw2220 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[02:15:24] <icinga-wm>	 RECOVERY - puppet last run on cp2016 is OK: OK: Puppet is currently enabled, last run 17 seconds ago with 0 failures
[02:16:14] <icinga-wm>	 PROBLEM - puppet last run on db1050 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[02:20:41] <logmsgbot>	 !log l10nupdate@tin scap sync-l10n completed (1.29.0-wmf.17) (duration: 07m 20s)
[02:20:50] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[02:24:24] <icinga-wm>	 RECOVERY - puppet last run on rdb2003 is OK: OK: Puppet is currently enabled, last run 10 seconds ago with 0 failures
[02:24:24] <icinga-wm>	 RECOVERY - puppet last run on mw2208 is OK: OK: Puppet is currently enabled, last run 6 seconds ago with 0 failures
[02:24:34] <icinga-wm>	 PROBLEM - puppet last run on db2011 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[02:26:05] <logmsgbot>	 !log l10nupdate@tin ResourceLoader cache refresh completed at Sun Mar 26 02:26:05 UTC 2017 (duration 5m 25s)
[02:26:12] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[02:32:04] <icinga-wm>	 PROBLEM - puppet last run on mw1261 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[02:34:24] <icinga-wm>	 PROBLEM - puppet last run on mw2205 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[02:43:34] <icinga-wm>	 PROBLEM - Postgres Replication Lag on maps2002 is CRITICAL: CRITICAL - Rep Delay is: 1905.995407 Seconds
[02:44:14] <icinga-wm>	 RECOVERY - puppet last run on db1050 is OK: OK: Puppet is currently enabled, last run 5 seconds ago with 0 failures
[02:45:34] <icinga-wm>	 RECOVERY - Postgres Replication Lag on maps2002 is OK: OK - Rep Delay is: 0.0 Seconds
[02:46:24] <icinga-wm>	 PROBLEM - puppet last run on mw2196 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[02:46:34] <icinga-wm>	 PROBLEM - puppet last run on rdb2005 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[02:47:24] <icinga-wm>	 PROBLEM - puppet last run on mw2093 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[02:47:44] <icinga-wm>	 PROBLEM - Codfw HTTP 5xx reqs/min on graphite1001 is CRITICAL: CRITICAL: 11.11% of data above the critical threshold [1000.0]
[02:48:14] <icinga-wm>	 PROBLEM - Eqiad HTTP 5xx reqs/min on graphite1001 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [1000.0]
[02:51:24] <icinga-wm>	 PROBLEM - Misc HTTP 5xx reqs/min on graphite1001 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [1000.0]
[02:52:14] <icinga-wm>	 PROBLEM - Eqiad HTTP 5xx reqs/min on graphite1001 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [1000.0]
[02:52:34] <icinga-wm>	 RECOVERY - puppet last run on db2011 is OK: OK: Puppet is currently enabled, last run 47 seconds ago with 0 failures
[02:54:04] <icinga-wm>	 PROBLEM - puppet last run on wtp1013 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[02:57:24] <icinga-wm>	 PROBLEM - puppet last run on es2013 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[02:57:25] <icinga-wm>	 PROBLEM - puppet last run on mw2228 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[03:00:04] <icinga-wm>	 RECOVERY - puppet last run on mw1261 is OK: OK: Puppet is currently enabled, last run 25 seconds ago with 0 failures
[03:02:25] <icinga-wm>	 RECOVERY - puppet last run on mw2205 is OK: OK: Puppet is currently enabled, last run 4 seconds ago with 0 failures
[03:06:14] <icinga-wm>	 RECOVERY - Eqiad HTTP 5xx reqs/min on graphite1001 is OK: OK: Less than 1.00% above the threshold [250.0]
[03:07:44] <icinga-wm>	 RECOVERY - Codfw HTTP 5xx reqs/min on graphite1001 is OK: OK: Less than 1.00% above the threshold [250.0]
[03:08:24] <icinga-wm>	 RECOVERY - Misc HTTP 5xx reqs/min on graphite1001 is OK: OK: Less than 1.00% above the threshold [250.0]
[03:13:24] <icinga-wm>	 RECOVERY - puppet last run on mw2220 is OK: OK: Puppet is currently enabled, last run 43 seconds ago with 0 failures
[03:13:25] <icinga-wm>	 PROBLEM - puppet last run on mw2201 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[03:15:24] <icinga-wm>	 RECOVERY - puppet last run on mw2093 is OK: OK: Puppet is currently enabled, last run 27 seconds ago with 0 failures
[03:15:34] <icinga-wm>	 RECOVERY - puppet last run on rdb2005 is OK: OK: Puppet is currently enabled, last run 19 seconds ago with 0 failures
[03:16:24] <icinga-wm>	 PROBLEM - puppet last run on mc2024 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[03:16:24] <icinga-wm>	 RECOVERY - puppet last run on mw2196 is OK: OK: Puppet is currently enabled, last run 50 seconds ago with 0 failures
[03:21:04] <icinga-wm>	 RECOVERY - puppet last run on wtp1013 is OK: OK: Puppet is currently enabled, last run 32 seconds ago with 0 failures
[03:23:14] <icinga-wm>	 PROBLEM - MariaDB Slave Lag: s1 on dbstore1002 is CRITICAL: CRITICAL slave_sql_lag Replication lag: 634.03 seconds
[03:25:14] <icinga-wm>	 RECOVERY - MariaDB Slave Lag: s1 on dbstore1002 is OK: OK slave_sql_lag Replication lag: 298.18 seconds
[03:26:24] <icinga-wm>	 RECOVERY - puppet last run on es2013 is OK: OK: Puppet is currently enabled, last run 29 seconds ago with 0 failures
[03:26:25] <icinga-wm>	 RECOVERY - puppet last run on mw2228 is OK: OK: Puppet is currently enabled, last run 6 seconds ago with 0 failures
[03:26:25] <icinga-wm>	 PROBLEM - puppet last run on ms-be2025 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[03:27:24] <icinga-wm>	 PROBLEM - puppet last run on mw2129 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[03:42:24] <icinga-wm>	 RECOVERY - puppet last run on mw2201 is OK: OK: Puppet is currently enabled, last run 51 seconds ago with 0 failures
[03:42:34] <icinga-wm>	 PROBLEM - puppet last run on mw2102 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[03:43:24] <icinga-wm>	 PROBLEM - Postgres Replication Lag on maps2003 is CRITICAL: CRITICAL - Rep Delay is: 3021.101657 Seconds
[03:44:24] <icinga-wm>	 PROBLEM - Postgres Replication Lag on maps2004 is CRITICAL: CRITICAL - Rep Delay is: 3080.403793 Seconds
[03:44:24] <icinga-wm>	 RECOVERY - puppet last run on mc2024 is OK: OK: Puppet is currently enabled, last run 32 seconds ago with 0 failures
[03:44:24] <icinga-wm>	 RECOVERY - Postgres Replication Lag on maps2003 is OK: OK - Rep Delay is: 0.0 Seconds
[03:45:24] <icinga-wm>	 RECOVERY - Postgres Replication Lag on maps2004 is OK: OK - Rep Delay is: 0.0 Seconds
[03:52:24] <icinga-wm>	 PROBLEM - puppet last run on maps-test2001 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[03:54:24] <icinga-wm>	 RECOVERY - puppet last run on ms-be2025 is OK: OK: Puppet is currently enabled, last run 1 second ago with 0 failures
[03:55:24] <icinga-wm>	 RECOVERY - puppet last run on mw2129 is OK: OK: Puppet is currently enabled, last run 21 seconds ago with 0 failures
[04:01:24] <icinga-wm>	 PROBLEM - puppet last run on mc2025 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[04:01:24] <icinga-wm>	 PROBLEM - puppet last run on wtp2003 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[04:01:34] <icinga-wm>	 PROBLEM - puppet last run on maps-test2004 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[04:10:34] <icinga-wm>	 RECOVERY - puppet last run on mw2102 is OK: OK: Puppet is currently enabled, last run 27 seconds ago with 0 failures
[04:13:44] <icinga-wm>	 PROBLEM - mailman I/O stats on fermium is CRITICAL: CRITICAL - I/O stats: Transfers/Sec=4910.10 Read Requests/Sec=3441.90 Write Requests/Sec=4.60 KBytes Read/Sec=13819.20 KBytes_Written/Sec=150.40
[04:15:34] <icinga-wm>	 PROBLEM - Postgres Replication Lag on maps2002 is CRITICAL: CRITICAL - Rep Delay is: 4948.303184 Seconds
[04:16:34] <icinga-wm>	 RECOVERY - Postgres Replication Lag on maps2002 is OK: OK - Rep Delay is: 0.0 Seconds
[04:18:24] <icinga-wm>	 PROBLEM - puppet last run on mw2260 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[04:18:24] <icinga-wm>	 PROBLEM - puppet last run on mw2182 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[04:19:25] <icinga-wm>	 PROBLEM - puppet last run on cp3030 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[04:20:24] <icinga-wm>	 PROBLEM - Postgres Replication Lag on maps2004 is CRITICAL: CRITICAL - Rep Delay is: 5241.463255 Seconds
[04:21:24] <icinga-wm>	 RECOVERY - puppet last run on maps-test2001 is OK: OK: Puppet is currently enabled, last run 58 seconds ago with 0 failures
[04:21:24] <icinga-wm>	 RECOVERY - Postgres Replication Lag on maps2004 is OK: OK - Rep Delay is: 0.0 Seconds
[04:21:44] <icinga-wm>	 RECOVERY - mailman I/O stats on fermium is OK: OK - I/O stats: Transfers/Sec=11.60 Read Requests/Sec=0.90 Write Requests/Sec=4.20 KBytes Read/Sec=8.00 KBytes_Written/Sec=150.00
[04:26:34] <icinga-wm>	 RECOVERY - puppet last run on maps-test2004 is OK: OK: Puppet is currently enabled, last run 52 seconds ago with 0 failures
[04:28:24] <icinga-wm>	 PROBLEM - puppet last run on mw2136 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[04:28:34] <icinga-wm>	 PROBLEM - puppet last run on mw2095 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[04:29:24] <icinga-wm>	 RECOVERY - puppet last run on wtp2003 is OK: OK: Puppet is currently enabled, last run 1 second ago with 0 failures
[04:30:24] <icinga-wm>	 RECOVERY - puppet last run on mc2025 is OK: OK: Puppet is currently enabled, last run 55 seconds ago with 0 failures
[04:30:24] <icinga-wm>	 PROBLEM - Postgres Replication Lag on maps2004 is CRITICAL: CRITICAL - Rep Delay is: 5840.67494 Seconds
[04:31:24] <icinga-wm>	 RECOVERY - Postgres Replication Lag on maps2004 is OK: OK - Rep Delay is: 0.0 Seconds
[04:34:34] <icinga-wm>	 PROBLEM - puppet last run on mw2204 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[04:44:24] <icinga-wm>	 PROBLEM - puppet last run on mw2220 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[04:45:24] <icinga-wm>	 PROBLEM - Postgres Replication Lag on maps2003 is CRITICAL: CRITICAL - Rep Delay is: 6742.689169 Seconds
[04:46:24] <icinga-wm>	 RECOVERY - Postgres Replication Lag on maps2003 is OK: OK - Rep Delay is: 0.0 Seconds
[04:46:24] <icinga-wm>	 RECOVERY - puppet last run on mw2260 is OK: OK: Puppet is currently enabled, last run 35 seconds ago with 0 failures
[04:46:24] <icinga-wm>	 RECOVERY - puppet last run on mw2182 is OK: OK: Puppet is currently enabled, last run 28 seconds ago with 0 failures
[04:48:24] <icinga-wm>	 RECOVERY - puppet last run on cp3030 is OK: OK: Puppet is currently enabled, last run 47 seconds ago with 0 failures
[04:53:34] <icinga-wm>	 PROBLEM - puppet last run on mw2186 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[04:56:34] <icinga-wm>	 RECOVERY - puppet last run on mw2095 is OK: OK: Puppet is currently enabled, last run 12 seconds ago with 0 failures
[04:57:24] <icinga-wm>	 RECOVERY - puppet last run on mw2136 is OK: OK: Puppet is currently enabled, last run 58 seconds ago with 0 failures
[04:59:24] <icinga-wm>	 PROBLEM - Postgres Replication Lag on maps2004 is CRITICAL: CRITICAL - Rep Delay is: 7580.510725 Seconds
[05:00:24] <icinga-wm>	 RECOVERY - Postgres Replication Lag on maps2004 is OK: OK - Rep Delay is: 0.0 Seconds
[05:02:34] <icinga-wm>	 RECOVERY - puppet last run on mw2204 is OK: OK: Puppet is currently enabled, last run 47 seconds ago with 0 failures
[05:05:24] <icinga-wm>	 PROBLEM - puppet last run on elastic2015 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[05:12:24] <icinga-wm>	 RECOVERY - puppet last run on mw2220 is OK: OK: Puppet is currently enabled, last run 22 seconds ago with 0 failures
[05:14:24] <icinga-wm>	 PROBLEM - puppet last run on elastic2017 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[05:17:24] <icinga-wm>	 PROBLEM - puppet last run on mw2096 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[05:22:34] <icinga-wm>	 RECOVERY - puppet last run on mw2186 is OK: OK: Puppet is currently enabled, last run 57 seconds ago with 0 failures
[05:26:24] <icinga-wm>	 PROBLEM - puppet last run on ms-be2020 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[05:26:24] <icinga-wm>	 PROBLEM - puppet last run on maps2003 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[05:26:34] <icinga-wm>	 PROBLEM - puppet last run on mw2234 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[05:33:24] <icinga-wm>	 RECOVERY - puppet last run on elastic2015 is OK: OK: Puppet is currently enabled, last run 19 seconds ago with 0 failures
[05:33:34] <icinga-wm>	 PROBLEM - Postgres Replication Lag on maps2002 is CRITICAL: CRITICAL - Rep Delay is: 9628.306445 Seconds
[05:35:34] <icinga-wm>	 RECOVERY - Postgres Replication Lag on maps2002 is OK: OK - Rep Delay is: 0.0 Seconds
[05:36:24] <icinga-wm>	 PROBLEM - puppet last run on mw2128 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[05:43:24] <icinga-wm>	 RECOVERY - puppet last run on elastic2017 is OK: OK: Puppet is currently enabled, last run 27 seconds ago with 0 failures
[05:44:24] <icinga-wm>	 PROBLEM - Postgres Replication Lag on maps2003 is CRITICAL: CRITICAL - Rep Delay is: 10281.299158 Seconds
[05:45:24] <icinga-wm>	 RECOVERY - puppet last run on mw2096 is OK: OK: Puppet is currently enabled, last run 14 seconds ago with 0 failures
[05:45:24] <icinga-wm>	 RECOVERY - Postgres Replication Lag on maps2003 is OK: OK - Rep Delay is: 0.0 Seconds
[05:46:34] <icinga-wm>	 PROBLEM - puppet last run on rdb2005 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[05:48:24] <icinga-wm>	 PROBLEM - Postgres Replication Lag on maps2003 is CRITICAL: CRITICAL - Rep Delay is: 10521.518144 Seconds
[05:49:24] <icinga-wm>	 RECOVERY - Postgres Replication Lag on maps2003 is OK: OK - Rep Delay is: 0.0 Seconds
[05:54:24] <icinga-wm>	 RECOVERY - puppet last run on ms-be2020 is OK: OK: Puppet is currently enabled, last run 30 seconds ago with 0 failures
[05:54:24] <icinga-wm>	 RECOVERY - puppet last run on maps2003 is OK: OK: Puppet is currently enabled, last run 22 seconds ago with 0 failures
[06:04:24] <icinga-wm>	 RECOVERY - puppet last run on mw2128 is OK: OK: Puppet is currently enabled, last run 18 seconds ago with 0 failures
[06:05:24] <icinga-wm>	 PROBLEM - puppet last run on mw2216 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[06:14:34] <icinga-wm>	 PROBLEM - Postgres Replication Lag on maps2002 is CRITICAL: CRITICAL - Rep Delay is: 12088.503986 Seconds
[06:15:34] <icinga-wm>	 RECOVERY - puppet last run on rdb2005 is OK: OK: Puppet is currently enabled, last run 1 second ago with 0 failures
[06:15:34] <icinga-wm>	 RECOVERY - Postgres Replication Lag on maps2002 is OK: OK - Rep Delay is: 0.0 Seconds
[06:17:24] <icinga-wm>	 PROBLEM - puppet last run on db2054 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[06:17:24] <icinga-wm>	 PROBLEM - puppet last run on mw2184 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[06:22:24] <icinga-wm>	 PROBLEM - Postgres Replication Lag on maps2003 is CRITICAL: CRITICAL - Rep Delay is: 12561.438914 Seconds
[06:23:24] <icinga-wm>	 RECOVERY - Postgres Replication Lag on maps2003 is OK: OK - Rep Delay is: 0.0 Seconds
[06:24:34] <icinga-wm>	 RECOVERY - puppet last run on mw2234 is OK: OK: Puppet is currently enabled, last run 13 seconds ago with 0 failures
[06:26:24] <icinga-wm>	 PROBLEM - puppet last run on elastic2006 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[06:27:34] <icinga-wm>	 PROBLEM - puppet last run on ms-be2026 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[06:27:34] <icinga-wm>	 PROBLEM - restbase endpoints health on restbase2002 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds.
[06:27:34] <icinga-wm>	 PROBLEM - mobileapps endpoints health on scb2003 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds.
[06:27:34] <icinga-wm>	 PROBLEM - mobileapps endpoints health on scb2006 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds.
[06:27:34] <icinga-wm>	 PROBLEM - mobileapps endpoints health on scb2004 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds.
[06:28:24] <icinga-wm>	 PROBLEM - puppet last run on mw2254 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[06:28:34] <icinga-wm>	 PROBLEM - puppet last run on achernar is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds.
[06:29:24] <icinga-wm>	 PROBLEM - puppet last run on mc2035 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[06:29:34] <icinga-wm>	 RECOVERY - restbase endpoints health on restbase2002 is OK: All endpoints are healthy
[06:29:34] <icinga-wm>	 RECOVERY - mobileapps endpoints health on scb2006 is OK: All endpoints are healthy
[06:30:34] <icinga-wm>	 RECOVERY - puppet last run on achernar is OK: OK: Puppet is currently enabled, last run 30 minutes ago with 0 failures
[06:30:34] <icinga-wm>	 RECOVERY - mobileapps endpoints health on scb2004 is OK: All endpoints are healthy
[06:30:34] <icinga-wm>	 RECOVERY - mobileapps endpoints health on scb2003 is OK: All endpoints are healthy
[06:34:24] <icinga-wm>	 RECOVERY - puppet last run on mw2216 is OK: OK: Puppet is currently enabled, last run 52 seconds ago with 0 failures
[06:38:24] <icinga-wm>	 PROBLEM - puppet last run on db2036 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[06:45:24] <icinga-wm>	 RECOVERY - puppet last run on db2054 is OK: OK: Puppet is currently enabled, last run 25 seconds ago with 0 failures
[06:45:24] <icinga-wm>	 RECOVERY - puppet last run on mw2184 is OK: OK: Puppet is currently enabled, last run 29 seconds ago with 0 failures
[06:47:24] <icinga-wm>	 PROBLEM - puppet last run on restbase2002 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[06:53:24] <icinga-wm>	 PROBLEM - Postgres Replication Lag on maps2003 is CRITICAL: CRITICAL - Rep Delay is: 14421.279384 Seconds
[06:54:24] <icinga-wm>	 RECOVERY - puppet last run on elastic2006 is OK: OK: Puppet is currently enabled, last run 45 seconds ago with 0 failures
[06:54:24] <icinga-wm>	 RECOVERY - Postgres Replication Lag on maps2003 is OK: OK - Rep Delay is: 0.0 Seconds
[06:55:34] <icinga-wm>	 RECOVERY - puppet last run on ms-be2026 is OK: OK: Puppet is currently enabled, last run 9 seconds ago with 0 failures
[06:57:24] <icinga-wm>	 RECOVERY - puppet last run on mw2254 is OK: OK: Puppet is currently enabled, last run 56 seconds ago with 0 failures
[06:57:25] <icinga-wm>	 RECOVERY - puppet last run on mc2035 is OK: OK: Puppet is currently enabled, last run 23 seconds ago with 0 failures
[06:58:24] <icinga-wm>	 PROBLEM - puppet last run on mc2026 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[07:06:24] <icinga-wm>	 RECOVERY - puppet last run on db2036 is OK: OK: Puppet is currently enabled, last run 52 seconds ago with 0 failures
[07:08:14] <icinga-wm>	 PROBLEM - puppet last run on lvs2001 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[07:08:24] <icinga-wm>	 PROBLEM - Postgres Replication Lag on maps2003 is CRITICAL: CRITICAL - Rep Delay is: 15321.177238 Seconds
[07:10:24] <icinga-wm>	 RECOVERY - Postgres Replication Lag on maps2003 is OK: OK - Rep Delay is: 0.0 Seconds
[07:16:24] <icinga-wm>	 PROBLEM - puppet last run on mc2024 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[07:16:24] <icinga-wm>	 RECOVERY - puppet last run on restbase2002 is OK: OK: Puppet is currently enabled, last run 38 seconds ago with 0 failures
[07:25:24] <icinga-wm>	 PROBLEM - puppet last run on ganeti2003 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[07:26:24] <icinga-wm>	 RECOVERY - puppet last run on mc2026 is OK: OK: Puppet is currently enabled, last run 25 seconds ago with 0 failures
[07:28:34] <icinga-wm>	 PROBLEM - Postgres Replication Lag on maps2002 is CRITICAL: CRITICAL - Rep Delay is: 16528.294609 Seconds
[07:29:34] <icinga-wm>	 RECOVERY - Postgres Replication Lag on maps2002 is OK: OK - Rep Delay is: 0.0 Seconds
[07:35:24] <icinga-wm>	 PROBLEM - puppet last run on db2019 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[07:36:15] <icinga-wm>	 RECOVERY - puppet last run on lvs2001 is OK: OK: Puppet is currently enabled, last run 23 seconds ago with 0 failures
[07:39:34] <icinga-wm>	 PROBLEM - Postgres Replication Lag on maps2002 is CRITICAL: CRITICAL - Rep Delay is: 17188.511203 Seconds
[07:40:34] <icinga-wm>	 RECOVERY - Postgres Replication Lag on maps2002 is OK: OK - Rep Delay is: 0.0 Seconds
[07:44:24] <icinga-wm>	 PROBLEM - puppet last run on elastic2017 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[07:44:24] <icinga-wm>	 RECOVERY - puppet last run on mc2024 is OK: OK: Puppet is currently enabled, last run 13 seconds ago with 0 failures
[07:44:25] <icinga-wm>	 PROBLEM - puppet last run on ms-be2033 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[07:47:24] <icinga-wm>	 PROBLEM - puppet last run on mw2127 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[07:49:34] <icinga-wm>	 PROBLEM - Postgres Replication Lag on maps2002 is CRITICAL: CRITICAL - Rep Delay is: 17789.179144 Seconds
[07:50:34] <icinga-wm>	 RECOVERY - Postgres Replication Lag on maps2002 is OK: OK - Rep Delay is: 0.0 Seconds
[07:53:24] <icinga-wm>	 RECOVERY - puppet last run on ganeti2003 is OK: OK: Puppet is currently enabled, last run 32 seconds ago with 0 failures
[07:58:24] <icinga-wm>	 PROBLEM - puppet last run on mw2254 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[07:58:24] <icinga-wm>	 PROBLEM - puppet last run on cp4010 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[08:00:24] <icinga-wm>	 PROBLEM - Postgres Replication Lag on maps2003 is CRITICAL: CRITICAL - Rep Delay is: 18441.504615 Seconds
[08:03:24] <icinga-wm>	 RECOVERY - puppet last run on db2019 is OK: OK: Puppet is currently enabled, last run 22 seconds ago with 0 failures
[08:04:24] <icinga-wm>	 RECOVERY - Postgres Replication Lag on maps2003 is OK: OK - Rep Delay is: 0.0 Seconds
[08:07:15] <icinga-wm>	 PROBLEM - puppet last run on ms-be2018 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[08:07:24] <icinga-wm>	 PROBLEM - puppet last run on cp4008 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[08:13:24] <icinga-wm>	 RECOVERY - puppet last run on elastic2017 is OK: OK: Puppet is currently enabled, last run 43 seconds ago with 0 failures
[08:13:24] <icinga-wm>	 RECOVERY - puppet last run on ms-be2033 is OK: OK: Puppet is currently enabled, last run 54 seconds ago with 0 failures
[08:13:34] <icinga-wm>	 PROBLEM - Postgres Replication Lag on maps2002 is CRITICAL: CRITICAL - Rep Delay is: 19228.210221 Seconds
[08:14:34] <icinga-wm>	 RECOVERY - Postgres Replication Lag on maps2002 is OK: OK - Rep Delay is: 0.0 Seconds
[08:16:24] <icinga-wm>	 RECOVERY - puppet last run on mw2127 is OK: OK: Puppet is currently enabled, last run 49 seconds ago with 0 failures
[08:17:24] <icinga-wm>	 PROBLEM - puppet last run on mw2123 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[08:18:24] <icinga-wm>	 PROBLEM - puppet last run on mw2182 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[08:26:24] <icinga-wm>	 RECOVERY - puppet last run on mw2254 is OK: OK: Puppet is currently enabled, last run 7 seconds ago with 0 failures
[08:27:25] <icinga-wm>	 RECOVERY - puppet last run on cp4010 is OK: OK: Puppet is currently enabled, last run 56 seconds ago with 0 failures
[08:28:24] <icinga-wm>	 PROBLEM - puppet last run on db2056 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[08:35:15] <icinga-wm>	 RECOVERY - puppet last run on ms-be2018 is OK: OK: Puppet is currently enabled, last run 18 seconds ago with 0 failures
[08:35:25] <icinga-wm>	 RECOVERY - puppet last run on cp4008 is OK: OK: Puppet is currently enabled, last run 8 seconds ago with 0 failures
[08:37:24] <icinga-wm>	 PROBLEM - puppet last run on restbase-test2003 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[08:45:24] <icinga-wm>	 RECOVERY - puppet last run on mw2123 is OK: OK: Puppet is currently enabled, last run 35 seconds ago with 0 failures
[08:46:24] <icinga-wm>	 PROBLEM - puppet last run on db2059 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[08:46:24] <icinga-wm>	 RECOVERY - puppet last run on mw2182 is OK: OK: Puppet is currently enabled, last run 39 seconds ago with 0 failures
[08:47:24] <icinga-wm>	 PROBLEM - puppet last run on mw2096 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[08:54:34] <icinga-wm>	 PROBLEM - Postgres Replication Lag on maps2002 is CRITICAL: CRITICAL - Rep Delay is: 21688.178774 Seconds
[08:55:34] <icinga-wm>	 RECOVERY - Postgres Replication Lag on maps2002 is OK: OK - Rep Delay is: 0.0 Seconds
[08:57:24] <icinga-wm>	 PROBLEM - puppet last run on db1046 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[08:58:34] <icinga-wm>	 PROBLEM - Postgres Replication Lag on maps2002 is CRITICAL: CRITICAL - Rep Delay is: 21928.316058 Seconds
[08:59:34] <icinga-wm>	 RECOVERY - Postgres Replication Lag on maps2002 is OK: OK - Rep Delay is: 0.0 Seconds
[09:00:24] <icinga-wm>	 PROBLEM - Postgres Replication Lag on maps2004 is CRITICAL: CRITICAL - Rep Delay is: 22040.506656 Seconds
[09:02:24] <icinga-wm>	 RECOVERY - Postgres Replication Lag on maps2004 is OK: OK - Rep Delay is: 0.0 Seconds
[09:05:24] <icinga-wm>	 RECOVERY - puppet last run on restbase-test2003 is OK: OK: Puppet is currently enabled, last run 33 seconds ago with 0 failures
[09:07:24] <icinga-wm>	 PROBLEM - puppet last run on db2018 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[09:11:54] <icinga-wm>	 PROBLEM - puppet last run on wtp1018 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[09:13:34] <icinga-wm>	 PROBLEM - Postgres Replication Lag on maps2002 is CRITICAL: CRITICAL - Rep Delay is: 22828.262644 Seconds
[09:14:24] <icinga-wm>	 RECOVERY - puppet last run on db2059 is OK: OK: Puppet is currently enabled, last run 18 seconds ago with 0 failures
[09:14:34] <icinga-wm>	 RECOVERY - Postgres Replication Lag on maps2002 is OK: OK - Rep Delay is: 0.0 Seconds
[09:15:34] <icinga-wm>	 PROBLEM - puppet last run on mw2117 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[09:16:24] <icinga-wm>	 RECOVERY - puppet last run on mw2096 is OK: OK: Puppet is currently enabled, last run 45 seconds ago with 0 failures
[09:18:24] <icinga-wm>	 PROBLEM - Postgres Replication Lag on maps2004 is CRITICAL: CRITICAL - Rep Delay is: 23120.595481 Seconds
[09:19:24] <icinga-wm>	 RECOVERY - Postgres Replication Lag on maps2004 is OK: OK - Rep Delay is: 0.0 Seconds
[09:25:24] <icinga-wm>	 RECOVERY - puppet last run on db1046 is OK: OK: Puppet is currently enabled, last run 3 seconds ago with 0 failures
[09:25:24] <icinga-wm>	 PROBLEM - puppet last run on cp4009 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[09:26:24] <icinga-wm>	 RECOVERY - puppet last run on db2056 is OK: OK: Puppet is currently enabled, last run 43 seconds ago with 0 failures
[09:27:24] <icinga-wm>	 PROBLEM - Postgres Replication Lag on maps2003 is CRITICAL: CRITICAL - Rep Delay is: 23661.515867 Seconds
[09:28:24] <icinga-wm>	 RECOVERY - Postgres Replication Lag on maps2003 is OK: OK - Rep Delay is: 0.0 Seconds
[09:36:24] <icinga-wm>	 RECOVERY - puppet last run on db2018 is OK: OK: Puppet is currently enabled, last run 38 seconds ago with 0 failures
[09:38:24] <icinga-wm>	 PROBLEM - puppet last run on restbase2001 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[09:38:34] <icinga-wm>	 PROBLEM - puppet last run on mw2218 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[09:38:54] <icinga-wm>	 RECOVERY - puppet last run on wtp1018 is OK: OK: Puppet is currently enabled, last run 5 seconds ago with 0 failures
[09:41:24] <icinga-wm>	 PROBLEM - puppet last run on mc2019 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[09:43:26] <wikibugs__>	 (03PS2) 10Urbanecm: Add autopatrolled group to svwiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/344393 (https://phabricator.wikimedia.org/T161210)
[09:44:34] <icinga-wm>	 PROBLEM - Postgres Replication Lag on maps2002 is CRITICAL: CRITICAL - Rep Delay is: 24688.24095 Seconds
[09:44:34] <icinga-wm>	 RECOVERY - puppet last run on mw2117 is OK: OK: Puppet is currently enabled, last run 3 seconds ago with 0 failures
[09:46:34] <icinga-wm>	 RECOVERY - Postgres Replication Lag on maps2002 is OK: OK - Rep Delay is: 0.0 Seconds
[09:49:34] <icinga-wm>	 PROBLEM - Postgres Replication Lag on maps2002 is CRITICAL: CRITICAL - Rep Delay is: 24988.497076 Seconds
[09:50:34] <icinga-wm>	 RECOVERY - Postgres Replication Lag on maps2002 is OK: OK - Rep Delay is: 0.0 Seconds
[09:51:24] <icinga-wm>	 PROBLEM - puppet last run on db2041 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[09:51:24] <icinga-wm>	 PROBLEM - puppet last run on db2023 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[09:51:24] <icinga-wm>	 PROBLEM - Postgres Replication Lag on maps2004 is CRITICAL: CRITICAL - Rep Delay is: 25100.826948 Seconds
[09:54:24] <icinga-wm>	 RECOVERY - puppet last run on cp4009 is OK: OK: Puppet is currently enabled, last run 12 seconds ago with 0 failures
[09:54:54] <icinga-wm>	 PROBLEM - puppet last run on db1074 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[09:55:24] <icinga-wm>	 RECOVERY - Postgres Replication Lag on maps2004 is OK: OK - Rep Delay is: 0.0 Seconds
[09:59:24] <icinga-wm>	 PROBLEM - puppet last run on cp4018 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[10:00:24] <icinga-wm>	 PROBLEM - puppet last run on ms-be2015 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[10:00:24] <icinga-wm>	 PROBLEM - puppet last run on maps2004 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[10:00:24] <icinga-wm>	 PROBLEM - puppet last run on db2062 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[10:05:04] <icinga-wm>	 PROBLEM - puppet last run on xenon is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[10:05:24] <icinga-wm>	 PROBLEM - Postgres Replication Lag on maps2004 is CRITICAL: CRITICAL - Rep Delay is: 25940.407651 Seconds
[10:06:24] <icinga-wm>	 RECOVERY - Postgres Replication Lag on maps2004 is OK: OK - Rep Delay is: 0.0 Seconds
[10:06:25] <_joe_>	 !log restarting apache2 on puppetmaster2002, passenger probably stuck
[10:06:31] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[10:06:34] <icinga-wm>	 RECOVERY - puppet last run on mw2218 is OK: OK: Puppet is currently enabled, last run 4 seconds ago with 0 failures
[10:07:24] <icinga-wm>	 RECOVERY - puppet last run on restbase2001 is OK: OK: Puppet is currently enabled, last run 51 seconds ago with 0 failures
[10:09:24] <icinga-wm>	 PROBLEM - Postgres Replication Lag on maps2004 is CRITICAL: CRITICAL - Rep Delay is: 26180.530755 Seconds
[10:09:24] <icinga-wm>	 RECOVERY - puppet last run on mc2019 is OK: OK: Puppet is currently enabled, last run 10 seconds ago with 0 failures
[10:10:24] <icinga-wm>	 RECOVERY - Postgres Replication Lag on maps2004 is OK: OK - Rep Delay is: 0.0 Seconds
[10:16:24] <icinga-wm>	 PROBLEM - puppet last run on ms-be2032 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[10:19:24] <icinga-wm>	 RECOVERY - puppet last run on db2041 is OK: OK: Puppet is currently enabled, last run 28 seconds ago with 0 failures
[10:19:24] <icinga-wm>	 RECOVERY - puppet last run on db2023 is OK: OK: Puppet is currently enabled, last run 7 seconds ago with 0 failures
[10:19:24] <icinga-wm>	 PROBLEM - Postgres Replication Lag on maps2003 is CRITICAL: CRITICAL - Rep Delay is: 26781.623598 Seconds
[10:20:24] <icinga-wm>	 RECOVERY - Postgres Replication Lag on maps2003 is OK: OK - Rep Delay is: 0.0 Seconds
[10:21:54] <icinga-wm>	 RECOVERY - puppet last run on db1074 is OK: OK: Puppet is currently enabled, last run 1 second ago with 0 failures
[10:24:34] <icinga-wm>	 PROBLEM - Postgres Replication Lag on maps2002 is CRITICAL: CRITICAL - Rep Delay is: 27088.47625 Seconds
[10:25:24] <icinga-wm>	 PROBLEM - puppet last run on cp4009 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[10:25:34] <icinga-wm>	 RECOVERY - Postgres Replication Lag on maps2002 is OK: OK - Rep Delay is: 0.0 Seconds
[10:27:24] <icinga-wm>	 PROBLEM - puppet last run on wtp2008 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[10:27:24] <icinga-wm>	 PROBLEM - Postgres Replication Lag on maps2004 is CRITICAL: CRITICAL - Rep Delay is: 27260.874986 Seconds
[10:28:24] <icinga-wm>	 RECOVERY - puppet last run on db2062 is OK: OK: Puppet is currently enabled, last run 20 seconds ago with 0 failures
[10:28:24] <icinga-wm>	 RECOVERY - Postgres Replication Lag on maps2004 is OK: OK - Rep Delay is: 0.0 Seconds
[10:29:24] <icinga-wm>	 RECOVERY - puppet last run on ms-be2015 is OK: OK: Puppet is currently enabled, last run 52 seconds ago with 0 failures
[10:29:24] <icinga-wm>	 RECOVERY - puppet last run on maps2004 is OK: OK: Puppet is currently enabled, last run 46 seconds ago with 0 failures
[10:29:24] <icinga-wm>	 RECOVERY - puppet last run on cp4018 is OK: OK: Puppet is currently enabled, last run 54 seconds ago with 0 failures
[10:33:24] <icinga-wm>	 PROBLEM - Postgres Replication Lag on maps2004 is CRITICAL: CRITICAL - Rep Delay is: 27621.417154 Seconds
[10:34:04] <icinga-wm>	 RECOVERY - puppet last run on xenon is OK: OK: Puppet is currently enabled, last run 34 seconds ago with 0 failures
[10:35:24] <icinga-wm>	 RECOVERY - Postgres Replication Lag on maps2004 is OK: OK - Rep Delay is: 0.0 Seconds
[10:39:34] <icinga-wm>	 PROBLEM - Postgres Replication Lag on maps2002 is CRITICAL: CRITICAL - Rep Delay is: 27988.567078 Seconds
[10:40:34] <icinga-wm>	 RECOVERY - Postgres Replication Lag on maps2002 is OK: OK - Rep Delay is: 0.0 Seconds
[10:42:34] <icinga-wm>	 PROBLEM - puppet last run on restbase2007 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[10:44:24] <icinga-wm>	 RECOVERY - puppet last run on ms-be2032 is OK: OK: Puppet is currently enabled, last run 29 seconds ago with 0 failures
[10:52:24] <icinga-wm>	 PROBLEM - puppet last run on elastic2028 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[10:53:24] <icinga-wm>	 PROBLEM - puppet last run on ms-be2030 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[10:53:34] <icinga-wm>	 PROBLEM - puppet last run on db2011 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[10:55:24] <icinga-wm>	 RECOVERY - puppet last run on wtp2008 is OK: OK: Puppet is currently enabled, last run 50 seconds ago with 0 failures
[10:55:24] <icinga-wm>	 PROBLEM - puppet last run on mw2248 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[10:55:24] <icinga-wm>	 RECOVERY - puppet last run on cp4009 is OK: OK: Puppet is currently enabled, last run 50 seconds ago with 0 failures
[10:55:34] <icinga-wm>	 PROBLEM - Postgres Replication Lag on maps2002 is CRITICAL: CRITICAL - Rep Delay is: 28948.145463 Seconds
[10:56:34] <icinga-wm>	 RECOVERY - Postgres Replication Lag on maps2002 is OK: OK - Rep Delay is: 0.0 Seconds
[11:08:24] <icinga-wm>	 PROBLEM - puppet last run on mw2241 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[11:11:34] <icinga-wm>	 RECOVERY - puppet last run on restbase2007 is OK: OK: Puppet is currently enabled, last run 56 seconds ago with 0 failures
[11:17:24] <icinga-wm>	 PROBLEM - puppet last run on restbase2002 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[11:21:24] <icinga-wm>	 RECOVERY - puppet last run on elastic2028 is OK: OK: Puppet is currently enabled, last run 0 seconds ago with 0 failures
[11:22:24] <icinga-wm>	 RECOVERY - puppet last run on ms-be2030 is OK: OK: Puppet is currently enabled, last run 38 seconds ago with 0 failures
[11:22:34] <icinga-wm>	 RECOVERY - puppet last run on db2011 is OK: OK: Puppet is currently enabled, last run 37 seconds ago with 0 failures
[11:23:24] <icinga-wm>	 RECOVERY - puppet last run on mw2248 is OK: OK: Puppet is currently enabled, last run 30 seconds ago with 0 failures
[11:27:14] <icinga-wm>	 PROBLEM - puppet last run on subra is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[11:27:24] <icinga-wm>	 PROBLEM - puppet last run on elastic2007 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[11:28:24] <icinga-wm>	 PROBLEM - puppet last run on mw2254 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[11:28:24] <icinga-wm>	 PROBLEM - puppet last run on mw2146 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[11:37:24] <icinga-wm>	 RECOVERY - puppet last run on mw2241 is OK: OK: Puppet is currently enabled, last run 30 seconds ago with 0 failures
[11:39:24] <icinga-wm>	 PROBLEM - puppet last run on lvs2005 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[11:46:24] <icinga-wm>	 RECOVERY - puppet last run on restbase2002 is OK: OK: Puppet is currently enabled, last run 55 seconds ago with 0 failures
[11:47:14] <icinga-wm>	 PROBLEM - puppet last run on wtp1001 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[11:48:24] <icinga-wm>	 PROBLEM - puppet last run on cp4014 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[11:54:24] <icinga-wm>	 PROBLEM - Postgres Replication Lag on maps2003 is CRITICAL: CRITICAL - Rep Delay is: 32481.492336 Seconds
[11:54:24] <icinga-wm>	 PROBLEM - Postgres Replication Lag on maps2004 is CRITICAL: CRITICAL - Rep Delay is: 32481.505374 Seconds
[11:55:15] <icinga-wm>	 RECOVERY - puppet last run on subra is OK: OK: Puppet is currently enabled, last run 15 seconds ago with 0 failures
[11:55:24] <icinga-wm>	 RECOVERY - Postgres Replication Lag on maps2003 is OK: OK - Rep Delay is: 0.0 Seconds
[11:56:24] <icinga-wm>	 RECOVERY - puppet last run on elastic2007 is OK: OK: Puppet is currently enabled, last run 34 seconds ago with 0 failures
[11:56:24] <icinga-wm>	 RECOVERY - puppet last run on mw2254 is OK: OK: Puppet is currently enabled, last run 28 seconds ago with 0 failures
[11:56:24] <icinga-wm>	 PROBLEM - puppet last run on sarin is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[11:56:24] <icinga-wm>	 RECOVERY - Postgres Replication Lag on maps2004 is OK: OK - Rep Delay is: 0.0 Seconds
[11:57:16] <godog>	 w/go lindseyanne 
[11:57:20] <godog>	 nope!
[11:57:25] <icinga-wm>	 RECOVERY - puppet last run on mw2146 is OK: OK: Puppet is currently enabled, last run 56 seconds ago with 0 failures
[11:57:34] <icinga-wm>	 PROBLEM - puppet last run on ms-be2026 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[12:07:24] <icinga-wm>	 RECOVERY - puppet last run on lvs2005 is OK: OK: Puppet is currently enabled, last run 17 seconds ago with 0 failures
[12:09:24] <icinga-wm>	 PROBLEM - Postgres Replication Lag on maps2004 is CRITICAL: CRITICAL - Rep Delay is: 33381.233532 Seconds
[12:10:24] <icinga-wm>	 RECOVERY - Postgres Replication Lag on maps2004 is OK: OK - Rep Delay is: 0.0 Seconds
[12:14:24] <icinga-wm>	 PROBLEM - Postgres Replication Lag on maps2003 is CRITICAL: CRITICAL - Rep Delay is: 33681.333263 Seconds
[12:14:24] <icinga-wm>	 PROBLEM - Postgres Replication Lag on maps2004 is CRITICAL: CRITICAL - Rep Delay is: 33681.356509 Seconds
[12:15:24] <icinga-wm>	 RECOVERY - Postgres Replication Lag on maps2003 is OK: OK - Rep Delay is: 0.0 Seconds
[12:15:24] <icinga-wm>	 RECOVERY - Postgres Replication Lag on maps2004 is OK: OK - Rep Delay is: 0.0 Seconds
[12:16:15] <icinga-wm>	 RECOVERY - puppet last run on wtp1001 is OK: OK: Puppet is currently enabled, last run 43 seconds ago with 0 failures
[12:16:24] <icinga-wm>	 PROBLEM - puppet last run on mw2163 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[12:16:24] <icinga-wm>	 RECOVERY - puppet last run on cp4014 is OK: OK: Puppet is currently enabled, last run 32 seconds ago with 0 failures
[12:23:24] <icinga-wm>	 PROBLEM - Postgres Replication Lag on maps2003 is CRITICAL: CRITICAL - Rep Delay is: 34221.614151 Seconds
[12:24:24] <icinga-wm>	 RECOVERY - puppet last run on sarin is OK: OK: Puppet is currently enabled, last run 2 seconds ago with 0 failures
[12:24:24] <icinga-wm>	 RECOVERY - Postgres Replication Lag on maps2003 is OK: OK - Rep Delay is: 0.0 Seconds
[12:24:34] <icinga-wm>	 PROBLEM - puppet last run on heze is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[12:25:34] <icinga-wm>	 RECOVERY - puppet last run on ms-be2026 is OK: OK: Puppet is currently enabled, last run 19 seconds ago with 0 failures
[12:26:24] <icinga-wm>	 PROBLEM - puppet last run on es2002 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[12:28:34] <icinga-wm>	 PROBLEM - Postgres Replication Lag on maps2002 is CRITICAL: CRITICAL - Rep Delay is: 34528.321103 Seconds
[12:29:34] <icinga-wm>	 RECOVERY - Postgres Replication Lag on maps2002 is OK: OK - Rep Delay is: 0.0 Seconds
[12:34:34] <icinga-wm>	 PROBLEM - puppet last run on mw2108 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[12:39:34] <icinga-wm>	 PROBLEM - puppet last run on mw2192 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[12:42:25] <icinga-wm>	 PROBLEM - puppet last run on cp3034 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[12:45:24] <icinga-wm>	 RECOVERY - puppet last run on mw2163 is OK: OK: Puppet is currently enabled, last run 43 seconds ago with 0 failures
[12:48:24] <icinga-wm>	 PROBLEM - puppet last run on cp4014 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[12:52:34] <icinga-wm>	 PROBLEM - Postgres Replication Lag on maps2002 is CRITICAL: CRITICAL - Rep Delay is: 35968.132528 Seconds
[12:53:24] <icinga-wm>	 RECOVERY - puppet last run on es2002 is OK: OK: Puppet is currently enabled, last run 13 seconds ago with 0 failures
[12:53:34] <icinga-wm>	 RECOVERY - Postgres Replication Lag on maps2002 is OK: OK - Rep Delay is: 0.0 Seconds
[12:53:34] <icinga-wm>	 RECOVERY - puppet last run on heze is OK: OK: Puppet is currently enabled, last run 51 seconds ago with 0 failures
[12:58:24] <icinga-wm>	 PROBLEM - puppet last run on mc2026 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[12:58:25] <icinga-wm>	 PROBLEM - puppet last run on restbase2006 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[12:59:24] <icinga-wm>	 PROBLEM - puppet last run on wtp2004 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[12:59:34] <icinga-wm>	 PROBLEM - puppet last run on restbase2008 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[13:02:24] <icinga-wm>	 PROBLEM - Postgres Replication Lag on maps2004 is CRITICAL: CRITICAL - Rep Delay is: 36561.239375 Seconds
[13:02:34] <icinga-wm>	 RECOVERY - puppet last run on mw2108 is OK: OK: Puppet is currently enabled, last run 48 seconds ago with 0 failures
[13:03:24] <icinga-wm>	 RECOVERY - Postgres Replication Lag on maps2004 is OK: OK - Rep Delay is: 0.0 Seconds
[13:07:34] <icinga-wm>	 RECOVERY - puppet last run on mw2192 is OK: OK: Puppet is currently enabled, last run 8 seconds ago with 0 failures
[13:08:54] <icinga-wm>	 PROBLEM - Postgres Replication Lag on maps1004 is CRITICAL: CRITICAL - Rep Delay is: 39363.651989 Seconds
[13:08:54] <icinga-wm>	 PROBLEM - Postgres Replication Lag on maps1003 is CRITICAL: CRITICAL - Rep Delay is: 39363.667557 Seconds
[13:09:14] <icinga-wm>	 PROBLEM - Postgres Replication Lag on maps1002 is CRITICAL: CRITICAL - Rep Delay is: 39387.259933 Seconds
[13:10:54] <icinga-wm>	 RECOVERY - Postgres Replication Lag on maps1004 is OK: OK - Rep Delay is: 0.0 Seconds
[13:10:55] <icinga-wm>	 RECOVERY - Postgres Replication Lag on maps1003 is OK: OK - Rep Delay is: 0.0 Seconds
[13:11:14] <icinga-wm>	 RECOVERY - Postgres Replication Lag on maps1002 is OK: OK - Rep Delay is: 0.0 Seconds
[13:11:24] <icinga-wm>	 RECOVERY - puppet last run on cp3034 is OK: OK: Puppet is currently enabled, last run 7 seconds ago with 0 failures
[13:14:24] <icinga-wm>	 PROBLEM - puppet last run on mw2220 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[13:17:24] <icinga-wm>	 RECOVERY - puppet last run on cp4014 is OK: OK: Puppet is currently enabled, last run 39 seconds ago with 0 failures
[13:25:14] <icinga-wm>	 PROBLEM - puppet last run on mc1030 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[13:25:24] <icinga-wm>	 PROBLEM - puppet last run on ms-fe2007 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[13:26:24] <icinga-wm>	 RECOVERY - puppet last run on mc2026 is OK: OK: Puppet is currently enabled, last run 52 seconds ago with 0 failures
[13:26:24] <icinga-wm>	 RECOVERY - puppet last run on restbase2006 is OK: OK: Puppet is currently enabled, last run 12 seconds ago with 0 failures
[13:26:54] <icinga-wm>	 PROBLEM - puppet last run on db1015 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[13:27:34] <icinga-wm>	 RECOVERY - puppet last run on restbase2008 is OK: OK: Puppet is currently enabled, last run 26 seconds ago with 0 failures
[13:28:24] <icinga-wm>	 RECOVERY - puppet last run on wtp2004 is OK: OK: Puppet is currently enabled, last run 58 seconds ago with 0 failures
[13:37:24] <icinga-wm>	 PROBLEM - puppet last run on db2018 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[13:41:24] <icinga-wm>	 PROBLEM - puppet last run on elastic2027 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[13:42:24] <icinga-wm>	 RECOVERY - puppet last run on mw2220 is OK: OK: Puppet is currently enabled, last run 11 seconds ago with 0 failures
[13:49:24] <icinga-wm>	 PROBLEM - puppet last run on ms-be2009 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[13:53:14] <icinga-wm>	 RECOVERY - puppet last run on mc1030 is OK: OK: Puppet is currently enabled, last run 9 seconds ago with 0 failures
[13:54:24] <icinga-wm>	 RECOVERY - puppet last run on ms-fe2007 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[13:55:54] <icinga-wm>	 RECOVERY - puppet last run on db1015 is OK: OK: Puppet is currently enabled, last run 24 seconds ago with 0 failures
[13:58:24] <icinga-wm>	 PROBLEM - puppet last run on es2013 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[14:06:24] <icinga-wm>	 RECOVERY - puppet last run on db2018 is OK: OK: Puppet is currently enabled, last run 47 seconds ago with 0 failures
[14:09:24] <icinga-wm>	 RECOVERY - puppet last run on elastic2027 is OK: OK: Puppet is currently enabled, last run 29 seconds ago with 0 failures
[14:12:24] <icinga-wm>	 PROBLEM - puppet last run on maps-test2002 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[14:18:24] <icinga-wm>	 RECOVERY - puppet last run on ms-be2009 is OK: OK: Puppet is currently enabled, last run 41 seconds ago with 0 failures
[14:23:24] <icinga-wm>	 PROBLEM - puppet last run on ms-be2030 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[14:27:24] <icinga-wm>	 RECOVERY - puppet last run on es2013 is OK: OK: Puppet is currently enabled, last run 48 seconds ago with 0 failures
[14:32:24] <icinga-wm>	 PROBLEM - puppet last run on db2066 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[14:38:54] <icinga-wm>	 PROBLEM - puppet last run on es1015 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[14:40:24] <icinga-wm>	 RECOVERY - puppet last run on maps-test2002 is OK: OK: Puppet is currently enabled, last run 12 seconds ago with 0 failures
[14:41:24] <icinga-wm>	 PROBLEM - puppet last run on mw2194 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[14:50:24] <icinga-wm>	 PROBLEM - puppet last run on mw2200 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[14:52:24] <icinga-wm>	 RECOVERY - puppet last run on ms-be2030 is OK: OK: Puppet is currently enabled, last run 28 seconds ago with 0 failures
[15:00:24] <icinga-wm>	 RECOVERY - puppet last run on db2066 is OK: OK: Puppet is currently enabled, last run 30 seconds ago with 0 failures
[15:00:24] <icinga-wm>	 PROBLEM - puppet last run on cp4018 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[15:06:54] <icinga-wm>	 RECOVERY - puppet last run on es1015 is OK: OK: Puppet is currently enabled, last run 3 seconds ago with 0 failures
[15:10:24] <icinga-wm>	 RECOVERY - puppet last run on mw2194 is OK: OK: Puppet is currently enabled, last run 51 seconds ago with 0 failures
[15:15:24] <icinga-wm>	 PROBLEM - puppet last run on ms-be2006 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[15:17:24] <icinga-wm>	 PROBLEM - puppet last run on db2054 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[15:18:24] <icinga-wm>	 RECOVERY - puppet last run on mw2200 is OK: OK: Puppet is currently enabled, last run 8 seconds ago with 0 failures
[15:21:24] <icinga-wm>	 PROBLEM - puppet last run on mc1025 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[15:25:24] <icinga-wm>	 PROBLEM - puppet last run on mw2145 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[15:27:24] <icinga-wm>	 PROBLEM - puppet last run on mw2158 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[15:28:24] <icinga-wm>	 RECOVERY - puppet last run on cp4018 is OK: OK: Puppet is currently enabled, last run 23 seconds ago with 0 failures
[15:29:24] <icinga-wm>	 PROBLEM - puppet last run on db2068 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[15:40:04] <icinga-wm>	 PROBLEM - SSH on ms-be2005 is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[15:40:54] <icinga-wm>	 RECOVERY - SSH on ms-be2005 is OK: SSH OK - OpenSSH_6.6.1p1 Ubuntu-2ubuntu2.8 (protocol 2.0)
[15:43:24] <icinga-wm>	 RECOVERY - puppet last run on ms-be2006 is OK: OK: Puppet is currently enabled, last run 12 seconds ago with 0 failures
[15:45:24] <icinga-wm>	 RECOVERY - puppet last run on db2054 is OK: OK: Puppet is currently enabled, last run 43 seconds ago with 0 failures
[15:50:18] <eddiegp>	 !instance
[15:50:18] <wm-bot>	 https://labsconsole.wikimedia.org/wiki/Instances
[15:50:22] <eddiegp>	 !instance del
[15:50:22] <wm-bot>	 Successfully removed instance
[15:50:24] <icinga-wm>	 RECOVERY - puppet last run on mc1025 is OK: OK: Puppet is currently enabled, last run 28 seconds ago with 0 failures
[15:50:29] <eddiegp>	 !instance is https://wikitech.wikimedia.org/wiki/Help:Instances
[15:50:30] <wm-bot>	 Key was added
[15:50:39] <eddiegp>	 !git del
[15:50:39] <wm-bot>	 Successfully removed git
[15:50:47] <eddiegp>	 !git is https://wikitech.wikimedia.org/wiki/Help:Git
[15:50:47] <wm-bot>	 Key was added
[15:51:42] <eddiegp>	 !amend del
[15:51:43] <wm-bot>	 Successfully removed amend
[15:51:48] <eddiegp>	 !amend is https://www.mediawiki.org/wiki/Gerrit/Tutorial#Amending_a_change_.28your_own_or_someone_else.27s.29
[15:51:48] <wm-bot>	 Key was added
[15:54:24] <icinga-wm>	 RECOVERY - puppet last run on mw2145 is OK: OK: Puppet is currently enabled, last run 2 seconds ago with 0 failures
[15:54:42] <eddiegp>	 !security
[15:54:42] <wm-bot>	 https://labsconsole.wikimedia.org/wiki/Security_Groups
[15:54:47] <eddiegp>	 !security del
[15:54:47] <wm-bot>	 Successfully removed security
[15:54:53] <eddiegp>	 !security is https://wikitech.wikimedia.org/wiki/Help:Security_groups
[15:54:54] <wm-bot>	 Key was added
[15:55:24] <icinga-wm>	 RECOVERY - puppet last run on mw2158 is OK: OK: Puppet is currently enabled, last run 28 seconds ago with 0 failures
[15:57:32] <eddiegp>	 !labs
[15:57:32] <wm-bot>	 https://labsconsole.wikimedia.org/wiki/
[15:57:36] <eddiegp>	 !labs del
[15:57:36] <wm-bot>	 Successfully removed labs
[15:57:51] <eddiegp>	 !labs is https://wikitech.wikimedia.org/wiki/$1
[15:57:51] <wm-bot>	 Key was added
[15:58:24] <icinga-wm>	 RECOVERY - puppet last run on db2068 is OK: OK: Puppet is currently enabled, last run 17 seconds ago with 0 failures
[15:58:28] <eddiegp>	 !projects
[15:58:29] <wm-bot>	 https://labsconsole.wikimedia.org/wiki/Special:Ask/-5B-5BResource-20Type::project-5D-5D/-3FMember
[15:58:32] <eddiegp>	 !projects del
[15:58:33] <wm-bot>	 Successfully removed projects
[15:59:09] <eddiegp>	 !projects is https://wikitech.wikimedia.org/wiki/Special:Ask/-5B-5BResource-20Type::project-5D-5D/-3FMember
[15:59:09] <wm-bot>	 Key was added
[15:59:39] <eddiegp>	 !instancelist del
[15:59:40] <wm-bot>	 Successfully removed instancelist
[15:59:49] <eddiegp>	 !instancelist is https://wikitech.wikimedia.org/w/index.php?title=Special:Ask&offset=0&limit=100&q=[[Resource+Type%3A%3Ainstance]]&p=format%3Dbroadtable&po=%3FInstance+Name%0A%3FInstance+Type%0A%3FProject%0A%3FImage+Id%0A%3FFQDN%0A%3FLaunch+Time%0A%3FPuppet+Class%0A%3FModification+date%0A%3FInstance+Host%0A%3FNumber+of+CPUs%0A%3FRAM+Size%0A%3FAmount+of+Storage%0A
[15:59:49] <wm-bot>	 Key was added
[16:00:48] <eddiegp>	 !instance-json del
[16:00:49] <wm-bot>	 Successfully removed instance-json
[16:00:55] <eddiegp>	 !instance-del is https://wikitech.wikimedia.org/wiki/Special:Ask/-5B-5BResource-20Type::instance-5D-5D/-3FInstance-20Name/-3FInstance-20Type/-3FProject/-3FImage-20Id/-3FFQDN/-3FLaunch-20Time/-3FPuppet-20Class/-3FModification-20date/-3FInstance-20Host/-3FNumber-20of-20CPUs/-3FRAM-20Size/-3FAmount-20of-20Storage/limit%3D500/format%3Djson
[16:00:56] <wm-bot>	 Key was added
[16:02:54] <icinga-wm>	 PROBLEM - puppet last run on notebook1002 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[16:03:19] <eddiegp>	 !pathconflict del
[16:03:20] <wm-bot>	 Successfully removed pathconflict
[16:03:28] <eddiegp>	 !pathconflict is https://wikitech.wikimedia.org/wiki/Git#Fixing_a_path_conflict
[16:03:28] <wm-bot>	 Key was added
[16:03:40] <eddiegp>	 !group del
[16:03:40] <wm-bot>	 Successfully removed group
[16:04:09] <eddiegp>	 !terminology del
[16:04:09] <wm-bot>	 Successfully removed terminology
[16:04:17] <eddiegp>	 !terminology is https://wikitech.wikimedia.org/wiki/Help:Terminology
[16:04:17] <wm-bot>	 Key was added
[16:05:04] <eddiegp>	 !access
[16:05:04] <wm-bot>	 https://labsconsole.wikimedia.org/wiki/Access#Accessing_public_and_private_instances
[16:05:07] <eddiegp>	 !access del
[16:05:07] <wm-bot>	 Successfully removed access
[16:05:13] <eddiegp>	 !access is https://wikitech.wikimedia.org/wiki/Access#Accessing_public_and_private_instances
[16:05:14] <wm-bot>	 Key was added
[16:07:04] <eddiegp>	 !stucked del
[16:07:04] <wm-bot>	 Successfully removed stucked
[16:07:58] <eddiegp>	 !socks-proxy del
[16:07:58] <wm-bot>	 Successfully removed socks-proxy
[16:08:23] <eddiegp>	 !socks-proxy is see !access ssh <user>@bastion.wmflabs.org -D <localport>
[16:08:24] <wm-bot>	 Key was added
[16:08:37] <eddiegp>	 !sudo
[16:08:37] <wm-bot>	 You have sudo in any project that you are a member of, excluding global projects (like bastion). Your sudo password is your labsconsole wiki password.
[16:08:47] <eddiegp>	 !sudo del
[16:08:48] <wm-bot>	 Successfully removed sudo
[16:08:58] <eddiegp>	 !sudo is You have sudo in any project that you are a member of, excluding global projects (like bastion). Your sudo password is your wikitech wiki password.
[16:08:58] <wm-bot>	 Key was added
[16:30:54] <icinga-wm>	 RECOVERY - puppet last run on notebook1002 is OK: OK: Puppet is currently enabled, last run 45 seconds ago with 0 failures
[16:45:24] <icinga-wm>	 PROBLEM - puppet last run on cp3047 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[17:11:24] <icinga-wm>	 PROBLEM - Check health of redis instance on 6479 on rdb2005 is CRITICAL: CRITICAL: replication_delay is 621 600 - REDIS 2.8.17 on 127.0.0.1:6479 has 1 databases (db0) with 3622435 keys, up 3 days 56 minutes - replication_delay is 621
[17:14:24] <icinga-wm>	 RECOVERY - puppet last run on cp3047 is OK: OK: Puppet is currently enabled, last run 1 second ago with 0 failures
[17:15:24] <icinga-wm>	 RECOVERY - Check health of redis instance on 6479 on rdb2005 is OK: OK: REDIS 2.8.17 on 127.0.0.1:6479 has 1 databases (db0) with 3589847 keys, up 3 days 1 hours - replication_delay is 0
[17:18:54] <icinga-wm>	 PROBLEM - Disk space on prometheus1001 is CRITICAL: DISK CRITICAL - free space: /srv 6863 MB (3% inode=86%)
[17:43:04] <icinga-wm>	 PROBLEM - puppet last run on dbstore1001 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:12:04] <icinga-wm>	 RECOVERY - puppet last run on dbstore1001 is OK: OK: Puppet is currently enabled, last run 40 seconds ago with 0 failures
[18:17:14] <icinga-wm>	 PROBLEM - puppet last run on ms-be1015 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:32:54] <icinga-wm>	 PROBLEM - puppet last run on meitnerium is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[18:45:14] <icinga-wm>	 RECOVERY - puppet last run on ms-be1015 is OK: OK: Puppet is currently enabled, last run 17 seconds ago with 0 failures
[18:52:16] <wikibugs>	 06Operations, 10Datasets-General-or-Unknown, 10Dumps-Generation, 06Labs, 10hardware-requests: Eqiad: Hardware request for labstore1006/7, dataset1002/3 - https://phabricator.wikimedia.org/T161311#3132234 (10bd808) p:05Triage>03Normal
[18:54:30] <wikibugs__>	 06Operations, 10DNS, 06Discovery, 06Labs, and 3 others: multi-component wmflabs.org subdomains doesn't work under simple wildcard TLS cert - https://phabricator.wikimedia.org/T161256#3132236 (10bd808) p:05Triage>03Normal
[19:00:54] <icinga-wm>	 RECOVERY - puppet last run on meitnerium is OK: OK: Puppet is currently enabled, last run 55 seconds ago with 0 failures
[20:59:43] <wikibugs>	 06Operations, 10Gerrit, 06Release-Engineering-Team, 07LDAP: Remove user gerrit2 from ldap - https://phabricator.wikimedia.org/T160122#3132441 (10bd808)
[22:28:14] <icinga-wm>	 PROBLEM - puppet last run on neodymium is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[22:56:14] <icinga-wm>	 RECOVERY - puppet last run on neodymium is OK: OK: Puppet is currently enabled, last run 1 second ago with 0 failures
[23:18:40] <wikibugs>	 07Puppet, 06Labs, 10MediaWiki-extensions-OpenStackManager, 07Documentation: doc.wikimedia.org puppet documentation for labs/lvm/srv.html gives a 404 - https://phabricator.wikimedia.org/T119329#3132598 (10bd808) 05Open>03declined https://doc.wikimedia.org/puppet/puppet_classes/role_3A_3Alabs_3A_3Alvm_3A...