[00:06:23] (03CR) 10Krinkle: varnish: Avoid std.fileread() and use new errorpage template (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/350966 (https://phabricator.wikimedia.org/T113114) (owner: 10Krinkle) [00:26:50] (03PS1) 10Herron: Lists: Change exim filter for spam observed from qq.com [puppet] - 10https://gerrit.wikimedia.org/r/365424 (https://phabricator.wikimedia.org/T170601) [00:28:28] (03CR) 10Herron: [C: 032] Lists: Change exim filter for spam observed from qq.com [puppet] - 10https://gerrit.wikimedia.org/r/365424 (https://phabricator.wikimedia.org/T170601) (owner: 10Herron) [00:35:39] (03CR) 10Herron: "Looks like it finished after all :)" [puppet] - 10https://gerrit.wikimedia.org/r/364827 (https://phabricator.wikimedia.org/T170462) (owner: 10Herron) [02:22:24] (03PS1) 10Dzahn: cache::misc: add director for netmon1003 [puppet] - 10https://gerrit.wikimedia.org/r/365428 (https://phabricator.wikimedia.org/T170653) [02:22:43] (03CR) 10jerkins-bot: [V: 04-1] cache::misc: add director for netmon1003 [puppet] - 10https://gerrit.wikimedia.org/r/365428 (https://phabricator.wikimedia.org/T170653) (owner: 10Dzahn) [02:23:25] (03PS2) 10Dzahn: cache::misc: add director for netmon1003 [puppet] - 10https://gerrit.wikimedia.org/r/365428 (https://phabricator.wikimedia.org/T170653) [02:26:36] (03PS1) 10Dzahn: cache::misc: switch servermon from netmon1001 to netmon1003 [puppet] - 10https://gerrit.wikimedia.org/r/365429 (https://phabricator.wikimedia.org/T170653) [02:26:58] (03CR) 10jerkins-bot: [V: 04-1] cache::misc: switch servermon from netmon1001 to netmon1003 [puppet] - 10https://gerrit.wikimedia.org/r/365429 (https://phabricator.wikimedia.org/T170653) (owner: 10Dzahn) [02:31:42] (03CR) 10Dzahn: [C: 032] cache::misc: add director for netmon1003 [puppet] - 10https://gerrit.wikimedia.org/r/365428 (https://phabricator.wikimedia.org/T170653) (owner: 10Dzahn) [02:39:42] (03PS2) 10Dzahn: cache::misc: switch servermon from netmon1001 to netmon1003 [puppet] - 10https://gerrit.wikimedia.org/r/365429 (https://phabricator.wikimedia.org/T170653) [02:44:02] (03CR) 10Dzahn: [C: 032] cache::misc: switch servermon from netmon1001 to netmon1003 [puppet] - 10https://gerrit.wikimedia.org/r/365429 (https://phabricator.wikimedia.org/T170653) (owner: 10Dzahn) [02:46:12] !log netmon1001 - stopping "make_updates" cron , migration to netmon1003, flipping cache::backend to netmon1003 (T170653) [02:46:22] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [02:46:23] T170653: create netmon1003, migrate servermon from netmon1001 to netmon1003 - https://phabricator.wikimedia.org/T170653 [02:53:24] !log servermon - switched to netmon1003 backend (jessie ganeti) [02:53:33] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [02:58:06] (03PS1) 10Dzahn: servermon: remove role from netmon1001 [puppet] - 10https://gerrit.wikimedia.org/r/365432 (https://phabricator.wikimedia.org/T170653) [02:59:08] (03PS2) 10Dzahn: servermon: remove role from netmon1001 [puppet] - 10https://gerrit.wikimedia.org/r/365432 (https://phabricator.wikimedia.org/T170653) [03:00:45] (03CR) 10Dzahn: [C: 032] servermon: remove role from netmon1001 [puppet] - 10https://gerrit.wikimedia.org/r/365432 (https://phabricator.wikimedia.org/T170653) (owner: 10Dzahn) [03:04:20] 10Operations, 10monitoring, 10Patch-For-Review: create netmon1003, migrate servermon from netmon1001 to netmon1003 - https://phabricator.wikimedia.org/T170653#3440997 (10Dzahn) 05Open>03Resolved done. Apache site has been removed on netmon1001 and here it is working as before from new backend: https:/... [03:06:01] 10Operations, 10monitoring, 10Patch-For-Review: setup netmon1002.wikimedia.org - https://phabricator.wikimedia.org/T159756#3441000 (10Dzahn) servermon is now on netmon1003 (T170653) (ganeti jessie) VM instead, so it will not move to netmon1002 (stretch) yet. therefore checking that box here. [03:06:28] 10Operations, 10monitoring, 10Patch-For-Review: setup netmon1002.wikimedia.org - https://phabricator.wikimedia.org/T159756#3441002 (10Dzahn) [03:27:25] PROBLEM - MariaDB Slave Lag: s1 on dbstore1002 is CRITICAL: CRITICAL slave_sql_lag Replication lag: 675.72 seconds [03:49:36] RECOVERY - MariaDB Slave Lag: s1 on dbstore1002 is OK: OK slave_sql_lag Replication lag: 233.67 seconds [03:56:55] PROBLEM - Make sure enwiki dumps are not empty on checker.tools.wmflabs.org is CRITICAL: HTTP CRITICAL: HTTP/1.1 503 SERVICE UNAVAILABLE - string OK not found on http://checker.tools.wmflabs.org:80/dumps - 288 bytes in 0.011 second response time [03:57:22] (03PS1) 10Dzahn: servermon: restrict http access to prod networks [puppet] - 10https://gerrit.wikimedia.org/r/365433 (https://phabricator.wikimedia.org/T170653) [03:58:27] (03CR) 10jerkins-bot: [V: 04-1] servermon: restrict http access to prod networks [puppet] - 10https://gerrit.wikimedia.org/r/365433 (https://phabricator.wikimedia.org/T170653) (owner: 10Dzahn) [03:58:45] PROBLEM - High lag on wdqs1001 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [1800.0] [04:00:22] (03PS2) 10Dzahn: servermon: restrict http access to prod networks [puppet] - 10https://gerrit.wikimedia.org/r/365433 (https://phabricator.wikimedia.org/T170653) [04:03:40] (03PS3) 10Dzahn: servermon: restrict http access to prod networks [puppet] - 10https://gerrit.wikimedia.org/r/365433 (https://phabricator.wikimedia.org/T170653) [04:03:45] PROBLEM - High lag on wdqs1002 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [1800.0] [04:04:24] (03CR) 10Krinkle: "RE: jawiktionary - LGTM." [mediawiki-config] - 10https://gerrit.wikimedia.org/r/365092 (https://phabricator.wikimedia.org/T150618) (owner: 10Urbanecm) [04:05:03] (03CR) 10Dzahn: [C: 032] servermon: restrict http access to prod networks [puppet] - 10https://gerrit.wikimedia.org/r/365433 (https://phabricator.wikimedia.org/T170653) (owner: 10Dzahn) [04:07:02] (03CR) 10Dzahn: "change applied on netmon1003 - https://servermon.wikimedia.org/ works just like before" [puppet] - 10https://gerrit.wikimedia.org/r/365433 (https://phabricator.wikimedia.org/T170653) (owner: 10Dzahn) [04:08:06] PROBLEM - High lag on wdqs1003 is CRITICAL: CRITICAL: 34.48% of data above the critical threshold [1800.0] [04:09:35] PROBLEM - mailman I/O stats on fermium is CRITICAL: CRITICAL - I/O stats: Transfers/Sec=3606.10 Read Requests/Sec=517.00 Write Requests/Sec=1.90 KBytes Read/Sec=53863.60 KBytes_Written/Sec=47.60 [04:16:45] RECOVERY - mailman I/O stats on fermium is OK: OK - I/O stats: Transfers/Sec=101.90 Read Requests/Sec=179.80 Write Requests/Sec=10.60 KBytes Read/Sec=2638.80 KBytes_Written/Sec=580.80 [04:19:57] (03PS1) 10Dzahn: smokeping: restrict http access to prod networks [puppet] - 10https://gerrit.wikimedia.org/r/365434 [04:25:35] (03PS1) 10Dzahn: cache::misc: remove unused director netmon1001 [puppet] - 10https://gerrit.wikimedia.org/r/365435 [05:20:51] RECOVERY - MariaDB Slave Lag: s5 on dbstore2001 is OK: OK slave_sql_lag Replication lag: 89843.62 seconds [07:02:21] PROBLEM - Restbase edge ulsfo on text-lb.ulsfo.wikimedia.org is CRITICAL: /api/rest_v1/transform/wikitext/to/html{/title}{/revision} (Transform wikitext to html) timed out before a response was received [07:03:11] RECOVERY - Restbase edge ulsfo on text-lb.ulsfo.wikimedia.org is OK: All endpoints are healthy [07:44:31] PROBLEM - puppet last run on ocg1001 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [07:44:41] PROBLEM - puppet last run on labvirt1015 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [07:44:51] PROBLEM - puppet last run on mwdebug1002 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [07:44:53] PROBLEM - puppet last run on analytics1061 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [07:44:53] PROBLEM - puppet last run on db1085 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [07:45:11] PROBLEM - puppet last run on restbase-dev1002 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [07:45:21] PROBLEM - puppet last run on mw1231 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [07:45:22] PROBLEM - puppet last run on ms-be1015 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [07:45:32] PROBLEM - puppet last run on mw1294 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [07:45:51] PROBLEM - puppet last run on analytics1052 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [07:45:51] PROBLEM - puppet last run on db1059 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [07:45:52] PROBLEM - puppet last run on wtp1003 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [07:46:01] PROBLEM - puppet last run on rdb1008 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [07:46:03] PROBLEM - puppet last run on thumbor1003 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [07:46:03] PROBLEM - puppet last run on cp1046 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [07:46:03] PROBLEM - puppet last run on db1048 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [07:46:11] PROBLEM - puppet last run on labstore1003 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [07:46:11] PROBLEM - puppet last run on darmstadtium is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [07:46:11] PROBLEM - puppet last run on analytics1003 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [07:46:11] PROBLEM - puppet last run on aqs1005 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [07:46:21] PROBLEM - puppet last run on db1087 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [07:46:21] PROBLEM - puppet last run on mw1215 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [07:46:21] PROBLEM - puppet last run on ms-fe1008 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [07:46:21] PROBLEM - puppet last run on dysprosium is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [07:46:22] PROBLEM - puppet last run on acamar is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [07:46:31] PROBLEM - puppet last run on cp1054 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [07:46:31] PROBLEM - puppet last run on scb1003 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [07:46:31] PROBLEM - puppet last run on db1105 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [07:46:51] PROBLEM - puppet last run on db1061 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [07:48:01] PROBLEM - puppet last run on elastic1018 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [07:48:02] PROBLEM - puppet last run on ms-be1035 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [07:48:11] PROBLEM - puppet last run on cp3040 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [07:48:21] PROBLEM - puppet last run on ms-be1026 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [07:48:24] PROBLEM - puppet last run on mw1225 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [07:48:31] PROBLEM - puppet last run on labvirt1003 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [07:48:32] PROBLEM - puppet last run on mw1245 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [07:48:32] PROBLEM - puppet last run on prometheus1003 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [07:48:41] PROBLEM - puppet last run on mw1275 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [07:48:41] PROBLEM - puppet last run on elastic1039 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [07:48:41] PROBLEM - puppet last run on einsteinium is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [07:48:42] PROBLEM - puppet last run on lvs1004 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [07:48:42] PROBLEM - puppet last run on labsdb1004 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [07:48:51] PROBLEM - puppet last run on netmon2001 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [07:48:51] PROBLEM - puppet last run on mw1274 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [07:48:51] PROBLEM - puppet last run on db1083 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [07:48:51] PROBLEM - puppet last run on mw1165 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [07:48:52] PROBLEM - puppet last run on fermium is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [07:48:52] PROBLEM - puppet last run on db1084 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [07:48:52] PROBLEM - puppet last run on labtestcontrol2001 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [07:48:53] PROBLEM - puppet last run on dbstore1001 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [07:48:53] PROBLEM - puppet last run on restbase1017 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [07:48:54] PROBLEM - puppet last run on nescio is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [07:49:01] PROBLEM - puppet last run on labstore1005 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [07:49:01] PROBLEM - puppet last run on mw1285 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [07:49:01] PROBLEM - puppet last run on wtp1039 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [07:49:01] PROBLEM - puppet last run on wtp1009 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [07:49:01] PROBLEM - puppet last run on cp1047 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [07:49:02] PROBLEM - puppet last run on db1049 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [07:49:02] PROBLEM - puppet last run on analytics1064 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [07:49:03] PROBLEM - puppet last run on nitrogen is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [07:49:03] PROBLEM - puppet last run on stat1003 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [07:49:04] PROBLEM - puppet last run on lvs3003 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [07:49:11] PROBLEM - puppet last run on ms-be1029 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [07:49:11] PROBLEM - puppet last run on mw1277 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [07:49:11] PROBLEM - puppet last run on ms-be1036 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [07:49:11] PROBLEM - puppet last run on labvirt1001 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [07:49:12] PROBLEM - puppet last run on db1103 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [07:49:12] PROBLEM - puppet last run on elastic1028 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [07:49:12] PROBLEM - puppet last run on dbproxy1008 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [07:49:21] PROBLEM - puppet last run on cp3007 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [07:49:21] PROBLEM - puppet last run on kubernetes1003 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [07:49:21] PROBLEM - puppet last run on cp3042 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [07:49:31] PROBLEM - puppet last run on mw1214 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [07:49:31] PROBLEM - puppet last run on db1037 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [07:49:32] PROBLEM - puppet last run on ms-be1018 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [07:49:32] PROBLEM - puppet last run on mw1247 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [07:49:32] PROBLEM - puppet last run on kafka1014 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [07:49:41] PROBLEM - puppet last run on restbase1014 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [07:49:41] PROBLEM - puppet last run on mw1216 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [07:49:41] PROBLEM - puppet last run on analytics1063 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [07:49:41] PROBLEM - puppet last run on mw1248 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [07:49:41] PROBLEM - puppet last run on lvs1006 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [07:49:42] PROBLEM - puppet last run on mw1305 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [07:49:42] PROBLEM - puppet last run on mendelevium is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [07:49:51] PROBLEM - puppet last run on analytics1054 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [07:50:11] PROBLEM - puppet last run on cp3045 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [07:50:31] PROBLEM - puppet last run on ganeti1004 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [07:50:31] PROBLEM - puppet last run on mw1235 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [07:50:32] PROBLEM - puppet last run on labsdb1009 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [07:50:32] PROBLEM - puppet last run on elastic1033 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [07:50:32] PROBLEM - puppet last run on analytics1066 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [07:50:41] PROBLEM - puppet last run on wtp1020 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [07:50:41] PROBLEM - puppet last run on wtp1021 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [07:50:41] PROBLEM - puppet last run on wtp1017 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [07:50:42] PROBLEM - puppet last run on cp3041 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [07:50:42] PROBLEM - puppet last run on lvs3004 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [07:50:51] PROBLEM - puppet last run on mc1030 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [07:50:51] PROBLEM - puppet last run on mc1032 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [07:50:51] PROBLEM - puppet last run on db1063 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [07:50:52] PROBLEM - puppet last run on mw1257 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [07:50:52] PROBLEM - puppet last run on etcd1006 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [07:50:52] PROBLEM - puppet last run on etcd1003 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [07:50:52] PROBLEM - puppet last run on rdb1001 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [07:50:53] PROBLEM - puppet last run on analytics1029 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [07:50:53] PROBLEM - puppet last run on cp3005 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [07:50:54] PROBLEM - puppet last run on analytics1036 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [07:50:54] PROBLEM - puppet last run on db1067 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [07:51:01] PROBLEM - puppet last run on db1098 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [07:51:01] PROBLEM - puppet last run on es1017 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [07:51:01] PROBLEM - puppet last run on elastic1037 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [07:51:02] PROBLEM - puppet last run on wtp1019 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [07:51:11] PROBLEM - puppet last run on wtp1041 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [07:51:11] PROBLEM - puppet last run on rcs1002 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [07:51:11] PROBLEM - puppet last run on db1050 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [07:51:11] PROBLEM - puppet last run on mc1024 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [07:51:12] PROBLEM - puppet last run on krypton is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [07:51:12] PROBLEM - puppet last run on mc1021 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [07:51:12] PROBLEM - puppet last run on ganeti1002 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [07:51:13] PROBLEM - puppet last run on mw1163 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [07:51:14] PROBLEM - puppet last run on mw1213 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [07:51:14] PROBLEM - puppet last run on db1104 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [07:51:21] PROBLEM - puppet last run on restbase1010 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [07:51:31] PROBLEM - puppet last run on snapshot1007 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [07:51:41] PROBLEM - puppet last run on mw1221 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [07:51:41] PROBLEM - puppet last run on mw1249 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [07:51:41] PROBLEM - puppet last run on elastic1020 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [07:52:21] PROBLEM - puppet last run on bohrium is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [07:52:41] PROBLEM - puppet last run on analytics1049 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [07:52:41] PROBLEM - puppet last run on labtestservices2002 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [07:52:41] PROBLEM - puppet last run on cp1071 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [07:52:41] PROBLEM - puppet last run on mw1279 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [07:52:41] PROBLEM - puppet last run on mw1293 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [07:52:42] PROBLEM - puppet last run on aqs1004 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [07:52:42] PROBLEM - puppet last run on elastic1019 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [07:52:51] PROBLEM - puppet last run on mw1164 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [07:52:53] PROBLEM - puppet last run on db1015 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [07:52:53] PROBLEM - puppet last run on labvirt1012 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [07:52:53] PROBLEM - puppet last run on mw1205 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [07:52:53] PROBLEM - puppet last run on es1015 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [07:53:01] PROBLEM - puppet last run on cp3046 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [07:53:01] PROBLEM - puppet last run on cp3004 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [07:53:01] PROBLEM - puppet last run on db1100 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [07:53:01] PROBLEM - puppet last run on baham is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [07:53:01] PROBLEM - puppet last run on labvirt1008 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [07:53:02] PROBLEM - puppet last run on db1029 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [07:53:02] PROBLEM - puppet last run on rdb1002 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [07:53:03] PROBLEM - puppet last run on ms-be1021 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [07:53:03] PROBLEM - puppet last run on db1068 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [07:53:04] PROBLEM - puppet last run on elastic1034 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [07:53:11] PROBLEM - puppet last run on cp3043 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [07:53:11] PROBLEM - puppet last run on notebook1001 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [07:53:11] PROBLEM - puppet last run on elastic1049 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [07:53:11] PROBLEM - puppet last run on cp3038 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [07:53:21] PROBLEM - puppet last run on mw1301 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [07:53:21] PROBLEM - puppet last run on kafka1022 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [07:53:21] PROBLEM - puppet last run on elastic1047 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [07:53:21] PROBLEM - puppet last run on analytics1051 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [07:53:21] PROBLEM - puppet last run on cp1059 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [07:53:22] PROBLEM - puppet last run on elastic1035 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [07:53:22] PROBLEM - puppet last run on wdqs1003 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [07:53:23] PROBLEM - puppet last run on db1020 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [07:53:24] PROBLEM - puppet last run on ores1008 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [07:53:24] PROBLEM - puppet last run on mw1203 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [07:53:25] PROBLEM - puppet last run on thumbor1002 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [07:53:25] PROBLEM - puppet last run on ms-be1023 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [07:53:26] PROBLEM - puppet last run on labvirt1016 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [07:53:26] PROBLEM - puppet last run on cp1053 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [07:53:31] PROBLEM - puppet last run on mw1230 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [07:53:41] PROBLEM - puppet last run on db1089 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [07:53:41] PROBLEM - puppet last run on labsdb1011 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [07:53:41] PROBLEM - puppet last run on db1056 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [07:53:41] PROBLEM - puppet last run on restbase1018 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [07:53:41] PROBLEM - puppet last run on db1073 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [07:53:42] PROBLEM - puppet last run on mw1212 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [07:53:42] PROBLEM - puppet last run on mw1295 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [07:53:51] PROBLEM - puppet last run on planet1001 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [07:53:51] PROBLEM - puppet last run on wtp1027 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [07:53:51] PROBLEM - puppet last run on wtp1001 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [07:54:01] PROBLEM - puppet last run on diadem is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [07:54:01] PROBLEM - puppet last run on analytics1068 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [07:54:01] PROBLEM - puppet last run on ganeti1006 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [07:54:11] PROBLEM - puppet last run on wtp1012 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [07:54:11] PROBLEM - puppet last run on ms-be1032 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [07:54:12] PROBLEM - puppet last run on radium is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [08:13:52] RECOVERY - puppet last run on labvirt1015 is OK: OK: Puppet is currently enabled, last run 15 seconds ago with 0 failures [08:14:02] RECOVERY - puppet last run on db1085 is OK: OK: Puppet is currently enabled, last run 54 seconds ago with 0 failures [08:14:02] RECOVERY - puppet last run on mwdebug1002 is OK: OK: Puppet is currently enabled, last run 5 seconds ago with 0 failures [08:14:02] RECOVERY - puppet last run on analytics1061 is OK: OK: Puppet is currently enabled, last run 46 seconds ago with 0 failures [08:14:11] RECOVERY - puppet last run on thumbor1003 is OK: OK: Puppet is currently enabled, last run 9 seconds ago with 0 failures [08:14:12] RECOVERY - puppet last run on rdb1008 is OK: OK: Puppet is currently enabled, last run 11 seconds ago with 0 failures [08:14:21] PROBLEM - puppet last run on rhodium is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [08:14:22] RECOVERY - puppet last run on elastic1028 is OK: OK: Puppet is currently enabled, last run 2 seconds ago with 0 failures [08:14:31] RECOVERY - puppet last run on restbase-dev1002 is OK: OK: Puppet is currently enabled, last run 18 seconds ago with 0 failures [08:14:31] RECOVERY - puppet last run on aqs1005 is OK: OK: Puppet is currently enabled, last run 10 seconds ago with 0 failures [08:14:41] PROBLEM - puppet last run on graphite1001 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [08:14:42] PROBLEM - puppet last run on mc1035 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [08:14:51] RECOVERY - puppet last run on ocg1001 is OK: OK: Puppet is currently enabled, last run 32 seconds ago with 0 failures [08:14:51] PROBLEM - puppet last run on elastic1029 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [08:14:51] PROBLEM - puppet last run on mw1232 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [08:14:51] PROBLEM - puppet last run on analytics1034 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [08:14:52] PROBLEM - puppet last run on mw1207 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [08:15:01] PROBLEM - puppet last run on ms-be1034 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [08:15:01] RECOVERY - puppet last run on analytics1052 is OK: OK: Puppet is currently enabled, last run 33 seconds ago with 0 failures [08:15:11] RECOVERY - puppet last run on db1061 is OK: OK: Puppet is currently enabled, last run 42 seconds ago with 0 failures [08:15:11] RECOVERY - puppet last run on wtp1003 is OK: OK: Puppet is currently enabled, last run 47 seconds ago with 0 failures [08:15:12] PROBLEM - puppet last run on ms-be3003 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [08:15:21] PROBLEM - puppet last run on bast3002 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [08:15:21] RECOVERY - puppet last run on db1048 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [08:15:21] RECOVERY - puppet last run on labstore1003 is OK: OK: Puppet is currently enabled, last run 1 second ago with 0 failures [08:15:22] RECOVERY - puppet last run on labvirt1001 is OK: OK: Puppet is currently enabled, last run 37 seconds ago with 0 failures [08:15:31] RECOVERY - puppet last run on ms-be1029 is OK: OK: Puppet is currently enabled, last run 47 seconds ago with 0 failures [08:15:31] PROBLEM - puppet last run on stat1006 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [08:15:31] PROBLEM - puppet last run on analytics1058 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [08:15:31] RECOVERY - puppet last run on mw1215 is OK: OK: Puppet is currently enabled, last run 28 seconds ago with 0 failures [08:15:31] PROBLEM - puppet last run on mw1202 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [08:15:32] RECOVERY - puppet last run on ms-fe1008 is OK: OK: Puppet is currently enabled, last run 33 seconds ago with 0 failures [08:15:32] PROBLEM - puppet last run on db1039 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [08:15:41] RECOVERY - puppet last run on cp1054 is OK: OK: Puppet is currently enabled, last run 36 seconds ago with 0 failures [08:15:42] RECOVERY - puppet last run on acamar is OK: OK: Puppet is currently enabled, last run 28 seconds ago with 0 failures [08:15:51] RECOVERY - puppet last run on db1105 is OK: OK: Puppet is currently enabled, last run 48 seconds ago with 0 failures [08:15:51] RECOVERY - puppet last run on mw1275 is OK: OK: Puppet is currently enabled, last run 20 seconds ago with 0 failures [08:15:51] RECOVERY - puppet last run on mw1216 is OK: OK: Puppet is currently enabled, last run 29 seconds ago with 0 failures [08:16:02] RECOVERY - puppet last run on etcd1006 is OK: OK: Puppet is currently enabled, last run 32 seconds ago with 0 failures [08:16:02] RECOVERY - puppet last run on dbstore1001 is OK: OK: Puppet is currently enabled, last run 5 seconds ago with 0 failures [08:16:11] RECOVERY - puppet last run on mw1285 is OK: OK: Puppet is currently enabled, last run 58 seconds ago with 0 failures [08:16:32] RECOVERY - puppet last run on cp3040 is OK: OK: Puppet is currently enabled, last run 1 second ago with 0 failures [08:16:41] RECOVERY - puppet last run on db1037 is OK: OK: Puppet is currently enabled, last run 29 seconds ago with 0 failures [08:16:42] RECOVERY - puppet last run on mw1247 is OK: OK: Puppet is currently enabled, last run 25 seconds ago with 0 failures [08:16:51] RECOVERY - puppet last run on mw1245 is OK: OK: Puppet is currently enabled, last run 44 seconds ago with 0 failures [08:16:51] RECOVERY - puppet last run on ms-be1018 is OK: OK: Puppet is currently enabled, last run 30 seconds ago with 0 failures [08:16:51] RECOVERY - puppet last run on elastic1039 is OK: OK: Puppet is currently enabled, last run 48 seconds ago with 0 failures [08:16:51] RECOVERY - puppet last run on aqs1004 is OK: OK: Puppet is currently enabled, last run 42 seconds ago with 0 failures [08:16:51] RECOVERY - puppet last run on analytics1063 is OK: OK: Puppet is currently enabled, last run 50 seconds ago with 0 failures [08:16:53] RECOVERY - puppet last run on wtp1020 is OK: OK: Puppet is currently enabled, last run 9 seconds ago with 0 failures [08:16:54] RECOVERY - puppet last run on mw1248 is OK: OK: Puppet is currently enabled, last run 19 seconds ago with 0 failures [08:16:54] RECOVERY - puppet last run on mw1305 is OK: OK: Puppet is currently enabled, last run 26 seconds ago with 0 failures [08:16:54] RECOVERY - puppet last run on lvs1006 is OK: OK: Puppet is currently enabled, last run 56 seconds ago with 0 failures [08:16:54] RECOVERY - puppet last run on lvs1004 is OK: OK: Puppet is currently enabled, last run 13 seconds ago with 0 failures [08:17:01] RECOVERY - puppet last run on cp3041 is OK: OK: Puppet is currently enabled, last run 28 seconds ago with 0 failures [08:17:02] RECOVERY - puppet last run on db1084 is OK: OK: Puppet is currently enabled, last run 25 seconds ago with 0 failures [08:17:11] RECOVERY - puppet last run on mw1165 is OK: OK: Puppet is currently enabled, last run 50 seconds ago with 0 failures [08:17:11] RECOVERY - puppet last run on fermium is OK: OK: Puppet is currently enabled, last run 53 seconds ago with 0 failures [08:17:11] RECOVERY - puppet last run on analytics1036 is OK: OK: Puppet is currently enabled, last run 7 seconds ago with 0 failures [08:17:11] RECOVERY - puppet last run on netmon2001 is OK: OK: Puppet is currently enabled, last run 13 seconds ago with 0 failures [08:17:11] RECOVERY - puppet last run on labtestcontrol2001 is OK: OK: Puppet is currently enabled, last run 30 seconds ago with 0 failures [08:17:12] RECOVERY - puppet last run on cp3005 is OK: OK: Puppet is currently enabled, last run 1 second ago with 0 failures [08:17:21] RECOVERY - puppet last run on wtp1009 is OK: OK: Puppet is currently enabled, last run 8 seconds ago with 0 failures [08:17:21] RECOVERY - puppet last run on cp1047 is OK: OK: Puppet is currently enabled, last run 32 seconds ago with 0 failures [08:17:21] RECOVERY - puppet last run on analytics1064 is OK: OK: Puppet is currently enabled, last run 38 seconds ago with 0 failures [08:17:21] RECOVERY - puppet last run on nitrogen is OK: OK: Puppet is currently enabled, last run 24 seconds ago with 0 failures [08:17:31] RECOVERY - puppet last run on ganeti1002 is OK: OK: Puppet is currently enabled, last run 4 seconds ago with 0 failures [08:17:31] RECOVERY - puppet last run on db1103 is OK: OK: Puppet is currently enabled, last run 18 seconds ago with 0 failures [08:17:31] RECOVERY - puppet last run on mw1163 is OK: OK: Puppet is currently enabled, last run 8 seconds ago with 0 failures [08:17:31] RECOVERY - puppet last run on ms-be1035 is OK: OK: Puppet is currently enabled, last run 43 seconds ago with 0 failures [08:17:31] RECOVERY - puppet last run on dbproxy1008 is OK: OK: Puppet is currently enabled, last run 8 seconds ago with 0 failures [08:17:32] RECOVERY - puppet last run on bohrium is OK: OK: Puppet is currently enabled, last run 6 seconds ago with 0 failures [08:17:32] RECOVERY - puppet last run on cp3045 is OK: OK: Puppet is currently enabled, last run 46 seconds ago with 0 failures [08:17:41] RECOVERY - puppet last run on mw1225 is OK: OK: Puppet is currently enabled, last run 54 seconds ago with 0 failures [08:17:41] RECOVERY - puppet last run on mw1214 is OK: OK: Puppet is currently enabled, last run 25 seconds ago with 0 failures [08:17:41] RECOVERY - puppet last run on ganeti1004 is OK: OK: Puppet is currently enabled, last run 54 seconds ago with 0 failures [08:17:41] RECOVERY - puppet last run on ms-be1026 is OK: OK: Puppet is currently enabled, last run 8 seconds ago with 0 failures [08:17:42] RECOVERY - puppet last run on cp3042 is OK: OK: Puppet is currently enabled, last run 11 seconds ago with 0 failures [08:17:42] RECOVERY - puppet last run on labvirt1003 is OK: OK: Puppet is currently enabled, last run 7 seconds ago with 0 failures [08:17:51] RECOVERY - puppet last run on kafka1014 is OK: OK: Puppet is currently enabled, last run 22 seconds ago with 0 failures [08:17:51] RECOVERY - puppet last run on analytics1066 is OK: OK: Puppet is currently enabled, last run 54 seconds ago with 0 failures [08:17:51] RECOVERY - puppet last run on mw1294 is OK: OK: Puppet is currently enabled, last run 43 seconds ago with 0 failures [08:18:01] RECOVERY - puppet last run on mendelevium is OK: OK: Puppet is currently enabled, last run 30 seconds ago with 0 failures [08:18:02] RECOVERY - puppet last run on db1083 is OK: OK: Puppet is currently enabled, last run 55 seconds ago with 0 failures [08:18:11] RECOVERY - puppet last run on db1067 is OK: OK: Puppet is currently enabled, last run 19 seconds ago with 0 failures [08:18:11] RECOVERY - puppet last run on labvirt1008 is OK: OK: Puppet is currently enabled, last run 18 seconds ago with 0 failures [08:18:12] RECOVERY - puppet last run on wtp1039 is OK: OK: Puppet is currently enabled, last run 37 seconds ago with 0 failures [08:18:21] RECOVERY - puppet last run on ms-be1021 is OK: OK: Puppet is currently enabled, last run 45 seconds ago with 0 failures [08:18:22] RECOVERY - puppet last run on mw1277 is OK: OK: Puppet is currently enabled, last run 41 seconds ago with 0 failures [08:18:22] RECOVERY - puppet last run on mc1021 is OK: OK: Puppet is currently enabled, last run 6 seconds ago with 0 failures [08:18:31] RECOVERY - puppet last run on lvs3003 is OK: OK: Puppet is currently enabled, last run 25 seconds ago with 0 failures [08:18:32] RECOVERY - puppet last run on cp3038 is OK: OK: Puppet is currently enabled, last run 15 seconds ago with 0 failures [08:18:32] RECOVERY - puppet last run on ms-be1036 is OK: OK: Puppet is currently enabled, last run 26 seconds ago with 0 failures [08:18:41] RECOVERY - puppet last run on kubernetes1003 is OK: OK: Puppet is currently enabled, last run 56 seconds ago with 0 failures [08:18:41] RECOVERY - puppet last run on ms-be1023 is OK: OK: Puppet is currently enabled, last run 33 seconds ago with 0 failures [08:18:51] RECOVERY - puppet last run on labsdb1009 is OK: OK: Puppet is currently enabled, last run 4 seconds ago with 0 failures [08:18:54] RECOVERY - puppet last run on elastic1033 is OK: OK: Puppet is currently enabled, last run 35 seconds ago with 0 failures [08:18:54] RECOVERY - puppet last run on restbase1014 is OK: OK: Puppet is currently enabled, last run 53 seconds ago with 0 failures [08:19:01] RECOVERY - puppet last run on wtp1021 is OK: OK: Puppet is currently enabled, last run 30 seconds ago with 0 failures [08:19:01] RECOVERY - puppet last run on elastic1019 is OK: OK: Puppet is currently enabled, last run 59 seconds ago with 0 failures [08:19:01] RECOVERY - puppet last run on mw1164 is OK: OK: Puppet is currently enabled, last run 46 seconds ago with 0 failures [08:19:11] RECOVERY - puppet last run on mc1030 is OK: OK: Puppet is currently enabled, last run 30 seconds ago with 0 failures [08:19:11] RECOVERY - puppet last run on mw1257 is OK: OK: Puppet is currently enabled, last run 17 seconds ago with 0 failures [08:19:11] RECOVERY - puppet last run on mw1274 is OK: OK: Puppet is currently enabled, last run 38 seconds ago with 0 failures [08:19:11] RECOVERY - puppet last run on db1063 is OK: OK: Puppet is currently enabled, last run 30 seconds ago with 0 failures [08:19:11] RECOVERY - puppet last run on mw1205 is OK: OK: Puppet is currently enabled, last run 45 seconds ago with 0 failures [08:19:12] RECOVERY - puppet last run on analytics1054 is OK: OK: Puppet is currently enabled, last run 46 seconds ago with 0 failures [08:19:12] RECOVERY - puppet last run on etcd1003 is OK: OK: Puppet is currently enabled, last run 30 seconds ago with 0 failures [08:19:13] RECOVERY - puppet last run on restbase1017 is OK: OK: Puppet is currently enabled, last run 51 seconds ago with 0 failures [08:19:13] RECOVERY - puppet last run on labstore1005 is OK: OK: Puppet is currently enabled, last run 56 seconds ago with 0 failures [08:19:14] RECOVERY - puppet last run on elastic1037 is OK: OK: Puppet is currently enabled, last run 33 seconds ago with 0 failures [08:19:21] RECOVERY - puppet last run on db1049 is OK: OK: Puppet is currently enabled, last run 47 seconds ago with 0 failures [08:19:21] RECOVERY - puppet last run on stat1003 is OK: OK: Puppet is currently enabled, last run 30 seconds ago with 0 failures [08:19:21] RECOVERY - puppet last run on nescio is OK: OK: Puppet is currently enabled, last run 55 seconds ago with 0 failures [08:19:22] RECOVERY - puppet last run on rcs1002 is OK: OK: Puppet is currently enabled, last run 25 seconds ago with 0 failures [08:19:22] RECOVERY - puppet last run on db1050 is OK: OK: Puppet is currently enabled, last run 13 seconds ago with 0 failures [08:19:31] RECOVERY - puppet last run on krypton is OK: OK: Puppet is currently enabled, last run 4 seconds ago with 0 failures [08:19:31] RECOVERY - puppet last run on db1104 is OK: OK: Puppet is currently enabled, last run 15 seconds ago with 0 failures [08:19:32] RECOVERY - puppet last run on mw1203 is OK: OK: Puppet is currently enabled, last run 30 seconds ago with 0 failures [08:19:32] RECOVERY - puppet last run on ores1008 is OK: OK: Puppet is currently enabled, last run 34 seconds ago with 0 failures [08:19:41] RECOVERY - puppet last run on cp3007 is OK: OK: Puppet is currently enabled, last run 51 seconds ago with 0 failures [08:19:42] RECOVERY - puppet last run on snapshot1007 is OK: OK: Puppet is currently enabled, last run 57 seconds ago with 0 failures [08:19:51] RECOVERY - puppet last run on mw1221 is OK: OK: Puppet is currently enabled, last run 46 seconds ago with 0 failures [08:19:51] RECOVERY - puppet last run on mw1249 is OK: OK: Puppet is currently enabled, last run 34 seconds ago with 0 failures [08:19:51] RECOVERY - puppet last run on elastic1020 is OK: OK: Puppet is currently enabled, last run 24 seconds ago with 0 failures [08:19:52] RECOVERY - puppet last run on analytics1049 is OK: OK: Puppet is currently enabled, last run 9 seconds ago with 0 failures [08:19:52] RECOVERY - puppet last run on labtestservices2002 is OK: OK: Puppet is currently enabled, last run 38 seconds ago with 0 failures [08:20:11] RECOVERY - puppet last run on mc1032 is OK: OK: Puppet is currently enabled, last run 18 seconds ago with 0 failures [08:20:11] RECOVERY - puppet last run on lvs3004 is OK: OK: Puppet is currently enabled, last run 14 seconds ago with 0 failures [08:20:11] RECOVERY - puppet last run on analytics1029 is OK: OK: Puppet is currently enabled, last run 56 seconds ago with 0 failures [08:20:11] RECOVERY - puppet last run on rdb1001 is OK: OK: Puppet is currently enabled, last run 59 seconds ago with 0 failures [08:20:11] RECOVERY - puppet last run on diadem is OK: OK: Puppet is currently enabled, last run 28 seconds ago with 0 failures [08:20:12] RECOVERY - puppet last run on db1098 is OK: OK: Puppet is currently enabled, last run 48 seconds ago with 0 failures [08:20:12] RECOVERY - puppet last run on es1017 is OK: OK: Puppet is currently enabled, last run 45 seconds ago with 0 failures [08:20:21] RECOVERY - puppet last run on db1068 is OK: OK: Puppet is currently enabled, last run 11 seconds ago with 0 failures [08:20:21] RECOVERY - puppet last run on wtp1019 is OK: OK: Puppet is currently enabled, last run 10 seconds ago with 0 failures [08:20:22] RECOVERY - puppet last run on elastic1049 is OK: OK: Puppet is currently enabled, last run 34 seconds ago with 0 failures [08:20:22] RECOVERY - puppet last run on wtp1041 is OK: OK: Puppet is currently enabled, last run 35 seconds ago with 0 failures [08:20:31] RECOVERY - puppet last run on mc1024 is OK: OK: Puppet is currently enabled, last run 56 seconds ago with 0 failures [08:20:31] RECOVERY - puppet last run on cp3043 is OK: OK: Puppet is currently enabled, last run 5 seconds ago with 0 failures [08:20:31] RECOVERY - puppet last run on mw1213 is OK: OK: Puppet is currently enabled, last run 57 seconds ago with 0 failures [08:20:31] RECOVERY - puppet last run on cp1059 is OK: OK: Puppet is currently enabled, last run 26 seconds ago with 0 failures [08:20:32] RECOVERY - puppet last run on wdqs1003 is OK: OK: Puppet is currently enabled, last run 11 seconds ago with 0 failures [08:20:32] RECOVERY - puppet last run on restbase1010 is OK: OK: Puppet is currently enabled, last run 1 second ago with 0 failures [08:20:41] RECOVERY - puppet last run on thumbor1002 is OK: OK: Puppet is currently enabled, last run 18 seconds ago with 0 failures [08:20:41] RECOVERY - puppet last run on mw1230 is OK: OK: Puppet is currently enabled, last run 28 seconds ago with 0 failures [08:20:51] RECOVERY - puppet last run on db1089 is OK: OK: Puppet is currently enabled, last run 23 seconds ago with 0 failures [08:20:51] RECOVERY - puppet last run on mw1235 is OK: OK: Puppet is currently enabled, last run 31 seconds ago with 0 failures [08:21:01] RECOVERY - puppet last run on wtp1017 is OK: OK: Puppet is currently enabled, last run 42 seconds ago with 0 failures [08:21:11] RECOVERY - puppet last run on analytics1068 is OK: OK: Puppet is currently enabled, last run 1 second ago with 0 failures [08:21:11] RECOVERY - puppet last run on es1015 is OK: OK: Puppet is currently enabled, last run 17 seconds ago with 0 failures [08:21:11] RECOVERY - puppet last run on db1100 is OK: OK: Puppet is currently enabled, last run 43 seconds ago with 0 failures [08:21:21] RECOVERY - puppet last run on elastic1034 is OK: OK: Puppet is currently enabled, last run 9 seconds ago with 0 failures [08:21:31] RECOVERY - puppet last run on radium is OK: OK: Puppet is currently enabled, last run 8 seconds ago with 0 failures [08:21:31] RECOVERY - puppet last run on mw1301 is OK: OK: Puppet is currently enabled, last run 18 seconds ago with 0 failures [08:21:51] RECOVERY - puppet last run on restbase1018 is OK: OK: Puppet is currently enabled, last run 20 seconds ago with 0 failures [08:21:52] RECOVERY - puppet last run on db1056 is OK: OK: Puppet is currently enabled, last run 20 seconds ago with 0 failures [08:21:52] RECOVERY - puppet last run on cp1071 is OK: OK: Puppet is currently enabled, last run 9 seconds ago with 0 failures [08:21:52] RECOVERY - puppet last run on db1073 is OK: OK: Puppet is currently enabled, last run 26 seconds ago with 0 failures [08:21:52] RECOVERY - puppet last run on mw1279 is OK: OK: Puppet is currently enabled, last run 23 seconds ago with 0 failures [08:22:01] RECOVERY - puppet last run on wtp1027 is OK: OK: Puppet is currently enabled, last run 27 seconds ago with 0 failures [08:22:01] RECOVERY - puppet last run on db1015 is OK: OK: Puppet is currently enabled, last run 59 seconds ago with 0 failures [08:22:11] RECOVERY - puppet last run on labvirt1012 is OK: OK: Puppet is currently enabled, last run 17 seconds ago with 0 failures [08:22:11] RECOVERY - puppet last run on ganeti1006 is OK: OK: Puppet is currently enabled, last run 53 seconds ago with 0 failures [08:22:21] RECOVERY - puppet last run on baham is OK: OK: Puppet is currently enabled, last run 24 seconds ago with 0 failures [08:22:21] RECOVERY - puppet last run on wtp1012 is OK: OK: Puppet is currently enabled, last run 9 seconds ago with 0 failures [08:22:21] RECOVERY - puppet last run on db1029 is OK: OK: Puppet is currently enabled, last run 42 seconds ago with 0 failures [08:22:21] RECOVERY - puppet last run on rdb1002 is OK: OK: Puppet is currently enabled, last run 40 seconds ago with 0 failures [08:22:21] RECOVERY - puppet last run on cp3046 is OK: OK: Puppet is currently enabled, last run 32 seconds ago with 0 failures [08:22:22] RECOVERY - puppet last run on cp3004 is OK: OK: Puppet is currently enabled, last run 41 seconds ago with 0 failures [08:22:22] RECOVERY - puppet last run on notebook1001 is OK: OK: Puppet is currently enabled, last run 47 seconds ago with 0 failures [08:22:31] RECOVERY - puppet last run on kafka1022 is OK: OK: Puppet is currently enabled, last run 1 second ago with 0 failures [08:22:31] RECOVERY - puppet last run on elastic1047 is OK: OK: Puppet is currently enabled, last run 35 seconds ago with 0 failures [08:22:31] RECOVERY - puppet last run on analytics1051 is OK: OK: Puppet is currently enabled, last run 31 seconds ago with 0 failures [08:22:32] RECOVERY - puppet last run on db1020 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [08:22:32] RECOVERY - puppet last run on elastic1035 is OK: OK: Puppet is currently enabled, last run 19 seconds ago with 0 failures [08:22:41] RECOVERY - puppet last run on cp1053 is OK: OK: Puppet is currently enabled, last run 47 seconds ago with 0 failures [08:22:41] RECOVERY - puppet last run on labvirt1016 is OK: OK: Puppet is currently enabled, last run 34 seconds ago with 0 failures [08:22:51] RECOVERY - puppet last run on labsdb1011 is OK: OK: Puppet is currently enabled, last run 32 seconds ago with 0 failures [08:22:53] RECOVERY - puppet last run on mw1212 is OK: OK: Puppet is currently enabled, last run 44 seconds ago with 0 failures [08:23:01] RECOVERY - puppet last run on planet1001 is OK: OK: Puppet is currently enabled, last run 59 seconds ago with 0 failures [08:23:02] RECOVERY - puppet last run on wtp1001 is OK: OK: Puppet is currently enabled, last run 56 seconds ago with 0 failures [08:23:31] RECOVERY - puppet last run on ms-be1032 is OK: OK: Puppet is currently enabled, last run 44 seconds ago with 0 failures [08:24:01] RECOVERY - puppet last run on mw1295 is OK: OK: Puppet is currently enabled, last run 17 seconds ago with 0 failures [08:25:01] RECOVERY - puppet last run on mw1293 is OK: OK: Puppet is currently enabled, last run 40 seconds ago with 0 failures [08:27:51] PROBLEM - puppet last run on conf1006 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [08:27:51] PROBLEM - puppet last run on mw1287 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [08:28:01] PROBLEM - puppet last run on bast4001 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [08:28:01] PROBLEM - puppet last run on netmon1002 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [08:28:11] PROBLEM - puppet last run on prometheus2004 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [08:28:11] PROBLEM - puppet last run on etcd1005 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [08:28:11] PROBLEM - puppet last run on db1090 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [08:28:12] PROBLEM - puppet last run on analytics1043 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [08:28:12] PROBLEM - puppet last run on labvirt1013 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [08:28:21] PROBLEM - puppet last run on prometheus1004 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [08:28:31] PROBLEM - puppet last run on tin is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [08:28:31] PROBLEM - puppet last run on mw1280 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [08:28:31] PROBLEM - puppet last run on cp1064 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [08:28:32] PROBLEM - puppet last run on wtp1030 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [08:28:41] PROBLEM - puppet last run on cp3049 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [08:28:41] PROBLEM - puppet last run on kubestagetcd1002 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [08:28:51] PROBLEM - puppet last run on mx2001 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [08:28:51] PROBLEM - puppet last run on elastic1044 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [08:29:01] PROBLEM - puppet last run on scb1004 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [08:29:01] PROBLEM - puppet last run on elastic1041 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [08:29:01] PROBLEM - puppet last run on lvs1011 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [08:29:01] PROBLEM - puppet last run on aqs1009 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [08:29:01] PROBLEM - puppet last run on cp1058 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [08:29:02] PROBLEM - puppet last run on mw1270 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [08:29:02] PROBLEM - puppet last run on druid1001 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [08:30:12] PROBLEM - puppet last run on snapshot1001 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [08:30:22] PROBLEM - puppet last run on db1052 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [08:30:22] PROBLEM - puppet last run on meitnerium is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [08:30:31] PROBLEM - puppet last run on analytics1032 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [08:30:32] PROBLEM - puppet last run on mw1242 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [08:30:41] PROBLEM - puppet last run on wtp1011 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [08:30:41] PROBLEM - puppet last run on mw1306 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [08:30:41] PROBLEM - puppet last run on ms-be1038 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [08:30:41] PROBLEM - puppet last run on labsdb1003 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [08:30:42] PROBLEM - puppet last run on db1009 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [08:30:51] PROBLEM - puppet last run on labtestcontrol2003 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [08:30:51] PROBLEM - puppet last run on mw1199 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [08:30:51] PROBLEM - puppet last run on ms-be1024 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [08:31:01] PROBLEM - puppet last run on ms-be1019 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [08:31:01] PROBLEM - puppet last run on labpuppetmaster1001 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [08:31:01] PROBLEM - puppet last run on elastic1040 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [08:31:01] PROBLEM - puppet last run on mw1192 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [08:31:01] PROBLEM - puppet last run on mw1288 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [08:31:02] PROBLEM - puppet last run on analytics1069 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [08:31:02] PROBLEM - puppet last run on analytics1059 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [08:31:03] PROBLEM - puppet last run on analytics1055 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [08:31:03] PROBLEM - puppet last run on analytics1053 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [08:31:04] PROBLEM - puppet last run on labvirt1004 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [08:31:04] PROBLEM - puppet last run on lvs1009 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [08:31:11] PROBLEM - puppet last run on ms-be1013 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [08:31:11] PROBLEM - puppet last run on wtp1047 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [08:31:11] PROBLEM - puppet last run on chromium is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [08:31:11] PROBLEM - puppet last run on mw1290 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [08:31:11] PROBLEM - puppet last run on analytics1001 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [08:31:12] PROBLEM - puppet last run on cp1063 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [08:31:12] PROBLEM - puppet last run on puppetmaster2002 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [08:31:13] PROBLEM - puppet last run on cp3039 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [08:31:21] PROBLEM - puppet last run on prometheus2003 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [08:31:21] PROBLEM - puppet last run on druid1003 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [08:31:21] PROBLEM - puppet last run on etcd1002 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [08:31:21] PROBLEM - puppet last run on mw1266 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [08:31:21] PROBLEM - puppet last run on labvirt1014 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [08:31:22] PROBLEM - puppet last run on analytics1035 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [08:31:22] PROBLEM - puppet last run on analytics1041 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [08:31:23] PROBLEM - puppet last run on es1018 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [08:31:23] PROBLEM - puppet last run on analytics1047 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [08:31:24] PROBLEM - puppet last run on pollux is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [08:31:24] PROBLEM - puppet last run on mw1261 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [08:31:25] PROBLEM - puppet last run on mw1188 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [08:31:25] PROBLEM - puppet last run on mw1272 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [08:31:26] PROBLEM - puppet last run on kafka1003 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [08:31:37] PROBLEM - puppet last run on wtp1045 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [08:31:37] PROBLEM - puppet last run on wtp1028 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [08:31:38] PROBLEM - puppet last run on logstash1005 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [08:31:38] PROBLEM - puppet last run on mc1022 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [08:31:39] PROBLEM - puppet last run on ms-be3002 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [08:31:39] PROBLEM - puppet last run on elastic1048 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [08:31:40] PROBLEM - puppet last run on roentgenium is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [08:31:42] PROBLEM - puppet last run on mw1194 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [08:31:42] PROBLEM - puppet last run on aqs1006 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [08:31:42] PROBLEM - puppet last run on silver is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [08:31:42] PROBLEM - puppet last run on db1046 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [08:31:42] PROBLEM - puppet last run on mw1181 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [08:31:43] PROBLEM - puppet last run on mw1233 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [08:31:52] PROBLEM - puppet last run on puppetmaster2001 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [08:32:01] PROBLEM - puppet last run on labpuppetmaster1002 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [08:32:01] PROBLEM - puppet last run on ms-be3001 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [08:32:01] PROBLEM - puppet last run on es1011 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [08:32:11] PROBLEM - puppet last run on dbproxy1007 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [08:32:11] PROBLEM - puppet last run on db1018 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [08:32:21] PROBLEM - puppet last run on seaborgium is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [08:32:21] PROBLEM - puppet last run on dbproxy1004 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [08:33:20] 10Operations, 10Wikimedia-Language-setup, 10Wikimedia-Site-requests, 10Patch-For-Review, 10User-Urbanecm: Create Dinka Wikipedia - https://phabricator.wikimedia.org/T168518#3441079 (10Dcljr) >>! In T168518#3439529, @Koavf wrote: > it's still not live on Wikidata for interwiki links. What is the timeline... [08:34:51] PROBLEM - puppet last run on mw1180 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [08:35:31] PROBLEM - puppet last run on elastic1031 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [08:35:31] PROBLEM - puppet last run on ms-fe1006 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [08:35:31] PROBLEM - puppet last run on lvs1005 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [08:35:31] PROBLEM - puppet last run on elastic1026 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [08:35:31] PROBLEM - puppet last run on mwdebug1001 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [08:35:32] PROBLEM - puppet last run on lvs1003 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [08:35:41] PROBLEM - puppet last run on mw1256 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [08:35:41] PROBLEM - puppet last run on db1075 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [08:35:41] PROBLEM - puppet last run on mw1201 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [08:35:41] PROBLEM - puppet last run on mw1161 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [08:35:41] PROBLEM - puppet last run on restbase1013 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [08:35:42] PROBLEM - puppet last run on install2002 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [08:35:51] PROBLEM - puppet last run on helium is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [08:35:51] PROBLEM - puppet last run on mw1226 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [08:35:51] PROBLEM - puppet last run on cp3034 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [08:35:51] PROBLEM - puppet last run on cp3037 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [08:35:51] PROBLEM - puppet last run on achernar is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [08:35:52] PROBLEM - puppet last run on lvs3002 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [08:35:52] PROBLEM - puppet last run on db1036 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [08:35:53] PROBLEM - puppet last run on labvirt1011 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [08:35:53] PROBLEM - puppet last run on conf1005 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [08:35:54] PROBLEM - puppet last run on db1021 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [08:36:01] PROBLEM - puppet last run on dbmonitor1001 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [08:36:02] PROBLEM - puppet last run on druid1002 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [08:36:02] PROBLEM - puppet last run on francium is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [08:36:02] PROBLEM - puppet last run on mc1026 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [08:36:02] PROBLEM - puppet last run on mw1303 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [08:36:02] PROBLEM - puppet last run on cp1065 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [08:36:02] PROBLEM - puppet last run on analytics1033 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [08:36:03] PROBLEM - puppet last run on dubnium is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [08:36:03] PROBLEM - puppet last run on analytics1056 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [08:36:04] PROBLEM - puppet last run on labweb1002 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [08:36:04] PROBLEM - puppet last run on cp1049 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [08:36:05] PROBLEM - puppet last run on labstore1004 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [08:36:17] PROBLEM - puppet last run on labvirt1018 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [08:36:21] PROBLEM - puppet last run on db1082 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [08:36:21] PROBLEM - puppet last run on db1074 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [08:36:21] PROBLEM - puppet last run on dumpsdata1001 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [08:36:21] PROBLEM - puppet last run on mw1246 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [08:36:21] PROBLEM - puppet last run on uranium is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [08:36:22] PROBLEM - puppet last run on labvirt1007 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [08:36:22] PROBLEM - puppet last run on aqs1007 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [08:36:23] PROBLEM - puppet last run on puppetmaster1002 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [08:36:23] PROBLEM - puppet last run on analytics1040 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [08:36:24] PROBLEM - puppet last run on mw1218 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [08:36:24] PROBLEM - puppet last run on alsafi is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [08:36:25] PROBLEM - puppet last run on contint2001 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [08:36:26] PROBLEM - puppet last run on db1092 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [08:36:26] PROBLEM - puppet last run on cp1055 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [08:36:27] PROBLEM - puppet last run on db1026 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [08:36:31] PROBLEM - puppet last run on cp1067 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [08:36:31] PROBLEM - puppet last run on pc1005 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [08:36:31] PROBLEM - puppet last run on wtp1015 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [08:36:31] PROBLEM - puppet last run on analytics1039 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [08:36:31] PROBLEM - puppet last run on analytics1050 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [08:36:43] PROBLEM - puppet last run on maps1003 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [08:36:43] PROBLEM - puppet last run on mw1300 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [08:36:44] PROBLEM - puppet last run on mw1240 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [08:36:44] PROBLEM - puppet last run on kafka1012 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [08:36:45] PROBLEM - puppet last run on relforge1001 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [08:36:51] PROBLEM - puppet last run on mw1168 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [08:37:01] PROBLEM - puppet last run on db1095 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [08:37:01] PROBLEM - puppet last run on db1081 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [08:37:01] PROBLEM - puppet last run on mw1190 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [08:37:02] PROBLEM - puppet last run on mc1036 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [08:37:02] PROBLEM - puppet last run on kubestage1001 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [08:37:02] PROBLEM - puppet last run on ores1006 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [08:37:11] PROBLEM - puppet last run on db1034 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [08:37:11] PROBLEM - puppet last run on praseodymium is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [08:37:11] PROBLEM - puppet last run on elastic1046 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [08:37:11] PROBLEM - puppet last run on stat1005 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [08:37:12] PROBLEM - puppet last run on lvs1001 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [08:37:21] PROBLEM - puppet last run on graphite1003 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [08:37:23] PROBLEM - puppet last run on ms-be3004 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [08:37:23] PROBLEM - puppet last run on restbase1015 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [08:37:23] PROBLEM - puppet last run on mw1198 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [08:37:23] PROBLEM - puppet last run on kubernetes1004 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [08:37:31] PROBLEM - puppet last run on cp1062 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [08:37:31] PROBLEM - puppet last run on cp1066 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [08:37:31] PROBLEM - puppet last run on pc1006 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [08:37:32] PROBLEM - puppet last run on labvirt1002 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [08:37:41] PROBLEM - puppet last run on mw1241 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [08:37:41] PROBLEM - puppet last run on cp3031 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [08:37:41] PROBLEM - puppet last run on mw1250 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [08:38:01] PROBLEM - puppet last run on labcontrol1002 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [08:38:01] PROBLEM - puppet last run on restbase1007 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [08:38:12] PROBLEM - puppet last run on cp1073 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [08:38:31] PROBLEM - puppet last run on relforge1002 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [08:38:32] PROBLEM - puppet last run on ores1001 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [08:38:51] PROBLEM - puppet last run on restbase1011 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [08:41:11] RECOVERY - puppet last run on prometheus1003 is OK: OK: Puppet is currently enabled, last run 6 seconds ago with 0 failures [08:41:21] RECOVERY - puppet last run on ms-be1034 is OK: OK: Puppet is currently enabled, last run 42 seconds ago with 0 failures [08:41:41] RECOVERY - puppet last run on rhodium is OK: OK: Puppet is currently enabled, last run 20 seconds ago with 0 failures [08:42:12] RECOVERY - puppet last run on mc1035 is OK: OK: Puppet is currently enabled, last run 30 seconds ago with 0 failures [08:42:12] RECOVERY - puppet last run on elastic1029 is OK: OK: Puppet is currently enabled, last run 32 seconds ago with 0 failures [08:42:12] RECOVERY - puppet last run on analytics1034 is OK: OK: Puppet is currently enabled, last run 32 seconds ago with 0 failures [08:42:31] RECOVERY - puppet last run on db1059 is OK: OK: Puppet is currently enabled, last run 24 seconds ago with 0 failures [08:42:41] RECOVERY - puppet last run on ms-be3003 is OK: OK: Puppet is currently enabled, last run 7 seconds ago with 0 failures [08:42:51] RECOVERY - puppet last run on mw1202 is OK: OK: Puppet is currently enabled, last run 7 seconds ago with 0 failures [08:42:51] RECOVERY - puppet last run on stat1006 is OK: OK: Puppet is currently enabled, last run 22 seconds ago with 0 failures [08:42:51] RECOVERY - puppet last run on db1039 is OK: OK: Puppet is currently enabled, last run 48 seconds ago with 0 failures [08:42:51] RECOVERY - puppet last run on dysprosium is OK: OK: Puppet is currently enabled, last run 37 seconds ago with 0 failures [08:42:51] RECOVERY - puppet last run on mw1231 is OK: OK: Puppet is currently enabled, last run 3 seconds ago with 0 failures [08:43:01] RECOVERY - puppet last run on ms-be1015 is OK: OK: Puppet is currently enabled, last run 17 seconds ago with 0 failures [08:43:02] RECOVERY - puppet last run on graphite1001 is OK: OK: Puppet is currently enabled, last run 57 seconds ago with 0 failures [08:43:21] RECOVERY - puppet last run on mw1207 is OK: OK: Puppet is currently enabled, last run 32 seconds ago with 0 failures [08:43:41] RECOVERY - puppet last run on elastic1018 is OK: OK: Puppet is currently enabled, last run 50 seconds ago with 0 failures [08:43:41] RECOVERY - puppet last run on cp1046 is OK: OK: Puppet is currently enabled, last run 45 seconds ago with 0 failures [08:43:41] RECOVERY - puppet last run on darmstadtium is OK: OK: Puppet is currently enabled, last run 20 seconds ago with 0 failures [08:43:51] RECOVERY - puppet last run on db1087 is OK: OK: Puppet is currently enabled, last run 24 seconds ago with 0 failures [08:43:51] RECOVERY - puppet last run on analytics1058 is OK: OK: Puppet is currently enabled, last run 23 seconds ago with 0 failures [08:44:02] RECOVERY - puppet last run on scb1003 is OK: OK: Puppet is currently enabled, last run 53 seconds ago with 0 failures [08:44:11] RECOVERY - puppet last run on mw1232 is OK: OK: Puppet is currently enabled, last run 33 seconds ago with 0 failures [08:44:21] RECOVERY - puppet last run on labsdb1004 is OK: OK: Puppet is currently enabled, last run 38 seconds ago with 0 failures [08:44:42] RECOVERY - puppet last run on analytics1003 is OK: OK: Puppet is currently enabled, last run 43 seconds ago with 0 failures [08:44:51] RECOVERY - puppet last run on bast3002 is OK: OK: Puppet is currently enabled, last run 56 seconds ago with 0 failures [08:52:31] RECOVERY - puppet last run on prometheus2004 is OK: OK: Puppet is currently enabled, last run 52 seconds ago with 0 failures [08:54:31] RECOVERY - puppet last run on labvirt1013 is OK: OK: Puppet is currently enabled, last run 4 seconds ago with 0 failures [08:54:41] PROBLEM - puppet last run on mw1220 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [08:54:51] RECOVERY - puppet last run on wtp1030 is OK: OK: Puppet is currently enabled, last run 12 seconds ago with 0 failures [08:55:11] RECOVERY - puppet last run on mx2001 is OK: OK: Puppet is currently enabled, last run 8 seconds ago with 0 failures [08:55:11] RECOVERY - puppet last run on cp1058 is OK: OK: Puppet is currently enabled, last run 3 seconds ago with 0 failures [08:56:02] RECOVERY - puppet last run on conf1006 is OK: OK: Puppet is currently enabled, last run 17 seconds ago with 0 failures [08:56:02] RECOVERY - puppet last run on cp3049 is OK: OK: Puppet is currently enabled, last run 30 seconds ago with 0 failures [08:56:11] RECOVERY - puppet last run on mw1287 is OK: OK: Puppet is currently enabled, last run 33 seconds ago with 0 failures [08:56:11] RECOVERY - puppet last run on elastic1041 is OK: OK: Puppet is currently enabled, last run 13 seconds ago with 0 failures [08:56:11] RECOVERY - puppet last run on mw1288 is OK: OK: Puppet is currently enabled, last run 41 seconds ago with 0 failures [08:56:12] RECOVERY - puppet last run on lvs1011 is OK: OK: Puppet is currently enabled, last run 10 seconds ago with 0 failures [08:56:12] RECOVERY - puppet last run on druid1001 is OK: OK: Puppet is currently enabled, last run 16 seconds ago with 0 failures [08:56:21] RECOVERY - puppet last run on netmon1002 is OK: OK: Puppet is currently enabled, last run 41 seconds ago with 0 failures [08:56:21] RECOVERY - puppet last run on cp1063 is OK: OK: Puppet is currently enabled, last run 52 seconds ago with 0 failures [08:56:31] RECOVERY - puppet last run on db1090 is OK: OK: Puppet is currently enabled, last run 0 seconds ago with 0 failures [08:56:31] RECOVERY - puppet last run on etcd1005 is OK: OK: Puppet is currently enabled, last run 10 seconds ago with 0 failures [08:56:31] RECOVERY - puppet last run on etcd1002 is OK: OK: Puppet is currently enabled, last run 12 seconds ago with 0 failures [08:56:31] RECOVERY - puppet last run on analytics1043 is OK: OK: Puppet is currently enabled, last run 39 seconds ago with 0 failures [08:56:31] RECOVERY - puppet last run on prometheus1004 is OK: OK: Puppet is currently enabled, last run 25 seconds ago with 0 failures [08:56:32] RECOVERY - puppet last run on ms-be1014 is OK: OK: Puppet is currently enabled, last run 35 seconds ago with 0 failures [08:56:41] RECOVERY - puppet last run on db1094 is OK: OK: Puppet is currently enabled, last run 48 seconds ago with 0 failures [08:56:42] RECOVERY - puppet last run on db1016 is OK: OK: Puppet is currently enabled, last run 0 seconds ago with 0 failures [08:56:51] RECOVERY - puppet last run on wtp1028 is OK: OK: Puppet is currently enabled, last run 46 seconds ago with 0 failures [08:56:51] RECOVERY - puppet last run on tin is OK: OK: Puppet is currently enabled, last run 46 seconds ago with 0 failures [08:56:51] RECOVERY - puppet last run on cp1064 is OK: OK: Puppet is currently enabled, last run 26 seconds ago with 0 failures [08:56:51] RECOVERY - puppet last run on mw1280 is OK: OK: Puppet is currently enabled, last run 31 seconds ago with 0 failures [08:57:01] RECOVERY - puppet last run on labsdb1003 is OK: OK: Puppet is currently enabled, last run 54 seconds ago with 0 failures [08:57:01] RECOVERY - puppet last run on kubestagetcd1002 is OK: OK: Puppet is currently enabled, last run 24 seconds ago with 0 failures [08:57:02] RECOVERY - puppet last run on mw1199 is OK: OK: Puppet is currently enabled, last run 44 seconds ago with 0 failures [08:57:11] RECOVERY - puppet last run on elastic1044 is OK: OK: Puppet is currently enabled, last run 18 seconds ago with 0 failures [08:57:11] RECOVERY - puppet last run on labtestcontrol2003 is OK: OK: Puppet is currently enabled, last run 50 seconds ago with 0 failures [08:57:11] RECOVERY - puppet last run on scb1004 is OK: OK: Puppet is currently enabled, last run 45 seconds ago with 0 failures [08:57:12] RECOVERY - puppet last run on labvirt1004 is OK: OK: Puppet is currently enabled, last run 11 seconds ago with 0 failures [08:57:12] RECOVERY - puppet last run on aqs1009 is OK: OK: Puppet is currently enabled, last run 45 seconds ago with 0 failures [08:57:12] RECOVERY - puppet last run on mw1270 is OK: OK: Puppet is currently enabled, last run 40 seconds ago with 0 failures [08:57:21] RECOVERY - puppet last run on mw1290 is OK: OK: Puppet is currently enabled, last run 12 seconds ago with 0 failures [08:57:21] RECOVERY - puppet last run on ms-be1013 is OK: OK: Puppet is currently enabled, last run 9 seconds ago with 0 failures [08:57:21] RECOVERY - puppet last run on dbproxy1007 is OK: OK: Puppet is currently enabled, last run 16 seconds ago with 0 failures [08:57:31] RECOVERY - puppet last run on bast4001 is OK: OK: Puppet is currently enabled, last run 10 seconds ago with 0 failures [08:57:31] RECOVERY - puppet last run on labvirt1014 is OK: OK: Puppet is currently enabled, last run 50 seconds ago with 0 failures [08:57:33] RECOVERY - puppet last run on druid1003 is OK: OK: Puppet is currently enabled, last run 2 seconds ago with 0 failures [08:57:33] RECOVERY - puppet last run on seaborgium is OK: OK: Puppet is currently enabled, last run 21 seconds ago with 0 failures [08:57:41] RECOVERY - puppet last run on labservices1002 is OK: OK: Puppet is currently enabled, last run 13 seconds ago with 0 failures [08:57:41] RECOVERY - puppet last run on db1052 is OK: OK: Puppet is currently enabled, last run 44 seconds ago with 0 failures [08:57:41] RECOVERY - puppet last run on mw1304 is OK: OK: Puppet is currently enabled, last run 27 seconds ago with 0 failures [08:57:41] RECOVERY - puppet last run on meitnerium is OK: OK: Puppet is currently enabled, last run 46 seconds ago with 0 failures [08:57:41] RECOVERY - puppet last run on wtp1005 is OK: OK: Puppet is currently enabled, last run 33 seconds ago with 0 failures [08:57:42] RECOVERY - puppet last run on cp3036 is OK: OK: Puppet is currently enabled, last run 29 seconds ago with 0 failures [08:57:42] RECOVERY - puppet last run on analytics1032 is OK: OK: Puppet is currently enabled, last run 21 seconds ago with 0 failures [08:57:51] RECOVERY - puppet last run on mc1033 is OK: OK: Puppet is currently enabled, last run 13 seconds ago with 0 failures [08:57:51] RECOVERY - puppet last run on mc1022 is OK: OK: Puppet is currently enabled, last run 43 seconds ago with 0 failures [08:57:51] RECOVERY - puppet last run on mw1242 is OK: OK: Puppet is currently enabled, last run 55 seconds ago with 0 failures [08:57:52] RECOVERY - puppet last run on db1046 is OK: OK: Puppet is currently enabled, last run 29 seconds ago with 0 failures [08:58:11] RECOVERY - puppet last run on labpuppetmaster1001 is OK: OK: Puppet is currently enabled, last run 42 seconds ago with 0 failures [08:58:12] RECOVERY - puppet last run on analytics1069 is OK: OK: Puppet is currently enabled, last run 14 seconds ago with 0 failures [08:58:21] RECOVERY - puppet last run on chromium is OK: OK: Puppet is currently enabled, last run 22 seconds ago with 0 failures [08:58:21] RECOVERY - puppet last run on wtp1047 is OK: OK: Puppet is currently enabled, last run 28 seconds ago with 0 failures [08:58:31] RECOVERY - puppet last run on snapshot1001 is OK: OK: Puppet is currently enabled, last run 54 seconds ago with 0 failures [08:58:31] RECOVERY - puppet last run on puppetmaster2002 is OK: OK: Puppet is currently enabled, last run 9 seconds ago with 0 failures [08:58:31] RECOVERY - puppet last run on mw1266 is OK: OK: Puppet is currently enabled, last run 35 seconds ago with 0 failures [08:58:41] RECOVERY - puppet last run on elastic1030 is OK: OK: Puppet is currently enabled, last run 58 seconds ago with 0 failures [08:58:42] RECOVERY - puppet last run on elastic1024 is OK: OK: Puppet is currently enabled, last run 24 seconds ago with 0 failures [08:58:42] RECOVERY - puppet last run on rutherfordium is OK: OK: Puppet is currently enabled, last run 34 seconds ago with 0 failures [08:58:42] RECOVERY - puppet last run on labvirt1006 is OK: OK: Puppet is currently enabled, last run 50 seconds ago with 0 failures [08:58:51] RECOVERY - puppet last run on hafnium is OK: OK: Puppet is currently enabled, last run 7 seconds ago with 0 failures [08:58:51] RECOVERY - puppet last run on wtp1045 is OK: OK: Puppet is currently enabled, last run 32 seconds ago with 0 failures [08:58:51] RECOVERY - puppet last run on logstash1005 is OK: OK: Puppet is currently enabled, last run 35 seconds ago with 0 failures [08:58:51] RECOVERY - puppet last run on elastic1048 is OK: OK: Puppet is currently enabled, last run 9 seconds ago with 0 failures [08:58:51] RECOVERY - puppet last run on mw1194 is OK: OK: Puppet is currently enabled, last run 34 seconds ago with 0 failures [08:58:52] RECOVERY - puppet last run on aqs1006 is OK: OK: Puppet is currently enabled, last run 50 seconds ago with 0 failures [08:59:01] RECOVERY - puppet last run on mw1233 is OK: OK: Puppet is currently enabled, last run 25 seconds ago with 0 failures [08:59:01] RECOVERY - puppet last run on ms-be3002 is OK: OK: Puppet is currently enabled, last run 17 seconds ago with 0 failures [08:59:01] RECOVERY - puppet last run on db1009 is OK: OK: Puppet is currently enabled, last run 2 seconds ago with 0 failures [08:59:11] RECOVERY - puppet last run on ms-be1024 is OK: OK: Puppet is currently enabled, last run 1 second ago with 0 failures [08:59:11] RECOVERY - puppet last run on labpuppetmaster1002 is OK: OK: Puppet is currently enabled, last run 1 second ago with 0 failures [08:59:12] RECOVERY - puppet last run on puppetmaster2001 is OK: OK: Puppet is currently enabled, last run 7 seconds ago with 0 failures [08:59:12] RECOVERY - puppet last run on elastic1040 is OK: OK: Puppet is currently enabled, last run 20 seconds ago with 0 failures [08:59:12] RECOVERY - puppet last run on analytics1053 is OK: OK: Puppet is currently enabled, last run 13 seconds ago with 0 failures [08:59:12] RECOVERY - puppet last run on analytics1055 is OK: OK: Puppet is currently enabled, last run 8 seconds ago with 0 failures [08:59:21] RECOVERY - puppet last run on db1018 is OK: OK: Puppet is currently enabled, last run 26 seconds ago with 0 failures [08:59:31] RECOVERY - puppet last run on analytics1041 is OK: OK: Puppet is currently enabled, last run 25 seconds ago with 0 failures [08:59:31] RECOVERY - puppet last run on analytics1035 is OK: OK: Puppet is currently enabled, last run 17 seconds ago with 0 failures [08:59:31] RECOVERY - puppet last run on mw1188 is OK: OK: Puppet is currently enabled, last run 6 seconds ago with 0 failures [08:59:32] RECOVERY - puppet last run on mw1261 is OK: OK: Puppet is currently enabled, last run 0 seconds ago with 0 failures [08:59:32] RECOVERY - puppet last run on mw1272 is OK: OK: Puppet is currently enabled, last run 27 seconds ago with 0 failures [08:59:32] RECOVERY - puppet last run on kafka1003 is OK: OK: Puppet is currently enabled, last run 56 seconds ago with 0 failures [08:59:41] RECOVERY - puppet last run on pollux is OK: OK: Puppet is currently enabled, last run 30 seconds ago with 0 failures [08:59:41] RECOVERY - puppet last run on labcontrol1004 is OK: OK: Puppet is currently enabled, last run 1 second ago with 0 failures [08:59:41] RECOVERY - puppet last run on cp3039 is OK: OK: Puppet is currently enabled, last run 38 seconds ago with 0 failures [08:59:41] RECOVERY - puppet last run on pc1004 is OK: OK: Puppet is currently enabled, last run 18 seconds ago with 0 failures [08:59:42] RECOVERY - puppet last run on cp3008 is OK: OK: Puppet is currently enabled, last run 28 seconds ago with 0 failures [08:59:51] RECOVERY - puppet last run on elastic1017 is OK: OK: Puppet is currently enabled, last run 13 seconds ago with 0 failures [08:59:51] RECOVERY - puppet last run on mx1001 is OK: OK: Puppet is currently enabled, last run 13 seconds ago with 0 failures [08:59:51] RECOVERY - puppet last run on wtp1024 is OK: OK: Puppet is currently enabled, last run 52 seconds ago with 0 failures [08:59:51] RECOVERY - puppet last run on roentgenium is OK: OK: Puppet is currently enabled, last run 6 seconds ago with 0 failures [09:00:01] RECOVERY - puppet last run on mw1306 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [09:00:02] RECOVERY - puppet last run on ms-be1038 is OK: OK: Puppet is currently enabled, last run 20 seconds ago with 0 failures [09:00:12] RECOVERY - puppet last run on ms-be1019 is OK: OK: Puppet is currently enabled, last run 7 seconds ago with 0 failures [09:00:12] RECOVERY - puppet last run on mw1192 is OK: OK: Puppet is currently enabled, last run 46 seconds ago with 0 failures [09:00:12] RECOVERY - puppet last run on analytics1059 is OK: OK: Puppet is currently enabled, last run 52 seconds ago with 0 failures [09:00:21] RECOVERY - puppet last run on es1011 is OK: OK: Puppet is currently enabled, last run 23 seconds ago with 0 failures [09:00:21] RECOVERY - puppet last run on lvs1009 is OK: OK: Puppet is currently enabled, last run 29 seconds ago with 0 failures [09:00:22] RECOVERY - puppet last run on ms-be3001 is OK: OK: Puppet is currently enabled, last run 46 seconds ago with 0 failures [09:00:31] RECOVERY - puppet last run on analytics1047 is OK: OK: Puppet is currently enabled, last run 20 seconds ago with 0 failures [09:00:31] RECOVERY - puppet last run on es1018 is OK: OK: Puppet is currently enabled, last run 41 seconds ago with 0 failures [09:00:41] RECOVERY - puppet last run on dbproxy1004 is OK: OK: Puppet is currently enabled, last run 5 seconds ago with 0 failures [09:00:42] RECOVERY - puppet last run on labnodepool1001 is OK: OK: Puppet is currently enabled, last run 52 seconds ago with 0 failures [09:00:51] RECOVERY - puppet last run on scb1002 is OK: OK: Puppet is currently enabled, last run 17 seconds ago with 0 failures [09:00:52] RECOVERY - puppet last run on wtp1011 is OK: OK: Puppet is currently enabled, last run 54 seconds ago with 0 failures [09:01:01] RECOVERY - puppet last run on silver is OK: OK: Puppet is currently enabled, last run 54 seconds ago with 0 failures [09:01:01] RECOVERY - puppet last run on mw1181 is OK: OK: Puppet is currently enabled, last run 10 seconds ago with 0 failures [09:01:11] RECOVERY - puppet last run on mw1180 is OK: OK: Puppet is currently enabled, last run 4 seconds ago with 0 failures [09:01:12] RECOVERY - puppet last run on druid1002 is OK: OK: Puppet is currently enabled, last run 26 seconds ago with 0 failures [09:01:12] RECOVERY - puppet last run on mc1026 is OK: OK: Puppet is currently enabled, last run 20 seconds ago with 0 failures [09:01:12] RECOVERY - puppet last run on cp1065 is OK: OK: Puppet is currently enabled, last run 13 seconds ago with 0 failures [09:01:21] RECOVERY - puppet last run on netmon1003 is OK: OK: Puppet is currently enabled, last run 26 seconds ago with 0 failures [09:01:21] RECOVERY - puppet last run on wtp1023 is OK: OK: Puppet is currently enabled, last run 10 seconds ago with 0 failures [09:01:21] RECOVERY - puppet last run on wtp1033 is OK: OK: Puppet is currently enabled, last run 47 seconds ago with 0 failures [09:01:22] RECOVERY - puppet last run on wtp1034 is OK: OK: Puppet is currently enabled, last run 30 seconds ago with 0 failures [09:01:22] RECOVERY - puppet last run on analytics1001 is OK: OK: Puppet is currently enabled, last run 59 seconds ago with 0 failures [09:01:32] RECOVERY - puppet last run on dumpsdata1001 is OK: OK: Puppet is currently enabled, last run 23 seconds ago with 0 failures [09:01:41] RECOVERY - puppet last run on prometheus2003 is OK: OK: Puppet is currently enabled, last run 16 seconds ago with 0 failures [09:01:41] RECOVERY - puppet last run on cp1067 is OK: OK: Puppet is currently enabled, last run 34 seconds ago with 0 failures [09:01:41] RECOVERY - puppet last run on pc1005 is OK: OK: Puppet is currently enabled, last run 12 seconds ago with 0 failures [09:01:41] RECOVERY - puppet last run on elastic1031 is OK: OK: Puppet is currently enabled, last run 4 seconds ago with 0 failures [09:02:01] RECOVERY - puppet last run on mw1201 is OK: OK: Puppet is currently enabled, last run 35 seconds ago with 0 failures [09:02:01] RECOVERY - puppet last run on kafka1012 is OK: OK: Puppet is currently enabled, last run 46 seconds ago with 0 failures [09:02:01] RECOVERY - puppet last run on mw1226 is OK: OK: Puppet is currently enabled, last run 19 seconds ago with 0 failures [09:02:02] RECOVERY - puppet last run on install2002 is OK: OK: Puppet is currently enabled, last run 6 seconds ago with 0 failures [09:02:11] RECOVERY - puppet last run on labvirt1011 is OK: OK: Puppet is currently enabled, last run 3 seconds ago with 0 failures [09:02:13] RECOVERY - puppet last run on db1036 is OK: OK: Puppet is currently enabled, last run 19 seconds ago with 0 failures [09:02:13] RECOVERY - puppet last run on db1095 is OK: OK: Puppet is currently enabled, last run 31 seconds ago with 0 failures [09:02:13] RECOVERY - puppet last run on lvs3002 is OK: OK: Puppet is currently enabled, last run 14 seconds ago with 0 failures [09:02:21] RECOVERY - puppet last run on mc1036 is OK: OK: Puppet is currently enabled, last run 18 seconds ago with 0 failures [09:02:21] RECOVERY - puppet last run on cp1049 is OK: OK: Puppet is currently enabled, last run 56 seconds ago with 0 failures [09:02:21] RECOVERY - puppet last run on mw1210 is OK: OK: Puppet is currently enabled, last run 1 second ago with 0 failures [09:02:21] RECOVERY - puppet last run on dbproxy1011 is OK: OK: Puppet is currently enabled, last run 29 seconds ago with 0 failures [09:02:21] RECOVERY - puppet last run on wtp1010 is OK: OK: Puppet is currently enabled, last run 20 seconds ago with 0 failures [09:02:22] RECOVERY - puppet last run on etcd1001 is OK: OK: Puppet is currently enabled, last run 4 seconds ago with 0 failures [09:02:31] RECOVERY - puppet last run on dbmonitor2001 is OK: OK: Puppet is currently enabled, last run 6 seconds ago with 0 failures [09:02:31] RECOVERY - puppet last run on labvirt1018 is OK: OK: Puppet is currently enabled, last run 40 seconds ago with 0 failures [09:02:32] RECOVERY - puppet last run on puppetmaster1002 is OK: OK: Puppet is currently enabled, last run 54 seconds ago with 0 failures [09:02:41] RECOVERY - puppet last run on cp1055 is OK: OK: Puppet is currently enabled, last run 17 seconds ago with 0 failures [09:02:41] RECOVERY - puppet last run on alsafi is OK: OK: Puppet is currently enabled, last run 25 seconds ago with 0 failures [09:02:42] RECOVERY - puppet last run on elastic1026 is OK: OK: Puppet is currently enabled, last run 7 seconds ago with 0 failures [09:02:51] RECOVERY - puppet last run on mw1278 is OK: OK: Puppet is currently enabled, last run 10 seconds ago with 0 failures [09:02:51] RECOVERY - puppet last run on dataset1001 is OK: OK: Puppet is currently enabled, last run 13 seconds ago with 0 failures [09:02:52] RECOVERY - puppet last run on rdb1005 is OK: OK: Puppet is currently enabled, last run 11 seconds ago with 0 failures [09:02:52] RECOVERY - puppet last run on mw1256 is OK: OK: Puppet is currently enabled, last run 39 seconds ago with 0 failures [09:02:52] RECOVERY - puppet last run on wtp1016 is OK: OK: Puppet is currently enabled, last run 9 seconds ago with 0 failures [09:03:02] RECOVERY - puppet last run on restbase1013 is OK: OK: Puppet is currently enabled, last run 30 seconds ago with 0 failures [09:03:02] RECOVERY - puppet last run on helium is OK: OK: Puppet is currently enabled, last run 20 seconds ago with 0 failures [09:03:11] RECOVERY - puppet last run on conf1005 is OK: OK: Puppet is currently enabled, last run 12 seconds ago with 0 failures [09:03:11] RECOVERY - puppet last run on db1021 is OK: OK: Puppet is currently enabled, last run 36 seconds ago with 0 failures [09:03:12] RECOVERY - puppet last run on dbmonitor1001 is OK: OK: Puppet is currently enabled, last run 35 seconds ago with 0 failures [09:03:21] RECOVERY - puppet last run on analytics1056 is OK: OK: Puppet is currently enabled, last run 42 seconds ago with 0 failures [09:03:21] RECOVERY - puppet last run on dubnium is OK: OK: Puppet is currently enabled, last run 59 seconds ago with 0 failures [09:03:21] RECOVERY - puppet last run on mw1167 is OK: OK: Puppet is currently enabled, last run 54 seconds ago with 0 failures [09:03:21] RECOVERY - puppet last run on cp1045 is OK: OK: Puppet is currently enabled, last run 48 seconds ago with 0 failures [09:03:21] RECOVERY - puppet last run on wtp1004 is OK: OK: Puppet is currently enabled, last run 31 seconds ago with 0 failures [09:03:22] RECOVERY - puppet last run on etherpad1001 is OK: OK: Puppet is currently enabled, last run 54 seconds ago with 0 failures [09:03:31] RECOVERY - puppet last run on dbproxy1003 is OK: OK: Puppet is currently enabled, last run 54 seconds ago with 0 failures [09:03:31] RECOVERY - puppet last run on cp1060 is OK: OK: Puppet is currently enabled, last run 22 seconds ago with 0 failures [09:03:31] RECOVERY - puppet last run on lithium is OK: OK: Puppet is currently enabled, last run 48 seconds ago with 0 failures [09:03:31] RECOVERY - puppet last run on db1074 is OK: OK: Puppet is currently enabled, last run 48 seconds ago with 0 failures [09:03:32] RECOVERY - puppet last run on graphite1003 is OK: OK: Puppet is currently enabled, last run 25 seconds ago with 0 failures [09:03:41] RECOVERY - puppet last run on labvirt1007 is OK: OK: Puppet is currently enabled, last run 31 seconds ago with 0 failures [09:03:41] RECOVERY - puppet last run on analytics1040 is OK: OK: Puppet is currently enabled, last run 49 seconds ago with 0 failures [09:03:41] RECOVERY - puppet last run on db1092 is OK: OK: Puppet is currently enabled, last run 50 seconds ago with 0 failures [09:03:41] RECOVERY - puppet last run on cp1066 is OK: OK: Puppet is currently enabled, last run 6 seconds ago with 0 failures [09:03:42] RECOVERY - puppet last run on wtp1015 is OK: OK: Puppet is currently enabled, last run 4 seconds ago with 0 failures [09:03:51] RECOVERY - puppet last run on iridium is OK: OK: Puppet is currently enabled, last run 49 seconds ago with 0 failures [09:03:51] RECOVERY - puppet last run on ms-fe1006 is OK: OK: Puppet is currently enabled, last run 25 seconds ago with 0 failures [09:03:51] RECOVERY - puppet last run on analytics1039 is OK: OK: Puppet is currently enabled, last run 8 seconds ago with 0 failures [09:03:51] RECOVERY - puppet last run on mwdebug1001 is OK: OK: Puppet is currently enabled, last run 45 seconds ago with 0 failures [09:03:51] RECOVERY - puppet last run on lvs1005 is OK: OK: Puppet is currently enabled, last run 43 seconds ago with 0 failures [09:03:52] RECOVERY - puppet last run on labvirt1002 is OK: OK: Puppet is currently enabled, last run 14 seconds ago with 0 failures [09:03:52] RECOVERY - puppet last run on elastic1042 is OK: OK: Puppet is currently enabled, last run 15 seconds ago with 0 failures [09:03:53] RECOVERY - puppet last run on lvs1003 is OK: OK: Puppet is currently enabled, last run 43 seconds ago with 0 failures [09:04:01] RECOVERY - puppet last run on db1075 is OK: OK: Puppet is currently enabled, last run 11 seconds ago with 0 failures [09:04:01] RECOVERY - puppet last run on mw1240 is OK: OK: Puppet is currently enabled, last run 44 seconds ago with 0 failures [09:04:01] RECOVERY - puppet last run on relforge1001 is OK: OK: Puppet is currently enabled, last run 32 seconds ago with 0 failures [09:04:01] RECOVERY - puppet last run on mw1161 is OK: OK: Puppet is currently enabled, last run 59 seconds ago with 0 failures [09:04:01] RECOVERY - puppet last run on mw1168 is OK: OK: Puppet is currently enabled, last run 51 seconds ago with 0 failures [09:04:11] RECOVERY - puppet last run on achernar is OK: OK: Puppet is currently enabled, last run 48 seconds ago with 0 failures [09:04:11] RECOVERY - puppet last run on mw1190 is OK: OK: Puppet is currently enabled, last run 43 seconds ago with 0 failures [09:04:12] RECOVERY - puppet last run on cp3037 is OK: OK: Puppet is currently enabled, last run 43 seconds ago with 0 failures [09:04:12] RECOVERY - puppet last run on cp3034 is OK: OK: Puppet is currently enabled, last run 42 seconds ago with 0 failures [09:04:21] RECOVERY - puppet last run on mw1303 is OK: OK: Puppet is currently enabled, last run 27 seconds ago with 0 failures [09:04:21] RECOVERY - puppet last run on labweb1002 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [09:04:21] RECOVERY - puppet last run on labstore1004 is OK: OK: Puppet is currently enabled, last run 18 seconds ago with 0 failures [09:04:22] RECOVERY - puppet last run on elastic1046 is OK: OK: Puppet is currently enabled, last run 3 seconds ago with 0 failures [09:04:31] RECOVERY - puppet last run on stat1005 is OK: OK: Puppet is currently enabled, last run 20 seconds ago with 0 failures [09:04:31] RECOVERY - puppet last run on dbproxy1001 is OK: OK: Puppet is currently enabled, last run 15 seconds ago with 0 failures [09:04:31] RECOVERY - puppet last run on labsdb1007 is OK: OK: Puppet is currently enabled, last run 10 seconds ago with 0 failures [09:04:31] RECOVERY - puppet last run on db1082 is OK: OK: Puppet is currently enabled, last run 13 seconds ago with 0 failures [09:04:32] RECOVERY - puppet last run on uranium is OK: OK: Puppet is currently enabled, last run 44 seconds ago with 0 failures [09:04:34] RECOVERY - puppet last run on mw1246 is OK: OK: Puppet is currently enabled, last run 10 seconds ago with 0 failures [09:04:41] RECOVERY - puppet last run on mw1218 is OK: OK: Puppet is currently enabled, last run 23 seconds ago with 0 failures [09:04:41] RECOVERY - puppet last run on kubernetes1004 is OK: OK: Puppet is currently enabled, last run 8 seconds ago with 0 failures [09:04:41] RECOVERY - puppet last run on db1026 is OK: OK: Puppet is currently enabled, last run 18 seconds ago with 0 failures [09:04:42] RECOVERY - puppet last run on pc1006 is OK: OK: Puppet is currently enabled, last run 46 seconds ago with 0 failures [09:04:51] RECOVERY - puppet last run on analytics1050 is OK: OK: Puppet is currently enabled, last run 58 seconds ago with 0 failures [09:04:51] RECOVERY - puppet last run on ores1001 is OK: OK: Puppet is currently enabled, last run 8 seconds ago with 0 failures [09:04:51] RECOVERY - puppet last run on ms1001 is OK: OK: Puppet is currently enabled, last run 18 seconds ago with 0 failures [09:04:51] RECOVERY - puppet last run on db1030 is OK: OK: Puppet is currently enabled, last run 39 seconds ago with 0 failures [09:04:51] RECOVERY - puppet last run on elastic1025 is OK: OK: Puppet is currently enabled, last run 26 seconds ago with 0 failures [09:05:01] RECOVERY - puppet last run on mw1250 is OK: OK: Puppet is currently enabled, last run 23 seconds ago with 0 failures [09:05:01] RECOVERY - puppet last run on maps1003 is OK: OK: Puppet is currently enabled, last run 44 seconds ago with 0 failures [09:05:01] RECOVERY - puppet last run on mw1300 is OK: OK: Puppet is currently enabled, last run 36 seconds ago with 0 failures [09:05:11] RECOVERY - puppet last run on db1081 is OK: OK: Puppet is currently enabled, last run 24 seconds ago with 0 failures [09:05:12] RECOVERY - puppet last run on labcontrol1002 is OK: OK: Puppet is currently enabled, last run 28 seconds ago with 0 failures [09:05:21] RECOVERY - puppet last run on restbase1007 is OK: OK: Puppet is currently enabled, last run 17 seconds ago with 0 failures [09:05:21] RECOVERY - puppet last run on francium is OK: OK: Puppet is currently enabled, last run 48 seconds ago with 0 failures [09:05:21] RECOVERY - puppet last run on analytics1033 is OK: OK: Puppet is currently enabled, last run 55 seconds ago with 0 failures [09:05:21] RECOVERY - puppet last run on kubestage1001 is OK: OK: Puppet is currently enabled, last run 14 seconds ago with 0 failures [09:05:21] RECOVERY - puppet last run on ores1006 is OK: OK: Puppet is currently enabled, last run 13 seconds ago with 0 failures [09:05:22] RECOVERY - puppet last run on db1034 is OK: OK: Puppet is currently enabled, last run 12 seconds ago with 0 failures [09:05:22] RECOVERY - puppet last run on praseodymium is OK: OK: Puppet is currently enabled, last run 41 seconds ago with 0 failures [09:05:31] RECOVERY - puppet last run on lvs1001 is OK: OK: Puppet is currently enabled, last run 37 seconds ago with 0 failures [09:05:32] RECOVERY - puppet last run on ms-be1037 is OK: OK: Puppet is currently enabled, last run 57 seconds ago with 0 failures [09:05:41] RECOVERY - puppet last run on aqs1007 is OK: OK: Puppet is currently enabled, last run 6 seconds ago with 0 failures [09:05:41] RECOVERY - puppet last run on restbase1015 is OK: OK: Puppet is currently enabled, last run 23 seconds ago with 0 failures [09:05:41] RECOVERY - puppet last run on contint2001 is OK: OK: Puppet is currently enabled, last run 5 seconds ago with 0 failures [09:05:51] RECOVERY - puppet last run on ms-be3004 is OK: OK: Puppet is currently enabled, last run 28 seconds ago with 0 failures [09:05:51] RECOVERY - puppet last run on relforge1002 is OK: OK: Puppet is currently enabled, last run 21 seconds ago with 0 failures [09:05:51] RECOVERY - puppet last run on mw1241 is OK: OK: Puppet is currently enabled, last run 11 seconds ago with 0 failures [09:06:02] RECOVERY - puppet last run on cp3031 is OK: OK: Puppet is currently enabled, last run 41 seconds ago with 0 failures [09:06:31] RECOVERY - puppet last run on cp1073 is OK: OK: Puppet is currently enabled, last run 59 seconds ago with 0 failures [09:06:41] RECOVERY - puppet last run on mw1198 is OK: OK: Puppet is currently enabled, last run 54 seconds ago with 0 failures [09:06:41] RECOVERY - puppet last run on cp1062 is OK: OK: Puppet is currently enabled, last run 51 seconds ago with 0 failures [09:07:02] RECOVERY - puppet last run on restbase1011 is OK: OK: Puppet is currently enabled, last run 14 seconds ago with 0 failures [09:23:51] RECOVERY - puppet last run on mw1220 is OK: OK: Puppet is currently enabled, last run 35 seconds ago with 0 failures [09:30:51] PROBLEM - puppet last run on analytics1060 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [09:30:51] PROBLEM - puppet last run on mw1188 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [09:30:52] PROBLEM - puppet last run on labcontrol1004 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [09:31:01] PROBLEM - puppet last run on elastic1024 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [09:31:02] PROBLEM - puppet last run on mx1001 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [09:31:11] PROBLEM - puppet last run on mw1306 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [09:31:31] PROBLEM - puppet last run on ms-be1024 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [09:31:32] PROBLEM - puppet last run on analytics1069 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [09:32:01] PROBLEM - puppet last run on restbase1008 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [09:32:21] PROBLEM - puppet last run on mw1262 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [09:32:31] PROBLEM - puppet last run on conf1004 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [09:32:34] PROBLEM - puppet last run on analytics1059 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [09:32:34] PROBLEM - puppet last run on es1016 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [09:32:34] PROBLEM - puppet last run on ores1002 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [09:32:41] PROBLEM - puppet last run on wtp1014 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [09:32:41] PROBLEM - puppet last run on wtp1023 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [09:32:41] PROBLEM - puppet last run on puppetmaster2001 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [09:32:41] PROBLEM - puppet last run on wtp1033 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [09:32:41] PROBLEM - puppet last run on db1031 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [09:32:42] PROBLEM - puppet last run on analytics1001 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [09:32:42] PROBLEM - puppet last run on mw1208 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [09:32:51] PROBLEM - puppet last run on dumpsdata1001 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [09:32:51] PROBLEM - puppet last run on analytics1041 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [09:32:51] PROBLEM - puppet last run on analytics1042 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [09:32:51] PROBLEM - puppet last run on analytics1047 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [09:32:51] PROBLEM - puppet last run on es1018 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [09:32:52] PROBLEM - puppet last run on mw1261 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [09:32:52] PROBLEM - puppet last run on prometheus1004 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [09:32:53] PROBLEM - puppet last run on lvs3001 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [09:32:53] PROBLEM - puppet last run on cp1067 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [09:32:54] PROBLEM - puppet last run on ms-be3001 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [09:33:01] PROBLEM - puppet last run on mw1286 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [09:33:01] PROBLEM - puppet last run on pc1004 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [09:33:01] PROBLEM - puppet last run on bast4001 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [09:33:01] PROBLEM - puppet last run on prometheus2003 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [09:33:01] PROBLEM - puppet last run on pollux is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [09:33:02] PROBLEM - puppet last run on labnodepool1001 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [09:33:02] PROBLEM - puppet last run on db1038 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [09:33:03] PROBLEM - puppet last run on scb1002 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [09:33:03] PROBLEM - puppet last run on neodymium is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [09:33:11] PROBLEM - puppet last run on wtp1024 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [09:33:11] PROBLEM - puppet last run on logstash1005 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [09:33:11] PROBLEM - puppet last run on roentgenium is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [09:33:11] PROBLEM - puppet last run on dataset1001 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [09:33:11] PROBLEM - puppet last run on labnet1001 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [09:33:12] PROBLEM - puppet last run on aqs1006 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [09:33:12] PROBLEM - puppet last run on lvs1002 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [09:33:13] PROBLEM - puppet last run on wtp1036 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [09:33:13] PROBLEM - puppet last run on lvs1012 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [09:33:14] PROBLEM - puppet last run on cp3008 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [09:33:14] PROBLEM - puppet last run on mw1181 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [09:33:21] PROBLEM - puppet last run on mw1226 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [09:33:21] PROBLEM - puppet last run on db1009 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [09:33:31] PROBLEM - puppet last run on labpuppetmaster1002 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [09:33:31] PROBLEM - puppet last run on mw1192 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [09:33:31] PROBLEM - puppet last run on cp1049 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [09:33:32] PROBLEM - puppet last run on ms-be1019 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [09:33:41] PROBLEM - puppet last run on cp3032 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [09:33:41] PROBLEM - puppet last run on wtp1042 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [09:37:18] hmm [09:37:31] is that known? [09:38:38] any ops ^^ [09:39:29] 10Operations, 10MW-1.30-release-notes, 10Wikimedia-Language-setup, 10Wikimedia-Site-requests, 10Patch-For-Review: Create Atikamekw Wikipedia - https://phabricator.wikimedia.org/T167714#3441150 (10Dcljr) Sorry for butting in here, but from a non-admin's perspective, AFAICT interwikis seem to be working fi... [09:42:41] PROBLEM - puppet last run on cobalt is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [09:42:43] PROBLEM - puppet last run on analytics1034 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [09:42:43] PROBLEM - puppet last run on db1054 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [09:42:43] PROBLEM - puppet last run on wtp1037 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [09:42:43] i will create a task [09:42:47] hmm [09:42:51] cobalt == gerrit [09:42:52] PROBLEM - puppet last run on db1085 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [09:43:01] PROBLEM - puppet last run on kubestage1002 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [09:43:01] PROBLEM - puppet last run on analytics1030 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [09:43:01] PROBLEM - puppet last run on db1097 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [09:43:01] PROBLEM - puppet last run on mw1296 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [09:43:02] PROBLEM - puppet last run on logstash1004 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [09:43:02] PROBLEM - puppet last run on ms-be1034 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [09:43:11] PROBLEM - puppet last run on oresrdb1001 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [09:43:11] PROBLEM - puppet last run on mw1252 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [09:43:11] PROBLEM - puppet last run on copper is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [09:43:11] PROBLEM - puppet last run on scb1001 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [09:43:11] PROBLEM - puppet last run on oresrdb1002 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [09:43:12] PROBLEM - puppet last run on ganeti1003 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [09:43:12] PROBLEM - puppet last run on db1091 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [09:43:13] PROBLEM - puppet last run on mc1031 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [09:43:13] PROBLEM - puppet last run on stat1002 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [09:43:14] PROBLEM - puppet last run on labvirt1017 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [09:43:21] PROBLEM - puppet last run on mw1283 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [09:43:21] PROBLEM - puppet last run on mw1251 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [09:43:21] PROBLEM - puppet last run on db1072 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [09:43:21] PROBLEM - puppet last run on mw1269 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [09:43:31] PROBLEM - puppet last run on mw1282 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [09:43:31] PROBLEM - puppet last run on db1071 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [09:43:31] PROBLEM - puppet last run on elastic1027 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [09:43:32] PROBLEM - puppet last run on mw1239 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [09:43:41] PROBLEM - puppet last run on elastic1038 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [09:43:41] PROBLEM - puppet last run on ganeti1001 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [09:43:41] PROBLEM - puppet last run on mc1035 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [09:43:41] PROBLEM - puppet last run on mw1255 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [09:43:41] PROBLEM - puppet last run on prometheus1003 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [09:43:42] PROBLEM - puppet last run on bast3002 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [09:43:42] PROBLEM - puppet last run on db1099 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [09:43:43] PROBLEM - puppet last run on elastic1029 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [09:44:01] PROBLEM - puppet last run on analytics1048 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [09:44:01] PROBLEM - puppet last run on analytics1061 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [09:44:01] PROBLEM - puppet last run on ores1005 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [09:44:02] PROBLEM - puppet last run on wtp1046 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [09:44:11] PROBLEM - puppet last run on db1096 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [09:44:12] PROBLEM - puppet last run on rhodium is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [09:44:21] PROBLEM - puppet last run on stat1006 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [09:44:21] PROBLEM - puppet last run on db1039 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [09:44:22] PROBLEM - puppet last run on dysprosium is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [09:44:43] 10Operations: A lot of puppet errors on prod machines - https://phabricator.wikimedia.org/T170738#3441151 (10Paladox) [09:44:52] 10Operations: A lot of puppet errors on prod machines - https://phabricator.wikimedia.org/T170738#3441163 (10Paladox) p:05Triage>03High [09:45:31] 10Operations: A lot of puppet errors on prod machines - https://phabricator.wikimedia.org/T170738#3441151 (10Paladox) [09:46:07] paladox: checking, it seems some distress from puppetdb [09:46:14] thanks :) [09:47:51] PROBLEM - puppet last run on lvs1006 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [09:48:01] PROBLEM - puppet last run on etcd1006 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [09:48:01] PROBLEM - puppet last run on mw1165 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [09:48:04] PROBLEM - puppet last run on wtp1022 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [09:48:11] PROBLEM - puppet last run on poolcounter1002 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [09:48:12] PROBLEM - puppet last run on netmon2001 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [09:48:21] PROBLEM - puppet last run on elastic1050 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [09:48:21] PROBLEM - puppet last run on db1103 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [09:48:21] PROBLEM - puppet last run on dbproxy1008 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [09:48:21] PROBLEM - puppet last run on labtestcontrol2001 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [09:48:31] PROBLEM - puppet last run on cp3041 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [09:48:31] PROBLEM - puppet last run on mw1225 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [09:48:31] PROBLEM - puppet last run on db1037 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [09:48:32] RECOVERY - High lag on wdqs1003 is OK: OK: Less than 30.00% above the threshold [600.0] [09:48:32] PROBLEM - puppet last run on mw1214 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [09:48:32] PROBLEM - puppet last run on ms-be1035 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [09:48:41] PROBLEM - puppet last run on mw1245 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [09:48:41] PROBLEM - puppet last run on mw1247 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [09:48:41] PROBLEM - puppet last run on cp3005 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [09:48:41] PROBLEM - puppet last run on db1044 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [09:48:42] PROBLEM - puppet last run on kubernetes1002 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [09:48:42] PROBLEM - puppet last run on aqs1004 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [09:48:51] PROBLEM - puppet last run on wtp1020 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [09:48:51] PROBLEM - puppet last run on mw1305 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [09:48:51] PROBLEM - puppet last run on mendelevium is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [09:48:51] PROBLEM - puppet last run on cp3045 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [09:48:52] PROBLEM - puppet last run on acamar is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [09:49:01] PROBLEM - puppet last run on ms-be1018 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [09:49:01] PROBLEM - puppet last run on restbase1017 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [09:49:01] PROBLEM - puppet last run on analytics1036 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [09:49:01] PROBLEM - puppet last run on dbstore1001 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [09:49:11] PROBLEM - puppet last run on wtp1009 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [09:49:12] PROBLEM - puppet last run on cp1047 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [09:49:12] PROBLEM - puppet last run on mw1277 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [09:49:51] PROBLEM - puppet last run on cp1068 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [09:50:11] PROBLEM - puppet last run on cp3042 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [09:50:11] PROBLEM - puppet last run on db1049 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [09:50:11] PROBLEM - puppet last run on wtp1039 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [09:50:12] PROBLEM - puppet last run on ms-be1021 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [09:50:12] PROBLEM - puppet last run on rcs1002 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [09:50:21] PROBLEM - puppet last run on mc1021 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [09:50:21] PROBLEM - puppet last run on ganeti1002 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [09:50:21] PROBLEM - puppet last run on mw1163 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [09:50:21] PROBLEM - puppet last run on db1104 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [09:50:21] PROBLEM - puppet last run on mw1203 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [09:50:22] PROBLEM - puppet last run on ores1008 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [09:50:31] PROBLEM - puppet last run on snapshot1007 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [09:50:41] PROBLEM - puppet last run on labvirt1003 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [09:50:41] PROBLEM - puppet last run on labsdb1010 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [09:50:41] PROBLEM - puppet last run on mw1221 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [09:50:41] PROBLEM - puppet last run on ms-be1023 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [09:50:42] PROBLEM - puppet last run on mw1243 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [09:50:42] PROBLEM - puppet last run on elastic1033 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [09:50:42] PROBLEM - puppet last run on mw1249 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [09:50:43] PROBLEM - puppet last run on kafka1014 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [09:50:43] PROBLEM - puppet last run on ms-be1026 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [09:50:51] PROBLEM - puppet last run on db1080 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [09:50:51] PROBLEM - puppet last run on mw1183 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [09:50:51] PROBLEM - puppet last run on ms-be1036 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [09:50:51] PROBLEM - puppet last run on analytics1049 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [09:50:51] PROBLEM - puppet last run on nescio is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [09:50:52] PROBLEM - puppet last run on analytics1037 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [09:50:52] PROBLEM - puppet last run on mw1248 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [09:50:53] PROBLEM - puppet last run on wtp1021 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [09:50:53] PROBLEM - puppet last run on mw1264 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [09:50:54] PROBLEM - puppet last run on restbase1014 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [09:50:54] PROBLEM - puppet last run on elastic1019 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [09:50:55] PROBLEM - puppet last run on wtp1035 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [09:50:55] PROBLEM - puppet last run on mw1164 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [09:50:56] PROBLEM - puppet last run on lvs3003 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [09:51:01] PROBLEM - puppet last run on cp3038 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [09:51:01] PROBLEM - puppet last run on labtestservices2002 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [09:51:01] PROBLEM - puppet last run on cp3040 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [09:51:01] PROBLEM - puppet last run on db1083 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [09:51:01] PROBLEM - puppet last run on mc1030 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [09:51:02] PROBLEM - puppet last run on mc1032 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [09:51:21] PROBLEM - puppet last run on mc1024 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [09:51:31] PROBLEM - puppet last run on lvs3004 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [09:55:02] RECOVERY - puppet last run on prometheus1004 is OK: OK: Puppet is currently enabled, last run 55 seconds ago with 0 failures [09:56:21] RECOVERY - puppet last run on bast4001 is OK: OK: Puppet is currently enabled, last run 50 seconds ago with 0 failures [09:57:51] RECOVERY - puppet last run on analytics1069 is OK: OK: Puppet is currently enabled, last run 41 seconds ago with 0 failures [09:58:11] RECOVERY - puppet last run on elastic1024 is OK: OK: Puppet is currently enabled, last run 9 seconds ago with 0 failures [09:58:22] RECOVERY - puppet last run on mw1306 is OK: OK: Puppet is currently enabled, last run 4 seconds ago with 0 failures [09:58:51] RECOVERY - puppet last run on ms-be1024 is OK: OK: Puppet is currently enabled, last run 30 seconds ago with 0 failures [09:59:01] RECOVERY - puppet last run on puppetmaster2001 is OK: OK: Puppet is currently enabled, last run 14 seconds ago with 0 failures [09:59:11] RECOVERY - puppet last run on analytics1060 is OK: OK: Puppet is currently enabled, last run 11 seconds ago with 0 failures [09:59:11] RECOVERY - puppet last run on analytics1041 is OK: OK: Puppet is currently enabled, last run 32 seconds ago with 0 failures [09:59:11] RECOVERY - puppet last run on es1018 is OK: OK: Puppet is currently enabled, last run 9 seconds ago with 0 failures [09:59:11] RECOVERY - puppet last run on labcontrol1004 is OK: OK: Puppet is currently enabled, last run 31 seconds ago with 0 failures [09:59:12] RECOVERY - puppet last run on pc1004 is OK: OK: Puppet is currently enabled, last run 11 seconds ago with 0 failures [09:59:22] RECOVERY - puppet last run on mx1001 is OK: OK: Puppet is currently enabled, last run 24 seconds ago with 0 failures [09:59:22] RECOVERY - puppet last run on pollux is OK: OK: Puppet is currently enabled, last run 32 seconds ago with 0 failures [09:59:22] RECOVERY - puppet last run on logstash1005 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [09:59:22] RECOVERY - puppet last run on roentgenium is OK: OK: Puppet is currently enabled, last run 17 seconds ago with 0 failures [09:59:31] RECOVERY - puppet last run on aqs1006 is OK: OK: Puppet is currently enabled, last run 48 seconds ago with 0 failures [09:59:32] RECOVERY - puppet last run on db1009 is OK: OK: Puppet is currently enabled, last run 46 seconds ago with 0 failures [09:59:51] RECOVERY - puppet last run on labpuppetmaster1002 is OK: OK: Puppet is currently enabled, last run 51 seconds ago with 0 failures [09:59:51] RECOVERY - puppet last run on mw1192 is OK: OK: Puppet is currently enabled, last run 26 seconds ago with 0 failures [09:59:51] RECOVERY - puppet last run on analytics1059 is OK: OK: Puppet is currently enabled, last run 35 seconds ago with 0 failures [09:59:51] RECOVERY - puppet last run on ms-be1019 is OK: OK: Puppet is currently enabled, last run 2 seconds ago with 0 failures [09:59:51] RECOVERY - puppet last run on es1016 is OK: OK: Puppet is currently enabled, last run 2 seconds ago with 0 failures [09:59:52] RECOVERY - puppet last run on wtp1014 is OK: OK: Puppet is currently enabled, last run 39 seconds ago with 0 failures [09:59:52] RECOVERY - puppet last run on wtp1042 is OK: OK: Puppet is currently enabled, last run 4 seconds ago with 0 failures [10:00:01] RECOVERY - puppet last run on analytics1001 is OK: OK: Puppet is currently enabled, last run 1 second ago with 0 failures [10:00:11] RECOVERY - puppet last run on analytics1047 is OK: OK: Puppet is currently enabled, last run 40 seconds ago with 0 failures [10:00:11] RECOVERY - puppet last run on mw1261 is OK: OK: Puppet is currently enabled, last run 54 seconds ago with 0 failures [10:00:11] RECOVERY - puppet last run on mw1188 is OK: OK: Puppet is currently enabled, last run 56 seconds ago with 0 failures [10:00:21] RECOVERY - puppet last run on restbase1008 is OK: OK: Puppet is currently enabled, last run 4 seconds ago with 0 failures [10:00:21] RECOVERY - puppet last run on labnodepool1001 is OK: OK: Puppet is currently enabled, last run 39 seconds ago with 0 failures [10:00:21] RECOVERY - puppet last run on ms-be3001 is OK: OK: Puppet is currently enabled, last run 41 seconds ago with 0 failures [10:00:21] RECOVERY - puppet last run on wtp1024 is OK: OK: Puppet is currently enabled, last run 50 seconds ago with 0 failures [10:00:31] RECOVERY - puppet last run on lvs1012 is OK: OK: Puppet is currently enabled, last run 43 seconds ago with 0 failures [10:00:31] RECOVERY - puppet last run on wtp1036 is OK: OK: Puppet is currently enabled, last run 0 seconds ago with 0 failures [10:00:31] RECOVERY - puppet last run on mw1181 is OK: OK: Puppet is currently enabled, last run 18 seconds ago with 0 failures [10:00:41] RECOVERY - puppet last run on cp3008 is OK: OK: Puppet is currently enabled, last run 47 seconds ago with 0 failures [10:00:41] RECOVERY - puppet last run on mw1262 is OK: OK: Puppet is currently enabled, last run 20 seconds ago with 0 failures [10:00:51] RECOVERY - puppet last run on conf1004 is OK: OK: Puppet is currently enabled, last run 31 seconds ago with 0 failures [10:00:51] RECOVERY - puppet last run on ores1002 is OK: OK: Puppet is currently enabled, last run 21 seconds ago with 0 failures [10:01:01] RECOVERY - puppet last run on wtp1033 is OK: OK: Puppet is currently enabled, last run 25 seconds ago with 0 failures [10:01:01] RECOVERY - puppet last run on mw1208 is OK: OK: Puppet is currently enabled, last run 9 seconds ago with 0 failures [10:01:11] RECOVERY - puppet last run on analytics1042 is OK: OK: Puppet is currently enabled, last run 40 seconds ago with 0 failures [10:01:12] RECOVERY - puppet last run on mw1286 is OK: OK: Puppet is currently enabled, last run 9 seconds ago with 0 failures [10:01:21] RECOVERY - puppet last run on lvs3001 is OK: OK: Puppet is currently enabled, last run 58 seconds ago with 0 failures [10:01:21] RECOVERY - puppet last run on db1038 is OK: OK: Puppet is currently enabled, last run 38 seconds ago with 0 failures [10:01:21] RECOVERY - puppet last run on prometheus2003 is OK: OK: Puppet is currently enabled, last run 59 seconds ago with 0 failures [10:01:21] RECOVERY - puppet last run on scb1002 is OK: OK: Puppet is currently enabled, last run 44 seconds ago with 0 failures [10:01:22] RECOVERY - puppet last run on neodymium is OK: OK: Puppet is currently enabled, last run 57 seconds ago with 0 failures [10:01:22] RECOVERY - puppet last run on labnet1001 is OK: OK: Puppet is currently enabled, last run 23 seconds ago with 0 failures [10:01:31] RECOVERY - puppet last run on lvs1002 is OK: OK: Puppet is currently enabled, last run 49 seconds ago with 0 failures [10:01:51] RECOVERY - puppet last run on cp1049 is OK: OK: Puppet is currently enabled, last run 23 seconds ago with 0 failures [10:02:01] RECOVERY - puppet last run on wtp1023 is OK: OK: Puppet is currently enabled, last run 57 seconds ago with 0 failures [10:02:01] RECOVERY - puppet last run on db1031 is OK: OK: Puppet is currently enabled, last run 34 seconds ago with 0 failures [10:02:02] RECOVERY - puppet last run on cp3032 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [10:02:11] RECOVERY - puppet last run on dumpsdata1001 is OK: OK: Puppet is currently enabled, last run 59 seconds ago with 0 failures [10:02:11] RECOVERY - puppet last run on cp1067 is OK: OK: Puppet is currently enabled, last run 32 seconds ago with 0 failures [10:02:31] RECOVERY - puppet last run on dataset1001 is OK: OK: Puppet is currently enabled, last run 40 seconds ago with 0 failures [10:02:31] RECOVERY - puppet last run on mw1226 is OK: OK: Puppet is currently enabled, last run 36 seconds ago with 0 failures [10:06:41] !log disable puppet on the entire fleet for puppetdb debugging on nitrogen [10:06:52] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [10:14:31] RECOVERY - High lag on wdqs1001 is OK: OK: Less than 30.00% above the threshold [600.0] [10:26:52] 10Operations, 10Puppet: PuppetDB misbehaving on 2017-07-15 - https://phabricator.wikimedia.org/T170740#3441190 (10akosiaris) [10:27:17] 10Operations, 10Puppet: PuppetDB misbehaving on 2017-07-15 - https://phabricator.wikimedia.org/T170740#3441202 (10akosiaris) p:05Triage>03High [10:39:11] RECOVERY - puppet last run on prometheus1003 is OK: OK: Puppet is currently enabled, last run 13 seconds ago with 0 failures [10:39:32] RECOVERY - puppet last run on mw1252 is OK: OK: Puppet is currently enabled, last run 24 seconds ago with 0 failures [10:39:32] RECOVERY - puppet last run on oresrdb1001 is OK: OK: Puppet is currently enabled, last run 26 seconds ago with 0 failures [10:39:32] RECOVERY - puppet last run on copper is OK: OK: Puppet is currently enabled, last run 57 seconds ago with 0 failures [10:39:32] RECOVERY - puppet last run on oresrdb1002 is OK: OK: Puppet is currently enabled, last run 45 seconds ago with 0 failures [10:39:32] RECOVERY - puppet last run on db1091 is OK: OK: Puppet is currently enabled, last run 23 seconds ago with 0 failures [10:39:41] RECOVERY - puppet last run on mw1283 is OK: OK: Puppet is currently enabled, last run 29 seconds ago with 0 failures [10:39:42] RECOVERY - puppet last run on mw1269 is OK: OK: Puppet is currently enabled, last run 8 seconds ago with 0 failures [10:40:01] RECOVERY - puppet last run on mw1282 is OK: OK: Puppet is currently enabled, last run 39 seconds ago with 0 failures [10:40:01] RECOVERY - puppet last run on elastic1038 is OK: OK: Puppet is currently enabled, last run 18 seconds ago with 0 failures [10:40:11] RECOVERY - puppet last run on mw1255 is OK: OK: Puppet is currently enabled, last run 33 seconds ago with 0 failures [10:40:11] RECOVERY - puppet last run on cobalt is OK: OK: Puppet is currently enabled, last run 42 seconds ago with 0 failures [10:40:21] RECOVERY - puppet last run on wtp1037 is OK: OK: Puppet is currently enabled, last run 44 seconds ago with 0 failures [10:40:22] RECOVERY - puppet last run on kubestage1002 is OK: OK: Puppet is currently enabled, last run 59 seconds ago with 0 failures [10:40:22] RECOVERY - puppet last run on analytics1030 is OK: OK: Puppet is currently enabled, last run 29 seconds ago with 0 failures [10:40:31] RECOVERY - puppet last run on db1097 is OK: OK: Puppet is currently enabled, last run 31 seconds ago with 0 failures [10:40:32] RECOVERY - puppet last run on scb1001 is OK: OK: Puppet is currently enabled, last run 33 seconds ago with 0 failures [10:40:32] RECOVERY - puppet last run on ganeti1003 is OK: OK: Puppet is currently enabled, last run 44 seconds ago with 0 failures [10:40:41] RECOVERY - puppet last run on mc1031 is OK: OK: Puppet is currently enabled, last run 48 seconds ago with 0 failures [10:40:41] RECOVERY - puppet last run on labvirt1017 is OK: OK: Puppet is currently enabled, last run 42 seconds ago with 0 failures [10:40:41] RECOVERY - puppet last run on mw1251 is OK: OK: Puppet is currently enabled, last run 11 seconds ago with 0 failures [10:41:01] RECOVERY - puppet last run on elastic1027 is OK: OK: Puppet is currently enabled, last run 11 seconds ago with 0 failures [10:41:01] RECOVERY - puppet last run on mw1239 is OK: OK: Puppet is currently enabled, last run 31 seconds ago with 0 failures [10:41:02] RECOVERY - puppet last run on ganeti1001 is OK: OK: Puppet is currently enabled, last run 45 seconds ago with 0 failures [10:41:02] RECOVERY - puppet last run on mc1035 is OK: OK: Puppet is currently enabled, last run 18 seconds ago with 0 failures [10:41:22] RECOVERY - puppet last run on bast3002 is OK: OK: Puppet is currently enabled, last run 1 second ago with 0 failures [10:41:31] RECOVERY - puppet last run on ores1005 is OK: OK: Puppet is currently enabled, last run 26 seconds ago with 0 failures [10:41:31] RECOVERY - puppet last run on logstash1004 is OK: OK: Puppet is currently enabled, last run 11 seconds ago with 0 failures [10:41:41] RECOVERY - puppet last run on rhodium is OK: OK: Puppet is currently enabled, last run 48 seconds ago with 0 failures [10:41:41] RECOVERY - puppet last run on ms-be1034 is OK: OK: Puppet is currently enabled, last run 41 seconds ago with 0 failures [10:41:42] RECOVERY - puppet last run on db1072 is OK: OK: Puppet is currently enabled, last run 37 seconds ago with 0 failures [10:41:42] RECOVERY - puppet last run on db1039 is OK: OK: Puppet is currently enabled, last run 1 second ago with 0 failures [10:41:51] RECOVERY - puppet last run on dysprosium is OK: OK: Puppet is currently enabled, last run 6 seconds ago with 0 failures [10:41:52] RECOVERY - puppet last run on db1071 is OK: OK: Puppet is currently enabled, last run 36 seconds ago with 0 failures [10:42:05] !log puppetdb restarted on nitrogen, puppet agents re-enabled - T170740 [10:42:11] RECOVERY - puppet last run on db1099 is OK: OK: Puppet is currently enabled, last run 47 seconds ago with 0 failures [10:42:11] RECOVERY - puppet last run on einsteinium is OK: OK: Puppet is currently enabled, last run 44 seconds ago with 0 failures [10:42:11] RECOVERY - puppet last run on elastic1029 is OK: OK: Puppet is currently enabled, last run 40 seconds ago with 0 failures [10:42:11] RECOVERY - puppet last run on analytics1034 is OK: OK: Puppet is currently enabled, last run 53 seconds ago with 0 failures [10:42:12] RECOVERY - puppet last run on db1054 is OK: OK: Puppet is currently enabled, last run 56 seconds ago with 0 failures [10:42:14] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [10:42:14] T170740: PuppetDB misbehaving on 2017-07-15 - https://phabricator.wikimedia.org/T170740 [10:42:21] RECOVERY - puppet last run on db1085 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [10:42:22] RECOVERY - puppet last run on analytics1048 is OK: OK: Puppet is currently enabled, last run 48 seconds ago with 0 failures [10:42:31] RECOVERY - puppet last run on analytics1061 is OK: OK: Puppet is currently enabled, last run 53 seconds ago with 0 failures [10:42:31] RECOVERY - puppet last run on mw1296 is OK: OK: Puppet is currently enabled, last run 15 seconds ago with 0 failures [10:42:31] RECOVERY - puppet last run on wtp1046 is OK: OK: Puppet is currently enabled, last run 20 seconds ago with 0 failures [10:42:38] paladox: (see sal) all back to normal now [10:42:41] RECOVERY - puppet last run on db1096 is OK: OK: Puppet is currently enabled, last run 59 seconds ago with 0 failures [10:42:42] RECOVERY - puppet last run on stat1002 is OK: OK: Puppet is currently enabled, last run 20 seconds ago with 0 failures [10:42:42] RECOVERY - puppet last run on stat1006 is OK: OK: Puppet is currently enabled, last run 20 seconds ago with 0 failures [10:42:49] elukey thanks :) [10:42:54] a lot of spam from icinga now but all recoveries :) [10:43:00] :) [10:43:22] 10Operations, 10Puppet: PuppetDB misbehaving on 2017-07-15 - https://phabricator.wikimedia.org/T170740#3441210 (10Peachey88) [10:43:24] 10Operations: A lot of puppet errors on prod machines - https://phabricator.wikimedia.org/T170738#3441212 (10Peachey88) [10:44:11] RECOVERY - puppet last run on db1044 is OK: OK: Puppet is currently enabled, last run 6 seconds ago with 0 failures [10:44:32] RECOVERY - puppet last run on poolcounter1002 is OK: OK: Puppet is currently enabled, last run 37 seconds ago with 0 failures [10:45:11] RECOVERY - puppet last run on mw1247 is OK: OK: Puppet is currently enabled, last run 2 seconds ago with 0 failures [10:45:11] RECOVERY - puppet last run on cp3041 is OK: OK: Puppet is currently enabled, last run 1 second ago with 0 failures [10:45:21] RECOVERY - puppet last run on lvs1006 is OK: OK: Puppet is currently enabled, last run 17 seconds ago with 0 failures [10:45:22] RECOVERY - puppet last run on etcd1006 is OK: OK: Puppet is currently enabled, last run 43 seconds ago with 0 failures [10:45:31] RECOVERY - puppet last run on analytics1036 is OK: OK: Puppet is currently enabled, last run 4 seconds ago with 0 failures [10:45:31] RECOVERY - puppet last run on mw1165 is OK: OK: Puppet is currently enabled, last run 32 seconds ago with 0 failures [10:45:31] RECOVERY - puppet last run on wtp1022 is OK: OK: Puppet is currently enabled, last run 0 seconds ago with 0 failures [10:45:32] RECOVERY - puppet last run on acamar is OK: OK: Puppet is currently enabled, last run 58 seconds ago with 0 failures [10:45:32] RECOVERY - puppet last run on ms-be1018 is OK: OK: Puppet is currently enabled, last run 29 seconds ago with 0 failures [10:45:41] RECOVERY - puppet last run on cp3045 is OK: OK: Puppet is currently enabled, last run 17 seconds ago with 0 failures [10:45:41] RECOVERY - puppet last run on cp3040 is OK: OK: Puppet is currently enabled, last run 26 seconds ago with 0 failures [10:45:51] RECOVERY - puppet last run on db1103 is OK: OK: Puppet is currently enabled, last run 2 seconds ago with 0 failures [10:45:51] RECOVERY - puppet last run on elastic1050 is OK: OK: Puppet is currently enabled, last run 7 seconds ago with 0 failures [10:46:01] RECOVERY - puppet last run on labtestcontrol2001 is OK: OK: Puppet is currently enabled, last run 4 seconds ago with 0 failures [10:46:01] RECOVERY - puppet last run on mw1225 is OK: OK: Puppet is currently enabled, last run 41 seconds ago with 0 failures [10:46:01] RECOVERY - puppet last run on db1037 is OK: OK: Puppet is currently enabled, last run 54 seconds ago with 0 failures [10:46:02] RECOVERY - puppet last run on mw1245 is OK: OK: Puppet is currently enabled, last run 45 seconds ago with 0 failures [10:46:11] RECOVERY - puppet last run on kafka1014 is OK: OK: Puppet is currently enabled, last run 7 seconds ago with 0 failures [10:46:11] RECOVERY - puppet last run on kubernetes1002 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [10:46:11] RECOVERY - puppet last run on aqs1004 is OK: OK: Puppet is currently enabled, last run 45 seconds ago with 0 failures [10:46:11] RECOVERY - puppet last run on ms-be1035 is OK: OK: Puppet is currently enabled, last run 53 seconds ago with 0 failures [10:46:11] RECOVERY - puppet last run on ms-be1026 is OK: OK: Puppet is currently enabled, last run 5 seconds ago with 0 failures [10:46:12] RECOVERY - puppet last run on mw1248 is OK: OK: Puppet is currently enabled, last run 43 seconds ago with 0 failures [10:46:12] RECOVERY - puppet last run on wtp1020 is OK: OK: Puppet is currently enabled, last run 42 seconds ago with 0 failures [10:46:13] RECOVERY - puppet last run on mw1305 is OK: OK: Puppet is currently enabled, last run 44 seconds ago with 0 failures [10:46:21] RECOVERY - puppet last run on mendelevium is OK: OK: Puppet is currently enabled, last run 35 seconds ago with 0 failures [10:46:21] RECOVERY - puppet last run on wtp1035 is OK: OK: Puppet is currently enabled, last run 38 seconds ago with 0 failures [10:46:22] RECOVERY - puppet last run on db1083 is OK: OK: Puppet is currently enabled, last run 17 seconds ago with 0 failures [10:46:31] RECOVERY - puppet last run on dbstore1001 is OK: OK: Puppet is currently enabled, last run 53 seconds ago with 0 failures [10:46:31] RECOVERY - puppet last run on cp3005 is OK: OK: Puppet is currently enabled, last run 6 seconds ago with 0 failures [10:46:32] RECOVERY - puppet last run on wtp1009 is OK: OK: Puppet is currently enabled, last run 19 seconds ago with 0 failures [10:46:41] RECOVERY - puppet last run on cp1047 is OK: OK: Puppet is currently enabled, last run 10 seconds ago with 0 failures [10:46:41] RECOVERY - puppet last run on mw1277 is OK: OK: Puppet is currently enabled, last run 3 seconds ago with 0 failures [10:46:41] RECOVERY - puppet last run on ganeti1002 is OK: OK: Puppet is currently enabled, last run 0 seconds ago with 0 failures [10:46:51] RECOVERY - puppet last run on mw1163 is OK: OK: Puppet is currently enabled, last run 45 seconds ago with 0 failures [10:46:53] RECOVERY - puppet last run on dbproxy1008 is OK: OK: Puppet is currently enabled, last run 50 seconds ago with 0 failures [10:46:53] RECOVERY - puppet last run on cp3042 is OK: OK: Puppet is currently enabled, last run 7 seconds ago with 0 failures [10:46:53] RECOVERY - puppet last run on netmon2001 is OK: OK: Puppet is currently enabled, last run 50 seconds ago with 0 failures [10:47:01] RECOVERY - puppet last run on labvirt1003 is OK: OK: Puppet is currently enabled, last run 33 seconds ago with 0 failures [10:47:01] RECOVERY - puppet last run on mw1214 is OK: OK: Puppet is currently enabled, last run 16 seconds ago with 0 failures [10:47:02] RECOVERY - puppet last run on mw1243 is OK: OK: Puppet is currently enabled, last run 20 seconds ago with 0 failures [10:47:11] RECOVERY - puppet last run on elastic1033 is OK: OK: Puppet is currently enabled, last run 31 seconds ago with 0 failures [10:47:11] RECOVERY - puppet last run on db1080 is OK: OK: Puppet is currently enabled, last run 7 seconds ago with 0 failures [10:47:11] RECOVERY - puppet last run on mw1183 is OK: OK: Puppet is currently enabled, last run 58 seconds ago with 0 failures [10:47:21] RECOVERY - puppet last run on mw1164 is OK: OK: Puppet is currently enabled, last run 7 seconds ago with 0 failures [10:47:21] RECOVERY - puppet last run on ms-be1036 is OK: OK: Puppet is currently enabled, last run 53 seconds ago with 0 failures [10:47:31] RECOVERY - puppet last run on mw1205 is OK: OK: Puppet is currently enabled, last run 3 seconds ago with 0 failures [10:47:31] RECOVERY - puppet last run on stat1003 is OK: OK: Puppet is currently enabled, last run 28 seconds ago with 0 failures [10:47:31] RECOVERY - puppet last run on labstore1005 is OK: OK: Puppet is currently enabled, last run 0 seconds ago with 0 failures [10:47:31] RECOVERY - puppet last run on db1067 is OK: OK: Puppet is currently enabled, last run 19 seconds ago with 0 failures [10:47:32] RECOVERY - puppet last run on db1049 is OK: OK: Puppet is currently enabled, last run 46 seconds ago with 0 failures [10:47:32] RECOVERY - puppet last run on nescio is OK: OK: Puppet is currently enabled, last run 24 seconds ago with 0 failures [10:47:41] RECOVERY - puppet last run on wtp1039 is OK: OK: Puppet is currently enabled, last run 43 seconds ago with 0 failures [10:47:41] RECOVERY - puppet last run on cp3038 is OK: OK: Puppet is currently enabled, last run 27 seconds ago with 0 failures [10:47:41] RECOVERY - puppet last run on lvs3003 is OK: OK: Puppet is currently enabled, last run 23 seconds ago with 0 failures [10:47:41] RECOVERY - puppet last run on ms-be1021 is OK: OK: Puppet is currently enabled, last run 13 seconds ago with 0 failures [10:47:41] RECOVERY - puppet last run on mc1021 is OK: OK: Puppet is currently enabled, last run 19 seconds ago with 0 failures [10:47:51] RECOVERY - puppet last run on cp3007 is OK: OK: Puppet is currently enabled, last run 24 seconds ago with 0 failures [10:48:01] RECOVERY - puppet last run on snapshot1007 is OK: OK: Puppet is currently enabled, last run 7 seconds ago with 0 failures [10:48:02] RECOVERY - puppet last run on labsdb1010 is OK: OK: Puppet is currently enabled, last run 10 seconds ago with 0 failures [10:48:11] RECOVERY - puppet last run on ms-be1023 is OK: OK: Puppet is currently enabled, last run 27 seconds ago with 0 failures [10:48:12] RECOVERY - puppet last run on analytics1037 is OK: OK: Puppet is currently enabled, last run 7 seconds ago with 0 failures [10:48:21] RECOVERY - puppet last run on wtp1021 is OK: OK: Puppet is currently enabled, last run 30 seconds ago with 0 failures [10:48:21] RECOVERY - puppet last run on elastic1019 is OK: OK: Puppet is currently enabled, last run 46 seconds ago with 0 failures [10:48:21] RECOVERY - puppet last run on restbase1014 is OK: OK: Puppet is currently enabled, last run 55 seconds ago with 0 failures [10:48:31] RECOVERY - puppet last run on mc1030 is OK: OK: Puppet is currently enabled, last run 10 seconds ago with 0 failures [10:48:31] RECOVERY - puppet last run on restbase1017 is OK: OK: Puppet is currently enabled, last run 53 seconds ago with 0 failures [10:48:31] RECOVERY - puppet last run on mw1274 is OK: OK: Puppet is currently enabled, last run 44 seconds ago with 0 failures [10:48:31] RECOVERY - puppet last run on mw1186 is OK: OK: Puppet is currently enabled, last run 43 seconds ago with 0 failures [10:48:32] RECOVERY - puppet last run on analytics1054 is OK: OK: Puppet is currently enabled, last run 51 seconds ago with 0 failures [10:48:32] RECOVERY - puppet last run on kubestagetcd1001 is OK: OK: Puppet is currently enabled, last run 17 seconds ago with 0 failures [10:48:32] RECOVERY - puppet last run on labvirt1008 is OK: OK: Puppet is currently enabled, last run 35 seconds ago with 0 failures [10:48:41] RECOVERY - puppet last run on rcs1002 is OK: OK: Puppet is currently enabled, last run 45 seconds ago with 0 failures [10:48:51] RECOVERY - puppet last run on mw1203 is OK: OK: Puppet is currently enabled, last run 35 seconds ago with 0 failures [10:48:52] RECOVERY - puppet last run on prometheus2004 is OK: OK: Puppet is currently enabled, last run 22 seconds ago with 0 failures [10:48:52] RECOVERY - puppet last run on ores1008 is OK: OK: Puppet is currently enabled, last run 50 seconds ago with 0 failures [10:49:11] RECOVERY - puppet last run on mw1221 is OK: OK: Puppet is currently enabled, last run 25 seconds ago with 0 failures [10:49:11] RECOVERY - puppet last run on mw1249 is OK: OK: Puppet is currently enabled, last run 28 seconds ago with 0 failures [10:49:12] RECOVERY - puppet last run on analytics1049 is OK: OK: Puppet is currently enabled, last run 15 seconds ago with 0 failures [10:49:21] RECOVERY - puppet last run on mw1264 is OK: OK: Puppet is currently enabled, last run 43 seconds ago with 0 failures [10:49:31] RECOVERY - puppet last run on mc1032 is OK: OK: Puppet is currently enabled, last run 4 seconds ago with 0 failures [10:49:31] RECOVERY - puppet last run on mw1257 is OK: OK: Puppet is currently enabled, last run 49 seconds ago with 0 failures [10:49:31] RECOVERY - puppet last run on analytics1029 is OK: OK: Puppet is currently enabled, last run 35 seconds ago with 0 failures [10:49:32] RECOVERY - puppet last run on db1098 is OK: OK: Puppet is currently enabled, last run 29 seconds ago with 0 failures [10:49:41] RECOVERY - puppet last run on aluminium is OK: OK: Puppet is currently enabled, last run 53 seconds ago with 0 failures [10:49:42] RECOVERY - puppet last run on labtestservices2002 is OK: OK: Puppet is currently enabled, last run 53 seconds ago with 0 failures [10:49:42] RECOVERY - puppet last run on es1017 is OK: OK: Puppet is currently enabled, last run 31 seconds ago with 0 failures [10:49:51] RECOVERY - puppet last run on mc1024 is OK: OK: Puppet is currently enabled, last run 42 seconds ago with 0 failures [10:49:51] RECOVERY - puppet last run on db1104 is OK: OK: Puppet is currently enabled, last run 52 seconds ago with 0 failures [10:50:21] RECOVERY - puppet last run on lvs3004 is OK: OK: Puppet is currently enabled, last run 53 seconds ago with 0 failures [11:17:31] RECOVERY - puppet last run on cp1068 is OK: OK: Puppet is currently enabled, last run 57 seconds ago with 0 failures [14:36:39] (03PS1) 10Reedy: Add din to InterwikiSortOrders [mediawiki-config] - 10https://gerrit.wikimedia.org/r/365451 (https://phabricator.wikimedia.org/T168518) [14:41:54] (03CR) 10Amire80: "The native name is "Thuɔŋjäŋ". In this language "Th" is considered a single letter coming after T. The creator of the wiki asked to put it" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/365451 (https://phabricator.wikimedia.org/T168518) (owner: 10Reedy) [14:42:52] (03CR) 10Reedy: "Unfortunately, this whole thing is mostly undocumented black magic" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/365451 (https://phabricator.wikimedia.org/T168518) (owner: 10Reedy) [14:46:21] (03CR) 10Amire80: "Totally, but let's make it simple: In arrays where "fi" (Finnish, native name Suomi) is in the area of "S" rather than "F", can "din" be t" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/365451 (https://phabricator.wikimedia.org/T168518) (owner: 10Reedy) [15:58:13] (03CR) 10Reedy: "Feel free to amend :)" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/365451 (https://phabricator.wikimedia.org/T168518) (owner: 10Reedy) [17:26:16] (03CR) 10Krinkle: "RE: puppet 3 - do we not run the same version in Beta?" [puppet] - 10https://gerrit.wikimedia.org/r/350966 (https://phabricator.wikimedia.org/T113114) (owner: 10Krinkle) [20:17:42] RECOVERY - Make sure enwiki dumps are not empty on checker.tools.wmflabs.org is OK: HTTP OK: HTTP/1.1 200 OK - 166 bytes in 0.012 second response time [22:32:40] robh: on duty? [22:35:06] Doesn't mean he's here on a weekend :P [22:36:09] well, i need someone with access to exim config [22:36:23] as the new wave of spam is raising