[00:02:51] Broly: are you trying to send indivually or both at the same time/have others CCed in? [00:03:58] tried both and it fails [00:22:28] PROBLEM - puppet last run on kafka1013 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [00:51:28] RECOVERY - puppet last run on kafka1013 is OK: OK: Puppet is currently enabled, last run 59 seconds ago with 0 failures [01:02:31] (03PS6) 10Paladox: Gerrit: Convert from utf8 to utf8mb4 [puppet] - 10https://gerrit.wikimedia.org/r/328571 (https://phabricator.wikimedia.org/T153899) [01:03:25] (03PS1) 10BryanDavis: vagrant: remove setup.sh call [puppet] - 10https://gerrit.wikimedia.org/r/329723 [01:03:27] (03PS1) 10BryanDavis: vagrant: add sudo rules for Vagrant 1.9.1 [puppet] - 10https://gerrit.wikimedia.org/r/329724 (https://phabricator.wikimedia.org/T122735) [01:04:45] (03PS3) 10Paladox: Gerrit: Enable config localUsernameToLowerCase [puppet] - 10https://gerrit.wikimedia.org/r/326150 (https://phabricator.wikimedia.org/T152640) [01:05:23] (03PS20) 10Paladox: Gerrit: Enable logstash in gerrit [puppet] - 10https://gerrit.wikimedia.org/r/326177 (https://phabricator.wikimedia.org/T141324) [01:05:36] (03PS4) 10Paladox: Wikitech: Switching from using SemanticForms to PageForms extension [mediawiki-config] - 10https://gerrit.wikimedia.org/r/319131 (https://phabricator.wikimedia.org/T149749) [01:09:03] Broly: does your email service and application try to automatically cc yourself onto the email? [01:09:22] nope [01:12:01] I would email it to -owner in that case and explain you are having issues [01:12:19] https://github.com/ghoneycutt/wikimedia-puppet/blob/master/templates/exim/system_filter.conf.erb seems to be where it's from. i don't know if the parser has arbcom-l clerks-l added to lists it shouldn't [01:15:52] (03PS19) 10Paladox: Add support for searching gerrit using bug:T1 [puppet] - 10https://gerrit.wikimedia.org/r/308753 (https://phabricator.wikimedia.org/T85002) [01:16:18] PROBLEM - puppet last run on db1066 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [01:20:27] Broly: it should, They are both private lists, which contents shouldn't ever be crossposted to the public lists [01:21:03] i'm sure the users on those lists are private yeah [01:21:23] but arbcom-l itself shouldn't be private, because then the public can't contact them (isn't that what's happening here?) [01:22:10] i think the code resolves the users on the list, which should be private, but it shouldn't have arbcom-l and clerks-l on that list with the private members (i think this would be an error someone would get trying to email arbcom-l members from the 'outside') [01:23:03] it stops people emailing arbcom and a public list such as wikimania-l to prevent leaks of confidential data [01:23:27] the wikipedia site says to mail those addresses, though [01:23:37] so they shouldn't be on that list, as it states the email would be an alternative to posting [01:23:45] so it should be open to receive emails from the public [01:24:04] the recipients that refers to are the email addresses for the mailing lists, not the indivual members of the list [01:24:18] Broly: they are open for people to email to [01:24:37] but that error i'm getting says it's rejected because the list has both public and private [01:24:39] and it shouldn't [01:24:47] arbcom-l should be public [01:25:27] whatever list it's resolving for incoming public emails (i.e. outside the domain), shouldn't have anything more than the *-l i think, and that's why ti's getting rejected. i'm only mailing the *-l [01:25:51] but i think ti's adding more than the -l (private users) and thus rejecting it [01:26:09] I just emailed it, and its working (i'm not a member) [01:26:26] i tried both a hotmail and other domain, both gave errors [01:26:50] let me try again [01:28:39] ahh someone got back to me. they said i can't mail both lists at once. i guess that's what you were saying. is there a timestamp issue that arises with sending both in the 'to' field? [01:28:43] i didn't use CC [01:31:06] https://lists.wikimedia.org/mailman/listinfo/arbcom-l claims that arbcom-l is private [01:31:15] https://hostr.co/file/970/QQpoHwos0Vka/arbcom1b.PNG https://hostr.co/file/970/dcW5lNpaoUb3/arbcom2.PNG [01:32:27] i don't know. i sent it to owner as p858snake suggested and that worked better, as mr kevin li forwarded it to the arbcom list [01:32:40] would rather not fill up their inbox with more clutter as a test [01:32:52] he said you have to send it to one list at a time [01:44:18] RECOVERY - puppet last run on db1066 is OK: OK: Puppet is currently enabled, last run 16 seconds ago with 0 failures [02:24:28] PROBLEM - puppet last run on sca1003 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [02:28:54] 06Operations, 10MediaWiki-Export-or-Import, 10Wikimedia-General-or-Unknown: Special:Import error: "Import failed: Could not open import file" - https://phabricator.wikimedia.org/T17000#2908075 (10TTO) >>! In T17000#2907375, @Joe wrote: > But I have a more fundamental question: why do you all think this issu... [02:39:20] 06Operations, 10MediaWiki-Export-or-Import, 10Wikimedia-General-or-Unknown: Special:Import error: "Import failed: Could not open import file" - https://phabricator.wikimedia.org/T17000#191958 (10Peachey88) Don't we already have maintenance scripts that allow these large ones to be done server side? [02:52:28] RECOVERY - puppet last run on sca1003 is OK: OK: Puppet is currently enabled, last run 41 seconds ago with 0 failures [03:05:18] PROBLEM - Postgres Replication Lag on maps1003 is CRITICAL: CRITICAL - Rep Delay is: 1800.515395 Seconds [03:06:18] RECOVERY - Postgres Replication Lag on maps1003 is OK: OK - Rep Delay is: 0.0 Seconds [03:22:38] PROBLEM - MariaDB Slave Lag: s1 on dbstore1002 is CRITICAL: CRITICAL slave_sql_lag Replication lag: 603.80 seconds [03:28:38] RECOVERY - MariaDB Slave Lag: s1 on dbstore1002 is OK: OK slave_sql_lag Replication lag: 298.78 seconds [03:49:58] PROBLEM - citoid endpoints health on scb2003 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [03:49:58] PROBLEM - citoid endpoints health on scb2004 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [03:49:58] PROBLEM - citoid endpoints health on scb2002 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [03:50:48] RECOVERY - citoid endpoints health on scb2003 is OK: All endpoints are healthy [03:50:48] RECOVERY - citoid endpoints health on scb2004 is OK: All endpoints are healthy [03:50:48] RECOVERY - citoid endpoints health on scb2002 is OK: All endpoints are healthy [04:07:58] PROBLEM - mailman I/O stats on fermium is CRITICAL: CRITICAL - I/O stats: Transfers/Sec=4684.60 Read Requests/Sec=2813.00 Write Requests/Sec=0.60 KBytes Read/Sec=32482.80 KBytes_Written/Sec=27.20 [04:16:58] RECOVERY - mailman I/O stats on fermium is OK: OK - I/O stats: Transfers/Sec=0.70 Read Requests/Sec=0.20 Write Requests/Sec=0.50 KBytes Read/Sec=0.80 KBytes_Written/Sec=18.40 [04:54:31] (03PS5) 10BryanDavis: [WIP] Provision MediaWiki-Vagrant on Jessie hosts [puppet] - 10https://gerrit.wikimedia.org/r/245920 [05:01:38] PROBLEM - puppet last run on sca2003 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [05:04:18] PROBLEM - puppet last run on labnodepool1001 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [05:30:38] RECOVERY - puppet last run on sca2003 is OK: OK: Puppet is currently enabled, last run 59 seconds ago with 0 failures [05:32:18] RECOVERY - puppet last run on labnodepool1001 is OK: OK: Puppet is currently enabled, last run 32 seconds ago with 0 failures [05:46:18] PROBLEM - puppet last run on db1066 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [06:15:18] RECOVERY - puppet last run on db1066 is OK: OK: Puppet is currently enabled, last run 46 seconds ago with 0 failures [06:22:20] 06Operations, 10MediaWiki-Export-or-Import, 10Wikimedia-General-or-Unknown: Special:Import error: "Import failed: Could not open import file" - https://phabricator.wikimedia.org/T17000#2908189 (10TTO) >>! In T17000#2908079, @Peachey88 wrote: > Don't we already have maintenance scripts that allow these large... [06:37:38] PROBLEM - Check HHVM threads for leakage on mw1168 is CRITICAL: CRITICAL: HHVM has more than double threads running or queued than apache has busy workers [06:45:18] PROBLEM - Check HHVM threads for leakage on mw1260 is CRITICAL: CRITICAL: HHVM has more than double threads running or queued than apache has busy workers [06:48:18] PROBLEM - Check HHVM threads for leakage on mw1169 is CRITICAL: CRITICAL: HHVM has more than double threads running or queued than apache has busy workers [06:56:18] PROBLEM - Check HHVM threads for leakage on mw1259 is CRITICAL: CRITICAL: HHVM has more than double threads running or queued than apache has busy workers [07:40:01] (03PS1) 10Tim Landscheidt: docker: Indent @ssl_settings in NGINX configuration [puppet] - 10https://gerrit.wikimedia.org/r/329735 [07:40:26] (03PS1) 10Tim Landscheidt: dumps: Indent @ssl_settings in NGINX configurations [puppet] - 10https://gerrit.wikimedia.org/r/329736 [07:41:25] (03PS1) 10Tim Landscheidt: ganglia: Indent @ssl_settings in Apache configuration [puppet] - 10https://gerrit.wikimedia.org/r/329737 [07:41:48] (03PS1) 10Tim Landscheidt: gerrit: Indent @ssl_settings in Apache configuration [puppet] - 10https://gerrit.wikimedia.org/r/329738 [07:42:11] (03PS1) 10Tim Landscheidt: icinga: Indent @ssl_settings in Apache configuration [puppet] - 10https://gerrit.wikimedia.org/r/329739 [07:42:32] (03PS1) 10Tim Landscheidt: install_server: Indent @ssl_settings in NGINX configuration [puppet] - 10https://gerrit.wikimedia.org/r/329740 [07:42:51] (03PS1) 10Tim Landscheidt: librenms: Indent @ssl_settings in Apache configuration [puppet] - 10https://gerrit.wikimedia.org/r/329741 [07:43:13] (03PS1) 10Tim Landscheidt: mailman: Indent @ssl_settings in Apache configuration [puppet] - 10https://gerrit.wikimedia.org/r/329742 [07:43:30] (03PS1) 10Tim Landscheidt: mirrors: Indent @ssl_settings in NGINX configuration [puppet] - 10https://gerrit.wikimedia.org/r/329743 [07:43:50] (03PS1) 10Tim Landscheidt: openstack: Indent @ssl_settings in Apache configuration [puppet] - 10https://gerrit.wikimedia.org/r/329744 [07:44:23] (03PS1) 10Tim Landscheidt: puppetmaster: Indent @ssl_settings in Apache and NGINX configurations [puppet] - 10https://gerrit.wikimedia.org/r/329745 [07:44:51] (03PS1) 10Tim Landscheidt: requesttracker: Indent @ssl_settings in Apache configuration [puppet] - 10https://gerrit.wikimedia.org/r/329746 [07:45:10] (03PS1) 10Tim Landscheidt: tendril: Indent @ssl_settings in Apache configuration [puppet] - 10https://gerrit.wikimedia.org/r/329747 [07:45:35] (03PS1) 10Tim Landscheidt: toolserver_legacy: Indent @ssl_settings in Apache configuration [puppet] - 10https://gerrit.wikimedia.org/r/329748 [07:47:02] (03PS1) 10Tim Landscheidt: Tools: Indent @ssl_settings in NGINX configuration [puppet] - 10https://gerrit.wikimedia.org/r/329749 [07:48:02] (03PS1) 10Tim Landscheidt: dynamicproxy: Indent @ssl_settings in NGINX configurations [puppet] - 10https://gerrit.wikimedia.org/r/329750 [07:49:18] RECOVERY - OCG health on ocg1002 is OK: OK: ocg_job_status 791864 msg: ocg_render_job_queue 0 msg [07:49:18] RECOVERY - OCG health on ocg1003 is OK: OK: ocg_job_status 791818 msg: ocg_render_job_queue 0 msg [07:49:38] RECOVERY - OCG health on ocg1001 is OK: OK: ocg_job_status 787542 msg: ocg_render_job_queue 0 msg [07:50:38] PROBLEM - puppet last run on lvs3004 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [08:18:58] RECOVERY - puppet last run on lvs3004 is OK: OK: Puppet is currently enabled, last run 15 seconds ago with 0 failures [08:38:38] PROBLEM - puppet last run on tin is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [08:42:28] RECOVERY - Check HHVM threads for leakage on mw1259 is OK: OK [09:06:38] RECOVERY - puppet last run on tin is OK: OK: Puppet is currently enabled, last run 9 seconds ago with 0 failures [09:21:38] PROBLEM - puppet last run on db1024 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [09:36:38] RECOVERY - Check HHVM threads for leakage on mw1169 is OK: OK [09:44:38] RECOVERY - Check HHVM threads for leakage on mw1260 is OK: OK [09:49:38] RECOVERY - puppet last run on db1024 is OK: OK: Puppet is currently enabled, last run 4 seconds ago with 0 failures [10:06:38] RECOVERY - Check HHVM threads for leakage on mw1168 is OK: OK [10:29:28] PROBLEM - puppet last run on stat1003 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [10:58:28] RECOVERY - puppet last run on stat1003 is OK: OK: Puppet is currently enabled, last run 47 seconds ago with 0 failures [10:58:28] PROBLEM - puppet last run on elastic1039 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [11:10:48] PROBLEM - puppet last run on sca2004 is CRITICAL: CRITICAL: Puppet has 27 failures. Last run 2 minutes ago with 27 failures. Failed resources (up to 3 shown): Exec[eth0_v6_token],Package[wipe],Package[zotero/translators],Package[zotero/translation-server] [11:27:28] (03Draft2) 10Matěj Suchánek: Update Wikidata property blacklist [mediawiki-config] - 10https://gerrit.wikimedia.org/r/329762 [11:27:28] RECOVERY - puppet last run on elastic1039 is OK: OK: Puppet is currently enabled, last run 42 seconds ago with 0 failures [11:29:30] (03CR) 10Matěj Suchánek: Update Wikidata property blacklist (031 comment) [mediawiki-config] - 10https://gerrit.wikimedia.org/r/329762 (owner: 10Matěj Suchánek) [11:37:38] RECOVERY - puppet last run on sca2004 is OK: OK: Puppet is currently enabled, last run 36 seconds ago with 0 failures [11:43:58] PROBLEM - puppet last run on cp3034 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [12:11:58] RECOVERY - puppet last run on cp3034 is OK: OK: Puppet is currently enabled, last run 13 seconds ago with 0 failures [12:29:48] PROBLEM - puppet last run on bast4001 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [12:35:38] PROBLEM - puppet last run on db1061 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [12:58:48] RECOVERY - puppet last run on bast4001 is OK: OK: Puppet is currently enabled, last run 4 seconds ago with 0 failures [13:03:38] RECOVERY - puppet last run on db1061 is OK: OK: Puppet is currently enabled, last run 26 seconds ago with 0 failures [13:09:58] (03PS2) 10MarcoAurelio: Set $wgAbuseFilterNotificationsPrivate = true; for Meta-Wiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/329600 (https://phabricator.wikimedia.org/T154358) [13:39:48] PROBLEM - puppet last run on sca2004 is CRITICAL: CRITICAL: Puppet has 27 failures. Last run 2 minutes ago with 27 failures. Failed resources (up to 3 shown): Exec[eth0_v6_token],Package[wipe],Package[zotero/translators],Package[zotero/translation-server] [14:01:30] (03PS1) 10Alexandros Kosiaris: Disable ntp across the fleet for leap second [puppet] - 10https://gerrit.wikimedia.org/r/329780 [14:04:09] (03CR) 10Alexandros Kosiaris: [C: 032] Disable ntp across the fleet for leap second [puppet] - 10https://gerrit.wikimedia.org/r/329780 (owner: 10Alexandros Kosiaris) [14:07:35] (03PS1) 10Alexandros Kosiaris: Disable NTP checks during the leap second [puppet] - 10https://gerrit.wikimedia.org/r/329782 [14:07:45] (03CR) 10Alexandros Kosiaris: [V: 032 C: 032] Disable NTP checks during the leap second [puppet] - 10https://gerrit.wikimedia.org/r/329782 (owner: 10Alexandros Kosiaris) [14:07:48] RECOVERY - puppet last run on sca2004 is OK: OK: Puppet is currently enabled, last run 48 seconds ago with 0 failures [14:08:43] !log disable ircecho for a while to avoid NTP alert storm [14:08:46] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [14:32:08] !log reenable puppet and ircecho on einsteinium. NTP disabling has propagated throughout the fleet [14:32:11] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [14:32:50] PROBLEM - puppet last run on sca2003 is CRITICAL: CRITICAL: Puppet has 27 failures. Last run 2 minutes ago with 27 failures. Failed resources (up to 3 shown): Exec[eth0_v6_token],Package[wipe],Package[zotero/translators],Package[zotero/translation-server] [14:33:40] PROBLEM - NTP peers on nescio is CRITICAL: NTP CRITICAL: No response from NTP server [14:40:19] (03PS1) 10Alexandros Kosiaris: nescio and maerlant are NTP servers too [puppet] - 10https://gerrit.wikimedia.org/r/329784 [14:40:38] (03CR) 10Alexandros Kosiaris: [V: 032 C: 032] nescio and maerlant are NTP servers too [puppet] - 10https://gerrit.wikimedia.org/r/329784 (owner: 10Alexandros Kosiaris) [14:45:38] RECOVERY - NTP peers on maerlant is OK: NTP OK: Offset -0.000696 secs [14:45:38] RECOVERY - NTP peers on nescio is OK: NTP OK: Offset 0.000315 secs [14:46:59] (03PS1) 10Alexandros Kosiaris: Revert NTP disabling for leap second [puppet] - 10https://gerrit.wikimedia.org/r/329785 [14:56:28] PROBLEM - puppet last run on elastic1020 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [14:58:58] RECOVERY - puppet last run on sca2003 is OK: OK: Puppet is currently enabled, last run 16 seconds ago with 0 failures [15:05:28] PROBLEM - puppet last run on sca1004 is CRITICAL: CRITICAL: Puppet has 27 failures. Last run 2 minutes ago with 27 failures. Failed resources (up to 3 shown): Exec[eth0_v6_token],Package[wipe],Package[zotero/translators],Package[zotero/translation-server] [15:24:28] RECOVERY - puppet last run on elastic1020 is OK: OK: Puppet is currently enabled, last run 45 seconds ago with 0 failures [15:32:38] RECOVERY - puppet last run on sca1004 is OK: OK: Puppet is currently enabled, last run 20 seconds ago with 0 failures [15:33:50] <_joe_> akosiaris: we did disable it after all [15:56:47] 06Operations, 10DBA, 06Labs, 07Tracking: Database replication problems - production and labs (tracking) - https://phabricator.wikimedia.org/T50930#2908670 (10AlexMonk-WMF) [16:24:28] PROBLEM - puppet last run on sca1003 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [16:52:38] RECOVERY - puppet last run on sca1003 is OK: OK: Puppet is currently enabled, last run 41 seconds ago with 0 failures [17:01:48] PROBLEM - puppet last run on db1041 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [17:03:29] (03PS4) 10Tim Landscheidt: Tools: Outfactor the configuration for outgoing HBA connections [puppet] - 10https://gerrit.wikimedia.org/r/267832 [17:09:00] (03PS6) 10Tim Landscheidt: Tools: Fix argument quoting in jlocal [puppet] - 10https://gerrit.wikimedia.org/r/266935 [17:29:48] RECOVERY - puppet last run on db1041 is OK: OK: Puppet is currently enabled, last run 46 seconds ago with 0 failures [17:39:18] (03CR) 10Tim Landscheidt: [C: 031] puppetmaster: Cleanup unused vars / crons in labs puppetmaster [puppet] - 10https://gerrit.wikimedia.org/r/312317 (owner: 10Yuvipanda) [17:58:08] PROBLEM - puppet last run on cp3048 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [18:26:08] RECOVERY - puppet last run on cp3048 is OK: OK: Puppet is currently enabled, last run 7 seconds ago with 0 failures [18:33:38] PROBLEM - puppet last run on db1054 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [19:01:38] RECOVERY - puppet last run on db1054 is OK: OK: Puppet is currently enabled, last run 11 seconds ago with 0 failures [19:16:15] (03CR) 10Tim Landscheidt: [V: 031 C: 031] Add fake clushuser keypair [labs/private] - 10https://gerrit.wikimedia.org/r/325050 (owner: 10Merlijn van Deen) [19:23:32] (03PS3) 10Tim Landscheidt: Revert "tools: store verbose logrotate logs" [puppet] - 10https://gerrit.wikimedia.org/r/329217 (https://phabricator.wikimedia.org/T96007) [19:50:45] (03CR) 10Tim Landscheidt: [C: 04-1] "nova-common is (now) installed by the openstack module, so before this could get merged, it must be tested that either nova-common's sudo " [puppet] - 10https://gerrit.wikimedia.org/r/180513 (owner: 10Faidon Liambotis) [20:47:17] 06Operations, 10Gerrit, 06Release-Engineering-Team, 07Upstream: Gerrit: Restarting gerrit could lead to data loss + maybe accounts - https://phabricator.wikimedia.org/T154205#2909008 (10Paladox) Actually T153079 had nothing to do with the index but was the submodule that was the problem. But other users n... [22:04:38] PROBLEM - puppet last run on sca1004 is CRITICAL: CRITICAL: Puppet has 27 failures. Last run 2 minutes ago with 27 failures. Failed resources (up to 3 shown): Exec[eth0_v6_token],Package[wipe],Package[zotero/translators],Package[zotero/translation-server] [22:32:38] RECOVERY - puppet last run on sca1004 is OK: OK: Puppet is currently enabled, last run 50 seconds ago with 0 failures [22:55:48] PROBLEM - puppet last run on ms-be1003 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [23:15:38] PROBLEM - puppet last run on kafka1001 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [23:22:48] PROBLEM - puppet last run on ms-be1009 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [23:24:38] PROBLEM - puppet last run on sca1003 is CRITICAL: CRITICAL: Puppet has 27 failures. Last run 2 minutes ago with 27 failures. Failed resources (up to 3 shown): Exec[eth0_v6_token],Package[wipe],Package[zotero/translators],Package[zotero/translation-server] [23:24:48] RECOVERY - puppet last run on ms-be1003 is OK: OK: Puppet is currently enabled, last run 54 seconds ago with 0 failures [23:43:38] RECOVERY - puppet last run on kafka1001 is OK: OK: Puppet is currently enabled, last run 10 seconds ago with 0 failures [23:50:48] RECOVERY - puppet last run on ms-be1009 is OK: OK: Puppet is currently enabled, last run 2 seconds ago with 0 failures [23:52:38] RECOVERY - puppet last run on sca1003 is OK: OK: Puppet is currently enabled, last run 2 seconds ago with 0 failures