[00:01:01] <icinga-miraheze>	 PROBLEM - mw183 Current Load on mw183 is WARNING: LOAD WARNING - total load average: 21.36, 21.34, 18.74
[00:01:40] <MirahezeLSBot>	 !log [wwr@test151] starting deploy of {'config': True} to test151
[00:01:41] <MirahezeLSBot>	 !log [wwr@test151] finished deploy of {'config': True} to test151 - SUCCESS in 0s
[00:01:43] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[00:01:47] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[00:01:59] <MirahezeLSBot>	 !log [wwr@test151] sudo -u www-data php /srv/mediawiki/1.45/maintenance/run.php Bucket:SetupDBPermission --wiki=exttestwikibeta (END - exit=0)
[00:02:01] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[00:03:00] <icinga-miraheze>	 RECOVERY - mw183 Current Load on mw183 is OK: LOAD OK - total load average: 12.15, 17.80, 17.75
[00:06:20] <icinga-miraheze>	 [Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.wikitide.net/d/GtxbP1Xnk?from=1772492750000&orgId=1&to=1772496380035
[00:31:20] <icinga-miraheze>	 [Grafana] RESOLVED: PHP-FPM Worker Usage High https://grafana.wikitide.net/d/GtxbP1Xnk?from=1772492750000&orgId=1&to=1772497850000
[00:34:07] <MirahezeLSBot>	 !log [skye@test151] starting deploy of {'folders': '1.45/extensions/Bucket'} to test151
[00:34:08] <MirahezeLSBot>	 !log [skye@test151] finished deploy of {'folders': '1.45/extensions/Bucket'} to test151 - SUCCESS in 0s
[00:34:25] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[00:34:39] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[00:34:58] <Skye>	 lmfao i forgot the grant all part i am so smart
[00:36:26] <icinga-miraheze>	 PROBLEM - mw201 Current Load on mw201 is WARNING: LOAD WARNING - total load average: 19.85, 20.62, 20.21
[00:38:15] <MirahezeLSBot>	 !log [skye@test151] starting deploy of {'folders': '1.45/extensions/Bucket'} to test151
[00:38:16] <MirahezeLSBot>	 !log [skye@test151] finished deploy of {'folders': '1.45/extensions/Bucket'} to test151 - SUCCESS in 0s
[00:38:19] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[00:38:22] <icinga-miraheze>	 RECOVERY - mw201 Current Load on mw201 is OK: LOAD OK - total load average: 16.40, 19.26, 19.77
[00:38:30] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[00:38:34] <Skye>	 the grant part*
[00:44:12] <icinga-miraheze>	 PROBLEM - mw201 Current Load on mw201 is WARNING: LOAD WARNING - total load average: 21.61, 20.47, 20.11
[00:46:08] <icinga-miraheze>	 RECOVERY - mw201 Current Load on mw201 is OK: LOAD OK - total load average: 19.46, 20.15, 20.06
[00:48:17] <MirahezeLSBot>	 !log [skye@test151] starting deploy of {'folders': '1.45/extensions/Bucket'} to test151
[00:48:18] <MirahezeLSBot>	 !log [skye@test151] finished deploy of {'folders': '1.45/extensions/Bucket'} to test151 - SUCCESS in 0s
[00:48:21] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[00:48:25] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[00:50:01] <icinga-miraheze>	 PROBLEM - mw201 Current Load on mw201 is WARNING: LOAD WARNING - total load average: 22.81, 21.25, 20.48
[00:51:57] <icinga-miraheze>	 RECOVERY - mw201 Current Load on mw201 is OK: LOAD OK - total load average: 20.06, 20.09, 20.10
[01:04:39] <MacFanBot>	 [02puppet] 07Universal-Omega merged 07MacFan4000's pull request #4812: replace my keys with an ed25519 (07miraheze:03main...07MacFan4000:03patch-75) 13https://github.com/miraheze/puppet/pull/4812
[01:04:39] <MacFanBot>	 [02puppet] 07Universal-Omega pushed 1 new commit to 03main 13https://github.com/miraheze/puppet/commit/947cc3a87f36361ee921119fd74790efec5b34f0
[01:04:39] <MacFanBot>	 02puppet/03main 07MacFan4000 03947cc3a replace my keys with an ed25519 (#4812)
[01:05:30] <icinga-miraheze>	 PROBLEM - mw201 Current Load on mw201 is WARNING: LOAD WARNING - total load average: 22.49, 21.56, 20.54
[01:07:35] <MirahezeLSBot>	 !log [wwr@test151] sudo -u www-data php /srv/mediawiki/1.45/maintenance/run.php Bucket:SetupDBPermission --wiki=exttestwikibeta (END - exit=0)
[01:07:43] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[01:09:21] <icinga-miraheze>	 PROBLEM - mw201 Current Load on mw201 is CRITICAL: LOAD CRITICAL - total load average: 24.56, 22.46, 21.04
[01:11:17] <icinga-miraheze>	 RECOVERY - mw201 Current Load on mw201 is OK: LOAD OK - total load average: 16.61, 19.87, 20.25
[01:21:50] <icinga-miraheze>	 [Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.wikitide.net/d/GtxbP1Xnk?from=1772497280000&orgId=1&to=1772500910033
[01:40:25] <MirahezeLSBot>	 !log [wwr@test151] sudo -u www-data php /srv/mediawiki/1.45/maintenance/run.php Bucket:RefreshAllBuckets --wiki=exttestwikibeta (END - exit=0)
[01:40:40] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[01:56:50] <icinga-miraheze>	 [Grafana] RESOLVED: PHP-FPM Worker Usage High https://grafana.wikitide.net/d/GtxbP1Xnk?from=1772499140000&orgId=1&to=1772502740000
[02:22:53] <MacFanBot>	 [02ssl] 07pskyechology pushed 1 new commit to 03main 13https://github.com/miraheze/ssl/commit/84783c1a03614afbd3dee60c231d887f84f956fd
[02:22:54] <MacFanBot>	 02ssl/03main 07Skye 0384783c1 T15044: Redirect kenpendium.miraheze.org to kenpendium.org
[10:30:42] <MirahezeLSBot>	 !log [wwr@test151] starting deploy of {'pull': 'config', 'config': True, 'l10n': True, 'extension_list': True, 'force_upgrade': True, 'versions': '1.45', 'upgrade_extensions': 'Bucket'} to test151
[10:30:43] <MirahezeLSBot>	 !log [wwr@test151] DEPLOY ABORTED: Non-Zero Exit Code in prep, see output.
[10:30:45] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[10:30:47] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[10:30:58] <MirahezeLSBot>	 !log [wwr@test151] starting deploy of {'l10n': True, 'extension_list': True, 'force_upgrade': True, 'versions': '1.45', 'upgrade_extensions': 'Bucket'} to test151
[10:31:00] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[10:31:51] <MirahezeLSBot>	 !log [wwr@test151] finished deploy of {'l10n': True, 'extension_list': True, 'force_upgrade': True, 'versions': '1.45', 'upgrade_extensions': 'Bucket'} to test151 - SUCCESS in 53s
[10:31:54] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[10:31:57] <MirahezeLSBot>	 !log [wwr@test151] starting deploy of {'config': True} to test151
[10:31:58] <MirahezeLSBot>	 !log [wwr@test151] finished deploy of {'config': True} to test151 - SUCCESS in 0s
[10:31:59] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[10:32:01] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[10:48:01] <MirahezeLSBot>	 !log [wwr@test151] starting deploy of {'config': True} to test151
[10:48:02] <MirahezeLSBot>	 !log [wwr@test151] finished deploy of {'config': True} to test151 - SUCCESS in 0s
[10:48:04] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[10:48:06] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[11:08:21] <MirahezeRelay>	 <thewwrnerdguy, replying to MirahezeLSBot> okay I think that's all the random things I can think of, still no clue why Bucket refuses to update anything except the one time it did update something
[11:34:41] <MirahezeRelay>	 <posix_memalign, replying to thewwrnerdguy> [1/2] Seems to be working?
[11:34:41] <MirahezeRelay>	 <posix_memalign, replying to thewwrnerdguy> [2/2] https://cdn.discordapp.com/attachments/808001911868489748/1478354927150694453/image.png?ex=69a818d1&is=69a6c751&hm=178cd886143c457867e8e8f2e1c064acc822033d7af9808851b8e7e15e0a5e57&
[11:35:17] <MirahezeRelay>	 <posix_memalign> [1/2] `{{#invoke:Bucket/put|put|drops|item=Main Page|quantity=Test|rarity=Test}}`
[11:35:18] <MirahezeRelay>	 <posix_memalign> [2/2] Using https://exttest.mirabeta.org/wiki/Module:Bucket/put?oldid=2511
[11:36:32] <MirahezeRelay>	 <posix_memalign> [1/2] Updates more-or-less instantly
[11:36:32] <MirahezeRelay>	 <posix_memalign> [2/2] https://cdn.discordapp.com/attachments/808001911868489748/1478355392311463998/image.png?ex=69a81940&is=69a6c7c0&hm=0f1a1e442b934f34262fc49c39a57f7f5196be7b161de6471bcd25ec419c333b&
[11:39:44] <MirahezeRelay>	 <posix_memalign> [1/2] Though imo Bucket is a terrible extension to start your tech journey with. Pretty complicated to set up and usage is not straightfoward either. 
[11:39:45] <MirahezeRelay>	 <posix_memalign> [2/2] I avoided taking on that extension initially due to the amount of moving parts, so it's great to see you and Skye doing the difficult part of the work.
[11:41:55] <MirahezeRelay>	 <posix_memalign, replying to posix_memalign> Here's a fun idea: add more features to this so that we can basically use Bucket as if it is Cargo.
[11:51:13] <MirahezeRelay>	 <thewwrnerdguy, replying to posix_memalign> tf
[11:52:46] <MirahezeRelay>	 <thewwrnerdguy> https://cdn.discordapp.com/attachments/808001911868489748/1478359478624846040/image.png?ex=69a81d0e&is=69a6cb8e&hm=6d4e59ffb6952f2ecaf11b35d6e25a11afabf502272e22f16b0c510b0ca7bcf8&
[11:52:53] <MirahezeRelay>	 <thewwrnerdguy> seems to not be working in mainspace
[11:57:06] <MirahezeRelay>	 <thewwrnerdguy, replying to posix_memalign> yeah that's definitely true, but also it is the latest extension to be approved so I might as well start testing it
[11:57:58] <MirahezeRelay>	 <thewwrnerdguy, replying to thewwrnerdguy> also https://exttest.mirabeta.org/wiki/Bucket:Bucket_issues doesn't exist
[12:21:12] <MirahezeRelay>	 <posix_memalign, replying to thewwrnerdguy> Very weird. I think exttest had namespace issues from the start, so it might work if you use a different wiki.
[12:21:57] <MirahezeRelay>	 <thewwrnerdguy, replying to posix_memalign> [1/2] I was about to say this right as you said that lmao
[12:21:58] <MirahezeRelay>	 <thewwrnerdguy, replying to posix_memalign> [2/2] > for bucket it looks like the `BucketWriteEnabledNamespaces` config setting isn't getting properly respected. It's enabled for main namespace but doesn't appear to do anything there, and I tried disabling it for the user namespace and it still works fine there. You can go to any user page and you'll see "View buckets" in the sidebar
[12:38:32] <MirahezeRelay>	 <posix_memalign, replying to thewwrnerdguy> It's not configured correctly. Run `shell exttestwikibeta` on test151 and check the config.
[12:38:53] <MirahezeRelay>	 <posix_memalign> [1/14] ```
[12:38:53] <MirahezeRelay>	 <posix_memalign> [2/14] petramagna@test151 ~ [2]> shell exttestwikibeta
[12:38:53] <MirahezeRelay>	 <posix_memalign> [3/14] Will execute:
[12:38:54] <MirahezeRelay>	 <posix_memalign> [4/14] sudo -u www-data php /srv/mediawiki/1.45/maintenance/run.php shell --wiki=exttestwikibeta
[12:38:54] <MirahezeRelay>	 <posix_memalign> [5/14] > $wgBucketWriteEnabledNamespaces
[12:38:54] <MirahezeRelay>	 <posix_memalign> [6/14] = [
[12:38:55] <MirahezeRelay>	 <posix_memalign> [7/14]     0,
[12:38:55] <MirahezeRelay>	 <posix_memalign> [8/14]     1,
[12:38:55] <MirahezeRelay>	 <posix_memalign> [9/14]     4,
[12:38:56] <MirahezeRelay>	 <posix_memalign> [10/14]     6,
[12:38:56] <MirahezeRelay>	 <posix_memalign> [11/14]     12,
[12:38:57] <MirahezeRelay>	 <posix_memalign> [12/14]     14,
[12:38:57] <MirahezeRelay>	 <posix_memalign> [13/14]   ]
[12:38:58] <MirahezeRelay>	 <posix_memalign> [14/14] ```
[12:39:16] <MirahezeRelay>	 <posix_memalign> [1/22] On my local machine where this is functioning correctly.
[12:39:16] <MirahezeRelay>	 <posix_memalign> [2/22] ```
[12:39:16] <MirahezeRelay>	 <posix_memalign> [3/22] > $wgBucketWriteEnabledNamespaces
[12:39:17] <MirahezeRelay>	 <posix_memalign> [4/22] = [
[12:39:17] <MirahezeRelay>	 <posix_memalign> [5/22]     true,
[12:39:17] <MirahezeRelay>	 <posix_memalign> [6/22]     false,
[12:39:18] <MirahezeRelay>	 <posix_memalign> [7/22]     true,
[12:39:18] <MirahezeRelay>	 <posix_memalign> [8/22]     false,
[12:39:18] <MirahezeRelay>	 <posix_memalign> [9/22]     true,
[12:39:19] <MirahezeRelay>	 <posix_memalign> [10/22]     false,
[12:39:19] <MirahezeRelay>	 <posix_memalign> [11/22]     true,
[12:39:20] <MirahezeRelay>	 <posix_memalign> [12/22]     false,
[12:39:20] <MirahezeRelay>	 <posix_memalign> [13/22]     false,
[12:39:21] <MirahezeRelay>	 <posix_memalign> [14/22]     false,
[12:39:21] <MirahezeRelay>	 <posix_memalign> [15/22]     false,
[12:39:22] <MirahezeRelay>	 <posix_memalign> [16/22]     false,
[12:39:22] <MirahezeRelay>	 <posix_memalign> [17/22]     true,
[12:39:23] <MirahezeRelay>	 <posix_memalign> [18/22]     false,
[12:39:23] <MirahezeRelay>	 <posix_memalign> [19/22]     true,
[12:39:24] <MirahezeRelay>	 <posix_memalign> [20/22]     false,
[12:39:24] <MirahezeRelay>	 <posix_memalign> [21/22]   ]
[12:39:25] <MirahezeRelay>	 <posix_memalign> [22/22] ```
[12:39:59] <MirahezeRelay>	 <posix_memalign> It's supposed to be an associative array where each ns maps to a bool.
[12:40:25] <MirahezeRelay>	 <posix_memalign> It doesn't work for main namespace because `$wgBucketWriteEnabledNamespaces[NS_MAIN]` is set to 0 which is interpreted as false.
[12:44:10] <MirahezeRelay>	 <thewwrnerdguy> hmm what
[12:45:41] <MirahezeRelay>	 <posix_memalign> I think someone else in tech got tripped up by the same problem a few months ago. Just need to change how the namespace is configured in mw-config I think.
[12:50:31] <MirahezeLSBot>	 !log [wwr@test151] starting deploy of {'config': True} to test151
[12:50:32] <MirahezeLSBot>	 !log [wwr@test151] finished deploy of {'config': True} to test151 - SUCCESS in 0s
[12:50:33] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[12:50:35] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[12:50:57] <MirahezeRelay>	 <posix_memalign> Follow `wgVisualEditorAvailableNamespaces`'s format (if you haven't already)
[12:51:02] <MirahezeRelay>	 <posix_memalign> I think vestyle is the one to use
[12:53:21] <MirahezeLSBot>	 !log [wwr@test151] starting deploy of {'config': True} to test151
[12:53:22] <MirahezeLSBot>	 !log [wwr@test151] finished deploy of {'config': True} to test151 - SUCCESS in 0s
[12:53:23] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[12:53:25] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[12:55:00] <MirahezeRelay>	 <thewwrnerdguy> okay it works
[14:40:05] <icinga-miraheze>	 PROBLEM - mw201 Current Load on mw201 is WARNING: LOAD WARNING - total load average: 21.22, 14.76, 8.48
[14:42:04] <icinga-miraheze>	 RECOVERY - mw201 Current Load on mw201 is OK: LOAD OK - total load average: 15.01, 14.14, 8.99
[15:43:23] <MirahezeLSBot>	 !log [petramagna@mwtask171] sudo -u www-data php /srv/mediawiki/1.45/maintenance/run.php /srv/mediawiki/1.45/extensions/TitleKey/maintenance/rebuildTitleKeys.php --wiki=infalsuswiki (END - exit=0)
[15:43:25] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[15:43:41] <MirahezeLSBot>	 !log [petramagna@mwtask171] sudo -u www-data php /srv/mediawiki/1.45/maintenance/run.php /srv/mediawiki/1.45/extensions/TitleKey/maintenance/rebuildTitleKeys.php --wiki=arcaeawiki (END - exit=0)
[15:43:44] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[17:19:47] <MacFanBot>	 [02puppet] 07SomeMWDev commented on pull request #4535: Not needed right now and there are better solutions. 13https://github.com/miraheze/puppet/pull/4535#issuecomment-3992470453
[17:19:47] <MacFanBot>	 [02puppet] 07SomeMWDev closed pull request #4535: Allow adding security overrides to PrivateSettings.php (07miraheze:03main...07SomeMWDev:03mw-security-settings) 13https://github.com/miraheze/puppet/pull/4535
[17:26:11] <MacFanBot>	 [02puppet] 07Universal-Omega left a file comment in pull request #4800 0389669e1: Don't really need to use 2026 here, 2024 was only used above to rotate passwords using a different user to avoid downtime IIRC. Up to you though. If this is ready I can add password to private puppet and deploy this. 13https://github.com/miraheze/puppet/pull/4800#discussion_r2879571093
[17:32:24] <MacFanBot>	 [02puppet] 07Universal-Omega closed pull request #4590: matomo: Set enable_sql_optimize_queries to 0 (03main...03paladox-patch-3) 13https://github.com/miraheze/puppet/pull/4590
[17:32:26] <MacFanBot>	 [02puppet] 07Universal-Omega commented on pull request #4590: This was already done in #4697 13https://github.com/miraheze/puppet/pull/4590#issuecomment-3992542699
[17:32:29] <MacFanBot>	 [02puppet] 07Universal-Omega 04deleted 03paladox-patch-3 at 03286efc2 13https://api.github.com/repos/miraheze/puppet/commit/286efc2
[17:58:56] <MacFanBot>	 [02puppet] 07Universal-Omega created 03mwdeploy-cleanup (+1 new commit) 13https://github.com/miraheze/puppet/commit/09c8f4cf2286
[17:58:56] <MacFanBot>	 02puppet/03mwdeploy-cleanup 07CosmicAlpha 0309c8f4c mwdeploy: reduce duplication for upgrading extensions and skins…
[17:59:04] <MacFanBot>	 [02puppet] 07Universal-Omega opened pull request #4813: mwdeploy: reduce duplication for upgrading extensions and skins (03main...03mwdeploy-cleanup) 13https://github.com/miraheze/puppet/pull/4813
[18:00:33] <miraheze-github>	 miraheze/puppet - Universal-Omega the build passed.
[18:53:04] <icinga-miraheze>	 PROBLEM - mw153 Puppet on mw153 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_3d2png]
[18:55:33] <icinga-miraheze>	 PROBLEM - mw191 Puppet on mw191 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 4 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_femiwiki-deploy-1.45]
[18:55:54] <icinga-miraheze>	 PROBLEM - bots171 Puppet on bots171 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_Pywikibot-stable]
[18:59:57] <icinga-miraheze>	 PROBLEM - mw163 Puppet on mw163 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_3d2png]
[19:01:24] <icinga-miraheze>	 PROBLEM - puppet181 Check unit status of git_pull_puppet on puppet181 is CRITICAL: CRITICAL: Status of the systemd unit git_pull_puppet
[19:01:25] <icinga-miraheze>	 PROBLEM - puppet181 Check unit status of git_pull_ssl on puppet181 is CRITICAL: CRITICAL: Status of the systemd unit git_pull_ssl
[19:03:31] <icinga-miraheze>	 PROBLEM - puppet181 Check unit status of listdomains_github_push on puppet181 is CRITICAL: CRITICAL: Status of the systemd unit listdomains_github_push
[19:04:51] <icinga-miraheze>	 PROBLEM - mw181 Puppet on mw181 is CRITICAL: CRITICAL: Puppet has 2 failures. Last run 2 minutes ago with 2 failures. Failed resources (up to 3 shown): Exec[git_pull_femiwiki-deploy-1.45],Exec[git_pull_3d2png]
[19:07:50] <icinga-miraheze>	 PROBLEM - mw183 Puppet on mw183 is CRITICAL: CRITICAL: Puppet has 2 failures. Last run 2 minutes ago with 2 failures. Failed resources (up to 3 shown): Exec[git_pull_femiwiki-deploy-1.45],Exec[git_pull_3d2png]
[19:07:59] <icinga-miraheze>	 PROBLEM - mw182 Puppet on mw182 is CRITICAL: CRITICAL: Puppet has 2 failures. Last run 2 minutes ago with 2 failures. Failed resources (up to 3 shown): Exec[git_pull_femiwiki-deploy-1.45],Exec[git_pull_3d2png]
[19:13:12] <icinga-miraheze>	 PROBLEM - mw172 Puppet on mw172 is CRITICAL: CRITICAL: Puppet has 2 failures. Last run 3 minutes ago with 2 failures. Failed resources (up to 3 shown): Exec[git_pull_femiwiki-deploy-1.45],Exec[git_pull_3d2png]
[19:16:20] <icinga-miraheze>	 PROBLEM - puppet181 Puppet on puppet181 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_puppet]
[19:19:24] <icinga-miraheze>	 RECOVERY - puppet181 Check unit status of git_pull_puppet on puppet181 is OK: OK: Status of the systemd unit git_pull_puppet
[19:19:25] <icinga-miraheze>	 RECOVERY - puppet181 Check unit status of git_pull_ssl on puppet181 is OK: OK: Status of the systemd unit git_pull_ssl
[19:21:28] <icinga-miraheze>	 PROBLEM - mw193 Puppet on mw193 is CRITICAL: CRITICAL: Puppet has 2 failures. Last run 3 minutes ago with 2 failures. Failed resources (up to 3 shown): Exec[git_pull_femiwiki-deploy-1.45],Exec[git_pull_3d2png]
[19:21:33] <icinga-miraheze>	 RECOVERY - mw191 Puppet on mw191 is OK: OK: Puppet is currently enabled, last run 17 seconds ago with 0 failures
[19:23:04] <icinga-miraheze>	 RECOVERY - mw153 Puppet on mw153 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[19:29:11] <icinga-miraheze>	 PROBLEM - mw192 Puppet on mw192 is CRITICAL: CRITICAL: Puppet has 2 failures. Last run 2 minutes ago with 2 failures. Failed resources (up to 3 shown): Exec[git_pull_femiwiki-deploy-1.45],Exec[git_pull_3d2png]
[19:30:51] <icinga-miraheze>	 RECOVERY - mw181 Puppet on mw181 is OK: OK: Puppet is currently enabled, last run 2 seconds ago with 0 failures
[19:30:54] <icinga-miraheze>	 PROBLEM - mw151 Puppet on mw151 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_femiwiki-deploy-1.45]
[19:31:24] <icinga-miraheze>	 PROBLEM - puppet181 Check unit status of git_pull_puppet on puppet181 is CRITICAL: CRITICAL: Status of the systemd unit git_pull_puppet
[19:31:25] <icinga-miraheze>	 PROBLEM - puppet181 Check unit status of git_pull_ssl on puppet181 is CRITICAL: CRITICAL: Status of the systemd unit git_pull_ssl
[19:31:31] <icinga-miraheze>	 RECOVERY - puppet181 Check unit status of listdomains_github_push on puppet181 is OK: OK: Status of the systemd unit listdomains_github_push
[19:35:59] <icinga-miraheze>	 RECOVERY - mw182 Puppet on mw182 is OK: OK: Puppet is currently enabled, last run 50 seconds ago with 0 failures
[19:36:07] <icinga-miraheze>	 RECOVERY - mw183 Puppet on mw183 is OK: OK: Puppet is currently enabled, last run 45 seconds ago with 0 failures
[19:41:12] <icinga-miraheze>	 RECOVERY - mw172 Puppet on mw172 is OK: OK: Puppet is currently enabled, last run 35 seconds ago with 0 failures
[19:42:20] <icinga-miraheze>	 RECOVERY - puppet181 Puppet on puppet181 is OK: OK: Puppet is currently enabled, last run 40 seconds ago with 0 failures
[19:43:05] <icinga-miraheze>	 PROBLEM - mw202 Puppet on mw202 is CRITICAL: CRITICAL: Puppet has 2 failures. Last run 3 minutes ago with 2 failures. Failed resources (up to 3 shown): Exec[git_pull_femiwiki-deploy-1.45],Exec[git_pull_3d2png]
[19:47:32] <icinga-miraheze>	 PROBLEM - mw173 Puppet on mw173 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_3d2png]
[19:49:24] <icinga-miraheze>	 RECOVERY - puppet181 Check unit status of git_pull_puppet on puppet181 is OK: OK: Status of the systemd unit git_pull_puppet
[19:49:25] <icinga-miraheze>	 RECOVERY - puppet181 Check unit status of git_pull_ssl on puppet181 is OK: OK: Status of the systemd unit git_pull_ssl
[19:49:30] <icinga-miraheze>	 RECOVERY - mw193 Puppet on mw193 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[19:54:23] <icinga-miraheze>	 RECOVERY - bots171 Puppet on bots171 is OK: OK: Puppet is currently enabled, last run 52 seconds ago with 0 failures
[19:57:11] <icinga-miraheze>	 RECOVERY - mw192 Puppet on mw192 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[19:58:52] <icinga-miraheze>	 RECOVERY - mw151 Puppet on mw151 is OK: OK: Puppet is currently enabled, last run 29 seconds ago with 0 failures
[19:59:57] <icinga-miraheze>	 RECOVERY - mw163 Puppet on mw163 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[20:11:04] <icinga-miraheze>	 RECOVERY - mw202 Puppet on mw202 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[20:13:32] <icinga-miraheze>	 RECOVERY - mw173 Puppet on mw173 is OK: OK: Puppet is currently enabled, last run 32 seconds ago with 0 failures