[00:00:11] <icinga-miraheze>	 PROBLEM - cp171 Disk Space on cp171 is WARNING: DISK WARNING - free space: / 44223MiB (9.7% inode=100%);
[00:01:04] <MacFanBot>	 [02statichelp] 07WikiTideBot pushed 1 new commit to 03main 13https://github.com/miraheze/statichelp/commit/3d0f30d27d9164de1098f1a63721c0365fe53520
[00:01:04] <MacFanBot>	 02statichelp/03main 07WikiTideBot 033d0f30d Bot: Auto-update Tech namespace pages 2026-03-08 00:01:02
[00:04:11] <icinga-miraheze>	 RECOVERY - cp171 Disk Space on cp171 is OK: DISK OK - free space: / 87883MiB (19.4% inode=100%);
[00:19:50] <icinga-miraheze>	 PROBLEM - cp201 Disk Space on cp201 is WARNING: DISK WARNING - free space: / 45197MiB (10.0% inode=100%);
[00:39:10] <MacFanBot>	 [02TSPortal] 07renovate[bot] created 03renovate/lock-file-maintenance (+1 new commit) 13https://github.com/miraheze/TSPortal/commit/995e40ffa7c2
[00:39:11] <MacFanBot>	 02TSPortal/03renovate/lock-file-maintenance 07renovate[bot] 03995e40f Lock file maintenance
[01:05:50] <icinga-miraheze>	 RECOVERY - cp201 Disk Space on cp201 is OK: DISK OK - free space: / 87791MiB (19.4% inode=100%);
[01:55:21] <MacFanBot>	 [02mw-config] 07pixDeVl opened pull request #6340: Standardize/fix links to m:Phorge in extension descs (07miraheze:03main...07pixDeVl:03patch-5) 13https://github.com/miraheze/mw-config/pull/6340
[01:56:31] <miraheze-github>	 miraheze/mw-config - pixDeVl the build passed.
[01:57:00] <MacFanBot>	 [02mw-config] 07Universal-Omega left a file comment in pull request #6340 037ff46fa: This shouldn't be removed. 13https://github.com/miraheze/mw-config/pull/6340#discussion_r2901114939
[02:00:21] <MacFanBot>	 [02ssl] 07WikiTideBot pushed 1 new commit to 03main 13https://github.com/miraheze/ssl/commit/40fd1f7aa32b7d49b04982b931794ab1f5b26c99
[02:00:22] <MacFanBot>	 02ssl/03main 07WikiTideBot 0340fd1f7 Bot: Auto-update domain lists
[02:19:52] <miraheze-github>	 miraheze/mw-config - pixDeVl the build passed.
[04:47:14] <icinga-miraheze>	 PROBLEM - db161 Current Load on db161 is CRITICAL: LOAD CRITICAL - total load average: 22.68, 19.43, 9.05
[04:51:14] <icinga-miraheze>	 RECOVERY - db161 Current Load on db161 is OK: LOAD OK - total load average: 1.02, 9.05, 7.13
[05:36:08] <MacFanBot>	 [02TSPortal] 07renovate[bot] pushed 1 new commit to 03main 13https://github.com/miraheze/TSPortal/commit/995e40ffa7c2d4a43c8fa250a56894d554da3f2c
[05:36:09] <MacFanBot>	 02TSPortal/03main 07renovate[bot] 03995e40f Lock file maintenance
[05:36:10] <MacFanBot>	 [02TSPortal] 07renovate[bot] 04deleted 03renovate/lock-file-maintenance at 03995e40f 13https://api.github.com/repos/miraheze/TSPortal/commit/995e40f
[09:25:08] <icinga-miraheze>	 PROBLEM - swiftproxy161 HTTPS on swiftproxy161 is CRITICAL: connect to address 10.0.16.135 and port 443: Connection refusedHTTP CRITICAL - Unable to open TCP socket
[09:25:48] <icinga-miraheze>	 PROBLEM - swiftproxy161 Swift NGINX SSL check on swiftproxy161 is CRITICAL: connect to address localhost and port 443: Connection refused
[09:32:56] <icinga-miraheze>	 PROBLEM - swiftproxy171 Check unit status of swift_dispersion_stats on swiftproxy171 is CRITICAL: CRITICAL: Status of the systemd unit swift_dispersion_stats
[09:33:34] <icinga-miraheze>	 PROBLEM - swift-lb.wikitide.net - LetsEncrypt on sslhost is CRITICAL: connect to address swift-lb.wikitide.net and port 443: Connection refusedHTTP CRITICAL - Unable to open TCP socket
[09:33:56] <RhinosF1>	 Huh
[09:37:48] <icinga-miraheze>	 RECOVERY - swiftproxy161 Swift NGINX SSL check on swiftproxy161 is OK: OK - Certificate 'wikitide.net' will expire on Sun May 31 22:19:44 2026 +0000.TCP OK - 0.010 second response time on localhost port 443
[09:39:08] <icinga-miraheze>	 RECOVERY - swiftproxy161 HTTPS on swiftproxy161 is OK: HTTP OK: Status line output matched "HTTP/1.1 404" - 352 bytes in 0.029 second response time
[09:44:56] <icinga-miraheze>	 RECOVERY - swiftproxy171 Check unit status of swift_dispersion_stats on swiftproxy171 is OK: OK: Status of the systemd unit swift_dispersion_stats
[10:02:08] <icinga-miraheze>	 RECOVERY - swift-lb.wikitide.net - LetsEncrypt on sslhost is OK: OK - Certificate 'wikitide.net' will expire on Sun 31 May 2026 10:19:44 PM GMT +0000.
[11:40:38] <icinga-miraheze>	 PROBLEM - swiftproxy161 Current Load on swiftproxy161 is CRITICAL: LOAD CRITICAL - total load average: 8.58, 6.90, 4.48
[11:40:58] <icinga-miraheze>	 PROBLEM - swiftproxy171 Current Load on swiftproxy171 is CRITICAL: LOAD CRITICAL - total load average: 9.76, 7.43, 4.62
[11:42:38] <icinga-miraheze>	 RECOVERY - swiftproxy161 Current Load on swiftproxy161 is OK: LOAD OK - total load average: 4.75, 6.07, 4.47
[11:42:58] <icinga-miraheze>	 RECOVERY - swiftproxy171 Current Load on swiftproxy171 is OK: LOAD OK - total load average: 4.15, 6.21, 4.53
[12:56:32] <MacFanBot>	 [02TSPortal] 07renovate[bot] created 03renovate/github-actions (+1 new commit) 13https://github.com/miraheze/TSPortal/commit/d89358d92cf4
[12:56:32] <MacFanBot>	 02TSPortal/03renovate/github-actions 07renovate[bot] 03d89358d Update shivammathur/setup-php digest to a1e0f56
[15:21:52] <icinga-miraheze>	 PROBLEM - puppet181 Check unit status of git_pull_puppet on puppet181 is CRITICAL: CRITICAL: Status of the systemd unit git_pull_puppet
[15:23:08] <icinga-miraheze>	 PROBLEM - puppet181 Check unit status of git_pull_ssl on puppet181 is CRITICAL: CRITICAL: Status of the systemd unit git_pull_ssl
[15:29:08] <icinga-miraheze>	 RECOVERY - puppet181 Check unit status of git_pull_ssl on puppet181 is OK: OK: Status of the systemd unit git_pull_ssl
[15:29:52] <icinga-miraheze>	 RECOVERY - puppet181 Check unit status of git_pull_puppet on puppet181 is OK: OK: Status of the systemd unit git_pull_puppet
[15:41:49] <icinga-miraheze>	 PROBLEM - mw161 Puppet on mw161 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle.
[15:53:08] <icinga-miraheze>	 PROBLEM - puppet181 Check unit status of git_pull_ssl on puppet181 is CRITICAL: CRITICAL: Status of the systemd unit git_pull_ssl
[15:59:08] <icinga-miraheze>	 RECOVERY - puppet181 Check unit status of git_pull_ssl on puppet181 is OK: OK: Status of the systemd unit git_pull_ssl
[16:11:50] <icinga-miraheze>	 RECOVERY - mw161 Puppet on mw161 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[17:02:45] <icinga-miraheze>	 PROBLEM - ns2 Puppet on ns2 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): File[/etc/apt/apt.conf.d/50unattended-upgrades]
[17:03:04] <MacFanBot>	 [02puppet] 07MacFan4000 commented on pull request #4823: there is now a second key to be added. 13https://github.com/miraheze/puppet/pull/4823#issuecomment-4019558890
[17:18:18] <MacFanBot>	 [02TSPortal] 07renovate[bot] pushed 1 new commit to 03main 13https://github.com/miraheze/TSPortal/commit/d89358d92cf424e21e8fbe883b5e3ce93ec6d793
[17:18:19] <MacFanBot>	 02TSPortal/03main 07renovate[bot] 03d89358d Update shivammathur/setup-php digest to a1e0f56
[17:18:21] <MacFanBot>	 [02TSPortal] 07renovate[bot] 04deleted 03renovate/github-actions at 03d89358d 13https://api.github.com/repos/miraheze/TSPortal/commit/d89358d
[17:31:09] <icinga-miraheze>	 RECOVERY - ns2 Puppet on ns2 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[19:14:14] <icinga-miraheze>	 PROBLEM - llm191 Puppet on llm191 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[openwebui]
[19:42:34] <icinga-miraheze>	 RECOVERY - llm191 Puppet on llm191 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[20:00:13] <MacFanBot>	 [02mw-config] 07Universal-Omega created 03remove-wgPortableInfoboxUseHeadings (+1 new commit) 13https://github.com/miraheze/mw-config/commit/da410f3dbe61
[20:00:13] <MacFanBot>	 02mw-config/03remove-wgPortableInfoboxUseHeadings 07CosmicAlpha 03da410f3 Remove wgPortableInfoboxUseHeadings from ManageWiki…
[20:00:19] <MacFanBot>	 [02mw-config] 07Universal-Omega opened pull request #6341: Remove wgPortableInfoboxUseHeadings from ManageWiki (03main...03remove-wgPortableInfoboxUseHeadings) 13https://github.com/miraheze/mw-config/pull/6341
[20:01:04] <MacFanBot>	 [02mw-config] 07Universal-Omega pushed 1 new commit to 03remove-wgPortableInfoboxUseHeadings 13https://github.com/miraheze/mw-config/commit/67c6a67f03415f32d70d37644939b492a099e3f6
[20:01:06] <MacFanBot>	 02mw-config/03remove-wgPortableInfoboxUseHeadings 07CosmicAlpha 0367c6a67 false
[20:02:32] <miraheze-github>	 miraheze/mw-config - Universal-Omega the build passed.
[20:35:57] <MirahezeRelay>	 <cosmicalpha> !log created ldap accounts for wwr and pixldev
[20:35:58] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[20:40:16] <MacFanBot>	 [02TSPortal] 07renovate[bot] created 03renovate/github-actions (+1 new commit) 13https://github.com/miraheze/TSPortal/commit/39fa25e26650
[20:40:16] <MacFanBot>	 02TSPortal/03renovate/github-actions 07renovate[bot] 0339fa25e Update shivammathur/setup-php digest to 8724c1d
[21:07:55] <MirahezeLSBot>	 !log [somerandomdeveloper@test151] starting deploy of {'versions': '1.45', 'upgrade_extensions': 'Bucket'} to test151
[21:07:56] <MirahezeLSBot>	 !log [somerandomdeveloper@test151] DEPLOY ABORTED: Non-Zero Exit Code in prep, see output.
[21:07:57] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[21:07:59] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[21:08:58] <MacFanBot>	 [02puppet] 07Universal-Omega merged 07pskyechology's pull request #4823: T15058: Add pixldev to bastion and mediawiki-test-admins (07miraheze:03main...07pskyechology:03T15058) 13https://github.com/miraheze/puppet/pull/4823
[21:08:58] <MacFanBot>	 [02puppet] 07Universal-Omega pushed 1 new commit to 03main 13https://github.com/miraheze/puppet/commit/3ad253ed61ca1626fb6a7e61423290980695868b
[21:09:00] <MacFanBot>	 02puppet/03main 07Skye 033ad253e T15058: Add pixldev to bastion and mediawiki-test-admins (#4823)
[21:10:14] <MirahezeRelay>	 <abaddriverlol, replying to MirahezeLSBot> @cosmicalpha private patches aren't removed from the server after being removed from the repo
[21:10:34] <MirahezeRelay>	 <abaddriverlol> I ran puppet on puppet181 and then test151, the diff showed that the JSON was updated, but the patch is still there
[21:11:04] <MirahezeRelay>	 <cosmicalpha> They arent removed but are from private.json so it shouldnt run still. Its not to bad. Will look at getting them to remove also though.
[21:11:17] <MirahezeRelay>	 <abaddriverlol> [1/5] ```
[21:11:17] <MirahezeRelay>	 <abaddriverlol> [2/5] [somerandomdeveloper@test151:~]$ mwdeploy --servers=all --versions=all --upgrade-extensions=Bucket
[21:11:18] <MirahezeRelay>	 <abaddriverlol> [3/5] Failed to upgrade Bucket (exit code: 128).
[21:11:18] <MirahezeRelay>	 <abaddriverlol> [4/5] Exiting due to non-zero status.
[21:11:18] <MirahezeRelay>	 <abaddriverlol> [5/5] ```
[21:11:31] <MirahezeRelay>	 <abaddriverlol> oh wait sorry
[21:11:36] <MirahezeRelay>	 <abaddriverlol> there's a local patch applied using git
[21:11:50] <MirahezeRelay>	 <abaddriverlol> @pskyechology patches should go in mediawiki-repos now
[21:12:14] <MirahezeLSBot>	 !log [somerandomdeveloper@test151] starting deploy of {'versions': '1.45', 'upgrade_extensions': 'Bucket'} to test151
[21:12:15] <MirahezeLSBot>	 !log [somerandomdeveloper@test151] finished deploy of {'versions': '1.45', 'upgrade_extensions': 'Bucket'} to test151 - SUCCESS in 0s
[21:12:16] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[21:12:17] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[21:12:31] <MirahezeRelay>	 <abaddriverlol> removed, but the security patch wasn't applied anymore because of the git local patch...
[21:13:10] <MirahezeLSBot>	 !log [somerandomdeveloper@mwtask181] starting deploy of {'versions': '1.45', 'upgrade_extensions': 'Bucket'} to all
[21:13:12] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[21:13:38] <MirahezeLSBot>	 !log [somerandomdeveloper@mwtask181] finished deploy of {'versions': '1.45', 'upgrade_extensions': 'Bucket'} to all - SUCCESS in 27s
[21:13:40] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[21:15:24] <MirahezeRelay>	 <abaddriverlol, replying to abaddriverlol> oh that's because --folders doesn't apply it
[21:15:29] <MirahezeRelay>	 <abaddriverlol> we should really discourage using that
[21:16:40] <MirahezeRelay>	 <pskyechology, replying to abaddriverlol> in all fairness i was going to but i was half asleep and just wanted to let wwr continue testing and then forgot after waking up
[21:16:42] <MirahezeRelay>	 <rhinosf1> --folders should be an alias of --extensions
[21:16:59] <MirahezeRelay>	 <abaddriverlol, replying to rhinosf1> sometimes you just want to sync parts of core/extensions/skins
[21:17:02] <MirahezeRelay>	 <rhinosf1> But I suspect that's been lost in changes and complexity since it was first done
[21:17:09] <MirahezeRelay>	 <abaddriverlol> but I agree there should be some solution
[21:17:21] <MirahezeRelay>	 <rhinosf1> @abaddriverlol ye but I mean doing --folders on an extension folder should do the right thing
[21:17:29] <MirahezeRelay>	 <cosmicalpha> !log added reader roles for pixldev and wwr in Graylog
[21:17:30] <MirahezeRelay>	 <rhinosf1> If it doesn't for sec patches, it's a bug
[21:17:31] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[21:17:32] <MirahezeRelay>	 <abaddriverlol, replying to rhinosf1> it also should on a subfolder
[21:17:37] <MirahezeRelay>	 <rhinosf1, replying to abaddriverlol> Yes
[21:17:39] <MirahezeRelay>	 <abaddriverlol> and the parent folder
[21:17:54] <MirahezeRelay>	 <abaddriverlol> why don't we apply the patches via puppet instead of mwdeploy?
[21:18:01] <MirahezeRelay>	 <rhinosf1> It feels like the logic for --folders has probably been broken
[21:18:09] <MirahezeRelay>	 <abaddriverlol, replying to abaddriverlol> oh because upgrading wouldn't work anymore ig
[21:18:23] <MirahezeRelay>	 <rhinosf1> puppet has little to do with mediawiki deploys
[21:18:28] <MirahezeRelay>	 <rhinosf1> Other than the initial clone
[21:18:39] <MirahezeRelay>	 <abaddriverlol, replying to rhinosf1> it clones and pulls the folders, so it might as well apply patches
[21:19:08] <MirahezeRelay>	 <rhinosf1> Hmm
[21:19:13] <MirahezeRelay>	 <rhinosf1> I think maybe not
[21:19:26] <MirahezeRelay>	 <rhinosf1> But I really need to look at the design principles of mwdeploy
[21:19:40] <MirahezeRelay>	 <rhinosf1> Because the code has changed so much since the original and never been properly redone
[21:20:58] <MirahezeRelay>	 <rhinosf1> mwdeploy was originally basically a shorthand for a loop of rsync with a few sprinkles on top
[21:21:04] <MirahezeRelay>	 <rhinosf1> It isn't really that any more
[21:23:21] <icinga-miraheze>	 RECOVERY - test151 Puppet on test151 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[22:01:16] <MirahezeLSBot>	 !log [skye@mwtask171] sudo -u www-data php /srv/mediawiki/1.45/maintenance/run.php refreshlinks --wiki=battlenationswiki (START)
[22:01:17] <MirahezeLSBot>	 !log [skye@mwtask171] sudo -u www-data php /srv/mediawiki/1.45/maintenance/run.php refreshlinks --wiki=battlenationswiki (END - exit=256)
[22:01:18] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[22:01:20] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[22:01:25] <MirahezeLSBot>	 !log [skye@mwtask171] sudo -u www-data php /srv/mediawiki/1.45/maintenance/run.php refreshLinks --wiki=battlenationswiki (START)
[22:01:27] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[22:03:21] <MirahezeRelay>	 <pskyechology> 72k? no wonder rebuildall might've done an oopsie
[22:07:03] <BlankEclair>	 imagine having to use refreshLinksChunked
[22:33:18] <MirahezeRelay>	 <pskyechology, replying to BlankEclair> i am afraid of the concept of such a beast
[22:33:29] <BlankEclair>	 check my home folder
[22:33:31] <BlankEclair>	 it should be there
[22:41:50] <icinga-miraheze>	 PROBLEM - cp201 Disk Space on cp201 is WARNING: DISK WARNING - free space: / 45215MiB (10.0% inode=100%);
[22:56:38] <icinga-miraheze>	 PROBLEM - swiftproxy161 Current Load on swiftproxy161 is WARNING: LOAD WARNING - total load average: 7.52, 6.02, 4.76
[22:58:38] <icinga-miraheze>	 RECOVERY - swiftproxy161 Current Load on swiftproxy161 is OK: LOAD OK - total load average: 4.38, 5.71, 4.83
[23:10:56] <MirahezeLSBot>	 !log [petramagna@mwtask181] starting deploy of {'versions': '1.45', 'upgrade_extensions': 'RenderBlocking'} to all
[23:10:58] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[23:11:21] <MirahezeLSBot>	 !log [petramagna@mwtask181] finished deploy of {'versions': '1.45', 'upgrade_extensions': 'RenderBlocking'} to all - SUCCESS in 24s
[23:11:23] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log