[00:00:54] <MacFanBot>	 [02statichelp] 07WikiTideBot pushed 1 new commit to 03main 13https://github.com/miraheze/statichelp/commit/0d8b19ace4e7b88342b95106b803b49badc28441
[00:00:54] <MacFanBot>	 02statichelp/03main 07WikiTideBot 030d8b19a Bot: Auto-update Tech namespace pages 2026-03-07 00:00:51
[00:02:04] <icinga-miraheze>	 RECOVERY - matomo151 Check unit status of matomo-archiver-2 on matomo151 is OK: OK: Status of the systemd unit matomo-archiver-2
[00:30:11] <icinga-miraheze>	 PROBLEM - cp171 Disk Space on cp171 is WARNING: DISK WARNING - free space: / 44049MiB (9.7% inode=100%);
[00:34:11] <icinga-miraheze>	 RECOVERY - cp171 Disk Space on cp171 is OK: DISK OK - free space: / 80890MiB (17.8% inode=100%);
[05:00:21] <MacFanBot>	 [02ssl] 07WikiTideBot pushed 1 new commit to 03main 13https://github.com/miraheze/ssl/commit/366c39aae619fd33fc45b1e6eca818de63eb062e
[05:00:21] <MacFanBot>	 02ssl/03main 07WikiTideBot 03366c39a Bot: Auto-update domain lists
[05:30:22] <MacFanBot>	 [02ssl] 07WikiTideBot pushed 1 new commit to 03main 13https://github.com/miraheze/ssl/commit/5f6a257e4457659f35248c15ce852206348302ff
[05:30:22] <MacFanBot>	 02ssl/03main 07WikiTideBot 035f6a257 Bot: Auto-update domain lists
[06:39:14] <icinga-miraheze>	 PROBLEM - db161 Current Load on db161 is CRITICAL: LOAD CRITICAL - total load average: 27.45, 12.51, 5.70
[06:41:14] <icinga-miraheze>	 RECOVERY - db161 Current Load on db161 is OK: LOAD OK - total load average: 5.25, 8.90, 5.20
[09:57:07] <icinga-miraheze>	 PROBLEM - mw173 MediaWiki Rendering on mw173 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 8191 bytes in 2.706 second response time
[09:57:35] <icinga-miraheze>	 PROBLEM - cp201 Varnish Backends on cp201 is CRITICAL: 1 backends are down. mw173
[09:57:49] <icinga-miraheze>	 PROBLEM - cp171 Varnish Backends on cp171 is CRITICAL: 1 backends are down. mw173
[09:59:02] <icinga-miraheze>	 RECOVERY - mw173 MediaWiki Rendering on mw173 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.219 second response time
[09:59:35] <icinga-miraheze>	 RECOVERY - cp201 Varnish Backends on cp201 is OK: All 31 backends are healthy
[09:59:49] <icinga-miraheze>	 RECOVERY - cp171 Varnish Backends on cp171 is OK: All 31 backends are healthy
[10:21:17] <MirahezeRelay>	 <pskyechology> oookay
[10:43:13] <Skye>	 watch mwtask171 scream in a sec because i went crazy with parallelizing a script to get biggest swift containers
[10:46:00] <Skye>	 actually it might not because i am smart and killed it
[12:15:21] <MacFanBot>	 [02puppet] 07paladox created 03paladox-patch-2 (+1 new commit) 13https://github.com/miraheze/puppet/commit/e819ab56375b
[12:15:22] <MacFanBot>	 02puppet/03paladox-patch-2 07paladox 03e819ab5 Remove swiftobject151 from swift storage devices…
[12:15:25] <MacFanBot>	 [02puppet] 07paladox opened pull request #4821: Remove swiftobject151 from swift storage devices (03main...03paladox-patch-2) 13https://github.com/miraheze/puppet/pull/4821
[12:16:10] <MacFanBot>	 [02puppet] 07paladox pushed 1 new commit to 03main 13https://github.com/miraheze/puppet/commit/37b2e8b9f9a31360f2242475b18fdb214b4617f8
[12:16:12] <MacFanBot>	 02puppet/03main 07paladox 0337b2e8b Remove swiftobject151 from swift storage devices (#4821)…
[12:16:14] <MacFanBot>	 [02puppet] 07paladox merged pull request #4821: Remove swiftobject151 from swift storage devices (03main...03paladox-patch-2) 13https://github.com/miraheze/puppet/pull/4821
[12:16:16] <MacFanBot>	 [02puppet] 07paladox 04deleted 03paladox-patch-2 at 03e819ab5 13https://api.github.com/repos/miraheze/puppet/commit/e819ab5
[12:39:23] <MacFanBot>	 [02mw-config] 07paladox created 03paladox-patch-6 (+1 new commit) 13https://github.com/miraheze/mw-config/commit/c8fd0b1c018f
[12:39:23] <MacFanBot>	 02mw-config/03paladox-patch-6 07paladox 03c8fd0b1 Use order by in generateDatabaseLists for query…
[12:39:26] <MacFanBot>	 [02mw-config] 07paladox opened pull request #6338: Use order by in generateDatabaseLists for query (03main...03paladox-patch-6) 13https://github.com/miraheze/mw-config/pull/6338
[12:40:35] <miraheze-github>	 miraheze/mw-config - paladox the build passed.
[13:08:43] <MacFanBot>	 [02mw-config] 07paladox created 03paladox-patch-7 (+1 new commit) 13https://github.com/miraheze/mw-config/commit/93e2f65ce5e0
[13:08:44] <MacFanBot>	 02mw-config/03paladox-patch-7 07paladox 0393e2f65 Refactor wiki list processing for clarity and efficiency…
[13:08:46] <MacFanBot>	 [02mw-config] 07paladox opened pull request #6339: Refactor wiki list processing for clarity and efficiency (03main...03paladox-patch-7) 13https://github.com/miraheze/mw-config/pull/6339
[13:09:35] <MacFanBot>	 [02mw-config] 07github-actions[bot] pushed 1 new commit to 03paladox-patch-7 13https://github.com/miraheze/mw-config/commit/24239b0e582a1a55e345369483c702c3d1749504
[13:09:36] <MacFanBot>	 02mw-config/03paladox-patch-7 07github-actions 0324239b0 CI: lint code to MediaWiki standards…
[13:10:27] <miraheze-github>	 miraheze/mw-config - paladox the build passed.
[13:30:05] <MacFanBot>	 [02mw-config] 07paladox pushed 1 new commit to 03paladox-patch-7 13https://github.com/miraheze/mw-config/commit/a9f3818dfec040b3d709e00bce4a14161c785062
[13:30:05] <MacFanBot>	 02mw-config/03paladox-patch-7 07paladox 03a9f3818 Refactor generateDatabaseLists for efficiency
[13:30:51] <MacFanBot>	 [02mw-config] 07github-actions[bot] pushed 1 new commit to 03paladox-patch-7 13https://github.com/miraheze/mw-config/commit/d0a2bdf28bfcb0e55a3bf339703ab831e2a99088
[13:30:51] <MacFanBot>	 02mw-config/03paladox-patch-7 07github-actions 03d0a2bdf CI: lint code to MediaWiki standards…
[13:31:40] <miraheze-github>	 miraheze/mw-config - paladox the build passed.
[13:44:47] <MacFanBot>	 [02mw-config] 07paladox commented on pull request #6339: Before `Memory used: 31.48 MB` […] 13https://github.com/miraheze/mw-config/pull/6339#issuecomment-4016549880
[15:57:58] <MirahezeLSBot>	 !log [paladox@test151] starting deploy of {'l10n': True, 'versions': '1.45', 'upgrade_extensions': 'PortableInfobox'} to test151
[15:58:00] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[15:58:14] <MirahezeLSBot>	 !log [paladox@mwtask181] starting deploy of {'l10n': True, 'versions': '1.45', 'upgrade_extensions': 'PortableInfobox'} to all
[15:58:16] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[15:58:56] <MirahezeLSBot>	 !log [paladox@test151] finished deploy of {'l10n': True, 'versions': '1.45', 'upgrade_extensions': 'PortableInfobox'} to test151 - SUCCESS in 57s
[15:58:57] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[16:07:18] <MirahezeLSBot>	 !log [paladox@mwtask181] finished deploy of {'l10n': True, 'versions': '1.45', 'upgrade_extensions': 'PortableInfobox'} to all - SUCCESS in 543s
[16:07:20] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[16:30:22] <MacFanBot>	 [02ssl] 07WikiTideBot pushed 1 new commit to 03main 13https://github.com/miraheze/ssl/commit/c39a69e00ac4c4fe8b9588d8b12c63a1a1036931
[16:30:22] <MacFanBot>	 02ssl/03main 07WikiTideBot 03c39a69e Bot: Auto-update domain lists
[17:27:07] <MacFanBot>	 [02puppet] 07paladox created 03paladox-patch-2 (+1 new commit) 13https://github.com/miraheze/puppet/commit/5d5bbffde422
[17:27:09] <MacFanBot>	 02puppet/03paladox-patch-2 07paladox 035d5bbff Switch expirerer host to swiftobject161
[17:27:11] <MacFanBot>	 [02puppet] 07paladox opened pull request #4822: Switch expirerer host to swiftobject161 (03main...03paladox-patch-2) 13https://github.com/miraheze/puppet/pull/4822
[17:27:16] <MacFanBot>	 [02puppet] 07paladox pushed 1 new commit to 03main 13https://github.com/miraheze/puppet/commit/5d47abb47679b718db75717e2944ecf97c9162fa
[17:27:18] <MacFanBot>	 02puppet/03main 07paladox 035d47abb Switch expirerer host to swiftobject161 (#4822)
[17:27:20] <MacFanBot>	 [02puppet] 07paladox merged pull request #4822: Switch expirerer host to swiftobject161 (03main...03paladox-patch-2) 13https://github.com/miraheze/puppet/pull/4822
[17:27:22] <MacFanBot>	 [02puppet] 07paladox 04deleted 03paladox-patch-2 at 035d5bbff 13https://api.github.com/repos/miraheze/puppet/commit/5d5bbff
[17:36:47] <MirahezeRelay>	 <ro0tasxd> [1/4] https://cdn.discordapp.com/attachments/808001911868489748/1479895602589597858/image_9s4ctd.jpg?ex=69adb3ae&is=69ac622e&hm=e39c6bafb0a576220c3eb886d447109a300cff87f4801962d7fa02c5424896e6&
[17:36:48] <MirahezeRelay>	 <ro0tasxd> [2/4] https://cdn.discordapp.com/attachments/808001911868489748/1479895603109429299/1771854798364.jpg?ex=69adb3ae&is=69ac622e&hm=f178d17253ea2891af80bc8028fecbcbc53d1ce2fec36dd1ff2dde3fc76e609c&
[17:36:48] <MirahezeRelay>	 <ro0tasxd> [3/4] https://cdn.discordapp.com/attachments/808001911868489748/1479895603780780205/20260213_134054.jpg?ex=69adb3af&is=69ac622f&hm=ad7d01974d7e6dadac3b6dd98ff18c9b835942c2f51abab304b4e90e65b05857&
[17:36:48] <MirahezeRelay>	 <ro0tasxd> [4/4] https://cdn.discordapp.com/attachments/808001911868489748/1479895604627767439/image_ksuynz.jpg?ex=69adb3af&is=69ac622f&hm=5a76f7e9b3ca59327809d86150111cc041d0550d2b3e8197026bf11932b689a4&
[18:26:11] <MacFanBot>	 [02TSPortal] 07Universal-Omega created 03improve-ial-message (+1 new commit) 13https://github.com/miraheze/TSPortal/commit/438ccc4368b9
[18:26:11] <MacFanBot>	 02TSPortal/03improve-ial-message 07CosmicAlpha 03438ccc4 Improvements/fixes to IAL message…
[18:26:17] <MacFanBot>	 [02TSPortal] 07Universal-Omega opened pull request #62: Improvements/fixes to IAL message (03main...03improve-ial-message) 13https://github.com/miraheze/TSPortal/pull/62
[18:49:57] <MacFanBot>	 [02mw-config] 07Universal-Omega approved pull request #6338 13https://github.com/miraheze/mw-config/pull/6338#pullrequestreview-3909353270
[18:50:09] <MacFanBot>	 [02mw-config] 07Universal-Omega merged 07paladox's pull request #6338: Use order by in generateDatabaseLists for query (03main...03paladox-patch-6) 13https://github.com/miraheze/mw-config/pull/6338
[18:50:09] <MacFanBot>	 [02mw-config] 07Universal-Omega pushed 1 new commit to 03main 13https://github.com/miraheze/mw-config/commit/abf3013cc4f57cc0e880a3f21e71705f54a36ceb
[18:50:09] <MacFanBot>	 02mw-config/03main 07paladox 03abf3013 Use order by in generateDatabaseLists for query (#6338)…
[18:50:10] <MacFanBot>	 [02mw-config] 07Universal-Omega 04deleted 03paladox-patch-6 at 03c8fd0b1 13https://api.github.com/repos/miraheze/mw-config/commit/c8fd0b1
[18:51:04] <MirahezeLSBot>	 !log [universalomega@mwtask181] starting deploy of {'pull': 'config', 'config': True} to all
[18:51:06] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[18:51:29] <MirahezeLSBot>	 !log [universalomega@mwtask181] finished deploy of {'pull': 'config', 'config': True} to all - SUCCESS in 24s
[18:51:30] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[18:51:40] <miraheze-github>	 miraheze/mw-config - Universal-Omega the build passed.
[18:53:15] <MacFanBot>	 [02mw-config] 07Universal-Omega approved pull request #6339 13https://github.com/miraheze/mw-config/pull/6339#pullrequestreview-3909358397
[19:33:36] <icinga-miraheze>	 RECOVERY - test151 Puppet on test151 is OK: OK: Puppet is currently enabled, last run 31 seconds ago with 0 failures
[19:37:27] <MirahezeLSBot>	 !log [wwr@test151] starting deploy of {'l10n': True, 'extension_list': True, 'force_upgrade': True, 'versions': '1.45', 'upgrade_extensions': 'Bucket'} to test151
[19:37:28] <MirahezeLSBot>	 !log [wwr@test151] DEPLOY ABORTED: Non-Zero Exit Code in prep, see output.
[19:37:29] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[19:37:31] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[19:37:43] <MirahezeLSBot>	 !log [wwr@test151] starting deploy of {'config': True} to test151
[19:37:44] <MirahezeLSBot>	 !log [wwr@test151] finished deploy of {'config': True} to test151 - SUCCESS in 0s
[19:37:45] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[19:37:47] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[19:41:51] <MirahezeRelay>	 <thewwrnerdguy, replying to MirahezeLSBot> hmm not a very descriptive output (guess it didn't matter anyway)
[19:42:20] <MirahezeRelay>	 <abaddriverlol> the patch didn't apply
[19:42:26] <MirahezeRelay>	 <abaddriverlol> because it's been upstreamed
[19:42:28] <MirahezeRelay>	 <thewwrnerdguy> anyway looks like my config changes worked
[19:42:32] <MirahezeRelay>	 <abaddriverlol> I will remove it later
[20:05:45] <icinga-miraheze>	 PROBLEM - test151 Puppet on test151 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_checkout_MediaWiki config]
[20:06:42] <MirahezeRelay>	 <rhinosf1, replying to thewwrnerdguy> What was the error
[20:06:48] <MirahezeRelay>	 <rhinosf1> I can try make it better
[20:07:17] <MirahezeRelay>	 <thewwrnerdguy, replying to rhinosf1> code 128, that's literally it
[20:07:47] <MirahezeRelay>	 <rhinosf1, replying to thewwrnerdguy> Nothing in the terminal output above
[20:07:54] <MirahezeRelay>	 <rhinosf1> When it was applying the patch
[20:08:48] <MirahezeRelay>	 <thewwrnerdguy> nope just told me it was a non-zero exit code and it was aborted. unfortunately I already logged out of shell so I don't have the actual output but iirc it was 3 lines long
[20:09:02] <MirahezeRelay>	 <rhinosf1> Hmm
[20:09:09] <MirahezeRelay>	 <rhinosf1> That's not what it should do
[20:09:26] <MirahezeRelay>	 <rhinosf1> stdout should be captured and relayed to your
[20:22:24] <MacFanBot>	 [02mw-config] 07paladox merged pull request #6339: Refactor wiki list processing for clarity and efficiency (03main...03paladox-patch-7) 13https://github.com/miraheze/mw-config/pull/6339
[20:22:24] <MacFanBot>	 [02mw-config] 07paladox pushed 1 new commit to 03main 13https://github.com/miraheze/mw-config/commit/c96b7f9ac9fe5a3c8a7488f3fdf7abcf5c369639
[20:22:24] <MacFanBot>	 02mw-config/03main 07paladox 03c96b7f9 Refactor wiki list processing for clarity and efficiency (#6339)…
[20:22:26] <MacFanBot>	 [02mw-config] 07paladox 04deleted 03paladox-patch-7 at 03d0a2bdf 13https://api.github.com/repos/miraheze/mw-config/commit/d0a2bdf
[20:23:09] <MirahezeLSBot>	 !log [paladox@test151] starting deploy of {'pull': 'config', 'config': True} to test151
[20:23:10] <MirahezeLSBot>	 !log [paladox@test151] DEPLOY ABORTED: Non-Zero Exit Code in prep, see output.
[20:23:11] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[20:23:13] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[20:23:18] <MirahezeLSBot>	 !log [paladox@mwtask181] starting deploy of {'pull': 'config', 'config': True} to all
[20:23:20] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[20:23:41] <MirahezeLSBot>	 !log [paladox@mwtask181] finished deploy of {'pull': 'config', 'config': True} to all - SUCCESS in 23s
[20:23:43] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[20:23:55] <miraheze-github>	 miraheze/mw-config - paladox the build passed.
[21:25:23] <MacFanBot>	 [02puppet] 07pskyechology opened pull request #4823: T15058: Add pixldev to bastion and mediawiki-test-admins (07miraheze:03main...07pskyechology:03T15058) 13https://github.com/miraheze/puppet/pull/4823
[21:45:35] <icinga-miraheze>	 PROBLEM - cp201 Varnish Backends on cp201 is CRITICAL: 1 backends are down. mw163
[21:45:37] <icinga-miraheze>	 PROBLEM - mw163 HTTPS on mw163 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10004 milliseconds with 0 bytes received
[21:45:49] <icinga-miraheze>	 PROBLEM - cp171 Varnish Backends on cp171 is CRITICAL: 1 backends are down. mw163
[21:46:04] <icinga-miraheze>	 PROBLEM - mw163 PowerDNS Recursor on mw163 is CRITICAL: CRITICAL - Plugin timed out while executing system call
[21:46:04] <icinga-miraheze>	 PROBLEM - mw163 SSH on mw163 is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[21:46:20] <icinga-miraheze>	 PROBLEM - cp161 Varnish Backends on cp161 is CRITICAL: 1 backends are down. mw163
[21:47:00] <icinga-miraheze>	 PROBLEM - mw163 Current Load on mw163 is CRITICAL: LOAD CRITICAL - total load average: 62.72, 35.08, 17.58
[21:47:03] <icinga-miraheze>	 PROBLEM - mw163 php-fpm on mw163 is CRITICAL: CRITICAL - Plugin timed out after 10 seconds
[21:47:14] <icinga-miraheze>	 PROBLEM - mw163 MediaWiki Rendering on mw163 is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[21:47:20] <icinga-miraheze>	 PROBLEM - cp191 Varnish Backends on cp191 is CRITICAL: 1 backends are down. mw163
[21:47:58] <icinga-miraheze>	 RECOVERY - mw163 PowerDNS Recursor on mw163 is OK: DNS OK: 0.306 seconds response time. mw163.fsslc.wtnet returns 10.0.16.151
[21:48:00] <icinga-miraheze>	 RECOVERY - mw163 SSH on mw163 is OK: SSH OK - OpenSSH_10.0p2 (protocol 2.0)
[21:49:00] <icinga-miraheze>	 RECOVERY - mw163 php-fpm on mw163 is OK: PROCS OK: 25 processes with command name 'php-fpm8.4'
[21:49:13] <icinga-miraheze>	 RECOVERY - mw163 MediaWiki Rendering on mw163 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.171 second response time
[21:49:20] <icinga-miraheze>	 RECOVERY - cp191 Varnish Backends on cp191 is OK: All 31 backends are healthy
[21:49:35] <icinga-miraheze>	 RECOVERY - cp201 Varnish Backends on cp201 is OK: All 31 backends are healthy
[21:49:36] <icinga-miraheze>	 RECOVERY - mw163 HTTPS on mw163 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4308 bytes in 0.071 second response time
[21:49:49] <icinga-miraheze>	 RECOVERY - cp171 Varnish Backends on cp171 is OK: All 31 backends are healthy
[21:50:20] <icinga-miraheze>	 RECOVERY - cp161 Varnish Backends on cp161 is OK: All 31 backends are healthy
[21:52:51] <icinga-miraheze>	 RECOVERY - mw163 Current Load on mw163 is OK: LOAD OK - total load average: 6.19, 18.47, 16.38
[22:03:26] <MacFanBot>	 [02puppet] 07Universal-Omega commented on pull request #4823: Per their request this shouldn't be merged just yet until they are ready for on-boarding. 13https://github.com/miraheze/puppet/pull/4823#issuecomment-4017491783
[22:34:13] <MirahezeLSBot>	 !log [blankeclair@mwtask181] starting deploy of {'l10n': True, 'versions': '1.45', 'upgrade_extensions': 'ConfirmEdit'} to all
[22:34:15] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[22:36:13] <MirahezeLSBot>	 !log [blankeclair@mwtask181] finished deploy of {'l10n': True, 'versions': '1.45', 'upgrade_extensions': 'ConfirmEdit'} to all - SUCCESS in 120s
[22:36:15] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[23:48:09] <icinga-miraheze>	 PROBLEM - cp191 Disk Space on cp191 is WARNING: DISK WARNING - free space: / 44090MiB (9.7% inode=100%);
[23:54:09] <icinga-miraheze>	 RECOVERY - cp191 Disk Space on cp191 is OK: DISK OK - free space: / 88618MiB (19.5% inode=100%);