[00:01:17] [02statichelp] 07WikiTideBot pushed 1 new commit to 03main 13https://github.com/miraheze/statichelp/commit/031e5a9f5a8516b97909e01e8c584d814235de91 [00:01:17] 02statichelp/03main 07WikiTideBot 03031e5a9 Bot: Auto-update Tech namespace pages 2025-12-14 00:01:15 [00:13:54] RECOVERY - cp171 Disk Space on cp171 is OK: DISK OK - free space: / 73128MiB (16% inode=99%); [00:17:04] [02MirahezeMagic] 07SomeMWDev created 03tech-notifications (+1 new commit) 13https://github.com/miraheze/MirahezeMagic/commit/983970c270e7 [00:17:04] 02MirahezeMagic/03tech-notifications 07SomeRandomDeveloper 03983970c Add NotifyWikiUsers maintenance script… [00:17:29] [02MirahezeMagic] 07SomeMWDev opened pull request #613: Add NotifyWikiUsers maintenance script (03main...03tech-notifications) 13https://github.com/miraheze/MirahezeMagic/pull/613 [00:17:34] [02MirahezeMagic] 07coderabbitai[bot] commented on pull request #613:

[…] 13https://github.com/miraheze/MirahezeMagic/pull/613#issuecomment-3649965799 [00:18:06] RECOVERY - cp201 Disk Space on cp201 is OK: DISK OK - free space: / 70263MiB (15% inode=99%); [00:20:00] [02MirahezeMagic] 07SomeMWDev pushed 1 new commit to 03tech-notifications 13https://github.com/miraheze/MirahezeMagic/commit/fb44922dc2243c918fa9c2f2c80afc810b5a70df [00:20:00] 02MirahezeMagic/03tech-notifications 07SomeRandomDeveloper 03fb44922 add inheritDoc to hook handler [00:31:46] miraheze/MirahezeMagic - SomeMWDev the build passed. [00:39:19] [02MirahezeMagic] 07SomeMWDev pushed 2 new commits to 03tech-notifications 13https://github.com/miraheze/MirahezeMagic/compare/fb44922dc224...d989af4db7de [00:39:19] 02MirahezeMagic/03tech-notifications 07SomeRandomDeveloper 032b92933 Collapse groups loop into one query [00:39:21] 02MirahezeMagic/03tech-notifications 07SomeRandomDeveloper 03d989af4 Simplify logic and use fetchFieldValues [00:41:17] [02MirahezeMagic] 07SomeMWDev pushed 1 new commit to 03tech-notifications 13https://github.com/miraheze/MirahezeMagic/commit/ad6711df715728403b16dc2bd7009a19de1580bc [00:41:18] 02MirahezeMagic/03tech-notifications 07SomeRandomDeveloper 03ad6711d log eligible groups [00:46:40] [02MirahezeMagic] 07SomeMWDev retitled PR #613: "Add NotifyWikiUsers maintenance script" ➜ "Add maintenance script to notify privileged users on a wiki" 13https://github.com/miraheze/MirahezeMagic/pull/613 [00:53:27] miraheze/MirahezeMagic - SomeMWDev the build passed. [03:03:22] PROBLEM - puppet181 Disk Space on puppet181 is WARNING: DISK WARNING - free space: / 1873MiB (6% inode=79%); [03:07:22] RECOVERY - puppet181 Disk Space on puppet181 is OK: DISK OK - free space: / 4225MiB (13% inode=79%); [09:48:06] PROBLEM - db201 Disk Space on db201 is CRITICAL: DISK CRITICAL - free space: / 20984MiB (5.0% inode=100%); [10:28:06] PROBLEM - cp201 Disk Space on cp201 is WARNING: DISK WARNING - free space: / 49748MiB (10% inode=99%); [13:19:54] PROBLEM - cp171 Disk Space on cp171 is WARNING: DISK WARNING - free space: / 49830MiB (10% inode=99%); [14:22:29] PROBLEM - cp191 Disk Space on cp191 is WARNING: DISK WARNING - free space: / 49847MiB (10% inode=99%); [15:22:45] [02MirahezeMagic] 07SomeMWDev left a file comment in pull request #613 03ad6711d: It's used in some places, for example in emails: https://gerrit.wikimedia.org/g/mediawiki/extensions/Echo/+/1287d8c/includes/Formatters/EchoHtmlEmailFormatter.php#193 13https://github.com/miraheze/MirahezeMagic/pull/613#discussion_r2617240949 [15:38:59] [02MirahezeMagic] 07SomeMWDev pushed 2 new commits to 03tech-notifications 13https://github.com/miraheze/MirahezeMagic/compare/ad6711df7157...4ecebd56bc97 [15:38:59] 02MirahezeMagic/03tech-notifications 07SomeRandomDeveloper 032dae74f Send notifications via email by default [15:38:59] 02MirahezeMagic/03tech-notifications 07SomeRandomDeveloper 034ecebd5 Move to alert section [15:44:13] [02MirahezeMagic] 07SomeMWDev left a file comment in pull request #613 034ecebd5: Mailhog doesn't render HTML emails, but it's definitely used: […] 13https://github.com/miraheze/MirahezeMagic/pull/613#discussion_r2617262564 [15:47:23] [02MirahezeMagic] 07SomeMWDev pushed 1 new commit to 03tech-notifications 13https://github.com/miraheze/MirahezeMagic/commit/560a86b79ecdb80c1217e3a7b8c6048a37d58cf6 [15:47:23] 02MirahezeMagic/03tech-notifications 07SomeRandomDeveloper 03560a86b Use --message parameter instead of stdin if provided [15:50:08] !log [somerandomdeveloper@test151] starting deploy of {'folders': '1.45/extensions/MirahezeMagic'} to test151 [15:50:09] !log [somerandomdeveloper@test151] finished deploy of {'folders': '1.45/extensions/MirahezeMagic'} to test151 - SUCCESS in 0s [15:50:13] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [15:50:16] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [15:52:21] !log [somerandomdeveloper@test151] sudo -u www-data php /srv/mediawiki/1.45/maintenance/run.php MirahezeMagic:NotifyWikiUsers --wiki=randomstuffwikibeta --permission managewiki-extensions --header 'Test Tech Notification' --link 'https://issue-tracker.miraheze.org/' --message 'This is a test notification.' (END - exit=0) [15:52:26] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [15:52:41] [1/2] nice, works [15:52:42] [2/2] https://cdn.discordapp.com/attachments/808001911868489748/1449791214030164105/image.png?ex=69402ec9&is=693edd49&hm=1a6f04d3b087cb5ca27c46a5ace985a02e6d1dcfb0ce7ede1824e9d70122a264& [15:59:29] miraheze/MirahezeMagic - SomeMWDev the build passed. [16:09:08] [02MirahezeMagic] 07SomeMWDev pushed 1 new commit to 03tech-notifications 13https://github.com/miraheze/MirahezeMagic/commit/63ca0e8e4d9b48586fa5ca396e8f98b7a098163f [16:09:10] 02MirahezeMagic/03tech-notifications 07SomeRandomDeveloper 0363ca0e8 return early if no groups were found [16:21:07] miraheze/MirahezeMagic - SomeMWDev the build passed. [16:23:50] !log [somerandomdeveloper@test151] starting deploy of {'upgrade_vendor': True, 'world': True, 'l10n': True, 'versions': '1.44'} to test151 [16:23:54] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:29:56] !log [somerandomdeveloper@test151] finished deploy of {'upgrade_vendor': True, 'world': True, 'l10n': True, 'versions': '1.44'} to test151 - SUCCESS in 366s [16:30:00] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:40:58] !log [somerandomdeveloper@test151] starting deploy of {'upgrade_vendor': True, 'world': True, 'l10n': True, 'versions': '1.45'} to test151 [16:41:02] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:41:26] !log [somerandomdeveloper@test151] DEPLOY ABORTED: Non-Zero Exit Code in prep, see output. [16:41:30] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:41:56] !log [somerandomdeveloper@test151] starting deploy of {'upgrade_vendor': True, 'world': True, 'l10n': True, 'versions': '1.45'} to test151 [16:41:57] !log [somerandomdeveloper@test151] DEPLOY ABORTED: Canary check failed for meta.mirabeta.org@localhost [16:42:00] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:42:04] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:42:21] !log [somerandomdeveloper@test151] starting deploy of {'upgrade_vendor': True, 'world': True, 'l10n': True, 'force': True, 'versions': '1.45'} to test151 [16:42:25] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:42:46] beta is temporarily down because mirahezemagic's extension.json had local changes that were not deployed... will be fixed shortly [16:43:21] fixed [16:48:06] !log [somerandomdeveloper@test151] finished deploy of {'upgrade_vendor': True, 'world': True, 'l10n': True, 'force': True, 'versions': '1.45'} to test151 - SUCCESS in 345s [16:48:10] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:06:37] PROBLEM - mw183 MediaWiki Rendering on mw183 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 8191 bytes in 0.013 second response time [19:06:43] PROBLEM - mw202 HTTPS on mw202 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 502 [19:06:45] PROBLEM - cp201 Varnish Backends on cp201 is CRITICAL: 9 backends are down. mw151 mw161 mw162 mw181 mw153 mw163 mw192 mw193 mw201 [19:06:50] PROBLEM - mw203 MediaWiki Rendering on mw203 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 8191 bytes in 0.014 second response time [19:06:50] PROBLEM - mw193 HTTPS on mw193 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 502 [19:06:54] PROBLEM - mw183 HTTPS on mw183 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 502 [19:06:57] PROBLEM - mw182 MediaWiki Rendering on mw182 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 8191 bytes in 0.011 second response time [19:06:58] PROBLEM - cp171 Varnish Backends on cp171 is CRITICAL: 9 backends are down. mw151 mw161 mw162 mw181 mw153 mw163 mw192 mw193 mw201 [19:07:03] PROBLEM - mw201 MediaWiki Rendering on mw201 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 8191 bytes in 0.012 second response time [19:07:07] PROBLEM - cp201 HTTPS on cp201 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 502 [19:07:08] PROBLEM - mw152 HTTPS on mw152 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 502 [19:07:08] PROBLEM - cp191 Varnish Backends on cp191 is CRITICAL: 9 backends are down. mw151 mw161 mw162 mw181 mw153 mw163 mw192 mw193 mw201 [19:07:17] PROBLEM - mw172 HTTPS on mw172 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 502 [19:07:21] PROBLEM - mw193 MediaWiki Rendering on mw193 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 8191 bytes in 0.012 second response time [19:07:22] PROBLEM - mw151 MediaWiki Rendering on mw151 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 8191 bytes in 0.012 second response time [19:07:24] PROBLEM - mw162 MediaWiki Rendering on mw162 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:07:28] PROBLEM - mw172 MediaWiki Rendering on mw172 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 8191 bytes in 0.012 second response time [19:07:33] PROBLEM - cp191 HTTPS on cp191 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 502 [19:07:36] PROBLEM - mw153 HTTPS on mw153 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 502 [19:07:38] PROBLEM - cp171 HTTPS on cp171 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 502 [19:07:38] PROBLEM - mw182 HTTPS on mw182 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 502 [19:07:40] PROBLEM - mw192 MediaWiki Rendering on mw192 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:07:41] PROBLEM - mw171 MediaWiki Rendering on mw171 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 8191 bytes in 0.014 second response time [19:07:41] PROBLEM - mw203 HTTPS on mw203 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 502 [19:07:41] PROBLEM - mw192 HTTPS on mw192 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10004 milliseconds with 0 bytes received [19:07:43] PROBLEM - mw201 HTTPS on mw201 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 502 [19:07:45] PROBLEM - cp161 HTTPS on cp161 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10004 milliseconds with 0 bytes received [19:07:45] PROBLEM - mw163 HTTPS on mw163 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10003 milliseconds with 0 bytes received [19:07:46] PROBLEM - mw191 HTTPS on mw191 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 502 [19:07:47] PROBLEM - mw202 MediaWiki Rendering on mw202 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 8191 bytes in 0.011 second response time [19:07:50] ffs [19:07:56] PROBLEM - mw181 MediaWiki Rendering on mw181 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 8191 bytes in 0.012 second response time [19:07:58] PROBLEM - mw191 MediaWiki Rendering on mw191 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 8191 bytes in 0.015 second response time [19:07:59] PROBLEM - mw161 HTTPS on mw161 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 502 [19:08:07] PROBLEM - mw181 HTTPS on mw181 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 502 [19:08:07] PROBLEM - mw163 MediaWiki Rendering on mw163 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:08:20] PROBLEM - mw153 MediaWiki Rendering on mw153 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 8191 bytes in 0.011 second response time [19:08:25] PROBLEM - mw173 HTTPS on mw173 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 502 [19:08:27] PROBLEM - mw171 HTTPS on mw171 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 502 [19:08:27] which one is it today [19:08:31] PROBLEM - cp161 HTTP 4xx/5xx ERROR Rate on cp161 is CRITICAL: CRITICAL - NGINX Error Rate is 90% [19:08:37] PROBLEM - mw151 HTTPS on mw151 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 502 [19:08:38] PROBLEM - mw173 MediaWiki Rendering on mw173 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 8191 bytes in 0.014 second response time [19:08:42] PROBLEM - mw161 MediaWiki Rendering on mw161 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 8191 bytes in 0.012 second response time [19:08:50] PROBLEM - mw162 HTTPS on mw162 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 502 [19:08:54] PROBLEM - mw152 MediaWiki Rendering on mw152 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 8191 bytes in 0.011 second response time [19:09:17] PROBLEM - puppet181 Check unit status of listdomains_github_push on puppet181 is CRITICAL: CRITICAL: Status of the systemd unit listdomains_github_push [19:10:13] all dbs appear to be fine [19:11:15] disk space on db201 is critical [19:11:27] @rhinosf1 @paladox [19:12:55] Oh [19:13:12] i can't ssh into it [19:13:21] nvm i can now [19:14:41] I don't really have the time to look into this rn [19:14:44] is anybody else available [19:15:06] I'm looking [19:15:11] thank you [19:15:17] RECOVERY - puppet181 Check unit status of listdomains_github_push on puppet181 is OK: OK: Status of the systemd unit listdomains_github_push [19:15:19] [1/13] ``` [19:15:19] [2/13] [somerandomdeveloper@db201:~]$ sudo du -sh /srv/mariadb/* | sort -rh | head -n 10 [19:15:19] [3/13] 97G /srv/mariadb/parsercache [19:15:20] [4/13] 2.1G /srv/mariadb/ib_logfile0 [19:15:20] [5/13] 1.1G /srv/mariadb/mysql-bin.000316 [19:15:20] [6/13] 1.1G /srv/mariadb/mysql-bin.000315 [19:15:20] [7/13] 1.1G /srv/mariadb/mysql-bin.000314 [19:15:21] [8/13] 1.1G /srv/mariadb/mysql-bin.000313 [19:15:21] [9/13] 1.1G /srv/mariadb/mysql-bin.000312 [19:15:21] [10/13] 1.1G /srv/mariadb/mysql-bin.000311 [19:15:22] [11/13] 1.1G /srv/mariadb/mysql-bin.000310 [19:15:22] [12/13] 1.1G /srv/mariadb/mysql-bin.000309 [19:15:22] [13/13] ``` [19:16:40] Can a @MediaWiki Specialists depool db201 [19:16:53] on it [19:16:58] Oh is this the new pc db? [19:17:05] Fun [19:18:41] It doesn't look totally duff [19:18:58] It's 98% full not full [19:19:14] would that be the s1 cluster [19:19:17] PROBLEM - puppet181 Check unit status of listdomains_github_push on puppet181 is CRITICAL: CRITICAL: Status of the systemd unit listdomains_github_push [19:19:38] s1 is db192 and should probably never be despoiled [19:19:39] What is db201 serving [19:19:52] db201 is indeed only parsercache [19:19:54] nothing else [19:20:12] Can a full parsercache bring down the farm? [19:20:20] guh how would I depool it if it's not associated with a cluster [19:20:20] Don't even know if s1 depool is possible [19:20:25] Yes [19:20:29] You don't [19:20:48] ah cool I will go bully the users then [19:21:06] will announce shortly [19:21:10] we have moved the single point of failure to another single point of failure 😎 [19:21:16] Parser cache is 15 days [19:21:20] And we've ran out at 9 [19:21:22] great success [19:21:35] @agentisai how comfortable do you feel about expanding its disk [19:21:38] To like double [19:21:59] Cause even dropping PC to like 7 days im still going to need to wipe stuff [19:23:24] Who just did something [19:23:36] Cause disk has dropped from 98% to 59% [19:24:06] RECOVERY - db201 Disk Space on db201 is OK: DISK OK - free space: / 304225MiB (72.4% inode=100%); [19:24:37] RECOVERY - mw151 HTTPS on mw151 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4224 bytes in 0.062 second response time [19:24:37] RECOVERY - mw161 MediaWiki Rendering on mw161 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.178 second response time [19:24:41] RECOVERY - mw183 MediaWiki Rendering on mw183 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 4.267 second response time [19:24:42] RECOVERY - mw202 HTTPS on mw202 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4224 bytes in 4.690 second response time [19:24:42] RECOVERY - mw173 MediaWiki Rendering on mw173 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 3.662 second response time [19:24:44] RECOVERY - mw193 HTTPS on mw193 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4224 bytes in 0.063 second response time [19:24:45] RECOVERY - mw162 HTTPS on mw162 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4224 bytes in 0.056 second response time [19:24:53] RECOVERY - mw203 MediaWiki Rendering on mw203 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 3.111 second response time [19:24:53] RECOVERY - mw183 HTTPS on mw183 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4224 bytes in 2.490 second response time [19:24:56] RECOVERY - mw201 MediaWiki Rendering on mw201 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.213 second response time [19:24:56] RECOVERY - mw152 MediaWiki Rendering on mw152 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 2.103 second response time [19:25:01] RECOVERY - mw182 MediaWiki Rendering on mw182 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 3.589 second response time [19:25:02] RECOVERY - mw152 HTTPS on mw152 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4224 bytes in 0.719 second response time [19:25:03] We had 350GBs of bin logs… [19:25:07] RECOVERY - cp201 HTTPS on cp201 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4273 bytes in 0.062 second response time [19:25:12] RECOVERY - mw162 MediaWiki Rendering on mw162 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.187 second response time [19:25:17] RECOVERY - mw172 HTTPS on mw172 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4224 bytes in 0.070 second response time [19:25:17] RECOVERY - puppet181 Check unit status of listdomains_github_push on puppet181 is OK: OK: Status of the systemd unit listdomains_github_push [19:25:19] RECOVERY - mw163 HTTPS on mw163 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4224 bytes in 0.066 second response time [19:25:21] RECOVERY - mw193 MediaWiki Rendering on mw193 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.183 second response time [19:25:23] RECOVERY - mw151 MediaWiki Rendering on mw151 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.191 second response time [19:25:24] RECOVERY - mw192 MediaWiki Rendering on mw192 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.266 second response time [19:25:24] RECOVERY - mw192 HTTPS on mw192 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4224 bytes in 0.077 second response time [19:25:26] RECOVERY - cp161 HTTPS on cp161 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4328 bytes in 0.065 second response time [19:25:28] RECOVERY - mw172 MediaWiki Rendering on mw172 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.190 second response time [19:25:29] RECOVERY - mw153 HTTPS on mw153 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4224 bytes in 0.045 second response time [19:25:33] RECOVERY - cp191 HTTPS on cp191 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4295 bytes in 0.066 second response time [19:25:35] RECOVERY - mw201 HTTPS on mw201 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4224 bytes in 0.061 second response time [19:25:38] RECOVERY - cp171 HTTPS on cp171 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4273 bytes in 0.061 second response time [19:25:38] RECOVERY - mw182 HTTPS on mw182 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4224 bytes in 0.057 second response time [19:25:41] RECOVERY - mw203 HTTPS on mw203 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4224 bytes in 0.068 second response time [19:25:41] RECOVERY - mw171 MediaWiki Rendering on mw171 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.196 second response time [19:25:43] RECOVERY - mw163 MediaWiki Rendering on mw163 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.195 second response time [19:25:46] RECOVERY - mw191 HTTPS on mw191 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4224 bytes in 0.072 second response time [19:25:47] RECOVERY - mw202 MediaWiki Rendering on mw202 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.247 second response time [19:25:51] RECOVERY - mw161 HTTPS on mw161 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4224 bytes in 0.060 second response time [19:25:57] RECOVERY - mw181 MediaWiki Rendering on mw181 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.244 second response time [19:25:58] RECOVERY - mw191 MediaWiki Rendering on mw191 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.188 second response time [19:26:02] Bin logs should not be on for parser cache [19:26:07] RECOVERY - mw181 HTTPS on mw181 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4224 bytes in 0.070 second response time [19:26:09] I'm guessing you've switched them off [19:26:19] I remember Paladox saying he turned them off [19:26:20] RECOVERY - mw153 MediaWiki Rendering on mw153 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.189 second response time [19:26:25] RECOVERY - mw173 HTTPS on mw173 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4224 bytes in 0.056 second response time [19:26:27] RECOVERY - mw171 HTTPS on mw171 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4224 bytes in 0.050 second response time [19:26:27] seems to have accidentally been switched back on [19:26:31] RECOVERY - cp161 HTTP 4xx/5xx ERROR Rate on cp161 is OK: OK - NGINX Error Rate is 9% [19:26:45] RECOVERY - cp201 Varnish Backends on cp201 is OK: All 31 backends are healthy [19:26:46] Have we disabled it in the MariaDB config file or just cli level @agentisai [19:26:50] I don’t understand how db201 ran out [19:26:56] It has 500gb [19:26:58] RECOVERY - cp171 Varnish Backends on cp171 is OK: All 31 backends are healthy [19:27:08] RECOVERY - cp191 Varnish Backends on cp191 is OK: All 31 backends are healthy [19:27:11] @paladox it's disk has been rising pretty steadily ever since it was installed [19:27:28] I’ll have to check but it seems they were never disabled [19:27:45] That would be my guess off that graph [19:28:06] PROBLEM - cp201 Disk Space on cp201 is CRITICAL: DISK CRITICAL - free space: / 27169MiB (5% inode=99%); [19:28:06] They can't have been off for any meaningful length of time [19:28:13] suddenly reminded of the mass refreshLinks [19:28:13] why is she full again [19:28:22] @agentisai make sure the MariaDB config in puppet is set [19:28:27] Also !log whatever you did please [19:28:44] And next time talk out loud what you're thinking during an incident [19:28:57] I shouldn't be looking going somethings just changed and not know what [19:29:08] If necessary we can increase but I like to know why 500gb is almost out when it didn’t on db161 [19:29:15] !log cleared binlogs 0 to 316 on db201 to clear up 350GBs of space [19:29:19] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:29:35] Oh it was the bin logs? [19:29:38] Yes [19:31:20] Thanks for fixing it [19:31:56] Have we deleted bin logs or actually made sure they are off forever ? [19:32:20] Let’s make sure it’s off [19:32:30] (I’m mobile otherwise I would) [19:32:55] Looking at puppet, only db182 has them off [19:33:56] [02puppet] 07RhinosF1 created 03RhinosF1-patch-3 (+1 new commit) 13https://github.com/miraheze/puppet/commit/698bef12b338 [19:33:56] 02puppet/03RhinosF1-patch-3 07RhinosF1 03698bef1 disable binlogs on db201 (parser cache db) [19:34:07] [02puppet] 07RhinosF1 opened pull request #4628: disable binlogs on db201 (parser cache db) (03main...03RhinosF1-patch-3) 13https://github.com/miraheze/puppet/pull/4628 [19:34:13] https://github.com/miraheze/puppet/pull/4628 [19:34:13] [GitHub] [miraheze/puppet #4628] open PR by RhinosF1, created 8 seconds ago: disable binlogs on db201 (parser cache db) | (empty comment) [19:34:15] [02puppet] 07coderabbitai[bot] commented on pull request #4628:

[…] 13https://github.com/miraheze/puppet/pull/4628#issuecomment-3651878171 [19:34:17] Can someone approve [19:34:23] https://github.com/miraheze/puppet/blob/main/hieradata/hosts/db201.yaml#L7 [19:34:24] [GitHub] [miraheze/puppet] hieradata/hosts/db201.yaml @ main | L7: mariadb::config::sync_binlog: '0' [19:34:35] I think the config doesn’t do what it says on the tin [19:35:04] It just removes the configs relying on defaults [19:35:20] That config was added by you @paladox [19:35:42] Yeh, I realised when looking just now [19:39:03] You can add disable_log_bin to actually disable it [19:39:08] It was made default [19:39:17] how did nobody notice that disk space on db201 went critical earlier today [19:39:29] maybe we have too many cache proxy disk space warnings... [19:39:32] It says no parameter https://dba.stackexchange.com/questions/72770/disable-mysql-binary-logging-with-log-bin-variable [19:39:45] Fairly sure this has been a learning from before [19:40:22] if a db runs out of disk space we should have a big red warning that pings the infra role or sth like that [19:40:33] because that is probably never a false positive [19:40:47] [02puppet] 07RhinosF1 created 03RhinosF1-patch-4 (+1 new commit) 13https://github.com/miraheze/puppet/commit/c4a72ba5b88b [19:40:48] 02puppet/03RhinosF1-patch-4 07RhinosF1 03c4a72ba Fix disable binary logging [19:41:09] [02puppet] 07RhinosF1 opened pull request #4629: Fix disable binary logging (03main...03RhinosF1-patch-4) 13https://github.com/miraheze/puppet/pull/4629 [19:41:15] [02puppet] 07coderabbitai[bot] commented on pull request #4629:

[…] 13https://github.com/miraheze/puppet/pull/4629#issuecomment-3651888437 [19:41:19] @paladox https://github.com/miraheze/puppet/pull/4629 [19:41:19] [GitHub] [miraheze/puppet #4629] open PR by RhinosF1, created 12 seconds ago: Fix disable binary logging | https://dba.stackexchange.com/questions/72770/disable-mysql-binary-logging-with-log-bin-variable [19:42:20] @paladox / @agentisai suggest we merge both of them, restart MariaDB then flush all bin logs and ensure they stay gone [19:44:11] @paladox be explicit [19:44:19] A thumbs up does not mean yes go ahead [19:44:56] i'm still reviewing, you said you suggest we merge and thumbs up is going yeh i acknowledge and will do so [19:45:26] i'm now off mobile and can ssh in [19:45:38] just got back [19:46:45] I am ready whenever you are [19:48:37] i think it needs a = 1 [19:49:16] The stack overflow link explicitly says no parameters [19:49:21] And gives that as the example [19:49:49] yeh but that's for mysql, from a quick search it's suggesting = 1 but i think we can try your change first and see [19:49:59] [02puppet] 07paladox merged 07RhinosF1's pull request #4629: Fix disable binary logging (03main...03RhinosF1-patch-4) 13https://github.com/miraheze/puppet/pull/4629 [19:50:00] [02puppet] 07paladox pushed 1 new commit to 03main 13https://github.com/miraheze/puppet/commit/a564457fa2d55585b9b6cb10ccf8ee6de93e6d06 [19:50:01] 02puppet/03main 07RhinosF1 03a564457 Fix disable binary logging (#4629)… [19:50:03] [02puppet] 07paladox 04deleted 03RhinosF1-patch-4 at 03c4a72ba 13https://api.github.com/repos/miraheze/puppet/commit/c4a72ba [19:50:09] @paladox you deploying or me? [19:50:24] i will deploy [19:50:42] for restarting can i just restart? [19:50:46] @paladox https://github.com/miraheze/puppet/pull/4628 needs merging too [19:50:46] [GitHub] [miraheze/puppet #4628] open PR by RhinosF1, created 16 minutes, 41 seconds ago: disable binlogs on db201 (parser cache db) | (empty comment) [19:50:56] @paladox yes once puppet has ran off course [19:50:59]