[00:01:22] [02statichelp] 07WikiTideBot pushed 1 new commit to 03main 13https://github.com/miraheze/statichelp/commit/ce66398e59f71bdda63aa5c049de0347e33b3690 [00:01:22] 02statichelp/03main 07WikiTideBot 03ce66398 Bot: Auto-update Tech namespace pages 2025-07-20 00:01:20 [00:02:12] PROBLEM - cp161 HTTP 4xx/5xx ERROR Rate on cp161 is WARNING: WARNING - NGINX Error Rate is 48% [00:06:23] RECOVERY - cp191 Disk Space on cp191 is OK: DISK OK - free space: / 57829MiB (12% inode=99%); [00:07:00] RECOVERY - cp171 Disk Space on cp171 is OK: DISK OK - free space: / 57447MiB (12% inode=99%); [00:07:04] RECOVERY - cp201 Disk Space on cp201 is OK: DISK OK - free space: / 57171MiB (12% inode=99%); [00:08:12] RECOVERY - cp161 HTTP 4xx/5xx ERROR Rate on cp161 is OK: OK - NGINX Error Rate is 34% [01:32:13] [02DiscordNotifications] 07pskyechology opened pull request #106: Add image embed support, fix various bugs, and refactor several pieces of code (07miraheze:03main...07pskyechology:03refactor) 13https://github.com/miraheze/DiscordNotifications/pull/106 [01:32:20] [02DiscordNotifications] 07coderabbitai[bot] commented on pull request #106:
[…] 13https://github.com/miraheze/DiscordNotifications/pull/106#issuecomment-3092681115 [01:35:36] miraheze/DiscordNotifications - pskyechology the build has errored. [01:55:09] miraheze/DiscordNotifications - pskyechology the build has errored. [02:00:25] miraheze/DiscordNotifications - pskyechology the build has errored. [02:11:42] miraheze/DiscordNotifications - pskyechology the build has errored. [02:21:01] miraheze/DiscordNotifications - pskyechology the build has errored. [02:50:33] miraheze/DiscordNotifications - pskyechology the build has errored. [02:53:58] minor amount of build errors [02:54:24] only 1 was because my code was silly, trust [03:19:56] [Grafana] FIRING: The mediawiki JobQueue backlog is increasing by more than 100 jobs a minute over an extended time period https://grafana.wikitide.net/d/GtxbP1Xnk?from=1752977960000&orgId=1&to=1752981596681 [03:24:56] [Grafana] RESOLVED: MediaWiki Jobs are rapidly increasing https://grafana.wikitide.net/d/GtxbP1Xnk?from=1752978020000&orgId=1&to=1752981620000 [03:34:56] [Grafana] FIRING: The mediawiki job queue has more than 100000 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?from=1752978710000&orgId=1&to=1752982496684[Grafana] RESOLVED: MediaWiki Jobs are rapidly increasing https://grafana.wikitide.net/d/GtxbP1Xnk?from=1752978020000&orgId=1&to=1752981620000 [03:59:56] [Grafana] RESOLVED: High Job Queue Backlog https://grafana.wikitide.net/d/GtxbP1Xnk?from=1752978710000&orgId=1&to=1752983810000 [04:05:26] [02puppet] 07Universal-Omega created 03concurrency (+1 new commit) 13https://github.com/miraheze/puppet/commit/8057fbfb5f1f [04:05:27] 02puppet/03concurrency 07CosmicAlpha 038057fbf Increase concurrency to 100 for LocalGlobalUserPageCacheUpdateJob [04:05:32] [02puppet] 07Universal-Omega opened pull request #4452: Increase concurrency to 100 for LocalGlobalUserPageCacheUpdateJob (03main...03concurrency) 13https://github.com/miraheze/puppet/pull/4452 [04:05:39] [02puppet] 07coderabbitai[bot] commented on pull request #4452:
[…] 13https://github.com/miraheze/puppet/pull/4452#issuecomment-3093183760 [04:36:43] !log [universalomega@mwtask181] starting deploy of {'versions': ['1.43', '1.44'], 'upgrade_extensions': 'MirahezeMagic'} to all [04:36:46] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [04:37:32] !log [universalomega@mwtask181] finished deploy of {'versions': ['1.43', '1.44'], 'upgrade_extensions': 'MirahezeMagic'} to all - SUCCESS in 49s [04:37:35] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [07:34:49] PROBLEM - mwtask181 Current Load on mwtask181 is WARNING: LOAD WARNING - total load average: 20.99, 19.54, 14.62 [07:38:49] RECOVERY - mwtask181 Current Load on mwtask181 is OK: LOAD OK - total load average: 20.19, 19.98, 15.93 [07:42:49] PROBLEM - mwtask181 Current Load on mwtask181 is WARNING: LOAD WARNING - total load average: 20.47, 20.27, 16.98 [07:44:49] RECOVERY - mwtask181 Current Load on mwtask181 is OK: LOAD OK - total load average: 19.89, 20.22, 17.37 [08:06:21] first time i had a rename failure [08:06:38] Error occurred: Error in renaming Matomo references [08:06:39] Attempting rollback... [08:06:40] !log [blankeclair@mwtask181] sudo -u www-data php /srv/mediawiki/1.43/maintenance/run.php MirahezeMagic:RenameDatabase --wiki=loginwiki --rename --old=1819rpwiki --new=paxamericana1wiki --user='BlankEclair_(Miraheze)' (END - exit=256) [08:06:43] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [08:06:45] oh. [08:08:54] !log [blankeclair@mwtask181] sudo -u www-data php /srv/mediawiki/1.43/maintenance/run.php MirahezeMagic:RenameDatabase --wiki=loginwiki --old=1819rpwiki --new=paxamericana1wiki --user='BlankEclair_(Miraheze)' (END - exit=256) [08:08:57] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [08:11:03] !log loginwiki/c3/db171: DROP TABLE paxamericana1wiki; -- Verified that `SELECT TABLE_NAME FROM information_schema.TABLES WHERE TABLE_SCHEMA = 'paxamericana1wiki';` has 0 rows [08:11:06] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [08:11:15] !log addendum: i meant DROP DATABASE, oops [08:11:17] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [08:20:11] did you figure out why it failed @blankeclair ? [08:20:16] Error in renaming Matomo references [08:20:25] how did you fix it? [08:20:35] oh [08:20:39] you did two lol [08:20:50] or not [08:20:54] im tired lol [08:21:10] it was the same one twice [08:21:12] something broke but idk what exactlyyyyy [08:21:24] but it worked the second time? [08:21:29] oh, the second is the dry run [08:21:35] oh! [08:21:49] oh oh oh lol [08:21:57] I see what you did now. [08:22:03] Im guessing I broke it [08:22:18] https://github.com/miraheze/MatomoAnalytics/blob/058e7aa4173833605c5fdbbf9b898b6f8e9edeec/includes/MatomoAnalytics.php#L161 [08:22:19] [GitHub] [miraheze/MatomoAnalytics] includes/MatomoAnalytics.php @ 058e7aa4173833605c5fdbbf9b898b6f8e9edeec | L161:  if ( (string)$siteId === (string)self::getSiteID( $newDbName, disableCache: false ) ) { [08:22:21] why's this even false lol [08:22:24] woah, it shows the line [08:23:19] UO, wanna debug this? [08:23:22] https://issue-tracker.miraheze.org/T13973 [08:24:08] because if its not false than its always true. you want to compare the no cache with the cache to verify success. [08:24:16] ah ic [08:24:49] anyway I do guarantee it was https://github.com/miraheze/MatomoAnalytics/commit/582d696845803faea461556a9d462df24c00f25b that broke it though [08:24:50] [GitHub] [miraheze/MatomoAnalytics] Universal-Omega: Convert to QueryBuilder and other cleanup | 677 changes in 15 files | Authored 16 days, 13 hours ago | Committed 16 days, 13 hours ago [08:25:25] oh lol [08:25:31] guess i'll test on mirabeta then [08:25:37] I need to stop rewriting extensions in one PR and split them so we know which one breaks things lol [08:26:05] that one wasn't really a rewrite but still lol [08:26:38] !log [blankeclair@test151] sudo -u www-data php /srv/mediawiki/1.44/maintenance/run.php MirahezeMagic:RenameDatabase --wiki=loginwiki --rename --old=clairewikibeta --new=claire2wikibeta --user=BlankEclair (END - exit=0) [08:26:41] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [08:26:45] forgot about realms [08:27:06] !log [blankeclair@test151] sudo -u www-data php /srv/mediawiki/1.44/maintenance/run.php MirahezeMagic:RenameDatabase --wiki=loginwikibeta --rename --old=clairewikibeta --new=claire2wikibeta --user=BlankEclair (END - exit=256) [08:27:09] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [08:27:14] yep, matomo [08:28:15] i think you got it the other way 'round... [08:28:31] !log loginwikibeta: DROP DATABASE claire2wikibeta [08:28:34] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [08:30:49] [02MatomoAnalytics] 07BlankEclair created 03fix-rename (+1 new commit) 13https://github.com/miraheze/MatomoAnalytics/commit/f7406d8ce977 [08:30:49] 02MatomoAnalytics/03fix-rename 07Claire Elaina 03f7406d8 Fix renaming wikis… [08:31:01] Was just about to say lol [08:31:01] is there a non-crude way of testing this on mirabeta? [08:31:12] or do i checkout on the directory and run mwdeploy [08:31:52] just git pull it then mwdeploy with --force-upgrade [08:32:45] !log test151: git -C /srv/mediawiki-staging/1.44/extensions/MatomoAnalytics fetch origin fix-rename [08:32:47] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [08:32:48] !log test151: git -C /srv/mediawiki-staging/1.44/extensions/MatomoAnalytics checkout FETCH_HEAD [08:32:51] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [08:33:04] !log [blankeclair@test151] starting deploy of {'force_upgrade': True, 'versions': '1.44', 'upgrade_extensions': 'MatomoAnalytics'} to test151 [08:33:05] !log [blankeclair@test151] finished deploy of {'force_upgrade': True, 'versions': '1.44', 'upgrade_extensions': 'MatomoAnalytics'} to test151 - SUCCESS in 0s [08:33:07] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [08:33:10] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [08:33:27] !log [blankeclair@test151] sudo -u www-data php /srv/mediawiki/1.44/maintenance/run.php MirahezeMagic:RenameDatabase --wiki=loginwikibeta --rename --old=clairewikibeta --new=claire2wikibeta --user=BlankEclair (END - exit=256) [08:33:30] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [08:33:56] @blankeclair have we ever actually tried hyphens in db names btw? I can't remember ever trying it its just a legacy we can't do it lol [08:34:02] same tbh [08:34:04] but i wonder [08:34:11] can you have hyphens in mariadb names? [08:34:17] ugh i fucking don't get mwdeploy [08:34:45] !log [blankeclair@test151:/srv/mediawiki/1.44/extensions/MatomoAnalytics]$ sudo -u www-data rsync -vaP /srv/mediawiki{-staging,}/1.44/extensions/MatomoAnalytics/includes/ [08:34:48] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [08:35:09] !log loginwikibeta: DROP DATABASE claire2wikibeta; [08:35:12] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [08:35:34] !log [blankeclair@test151] sudo -u www-data php /srv/mediawiki/1.44/maintenance/run.php MirahezeMagic:RenameDatabase --wiki=loginwikibeta --rename --old=clairewikibeta --new=claire2wikibeta --user=BlankEclair (END - exit=256) [08:35:34] aaaaaaaaaaaaaaaaaa [08:35:36] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [08:35:52] very poorly it does it seems if you wrap it in backticks and always use backticks when referencing it but we could allow hyphens in subdomains and replace it with underscore in db names lol [08:36:05] touche [08:36:26] are matomo references even added on beta btw? [08:36:36] good question [08:36:39] but i'm trying [08:36:39] metawikibeta shows matomo for global production it seems lol [08:36:50] gg [08:37:00] ughhhhhh should i just test in prod [08:37:01] the global id 0 so it has every single production wiki on that one wiki lol [08:37:30] maybe [08:37:40] i'm blaming u if it breaks /j [08:37:47] not much can go wrong other than it still fails lol [08:37:55] [02MatomoAnalytics] 07BlankEclair opened pull request #167: Fix renaming wikis (03main...03fix-rename) 13https://github.com/miraheze/MatomoAnalytics/pull/167 [08:38:01] [02MatomoAnalytics] 07coderabbitai[bot] commented on pull request #167:
[…] 13https://github.com/miraheze/MatomoAnalytics/pull/167#issuecomment-3094357231 [08:39:16] who knew that renaming a wiki can be this tiring [08:39:35] now i lost my motivation to do other tasks xD [08:42:27] miraheze/MatomoAnalytics - BlankEclair the build passed. [08:42:37] [02MatomoAnalytics] 07BlankEclair pushed 1 new commit to 03main 13https://github.com/miraheze/MatomoAnalytics/commit/6d2fda4f77fd58bd6e2838cfb432f1427cf1c7d1 [08:42:37] 02MatomoAnalytics/03main 07Claire Elaina 036d2fda4 Fix renaming wikis (#167)… [08:42:39] [02MatomoAnalytics] 07BlankEclair merged pull request #167: Fix renaming wikis (03main...03fix-rename) 13https://github.com/miraheze/MatomoAnalytics/pull/167 [08:42:41] [02MatomoAnalytics] 07BlankEclair 04deleted 03fix-rename at 03f7406d8 13https://api.github.com/repos/miraheze/MatomoAnalytics/commit/f7406d8 [08:42:56] [Grafana] FIRING: The mediawiki JobQueue backlog is increasing by more than 100 jobs a minute over an extended time period https://grafana.wikitide.net/d/GtxbP1Xnk?from=1752997340000&orgId=1&to=1753000976682 [08:42:56] !log [blankeclair@mwtask181] starting deploy of {'versions': '1.43', 'upgrade_extensions': 'MatomoAnalytics'} to all [08:43:00] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [08:43:23] !log [blankeclair@mwtask181] finished deploy of {'versions': '1.43', 'upgrade_extensions': 'MatomoAnalytics'} to all - SUCCESS in 26s [08:43:25] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [08:43:27] we're parsing in [08:45:39] !log [blankeclair@mwtask181] sudo -u www-data php /srv/mediawiki/1.43/maintenance/run.php MirahezeMagic:RenameDatabase --wiki=loginwiki --rename --old=1819rpwiki --new=paxamericana1wiki --user='BlankEclair_(Miraheze)' (END - exit=0) [08:45:42] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [08:47:04] miraheze/MatomoAnalytics - BlankEclair the build passed. [08:47:56] [Grafana] RESOLVED: MediaWiki Jobs are rapidly increasing https://grafana.wikitide.net/d/GtxbP1Xnk?from=1752997400000&orgId=1&to=1753001000000 [09:09:49] !log [reception@mwtask181] sudo -u www-data php /srv/mediawiki/1.43/maintenance/run.php MirahezeMagic:RenameDatabase --wiki=loginwiki --rename --old=accishiwiki --new=vehiclesofbrickrigswiki --user=Reception123 (END - exit=256) [09:09:52] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [09:12:13] !log DROP DATABASE vehiclesofbrickrigswiki; (previously failed rename) [09:12:15] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [09:13:15] !log sudo -u www-data php /srv/mediawiki/1.43/maintenance/run.php reassignEdits --wiki=deadmau5wiki Wmyttmlimvty "It's moon" [09:13:18] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [09:14:21] !log [reception@mwtask181] sudo -u www-data php /srv/mediawiki/1.43/maintenance/run.php MirahezeMagic:RenameDatabase --wiki=loginwiki --rename --old=accishiwiki --new=vehiclesofbrickrigswiki --user=Reception123 (END - exit=0) [09:14:24] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [09:17:05] !log [reception@mwtask181] sudo -u www-data php /srv/mediawiki/1.43/maintenance/run.php MirahezeMagic:RenameDatabase --wiki=loginwiki --rename --old=synergycancerprojectwiki --new=synergycancerprotocolwiki --user=Reception123 (END - exit=0) [09:17:08] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [09:18:11] !log [reception@mwtask181] sudo -u www-data php /srv/mediawiki/1.43/maintenance/run.php MirahezeMagic:RenameDatabase --wiki=loginwiki --rename --old=telefoniaipredsessebastiansaavedrawiki --new=voipsaavedrawiki --user=Reception123 (END - exit=256) [09:18:13] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [09:21:17] !log [reception@mwtask181] sudo -u www-data php /srv/mediawiki/1.43/maintenance/run.php MirahezeMagic:RenameDatabase --wiki=loginwiki --rename --old=telefoniaipredessebastiansaavedrawiki --new=voipsaavedrawiki --user=Reception123 (END - exit=0) [09:21:20] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [09:23:02] !log [reception@mwtask181] sudo -u www-data php /srv/mediawiki/1.43/maintenance/run.php MirahezeMagic:RenameDatabase --wiki=loginwiki --rename --old=forkedaltversewiki --new=allohistoricaversaliswiki --user=Reception123 (END - exit=256) [09:23:05] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [09:23:58] !log MariaDB [(none)]> DROP DATABASE allohistoricaversaliswiki; (part of previously failed rename) [09:24:10] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [09:25:36] !log [reception@mwtask181] sudo -u www-data php /srv/mediawiki/1.43/maintenance/run.php MirahezeMagic:RenameDatabase --wiki=loginwiki --rename --old=forkedaltversewiki --new=allohistoricaversaliswiki --user=Reception123 (END - exit=0) [09:25:39] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [09:51:04] PROBLEM - cp201 Disk Space on cp201 is WARNING: DISK WARNING - free space: / 49840MiB (10% inode=99%); [10:01:27] miraheze/mw-config - Universal-Omega the build passed. [10:04:21] !log [universalomega@mwtask181] starting deploy of {'pull': 'config', 'config': True} to all [10:04:24] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [10:04:37] miraheze/mw-config - Universal-Omega the build passed. [10:04:46] !log [universalomega@mwtask181] finished deploy of {'pull': 'config', 'config': True} to all - SUCCESS in 25s [10:04:49] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [10:17:00] PROBLEM - cp171 Disk Space on cp171 is WARNING: DISK WARNING - free space: / 49872MiB (10% inode=99%); [10:36:23] PROBLEM - cp191 Disk Space on cp191 is WARNING: DISK WARNING - free space: / 49942MiB (10% inode=99%); [12:22:42] PROBLEM - wiki.frostforgenations.xyz - Cloudflare on sslhost is CRITICAL: Name or service not knownHTTP CRITICAL - Unable to open TCP socket [12:52:37] PROBLEM - wiki.frostforgenations.xyz - Cloudflare on sslhost is WARNING: WARNING - Certificate 'wiki.frostforgenations.xyz' expires in 21 day(s) (Sun 10 Aug 2025 10:39:35 PM GMT +0000).