[00:01:16] [02statichelp] 07WikiTideBot pushed 1 new commit to 03main 13https://github.com/miraheze/statichelp/commit/068420cd2376e3ef1ab84c22af484786b1365ee3 [00:01:16] 02statichelp/03main 07WikiTideBot 03068420c Bot: Auto-update Tech namespace pages 2026-01-30 00:01:13 [00:19:24] RECOVERY - cp201 Disk Space on cp201 is OK: DISK OK - free space: / 72737MiB (16.0% inode=100%); [00:27:50] RECOVERY - cp171 Disk Space on cp171 is OK: DISK OK - free space: / 72139MiB (15.9% inode=100%); [00:29:21] RECOVERY - cp191 Disk Space on cp191 is OK: DISK OK - free space: / 72850MiB (16.0% inode=100%); [00:57:30] PROBLEM - mwtask181 Check unit status of mediawiki_job_generate-sitemap-index on mwtask181 is CRITICAL: CRITICAL: Status of the systemd unit mediawiki_job_generate-sitemap-index [02:48:10] PROBLEM - test151 PowerDNS Recursor on test151 is CRITICAL: No response from DNS 127.0.0.1 [02:50:36] PROBLEM - test151 Puppet on test151 is WARNING: WARNING: Puppet is currently disabled, message: Universal Omega, last run 17 minutes ago with 0 failures [02:52:34] RECOVERY - test151 Puppet on test151 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [02:54:56] RECOVERY - test151 PowerDNS Recursor on test151 is OK: DNS OK: 0.031 seconds response time. test151.fsslc.wtnet returns 10.0.15.118 [03:02:41] [02puppet] 07Universal-Omega pushed 1 new commit to 03main 13https://github.com/miraheze/puppet/commit/7d13079988ec8f7299d116231ea91b8da89ffb9d [03:02:41] 02puppet/03main 07CosmicAlpha 037d13079 beta: use internal [03:08:22] [02puppet] 07Universal-Omega pushed 1 new commit to 03main 13https://github.com/miraheze/puppet/commit/2b68f98c19ba263455588cd6d9375be36f8b83be [03:08:23] 02puppet/03main 07CosmicAlpha 032b68f98 beta: no ipv6 [03:22:21] PROBLEM - ns1 Puppet on ns1 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_dns] [03:29:07] PROBLEM - ns1 NTP time on ns1 is CRITICAL: NTP CRITICAL: No response from NTP server [03:31:05] RECOVERY - ns1 NTP time on ns1 is OK: NTP OK: Offset 0.000296652317 secs [03:36:13] RECOVERY - ns1 Puppet on ns1 is OK: OK: Puppet is currently enabled, last run 10 seconds ago with 0 failures [03:39:29] !log fixed some networking issues on ns1 [03:39:31] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:42:42] RECOVERY - mw173 Check unit status of php8.2-fpm_check_restart on mw173 is OK: OK: Status of the systemd unit php8.2-fpm_check_restart [04:21:10] !log fixed some networking issues on ns2 [04:21:13] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [04:35:09] !log fixed hostname on mattermost2 [04:35:10] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [04:35:20] [02ssl] 07WikiTideBot pushed 1 new commit to 03main 13https://github.com/miraheze/ssl/commit/867d1ff7e2a262b1042e9dc800be27f15eecc8e6 [04:35:20] 02ssl/03main 07WikiTideBot 03867d1ff Bot: Auto-update domain lists [11:11:54] PROBLEM - test151 MediaWiki Rendering on test151 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:13:50] RECOVERY - test151 MediaWiki Rendering on test151 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.589 second response time [12:57:20] PROBLEM - puppet181 Check unit status of listdomains_github_push on puppet181 is CRITICAL: CRITICAL: Status of the systemd unit listdomains_github_push [13:15:20] RECOVERY - puppet181 Check unit status of listdomains_github_push on puppet181 is OK: OK: Status of the systemd unit listdomains_github_push [14:20:28] Just won $2,500 on KAIWIN.CC with promo code: GIFT πŸ’°πŸ’Έ @everyone [14:56:46] PROBLEM - test151 MediaWiki Rendering on test151 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:58:41] RECOVERY - test151 MediaWiki Rendering on test151 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.527 second response time [15:46:51] !log [skye@mwtask171] sudo -u www-data php /srv/mediawiki/1.44/maintenance/run.php MirahezeMagic:RenameDatabase --wiki=loginwiki --rename --old=tftworoguelikewiki --new=tf2roguelikewiki --user=Skye (END - exit=0) [15:46:53] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [15:47:15] !log [skye@mwtask171] sudo -u www-data php /srv/mediawiki/1.44/maintenance/run.php CreateWiki:SetContainersAccess --wiki=TF2Roguelikewiki (END - exit=0) [15:47:17] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [15:50:00] !log [skye@mwtask171] sudo -u www-data php /srv/mediawiki/1.44/maintenance/run.php MirahezeMagic:ResetWiki --wiki=loginwiki --dbname=payloadrobloxwiki --requester=Yalaicon (END - exit=0) [15:50:02] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:35:19] !log Cloudflare: Moved an ASN challenge to custom rules to allow for a static exemption [16:35:21] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:46:05] !log disable OATHAuth [16:46:07] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:46:47] might wanna specify that cause otherwise it feels like you disabled it for everyone [17:16:32] [02puppet] 07pskyechology opened pull request #4724: Map @pskyechology to Skye (07miraheze:03main...07pskyechology:03skye) 13https://github.com/miraheze/puppet/pull/4724 [17:16:41] [02puppet] 07coderabbitai[bot] commented on pull request #4724: - [ ] πŸ” Trigger a full review […] 13https://github.com/miraheze/puppet/pull/4724#issuecomment-3824799585 [17:19:54] miraheze/puppet - pskyechology the build passed. [17:31:19] !log disable OATHAuth meant for WebAuthn only (thanks to @pskyechology for reminding me I logged half a sentence) [17:31:21] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:36:08] [02mw-config] 07Universal-Omega pushed 1 new commit to 03sul3-support 13https://github.com/miraheze/mw-config/commit/b023097cd9fc7bc58954e90264014709f7c819ca [17:36:08] 02mw-config/03sul3-support 07CosmicAlpha 03b023097 Merge branch 'main' into sul3-support [17:37:06] miraheze/mw-config - Universal-Omega the build passed. [17:39:45] !log [skye@mwtask171] sudo -u www-data php /srv/mediawiki/1.44/maintenance/run.php purgeParserCache --wiki=payloadrobloxwiki --age=1 (END - exit=0) [17:39:46] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:00:13] [02mw-config] 07Universal-Omega merged pull request #6220: Migrate to SUL3 (03main...03sul3-support) 13https://github.com/miraheze/mw-config/pull/6220 [18:00:13] [02mw-config] 07Universal-Omega pushed 1 new commit to 03main 13https://github.com/miraheze/mw-config/commit/dbba0abdd31f6176136a9d588c116680efc3601e [18:00:13] 02mw-config/03main 07CosmicAlpha 03dbba0ab Migrate to SUL3 [18:00:15] [02mw-config] 07Universal-Omega 04deleted 03sul3-support at 03b023097 13https://api.github.com/repos/miraheze/mw-config/commit/b023097 [18:01:08] !log [universalomega@mwtask181] starting deploy of {'pull': 'config', 'config': True, 'force': True} to all [18:01:10] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:01:16] miraheze/mw-config - Universal-Omega the build passed. [18:01:31] !log [universalomega@mwtask181] finished deploy of {'pull': 'config', 'config': True, 'force': True} to all - SUCCESS in 23s [18:01:34] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:13:54] [02puppet] 07Universal-Omega created 03sul3-load (+1 new commit) 13https://github.com/miraheze/puppet/commit/0eaf48d5adac [18:13:54] 02puppet/03sul3-load 07CosmicAlpha 030eaf48d Allow load.php directly from SUL3 domain [18:14:00] [02puppet] 07Universal-Omega opened pull request #4725: Allow load.php directly from SUL3 domain (03main...03sul3-load) 13https://github.com/miraheze/puppet/pull/4725 [18:14:06] [02puppet] 07coderabbitai[bot] commented on pull request #4725: - [ ] πŸ” Trigger a full review […] 13https://github.com/miraheze/puppet/pull/4725#issuecomment-3825020205 [18:14:45] [02puppet] 07Universal-Omega merged pull request #4725: Allow load.php directly from SUL3 domain (03main...03sul3-load) 13https://github.com/miraheze/puppet/pull/4725 [18:14:46] [02puppet] 07Universal-Omega pushed 1 new commit to 03main 13https://github.com/miraheze/puppet/commit/6591f0ca98d68df79f08f40596a5a37a0333765f [18:14:46] 02puppet/03main 07CosmicAlpha 036591f0c Allow load.php directly from SUL3 domain [18:14:47] [02puppet] 07Universal-Omega 04deleted 03sul3-load at 030eaf48d 13https://api.github.com/repos/miraheze/puppet/commit/0eaf48d [18:27:44] [02mw-config] 07Universal-Omega created 03sul3-load (+1 new commit) 13https://github.com/miraheze/mw-config/commit/3ecb194f0979 [18:27:44] 02mw-config/03sul3-load 07CosmicAlpha 033ecb194 Allow load.php directly from SUL3 domain [18:27:50] [02mw-config] 07Universal-Omega opened pull request #6256: Allow load.php directly from SUL3 domain (03main...03sul3-load) 13https://github.com/miraheze/mw-config/pull/6256 [18:28:45] miraheze/mw-config - Universal-Omega the build passed. [18:30:52] [02mw-config] 07YTFGolf opened pull request #6257: T14875: add template- and founder-protection levels to mcsosirswiki (07miraheze:03main...07YTFGolf:03main) 13https://github.com/miraheze/mw-config/pull/6257 [18:31:48] miraheze/mw-config - YTFGolf the build passed. [18:32:48] [02mw-config] 07Universal-Omega merged pull request #6256: Allow load.php directly from SUL3 domain (03main...03sul3-load) 13https://github.com/miraheze/mw-config/pull/6256 [18:32:48] [02mw-config] 07Universal-Omega pushed 1 new commit to 03main 13https://github.com/miraheze/mw-config/commit/030f8584c502c34d375e3b2dfacd20cec8eeb776 [18:32:48] 02mw-config/03main 07CosmicAlpha 03030f858 Allow load.php directly from SUL3 domain [18:32:49] [02mw-config] 07Universal-Omega 04deleted 03sul3-load at 033ecb194 13https://api.github.com/repos/miraheze/mw-config/commit/3ecb194 [18:33:41] miraheze/mw-config - Universal-Omega the build passed. [18:33:46] [02mw-config] 07Universal-Omega created 03revert-6256-sul3-load (+1 new commit) 13https://github.com/miraheze/mw-config/commit/f5f725bc7a94 [18:33:46] 02mw-config/03revert-6256-sul3-load 07CosmicAlpha 03f5f725b Revert "Allow load.php directly from SUL3 domain"… [18:33:51] [02mw-config] 07Universal-Omega opened pull request #6258: Revert "Allow load.php directly from SUL3 domain" (03main...03revert-6256-sul3-load) 13https://github.com/miraheze/mw-config/pull/6258 [18:34:45] miraheze/mw-config - Universal-Omega the build passed. [18:38:36] PROBLEM - test151 Puppet on test151 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[nginx] [18:43:46] [02mw-config] 07Universal-Omega pushed 1 new commit to 03revert-6256-sul3-load 13https://github.com/miraheze/mw-config/commit/6d4cc675937c3afb3b77ea555604d62b3b3fa991 [18:43:47] 02mw-config/03revert-6256-sul3-load 07CosmicAlpha 036d4cc67 Update [18:44:20] [02mw-config] 07github-actions[bot] pushed 1 new commit to 03revert-6256-sul3-load 13https://github.com/miraheze/mw-config/commit/d21346910d2b6808f472c7f2d49aa6786bfd6aeb [18:44:21] 02mw-config/03revert-6256-sul3-load 07github-actions 03d213469 CI: lint code to MediaWiki standards… [18:44:53] miraheze/mw-config - Universal-Omega the build passed. [18:46:17] [02mw-config] 07Universal-Omega retitled PR #6258: "Revert "Allow load.php directly from SUL3 domain"" ➜ "Use main domain for load.php" 13https://github.com/miraheze/mw-config/pull/6258 [18:46:42] [02mw-config] 07Universal-Omega merged pull request #6258: Use main domain for load.php (03main...03revert-6256-sul3-load) 13https://github.com/miraheze/mw-config/pull/6258 [18:46:44] [02mw-config] 07Universal-Omega pushed 1 new commit to 03main 13https://github.com/miraheze/mw-config/commit/fab4dc336744b3949008389b1f755ec366983271 [18:46:46] [02mw-config] 07Universal-Omega 04deleted 03revert-6256-sul3-load at 03d213469 13https://api.github.com/repos/miraheze/mw-config/commit/d213469 [18:46:47] 02mw-config/03main 07CosmicAlpha 03fab4dc3 Use main domain for load.php [18:47:04] !log [universalomega@mwtask181] starting deploy of {'pull': 'config', 'config': True} to all [18:47:06] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:47:33] !log [universalomega@mwtask181] finished deploy of {'pull': 'config', 'config': True} to all - SUCCESS in 28s [18:47:35] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:47:37] miraheze/mw-config - Universal-Omega the build passed. [18:48:50] [02puppet] 07Universal-Omega created 03revert-4725-sul3-load (+1 new commit) 13https://github.com/miraheze/puppet/commit/cc970c37587d [18:48:50] 02puppet/03revert-4725-sul3-load 07CosmicAlpha 03cc970c3 Revert "Allow load.php directly from SUL3 domain"… [18:48:54] [02puppet] 07Universal-Omega opened pull request #4726: Revert "Allow load.php directly from SUL3 domain" (03main...03revert-4725-sul3-load) 13https://github.com/miraheze/puppet/pull/4726 [18:49:04] [02puppet] 07coderabbitai[bot] commented on pull request #4726: - [ ] πŸ” Trigger a full review […] 13https://github.com/miraheze/puppet/pull/4726#issuecomment-3825156640 [18:49:09] [02puppet] 07Universal-Omega merged pull request #4726: Revert "Allow load.php directly from SUL3 domain" (03main...03revert-4725-sul3-load) 13https://github.com/miraheze/puppet/pull/4726 [18:49:09] [02puppet] 07Universal-Omega pushed 1 new commit to 03main 13https://github.com/miraheze/puppet/commit/44e19fe6966783d89ac494908ac3cdc6888785b0 [18:49:09] 02puppet/03main 07CosmicAlpha 0344e19fe Revert "Allow load.php directly from SUL3 domain"… [18:49:11] [02puppet] 07Universal-Omega 04deleted 03revert-4725-sul3-load at 03cc970c3 13https://api.github.com/repos/miraheze/puppet/commit/cc970c3 [19:01:25] [02mw-config] 07Universal-Omega created 03sul3-fix (+1 new commit) 13https://github.com/miraheze/mw-config/commit/6b7cd734f08d [19:01:25] 02mw-config/03sul3-fix 07CosmicAlpha 036b7cd73 Fix SUL3 permission error when not using default article paths [19:01:31] [02mw-config] 07Universal-Omega opened pull request #6259: Fix SUL3 permission error when not using default article paths (03main...03sul3-fix) 13https://github.com/miraheze/mw-config/pull/6259 [19:02:24] miraheze/mw-config - Universal-Omega the build passed. [19:05:54] [02mw-config] 07Universal-Omega pushed 1 new commit to 03main 13https://github.com/miraheze/mw-config/commit/39c36fa514e43cb9d1c98822d092ee710801874a [19:05:55] 02mw-config/03main 07CosmicAlpha 0339c36fa Fix SUL3 permission error when not using default article paths [19:05:57] [02mw-config] 07Universal-Omega merged pull request #6259: Fix SUL3 permission error when not using default article paths (03main...03sul3-fix) 13https://github.com/miraheze/mw-config/pull/6259 [19:05:59] [02mw-config] 07Universal-Omega 04deleted 03sul3-fix at 036b7cd73 13https://api.github.com/repos/miraheze/mw-config/commit/6b7cd73 [19:06:12] !log [universalomega@mwtask181] starting deploy of {'pull': 'config', 'config': True} to all [19:06:15] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:06:43] !log [universalomega@mwtask181] finished deploy of {'pull': 'config', 'config': True} to all - SUCCESS in 30s [19:06:45] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:06:54] miraheze/mw-config - Universal-Omega the build passed. [19:13:24] PROBLEM - cp201 Disk Space on cp201 is WARNING: DISK WARNING - free space: / 45273MiB (10.0% inode=100%); [19:19:21] PROBLEM - cp191 Disk Space on cp191 is WARNING: DISK WARNING - free space: / 45361MiB (10.0% inode=100%); [19:25:50] PROBLEM - cp171 Disk Space on cp171 is WARNING: DISK WARNING - free space: / 45262MiB (10.0% inode=100%); [19:34:34] RECOVERY - test151 Puppet on test151 is OK: OK: Puppet is currently enabled, last run 11 seconds ago with 0 failures [19:41:47] PROBLEM - mwtask181 Puppet on mwtask181 is WARNING: WARNING: Puppet is currently disabled, message: Universal Omega, last run 8 minutes ago with 0 failures [20:03:56] RECOVERY - mwtask181 Puppet on mwtask181 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [20:19:38] [02puppet] 07Universal-Omega created 03auth-csp (+1 new commit) 13https://github.com/miraheze/puppet/commit/b99302967d5e [20:19:38] 02puppet/03auth-csp 07CosmicAlpha 03b993029 mediawiki: don't set CSP header on auth.miraheze.org… [20:19:44] [02puppet] 07Universal-Omega opened pull request #4727: mediawiki: don't set CSP header on auth.miraheze.org (03main...03auth-csp) 13https://github.com/miraheze/puppet/pull/4727 [20:19:51] [02puppet] 07coderabbitai[bot] commented on pull request #4727: - [ ] πŸ” Trigger a full review […] 13https://github.com/miraheze/puppet/pull/4727#issuecomment-3825564852 [20:27:40] [02mw-config] 07Universal-Omega created 03auth-csp (+1 new commit) 13https://github.com/miraheze/mw-config/commit/81ff93fdec90 [20:27:40] 02mw-config/03auth-csp 07CosmicAlpha 0381ff93f Set $wgCSPHeader on auth [20:27:46] [02mw-config] 07Universal-Omega opened pull request #6260: Set $wgCSPHeader on auth (03main...03auth-csp) 13https://github.com/miraheze/mw-config/pull/6260 [20:28:38] miraheze/mw-config - Universal-Omega the build passed. [20:31:03] PROBLEM - graylog161 Disk Space on graylog161 is WARNING: DISK WARNING - free space: / 1266MiB (7.0% inode=92%); [20:32:11] PROBLEM - mwtask181 Puppet on mwtask181 is WARNING: WARNING: Puppet is currently disabled, message: Universal Omega, last run 27 minutes ago with 0 failures [20:32:19] uh @cosmicalpha ^ [20:33:14] looking [20:33:31] a sudden increase https://grafana.wikitide.net/d/W9MIkA7iz/wikitide-cluster?orgId=1&from=now-6h&to=now&timezone=browser&var-job=node&var-node=graylog161.fsslc.wtnet&var-port=9100 [20:33:37] spike in cpu, spike in network [20:37:03] PROBLEM - graylog161 Disk Space on graylog161 is CRITICAL: DISK CRITICAL - free space: / 756MiB (4.2% inode=92%); [20:38:14] RECOVERY - mwtask181 Puppet on mwtask181 is OK: OK: Puppet is currently enabled, last run 27 seconds ago with 0 failures [20:40:11] [1/2] im sure this huge spike has nothing to do with it [20:40:11] [2/2] https://cdn.discordapp.com/attachments/808001911868489748/1466895793561534537/image.png?ex=697e68aa&is=697d172a&hm=d92e0ef912d0f44e982d4ebaa86bf9267c3e686dcd1764e70d7919d9028ee7a2& [20:42:13] seems to be caused by wikibase...? [20:48:33] [02puppet] 07Universal-Omega pushed 1 new commit to 03auth-csp 13https://github.com/miraheze/puppet/commit/33a21d4e374edc44753f3a2f41f6c865a33c4ce1 [20:48:34] 02puppet/03auth-csp 07CosmicAlpha 0333a21d4 Update [20:48:52] [02puppet] 07Universal-Omega pushed 1 new commit to 03auth-csp 13https://github.com/miraheze/puppet/commit/794e97935192c1cca8e196219ff87e0829927760 [20:48:52] 02puppet/03auth-csp 07CosmicAlpha 03794e979 - [20:54:58] PROBLEM - mw152 Puppet on mw152 is WARNING: WARNING: Puppet is currently disabled, message: Universal Omega, last run 21 minutes ago with 0 failures [20:55:11] PROBLEM - mw171 Puppet on mw171 is WARNING: WARNING: Puppet is currently disabled, message: Universal Omega, last run 5 minutes ago with 0 failures [20:55:17] PROBLEM - mwtask171 Puppet on mwtask171 is WARNING: WARNING: Puppet is currently disabled, message: Universal Omega, last run 13 minutes ago with 0 failures [20:55:20] PROBLEM - mw153 Puppet on mw153 is WARNING: WARNING: Puppet is currently disabled, message: Universal Omega, last run 4 minutes ago with 0 failures [20:55:33] PROBLEM - mw161 Puppet on mw161 is WARNING: WARNING: Puppet is currently disabled, message: Universal Omega, last run 15 minutes ago with 0 failures [20:55:41] PROBLEM - mw202 Puppet on mw202 is WARNING: WARNING: Puppet is currently disabled, message: Universal Omega, last run 16 minutes ago with 0 failures [20:55:46] PROBLEM - mwtask151 Puppet on mwtask151 is WARNING: WARNING: Puppet is currently disabled, message: Universal Omega, last run 12 minutes ago with 0 failures [20:55:47] PROBLEM - mw162 Puppet on mw162 is WARNING: WARNING: Puppet is currently disabled, message: Universal Omega, last run 16 minutes ago with 0 failures [20:55:49] PROBLEM - mw163 Puppet on mw163 is WARNING: WARNING: Puppet is currently disabled, message: Universal Omega, last run 27 minutes ago with 0 failures [20:55:55] PROBLEM - mw172 Puppet on mw172 is WARNING: WARNING: Puppet is currently disabled, message: Universal Omega, last run 15 minutes ago with 0 failures [20:55:58] PROBLEM - mw173 Puppet on mw173 is WARNING: WARNING: Puppet is currently disabled, message: Universal Omega, last run 12 minutes ago with 0 failures [20:56:02] PROBLEM - mw191 Puppet on mw191 is WARNING: WARNING: Puppet is currently disabled, message: Universal Omega, last run 4 minutes ago with 0 failures [20:56:06] PROBLEM - mw192 Puppet on mw192 is WARNING: WARNING: Puppet is currently disabled, message: Universal Omega, last run 29 minutes ago with 0 failures [20:56:09] PROBLEM - mw182 Puppet on mw182 is WARNING: WARNING: Puppet is currently disabled, message: Universal Omega, last run 20 minutes ago with 0 failures [20:56:11] PROBLEM - mw183 Puppet on mw183 is WARNING: WARNING: Puppet is currently disabled, message: Universal Omega, last run 20 minutes ago with 0 failures [20:56:16] PROBLEM - mw193 Puppet on mw193 is WARNING: WARNING: Puppet is currently disabled, message: Universal Omega, last run 8 minutes ago with 0 failures [20:56:24] PROBLEM - mw201 Puppet on mw201 is WARNING: WARNING: Puppet is currently disabled, message: Universal Omega, last run 22 minutes ago with 0 failures [20:56:29] PROBLEM - mw151 Puppet on mw151 is WARNING: WARNING: Puppet is currently disabled, message: Universal Omega, last run 28 minutes ago with 0 failures [20:56:31] PROBLEM - mw203 Puppet on mw203 is WARNING: WARNING: Puppet is currently disabled, message: Universal Omega, last run 14 minutes ago with 0 failures [20:56:45] PROBLEM - mw181 Puppet on mw181 is WARNING: WARNING: Puppet is currently disabled, message: Universal Omega, last run 25 minutes ago with 0 failures [20:56:49] PROBLEM - mwtask161 Puppet on mwtask161 is WARNING: WARNING: Puppet is currently disabled, message: Universal Omega, last run 14 minutes ago with 0 failures [21:03:26] [Grafana] FIRING: The mediawiki JobQueue backlog is increasing by more than 100 jobs a minute over an extended time period https://grafana.wikitide.net/d/GtxbP1Xnk?from=1769803370000&orgId=1&to=1769807006679 [21:08:26] [Grafana] RESOLVED: MediaWiki Jobs are rapidly increasing https://grafana.wikitide.net/d/GtxbP1Xnk?from=1769803430000&orgId=1&to=1769807030000 [21:08:36] PROBLEM - test151 Puppet on test151 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[nginx] [21:10:50] [02mw-config] 07Universal-Omega pushed 1 new commit to 03auth-csp 13https://github.com/miraheze/mw-config/commit/16a4d791b760945ac22693a79a41d6589241f719 [21:10:50] 02mw-config/03auth-csp 07CosmicAlpha 0316a4d79 Update [21:11:46] miraheze/mw-config - Universal-Omega the build passed. [21:24:09] PROBLEM - graylog161 Current Load on graylog161 is CRITICAL: LOAD CRITICAL - total load average: 14.31, 8.94, 6.02 [21:26:13] [02puppet] 07Universal-Omega created 03revert-4726-revert-4725-sul3-load (+1 new commit) 13https://github.com/miraheze/puppet/commit/32b208d54a1c [21:26:13] 02puppet/03revert-4726-revert-4725-sul3-load 07CosmicAlpha 0332b208d Revert "Revert "Allow load.php directly from SUL3 domain""… [21:26:30] [02puppet] 07Universal-Omega opened pull request #4728: Reapply "Allow load.php directly from SUL3 domain" (03main...03revert-4726-revert-4725-sul3-load) 13https://github.com/miraheze/puppet/pull/4728 [21:26:45] [02puppet] 07coderabbitai[bot] commented on pull request #4728: - [ ] πŸ” Trigger a full review […] 13https://github.com/miraheze/puppet/pull/4728#issuecomment-3825834109 [21:27:17] [02puppet] 07Universal-Omega merged pull request #4728: Reapply "Allow load.php directly from SUL3 domain" (03main...03revert-4726-revert-4725-sul3-load) 13https://github.com/miraheze/puppet/pull/4728 [21:27:17] [02puppet] 07Universal-Omega pushed 1 new commit to 03main 13https://github.com/miraheze/puppet/commit/0c69936702d452881f75fedf777e0c3720cdcdac [21:27:17] 02puppet/03main 07CosmicAlpha 030c69936 Reapply "Allow load.php directly from SUL3 domain" [21:27:18] [02puppet] 07Universal-Omega 04deleted 03revert-4726-revert-4725-sul3-load at 0332b208d 13https://api.github.com/repos/miraheze/puppet/commit/32b208d [21:30:13] [02mw-config] 07Universal-Omega created 03auth-load (+1 new commit) 13https://github.com/miraheze/mw-config/commit/503c38903651 [21:30:13] 02mw-config/03auth-load 07CosmicAlpha 03503c389 Use load.php directly from auth [21:30:18] [02mw-config] 07Universal-Omega opened pull request #6261: Use load.php directly from auth (03main...03auth-load) 13https://github.com/miraheze/mw-config/pull/6261 [21:31:09] miraheze/mw-config - Universal-Omega the build passed. [21:31:45] PROBLEM - puppet181 Check unit status of git_pull_puppet on puppet181 is CRITICAL: CRITICAL: Status of the systemd unit git_pull_puppet [21:33:03] PROBLEM - graylog161 Disk Space on graylog161 is WARNING: DISK WARNING - free space: / 989MiB (5.5% inode=92%); [21:35:03] PROBLEM - graylog161 Disk Space on graylog161 is CRITICAL: DISK CRITICAL - free space: / 447MiB (2.5% inode=92%); [21:35:11] RECOVERY - mw171 Puppet on mw171 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [21:35:11] RECOVERY - mw152 Puppet on mw152 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [21:35:18] RECOVERY - mwtask171 Puppet on mwtask171 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [21:35:20] RECOVERY - mw153 Puppet on mw153 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [21:35:33] RECOVERY - mw161 Puppet on mw161 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [21:35:42] RECOVERY - mw202 Puppet on mw202 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [21:35:45] RECOVERY - mwtask151 Puppet on mwtask151 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [21:35:52] RECOVERY - mw172 Puppet on mw172 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [21:35:55] RECOVERY - mw162 Puppet on mw162 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [21:35:57] RECOVERY - mw173 Puppet on mw173 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [21:36:02] RECOVERY - mw191 Puppet on mw191 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [21:36:06] RECOVERY - mw192 Puppet on mw192 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [21:36:09] RECOVERY - mw182 Puppet on mw182 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [21:36:10] RECOVERY - mw163 Puppet on mw163 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [21:36:11] RECOVERY - mw183 Puppet on mw183 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [21:36:15] RECOVERY - mw193 Puppet on mw193 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [21:36:22] RECOVERY - mw201 Puppet on mw201 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [21:36:29] RECOVERY - mw203 Puppet on mw203 is OK: OK: Puppet is currently enabled, last run 0 seconds ago with 0 failures [21:36:29] RECOVERY - mw151 Puppet on mw151 is OK: OK: Puppet is currently enabled, last run 5 seconds ago with 0 failures [21:36:45] RECOVERY - mw181 Puppet on mw181 is OK: OK: Puppet is currently enabled, last run 20 seconds ago with 0 failures [21:36:49] RECOVERY - mwtask161 Puppet on mwtask161 is OK: OK: Puppet is currently enabled, last run 22 seconds ago with 0 failures [21:38:03] [02mw-config] 07Universal-Omega merged pull request #6261: Use load.php directly from auth (03main...03auth-load) 13https://github.com/miraheze/mw-config/pull/6261 [21:38:05] [02mw-config] 07Universal-Omega pushed 1 new commit to 03main 13https://github.com/miraheze/mw-config/commit/ad7678804b4b684296fd52fa485ad34e2a69e4fa [21:38:07] [02mw-config] 07Universal-Omega 04deleted 03auth-load at 03503c389 13https://api.github.com/repos/miraheze/mw-config/commit/503c389 [21:38:08] PROBLEM - graylog161 Current Load on graylog161 is WARNING: LOAD WARNING - total load average: 4.89, 7.02, 7.63 [21:38:09] 02mw-config/03main 07CosmicAlpha 03ad76788 Use load.php directly from auth [21:38:41] !log [universalomega@mwtask181] starting deploy of {'pull': 'config', 'config': True} to all [21:38:43] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:39:01] miraheze/mw-config - Universal-Omega the build passed. [21:39:11] !log [universalomega@mwtask181] finished deploy of {'pull': 'config', 'config': True} to all - SUCCESS in 30s [21:39:13] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:39:45] RECOVERY - puppet181 Check unit status of git_pull_puppet on puppet181 is OK: OK: Status of the systemd unit git_pull_puppet [21:40:25] [02puppet] 07Universal-Omega closed pull request #4727: mediawiki: don't set CSP header on auth.miraheze.org (03main...03auth-csp) 13https://github.com/miraheze/puppet/pull/4727 [21:40:29] [02puppet] 07Universal-Omega 04deleted 03auth-csp at 03794e979 13https://api.github.com/repos/miraheze/puppet/commit/794e979 [21:40:45] [02puppet] 07Universal-Omega merged 07pskyechology's pull request #4724: Map @pskyechology to Skye (07miraheze:03main...07pskyechology:03skye) 13https://github.com/miraheze/puppet/pull/4724 [21:40:45] [02puppet] 07Universal-Omega pushed 1 new commit to 03main 13https://github.com/miraheze/puppet/commit/abae786566dbeaa68a42230aba15246ca22c01b1 [21:40:46] 02puppet/03main 07Skye 03abae786 Map @pskyechology to Skye (#4724)… [21:42:15] miraheze/puppet - Universal-Omega the build passed. [21:44:08] RECOVERY - graylog161 Current Load on graylog161 is OK: LOAD OK - total load average: 5.10, 5.29, 6.61 [21:50:41] !log [universalomega@test151] starting deploy of {'pull': 'config', 'config': True} to test151 [21:50:43] !log [universalomega@test151] finished deploy of {'pull': 'config', 'config': True} to test151 - SUCCESS in 1s [21:50:44] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:50:46] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:50:52] !log [universalomega@test151] starting deploy of {'pull': 'config', 'config': True, 'ignore_time': True} to test151 [21:50:53] !log [universalomega@test151] finished deploy of {'pull': 'config', 'config': True, 'ignore_time': True} to test151 - SUCCESS in 0s [21:50:54] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:50:56] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:54:19] [02puppet] 07paladox created 03paladox-patch-1 (+1 new commit) 13https://github.com/miraheze/puppet/commit/e5ef54caa7a5 [21:54:19] 02puppet/03paladox-patch-1 07paladox 03e5ef54c graylog161: Remove limit on client body size [21:54:22] [02puppet] 07paladox opened pull request #4729: graylog161: Remove limit on client body size (03main...03paladox-patch-1) 13https://github.com/miraheze/puppet/pull/4729 [21:54:31] [02puppet] 07paladox merged pull request #4729: graylog161: Remove limit on client body size (03main...03paladox-patch-1) 13https://github.com/miraheze/puppet/pull/4729 [21:54:33] [02puppet] 07coderabbitai[bot] commented on pull request #4729: - [ ] πŸ” Trigger a full review […] 13https://github.com/miraheze/puppet/pull/4729#issuecomment-3825951478 [21:54:35] [02puppet] 07paladox pushed 1 new commit to 03main 13https://github.com/miraheze/puppet/commit/05c21132fd97049e376ed9229b5496a367abfc4a [21:54:37] [02puppet] 07paladox 04deleted 03paladox-patch-1 at 03e5ef54c 13https://api.github.com/repos/miraheze/puppet/commit/e5ef54c [21:54:38] 02puppet/03main 07paladox 0305c2113 graylog161: Remove limit on client body size (#4729) [21:54:52] [02puppet] 07paladox created 03paladox-patch-1 (+1 new commit) 13https://github.com/miraheze/puppet/commit/a707068c0154 [21:54:52] 02puppet/03paladox-patch-1 07paladox 03a707068 os151: remove client boxy size limit [21:54:55] [02puppet] 07paladox opened pull request #4730: os151: remove client boxy size limit (03main...03paladox-patch-1) 13https://github.com/miraheze/puppet/pull/4730 [21:55:03] [02puppet] 07paladox pushed 1 new commit to 03main 13https://github.com/miraheze/puppet/commit/c1f73d57fc6d848abed3b00e9ef404c0b79ef0ed [21:55:05] 02puppet/03main 07paladox 03c1f73d5 os151: remove client boxy size limit (#4730) [21:55:07] [02puppet] 07paladox merged pull request #4730: os151: remove client boxy size limit (03main...03paladox-patch-1) 13https://github.com/miraheze/puppet/pull/4730 [21:55:09] [02puppet] 07coderabbitai[bot] commented on pull request #4730: - [ ] πŸ” Trigger a full review […] 13https://github.com/miraheze/puppet/pull/4730#issuecomment-3825953642 [21:55:11] [02puppet] 07paladox 04deleted 03paladox-patch-1 at 03a707068 13https://api.github.com/repos/miraheze/puppet/commit/a707068 [22:01:03] PROBLEM - graylog161 Disk Space on graylog161 is WARNING: DISK WARNING - free space: / 909MiB (5.1% inode=92%); [22:04:34] RECOVERY - test151 Puppet on test151 is OK: OK: Puppet is currently enabled, last run 22 seconds ago with 0 failures [22:20:28] !log [macfan@mwtask181] sudo -u www-data php /srv/mediawiki/1.44/maintenance/run.php MirahezeMagic:SendBulkEmails --wiki=metawiki --subject=WebAuthn on your account --body=/home/macfan/message.txt --to=/home/macfan=users.txt --from=MacFan4000 --reply-to=BeeBot --dry-run (END - exit=256) [22:20:30] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [22:20:48] !log [macfan@mwtask181] sudo -u www-data php /srv/mediawiki/1.44/maintenance/run.php MirahezeMagic:SendBulkEmails --wiki=metawiki --subject=WebAuthn on your account --body=/home/macfan/message.txt --to=/home/macfan/users.txt --from=MacFan4000 --reply-to=BeeBot --dry-run (END - exit=2) [22:20:50] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [22:21:03] RECOVERY - graylog161 Disk Space on graylog161 is OK: DISK OK - free space: / 1874MiB (10.4% inode=92%); [22:21:19] !log [macfan@mwtask181] sudo -u www-data php /srv/mediawiki/1.44/maintenance/run.php MirahezeMagic:SendBulkEmails --wiki=metawiki --subject=WebAuthn on your account --body=/home/macfan/message.txt --to=/home/macfan/users.txt --from=MacFan4000 --reply-to=BeeBot (END - exit=2) [22:21:21] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [22:28:45] !log [macfan@mwtask181] sudo -u www-data php /srv/mediawiki/1.44/maintenance/run.php MirahezeMagic:SendBulkEmails --wiki=metawiki --subject=WebAuthn on your account --body=/home/macfan/message.txt --to=/home/macfan/users.txt --from=Miraheze_Operations --reply-to=BeeBot (END - exit=0) [22:28:48] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [22:31:44] [02mw-config] 07Universal-Omega pushed 1 new commit to 03main 13https://github.com/miraheze/mw-config/commit/6620ba99a5a5e0ffa9acd21fc8b0f111bd5828c3 [22:31:44] 02mw-config/03main 07CosmicAlpha 036620ba9 Sample some log channels [22:32:38] miraheze/mw-config - Universal-Omega the build passed. [22:32:45] !log [universalomega@mwtask181] starting deploy of {'pull': 'config', 'config': True} to all [22:32:48] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [22:33:16] !log [universalomega@mwtask181] finished deploy of {'pull': 'config', 'config': True} to all - SUCCESS in 30s [22:33:18] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:15:47] [02mw-config] 07Universal-Omega closed pull request #6260: Set $wgCSPHeader on auth (03main...03auth-csp) 13https://github.com/miraheze/mw-config/pull/6260 [23:15:51] [02mw-config] 07Universal-Omega 04deleted 03auth-csp at 0316a4d79 13https://api.github.com/repos/miraheze/mw-config/commit/16a4d79 [23:17:11] [02mw-config] 07Universal-Omega pushed 1 new commit to 03paladox-patch-4 13https://github.com/miraheze/mw-config/commit/c593a87a765e5145287bb2c1b23a43af5a464415 [23:17:11] 02mw-config/03paladox-patch-4 07CosmicAlpha 03c593a87 Merge branch 'main' into paladox-patch-4 [23:18:01] miraheze/mw-config - Universal-Omega the build passed. [23:24:36] PROBLEM - test151 Puppet on test151 is WARNING: WARNING: Puppet is currently disabled, message: Universal Omega, last run 20 minutes ago with 0 failures [23:27:19] PROBLEM - test151 MediaWiki Rendering on test151 is CRITICAL: connect to address 10.0.15.118 and port 443: Connection refusedHTTP CRITICAL - Unable to open TCP socket [23:31:20] RECOVERY - test151 MediaWiki Rendering on test151 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.615 second response time