[00:17:50] RECOVERY - cp171 Disk Space on cp171 is OK: DISK OK - free space: / 71397MiB (15.7% inode=100%); [00:39:21] RECOVERY - cp191 Disk Space on cp191 is OK: DISK OK - free space: / 71319MiB (15.7% inode=100%); [00:45:24] RECOVERY - cp201 Disk Space on cp201 is OK: DISK OK - free space: / 71400MiB (15.7% inode=100%); [02:07:48] PROBLEM - mw152 Puppet on mw152 is CRITICAL: CRITICAL: Puppet has 2 failures. Last run 3 minutes ago with 2 failures. Failed resources (up to 3 shown): Exec[git_pull_femiwiki-deploy-1.44],Exec[git_pull_3d2png] [02:35:56] RECOVERY - mw152 Puppet on mw152 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [06:07:20] PROBLEM - puppet181 Check unit status of listdomains_github_push on puppet181 is CRITICAL: CRITICAL: Status of the systemd unit listdomains_github_push [06:09:24] !log [petramagna@test151] starting deploy of {'config': True, 'l10n': True, 'extension_list': True, 'versions': ['1.44', '1.45'], 'upgrade_extensions': 'RenderBlocking'} to test151 [06:09:26] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [06:15:20] RECOVERY - puppet181 Check unit status of listdomains_github_push on puppet181 is OK: OK: Status of the systemd unit listdomains_github_push [06:15:37] !log [petramagna@test151] finished deploy of {'config': True, 'l10n': True, 'extension_list': True, 'versions': ['1.44', '1.45'], 'upgrade_extensions': 'RenderBlocking'} to test151 - SUCCESS in 372s [06:15:38] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [06:20:29] miraheze/mw-config - lihaohong6 the build passed. [06:21:05] !log [petramagna@mwtask181] starting deploy of {'pull': 'config', 'config': True, 'l10n': True, 'extension_list': True, 'versions': '1.44', 'upgrade_extensions': 'RenderBlocking'} to all [06:21:06] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [06:30:30] !log [petramagna@mwtask181] finished deploy of {'pull': 'config', 'config': True, 'l10n': True, 'extension_list': True, 'versions': '1.44', 'upgrade_extensions': 'RenderBlocking'} to all - SUCCESS in 565s [06:30:31] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [07:04:01] PROBLEM - cp161 HTTP 4xx/5xx ERROR Rate on cp161 is WARNING: WARNING - NGINX Error Rate is 49% [07:11:36] RECOVERY - cp161 HTTP 4xx/5xx ERROR Rate on cp161 is OK: OK - NGINX Error Rate is 33% [07:45:46] PROBLEM - test151 MediaWiki Rendering on test151 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:45:51] !log [petramagna@test151] starting deploy of {'config': True} to test151 [07:45:52] !log [petramagna@test151] finished deploy of {'config': True} to test151 - SUCCESS in 0s [07:48:42] PROBLEM - cp161 HTTP 4xx/5xx ERROR Rate on cp161 is WARNING: WARNING - NGINX Error Rate is 41% [07:49:47] RECOVERY - test151 MediaWiki Rendering on test151 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.556 second response time [07:50:42] RECOVERY - cp161 HTTP 4xx/5xx ERROR Rate on cp161 is OK: OK - NGINX Error Rate is 33% [07:56:42] PROBLEM - cp161 HTTP 4xx/5xx ERROR Rate on cp161 is WARNING: WARNING - NGINX Error Rate is 40% [08:00:42] RECOVERY - cp161 HTTP 4xx/5xx ERROR Rate on cp161 is OK: OK - NGINX Error Rate is 35% [09:13:53] PROBLEM - llm191 Puppet on llm191 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[openwebui] [09:41:53] RECOVERY - llm191 Puppet on llm191 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [10:12:06] !log [skye@mwtask171] sudo -u www-data php /srv/mediawiki/1.44/maintenance/run.php MirahezeMagic:RenameDatabase --wiki=loginwiki --rename --old=avemujiarchivewiki --new=atsudoriwiki --user=Skye (END - exit=0) [10:12:07] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [10:16:56] !log [skye@mwtask171] sudo -u www-data php /srv/mediawiki/1.44/maintenance/run.php CreateWiki:SetContainersAccess --wiki=atsudoriwiki (END - exit=0) [10:16:57] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [10:20:45] !log [skye@mwtask171] sudo -u www-data php /srv/mediawiki/1.44/maintenance/run.php MirahezeMagic:RenameDatabase --wiki=loginwiki --rename --old=ffxivuawiki --new=facelessfoxwiki --user=Skye (END - exit=0) [10:20:47] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [10:21:25] !log [skye@mwtask171] sudo -u www-data php /srv/mediawiki/1.44/maintenance/run.php CreateWiki:SetContainersAccess --wiki=facelessfoxwiki (END - exit=0) [10:21:26] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [10:28:39] !log [skye@mwtask171] sudo -u www-data php /srv/mediawiki/1.44/maintenance/run.php MirahezeMagic:RenameDatabase --wiki=loginwiki --rename --old=projecthishuuwiki --new=aleth019wiki --user=Skye (END - exit=0) [10:28:41] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [10:29:11] !log [skye@mwtask171] sudo -u www-data php /srv/mediawiki/1.44/maintenance/run.php CreateWiki:SetContainersAccess --wiki=aleth019wiki (END - exit=0) [10:29:13] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [10:32:02] !log [skye@mwtask171] sudo -u www-data php /srv/mediawiki/1.44/maintenance/run.php MirahezeMagic:RenameDatabase --wiki=loginwiki --rename --old=mosquitianwiki --new=tmewiki --user=Skye (END - exit=0) [10:32:03] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [10:47:08] !log [skye@mwtask171] sudo -u www-data php /srv/mediawiki/1.44/maintenance/run.php CreateWiki:SetContainersAccess --wiki=tmewiki (END - exit=0) [10:47:10] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [10:50:49] !log [skye@mwtask171] sudo -u www-data php /srv/mediawiki/1.44/maintenance/run.php MirahezeMagic:RenameDatabase --wiki=loginwiki --rename --old=heraldiberiawiki --new=iberaldicawiki --user=Skye (END - exit=0) [10:50:50] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [10:51:27] !log [skye@mwtask171] sudo -u www-data php /srv/mediawiki/1.44/maintenance/run.php CreateWiki:SetContainersAccess --wiki=iberaldicawiki (END - exit=0) [10:51:28] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [10:53:09] [02TSPortal] 07Universal-Omega pushed 1 new commit to 03main 13https://github.com/miraheze/TSPortal/commit/3e0bd8c6333d51a40ba7c64335662c5fcb1b2a2e [10:53:09] 02TSPortal/03main 07CosmicAlpha 033e0bd8c Automerge renovate [10:53:51] [02TSPortal] 07renovate[bot] 04force-pushed 03renovate/all-deps from 03dbf99af to 032220d6a 13https://api.github.com/repos/miraheze/TSPortal/commits/renovate/all-deps [10:53:51] 02TSPortal/03renovate/all-deps 07renovate[bot] 032220d6a Update dependency laravel/framework to v12.49.0 [10:55:16] [02TSPortal] 07renovate[bot] merged pull request #54: Update dependency laravel/framework to v12.49.0 (03main...03renovate/all-deps) 13https://github.com/miraheze/TSPortal/pull/54 [10:55:16] [02TSPortal] 07renovate[bot] pushed 1 new commit to 03main 13https://github.com/miraheze/TSPortal/commit/298e85dc4eeac64214cef3b796cc7642e3d73082 [10:55:16] 02TSPortal/03main 07renovate[bot] 03298e85d Update dependency laravel/framework to v12.49.0 (#54)… [10:55:17] [02TSPortal] 07renovate[bot] 04deleted 03renovate/all-deps at 032220d6a 13https://api.github.com/repos/miraheze/TSPortal/commit/2220d6a [11:12:47] I purged the CF cache for bluearchive.wiki/w/load.php and bluearchive.miraheze.org/w/load.php due to broken regional caching. Probably not worth logging but want to leave a note here. [11:15:29] better to overlog than to underlog [11:15:40] You can log that if you want. [11:15:44] but in this case, not really necessary, yeah [11:15:57] Its not necessary but can do. [11:16:35] You can log anything you do if you want, unless there is some confidential reason not to. [11:17:02] Anyway it is 4:15AM. I need to sleep. [11:18:48] We really need a guide on when to log and when it is not necessary tbh. [11:26:15] when in doubt, log it [11:26:37] To note I often log even confidential stuff but just use generic logs rather than anything revealing just to note some action was done. [11:27:20] PROBLEM - puppet181 Check unit status of listdomains_github_push on puppet181 is CRITICAL: CRITICAL: Status of the systemd unit listdomains_github_push [11:28:34] for example: [11:28:36] !log meow~ [11:28:37] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [11:28:54] someone arrest this woman [11:29:06] in what country? [11:29:11] they are doubting their catgirlness [11:29:21] one that doesnt get you hatecrimed [11:29:26] yay!!! [11:29:44] maybe they'd give me hrt- [11:34:42] PROBLEM - cp161 HTTP 4xx/5xx ERROR Rate on cp161 is CRITICAL: CRITICAL - NGINX Error Rate is 66% [11:35:20] RECOVERY - puppet181 Check unit status of listdomains_github_push on puppet181 is OK: OK: Status of the systemd unit listdomains_github_push [11:36:42] PROBLEM - cp161 HTTP 4xx/5xx ERROR Rate on cp161 is WARNING: WARNING - NGINX Error Rate is 42% [11:38:42] RECOVERY - cp161 HTTP 4xx/5xx ERROR Rate on cp161 is OK: OK - NGINX Error Rate is 22% [11:43:10] !log [petramagna@mwtask181] sudo -u www-data php /srv/mediawiki/1.44/maintenance/run.php /srv/mediawiki/1.44/extensions/TitleKey/maintenance/rebuildTitleKeys.php --wiki=fromsoftwiki (END - exit=65280) [11:43:11] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [11:43:39] !log [petramagna@mwtask181] sudo -u www-data php /srv/mediawiki/1.44/maintenance/run.php /srv/mediawiki/1.44/extensions/TitleKey/maintenance/rebuildTitleKeys.php --wiki=fromsoftwikiwiki (END - exit=0) [11:43:41] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [11:45:39] this prolly why newly uploaded images aint being shown [12:05:40] please don't make things up [13:03:13] [02puppet] 07paladox merged pull request #4720: squid: set cache_dir and set it to 1gb (03main...03paladox-patch-1) 13https://github.com/miraheze/puppet/pull/4720 [13:03:14] [02puppet] 07paladox pushed 1 new commit to 03main 13https://github.com/miraheze/puppet/commit/e3a3032ae63c479261b8bc81fc5bb11cd9e852d4 [13:03:14] 02puppet/03main 07paladox 03e3a3032 squid: set cache_dir and set it to 1gb (#4720)… [13:03:16] [02puppet] 07paladox 04deleted 03paladox-patch-1 at 03406357f 13https://api.github.com/repos/miraheze/puppet/commit/406357f [13:06:51] PROBLEM - mw152 Puppet on mw152 is CRITICAL: CRITICAL: Puppet has 2 failures. Last run 2 minutes ago with 2 failures. Failed resources (up to 3 shown): Exec[git_pull_femiwiki-deploy-1.44],Exec[git_pull_3d2png] [13:07:20] PROBLEM - puppet181 Check unit status of listdomains_github_push on puppet181 is CRITICAL: CRITICAL: Status of the systemd unit listdomains_github_push [13:08:09] PROBLEM - mw182 Puppet on mw182 is CRITICAL: CRITICAL: Puppet has 2 failures. Last run 3 minutes ago with 2 failures. Failed resources (up to 3 shown): Exec[git_pull_femiwiki-deploy-1.44],Exec[git_pull_3d2png] [13:08:11] PROBLEM - mw183 Puppet on mw183 is CRITICAL: CRITICAL: Puppet has 2 failures. Last run 2 minutes ago with 2 failures. Failed resources (up to 3 shown): Exec[git_pull_femiwiki-deploy-1.44],Exec[git_pull_3d2png] [13:13:21] [02puppet] 07paladox pushed 1 new commit to 03main 13https://github.com/miraheze/puppet/commit/1c869f1c1af3c347e1602a0e5d9423b7f4b2f2d8 [13:13:21] 02puppet/03main 07paladox 031c869f1 Revert "squid: set cache_dir and set it to 1gb (#4720)"… [13:15:02] [02puppet] 07paladox created 03paladox-patch-1 (+1 new commit) 13https://github.com/miraheze/puppet/commit/34beae9e9816 [13:15:03] 02puppet/03paladox-patch-1 07paladox 0334beae9 squid: Set maximum_object_size to 1gb [13:15:07] [02puppet] 07paladox opened pull request #4721: squid: Set maximum_object_size to 1gb (03main...03paladox-patch-1) 13https://github.com/miraheze/puppet/pull/4721 [13:15:16] [02puppet] 07coderabbitai[bot] commented on pull request #4721: - [ ] 🔍 Trigger a full review […] 13https://github.com/miraheze/puppet/pull/4721#issuecomment-3811249668 [13:15:20] RECOVERY - puppet181 Check unit status of listdomains_github_push on puppet181 is OK: OK: Status of the systemd unit listdomains_github_push [13:15:26] [02puppet] 07paladox merged pull request #4721: squid: Set maximum_object_size to 1gb (03main...03paladox-patch-1) 13https://github.com/miraheze/puppet/pull/4721 [13:15:26] [02puppet] 07paladox pushed 1 new commit to 03main 13https://github.com/miraheze/puppet/commit/917fd1d90398a4643ac991f718ee52bd95078c2e [13:15:27] 02puppet/03main 07paladox 03917fd1d squid: Set maximum_object_size to 1gb (#4721) [13:15:29] [02puppet] 07paladox 04deleted 03paladox-patch-1 at 0334beae9 13https://api.github.com/repos/miraheze/puppet/commit/34beae9 [13:20:16] PROBLEM - mw193 Puppet on mw193 is CRITICAL: CRITICAL: Puppet has 2 failures. Last run 2 minutes ago with 2 failures. Failed resources (up to 3 shown): Exec[git_pull_femiwiki-deploy-1.44],Exec[git_pull_3d2png] [13:34:59] RECOVERY - mw152 Puppet on mw152 is OK: OK: Puppet is currently enabled, last run 40 seconds ago with 0 failures [13:36:09] RECOVERY - mw182 Puppet on mw182 is OK: OK: Puppet is currently enabled, last run 26 seconds ago with 0 failures [13:36:11] RECOVERY - mw183 Puppet on mw183 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [13:36:48] [02puppet] 07paladox created 03paladox-patch-1 (+1 new commit) 13https://github.com/miraheze/puppet/commit/7f21f7f62c13 [13:36:48] 02puppet/03paladox-patch-1 07paladox 037f21f7f dns: Disabling tracking for port 53 [13:36:53] [02puppet] 07paladox opened pull request #4722: dns: Disabling tracking for port 53 (03main...03paladox-patch-1) 13https://github.com/miraheze/puppet/pull/4722 [13:36:59] [02puppet] 07coderabbitai[bot] commented on pull request #4722: - [ ] 🔍 Trigger a full review […] 13https://github.com/miraheze/puppet/pull/4722#issuecomment-3811352539 [13:37:32] [02puppet] 07github-actions[bot] pushed 1 new commit to 03paladox-patch-1 13https://github.com/miraheze/puppet/commit/9f1f13827435ac923b576e13762fa33bb7b902b7 [13:37:32] 02puppet/03paladox-patch-1 07github-actions 039f1f138 CI: lint puppet code to standards… [13:39:02] [02puppet] 07paladox merged pull request #4722: dns: Disabling tracking for port 53 (03main...03paladox-patch-1) 13https://github.com/miraheze/puppet/pull/4722 [13:39:02] [02puppet] 07paladox pushed 1 new commit to 03main 13https://github.com/miraheze/puppet/commit/eee0c4e1d99c92e06f690f8bac15ec8999ccd317 [13:39:03] 02puppet/03main 07paladox 03eee0c4e dns: Disabling tracking for port 53 (#4722)… [13:39:05] [02puppet] 07paladox 04deleted 03paladox-patch-1 at 039f1f138 13https://api.github.com/repos/miraheze/puppet/commit/9f1f138 [13:48:16] RECOVERY - mw193 Puppet on mw193 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [14:09:20] PROBLEM - puppet181 Check unit status of listdomains_github_push on puppet181 is CRITICAL: CRITICAL: Status of the systemd unit listdomains_github_push [14:15:20] RECOVERY - puppet181 Check unit status of listdomains_github_push on puppet181 is OK: OK: Status of the systemd unit listdomains_github_push [17:15:46] PROBLEM - test151 MediaWiki Rendering on test151 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:17:42] RECOVERY - test151 MediaWiki Rendering on test151 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.597 second response time [17:19:08] [02TSPortal] 07Universal-Omega 04force-pushed 03standing-good from 03e48a46c to 0324aaf1b 13https://api.github.com/repos/miraheze/TSPortal/commits/standing-good [17:19:08] 02TSPortal/03standing-good 07CosmicAlpha 0324aaf1b Implement setting standing to good on successful appeal [17:19:50] PROBLEM - cp171 Disk Space on cp171 is WARNING: DISK WARNING - free space: / 45224MiB (10.0% inode=100%); [17:29:21] PROBLEM - cp191 Disk Space on cp191 is WARNING: DISK WARNING - free space: / 45356MiB (10.0% inode=100%); [17:47:24] PROBLEM - cp201 Disk Space on cp201 is WARNING: DISK WARNING - free space: / 45220MiB (10.0% inode=100%); [21:25:33] [02TSPortal] 07Universal-Omega pushed 1 new commit to 03main 13https://github.com/miraheze/TSPortal/commit/b9882514b825ac28e327ec3d7778975188788e7a [21:25:33] 02TSPortal/03main 07CosmicAlpha 03b988251 Minor updates to README [21:30:32] [02TSPortal] 07Universal-Omega pushed 1 new commit to 03main 13https://github.com/miraheze/TSPortal/commit/1d1984998619528a07b6e9fd82ee49c51530cd5c [21:30:32] 02TSPortal/03main 07CosmicAlpha 031d19849 More updates to README [21:56:55] PROBLEM - bots171 APT on bots171 is CRITICAL: APT CRITICAL: 2 packages available for upgrade (1 critical updates). [23:48:01] [02mw-config] 07YTFGolf opened pull request #6255: T14862 ECP (07miraheze:03main...07YTFGolf:03main) 13https://github.com/miraheze/mw-config/pull/6255 [23:48:56] miraheze/mw-config - YTFGolf the build passed. [23:57:24] [02mw-config] 07SomeMWDev approved pull request #6255 13https://github.com/miraheze/mw-config/pull/6255#pullrequestreview-3719756085 [23:58:11] [02mw-config] 07SomeMWDev merged 07YTFGolf's pull request #6255: T14862 ECP (07miraheze:03main...07YTFGolf:03main) 13https://github.com/miraheze/mw-config/pull/6255 [23:58:11] [02mw-config] 07SomeMWDev pushed 1 new commit to 03main 13https://github.com/miraheze/mw-config/commit/e07728a3a5528cb46d48ace4f2cba31c1d5d406e [23:58:11] 02mw-config/03main 07YTFGolf 03e07728a T14862: Add extended confirmed protection level for mcsosirswiki (#6255) [23:58:23] !log [somerandomdeveloper@mwtask181] starting deploy of {'pull': 'config', 'config': True} to all [23:58:25] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:58:53] !log [somerandomdeveloper@mwtask181] finished deploy of {'pull': 'config', 'config': True} to all - SUCCESS in 29s [23:58:54] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:59:00] miraheze/mw-config - SomeMWDev the build passed.