[00:03:26] PROBLEM - cp191 Disk Space on cp191 is CRITICAL: DISK CRITICAL - free space: / 26925MiB (5% inode=99%); [00:03:36] PROBLEM - cp171 Disk Space on cp171 is CRITICAL: DISK CRITICAL - free space: / 26992MiB (5% inode=99%); [00:09:20] RECOVERY - cp171 Disk Space on cp171 is OK: DISK OK - free space: / 55053MiB (12% inode=99%); [00:11:10] RECOVERY - cp191 Disk Space on cp191 is OK: DISK OK - free space: / 55345MiB (12% inode=99%); [00:41:34] [02puppet] 07Universal-Omega created 03systemd-monitor (+1 new commit) 13https://github.com/miraheze/puppet/commit/79a9fbd6d2f3 [00:41:34] 02puppet/03systemd-monitor 07CosmicAlpha 0379a9fbd systemd: add monitoring for timers [00:41:38] [02puppet] 07Universal-Omega opened pull request #4393: systemd: add monitoring for timers (03main...03systemd-monitor) 13https://github.com/miraheze/puppet/pull/4393 [00:41:43] [02puppet] 07coderabbitai[bot] commented on pull request #4393:

[…] 13https://github.com/miraheze/puppet/pull/4393#issuecomment-2974811283 [00:42:09] [02puppet] 07github-actions[bot] pushed 1 new commit to 03systemd-monitor 13https://github.com/miraheze/puppet/commit/7e067a4c3396aceeec3174daa5484565e8714f7c [00:42:11] 02puppet/03systemd-monitor 07github-actions 037e067a4 CI: lint puppet code to standards… [00:44:47] [02puppet] 07Universal-Omega pushed 1 new commit to 03systemd-monitor 13https://github.com/miraheze/puppet/commit/f4d57d8300ef8a6e8635b83e867f7801f9e36335 [00:44:47] 02puppet/03systemd-monitor 07CosmicAlpha 03f4d57d8 Add ensure [00:47:34] [02puppet] 07Universal-Omega pushed 1 new commit to 03systemd-monitor 13https://github.com/miraheze/puppet/commit/ec6e1ff0305eded2b335495acd2972fa64a76976 [00:47:34] 02puppet/03systemd-monitor 07CosmicAlpha 03ec6e1ff Add [00:48:48] [02puppet] 07Universal-Omega pushed 1 new commit to 03systemd-monitor 13https://github.com/miraheze/puppet/commit/127283ef198c7ad09d379684f48b33b069385457 [00:48:49] 02puppet/03systemd-monitor 07CosmicAlpha 03127283e Add [00:50:08] [02puppet] 07Universal-Omega pushed 1 new commit to 03systemd-monitor 13https://github.com/miraheze/puppet/commit/75dafce1ec311e7266d6e1cbc6b4bedb0c7717ce [00:50:08] 02puppet/03systemd-monitor 07CosmicAlpha 0375dafce Use .sh [00:50:32] [02puppet] 07Universal-Omega pushed 1 new commit to 03systemd-monitor 13https://github.com/miraheze/puppet/commit/e9e36bf4eb16ddbd6408473f3e80c7c156c20f68 [00:50:32] 02puppet/03systemd-monitor 07CosmicAlpha 03e9e36bf Use .sh [00:56:38] [02puppet] 07Universal-Omega pushed 1 new commit to 03systemd-monitor 13https://github.com/miraheze/puppet/commit/63f4e1f74747b7f4c15aa8cce7895f0665970998 [00:56:38] 02puppet/03systemd-monitor 07CosmicAlpha 0363f4e1f Fix [01:05:01] PROBLEM - mwtask181 Current Load on mwtask181 is CRITICAL: LOAD CRITICAL - total load average: 24.81, 16.65, 10.02 [01:17:01] PROBLEM - mwtask181 Current Load on mwtask181 is WARNING: LOAD WARNING - total load average: 14.29, 20.85, 16.91 [01:19:01] RECOVERY - mwtask181 Current Load on mwtask181 is OK: LOAD OK - total load average: 8.59, 16.47, 15.78 [04:59:43] PROBLEM - cp171 Disk Space on cp171 is WARNING: DISK WARNING - free space: / 49865MiB (10% inode=99%); [05:03:39] PROBLEM - Host mattermost1 is DOWN: PING CRITICAL - Packet loss = 100% [05:05:40] RECOVERY - Host mattermost1 is UP: PING OK - Packet loss = 0%, RTA = 30.38 ms [05:08:43] PROBLEM - cp191 Disk Space on cp191 is WARNING: DISK WARNING - free space: / 49915MiB (10% inode=99%); [05:40:53] [Grafana] FIRING: The estimated time for the MediaWiki JobQueue to clear is excessively high (8 hours) for an extended time period https://grafana.wikitide.net/d/GtxbP1Xnk?from=1750048820000&orgId=1&to=1750052453348 [05:45:53] [Grafana] RESOLVED: MediaWiki JobQueue is stalled https://grafana.wikitide.net/d/GtxbP1Xnk?from=1750049120000&orgId=1&to=1750052720000 [06:40:01] PROBLEM - mwtask181 Current Load on mwtask181 is CRITICAL: LOAD CRITICAL - total load average: 24.14, 20.35, 18.23 [06:43:51] PROBLEM - mwtask181 Current Load on mwtask181 is WARNING: LOAD WARNING - total load average: 23.95, 22.35, 19.52 [06:45:26] PROBLEM - mwtask151 Current Load on mwtask151 is WARNING: LOAD WARNING - total load average: 21.30, 18.49, 13.38 [06:45:46] PROBLEM - mwtask181 Current Load on mwtask181 is CRITICAL: LOAD CRITICAL - total load average: 25.65, 23.48, 20.27 [06:47:26] RECOVERY - mwtask151 Current Load on mwtask151 is OK: LOAD OK - total load average: 18.31, 18.56, 14.05 [06:47:40] PROBLEM - mwtask181 Current Load on mwtask181 is WARNING: LOAD WARNING - total load average: 22.65, 23.28, 20.59 [06:51:30] PROBLEM - mwtask181 Current Load on mwtask181 is CRITICAL: LOAD CRITICAL - total load average: 26.18, 24.18, 21.50 [07:09:01] PROBLEM - mwtask181 Current Load on mwtask181 is WARNING: LOAD WARNING - total load average: 22.88, 23.77, 23.57 [07:15:01] PROBLEM - mwtask181 Current Load on mwtask181 is CRITICAL: LOAD CRITICAL - total load average: 24.94, 24.06, 23.73 [07:23:01] PROBLEM - mwtask181 Current Load on mwtask181 is WARNING: LOAD WARNING - total load average: 22.16, 23.39, 23.74 [07:29:01] RECOVERY - mwtask181 Current Load on mwtask181 is OK: LOAD OK - total load average: 5.72, 13.38, 19.48 [08:51:01] PROBLEM - mwtask181 Current Load on mwtask181 is CRITICAL: LOAD CRITICAL - total load average: 25.65, 18.06, 10.73 [08:53:01] PROBLEM - mwtask181 Current Load on mwtask181 is WARNING: LOAD WARNING - total load average: 23.52, 20.12, 12.40 [08:55:01] PROBLEM - mwtask181 Current Load on mwtask181 is CRITICAL: LOAD CRITICAL - total load average: 24.77, 21.82, 13.97 [08:57:01] RECOVERY - mwtask181 Current Load on mwtask181 is OK: LOAD OK - total load average: 13.88, 19.53, 14.15 [08:59:51] PROBLEM - db181 Current Load on db181 is CRITICAL: LOAD CRITICAL - total load average: 6.57, 13.49, 6.98 [09:01:01] PROBLEM - mwtask181 Current Load on mwtask181 is CRITICAL: LOAD CRITICAL - total load average: 24.67, 22.16, 16.41 [09:01:51] RECOVERY - db181 Current Load on db181 is OK: LOAD OK - total load average: 1.36, 9.22, 6.20 [09:17:01] PROBLEM - mwtask181 Current Load on mwtask181 is WARNING: LOAD WARNING - total load average: 10.91, 20.98, 20.85 [09:19:01] RECOVERY - mwtask181 Current Load on mwtask181 is OK: LOAD OK - total load average: 7.91, 16.58, 19.25 [10:25:27] PROBLEM - cp191 Varnish Backends on cp191 is CRITICAL: 1 backends are down. mw192 [10:25:45] Weird [10:27:56] mw192 is sad [10:33:19] RECOVERY - cp191 Varnish Backends on cp191 is OK: All 31 backends are healthy [11:41:27] PROBLEM - cp171 Varnish Backends on cp171 is CRITICAL: 1 backends are down. mw192 [11:48:32] PROBLEM - cp161 Varnish Backends on cp161 is CRITICAL: 1 backends are down. mw192 [11:49:19] rip mw192 [11:49:23] RECOVERY - cp171 Varnish Backends on cp171 is OK: All 31 backends are healthy [11:50:18] !log reboot mw192 [11:50:21] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [11:52:29] RECOVERY - cp161 Varnish Backends on cp161 is OK: All 31 backends are healthy [11:54:10] [02puppet] 07paladox created 03paladox-patch-11 (+1 new commit) 13https://github.com/miraheze/puppet/commit/45bf15e690be [11:54:10] 02puppet/03paladox-patch-11 07paladox 0345bf15e matomo151: lower php::fpm::fpm_workers_multiplier: to 3… [11:54:14] [02puppet] 07paladox opened pull request #4394: matomo151: lower php::fpm::fpm_workers_multiplier: to 3 (03main...03paladox-patch-11) 13https://github.com/miraheze/puppet/pull/4394 [11:54:19] [02puppet] 07coderabbitai[bot] commented on pull request #4394:

[…] 13https://github.com/miraheze/puppet/pull/4394#issuecomment-2976275711 [11:54:28] [02puppet] 07paladox retitled PR #4394: "matomo151: lower php::fpm::fpm_workers_multiplier: to 3" ➜ "matomo151: lower php::fpm::fpm_workers_multiplier to 3" 13https://github.com/miraheze/puppet/pull/4394 [11:54:32] [02puppet] 07paladox merged pull request #4394: matomo151: lower php::fpm::fpm_workers_multiplier to 3 (03main...03paladox-patch-11) 13https://github.com/miraheze/puppet/pull/4394 [11:54:32] [02puppet] 07paladox pushed 1 new commit to 03main 13https://github.com/miraheze/puppet/commit/07b8bc569029495a2c862d3f2aa159578a7c83a6 [11:54:32] 02puppet/03main 07paladox 0307b8bc5 matomo151: lower php::fpm::fpm_workers_multiplier to 3 (#4394)… [11:54:34] [02puppet] 07paladox 04deleted 03paladox-patch-11 at 0345bf15e 13https://api.github.com/repos/miraheze/puppet/commit/45bf15e [12:00:55] [02puppet] 07paladox created 03paladox-patch-11 (+1 new commit) 13https://github.com/miraheze/puppet/commit/1b0dcd5655c6 [12:00:55] 02puppet/03paladox-patch-11 07paladox 031b0dcd5 matomo: set maxmeory_policy for redis to volatile-lru [12:00:59] [02puppet] 07paladox opened pull request #4395: matomo: set maxmeory_policy for redis to volatile-lru (03main...03paladox-patch-11) 13https://github.com/miraheze/puppet/pull/4395 [12:01:05] [02puppet] 07coderabbitai[bot] commented on pull request #4395:

[…] 13https://github.com/miraheze/puppet/pull/4395#issuecomment-2976312382 [12:01:11] [02puppet] 07paladox merged pull request #4395: matomo: set maxmeory_policy for redis to volatile-lru (03main...03paladox-patch-11) 13https://github.com/miraheze/puppet/pull/4395 [12:01:11] [02puppet] 07paladox pushed 1 new commit to 03main 13https://github.com/miraheze/puppet/commit/91fe416e8c48c8740c8375e48ea8f285529c742e [12:01:11] 02puppet/03main 07paladox 0391fe416 matomo: set maxmeory_policy for redis to volatile-lru (#4395) [12:01:13] [02puppet] 07paladox 04deleted 03paladox-patch-11 at 031b0dcd5 13https://api.github.com/repos/miraheze/puppet/commit/1b0dcd5 [12:08:10] !log reboot matomo151 [12:08:18] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [12:10:33] [02DiscordNotifications] 07translatewiki pushed 1 new commit to 03main 13https://github.com/miraheze/DiscordNotifications/commit/6052af2b94c325a0feaca70c6675682ce6dbbb17 [12:10:34] 02DiscordNotifications/03main 07translatewiki.net 036052af2 Localisation updates from https://translatewiki.net. [12:10:35] [02landing] 07translatewiki pushed 1 new commit to 03main 13https://github.com/miraheze/landing/commit/dab3ba03ee38de5130fdb1b095ee189d2aac79e3 [12:10:35] 02landing/03main 07translatewiki.net 03dab3ba0 Localisation updates from https://translatewiki.net. [12:10:37] [02CreateWiki] 07translatewiki pushed 1 new commit to 03main 13https://github.com/miraheze/CreateWiki/commit/4ed8a009a81098229df51a3e1482ec71b5a8a859 [12:10:39] [02MirahezeMagic] 07translatewiki pushed 1 new commit to 03main 13https://github.com/miraheze/MirahezeMagic/commit/363eed2f201d29948fe896b70c32537f43a47f57 [12:10:40] 02CreateWiki/03main 07translatewiki.net 034ed8a00 Localisation updates from https://translatewiki.net. [12:10:42] 02MirahezeMagic/03main 07translatewiki.net 03363eed2 Localisation updates from https://translatewiki.net. [12:11:23] miraheze/landing - translatewiki the build passed. [12:16:24] !log [somerandomdeveloper@test151] sudo -u www-data php /srv/mediawiki/1.44/maintenance/run.php /srv/mediawiki/1.44/maintenance/createAndPromote.php --wiki=morerandomstuffwikibeta Miraheze_Operations --bot (END - exit=256) [12:16:27] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [12:18:16] !log [somerandomdeveloper@test151] sudo -u www-data php /srv/mediawiki/1.44/maintenance/run.php /srv/mediawiki/1.44/maintenance/createAndPromote.php --wiki=morerandomstuffwikibeta RhinosF1 --bot (END - exit=256) [12:18:19] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [12:19:24] !log [somerandomdeveloper@test151] sudo -u www-data php /srv/mediawiki/1.44/maintenance/run.php /srv/mediawiki/1.44/maintenance/createAndPromote.php --wiki=randomstuffwikibeta SomeRandomDeveloperAlt --bot (END - exit=256) [12:19:27] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [12:21:16] !log [somerandomdeveloper@test151] sudo -u www-data php /srv/mediawiki/1.44/maintenance/run.php CentralAuth:createLocalAccount --wiki=randomstuffwikibeta SomeRandomDeveloperAlt (END - exit=0) [12:21:18] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [12:22:12] !log [somerandomdeveloper@mwtask181] sudo -u www-data php /srv/mediawiki/1.43/maintenance/run.php CentralAuth:createLocalAccount --wiki=battlenationswiki Miraheze_Operations (END - exit=0) [12:22:15] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [12:22:50] miraheze/MirahezeMagic - translatewiki the build passed. [12:24:05] !log [somerandomdeveloper@mwtask181] sudo -u www-data php /srv/mediawiki/1.43/maintenance/run.php createAndPromote --wiki=battlenationswiki Miraheze_Operations --force --bot (END - exit=0) [12:24:08] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [12:29:22] miraheze/CreateWiki - translatewiki the build passed. [12:30:24] !log [somerandomdeveloper@mwtask171] sudo -u www-data php /srv/mediawiki/1.43/maintenance/run.php /srv/mediawiki/1.43/maintenance/deleteBatch.php --wiki=battlenationswiki --i=1 --u=Miraheze_Operations --r='[[phab:T13542|Requested]]' /home/somerandomdeveloper/deletebatch.txt (START) [12:30:26] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [12:30:53] [Grafana] FIRING: The estimated time for the MediaWiki JobQueue to clear is excessively high (8 hours) for an extended time period https://grafana.wikitide.net/d/GtxbP1Xnk?from=1750073420000&orgId=1&to=1750077053344 [12:30:59] miraheze/DiscordNotifications - translatewiki the build passed. [12:40:53] [Grafana] RESOLVED: MediaWiki JobQueue is stalled https://grafana.wikitide.net/d/GtxbP1Xnk?from=1750074020000&orgId=1&to=1750077620000 [12:50:00] ^ should be less spammy [12:50:05] i fixed a data issue [16:15:24] [02ssl] 07MacFan4000 pushed 1 new commit to 03main 13https://github.com/miraheze/ssl/commit/5d32fd9871d43287a475cf0915c2fe379b0529b2 [16:15:24] 02ssl/03main 07MacFan4000 035d32fd9 add 3 domains [16:23:54] [02ssl] 07MacFan4000 pushed 1 new commit to 03main 13https://github.com/miraheze/ssl/commit/704d9662692d4205b504ea315fe7c42c0ddbee03 [16:23:54] 02ssl/03main 07MacFan4000 03704d966 add www.ecole.science [18:33:13] !log [reception@mwtask181] sudo -u www-data php /srv/mediawiki/1.43/maintenance/run.php MirahezeMagic:RenameDatabase --wiki=loginwiki --rename --old=princesscacashwiki --new=soloniuniwiki --user=Reception123 (END - exit=0) [18:33:16] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:37:02] !log [reception@mwtask181] sudo -u www-data php /srv/mediawiki/1.43/maintenance/run.php MirahezeMagic:RenameDatabase --wiki=loginwiki --rename --old=csgamedevtycoonwiki --new=gdtwikiwiki --user=Reception123 (END - exit=0) [18:37:05] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:39:40] !log [reception@mwtask181] sudo -u www-data php /srv/mediawiki/1.43/maintenance/run.php MirahezeMagic:RenameDatabase --wiki=loginwiki --rename --old=jtohxlprojectwiki --new=etohxlwiki --user=Reception123 (END - exit=0) [18:39:43] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:42:16] !log [reception@mwtask181] sudo -u www-data php /srv/mediawiki/1.43/maintenance/run.php MirahezeMagic:RenameDatabase --wiki=loginwiki --rename --old=taiwanrailwihiwiki --new=taiwanrailwikiwiki --user=Reception123 (END - exit=0) [18:42:19] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:43:32] PROBLEM - mwtask151 Current Load on mwtask151 is WARNING: LOAD WARNING - total load average: 23.68, 20.98, 14.74 [18:49:16] PROBLEM - mwtask151 Current Load on mwtask151 is CRITICAL: LOAD CRITICAL - total load average: 24.13, 23.11, 17.64 [18:50:34] PROBLEM - mwtask181 Current Load on mwtask181 is CRITICAL: LOAD CRITICAL - total load average: 25.82, 21.45, 16.14 [18:50:48] !log [reception@mwtask181] sudo -u www-data php /srv/mediawiki/1.43/maintenance/run.php MirahezeMagic:RenameDatabase --wiki=loginwiki --rename --old=pokenexuswiki --new=pokemoneclipsewiki --user=Reception123 (END - exit=0) [18:50:50] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:51:11] PROBLEM - mwtask151 Current Load on mwtask151 is WARNING: LOAD WARNING - total load average: 22.40, 22.76, 18.16 [18:52:29] PROBLEM - mwtask181 Current Load on mwtask181 is WARNING: LOAD WARNING - total load average: 23.95, 21.96, 16.94 [18:54:15] !log [reception@mwtask181] sudo -u www-data php /srv/mediawiki/1.43/maintenance/run.php MirahezeMagic:RenameDatabase --wiki=loginwiki --rename --old=fashiondreamerwiki --new=fashiondreamerwikiwiki --user=Reception123 (END - exit=0) [18:54:17] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:54:24] PROBLEM - mwtask181 Current Load on mwtask181 is CRITICAL: LOAD CRITICAL - total load average: 24.38, 23.03, 17.93 [18:56:19] RECOVERY - mwtask181 Current Load on mwtask181 is OK: LOAD OK - total load average: 14.41, 19.83, 17.37 [18:58:49] PROBLEM - mwtask151 Current Load on mwtask151 is CRITICAL: LOAD CRITICAL - total load average: 24.07, 23.15, 20.03 [19:04:33] PROBLEM - mwtask151 Current Load on mwtask151 is WARNING: LOAD WARNING - total load average: 21.91, 22.87, 20.95 [19:06:07] PROBLEM - mwtask181 Current Load on mwtask181 is CRITICAL: LOAD CRITICAL - total load average: 24.46, 20.57, 18.07 [19:07:20] [Grafana] FIRING: The mediawiki job queue has more than 100000 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?from=1750097210000&orgId=1&to=1750100840013 [19:08:07] PROBLEM - mwtask181 Current Load on mwtask181 is WARNING: LOAD WARNING - total load average: 23.98, 21.91, 18.88 [19:10:07] PROBLEM - mwtask181 Current Load on mwtask181 is CRITICAL: LOAD CRITICAL - total load average: 24.40, 22.81, 19.59 [19:12:07] PROBLEM - mwtask181 Current Load on mwtask181 is WARNING: LOAD WARNING - total load average: 22.42, 22.56, 19.89 [19:16:07] PROBLEM - mwtask181 Current Load on mwtask181 is CRITICAL: LOAD CRITICAL - total load average: 25.09, 23.67, 20.93 [19:17:20] [Grafana] RESOLVED: High Job Queue Backlog https://grafana.wikitide.net/d/GtxbP1Xnk?from=1750097210000&orgId=1&to=1750101410000 [19:24:01] PROBLEM - mwtask151 Current Load on mwtask151 is CRITICAL: LOAD CRITICAL - total load average: 24.13, 23.19, 22.39 [19:26:01] PROBLEM - mwtask151 Current Load on mwtask151 is WARNING: LOAD WARNING - total load average: 22.24, 22.84, 22.36 [19:34:07] PROBLEM - mwtask181 Current Load on mwtask181 is WARNING: LOAD WARNING - total load average: 23.52, 23.96, 23.30 [19:43:56] [02mw-config] 07SomeMWDev opened pull request #5987: Enable citizen command palette by default (07miraheze:03main...07SomeMWDev:03enable-citizen-command-palette) 13https://github.com/miraheze/mw-config/pull/5987 [19:44:42] miraheze/mw-config - SomeMWDev the build passed. [19:45:23] [02mw-config] 07SomeMWDev merged pull request #5987: Enable citizen command palette by default (07miraheze:03main...07SomeMWDev:03enable-citizen-command-palette) 13https://github.com/miraheze/mw-config/pull/5987 [19:45:23] [02mw-config] 07SomeMWDev pushed 1 new commit to 03main 13https://github.com/miraheze/mw-config/commit/9fe62836d382a8d1628f8782be650bde54662eba [19:45:23] 02mw-config/03main 07SomeRandomDeveloper 039fe6283 Enable citizen command palette by default (#5987)… [19:45:48] !log [somerandomdeveloper@test151] starting deploy of {'pull': 'config', 'config': True} to test151 [19:45:49] !log [somerandomdeveloper@test151] finished deploy of {'pull': 'config', 'config': True} to test151 - SUCCESS in 1s [19:45:51] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:45:53] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:46:07] PROBLEM - mwtask181 Current Load on mwtask181 is CRITICAL: LOAD CRITICAL - total load average: 24.14, 23.11, 23.07 [19:46:19] miraheze/mw-config - SomeMWDev the build passed. [19:46:33] !log [somerandomdeveloper@mwtask181] starting deploy of {'pull': 'config', 'config': True} to all [19:46:36] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:47:07] !log [somerandomdeveloper@mwtask181] finished deploy of {'pull': 'config', 'config': True} to all - SUCCESS in 33s [19:47:10] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:50:07] PROBLEM - mwtask181 Current Load on mwtask181 is WARNING: LOAD WARNING - total load average: 22.31, 23.38, 23.24 [20:00:01] RECOVERY - mwtask151 Current Load on mwtask151 is OK: LOAD OK - total load average: 4.69, 14.25, 19.46 [20:02:07] RECOVERY - mwtask181 Current Load on mwtask181 is OK: LOAD OK - total load average: 9.15, 16.15, 20.16 [22:05:41] [02puppet] 07Universal-Omega created 03reports-timer (+1 new commit) 13https://github.com/miraheze/puppet/commit/5339e45a8da1 [22:05:42] 02puppet/03reports-timer 07CosmicAlpha 035339e45 reports: convert task scheduler from cron to systemd timer [22:05:46] [02puppet] 07Universal-Omega opened pull request #4396: reports: convert task scheduler from cron to systemd timer (03main...03reports-timer) 13https://github.com/miraheze/puppet/pull/4396 [22:05:51] [02puppet] 07coderabbitai[bot] commented on pull request #4396:

[…] 13https://github.com/miraheze/puppet/pull/4396#issuecomment-2978303981 [22:06:17] [02puppet] 07github-actions[bot] pushed 1 new commit to 03reports-timer 13https://github.com/miraheze/puppet/commit/c2ba4e482e005b95d82e58d4ececf9e0cedbc14d [22:06:17] 02puppet/03reports-timer 07github-actions 03c2ba4e4 CI: lint puppet code to standards… [22:15:40] [02puppet] 07Universal-Omega pushed 1 new commit to 03reports-timer 13https://github.com/miraheze/puppet/commit/c03d813a80387c6e33e78242225a7985de874d4d [22:15:40] 02puppet/03reports-timer 07CosmicAlpha 03c03d813 Run every minute [22:20:39] [02puppet] 07Universal-Omega created 03backup-log-paths from 03main (+0 new commit) 13https://github.com/miraheze/puppet/compare/backup-log-paths [22:21:39] [02puppet] 07Universal-Omega pushed 1 new commit to 03backup-log-paths 13https://github.com/miraheze/puppet/commit/5b29479b84a5374f19ad3c485a11ac5323d67a5f [22:21:39] 02puppet/03backup-log-paths 07CosmicAlpha 035b29479 Fix paths for some backup log files [22:22:05] [02puppet] 07Universal-Omega opened pull request #4397: Fix paths for some backup log files (03main...03backup-log-paths) 13https://github.com/miraheze/puppet/pull/4397 [22:22:10] [02puppet] 07coderabbitai[bot] commented on pull request #4397:

[…] 13https://github.com/miraheze/puppet/pull/4397#issuecomment-2978332426 [22:28:07] [02puppet] 07Universal-Omega created 03swift-devicecheck-timer (+1 new commit) 13https://github.com/miraheze/puppet/commit/9599b70cb942 [22:28:07] 02puppet/03swift-devicecheck-timer 07CosmicAlpha 039599b70 swift: convert devicecheck cron to a systemd timer [22:28:11] [02puppet] 07Universal-Omega opened pull request #4398: swift: convert devicecheck cron to a systemd timer (03main...03swift-devicecheck-timer) 13https://github.com/miraheze/puppet/pull/4398 [22:28:28] [02puppet] 07coderabbitai[bot] commented on pull request #4398:

[…] 13https://github.com/miraheze/puppet/pull/4398#issuecomment-2978348036 [22:28:47] [02puppet] 07github-actions[bot] pushed 1 new commit to 03swift-devicecheck-timer 13https://github.com/miraheze/puppet/commit/e3b43def31b2b6bedc74a710440c98bc42615fe7 [22:28:47] 02puppet/03swift-devicecheck-timer 07github-actions 03e3b43de CI: lint puppet code to standards… [22:29:33] [02puppet] 07Universal-Omega pushed 1 new commit to 03swift-devicecheck-timer 13https://github.com/miraheze/puppet/commit/c80a5b71f05b388efe4d47273197ac3c224fb53c [22:29:34] 02puppet/03swift-devicecheck-timer 07CosmicAlpha 03c80a5b7 Remove [22:30:53] [02puppet] 07Universal-Omega pushed 1 new commit to 03swift-devicecheck-timer 13https://github.com/miraheze/puppet/commit/c06ce2eb9c8312ed3d97e1eb677f46ccfec725e6 [22:30:53] 02puppet/03swift-devicecheck-timer 07CosmicAlpha 03c06ce2e Use python3 [22:34:31] [02puppet] 07Universal-Omega merged pull request #4396: reports: convert task scheduler from cron to systemd timer (03main...03reports-timer) 13https://github.com/miraheze/puppet/pull/4396 [22:34:31] [02puppet] 07Universal-Omega pushed 1 new commit to 03main 13https://github.com/miraheze/puppet/commit/cfe8e5e3ff8947bca64d9e6a77ab2efa5d51ca78 [22:34:33] 02puppet/03main 07CosmicAlpha 03cfe8e5e reports: convert task scheduler from cron to systemd timer [22:34:35] [02puppet] 07Universal-Omega 04deleted 03reports-timer at 03c03d813 13https://api.github.com/repos/miraheze/puppet/commit/c03d813 [22:39:04] [02puppet] 07Universal-Omega merged pull request #4397: Fix paths for some backup log files (03main...03backup-log-paths) 13https://github.com/miraheze/puppet/pull/4397 [22:39:04] [02puppet] 07Universal-Omega pushed 1 new commit to 03main 13https://github.com/miraheze/puppet/commit/149362146209c9192c9ed3d46626fbb0203877ea [22:39:04] 02puppet/03main 07CosmicAlpha 031493621 Fix paths for some backup log files [22:39:05] [02puppet] 07Universal-Omega 04deleted 03backup-log-paths at 035b29479 13https://api.github.com/repos/miraheze/puppet/commit/5b29479 [22:42:07] [02puppet] 07Universal-Omega merged pull request #4398: swift: convert devicecheck cron to a systemd timer (03main...03swift-devicecheck-timer) 13https://github.com/miraheze/puppet/pull/4398 [22:42:08] [02puppet] 07Universal-Omega pushed 1 new commit to 03main 13https://github.com/miraheze/puppet/commit/6b64d3a78035dc985c0c986e27c5c27898746c8c [22:42:08] 02puppet/03main 07CosmicAlpha 036b64d3a swift: convert devicecheck cron to a systemd timer [22:42:09] [02puppet] 07Universal-Omega 04deleted 03swift-devicecheck-timer at 03c06ce2e 13https://api.github.com/repos/miraheze/puppet/commit/c06ce2e [22:44:42] !log remove /etc/cron.d/devicecheck on swiftobject* [22:44:44] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [22:45:26] !log remove www-data crontab on reports171 (all crons on reports171 are migrated to systemd timers) [22:45:28] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [22:46:19] PROBLEM - phorge171 Backups Phorge Static on phorge171 is CRITICAL: FILE_AGE CRITICAL: File not found - /var/log/phorge-backup/phorge-backup.log [22:52:57] [02puppet] 07Universal-Omega created 03varnishreqs-timer (+1 new commit) 13https://github.com/miraheze/puppet/commit/11d07355b514 [22:52:57] 02puppet/03varnishreqs-timer 07CosmicAlpha 0311d0735 varnishreqs: convert cron to systemd timer [22:53:01] [02puppet] 07Universal-Omega opened pull request #4399: varnishreqs: convert cron to systemd timer (03main...03varnishreqs-timer) 13https://github.com/miraheze/puppet/pull/4399 [22:53:06] [02puppet] 07coderabbitai[bot] commented on pull request #4399:

[…] 13https://github.com/miraheze/puppet/pull/4399#issuecomment-2978415292 [23:03:22] [02puppet] 07Universal-Omega created 03openldap-timer from 03main (+0 new commit) 13https://github.com/miraheze/puppet/compare/openldap-timer [23:04:55] [02puppet] 07Universal-Omega pushed 1 new commit to 03openldap-timer 13https://github.com/miraheze/puppet/commit/ea5b2ea2f30aedc0dbecd5d6c3238255bbcc9667 [23:04:55] 02puppet/03openldap-timer 07CosmicAlpha 03ea5b2ea openldap: convert cron to systemd timer [23:05:22] [02puppet] 07Universal-Omega opened pull request #4400: openldap: convert cron to systemd timer (03main...03openldap-timer) 13https://github.com/miraheze/puppet/pull/4400 [23:05:28] [02puppet] 07coderabbitai[bot] commented on pull request #4400:

[…] 13https://github.com/miraheze/puppet/pull/4400#issuecomment-2978434128 [23:07:11] PROBLEM - mon181 Backups Grafana on mon181 is CRITICAL: FILE_AGE CRITICAL: File not found - /var/log/grafana-backup/grafana-backup.log [23:14:28] PROBLEM - puppet181 Backups PuppetDB on puppet181 is CRITICAL: FILE_AGE CRITICAL: File not found - /var/log/puppetdb-backup/puppetdb.log [23:19:48] [02puppet] 07Universal-Omega merged pull request #4400: openldap: convert cron to systemd timer (03main...03openldap-timer) 13https://github.com/miraheze/puppet/pull/4400 [23:19:48] [02puppet] 07Universal-Omega pushed 1 new commit to 03main 13https://github.com/miraheze/puppet/commit/790d353dccfd8ba26eab45a33dee5fa332d74e9a [23:19:48] 02puppet/03main 07CosmicAlpha 03790d353 openldap: convert cron to systemd timer [23:19:50] [02puppet] 07Universal-Omega 04deleted 03openldap-timer at 03ea5b2ea 13https://api.github.com/repos/miraheze/puppet/commit/ea5b2ea [23:23:52] !log remove root crontab on ldap171 (all crons on ldap171 are migrated to systemd timers) [23:23:54] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:26:20] [02puppet] 07Universal-Omega merged pull request #4399: varnishreqs: convert cron to systemd timer (03main...03varnishreqs-timer) 13https://github.com/miraheze/puppet/pull/4399 [23:26:20] [02puppet] 07Universal-Omega pushed 1 new commit to 03main 13https://github.com/miraheze/puppet/commit/342793f1ab763d943aab7dac84442a4371e7a1be [23:26:20] 02puppet/03main 07CosmicAlpha 03342793f varnishreqs: convert cron to systemd timer [23:26:21] [02puppet] 07Universal-Omega 04deleted 03varnishreqs-timer at 0311d0735 13https://api.github.com/repos/miraheze/puppet/commit/11d0735 [23:28:00] !log remove root crontab on cp (all crons on cp are migrated to systemd timers) [23:28:03] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:30:44] [02puppet] 07Universal-Omega pushed 1 new commit to 03main 13https://github.com/miraheze/puppet/commit/48e8e863694ec831bc3ec1eaadb178f4e523815c [23:30:44] 02puppet/03main 07CosmicAlpha 0348e8e86 reports: disable logging for task scheduler… [23:38:47] [02puppet] 07Universal-Omega pushed 1 new commit to 03main 13https://github.com/miraheze/puppet/commit/cb23a8a786e9c5fd5bb15b22405c9d5430410b66 [23:38:47] 02puppet/03main 07CosmicAlpha 03cb23a8a Update prometheus_graylog_stats to timer [23:39:34] miraheze/puppet - Universal-Omega the build has errored. [23:40:24] [02puppet] 07Universal-Omega pushed 1 new commit to 03main 13https://github.com/miraheze/puppet/commit/cc046cc938f10131f571cfd47a441cf65c110100 [23:40:24] 02puppet/03main 07CosmicAlpha 03cc046cc Fix [23:42:32] !log remove root crontab on graylog161 (all crons on graylog161 are migrated to systemd timers) [23:42:35] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:45:21] [02puppet] 07Universal-Omega pushed 1 new commit to 03main 13https://github.com/miraheze/puppet/commit/d257f6157bea247e4d94346db38d28b46798817a [23:45:21] 02puppet/03main 07CosmicAlpha 03d257f61 Update prometheus_gdnsd_stats to timer [23:48:32] !log remove root crontab on ns (all crons on ns are migrated to systemd timers) [23:48:35] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log