[00:07:51] <icinga-miraheze>	 PROBLEM - os162 Current Load on os162 is WARNING: LOAD WARNING - total load average: 7.54, 5.16, 3.62
[00:09:50] <icinga-miraheze>	 RECOVERY - os162 Current Load on os162 is OK: LOAD OK - total load average: 4.32, 5.16, 3.83
[00:13:18] <icinga-miraheze>	 RECOVERY - cp38 Disk Space on cp38 is OK: DISK OK - free space: / 51309MiB (11% inode=99%);
[00:14:00] <icinga-miraheze>	 RECOVERY - cp37 Disk Space on cp37 is OK: DISK OK - free space: / 52250MiB (11% inode=99%);
[00:31:21] <MacFanBot>	 [02ManageWiki] 07Universal-Omega pushed 1 new commit to 03main 13https://github.com/miraheze/ManageWiki/commit/46c683fa8d6597a43600350802aa67c84701798c
[00:31:21] <MacFanBot>	 02ManageWiki/03main 07CosmicAlpha 0346c683f Rename variable
[00:36:06] <miraheze-github>	 miraheze/ManageWiki - Universal-Omega the build passed.
[00:49:16] <MacFanBot>	 [02ManageWiki] 07Universal-Omega created 03PermissionsError (+1 new commit) 13https://github.com/miraheze/ManageWiki/commit/331dcc6a0aca
[00:49:16] <MacFanBot>	 02ManageWiki/03PermissionsError 07CosmicAlpha 03331dcc6 FormFactory: use PermissionsError and be more clear for locked
[00:49:20] <MacFanBot>	 [02ManageWiki] 07Universal-Omega opened pull request #688: FormFactory: use PermissionsError and be more clear for locked (03main...03PermissionsError) 13https://github.com/miraheze/ManageWiki/pull/688
[00:49:25] <MacFanBot>	 [02ManageWiki] 07coderabbitai[bot] commented on pull request #688: --- […] 13https://github.com/miraheze/ManageWiki/pull/688#issuecomment-2907532369
[00:54:25] <miraheze-github>	 miraheze/ManageWiki - Universal-Omega the build passed.
[01:00:01] <MacFanBot>	 [02ManageWiki] 07Universal-Omega pushed 1 new commit to 03PermissionsError 13https://github.com/miraheze/ManageWiki/commit/af87dd0009842245150ee4c18eec19358d0853ab
[01:00:01] <MacFanBot>	 02ManageWiki/03PermissionsError 07CosmicAlpha 03af87dd0 Remove duplicate variable
[01:03:18] <icinga-miraheze>	 PROBLEM - cp38 Disk Space on cp38 is WARNING: DISK WARNING - free space: / 49885MiB (10% inode=99%);
[01:05:01] <miraheze-github>	 miraheze/ManageWiki - Universal-Omega the build passed.
[01:21:25] <icinga-miraheze>	 PROBLEM - os162 Current Load on os162 is WARNING: LOAD WARNING - total load average: 7.70, 6.77, 5.28
[01:23:24] <icinga-miraheze>	 RECOVERY - os162 Current Load on os162 is OK: LOAD OK - total load average: 5.79, 6.68, 5.44
[01:29:58] <MacFanBot>	 [02ssl] 07BlankEclair opened pull request #850: certs: Add exodusgame.wiki (07miraheze:03main...07BlankEclair:03requestssl-518) 13https://github.com/miraheze/ssl/pull/850
[01:33:43] <MacFanBot>	 [02ssl] 07BlankEclair merged pull request #850: certs: Add exodusgame.wiki (07miraheze:03main...07BlankEclair:03requestssl-518) 13https://github.com/miraheze/ssl/pull/850
[01:33:45] <MacFanBot>	 [02ssl] 07BlankEclair pushed 1 new commit to 03main 13https://github.com/miraheze/ssl/commit/4d3a518121471252a41fa9f06f35b526e30d34d0
[01:33:46] <MacFanBot>	 02ssl/03main 07Claire 034d3a518 certs: Add exodusgame.wiki (#850)…
[01:34:00] <MacFanBot>	 [02ssl] 07coderabbitai[bot] commented on pull request #850: A new configuration entry for the site "exodusgamewiki" was added to `certs.yaml`. The entry specifies the domain, sets Cloudflare as the certificate authority, and assigns the SSL certificate name "miraheze-origin-cert". No other files or […] 13https://github.com/miraheze/ssl/pull/850#issuecomment-2907545621
[01:34:03] <MacFanBot>	 [02ssl] 07coderabbitai[bot] left a review on pull request #850: **Actionable comments posted: 0** […] 13https://github.com/miraheze/ssl/pull/850#pullrequestreview-2866699981
[01:43:16] <icinga-miraheze>	 PROBLEM - os162 Current Load on os162 is WARNING: LOAD WARNING - total load average: 7.56, 6.67, 5.42
[01:45:16] <icinga-miraheze>	 RECOVERY - os162 Current Load on os162 is OK: LOAD OK - total load average: 3.70, 5.90, 5.31
[01:46:00] <icinga-miraheze>	 PROBLEM - cp37 Disk Space on cp37 is WARNING: DISK WARNING - free space: / 49765MiB (10% inode=99%);
[01:46:15] <icinga-miraheze>	 PROBLEM - cp38 Varnish Backends on cp38 is CRITICAL: 7 backends are down. mw161 mw171 mw182 mw153 mw173 mw193 mw203
[01:48:14] <icinga-miraheze>	 RECOVERY - cp38 Varnish Backends on cp38 is OK: All 31 backends are healthy
[01:59:41] <icinga-miraheze>	 PROBLEM - os162 Current Load on os162 is WARNING: LOAD WARNING - total load average: 7.25, 5.06, 4.29
[02:01:38] <icinga-miraheze>	 RECOVERY - os162 Current Load on os162 is OK: LOAD OK - total load average: 3.47, 4.48, 4.17
[02:11:22] <icinga-miraheze>	 PROBLEM - os162 Current Load on os162 is WARNING: LOAD WARNING - total load average: 7.45, 5.40, 4.37
[02:13:18] <icinga-miraheze>	 RECOVERY - os162 Current Load on os162 is OK: LOAD OK - total load average: 3.36, 4.93, 4.35
[02:49:48] <icinga-miraheze>	 PROBLEM - os162 Current Load on os162 is WARNING: LOAD WARNING - total load average: 7.47, 5.65, 4.26
[02:55:37] <icinga-miraheze>	 RECOVERY - os162 Current Load on os162 is OK: LOAD OK - total load average: 2.76, 5.58, 4.83
[04:23:15] <icinga-miraheze>	 PROBLEM - os162 Current Load on os162 is WARNING: LOAD WARNING - total load average: 7.46, 5.36, 3.46
[04:27:15] <icinga-miraheze>	 RECOVERY - os162 Current Load on os162 is OK: LOAD OK - total load average: 1.82, 4.40, 3.60
[04:39:15] <icinga-miraheze>	 PROBLEM - os162 Current Load on os162 is CRITICAL: LOAD CRITICAL - total load average: 8.25, 6.15, 4.51
[04:41:15] <icinga-miraheze>	 PROBLEM - os162 Current Load on os162 is WARNING: LOAD WARNING - total load average: 7.85, 6.70, 4.92
[04:43:15] <icinga-miraheze>	 RECOVERY - os162 Current Load on os162 is OK: LOAD OK - total load average: 5.87, 6.20, 4.95
[04:47:15] <icinga-miraheze>	 PROBLEM - os162 Current Load on os162 is WARNING: LOAD WARNING - total load average: 7.82, 7.09, 5.60
[04:49:14] <icinga-miraheze>	 RECOVERY - os162 Current Load on os162 is OK: LOAD OK - total load average: 1.43, 5.02, 5.04
[04:54:38] <icinga-miraheze>	 PROBLEM - cp37 Varnish Backends on cp37 is CRITICAL: 4 backends are down. mw151 mw163 mw173 mw192
[04:56:38] <icinga-miraheze>	 RECOVERY - cp37 Varnish Backends on cp37 is OK: All 31 backends are healthy
[06:47:49] <MacFanBot>	 [02mw-config] 07Universal-Omega pushed 1 new commit to 03main 13https://github.com/miraheze/mw-config/commit/ba4e984775f6ff361a1df7611706331ada728072
[06:47:50] <MacFanBot>	 02mw-config/03main 07CosmicAlpha 03ba4e984 Disable Mintydocs for security vulnerabilities
[06:48:22] <MirahezeLSBot>	 !log [universalomega@mwtask181] starting deploy of {'pull': 'config', 'config': True} to all
[06:48:24] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[06:48:41] <miraheze-github>	 miraheze/mw-config - Universal-Omega the build passed.
[06:48:47] <MirahezeLSBot>	 !log [universalomega@mwtask181] finished deploy of {'pull': 'config', 'config': True} to all - SUCCESS in 25s
[06:48:50] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[06:51:15] <icinga-miraheze>	 PROBLEM - os162 Current Load on os162 is WARNING: LOAD WARNING - total load average: 7.56, 6.12, 4.14
[06:53:14] <icinga-miraheze>	 RECOVERY - os162 Current Load on os162 is OK: LOAD OK - total load average: 2.39, 4.68, 3.86
[06:59:39] <MirahezeLSBot>	 !log [universalomega@mwtask181] starting deploy of {'versions': '1.43', 'upgrade_extensions': 'ManageWiki'} to all
[06:59:42] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[07:00:03] <MirahezeLSBot>	 !log [universalomega@mwtask181] finished deploy of {'versions': '1.43', 'upgrade_extensions': 'ManageWiki'} to all - SUCCESS in 24s
[07:00:06] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[07:02:40] <MacFanBot>	 [02CreateWiki] 07Universal-Omega pushed 1 new commit to 03main 13https://github.com/miraheze/CreateWiki/commit/43135df16a57e783c962b5a56ab466da566c981c
[07:02:40] <MacFanBot>	 02CreateWiki/03main 07CosmicAlpha 0343135df Use constant
[07:07:15] <icinga-miraheze>	 PROBLEM - os162 Current Load on os162 is WARNING: LOAD WARNING - total load average: 7.82, 5.32, 3.91
[07:09:15] <icinga-miraheze>	 RECOVERY - os162 Current Load on os162 is OK: LOAD OK - total load average: 4.05, 4.77, 3.89
[07:15:15] <icinga-miraheze>	 PROBLEM - os162 Current Load on os162 is WARNING: LOAD WARNING - total load average: 7.36, 6.50, 4.94
[07:17:15] <icinga-miraheze>	 RECOVERY - os162 Current Load on os162 is OK: LOAD OK - total load average: 4.07, 5.77, 4.87
[07:20:17] <miraheze-github>	 miraheze/CreateWiki - Universal-Omega the build passed.
[07:53:23] <MirahezeLSBot>	 !log [reception@mwtask181] sudo -u www-data php /srv/mediawiki/1.43/maintenance/run.php /srv/mediawiki/1.43/maintenance/importImages.php --wiki=animebattlearenawiki --summary=Imported from https://animebattlearenaaba.fandom.com --search-recursively /home/reception/animebattlearena (START)
[07:53:26] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[07:56:41] <MirahezeLSBot>	 !log [reception@mwtask181] sudo -u www-data php /srv/mediawiki/1.43/maintenance/run.php /srv/mediawiki/1.43/maintenance/importImages.php --wiki=animebattlearenawiki --summary=Imported from https://animebattlearenaaba.fandom.com --search-recursively /home/reception/animebattlearena (END - exit=0)
[07:56:45] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[07:57:00] <MirahezeLSBot>	 !log [reception@mwtask181] sudo -u www-data php /srv/mediawiki/1.43/maintenance/run.php /srv/mediawiki/1.43/maintenance/importImages.php --wiki=vgmarchiveswiki --summary=Imported from https://vgmarchives.fandom.com --search-recursively /home/reception/vgmarchives (START)
[07:57:02] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[07:57:54] <MirahezeLSBot>	 !log [reception@mwtask181] sudo -u www-data php /srv/mediawiki/1.43/maintenance/run.php /srv/mediawiki/1.43/maintenance/importImages.php --wiki=vgmarchiveswiki --summary=Imported from https://vgmarchives.fandom.com --search-recursively /home/reception/vgmarchives (END - exit=0)
[07:57:57] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[08:00:10] <icinga-miraheze>	 PROBLEM - os162 Current Load on os162 is WARNING: LOAD WARNING - total load average: 7.82, 5.95, 4.21
[08:04:03] <icinga-miraheze>	 RECOVERY - os162 Current Load on os162 is OK: LOAD OK - total load average: 3.33, 5.40, 4.46
[08:09:35] <MirahezeLSBot>	 !log [universalomega@mwtask181] starting deploy of {'versions': '1.43', 'upgrade_extensions': 'CreateWiki'} to all
[08:09:37] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[08:10:00] <MirahezeLSBot>	 !log [universalomega@mwtask181] finished deploy of {'versions': '1.43', 'upgrade_extensions': 'CreateWiki'} to all - SUCCESS in 25s
[08:10:03] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[08:12:45] <MacFanBot>	 [02ManageWiki] 07Universal-Omega pushed 1 new commit to 03main 13https://github.com/miraheze/ManageWiki/commit/518c5edd0ad9a68e69acf431a263603671985fef
[08:12:45] <MacFanBot>	 02ManageWiki/03main 07CosmicAlpha 03518c5ed FormFactory: use PermissionsError and be more clear for locked
[08:12:45] <MacFanBot>	 [02ManageWiki] 07Universal-Omega merged pull request #688: FormFactory: use PermissionsError and be more clear for locked (03main...03PermissionsError) 13https://github.com/miraheze/ManageWiki/pull/688
[08:12:46] <MacFanBot>	 [02ManageWiki] 07Universal-Omega 04deleted 03PermissionsError at 03af87dd0 13https://api.github.com/repos/miraheze/ManageWiki/commit/af87dd0
[08:15:43] <icinga-miraheze>	 PROBLEM - os162 Current Load on os162 is WARNING: LOAD WARNING - total load average: 7.58, 5.33, 4.32
[08:17:40] <icinga-miraheze>	 PROBLEM - os162 Current Load on os162 is CRITICAL: LOAD CRITICAL - total load average: 8.01, 6.16, 4.74
[08:17:48] <miraheze-github>	 miraheze/ManageWiki - Universal-Omega the build passed.
[08:18:27] <MirahezeLSBot>	 !log [reception@mwtask181] sudo -u www-data php /srv/mediawiki/1.43/maintenance/run.php /srv/mediawiki/1.43/maintenance/deleteBatch.php --wiki=animebattlearenawiki --reason=Fandom-era deletions T13738 /home/reception/animebattlearenadel.txt (START)
[08:18:28] <MirahezeLSBot>	 !log [reception@mwtask181] sudo -u www-data php /srv/mediawiki/1.43/maintenance/run.php /srv/mediawiki/1.43/maintenance/deleteBatch.php --wiki=animebattlearenawiki --reason=Fandom-era deletions T13738 /home/reception/animebattlearenadel.txt (END - exit=256)
[08:18:30] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[08:18:34] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[08:18:44] <MirahezeLSBot>	 !log [reception@mwtask181] sudo -u www-data php /srv/mediawiki/1.43/maintenance/run.php /srv/mediawiki/1.43/maintenance/deleteBatch.php --wiki=animebattlearenawiki --r=Fandom-era deletions T13738 /home/reception/animebattlearenadel.txt (START)
[08:18:47] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[08:19:36] <icinga-miraheze>	 RECOVERY - os162 Current Load on os162 is OK: LOAD OK - total load average: 2.60, 4.76, 4.39
[08:20:45] <MirahezeLSBot>	 !log [reception@mwtask181] sudo -u www-data php /srv/mediawiki/1.43/maintenance/run.php /srv/mediawiki/1.43/maintenance/deleteBatch.php --wiki=animebattlearenawiki --r=Fandom-era deletions T13738 /home/reception/animebattlearenadel.txt (END - exit=0)
[08:20:48] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[08:39:15] <icinga-miraheze>	 PROBLEM - os162 Current Load on os162 is WARNING: LOAD WARNING - total load average: 7.83, 5.88, 4.24
[08:41:15] <icinga-miraheze>	 RECOVERY - os162 Current Load on os162 is OK: LOAD OK - total load average: 5.11, 5.83, 4.44
[10:03:15] <icinga-miraheze>	 PROBLEM - os162 Current Load on os162 is WARNING: LOAD WARNING - total load average: 7.57, 5.43, 3.68
[10:07:14] <icinga-miraheze>	 RECOVERY - os162 Current Load on os162 is OK: LOAD OK - total load average: 2.55, 4.73, 3.88
[10:21:15] <icinga-miraheze>	 PROBLEM - os162 Current Load on os162 is WARNING: LOAD WARNING - total load average: 7.46, 5.77, 4.41
[10:23:15] <icinga-miraheze>	 RECOVERY - os162 Current Load on os162 is OK: LOAD OK - total load average: 3.11, 4.93, 4.29
[10:49:39] <icinga-miraheze>	 PROBLEM - cp38 Varnish Backends on cp38 is CRITICAL: 3 backends are down. mw172 mw163 mw201
[10:51:39] <icinga-miraheze>	 RECOVERY - cp38 Varnish Backends on cp38 is OK: All 31 backends are healthy
[11:31:15] <icinga-miraheze>	 PROBLEM - os162 Current Load on os162 is WARNING: LOAD WARNING - total load average: 7.52, 5.92, 4.42
[11:33:15] <icinga-miraheze>	 RECOVERY - os162 Current Load on os162 is OK: LOAD OK - total load average: 3.28, 5.14, 4.34
[12:31:15] <icinga-miraheze>	 PROBLEM - os162 Current Load on os162 is WARNING: LOAD WARNING - total load average: 7.65, 5.48, 3.70
[12:33:15] <icinga-miraheze>	 RECOVERY - os162 Current Load on os162 is OK: LOAD OK - total load average: 2.69, 4.62, 3.62
[13:41:15] <icinga-miraheze>	 PROBLEM - os162 Current Load on os162 is CRITICAL: LOAD CRITICAL - total load average: 8.08, 6.32, 4.51
[13:43:15] <icinga-miraheze>	 RECOVERY - os162 Current Load on os162 is OK: LOAD OK - total load average: 2.50, 5.02, 4.27
[13:56:44] <icinga-miraheze>	 PROBLEM - mwtask151 Current Load on mwtask151 is WARNING: LOAD WARNING - total load average: 22.12, 17.02, 10.15
[13:57:56] <icinga-miraheze>	 [Grafana] FIRING: The mediawiki JobQueue backlog is increasing by more than 100 jobs a minute over an extended time period https://grafana.wikitide.net/d/GtxbP1Xnk?from=1748177840000&orgId=1&to=1748181476704
[14:00:44] <icinga-miraheze>	 RECOVERY - mwtask151 Current Load on mwtask151 is OK: LOAD OK - total load average: 12.95, 16.89, 11.83
[14:02:56] <icinga-miraheze>	 [Grafana] RESOLVED: MediaWiki Jobs are rapidly increasing https://grafana.wikitide.net/d/GtxbP1Xnk?from=1748177840000&orgId=1&to=1748181740000
[14:03:06] <icinga-miraheze>	 PROBLEM - cp38 Disk Space on cp38 is CRITICAL: DISK CRITICAL - free space: / 27240MiB (5% inode=99%);
[14:07:15] <icinga-miraheze>	 PROBLEM - os162 Current Load on os162 is WARNING: LOAD WARNING - total load average: 7.19, 5.38, 3.82
[14:07:56] <icinga-miraheze>	 [Grafana] FIRING: The mediawiki job queue has more than 100000 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?from=1748178410000&orgId=1&to=1748182076715[Grafana] RESOLVED: MediaWiki Jobs are rapidly increasing https://grafana.wikitide.net/d/GtxbP1Xnk?from=1748178140000&orgId=1&to=1748181740000
[14:09:15] <icinga-miraheze>	 RECOVERY - os162 Current Load on os162 is OK: LOAD OK - total load average: 3.53, 4.99, 3.90
[14:15:15] <icinga-miraheze>	 PROBLEM - os162 Current Load on os162 is WARNING: LOAD WARNING - total load average: 7.51, 6.05, 4.61
[14:19:15] <icinga-miraheze>	 PROBLEM - os162 Current Load on os162 is CRITICAL: LOAD CRITICAL - total load average: 8.19, 7.15, 5.38
[14:21:15] <icinga-miraheze>	 PROBLEM - os162 Current Load on os162 is WARNING: LOAD WARNING - total load average: 7.68, 7.33, 5.67
[14:23:14] <icinga-miraheze>	 RECOVERY - os162 Current Load on os162 is OK: LOAD OK - total load average: 3.64, 6.05, 5.41
[14:50:40] <icinga-miraheze>	 PROBLEM - mon181 Current Load on mon181 is CRITICAL: LOAD CRITICAL - total load average: 14.10, 12.73, 5.73
[14:54:32] <icinga-miraheze>	 RECOVERY - mon181 Current Load on mon181 is OK: LOAD OK - total load average: 0.69, 6.18, 4.59
[14:55:35] <MirahezeLSBot>	 !log [paladox@mon181] systemctl disable prometheus-cloudflare-exporter
[14:55:38] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[15:11:15] <icinga-miraheze>	 PROBLEM - os162 Current Load on os162 is WARNING: LOAD WARNING - total load average: 7.65, 6.34, 4.67
[15:13:15] <icinga-miraheze>	 RECOVERY - os162 Current Load on os162 is OK: LOAD OK - total load average: 5.20, 5.92, 4.73
[15:17:57] <icinga-miraheze>	 [Grafana] FIRING: The mediawiki job queue has more than 100000 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?from=1748178410000&orgId=1&to=1748186277172[Grafana] FIRING: The mediawiki JobQueue backlog is increasing by more than 100 jobs a minute over an extended time period https://grafana.wikitide.net/d/GtxbP1Xnk?from=1748182640000&orgId=1&to=1748186277172
[15:22:57] <icinga-miraheze>	 [Grafana] FIRING: The mediawiki job queue has more than 100000 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?from=1748178410000&orgId=1&to=1748186577170[Grafana] RESOLVED: MediaWiki Jobs are rapidly increasing https://grafana.wikitide.net/d/GtxbP1Xnk?from=1748182760000&orgId=1&to=1748186360000
[15:24:10] <MirahezeRelay>	 <abaddriverlol> 910k jobs
[15:24:14] <MirahezeRelay>	 <abaddriverlol> sounds like a lot
[15:27:20] <MirahezeRelay>	 <abaddriverlol> https://cdn.discordapp.com/attachments/808001911868489748/1376220097286246561/Screenshot_20250525_172612_Firefox.jpg?ex=68348858&is=683336d8&hm=c4dc6044651fbb7dcc3f5c6e47c88b1165c18da270bb849fe4d082398a1791fd&
[15:37:57] <icinga-miraheze>	 [Grafana] FIRING: The mediawiki job queue has more than 100000 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?from=1748178410000&orgId=1&to=1748187477172[Grafana] FIRING: The estimated time for the MediaWiki JobQueue to clear is excessively high (8 hours) for an extended time period https://grafana.wikitide.net/d/GtxbP1Xnk?from=1748183600000&orgId=1&to=1748187477172[Grafana] RESOLVED: MediaWiki Jobs are rapidly increasing https
[15:37:57] <icinga-miraheze>	 .wikitide.net/d/GtxbP1Xnk?from=1748182760000&orgId=1&to=1748186360000
[15:47:57] <icinga-miraheze>	 [Grafana] FIRING: The mediawiki job queue has more than 100000 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?from=1748178410000&orgId=1&to=1748188077174[Grafana] RESOLVED: MediaWiki JobQueue is stalled https://grafana.wikitide.net/d/GtxbP1Xnk?from=1748184320000&orgId=1&to=1748187920000
[15:48:03] <icinga-miraheze>	 PROBLEM - cp37 Disk Space on cp37 is CRITICAL: DISK CRITICAL - free space: / 27202MiB (5% inode=99%);
[16:07:57] <icinga-miraheze>	 [Grafana] FIRING: The mediawiki job queue has more than 100000 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?from=1748178410000&orgId=1&to=1748189277176[Grafana] FIRING: The estimated time for the MediaWiki JobQueue to clear is excessively high (8 hours) for an extended time period https://grafana.wikitide.net/d/GtxbP1Xnk?from=1748185580000&orgId=1&to=1748189277176
[16:09:15] <icinga-miraheze>	 PROBLEM - os162 Current Load on os162 is CRITICAL: LOAD CRITICAL - total load average: 8.20, 6.20, 4.58
[16:11:15] <icinga-miraheze>	 PROBLEM - os162 Current Load on os162 is WARNING: LOAD WARNING - total load average: 7.80, 6.83, 5.02
[16:17:14] <icinga-miraheze>	 RECOVERY - os162 Current Load on os162 is OK: LOAD OK - total load average: 5.06, 6.13, 5.38
[17:19:15] <icinga-miraheze>	 PROBLEM - os162 Current Load on os162 is WARNING: LOAD WARNING - total load average: 7.63, 5.54, 3.86
[17:25:15] <icinga-miraheze>	 RECOVERY - os162 Current Load on os162 is OK: LOAD OK - total load average: 5.68, 6.46, 4.90
[17:47:57] <icinga-miraheze>	 [Grafana] FIRING: The mediawiki job queue has more than 100000 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?from=1748178410000&orgId=1&to=1748195277189[Grafana] RESOLVED: MediaWiki JobQueue is stalled https://grafana.wikitide.net/d/GtxbP1Xnk?from=1748191460000&orgId=1&to=1748195060000
[18:19:16] <MirahezeRelay>	 <cosmicalpha> !log added agent and raidarr to board group for email
[18:19:19] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[18:21:15] <icinga-miraheze>	 PROBLEM - os162 Current Load on os162 is WARNING: LOAD WARNING - total load average: 7.64, 6.22, 4.76
[18:28:03] <MacFanBot>	 [02mw-config] 07Universal-Omega pushed 1 new commit to 03main 13https://github.com/miraheze/mw-config/commit/6fef7ceb9b21f0d582529dab08790c6504650b6a
[18:28:03] <MacFanBot>	 02mw-config/03main 07CosmicAlpha 036fef7ce Add Agent and Raidarr to AccessIdsMap
[18:28:53] <miraheze-github>	 miraheze/mw-config - Universal-Omega the build passed.
[18:29:15] <icinga-miraheze>	 RECOVERY - os162 Current Load on os162 is OK: LOAD OK - total load average: 4.36, 6.09, 5.47
[18:30:26] <MirahezeLSBot>	 !log [universalomega@mwtask181] starting deploy of {'pull': 'config', 'config': True} to all
[18:30:29] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[18:30:55] <MirahezeLSBot>	 !log [universalomega@mwtask181] finished deploy of {'pull': 'config', 'config': True} to all - SUCCESS in 28s
[18:30:58] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[19:12:16] <icinga-miraheze>	 PROBLEM - mwtask181 Current Load on mwtask181 is CRITICAL: LOAD CRITICAL - total load average: 24.21, 18.00, 12.76
[19:16:16] <icinga-miraheze>	 PROBLEM - mwtask181 Current Load on mwtask181 is WARNING: LOAD WARNING - total load average: 23.34, 21.62, 15.52
[19:22:16] <icinga-miraheze>	 PROBLEM - mwtask181 Current Load on mwtask181 is CRITICAL: LOAD CRITICAL - total load average: 24.55, 23.32, 18.22
[19:24:16] <icinga-miraheze>	 PROBLEM - mwtask181 Current Load on mwtask181 is WARNING: LOAD WARNING - total load average: 23.77, 23.49, 18.90
[19:42:16] <icinga-miraheze>	 RECOVERY - mwtask181 Current Load on mwtask181 is OK: LOAD OK - total load average: 16.60, 20.05, 20.26
[19:52:16] <icinga-miraheze>	 PROBLEM - mwtask181 Current Load on mwtask181 is CRITICAL: LOAD CRITICAL - total load average: 25.56, 22.68, 21.17
[19:54:16] <icinga-miraheze>	 PROBLEM - mwtask181 Current Load on mwtask181 is WARNING: LOAD WARNING - total load average: 20.22, 22.01, 21.14
[20:04:16] <icinga-miraheze>	 RECOVERY - mwtask181 Current Load on mwtask181 is OK: LOAD OK - total load average: 10.14, 16.77, 19.47
[20:13:11] <icinga-miraheze>	 PROBLEM - mwtask181 Current Load on mwtask181 is WARNING: LOAD WARNING - total load average: 22.32, 20.51, 20.06
[20:15:09] <icinga-miraheze>	 RECOVERY - mwtask181 Current Load on mwtask181 is OK: LOAD OK - total load average: 19.21, 20.32, 20.07
[20:21:01] <icinga-miraheze>	 PROBLEM - mwtask181 Current Load on mwtask181 is CRITICAL: LOAD CRITICAL - total load average: 24.82, 22.35, 20.84
[20:23:00] <icinga-miraheze>	 PROBLEM - mwtask181 Current Load on mwtask181 is WARNING: LOAD WARNING - total load average: 23.14, 22.74, 21.18
[20:24:58] <icinga-miraheze>	 PROBLEM - mwtask181 Current Load on mwtask181 is CRITICAL: LOAD CRITICAL - total load average: 24.02, 23.02, 21.46
[20:26:56] <icinga-miraheze>	 PROBLEM - mwtask181 Current Load on mwtask181 is WARNING: LOAD WARNING - total load average: 21.58, 22.22, 21.34
[20:38:45] <icinga-miraheze>	 PROBLEM - mwtask181 Current Load on mwtask181 is CRITICAL: LOAD CRITICAL - total load average: 24.39, 23.04, 22.00
[20:40:43] <icinga-miraheze>	 PROBLEM - mwtask181 Current Load on mwtask181 is WARNING: LOAD WARNING - total load average: 20.06, 22.26, 21.87
[20:44:39] <icinga-miraheze>	 PROBLEM - mwtask181 Current Load on mwtask181 is CRITICAL: LOAD CRITICAL - total load average: 25.06, 22.10, 21.76
[20:46:37] <icinga-miraheze>	 PROBLEM - mwtask181 Current Load on mwtask181 is WARNING: LOAD WARNING - total load average: 23.13, 22.22, 21.84
[20:50:33] <icinga-miraheze>	 PROBLEM - mwtask181 Current Load on mwtask181 is CRITICAL: LOAD CRITICAL - total load average: 25.14, 22.85, 22.09
[20:52:31] <icinga-miraheze>	 PROBLEM - mwtask181 Current Load on mwtask181 is WARNING: LOAD WARNING - total load average: 20.26, 21.26, 21.57
[20:54:30] <icinga-miraheze>	 PROBLEM - mwtask181 Current Load on mwtask181 is CRITICAL: LOAD CRITICAL - total load average: 25.69, 22.96, 22.15
[20:58:26] <icinga-miraheze>	 PROBLEM - mwtask181 Current Load on mwtask181 is WARNING: LOAD WARNING - total load average: 19.59, 22.86, 22.41
[21:26:16] <icinga-miraheze>	 RECOVERY - mwtask181 Current Load on mwtask181 is OK: LOAD OK - total load average: 8.99, 17.19, 20.06
[21:47:57] <icinga-miraheze>	 [Grafana] FIRING: The mediawiki job queue has more than 100000 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?from=1748178410000&orgId=1&to=1748209677224
[22:15:39] <icinga-miraheze>	 PROBLEM - cp38 Varnish Backends on cp38 is CRITICAL: 3 backends are down. mw151 mw171 mw182
[22:17:39] <icinga-miraheze>	 RECOVERY - cp38 Varnish Backends on cp38 is OK: All 31 backends are healthy
[22:49:26] <MacFanBot>	 [02ManageWiki] 07Universal-Omega created 03form-service (+1 new commit) 13https://github.com/miraheze/ManageWiki/commit/dff9cbc9ce43
[22:49:26] <MacFanBot>	 02ManageWiki/03form-service 07CosmicAlpha 03dff9cbc Part 1 converting FormFactoryBuilder to service
[22:49:31] <MacFanBot>	 [02ManageWiki] 07Universal-Omega opened pull request #689: Part 1 converting FormFactoryBuilder to service (03main...03form-service) 13https://github.com/miraheze/ManageWiki/pull/689
[22:49:37] <MacFanBot>	 [02ManageWiki] 07coderabbitai[bot] commented on pull request #689: --- […] 13https://github.com/miraheze/ManageWiki/pull/689#issuecomment-2908134554
[22:50:07] <MacFanBot>	 [02ManageWiki] 07Universal-Omega pushed 1 new commit to 03form-service 13https://github.com/miraheze/ManageWiki/commit/e59ac197c9a7bbdebda4a45ca51ecb00b6e4bc92
[22:50:07] <MacFanBot>	 02ManageWiki/03form-service 07CosmicAlpha 03e59ac19 Fix
[22:51:35] <MacFanBot>	 [02ManageWiki] 07github-actions[bot] pushed 1 new commit to 03form-service 13https://github.com/miraheze/ManageWiki/commit/a4e318567b14cfdebb96c3eddb4c181cd72db663
[22:51:35] <MacFanBot>	 02ManageWiki/03form-service 07github-actions 03a4e3185 CI: lint code to MediaWiki standards…
[22:52:59] <miraheze-github>	 miraheze/ManageWiki - Universal-Omega the build has errored.
[23:11:39] <icinga-miraheze>	 PROBLEM - cp38 Varnish Backends on cp38 is CRITICAL: 6 backends are down. mw161 mw162 mw163 mw173 mw191 mw201
[23:13:39] <icinga-miraheze>	 RECOVERY - cp38 Varnish Backends on cp38 is OK: All 31 backends are healthy
[23:22:57] <icinga-miraheze>	 [Grafana] RESOLVED: High Job Queue Backlog https://grafana.wikitide.net/d/GtxbP1Xnk?from=1748178410000&orgId=1&to=1748215310000