[00:01:31] <MacFanBot>	 [02statichelp] 07WikiTideBot pushed 1 new commit to 03main 13https://github.com/miraheze/statichelp/commit/26cc71978252c55033a0bf108e12661a5eafec12
[00:01:31] <MacFanBot>	 02statichelp/03main 07WikiTideBot 0326cc719 Bot: Auto-update Tech namespace pages 2025-12-06 00:01:28
[00:12:44] <icinga-miraheze>	 RECOVERY - cp171 Disk Space on cp171 is OK: DISK OK - free space: / 70506MiB (15% inode=99%);
[00:13:47] <icinga-miraheze>	 RECOVERY - cp191 Disk Space on cp191 is OK: DISK OK - free space: / 69647MiB (15% inode=99%);
[00:14:12] <icinga-miraheze>	 RECOVERY - cp201 Disk Space on cp201 is OK: DISK OK - free space: / 71105MiB (15% inode=99%);
[01:15:12] <icinga-miraheze>	 PROBLEM - mwtask181 Current Load on mwtask181 is CRITICAL: LOAD CRITICAL - total load average: 25.45, 17.54, 12.50
[01:31:12] <icinga-miraheze>	 RECOVERY - mwtask181 Current Load on mwtask181 is OK: LOAD OK - total load average: 10.49, 19.83, 18.90
[01:37:12] <icinga-miraheze>	 PROBLEM - mwtask181 Current Load on mwtask181 is WARNING: LOAD WARNING - total load average: 23.94, 21.24, 19.50
[01:41:12] <icinga-miraheze>	 RECOVERY - mwtask181 Current Load on mwtask181 is OK: LOAD OK - total load average: 9.97, 17.73, 18.71
[03:45:20] <MacFanBot>	 [02ssl] 07WikiTideBot pushed 1 new commit to 03main 13https://github.com/miraheze/ssl/commit/6afcdbee68a0fbd7b97f49dff42851e35beed6d8
[03:45:20] <MacFanBot>	 02ssl/03main 07WikiTideBot 036afcdbe Bot: Auto-update domain lists
[09:42:53] <icinga-miraheze>	 [Grafana] FIRING: The estimated time for the MediaWiki JobQueue to clear is excessively high (8 hours) for an extended time period https://grafana.wikitide.net/d/GtxbP1Xnk?from=1765010540000&orgId=1&to=1765014173359
[09:47:53] <icinga-miraheze>	 [Grafana] RESOLVED: MediaWiki JobQueue is stalled https://grafana.wikitide.net/d/GtxbP1Xnk?from=1765010720000&orgId=1&to=1765014320000
[11:17:12] <icinga-miraheze>	 PROBLEM - mwtask181 Current Load on mwtask181 is CRITICAL: LOAD CRITICAL - total load average: 26.24, 20.60, 14.04
[11:23:12] <icinga-miraheze>	 PROBLEM - mwtask181 Current Load on mwtask181 is WARNING: LOAD WARNING - total load average: 14.59, 20.48, 16.52
[11:25:12] <icinga-miraheze>	 RECOVERY - mwtask181 Current Load on mwtask181 is OK: LOAD OK - total load average: 11.72, 17.33, 15.84
[11:29:12] <icinga-miraheze>	 PROBLEM - mwtask181 Current Load on mwtask181 is CRITICAL: LOAD CRITICAL - total load average: 26.04, 21.64, 17.88
[11:31:12] <icinga-miraheze>	 PROBLEM - mwtask181 Current Load on mwtask181 is WARNING: LOAD WARNING - total load average: 19.46, 20.61, 17.96
[11:33:12] <icinga-miraheze>	 RECOVERY - mwtask181 Current Load on mwtask181 is OK: LOAD OK - total load average: 18.19, 20.08, 18.11
[12:37:47] <icinga-miraheze>	 PROBLEM - cp191 Disk Space on cp191 is WARNING: DISK WARNING - free space: / 49905MiB (10% inode=99%);
[13:04:12] <icinga-miraheze>	 PROBLEM - cp201 Disk Space on cp201 is WARNING: DISK WARNING - free space: / 49762MiB (10% inode=99%);
[13:38:44] <icinga-miraheze>	 PROBLEM - cp171 Disk Space on cp171 is WARNING: DISK WARNING - free space: / 49799MiB (10% inode=99%);
[15:55:19] <icinga-miraheze>	 PROBLEM - kafka181 Disk Space on kafka181 is WARNING: DISK WARNING - free space: / 3129MiB (10% inode=95%);
[15:58:24] <MirahezeRelay>	 <abaddriverlol, replying to icinga-miraheze> that's a new one
[16:05:19] <icinga-miraheze>	 RECOVERY - kafka181 Disk Space on kafka181 is OK: DISK OK - free space: / 3159MiB (11% inode=95%);
[21:05:18] <MirahezeRelay>	 <paladox> !log increase kafka181 disk by 17gib (47gib = 50gb)
[21:05:22] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[21:14:40] <MacFanBot>	 [02puppet] 07paladox created 03paladox-patch-6 (+1 new commit) 13https://github.com/miraheze/puppet/commit/fafe70ccf64f
[21:14:41] <MacFanBot>	 02puppet/03paladox-patch-6 07paladox 03fafe70c Update kafka config
[21:14:44] <MacFanBot>	 [02puppet] 07paladox opened pull request #4623: Update kafka config (03main...03paladox-patch-6) 13https://github.com/miraheze/puppet/pull/4623
[21:14:49] <MacFanBot>	 [02puppet] 07coderabbitai[bot] commented on pull request #4623: <details> […] 13https://github.com/miraheze/puppet/pull/4623#issuecomment-3621153626
[21:17:44] <MacFanBot>	 [02puppet] 07paladox pushed 1 new commit to 03main 13https://github.com/miraheze/puppet/commit/72ed249c89826b10fb04488e1d8c7cbc1ca16d10
[21:17:44] <MacFanBot>	 02puppet/03main 07paladox 0372ed249 Update kafka config (#4623)…
[21:17:44] <MacFanBot>	 [02puppet] 07paladox merged pull request #4623: Update kafka config (03main...03paladox-patch-6) 13https://github.com/miraheze/puppet/pull/4623
[21:17:45] <MacFanBot>	 [02puppet] 07paladox 04deleted 03paladox-patch-6 at 03fafe70c 13https://api.github.com/repos/miraheze/puppet/commit/fafe70c
[21:22:56] <icinga-miraheze>	 [Grafana] FIRING: The mediawiki JobQueue backlog is increasing by more than 100 jobs a minute over an extended time period https://grafana.wikitide.net/d/GtxbP1Xnk?from=1765052540000&orgId=1&to=1765056176680
[21:27:56] <icinga-miraheze>	 [Grafana] RESOLVED: DatasourceNoData https://grafana.wikitide.net/d/GtxbP1Xnk?from=1765052540000&orgId=1&to=1765056320000
[22:05:47] <icinga-miraheze>	 PROBLEM - cp191 Disk Space on cp191 is CRITICAL: DISK CRITICAL - free space: / 26775MiB (5% inode=99%);
[22:15:47] <icinga-miraheze>	 RECOVERY - cp191 Disk Space on cp191 is OK: DISK OK - free space: / 69678MiB (15% inode=99%);
[22:22:27] <MirahezeLSBot>	 !log [blankeclair@mwtask181] Starting import for historyhelperwiki (XML: None; Images: images) (START)
[22:22:28] <MirahezeLSBot>	 !log [blankeclair@mwtask181] sudo -u www-data php /srv/mediawiki/1.44/maintenance/run.php importImages --wiki=historyhelperwiki --sleep=0 '--comment=Importing images from https://wiki.historyhelper.com.au ([[phorge:T14604|T14604]])' --search-recursively -- images (START)
[22:22:32] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[22:22:36] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[22:22:50] <MirahezeLSBot>	 !log [blankeclair@mwtask181] sudo -u www-data php /srv/mediawiki/1.44/maintenance/run.php importImages --wiki=historyhelperwiki --sleep=0 '--comment=Importing images from https://wiki.historyhelper.com.au ([[phorge:T14604|T14604]])' --search-recursively -- images (END - exit=0)
[22:22:51] <MirahezeLSBot>	 !log [blankeclair@mwtask181] sudo -u www-data php /srv/mediawiki/1.44/maintenance/run.php initSiteStats --wiki=historyhelperwiki --update (START)
[22:22:52] <MirahezeLSBot>	 !log [blankeclair@mwtask181] sudo -u www-data php /srv/mediawiki/1.44/maintenance/run.php initSiteStats --wiki=historyhelperwiki --update (END - exit=0)
[22:22:53] <MirahezeLSBot>	 !log [blankeclair@mwtask181] Finished import for historyhelperwiki (XML: None; Images: images) (END - exit=0)
[22:22:55] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[22:23:00] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[22:23:05] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[22:23:10] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[22:30:38] <MacFanBot>	 [02mw-config] 07BlankEclair opened pull request #6200: T14613: Set configuration settings for piggywiki (07miraheze:03main...07BlankEclair:03T14613) 13https://github.com/miraheze/mw-config/pull/6200
[22:31:29] <miraheze-github>	 miraheze/mw-config - BlankEclair the build passed.
[22:31:57] <MacFanBot>	 [02mw-config] 07BlankEclair merged pull request #6200: T14613: Set configuration settings for piggywiki (07miraheze:03main...07BlankEclair:03T14613) 13https://github.com/miraheze/mw-config/pull/6200
[22:31:58] <MacFanBot>	 [02mw-config] 07BlankEclair pushed 1 new commit to 03main 13https://github.com/miraheze/mw-config/commit/c5f672f64b690e79c3d6761b795e312148bcc140
[22:31:58] <MacFanBot>	 02mw-config/03main 07Claire Elaina 03c5f672f T14613: Set configuration settings for piggywiki (#6200)…
[22:32:06] <MirahezeLSBot>	 !log [blankeclair@mwtask181] starting deploy of {'pull': 'config', 'config': True} to all
[22:32:10] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[22:32:30] <MirahezeLSBot>	 !log [blankeclair@mwtask181] finished deploy of {'pull': 'config', 'config': True} to all - SUCCESS in 24s
[22:32:35] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[22:32:52] <MirahezeRelay>	 <bartomelow, replying to MacFanBot> https://cdn.discordapp.com/attachments/1382207711332143106/1446152102358290533/geewhiz.gif
[22:32:57] <miraheze-github>	 miraheze/mw-config - BlankEclair the build passed.
[22:34:59] <MirahezeRelay>	 <bartomelow> It takes a bit to load right
[22:58:33] <MirahezeRelay>	 <bartomelow, replying to MacFanBot> @blankeclair question, what does this do
[22:58:35] <MirahezeRelay>	 <bartomelow> Got curious
[22:59:03] <MirahezeRelay>	 <blankeclair, replying to bartomelow> it enables vector night mode, sets it as default, and disables that max widt hthingy
[22:59:40] <MirahezeRelay>	 <bartomelow, replying to blankeclair> How long does it take to do so?
[22:59:49] <MirahezeRelay>	 <blankeclair> should already apply; i deployed it
[23:00:08] <MirahezeRelay>	 <bartomelow> https://cdn.discordapp.com/attachments/808001911868489748/1446999680054595725/image.png?ex=693606f7&is=6934b577&hm=6a72867589635b8f8ab711d6d94d737763b5c9175d4aa325f5480fede8b64fc1&
[23:00:14] <MirahezeRelay>	 <bartomelow> Does it only work for some pages, and not others?
[23:06:44] <icinga-miraheze>	 PROBLEM - cp171 Disk Space on cp171 is CRITICAL: DISK CRITICAL - free space: / 26891MiB (5% inode=99%);
[23:06:54] <MirahezeRelay>	 <bartomelow> oh my PROBLEMS
[23:14:44] <icinga-miraheze>	 RECOVERY - cp171 Disk Space on cp171 is OK: DISK OK - free space: / 70921MiB (15% inode=99%);
[23:16:12] <icinga-miraheze>	 PROBLEM - cp201 Disk Space on cp201 is CRITICAL: DISK CRITICAL - free space: / 27155MiB (5% inode=99%);
[23:19:57] <MirahezeRelay>	 <bartomelow> iOH my PROBLEMS"!!!
[23:25:13] <MirahezeRelay>	 <bartomelow, replying to bartomelow> Actually I don't think it worked
[23:59:46] <MacFanBot>	 [02mw-config] 07BlankEclair opened pull request #6201: T14613: Fix $wgVectorMaxWidthOptions for piggywiki (07miraheze:03main...07BlankEclair:03T14613-2) 13https://github.com/miraheze/mw-config/pull/6201