[00:01:22] <MacFanBot>	 [02statichelp] 07WikiTideBot pushed 1 new commit to 03main 13https://github.com/miraheze/statichelp/commit/bb4fab7c68bbf21acc190fa49be2bd55c76dae1c
[00:01:22] <MacFanBot>	 02statichelp/03main 07WikiTideBot 03bb4fab7 Bot: Auto-update Tech namespace pages 2025-10-09 00:01:19
[00:09:48] <icinga-miraheze>	 RECOVERY - cp171 Disk Space on cp171 is OK: DISK OK - free space: / 54876MiB (12% inode=99%);
[00:11:18] <icinga-miraheze>	 RECOVERY - cp191 Disk Space on cp191 is OK: DISK OK - free space: / 55401MiB (12% inode=99%);
[00:11:49] <icinga-miraheze>	 RECOVERY - cp201 Disk Space on cp201 is OK: DISK OK - free space: / 54594MiB (12% inode=99%);
[03:02:59] <icinga-miraheze>	 [Grafana] FIRING: The mediawiki JobQueue backlog is increasing by more than 100 jobs a minute over an extended time period https://grafana.wikitide.net/d/GtxbP1Xnk?from=1759975280000&orgId=1&to=1759978979225
[03:04:05] <icinga-miraheze>	 PROBLEM - prometheus151 SSH on prometheus151 is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[03:04:46] <icinga-miraheze>	 PROBLEM - prometheus151 PowerDNS Recursor on prometheus151 is CRITICAL: CRITICAL - Plugin timed out while executing system call
[03:06:00] <icinga-miraheze>	 RECOVERY - prometheus151 SSH on prometheus151 is OK: SSH OK - OpenSSH_9.2p1 Debian-2+deb12u7 (protocol 2.0)
[03:06:47] <icinga-miraheze>	 PROBLEM - prometheus151 Current Load on prometheus151 is CRITICAL: LOAD CRITICAL - total load average: 11.45, 6.80, 3.07
[03:07:12] <icinga-miraheze>	 PROBLEM - prometheus151 APT on prometheus151 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds.
[03:07:16] <icinga-miraheze>	 PROBLEM - prometheus151 Puppet on prometheus151 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds.
[03:07:59] <icinga-miraheze>	 [Grafana] RESOLVED: DatasourceError https://grafana.wikitide.net/d/GtxbP1Xnk?from=1759975520000&orgId=1&to=1759979240000
[03:08:31] <icinga-miraheze>	 [Grafana] FIRING: The mediawiki JobQueue backlog is increasing by more than 100 jobs a minute over an extended time period https://grafana.wikitide.net/d/GtxbP1Xnk?from=1759975520000&orgId=1&to=1759979311694
[03:08:54] <icinga-miraheze>	 RECOVERY - prometheus151 PowerDNS Recursor on prometheus151 is OK: DNS OK: 2.519 seconds response time. prometheus151.fsslc.wtnet returns 10.0.15.116
[03:09:09] <icinga-miraheze>	 RECOVERY - prometheus151 APT on prometheus151 is OK: APT OK: 117 packages available for upgrade (0 critical updates).
[03:09:16] <icinga-miraheze>	 RECOVERY - prometheus151 Puppet on prometheus151 is OK: OK: Puppet is currently enabled, last run 34 minutes ago with 0 failures
[03:10:47] <icinga-miraheze>	 PROBLEM - prometheus151 Current Load on prometheus151 is WARNING: LOAD WARNING - total load average: 7.23, 7.43, 4.25
[03:12:46] <icinga-miraheze>	 PROBLEM - prometheus151 Current Load on prometheus151 is CRITICAL: LOAD CRITICAL - total load average: 9.53, 8.24, 4.93
[03:13:16] <icinga-miraheze>	 PROBLEM - prometheus151 PowerDNS Recursor on prometheus151 is CRITICAL: CRITICAL - Plugin timed out while executing system call
[03:13:17] <icinga-miraheze>	 PROBLEM - prometheus151 SSH on prometheus151 is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[03:13:31] <icinga-miraheze>	 [Grafana] FIRING: An unusually high number of threats are being reported by CloudFlare! https://grafana.wikitide.net/d/GtxbP1Xnk?from=1759975860000&orgId=1&to=1759979611694[Grafana] FIRING: A MediaWiki pool is sick according to CloudFlare https://grafana.wikitide.net/d/GtxbP1Xnk?from=1759975860000&orgId=1&to=1759979611694[Grafana] RESOLVED: DatasourceError https://grafana.wikitide.net/d/GtxbP1Xnk?from=1759975790000&orgId=1&to=1759979600
[03:13:31] <icinga-miraheze>	 a] RESOLVED: DatasourceNoData https://grafana.wikitide.net/d/GtxbP1Xnk?from=1759975700000&orgId=1&to=1759979360000
[03:15:16] <icinga-miraheze>	 RECOVERY - prometheus151 SSH on prometheus151 is OK: SSH OK - OpenSSH_9.2p1 Debian-2+deb12u7 (protocol 2.0)
[03:15:16] <icinga-miraheze>	 RECOVERY - prometheus151 PowerDNS Recursor on prometheus151 is OK: DNS OK: 2.966 seconds response time. prometheus151.fsslc.wtnet returns 10.0.15.116
[03:18:31] <icinga-miraheze>	 [Grafana] RESOLVED: MediaWiki Exception Rate https://grafana.wikitide.net/d/GtxbP1Xnk?from=1759976210000&orgId=1&to=1759979810000[Grafana] FIRING: An unusually high number of threats are being reported by CloudFlare! https://grafana.wikitide.net/d/GtxbP1Xnk?from=1759975860000&orgId=1&to=1759979911695[Grafana] FIRING: A MediaWiki pool is sick according to CloudFlare https://grafana.wikitide.net/d/GtxbP1Xnk?from=1759975860000&orgId=1&to=1
[03:18:32] <icinga-miraheze>	 95[Grafana] RESOLVED: DatasourceError https://grafana.wikitide.net/d/GtxbP1Xnk?from=1759975790000&orgId=1&to=1759979870000[Grafana] RESOLVED: DatasourceNoData https://grafana.wikitide.net/d/GtxbP1Xnk?from=1759976180000&orgId=1&to=1759979900000
[03:18:46] <icinga-miraheze>	 RECOVERY - prometheus151 Current Load on prometheus151 is OK: LOAD OK - total load average: 1.91, 6.11, 5.38
[03:51:54] <MacFanBot>	 [02mediawiki-repos] 07lihaohong6 opened pull request #97: T13897: install ImgTag extension (07miraheze:03main...07lihaohong6:03main) 13https://github.com/miraheze/mediawiki-repos/pull/97
[03:52:03] <MacFanBot>	 [02mediawiki-repos] 07coderabbitai[bot] commented on pull request #97: <details> […] 13https://github.com/miraheze/mediawiki-repos/pull/97#issuecomment-3384010523
[03:53:31] <icinga-miraheze>	 [Grafana] RESOLVED: DatasourceError https://grafana.wikitide.net/d/GtxbP1Xnk?from=1759975860000&orgId=1&to=1759981860000[Grafana] RESOLVED: DatasourceError https://grafana.wikitide.net/d/GtxbP1Xnk?from=1759975860000&orgId=1&to=1759981860000
[03:58:10] <MacFanBot>	 [02mw-config] 07lihaohong6 opened pull request #6138: T13897: install ImgTag extension (07miraheze:03main...07lihaohong6:03patch-2) 13https://github.com/miraheze/mw-config/pull/6138
[03:59:06] <miraheze-github>	 miraheze/mw-config - lihaohong6 the build passed.
[03:59:49] <icinga-miraheze>	 PROBLEM - cp201 Disk Space on cp201 is WARNING: DISK WARNING - free space: / 49806MiB (10% inode=99%);
[04:02:35] <miraheze-github>	 miraheze/mw-config - lihaohong6 the build passed.
[04:05:23] <miraheze-github>	 miraheze/mw-config - lihaohong6 the build passed.
[04:11:48] <icinga-miraheze>	 PROBLEM - cp171 Disk Space on cp171 is WARNING: DISK WARNING - free space: / 49844MiB (10% inode=99%);
[04:37:18] <icinga-miraheze>	 PROBLEM - cp191 Disk Space on cp191 is WARNING: DISK WARNING - free space: / 49901MiB (10% inode=99%);
[04:58:07] <icinga-miraheze>	 PROBLEM - puppet181 Check unit status of listdomains_github_push on puppet181 is CRITICAL: CRITICAL: Status of the systemd unit listdomains_github_push
[05:06:08] <icinga-miraheze>	 RECOVERY - puppet181 Check unit status of listdomains_github_push on puppet181 is OK: OK: Status of the systemd unit listdomains_github_push
[05:24:31] <MacFanBot>	 [02mw-config] 07lihaohong6 commented on pull request #6138: I think this PR is ready to be reviewed. @SomeMWDev in case you had concerns about turning off `$wgImgTagSanitizeDomain`. 13https://github.com/miraheze/mw-config/pull/6138#issuecomment-3384182817
[11:02:59] <icinga-miraheze>	 [Grafana] FIRING: The mediawiki JobQueue backlog is increasing by more than 100 jobs a minute over an extended time period https://grafana.wikitide.net/d/GtxbP1Xnk?from=1760004080000&orgId=1&to=1760007779358
[11:04:05] <icinga-miraheze>	 PROBLEM - prometheus151 SSH on prometheus151 is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[11:04:46] <icinga-miraheze>	 PROBLEM - prometheus151 Current Load on prometheus151 is CRITICAL: LOAD CRITICAL - total load average: 10.01, 5.76, 2.51
[11:05:58] <icinga-miraheze>	 RECOVERY - prometheus151 SSH on prometheus151 is OK: SSH OK - OpenSSH_9.2p1 Debian-2+deb12u7 (protocol 2.0)
[11:06:46] <icinga-miraheze>	 RECOVERY - prometheus151 Current Load on prometheus151 is OK: LOAD OK - total load average: 2.95, 4.52, 2.46
[11:07:59] <icinga-miraheze>	 [Grafana] RESOLVED: DatasourceError https://grafana.wikitide.net/d/GtxbP1Xnk?from=1760004080000&orgId=1&to=1760008010000[Grafana] RESOLVED: DatasourceNoData https://grafana.wikitide.net/d/GtxbP1Xnk?from=1760004320000&orgId=1&to=1760008040000
[12:19:10] <MacFanBot>	 [02landing] 07translatewiki pushed 1 new commit to 03main 13https://github.com/miraheze/landing/commit/524d3d20812da071f5f18167b8788670d67e210b
[12:19:10] <MacFanBot>	 02landing/03main 07translatewiki.net 03524d3d2 Localisation updates from https://translatewiki.net.
[12:20:00] <miraheze-github>	 miraheze/landing - translatewiki the build passed.
[13:18:33] <MirahezeLSBot>	 !log [skye@mwtask171] sudo -u www-data php /srv/mediawiki/1.44/maintenance/run.php CirrusSearch:UpdateSearchIndexConfig --wiki=falcomwiki --startOver (END - exit=0)
[13:18:36] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[13:49:15] <MirahezeLSBot>	 !log [macfan@mwtask181] sudo -u www-data php /srv/mediawiki/1.44/maintenance/run.php /srv/mediawiki/1.44/maintenance/rebuildall.php --wiki=utaitewiki (END - exit=0)
[13:49:16] <MirahezeLSBot>	 !log [macfan@mwtask181] sudo -u www-data php /srv/mediawiki/1.44/maintenance/run.php /srv/mediawiki/1.44/maintenance/initSiteStats.php --wiki=utaitewiki --update (END - exit=0)
[13:49:19] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[13:49:22] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[13:51:47] <MirahezeLSBot>	 !log [skye@mwtask171] sudo -u www-data php /srv/mediawiki/1.44/maintenance/run.php CirrusSearch:ForceSearchIndex --wiki=falcomwiki (END - exit=0)
[13:51:50] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[14:26:18] <MacFanBot>	 [02mediawiki-repos] 07pskyechology approved pull request #97 13https://github.com/miraheze/mediawiki-repos/pull/97#pullrequestreview-3319278536
[14:26:33] <MacFanBot>	 [02mediawiki-repos] 07pskyechology merged 07lihaohong6's pull request #97: T13897: install ImgTag extension (07miraheze:03main...07lihaohong6:03main) 13https://github.com/miraheze/mediawiki-repos/pull/97
[14:26:35] <MacFanBot>	 [02mediawiki-repos] 07pskyechology pushed 1 new commit to 03main 13https://github.com/miraheze/mediawiki-repos/commit/649c8988fc55150bda82f106c315b543d4a2b133
[14:26:37] <MacFanBot>	 02mediawiki-repos/03main 07Peter Li 03649c898 T13897: install ImgTag extension (#97)
[14:43:42] <icinga-miraheze>	 PROBLEM - mw181 HTTPS on mw181 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10004 milliseconds with 0 bytes received
[14:43:55] <icinga-miraheze>	 PROBLEM - cp161 HTTPS on cp161 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10004 milliseconds with 0 bytes received
[14:43:59] <icinga-miraheze>	 PROBLEM - cp191 Varnish Backends on cp191 is CRITICAL: 17 backends are down. mw151 mw152 mw161 mw162 mw172 mw181 mw182 mw153 mw163 mw173 mw183 mw191 mw192 mw193 mw201 mw202 mw203
[14:44:01] <icinga-miraheze>	 PROBLEM - mw201 MediaWiki Rendering on mw201 is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[14:44:30] <icinga-miraheze>	 PROBLEM - cp171 Varnish Backends on cp171 is CRITICAL: 9 backends are down. mw152 mw162 mw171 mw172 mw173 mw192 mw193 mw201 mw202
[14:44:39] <icinga-miraheze>	 PROBLEM - cp161 Varnish Backends on cp161 is CRITICAL: 8 backends are down. mw161 mw162 mw171 mw172 mw173 mw192 mw193 mw202
[14:45:06] <icinga-miraheze>	 PROBLEM - cp201 Varnish Backends on cp201 is CRITICAL: 1 backends are down. mw173
[14:45:12] <MirahezeRelay>	 <pskyechology> @rhinosf1 mooooooooom they killed miraheze
[14:45:36] <icinga-miraheze>	 RECOVERY - mw181 HTTPS on mw181 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4224 bytes in 0.064 second response time
[14:45:53] <icinga-miraheze>	 RECOVERY - cp161 HTTPS on cp161 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4328 bytes in 0.064 second response time
[14:45:54] <icinga-miraheze>	 RECOVERY - cp191 Varnish Backends on cp191 is OK: All 31 backends are healthy
[14:45:55] <MirahezeRelay>	 <rhinosf1, replying to pskyechology> Busy
[14:45:58] <icinga-miraheze>	 RECOVERY - mw201 MediaWiki Rendering on mw201 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.178 second response time
[14:46:30] <icinga-miraheze>	 RECOVERY - cp171 Varnish Backends on cp171 is OK: All 31 backends are healthy
[14:46:39] <icinga-miraheze>	 RECOVERY - cp161 Varnish Backends on cp161 is OK: All 31 backends are healthy
[14:47:05] <icinga-miraheze>	 RECOVERY - cp201 Varnish Backends on cp201 is OK: All 31 backends are healthy
[14:53:26] <MirahezeRelay>	 <pskyechology> wtf
[15:22:37] <MacFanBot>	 [02mediawiki-repos] 07pskyechology left a file comment in pull request #96 03f31dc80: Bit of a nitpick as it would work perfectly fine and might even be more correct (at least as far as Git documentation is concerned), but please remove .git from the end of the line. Literally every single other entry is like this as well. 13https://github.com/miraheze/mediawiki-repos/pull/96#discussion_r2417060729
[15:22:55] <MacFanBot>	 [02mediawiki-repos] 07pskyechology requested changes on pull request #96 13https://github.com/miraheze/mediawiki-repos/pull/96#pullrequestreview-3319375673
[15:31:46] <icinga-miraheze>	 PROBLEM - cp161 HTTP 4xx/5xx ERROR Rate on cp161 is WARNING: WARNING - NGINX Error Rate is 48%
[15:33:46] <icinga-miraheze>	 RECOVERY - cp161 HTTP 4xx/5xx ERROR Rate on cp161 is OK: OK - NGINX Error Rate is 19%
[17:02:59] <icinga-miraheze>	 [Grafana] FIRING: The mediawiki JobQueue backlog is increasing by more than 100 jobs a minute over an extended time period https://grafana.wikitide.net/d/GtxbP1Xnk?from=1760025680000&orgId=1&to=1760029379365
[17:04:05] <icinga-miraheze>	 PROBLEM - prometheus151 SSH on prometheus151 is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[17:04:46] <icinga-miraheze>	 PROBLEM - prometheus151 Current Load on prometheus151 is CRITICAL: LOAD CRITICAL - total load average: 10.55, 5.24, 2.21
[17:05:58] <icinga-miraheze>	 RECOVERY - prometheus151 SSH on prometheus151 is OK: SSH OK - OpenSSH_9.2p1 Debian-2+deb12u7 (protocol 2.0)
[17:06:46] <icinga-miraheze>	 RECOVERY - prometheus151 Current Load on prometheus151 is OK: LOAD OK - total load average: 3.21, 4.28, 2.24
[17:07:59] <icinga-miraheze>	 [Grafana] RESOLVED: DatasourceError https://grafana.wikitide.net/d/GtxbP1Xnk?from=1760025680000&orgId=1&to=1760029640000[Grafana] RESOLVED: DatasourceNoData https://grafana.wikitide.net/d/GtxbP1Xnk?from=1760025950000&orgId=1&to=1760029640000
[18:18:17] <MacFanBot>	 [02mediawiki-repos] 07songnguxyz left a file comment in pull request #96 03f31dc80: @pskyechology Getting into it now... 13https://github.com/miraheze/mediawiki-repos/pull/96#discussion_r2417603495
[18:20:52] <MirahezeRelay>	 <songngu.xyz> bruh, so it does ping in discord as well...
[18:23:52] <MirahezeRelay>	 <pskyechology> lmao nice
[18:24:27] <MacFanBot>	 [02mediawiki-repos] 07pskyechology approved pull request #96 13https://github.com/miraheze/mediawiki-repos/pull/96#pullrequestreview-3320158802
[18:24:53] <MacFanBot>	 [02mediawiki-repos] 07pskyechology merged 07songnguxyz's pull request #96: T13921: Install ParserPower (07miraheze:03main...07songnguxyz:03T13921) 13https://github.com/miraheze/mediawiki-repos/pull/96
[18:24:53] <MacFanBot>	 [02mediawiki-repos] 07pskyechology pushed 1 new commit to 03main 13https://github.com/miraheze/mediawiki-repos/commit/a91d7abc369f0ef166072b3ba8236f5296926f05
[18:24:53] <MacFanBot>	 02mediawiki-repos/03main 07Vũ Minh Nguyên 03a91d7ab T13921: Install ParserPower (#96)…
[18:26:10] <MirahezeRelay>	 <songngu.xyz> Now one more left at mw-config
[18:30:46] <icinga-miraheze>	 PROBLEM - db171 Current Load on db171 is WARNING: LOAD WARNING - total load average: 11.25, 8.03, 4.82
[18:32:46] <icinga-miraheze>	 PROBLEM - db171 Current Load on db171 is CRITICAL: LOAD CRITICAL - total load average: 46.87, 21.81, 10.10
[18:40:44] <icinga-miraheze>	 RECOVERY - db171 Current Load on db171 is OK: LOAD OK - total load average: 2.81, 9.21, 9.05
[18:55:31] <miraheze-github>	 miraheze/mw-config - songnguxyz the build passed.
[19:14:57] <MirahezeLSBot>	 !log [skye@test151] starting deploy of {'config': True} to test151
[19:14:58] <MirahezeLSBot>	 !log [skye@test151] finished deploy of {'config': True} to test151 - SUCCESS in 0s
[19:15:02] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[19:15:06] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[19:18:33] <MirahezeLSBot>	 !log [skye@test151] starting deploy of {'config': True, 'world': True, 'l10n': True, 'extension_list': True, 'versions': ['1.44', '1.45']} to test151
[19:18:36] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[19:19:33] <MirahezeRelay>	 <pskyechology> maybe it's better to deploy the extension before enabling it...
[19:26:55] <MirahezeRelay>	 <pskyechology> and that's why it blew up exttestwikibeta
[19:27:25] <MirahezeRelay>	 <pskyechology> it doesn't know what the magic word is because it's still rebuilding the localisation cache
[19:27:49] <MirahezeRelay>	 <pskyechology> (or any other scripts yet to run (you can't sue me if I'm wrong))
[19:28:10] <MirahezeRelay>	 <pskyechology> took a solid 5 min for 1.44 soo
[19:30:05] <MirahezeLSBot>	 !log [skye@test151] finished deploy of {'config': True, 'world': True, 'l10n': True, 'extension_list': True, 'versions': ['1.44', '1.45']} to test151 - SUCCESS in 692s
[19:30:09] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[19:31:38] <MirahezeRelay>	 <pskyechology> @posix_memalign can you confirm the behaviour of the #follow function on your local wiki (test case 4)? per [[Extension:ParserPower]] i have a feeling it's not supposed to return a link
[19:31:39] <MirahezeRelay>	 <Wiki-Bot#2998> <https://meta.miraheze.org/wiki/Extension:ParserPower?action=edit&redlink=1>
[19:31:49] <MirahezeRelay>	 <pskyechology> oops forgor the mw
[19:32:02] <MirahezeRelay>	 <pskyechology> [[mw:Extension:ParserPower]]
[19:32:03] <MirahezeRelay>	 <Wiki-Bot#2998> <https://www.mediawiki.org/wiki/Extension:ParserPower>
[19:33:03] <MirahezeRelay>	 <pskyechology> other test cases are a bit wonky as well
[19:35:32] <MirahezeRelay>	 <pskyechology, replying to pskyechology> oh nvm this might've been a fluke as it is fine now
[19:35:51] <MirahezeRelay>	 <posix_memalign> I just changd the test
[19:36:01] <MirahezeRelay>	 <pskyechology> ah, fair
[19:36:38] <MirahezeRelay>	 <posix_memalign> I'm rereading https://support.wiki.gg/wiki/ParserPower/basic_functions_and_tags right now. The tests were written without the extension installed, so it's understandable that some bad assumptions were made about how it works
[19:39:56] <MirahezeRelay>	 <pskyechology> [1/2] test case 7 seems to be broken due to a missing template
[19:39:56] <MirahezeRelay>	 <pskyechology> [2/2] 11 i am a bit concerned about
[19:43:33] <MirahezeRelay>	 <pskyechology> oh lol i see why 11 is messed up
[19:43:47] <MirahezeRelay>	 <pskyechology> six seven eight six seven eight
[19:44:19] <MirahezeRelay>	 <posix_memalign> Case 7 used the wrong parser function. `#argmap` -> `#iargmap` was missed possibly due to copy-pasting oversight.
[19:44:50] <MirahezeRelay>	 <pskyechology> whoopsie
[19:44:58] <MirahezeRelay>	 <pskyechology> yay it works
[19:45:20] <MirahezeRelay>	 <pskyechology> works for me, will approve and merge together with imgtag
[19:45:23] <MirahezeRelay>	 <pskyechology> and possibly others
[19:46:06] <MacFanBot>	 [02mw-config] 07pskyechology approved pull request #6137: (pushing with others later) 13https://github.com/miraheze/mw-config/pull/6137#pullrequestreview-3320410703
[19:46:50] <MirahezeRelay>	 <posix_memalign> I think the default extension changes weren't deployed to prod, so if you want to be extra cautious, you could create a new wiki on beta.
[19:47:35] <MirahezeRelay>	 <pskyechology, replying to posix_memalign> they weren't, yeah, didn't feel like logging onto mwtask at the time
[19:50:46] <MirahezeRelay>	 <abaddriverlol> config changes merged to main should always be deployed, I don't want to deploy an unrelated change at 2am and have something break I can't fix because there were undeployed commits
[19:51:44] <MirahezeRelay>	 <pskyechology, replying to abaddriverlol> this is such a valid point and I agree, and will do so in the future, but if adding/removing default extensions break smth we have bigger problems
[19:52:25] <MirahezeRelay>	 <pskyechology, replying to pskyechology> (ok this could happen if you removed CA but you aint doing that)
[19:52:45] <MirahezeRelay>	 <pskyechology> it works :3
[19:52:46] <MirahezeRelay>	 <abaddriverlol, replying to pskyechology> I feel like our infrastructure and MW in general are complicated enough that any minor change could break stuff
[19:53:16] <MirahezeRelay>	 <pskyechology, replying to abaddriverlol> the omnissiah called, we haven't prayed and sacrificed enough DB queries to him today
[19:53:44] <MirahezeRelay>	 <pskyechology> the curse of db171 having a nap will cast us all down
[19:57:10] <MirahezeRelay>	 <pskyechology> @abaddriverlol anyway if you could be so kind as to give the already-reviewed extensions (and good to be added to prod) a quick once-over before I introduce 70 new XSSes into prod https://issue-tracker.miraheze.org/tag/extensions/
[19:57:37] <MirahezeRelay>	 <pskyechology> not that I don't trust Redmin's recent reviews, but rather that your hawk-like XSS eyes might find something real quick
[20:00:17] <MirahezeRelay>	 <abaddriverlol, replying to pskyechology> I had 4-5 hours of sleep every night for the last three days (I had to wake up early but my sleep schedule didn't like to adjust to that) and will have to wake up in 8 hours, so I'm not sure I would find anything this way
[20:00:39] <MirahezeRelay>	 <pskyechology, replying to abaddriverlol> completely valid, rest well
[20:01:15] <MirahezeRelay>	 <abaddriverlol> I looked at ParserPower a few days ago and I think all parser functions return wikitext, so it's relatively unlikely there's an XSS (unless I missed something, I mainly searched for `->insertStripMarker` and `html`)
[20:06:15] <MirahezeLSBot>	 !log [skye@test151] starting deploy of {'config': True, 'world': True, 'l10n': True, 'extension_list': True, 'versions': ['1.44', '1.45']} to test151
[20:06:19] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[20:06:25] <MirahezeRelay>	 <pskyechology> fair enough
[20:08:38] <MirahezeLSBot>	 !log [skye@test151] finished deploy of {'config': True, 'world': True, 'l10n': True, 'extension_list': True, 'versions': ['1.44', '1.45']} to test151 - SUCCESS in 143s
[20:08:43] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[20:31:49] <icinga-miraheze>	 PROBLEM - cp201 Disk Space on cp201 is CRITICAL: DISK CRITICAL - free space: / 27199MiB (5% inode=99%);
[20:49:48] <icinga-miraheze>	 PROBLEM - cp171 Disk Space on cp171 is CRITICAL: DISK CRITICAL - free space: / 27190MiB (5% inode=99%);
[20:52:24] <MirahezeRelay>	 <pskyechology> @posix_memalign lazily-loaded images are not loaded until they're visible
[20:52:41] <MirahezeRelay>	 <pskyechology> ergo in the test case they will never be loaded and won't show up in the network tab either
[20:53:48] <MirahezeRelay>	 <posix_memalign> Yeah that's the expected behavior
[20:54:11] <MirahezeRelay>	 <posix_memalign> It was intended to be a solution to https://issue-tracker.miraheze.org/T13802, but I've since figured out a different solution.
[20:54:11] <MirahezeRelay>	 <pskyechology> ah i cannot read today
[20:54:21] <MirahezeRelay>	 <pskyechology, replying to pskyechology> (there was a "not" in there)
[20:55:31] <MirahezeRelay>	 <pskyechology> ok cool everything works lets see if I can squeeze more exts in for one big prod explosion
[20:56:06] <MacFanBot>	 [02mw-config] 07pskyechology approved pull request #6138: (will merge with others) 13https://github.com/miraheze/mw-config/pull/6138#pullrequestreview-3320616018
[20:57:18] <icinga-miraheze>	 PROBLEM - cp191 Disk Space on cp191 is CRITICAL: DISK CRITICAL - free space: / 27237MiB (5% inode=99%);
[20:57:36] <MirahezeRelay>	 <pskyechology> i do not have the energy to figure out a solution to WPTouch's logo problem rn so will be skipping that...
[20:58:58] <MirahezeRelay>	 <posix_memalign> [1/2] Btw I think mirabeta's api.php page exploded. Other apis are usable, though.
[20:58:58] <MirahezeRelay>	 <posix_memalign> [2/2] https://cdn.discordapp.com/attachments/808001911868489748/1425950691674820842/message.txt?ex=68e97391&is=68e82211&hm=49ac59c7fc53b4e1369ebf089b3ad670a5f6a969cd426d5c50619d72fcdffed1&
[20:59:26] <MirahezeRelay>	 <posix_memalign> It'll probably fix itself in a few weeks.
[21:00:24] <MirahezeRelay>	 <pskyechology> most likely
[21:27:27] <MirahezeRelay>	 <pskyechology> blegh I wont get around to packing if I do any more
[21:28:24] <MacFanBot>	 [02mw-config] 07pskyechology merged 07songnguxyz's pull request #6137: T13921: Install ParserPower extension (07miraheze:03main...07songnguxyz:03T13921) 13https://github.com/miraheze/mw-config/pull/6137
[21:28:25] <MacFanBot>	 [02mw-config] 07pskyechology pushed 1 new commit to 03main 13https://github.com/miraheze/mw-config/commit/6ef153ed99d78a5b6f5aa7fd10ece6bc5752b66a
[21:28:25] <MacFanBot>	 02mw-config/03main 07Vũ Minh Nguyên 036ef153e T13921: Install ParserPower extension (#6137)…
[21:28:43] <MacFanBot>	 [02mw-config] 07pskyechology merged 07lihaohong6's pull request #6138: T13897: install ImgTag extension (07miraheze:03main...07lihaohong6:03patch-2) 13https://github.com/miraheze/mw-config/pull/6138
[21:28:43] <MacFanBot>	 [02mw-config] 07pskyechology pushed 1 new commit to 03main 13https://github.com/miraheze/mw-config/commit/1a11997a2cce9cac3bc357eeb795acfdf5359894
[21:28:43] <MacFanBot>	 02mw-config/03main 07Peter Li 031a11997 T13897: install ImgTag extension (#6138)
[21:29:16] <MirahezeLSBot>	 !log [skye@mwtask181] starting deploy of {'pull': 'config', 'config': True, 'world': True, 'l10n': True, 'extension_list': True, 'versions': '1.44'} to all
[21:29:19] <miraheze-github>	 miraheze/mw-config - pskyechology the build passed.
[21:29:21] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[21:29:40] <miraheze-github>	 miraheze/mw-config - pskyechology the build passed.
[21:39:30] <MirahezeLSBot>	 !log [skye@mwtask181] finished deploy of {'pull': 'config', 'config': True, 'world': True, 'l10n': True, 'extension_list': True, 'versions': '1.44'} to all - SUCCESS in 613s
[21:39:34] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[21:52:39] <MirahezeRelay>	 <abaddriverlol> Does ParserPower need to be incompatible with PageForms?
[21:53:13] <MirahezeRelay>	 <abaddriverlol> <https://github.com/wiki-gg-oss/mediawiki-extensions-ParserPower/blob/e70e688fdebd592d97b5c9a424e56cb56a1003ad/src/ParserVariableRegistry.php#L151-L153> seems like it should be compatible
[21:53:13] <MacFanBot>	 [GitHub] [wiki-gg-oss/mediawiki-extensions-ParserPower] src/ParserVariableRegistry.php @ e70e688fdebd592d97b5c9a424e56cb56a1003ad | L151: 		if ( !defined( 'PF_VERSION' ) ) { […] (to L153)
[21:59:35] <MirahezeRelay>	 <abaddriverlol> cc @songngu.xyz
[23:02:59] <icinga-miraheze>	 [Grafana] FIRING: The mediawiki JobQueue backlog is increasing by more than 100 jobs a minute over an extended time period https://grafana.wikitide.net/d/GtxbP1Xnk?from=1760047280000&orgId=1&to=1760050979236
[23:04:05] <icinga-miraheze>	 PROBLEM - prometheus151 SSH on prometheus151 is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[23:04:46] <icinga-miraheze>	 PROBLEM - prometheus151 Current Load on prometheus151 is CRITICAL: LOAD CRITICAL - total load average: 10.93, 5.38, 2.29
[23:04:47] <icinga-miraheze>	 PROBLEM - prometheus151 PowerDNS Recursor on prometheus151 is CRITICAL: CRITICAL - Plugin timed out while executing system call
[23:05:58] <icinga-miraheze>	 RECOVERY - prometheus151 SSH on prometheus151 is OK: SSH OK - OpenSSH_9.2p1 Debian-2+deb12u7 (protocol 2.0)
[23:06:44] <icinga-miraheze>	 RECOVERY - prometheus151 PowerDNS Recursor on prometheus151 is OK: DNS OK: 0.045 seconds response time. prometheus151.fsslc.wtnet returns 10.0.15.116
[23:06:46] <icinga-miraheze>	 RECOVERY - prometheus151 Current Load on prometheus151 is OK: LOAD OK - total load average: 2.70, 4.18, 2.23
[23:07:59] <icinga-miraheze>	 [Grafana] RESOLVED: DatasourceError https://grafana.wikitide.net/d/GtxbP1Xnk?from=1760047280000&orgId=1&to=1760051240000[Grafana] FIRING: The mediawiki JobQueue backlog is increasing by more than 100 jobs a minute over an extended time period https://grafana.wikitide.net/d/GtxbP1Xnk?from=1760047550000&orgId=1&to=1760051279237
[23:10:15] <icinga-miraheze>	 PROBLEM - prometheus151 SSH on prometheus151 is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[23:11:06] <icinga-miraheze>	 PROBLEM - prometheus151 PowerDNS Recursor on prometheus151 is CRITICAL: CRITICAL - Plugin timed out while executing system call
[23:12:49] <icinga-miraheze>	 PROBLEM - prometheus151 Current Load on prometheus151 is CRITICAL: LOAD CRITICAL - total load average: 11.46, 7.97, 4.45
[23:12:59] <icinga-miraheze>	 [Grafana] FIRING: An unusually high number of threats are being reported by CloudFlare! https://grafana.wikitide.net/d/GtxbP1Xnk?from=1760047860000&orgId=1&to=1760051579237[Grafana] FIRING: A MediaWiki pool is sick according to CloudFlare https://grafana.wikitide.net/d/GtxbP1Xnk?from=1760047860000&orgId=1&to=1760051579237[Grafana] RESOLVED: DatasourceError https://grafana.wikitide.net/d/GtxbP1Xnk?from=1760047640000&orgId=1&to=1760051540
[23:12:59] <icinga-miraheze>	 a] RESOLVED: DatasourceNoData https://grafana.wikitide.net/d/GtxbP1Xnk?from=1760047550000&orgId=1&to=1760051330000
[23:13:34] <icinga-miraheze>	 PROBLEM - prometheus151 APT on prometheus151 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds.
[23:13:42] <icinga-miraheze>	 PROBLEM - prometheus151 Puppet on prometheus151 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds.
[23:13:44] <icinga-miraheze>	 PROBLEM - prometheus151 conntrack_table_size on prometheus151 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds.
[23:16:01] <icinga-miraheze>	 RECOVERY - prometheus151 conntrack_table_size on prometheus151 is OK: OK: nf_conntrack is 0 % full
[23:16:04] <icinga-miraheze>	 RECOVERY - prometheus151 Puppet on prometheus151 is OK: OK: Puppet is currently enabled, last run 10 minutes ago with 0 failures
[23:16:05] <icinga-miraheze>	 RECOVERY - prometheus151 APT on prometheus151 is OK: APT OK: 117 packages available for upgrade (0 critical updates).
[23:16:16] <icinga-miraheze>	 RECOVERY - prometheus151 SSH on prometheus151 is OK: SSH OK - OpenSSH_9.2p1 Debian-2+deb12u7 (protocol 2.0)
[23:17:18] <icinga-miraheze>	 RECOVERY - prometheus151 PowerDNS Recursor on prometheus151 is OK: DNS OK: 0.052 seconds response time. prometheus151.fsslc.wtnet returns 10.0.15.116
[23:17:59] <icinga-miraheze>	 [Grafana] RESOLVED: MediaWiki Exception Rate https://grafana.wikitide.net/d/GtxbP1Xnk?from=1760048210000&orgId=1&to=1760051810000[Grafana] FIRING: An unusually high number of threats are being reported by CloudFlare! https://grafana.wikitide.net/d/GtxbP1Xnk?from=1760047860000&orgId=1&to=1760051879237[Grafana] FIRING: A MediaWiki pool is sick according to CloudFlare https://grafana.wikitide.net/d/GtxbP1Xnk?from=1760047860000&orgId=1&to=1
[23:17:59] <icinga-miraheze>	 37[Grafana] FIRING: The mediawiki JobQueue backlog is increasing by more than 100 jobs a minute over an extended time period https://grafana.wikitide.net/d/GtxbP1Xnk?from=1760047640000&orgId=1&to=1760051879237[Grafana] FIRING: The mediawiki JobQueue backlog is increasing by more than 100 jobs a minute over an extended time period https://grafana.wikitide.net/d/GtxbP1Xnk?from=1760048210000&orgId=1&to=1760051879237
[23:18:47] <icinga-miraheze>	 PROBLEM - prometheus151 Current Load on prometheus151 is WARNING: LOAD WARNING - total load average: 3.25, 6.92, 5.43
[23:20:46] <icinga-miraheze>	 RECOVERY - prometheus151 Current Load on prometheus151 is OK: LOAD OK - total load average: 5.49, 6.66, 5.52
[23:22:59] <icinga-miraheze>	 [Grafana] RESOLVED: MediaWiki Exception Rate https://grafana.wikitide.net/d/GtxbP1Xnk?from=1760048510000&orgId=1&to=1760052110000[Grafana] FIRING: An unusually high number of threats are being reported by CloudFlare! https://grafana.wikitide.net/d/GtxbP1Xnk?from=1760047860000&orgId=1&to=1760052179238[Grafana] FIRING: A MediaWiki pool is sick according to CloudFlare https://grafana.wikitide.net/d/GtxbP1Xnk?from=1760047860000&orgId=1&to=1
[23:22:59] <icinga-miraheze>	 38[Grafana] RESOLVED: DatasourceError https://grafana.wikitide.net/d/GtxbP1Xnk?from=1760048330000&orgId=1&to=1760052050000[Grafana] RESOLVED: DatasourceNoData https://grafana.wikitide.net/d/GtxbP1Xnk?from=1760048210000&orgId=1&to=1760051900000
[23:30:12] <icinga-miraheze>	 PROBLEM - prometheus151 SSH on prometheus151 is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[23:31:49] <icinga-miraheze>	 PROBLEM - prometheus151 PowerDNS Recursor on prometheus151 is CRITICAL: CRITICAL - Plugin timed out while executing system call
[23:32:46] <icinga-miraheze>	 PROBLEM - prometheus151 Current Load on prometheus151 is CRITICAL: LOAD CRITICAL - total load average: 11.04, 8.06, 6.15
[23:33:33] <icinga-miraheze>	 PROBLEM - prometheus151 APT on prometheus151 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds.
[23:33:41] <icinga-miraheze>	 PROBLEM - prometheus151 Puppet on prometheus151 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds.
[23:33:42] <icinga-miraheze>	 PROBLEM - prometheus151 conntrack_table_size on prometheus151 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds.
[23:36:13] <icinga-miraheze>	 RECOVERY - prometheus151 SSH on prometheus151 is OK: SSH OK - OpenSSH_9.2p1 Debian-2+deb12u7 (protocol 2.0)
[23:36:13] <icinga-miraheze>	 RECOVERY - prometheus151 conntrack_table_size on prometheus151 is OK: OK: nf_conntrack is 0 % full
[23:36:16] <icinga-miraheze>	 RECOVERY - prometheus151 Puppet on prometheus151 is OK: OK: Puppet is currently enabled, last run 30 minutes ago with 0 failures
[23:36:18] <icinga-miraheze>	 RECOVERY - prometheus151 APT on prometheus151 is OK: APT OK: 117 packages available for upgrade (0 critical updates).
[23:37:59] <icinga-miraheze>	 [Grafana] RESOLVED: MediaWiki Exception Rate https://grafana.wikitide.net/d/GtxbP1Xnk?from=1760049230000&orgId=1&to=1760053070000[Grafana] FIRING: An unusually high number of threats are being reported by CloudFlare! https://grafana.wikitide.net/d/GtxbP1Xnk?from=1760047860000&orgId=1&to=1760053079239[Grafana] FIRING: A MediaWiki pool is sick according to CloudFlare https://grafana.wikitide.net/d/GtxbP1Xnk?from=1760047860000&orgId=1&to=1
[23:37:59] <icinga-miraheze>	 39[Grafana] FIRING: The mediawiki JobQueue backlog is increasing by more than 100 jobs a minute over an extended time period https://grafana.wikitide.net/d/GtxbP1Xnk?from=1760048870000&orgId=1&to=1760053079239[Grafana] FIRING: The mediawiki JobQueue backlog is increasing by more than 100 jobs a minute over an extended time period https://grafana.wikitide.net/d/GtxbP1Xnk?from=1760049440000&orgId=1&to=1760053079239
[23:38:01] <icinga-miraheze>	 RECOVERY - prometheus151 PowerDNS Recursor on prometheus151 is OK: DNS OK: 0.062 seconds response time. prometheus151.fsslc.wtnet returns 10.0.15.116
[23:38:46] <icinga-miraheze>	 PROBLEM - prometheus151 Current Load on prometheus151 is WARNING: LOAD WARNING - total load average: 3.78, 7.53, 6.85
[23:39:25] <MirahezeRelay>	 <pskyechology> those are some freaky errors
[23:40:48] <icinga-miraheze>	 PROBLEM - prometheus151 Current Load on prometheus151 is CRITICAL: LOAD CRITICAL - total load average: 8.57, 7.95, 7.08
[23:42:23] <icinga-miraheze>	 PROBLEM - prometheus151 PowerDNS Recursor on prometheus151 is CRITICAL: CRITICAL - Plugin timed out while executing system call
[23:42:23] <icinga-miraheze>	 PROBLEM - prometheus151 SSH on prometheus151 is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[23:42:59] <icinga-miraheze>	 [Grafana] FIRING: An unusually high number of threats are being reported by CloudFlare! https://grafana.wikitide.net/d/GtxbP1Xnk?from=1760047860000&orgId=1&to=1760053379240[Grafana] FIRING: A MediaWiki pool is sick according to CloudFlare https://grafana.wikitide.net/d/GtxbP1Xnk?from=1760047860000&orgId=1&to=1760053379240[Grafana] RESOLVED: DatasourceError https://grafana.wikitide.net/d/GtxbP1Xnk?from=1760049530000&orgId=1&to=1760053340
[23:42:59] <icinga-miraheze>	 a] RESOLVED: DatasourceNoData https://grafana.wikitide.net/d/GtxbP1Xnk?from=1760049440000&orgId=1&to=1760053220000
[23:45:10] <icinga-miraheze>	 PROBLEM - prometheus151 Puppet on prometheus151 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds.
[23:45:12] <icinga-miraheze>	 PROBLEM - prometheus151 APT on prometheus151 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds.
[23:47:27] <icinga-miraheze>	 RECOVERY - prometheus151 Puppet on prometheus151 is OK: OK: Puppet is currently enabled, last run 9 minutes ago with 0 failures
[23:47:28] <icinga-miraheze>	 RECOVERY - prometheus151 APT on prometheus151 is OK: APT OK: 117 packages available for upgrade (0 critical updates).
[23:47:59] <icinga-miraheze>	 [Grafana] FIRING: There has been a rise in the MediaWiki exception rate https://grafana.wikitide.net/d/GtxbP1Xnk?from=1760049890000&orgId=1&to=1760053679241[Grafana] FIRING: An unusually high number of threats are being reported by CloudFlare! https://grafana.wikitide.net/d/GtxbP1Xnk?from=1760047860000&orgId=1&to=1760053679241[Grafana] FIRING: A MediaWiki pool is sick according to CloudFlare https://grafana.wikitide.net/d/GtxbP1Xnk?from
[23:47:59] <icinga-miraheze>	 0000&orgId=1&to=1760053679241[Grafana] FIRING: The mediawiki JobQueue backlog is increasing by more than 100 jobs a minute over an extended time period https://grafana.wikitide.net/d/GtxbP1Xnk?from=1760049530000&orgId=1&to=1760053679241
[23:48:24] <icinga-miraheze>	 RECOVERY - prometheus151 SSH on prometheus151 is OK: SSH OK - OpenSSH_9.2p1 Debian-2+deb12u7 (protocol 2.0)
[23:48:36] <icinga-miraheze>	 RECOVERY - prometheus151 PowerDNS Recursor on prometheus151 is OK: DNS OK: 0.056 seconds response time. prometheus151.fsslc.wtnet returns 10.0.15.116
[23:52:41] <icinga-miraheze>	 PROBLEM - prometheus151 SSH on prometheus151 is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[23:52:58] <icinga-miraheze>	 PROBLEM - prometheus151 PowerDNS Recursor on prometheus151 is CRITICAL: CRITICAL - Plugin timed out while executing system call
[23:52:59] <icinga-miraheze>	 [Grafana] RESOLVED: MediaWiki Exception Rate https://grafana.wikitide.net/d/GtxbP1Xnk?from=1760050130000&orgId=1&to=1760053730000[Grafana] FIRING: An unusually high number of threats are being reported by CloudFlare! https://grafana.wikitide.net/d/GtxbP1Xnk?from=1760047860000&orgId=1&to=1760053979242[Grafana] FIRING: A MediaWiki pool is sick according to CloudFlare https://grafana.wikitide.net/d/GtxbP1Xnk?from=1760047860000&orgId=1&to=1
[23:52:59] <icinga-miraheze>	 42[Grafana] RESOLVED: DatasourceError https://grafana.wikitide.net/d/GtxbP1Xnk?from=1760050250000&orgId=1&to=1760053970000[Grafana] RESOLVED: DatasourceNoData https://grafana.wikitide.net/d/GtxbP1Xnk?from=1760050100000&orgId=1&to=1760053820000
[23:56:12] <icinga-miraheze>	 PROBLEM - prometheus151 APT on prometheus151 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds.
[23:56:21] <icinga-miraheze>	 PROBLEM - prometheus151 Puppet on prometheus151 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds.
[23:56:22] <icinga-miraheze>	 PROBLEM - prometheus151 conntrack_table_size on prometheus151 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds.
[23:58:44] <icinga-miraheze>	 RECOVERY - prometheus151 SSH on prometheus151 is OK: SSH OK - OpenSSH_9.2p1 Debian-2+deb12u7 (protocol 2.0)
[23:58:44] <icinga-miraheze>	 RECOVERY - prometheus151 conntrack_table_size on prometheus151 is OK: OK: nf_conntrack is 0 % full
[23:58:47] <icinga-miraheze>	 RECOVERY - prometheus151 Puppet on prometheus151 is OK: OK: Puppet is currently enabled, last run 21 minutes ago with 0 failures
[23:58:48] <icinga-miraheze>	 RECOVERY - prometheus151 APT on prometheus151 is OK: APT OK: 117 packages available for upgrade (0 critical updates).
[23:59:11] <icinga-miraheze>	 RECOVERY - prometheus151 PowerDNS Recursor on prometheus151 is OK: DNS OK: 0.093 seconds response time. prometheus151.fsslc.wtnet returns 10.0.15.116