[00:17:24] <icinga-miraheze>	 PROBLEM - cp161 HTTP 4xx/5xx ERROR Rate on cp161 is WARNING: WARNING - NGINX Error Rate is 43%
[00:19:21] <icinga-miraheze>	 RECOVERY - cp161 HTTP 4xx/5xx ERROR Rate on cp161 is OK: OK - NGINX Error Rate is 23%
[04:15:47] <MirahezeLSBot>	 !log [skye@mwtask171] sudo -u www-data php /srv/mediawiki/1.45/maintenance/run.php rebuildall --wiki=domainswiki (END - exit=0)
[04:15:48] <MirahezeLSBot>	 !log [skye@mwtask171] sudo -u www-data php /srv/mediawiki/1.45/maintenance/run.php initEditCount --wiki=domainswiki (START)
[04:15:49] <MirahezeLSBot>	 !log [skye@mwtask171] sudo -u www-data php /srv/mediawiki/1.45/maintenance/run.php initEditCount --wiki=domainswiki (END - exit=0)
[04:15:50] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[04:15:50] <MirahezeLSBot>	 !log [skye@mwtask171] sudo -u www-data php /srv/mediawiki/1.45/maintenance/run.php initSiteStats --wiki=domainswiki --update (START)
[04:15:51] <MirahezeLSBot>	 !log [skye@mwtask171] sudo -u www-data php /srv/mediawiki/1.45/maintenance/run.php initSiteStats --wiki=domainswiki --update (END - exit=0)
[04:15:52] <MirahezeLSBot>	 !log [skye@mwtask171] Finished import for domainswiki (XML: internetdomains.xml; Images: None) (END - exit=0)
[04:15:52] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[04:15:55] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[04:15:57] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[04:16:00] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[04:16:02] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[04:17:19] <MirahezeLSBot>	 !log [petramagna@test151] starting deploy of {'config': True} to test151
[04:17:20] <MirahezeLSBot>	 !log [petramagna@test151] finished deploy of {'config': True} to test151 - SUCCESS in 0s
[04:17:21] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[04:17:24] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[04:26:38] <miraheze-github>	 miraheze/mw-config - lihaohong6 the build passed.
[04:29:59] <MirahezeLSBot>	 !log [petramagna@mwtask181] starting deploy of {'pull': 'config', 'config': True} to all
[04:30:02] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[04:30:25] <MirahezeLSBot>	 !log [petramagna@mwtask181] finished deploy of {'pull': 'config', 'config': True} to all - SUCCESS in 26s
[04:30:28] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[04:31:15] <miraheze-github>	 miraheze/mw-config - lihaohong6 the build passed.
[07:11:24] <icinga-miraheze>	 PROBLEM - bast161 NTP time on bast161 is CRITICAL: NTP CRITICAL: No response from NTP server
[07:13:25] <icinga-miraheze>	 RECOVERY - bast161 NTP time on bast161 is OK: NTP OK: Offset -0.002618789673 secs
[07:53:18] <icinga-miraheze>	 PROBLEM - ns1 Puppet on ns1 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_dns]
[08:17:18] <icinga-miraheze>	 RECOVERY - ns1 Puppet on ns1 is OK: OK: Puppet is currently enabled, last run 33 seconds ago with 0 failures
[12:10:13] <MacFanBot>	 [02MirahezeMagic] 07translatewiki pushed 1 new commit to 03main 13https://github.com/miraheze/MirahezeMagic/commit/4861b0d16e4619b5af792a99cf6f5daa2f719819
[12:10:13] <MacFanBot>	 02MirahezeMagic/03main 07translatewiki.net 034861b0d Localisation updates from https://translatewiki.net.
[12:10:13] <MacFanBot>	 [02ManageWiki] 07translatewiki pushed 1 new commit to 03main 13https://github.com/miraheze/ManageWiki/commit/49597dd7b5190dbde6502b24759a6c00814d275c
[12:10:13] <MacFanBot>	 02ManageWiki/03main 07translatewiki.net 0349597dd Localisation updates from https://translatewiki.net.
[12:10:15] <MacFanBot>	 [02WikiDiscover] 07translatewiki pushed 1 new commit to 03main 13https://github.com/miraheze/WikiDiscover/commit/38ed579e7db48fce3afb84487f3b29a3d6707c28
[12:10:16] <MacFanBot>	 02WikiDiscover/03main 07translatewiki.net 0338ed579 Localisation updates from https://translatewiki.net.
[12:14:49] <miraheze-github>	 miraheze/WikiDiscover - translatewiki the build passed.
[12:15:56] <miraheze-github>	 miraheze/ManageWiki - translatewiki the build passed.
[12:17:42] <miraheze-github>	 miraheze/MirahezeMagic - translatewiki the build passed.
[17:37:07] <MacFanBot>	 [02python-functions] 07renovate[bot] created 03renovate/major-all-deps (+1 new commit) 13https://github.com/miraheze/python-functions/commit/33362c9b5136
[17:37:07] <MacFanBot>	 02python-functions/03renovate/major-all-deps 07renovate[bot] 0333362c9 chore(deps): update dependency mypy to v2
[20:02:21] <MirahezeLSBot>	 !log [void@puppet181] Upgraded packages on db182: libwireshark-data, libwsutil16, libwiretap15, libwireshark18, tshark, and wireshark-common
[20:02:24] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[20:02:41] <MirahezeLSBot>	 !log [void@puppet181] Upgraded packages on mon181: libwireshark-data, libwsutil16, libwiretap15, libwireshark18, tshark, and wireshark-common
[20:02:44] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[20:03:05] <icinga-miraheze>	 RECOVERY - mon181 APT on mon181 is OK: APT OK: 66 packages available for upgrade (0 critical updates).
[20:04:04] <icinga-miraheze>	 RECOVERY - db182 APT on db182 is OK: APT OK: 71 packages available for upgrade (0 critical updates).
[20:45:14] <icinga-miraheze>	 PROBLEM - db151 Current Load on db151 is WARNING: LOAD WARNING - total load average: 10.76, 6.43, 3.15
[20:47:14] <icinga-miraheze>	 PROBLEM - db151 Current Load on db151 is CRITICAL: LOAD CRITICAL - total load average: 14.06, 8.80, 4.40
[20:49:14] <icinga-miraheze>	 PROBLEM - db151 Current Load on db151 is WARNING: LOAD WARNING - total load average: 11.45, 9.74, 5.29
[20:51:14] <icinga-miraheze>	 RECOVERY - db151 Current Load on db151 is OK: LOAD OK - total load average: 9.63, 9.83, 5.88
[21:30:10] <MacFanBot>	 [02puppet] 07SomeMWDev created 03remove-srd-ssh-key-c (+1 new commit) 13https://github.com/miraheze/puppet/commit/44e23124e509
[21:30:10] <MacFanBot>	 02puppet/03remove-srd-ssh-key-c 07SomeRandomDeveloper 0344e2312 Remove one of my SSH keys
[21:30:19] <MacFanBot>	 [02puppet] 07SomeMWDev opened pull request #4843: Remove one of my SSH keys (03main...03remove-srd-ssh-key-c) 13https://github.com/miraheze/puppet/pull/4843
[21:30:27] <MacFanBot>	 [02puppet] 07SomeMWDev merged pull request #4843: Remove one of my SSH keys (03main...03remove-srd-ssh-key-c) 13https://github.com/miraheze/puppet/pull/4843
[21:30:27] <MacFanBot>	 [02puppet] 07SomeMWDev pushed 1 new commit to 03main 13https://github.com/miraheze/puppet/commit/10e2c1d4ac69b4989c93217ccbb025fa2610a96b
[21:30:27] <MacFanBot>	 02puppet/03main 07SomeRandomDeveloper 0310e2c1d Remove one of my SSH keys (#4843)
[21:30:29] <MacFanBot>	 [02puppet] 07SomeMWDev 04deleted 03remove-srd-ssh-key-c at 0344e2312 13https://api.github.com/repos/miraheze/puppet/commit/44e2312
[22:21:56] <icinga-miraheze>	 PROBLEM - db161 Current Load on db161 is CRITICAL: LOAD CRITICAL - total load average: 12.15, 8.90, 5.80
[22:23:55] <icinga-miraheze>	 RECOVERY - db161 Current Load on db161 is OK: LOAD OK - total load average: 10.18, 9.64, 6.47
[22:35:26] <icinga-miraheze>	 [Grafana] FIRING: The mediawiki JobQueue backlog is increasing by more than 100 jobs a minute over an extended time period https://grafana.wikitide.net/d/GtxbP1Xnk?from=1778189690000&orgId=1&to=1778193326678
[23:00:26] <icinga-miraheze>	 [Grafana] RESOLVED: MediaWiki Jobs are rapidly increasing https://grafana.wikitide.net/d/GtxbP1Xnk?from=1778191010000&orgId=1&to=1778194610000