[00:01:18] <MacFanBot>	 [02statichelp] 07WikiTideBot pushed 1 new commit to 03main 13https://github.com/miraheze/statichelp/commit/b75a7cb52adbb65182443560cd62972ca4c9cba2
[00:01:18] <MacFanBot>	 02statichelp/03main 07WikiTideBot 03b75a7cb Bot: Auto-update Tech namespace pages 2025-12-29 00:01:15
[00:21:16] <icinga-miraheze>	 RECOVERY - cp201 Disk Space on cp201 is OK: DISK OK - free space: / 69278MiB (15% inode=99%);
[02:22:21] <icinga-miraheze>	 PROBLEM - reports171 Current Load on reports171 is CRITICAL: LOAD CRITICAL - total load average: 5.03, 3.65, 1.52
[02:24:18] <icinga-miraheze>	 RECOVERY - reports171 Current Load on reports171 is OK: LOAD OK - total load average: 0.96, 2.50, 1.35
[04:14:26] <icinga-miraheze>	 PROBLEM - db161 Current Load on db161 is WARNING: LOAD WARNING - total load average: 6.61, 10.99, 5.88
[04:16:25] <icinga-miraheze>	 RECOVERY - db161 Current Load on db161 is OK: LOAD OK - total load average: 2.04, 7.92, 5.35
[09:33:51] <icinga-miraheze>	 PROBLEM - cp201 Disk Space on cp201 is WARNING: DISK WARNING - free space: / 49743MiB (10% inode=99%);
[10:05:07] <icinga-miraheze>	 PROBLEM - mw172 HTTPS on mw172 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10003 milliseconds with 0 bytes received
[10:05:14] <icinga-miraheze>	 PROBLEM - cp171 HTTPS on cp171 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10003 milliseconds with 0 bytes received
[10:05:26] <icinga-miraheze>	 PROBLEM - mw181 HTTPS on mw181 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 502
[10:05:41] <icinga-miraheze>	 PROBLEM - mw172 MediaWiki Rendering on mw172 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 8191 bytes in 0.011 second response time
[10:05:41] <icinga-miraheze>	 PROBLEM - cp191 Varnish Backends on cp191 is CRITICAL: 1 backends are down. mw153
[10:05:45] <icinga-miraheze>	 PROBLEM - mw153 HTTPS on mw153 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10002 milliseconds with 0 bytes received
[10:05:49] <RhinosF1>	 o/
[10:06:00] <icinga-miraheze>	 PROBLEM - mw203 HTTPS on mw203 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10004 milliseconds with 0 bytes received
[10:06:05] <icinga-miraheze>	 PROBLEM - mw161 HTTPS on mw161 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 502
[10:06:14] <icinga-miraheze>	 PROBLEM - cp201 Varnish Backends on cp201 is CRITICAL: 5 backends are down. mw151 mw162 mw153 mw163 mw193
[10:06:18] <icinga-miraheze>	 PROBLEM - mw202 HTTPS on mw202 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10004 milliseconds with 0 bytes received
[10:06:22] <icinga-miraheze>	 PROBLEM - mw201 HTTPS on mw201 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 502
[10:06:24] <icinga-miraheze>	 PROBLEM - mw183 HTTPS on mw183 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 502
[10:06:27] <icinga-miraheze>	 PROBLEM - mw201 MediaWiki Rendering on mw201 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 8191 bytes in 0.012 second response time
[10:06:32] <icinga-miraheze>	 PROBLEM - mw183 MediaWiki Rendering on mw183 is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[10:06:32] <icinga-miraheze>	 PROBLEM - test151 MediaWiki Rendering on test151 is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[10:06:33] <icinga-miraheze>	 PROBLEM - mw152 MediaWiki Rendering on mw152 is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[10:06:37] <icinga-miraheze>	 PROBLEM - mw192 MediaWiki Rendering on mw192 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 8191 bytes in 0.012 second response time
[10:06:39] <icinga-miraheze>	 PROBLEM - mw191 MediaWiki Rendering on mw191 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 8191 bytes in 0.011 second response time
[10:06:43] <icinga-miraheze>	 PROBLEM - mw161 MediaWiki Rendering on mw161 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 8191 bytes in 0.011 second response time
[10:06:43] <icinga-miraheze>	 PROBLEM - mw202 MediaWiki Rendering on mw202 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 8191 bytes in 0.012 second response time
[10:06:48] <icinga-miraheze>	 PROBLEM - cp171 Varnish Backends on cp171 is CRITICAL: 3 backends are down. mw151 mw162 mw163
[10:07:15] <icinga-miraheze>	 RECOVERY - mw172 HTTPS on mw172 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4224 bytes in 9.067 second response time
[10:07:27] <icinga-miraheze>	 RECOVERY - mw181 HTTPS on mw181 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4224 bytes in 1.396 second response time
[10:07:39] <icinga-miraheze>	 RECOVERY - mw172 MediaWiki Rendering on mw172 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.190 second response time
[10:07:40] <icinga-miraheze>	 RECOVERY - mw153 HTTPS on mw153 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4224 bytes in 0.058 second response time
[10:07:41] <icinga-miraheze>	 RECOVERY - cp191 Varnish Backends on cp191 is OK: All 31 backends are healthy
[10:07:57] <icinga-miraheze>	 RECOVERY - mw203 HTTPS on mw203 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4224 bytes in 0.067 second response time
[10:08:04] <icinga-miraheze>	 RECOVERY - mw161 HTTPS on mw161 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4224 bytes in 0.062 second response time
[10:08:12] <icinga-miraheze>	 RECOVERY - mw202 HTTPS on mw202 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4224 bytes in 0.069 second response time
[10:08:14] <icinga-miraheze>	 RECOVERY - cp201 Varnish Backends on cp201 is OK: All 31 backends are healthy
[10:08:17] <icinga-miraheze>	 RECOVERY - mw201 HTTPS on mw201 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4224 bytes in 0.062 second response time
[10:08:23] <icinga-miraheze>	 RECOVERY - mw201 MediaWiki Rendering on mw201 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.227 second response time
[10:08:24] <icinga-miraheze>	 RECOVERY - mw183 HTTPS on mw183 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4224 bytes in 0.069 second response time
[10:08:29] <icinga-miraheze>	 RECOVERY - mw183 MediaWiki Rendering on mw183 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.195 second response time
[10:08:32] <icinga-miraheze>	 RECOVERY - mw192 MediaWiki Rendering on mw192 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.185 second response time
[10:08:32] <icinga-miraheze>	 RECOVERY - mw152 MediaWiki Rendering on mw152 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.183 second response time
[10:08:34] <RhinosF1>	 Okay, everything looked shit
[10:08:41] <icinga-miraheze>	 RECOVERY - mw161 MediaWiki Rendering on mw161 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.887 second response time
[10:08:41] <icinga-miraheze>	 RECOVERY - mw202 MediaWiki Rendering on mw202 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.246 second response time
[10:08:48] <icinga-miraheze>	 RECOVERY - cp171 Varnish Backends on cp171 is OK: All 31 backends are healthy
[10:09:12] <icinga-miraheze>	 RECOVERY - cp171 HTTPS on cp171 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4273 bytes in 0.051 second response time
[10:09:17] <icinga-miraheze>	 PROBLEM - puppet181 Check unit status of listdomains_github_push on puppet181 is CRITICAL: CRITICAL: Status of the systemd unit listdomains_github_push
[10:10:32] <icinga-miraheze>	 RECOVERY - test151 MediaWiki Rendering on test151 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.679 second response time
[10:10:45] <icinga-miraheze>	 RECOVERY - mw191 MediaWiki Rendering on mw191 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.192 second response time
[10:15:17] <icinga-miraheze>	 RECOVERY - puppet181 Check unit status of listdomains_github_push on puppet181 is OK: OK: Status of the systemd unit listdomains_github_push
[12:14:30] <MacFanBot>	 [02landing] 07translatewiki pushed 1 new commit to 03main 13https://github.com/miraheze/landing/commit/afef4bc91f20678314035fa8a3a40a3843baee94
[12:14:30] <MacFanBot>	 02landing/03main 07translatewiki.net 03afef4bc Localisation updates from https://translatewiki.net.
[12:14:31] <MacFanBot>	 [02MirahezeMagic] 07translatewiki pushed 1 new commit to 03main 13https://github.com/miraheze/MirahezeMagic/commit/732e92b16f4aa355089d9eee179ecb7641d2751f
[12:14:32] <MacFanBot>	 02MirahezeMagic/03main 07translatewiki.net 03732e92b Localisation updates from https://translatewiki.net.
[12:14:33] <MacFanBot>	 [02MatomoAnalytics] 07translatewiki pushed 1 new commit to 03main 13https://github.com/miraheze/MatomoAnalytics/commit/429ec9297ad8eadc8c33e7f8cfad8a8656ca53b7
[12:14:35] <MacFanBot>	 02MatomoAnalytics/03main 07translatewiki.net 03429ec92 Localisation updates from https://translatewiki.net.
[12:14:37] <MacFanBot>	 [02ManageWiki] 07translatewiki pushed 1 new commit to 03main 13https://github.com/miraheze/ManageWiki/commit/590fe03d231f344e4eeef46836f64e8205a1fead
[12:14:39] <MacFanBot>	 02ManageWiki/03main 07translatewiki.net 03590fe03 Localisation updates from https://translatewiki.net.
[12:15:20] <miraheze-github>	 miraheze/landing - translatewiki the build passed.
[12:18:15] <miraheze-github>	 miraheze/MirahezeMagic - translatewiki the build has errored.
[12:19:24] <miraheze-github>	 miraheze/MatomoAnalytics - translatewiki the build passed.
[12:20:44] <miraheze-github>	 miraheze/ManageWiki - translatewiki the build passed.
[12:31:28] <icinga-miraheze>	 PROBLEM - cp191 Disk Space on cp191 is WARNING: DISK WARNING - free space: / 49876MiB (10% inode=99%);
[13:51:03] <icinga-miraheze>	 PROBLEM - cp171 Disk Space on cp171 is WARNING: DISK WARNING - free space: / 49789MiB (10% inode=99%);
[14:55:21] <MacFanBot>	 [02ssl] 07WikiTideBot pushed 1 new commit to 03main 13https://github.com/miraheze/ssl/commit/ad78ec59dec57cd16ab69be7781606526e4d4522
[14:55:21] <MacFanBot>	 02ssl/03main 07WikiTideBot 03ad78ec5 Bot: Auto-update domain lists
[15:24:39] <MacFanBot>	 [02RemovePII] 07SomeMWDev created 03remove-comments (+1 new commit) 13https://github.com/miraheze/RemovePII/commit/458a6479dae6
[15:24:39] <MacFanBot>	 02RemovePII/03remove-comments 07SomeRandomDeveloper 03458a647 Remove Comments table update…
[15:24:58] <MacFanBot>	 [02RemovePII] 07SomeMWDev opened pull request #115: Remove Comments table update (03main...03remove-comments) 13https://github.com/miraheze/RemovePII/pull/115
[15:25:04] <MacFanBot>	 [02RemovePII] 07coderabbitai[bot] commented on pull request #115: <details> […] 13https://github.com/miraheze/RemovePII/pull/115#issuecomment-3696814603
[15:29:43] <miraheze-github>	 miraheze/RemovePII - SomeMWDev the build passed.
[16:44:49] <MacFanBot>	 [02MirahezeMagic] 07Reception123 pushed 1 new commit to 03Reception123-patch-3 13https://github.com/miraheze/MirahezeMagic/commit/9950a6b01353809e4048f7ef79b1e955dcf82c7d
[16:44:49] <MacFanBot>	 02MirahezeMagic/03Reception123-patch-3 07Reception123 039950a6b Update RequestWiki.php
[16:48:03] <miraheze-github>	 miraheze/MirahezeMagic - Reception123 the build has errored.
[17:06:23] <icinga-miraheze>	 PROBLEM - test151 Puppet on test151 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_checkout_MediaWiki-REL1_45 MirahezeMagic]
[17:08:41] <icinga-miraheze>	 PROBLEM - mw182 Puppet on mw182 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[nginx]
[17:09:13] <icinga-miraheze>	 PROBLEM - mw183 Puppet on mw183 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[nginx]
[17:13:27] <MacFanBot>	 [02puppet] 07paladox created 03paladox-patch-6 (+1 new commit) 13https://github.com/miraheze/puppet/commit/101e791ca0a6
[17:13:27] <MacFanBot>	 02puppet/03paladox-patch-6 07paladox 03101e791 openldap: Fix reloading slapd service on cert update
[17:13:31] <MacFanBot>	 [02puppet] 07paladox opened pull request #4643: openldap: Fix reloading slapd service on cert update (03main...03paladox-patch-6) 13https://github.com/miraheze/puppet/pull/4643
[17:13:39] <MacFanBot>	 [02puppet] 07coderabbitai[bot] commented on pull request #4643: <details> […] 13https://github.com/miraheze/puppet/pull/4643#issuecomment-3697067229
[17:14:19] <MacFanBot>	 [02puppet] 07paladox pushed 1 new commit to 03paladox-patch-6 13https://github.com/miraheze/puppet/commit/18269dfb334dc655a856eb2032632184b173733a
[17:14:19] <MacFanBot>	 02puppet/03paladox-patch-6 07paladox 0318269df Change notify to notify_service for slapd
[17:14:58] <MacFanBot>	 [02puppet] 07paladox pushed 1 new commit to 03main 13https://github.com/miraheze/puppet/commit/c3eba21f261ee021f26a7098af518a7759b3d98a
[17:14:58] <MacFanBot>	 02puppet/03main 07paladox 03c3eba21 openldap: Fix reloading slapd service on cert update (#4643)
[17:15:00] <MacFanBot>	 [02puppet] 07paladox merged pull request #4643: openldap: Fix reloading slapd service on cert update (03main...03paladox-patch-6) 13https://github.com/miraheze/puppet/pull/4643
[17:15:02] <MacFanBot>	 [02puppet] 07paladox 04deleted 03paladox-patch-6 at 0318269df 13https://api.github.com/repos/miraheze/puppet/commit/18269df
[17:15:42] <MirahezeRelay>	 <abaddriverlol> does this fix the nginx error?
[17:16:05] <MirahezeRelay>	 <abaddriverlol> [1/4] ```
[17:16:06] <MirahezeRelay>	 <abaddriverlol> [2/4] Dec 29 17:05:30 mw182 puppet-run[4027607]: Dec 29 17:05:30 mw182 nginx[4031246]: 2025/12/29 17:05:30 [emerg] 4031246#4031246: cannot load certificate "/etc/ssl/localcerts/wikitide.net.crt": PEM_read_
[17:16:06] <MirahezeRelay>	 <abaddriverlol> [3/4] bio_X509_AUX() failed (SSL: error:0480006C:PEM routines::no start line:Expecting: TRUSTED CERTIFICATE)
[17:16:06] <MirahezeRelay>	 <abaddriverlol> [4/4] ```
[17:17:14] <MacFanBot>	 [02puppet] 07paladox pushed 1 new commit to 03main 13https://github.com/miraheze/puppet/commit/e2f6ce22240fd91556cee32c7de1de52a643417e
[17:17:14] <MacFanBot>	 02puppet/03main 07paladox 03e2f6ce2 Improve condition check for notify_service
[17:18:09] <icinga-miraheze>	 PROBLEM - cloud15 Puppet on cloud15 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle.
[17:18:23] <icinga-miraheze>	 PROBLEM - ldap171 Puppet on ldap171 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle.
[17:18:23] <icinga-miraheze>	 PROBLEM - ns1 Puppet on ns1 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle.
[17:18:42] <icinga-miraheze>	 PROBLEM - mw193 Puppet on mw193 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle.
[17:19:23] <MirahezeRelay>	 <abaddriverlol> `Error while evaluating a Function Call, 'defined' parameter 'vals' expects a value of type String, Type[CatalogEntry], or Type[Type], got Undef`
[17:19:40] <icinga-miraheze>	 PROBLEM - os151 Puppet on os151 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle.
[17:19:41] <icinga-miraheze>	 PROBLEM - mem151 Puppet on mem151 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle.
[17:23:12] <icinga-miraheze>	 PROBLEM - mw153 Puppet on mw153 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[nginx]
[17:23:22] <icinga-miraheze>	 PROBLEM - mw171 Puppet on mw171 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[nginx]
[17:24:48] <icinga-miraheze>	 PROBLEM - mw191 Puppet on mw191 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[nginx]
[17:27:02] <MacFanBot>	 [02puppet] 07paladox created 03paladox-patch-6 (+1 new commit) 13https://github.com/miraheze/puppet/commit/7302503f70a7
[17:27:02] <MacFanBot>	 02puppet/03paladox-patch-6 07paladox 037302503 Change notify_service type to Optional[String]
[17:27:05] <MacFanBot>	 [02puppet] 07paladox opened pull request #4644: Change notify_service type to Optional[String] (03main...03paladox-patch-6) 13https://github.com/miraheze/puppet/pull/4644
[17:27:15] <MacFanBot>	 [02puppet] 07coderabbitai[bot] commented on pull request #4644: <details> […] 13https://github.com/miraheze/puppet/pull/4644#issuecomment-3697094458
[17:27:36] <icinga-miraheze>	 PROBLEM - cp191 Puppet on cp191 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[haproxy]
[17:27:41] <MacFanBot>	 [02puppet] 07paladox pushed 1 new commit to 03paladox-patch-6 13https://github.com/miraheze/puppet/commit/b5b34fbb3061c2f1668f615ee52b61112789d473
[17:27:43] <MacFanBot>	 02puppet/03paladox-patch-6 07paladox 03b5b34fb Fix notify_service syntax for slapd service
[17:28:10] <MacFanBot>	 [02puppet] 07paladox merged pull request #4644: Change notify_service type to Optional[String] (03main...03paladox-patch-6) 13https://github.com/miraheze/puppet/pull/4644
[17:28:10] <MacFanBot>	 [02puppet] 07paladox pushed 1 new commit to 03main 13https://github.com/miraheze/puppet/commit/df438ada8e0446637bb917f44f378f422e6f0fbd
[17:28:10] <MacFanBot>	 02puppet/03main 07paladox 03df438ad Change notify_service type to Optional[String] (#4644)
[17:28:11] <MacFanBot>	 [02puppet] 07paladox 04deleted 03paladox-patch-6 at 03b5b34fb 13https://api.github.com/repos/miraheze/puppet/commit/b5b34fb
[17:28:41] <icinga-miraheze>	 PROBLEM - cp161 Puppet on cp161 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[nginx]
[17:29:32] <icinga-miraheze>	 PROBLEM - graylog161 graylog tls port 12210 ssl cert check on graylog161 is CRITICAL: CRITICAL - Cannot make SSL connection.80CBE8A9347F0000:error:0A000126:SSL routines:ssl3_read_n:unexpected eof while reading:../ssl/record/rec_layer_s3.c:322
[17:30:21] <icinga-miraheze>	 RECOVERY - ldap171 Puppet on ldap171 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[17:30:26] <icinga-miraheze>	 PROBLEM - mw192 Puppet on mw192 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[nginx]
[17:30:54] <RhinosF1>	 Huh
[17:31:17] <RhinosF1>	 @paladox: wtf are you deploying
[17:31:18] <icinga-miraheze>	 PROBLEM - mw163 Puppet on mw163 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[nginx]
[17:31:46] <MirahezeRelay>	 <paladox> i'm trying to fix puppet not restarting ldap when cert changes
[17:32:02] <icinga-miraheze>	 PROBLEM - mw151 Puppet on mw151 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[nginx]
[17:33:06] <MirahezeRelay>	 <paladox> @rhinosf1
[17:34:02] <icinga-miraheze>	 RECOVERY - mw151 Puppet on mw151 is OK: OK: Puppet is currently enabled, last run 1 second ago with 0 failures
[17:34:07] <icinga-miraheze>	 PROBLEM - test151 MediaWiki Rendering on test151 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 4257 bytes in 0.030 second response time
[17:34:40] <icinga-miraheze>	 RECOVERY - mw193 Puppet on mw193 is OK: OK: Puppet is currently enabled, last run 2 seconds ago with 0 failures
[17:34:47] <icinga-miraheze>	 RECOVERY - mw191 Puppet on mw191 is OK: OK: Puppet is currently enabled, last run 9 seconds ago with 0 failures
[17:35:10] <icinga-miraheze>	 RECOVERY - mw183 Puppet on mw183 is OK: OK: Puppet is currently enabled, last run 32 seconds ago with 0 failures
[17:35:14] <icinga-miraheze>	 RECOVERY - mw153 Puppet on mw153 is OK: OK: Puppet is currently enabled, last run 36 seconds ago with 0 failures
[17:35:16] <icinga-miraheze>	 RECOVERY - mw163 Puppet on mw163 is OK: OK: Puppet is currently enabled, last run 35 seconds ago with 0 failures
[17:35:23] <icinga-miraheze>	 RECOVERY - mw171 Puppet on mw171 is OK: OK: Puppet is currently enabled, last run 46 seconds ago with 0 failures
[17:35:46] <MirahezeRelay>	 <rhinosf1, replying to icinga-miraheze> No you're not
[17:35:59] <MirahezeRelay>	 <rhinosf1> It's a deployment freeze and you're breaking puppet everywhere
[17:36:26] <icinga-miraheze>	 RECOVERY - mw192 Puppet on mw192 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[17:36:41] <icinga-miraheze>	 RECOVERY - mw182 Puppet on mw182 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures
[17:38:00] <icinga-miraheze>	 PROBLEM - test151 Check unit status of mediawiki_job_mediamoderation-scan on test151 is CRITICAL: CRITICAL: Status of the systemd unit mediawiki_job_mediamoderation-scan
[17:38:07] <icinga-miraheze>	 RECOVERY - test151 MediaWiki Rendering on test151 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.692 second response time
[17:38:20] <MirahezeRelay>	 <rhinosf1> @paladox
[17:38:30] <MirahezeRelay>	 <rhinosf1> Get things stable please
[17:39:05] <MirahezeRelay>	 <paladox> it's stable, it was an error in the code that had no impact on service and it's fixed, i haven't managed to fix the service not being restarted but i'll stop.
[17:39:54] <MirahezeRelay>	 <abaddriverlol, replying to icinga-miraheze> ^ graylog has an issue as well which I assume is unrelated?
[17:42:43] <MirahezeRelay>	 <rhinosf1, replying to paladox> Okay, please stop deploying things that have a high chance of breaking stuff without testing
[17:42:50] <MirahezeRelay>	 <rhinosf1> I'll open a deployment freeze breach ticket
[17:44:54] <icinga-miraheze>	 RECOVERY - graylog161 graylog tls port 12210 ssl cert check on graylog161 is OK: OK - Certificate 'wikitide.net' will expire on Fri Mar 13 15:30:27 2026 +0000.TCP OK - 0.156 second response time on localhost port 12210
[17:45:22] <icinga-miraheze>	 RECOVERY - cp161 Puppet on cp161 is OK: OK: Puppet is currently enabled, last run 10 seconds ago with 0 failures
[17:45:28] <icinga-miraheze>	 RECOVERY - cp191 Puppet on cp191 is OK: OK: Puppet is currently enabled, last run 11 seconds ago with 0 failures
[17:46:09] <icinga-miraheze>	 RECOVERY - cloud15 Puppet on cloud15 is OK: OK: Puppet is currently enabled, last run 26 seconds ago with 0 failures
[17:47:39] <icinga-miraheze>	 RECOVERY - os151 Puppet on os151 is OK: OK: Puppet is currently enabled, last run 41 seconds ago with 0 failures
[17:47:55] <icinga-miraheze>	 RECOVERY - mem151 Puppet on mem151 is OK: OK: Puppet is currently enabled, last run 18 seconds ago with 0 failures
[17:48:26] <icinga-miraheze>	 RECOVERY - ns1 Puppet on ns1 is OK: OK: Puppet is currently enabled, last run 51 seconds ago with 0 failures
[17:49:24] <MirahezeRelay>	 <paladox> !log restart squid on bast161 (seems memory has been building up to a potential oom)
[17:49:28] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[17:49:52] <MirahezeRelay>	 <paladox> high load ^
[17:50:00] <icinga-miraheze>	 PROBLEM - llm191 Puppet on llm191 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[openwebui_pip_install]
[17:51:11] <icinga-miraheze>	 PROBLEM - mwtask161 Puppet on mwtask161 is CRITICAL: CRITICAL: Puppet has 2 failures. Last run 2 minutes ago with 2 failures. Failed resources (up to 3 shown): Exec[git_pull_femiwiki-deploy-1.44],Exec[git_pull_3d2png]
[17:52:21] <icinga-miraheze>	 PROBLEM - mwtask151 Puppet on mwtask151 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_3d2png]
[17:52:22] <icinga-miraheze>	 PROBLEM - puppet181 Check unit status of git_pull_ssl on puppet181 is CRITICAL: CRITICAL: Status of the systemd unit git_pull_ssl
[17:52:51] <icinga-miraheze>	 PROBLEM - puppet181 Check unit status of git_pull_puppet on puppet181 is CRITICAL: CRITICAL: Status of the systemd unit git_pull_puppet
[17:53:07] <icinga-miraheze>	 PROBLEM - mwtask171 Puppet on mwtask171 is CRITICAL: CRITICAL: Puppet has 3 failures. Last run 4 minutes ago with 3 failures. Failed resources (up to 3 shown): Exec[git_pull_mathoid],Exec[git_pull_femiwiki-deploy-1.44],Exec[git_pull_3d2png]
[18:00:22] <icinga-miraheze>	 RECOVERY - puppet181 Check unit status of git_pull_ssl on puppet181 is OK: OK: Status of the systemd unit git_pull_ssl
[18:00:51] <icinga-miraheze>	 RECOVERY - puppet181 Check unit status of git_pull_puppet on puppet181 is OK: OK: Status of the systemd unit git_pull_puppet
[18:12:00] <icinga-miraheze>	 RECOVERY - llm191 Puppet on llm191 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[18:13:04] <icinga-miraheze>	 RECOVERY - mwtask171 Puppet on mwtask171 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[18:13:11] <icinga-miraheze>	 RECOVERY - mwtask161 Puppet on mwtask161 is OK: OK: Puppet is currently enabled, last run 40 seconds ago with 0 failures
[18:14:21] <icinga-miraheze>	 RECOVERY - mwtask151 Puppet on mwtask151 is OK: OK: Puppet is currently enabled, last run 57 seconds ago with 0 failures
[18:26:51] <icinga-miraheze>	 PROBLEM - mwtask181 Current Load on mwtask181 is WARNING: LOAD WARNING - total load average: 21.21, 14.45, 8.79
[18:28:51] <icinga-miraheze>	 RECOVERY - mwtask181 Current Load on mwtask181 is OK: LOAD OK - total load average: 18.45, 16.08, 10.10
[18:34:00] <icinga-miraheze>	 RECOVERY - test151 Check unit status of mediawiki_job_mediamoderation-scan on test151 is OK: OK: Status of the systemd unit mediawiki_job_mediamoderation-scan
[18:34:51] <icinga-miraheze>	 PROBLEM - mwtask181 Current Load on mwtask181 is WARNING: LOAD WARNING - total load average: 23.64, 19.92, 13.48
[18:38:51] <icinga-miraheze>	 RECOVERY - mwtask181 Current Load on mwtask181 is OK: LOAD OK - total load average: 9.64, 16.86, 13.93
[19:15:28] <icinga-miraheze>	 RECOVERY - cp191 Disk Space on cp191 is OK: DISK OK - free space: / 75598MiB (16% inode=99%);
[19:31:51] <icinga-miraheze>	 PROBLEM - cp201 Disk Space on cp201 is CRITICAL: DISK CRITICAL - free space: / 27126MiB (5% inode=99%);
[21:15:03] <icinga-miraheze>	 RECOVERY - cp171 Disk Space on cp171 is OK: DISK OK - free space: / 75561MiB (16% inode=99%);
[22:53:16] <MirahezeLSBot>	 !log [macfan@mwtask181] sudo -u www-data php /srv/mediawiki/1.44/maintenance/run.php /srv/mediawiki/1.44/maintenance/importDump.php --wiki=arcaneodysseywikiwiki dump.xml --no-updates (START)
[22:53:22] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[23:21:31] <MirahezeLSBot>	 !log [macfan@mwtask181] sudo -u www-data php /srv/mediawiki/1.44/maintenance/run.php /srv/mediawiki/1.44/maintenance/importDump.php --wiki=arcaneodysseywikiwiki dump.xml --no-updates (END - exit=0)
[23:21:32] <MirahezeLSBot>	 !log [macfan@mwtask181] sudo -u www-data php /srv/mediawiki/1.44/maintenance/run.php /srv/mediawiki/1.44/maintenance/rebuildall.php --wiki=arcaneodysseywikiwiki (START)
[23:21:36] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[23:21:40] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log