[00:00:52] [02statichelp] 07WikiTideBot pushed 1 new commit to 03main 13https://github.com/miraheze/statichelp/commit/6477ba8df51b1dc01af4067a9cbc2e28635c6758 [00:00:52] 02statichelp/03main 07WikiTideBot 036477ba8 Bot: Auto-update Tech namespace pages 2026-03-02 00:00:50 [00:49:38] PROBLEM - cp191 Disk Space on cp191 is WARNING: DISK WARNING - free space: / 44387MiB (9.8% inode=100%); [00:53:38] RECOVERY - cp191 Disk Space on cp191 is OK: DISK OK - free space: / 88666MiB (19.5% inode=100%); [01:03:24] PROBLEM - cp201 Disk Space on cp201 is WARNING: DISK WARNING - free space: / 44644MiB (9.8% inode=100%); [01:13:24] RECOVERY - cp201 Disk Space on cp201 is OK: DISK OK - free space: / 88054MiB (19.4% inode=100%); [03:37:16] PROBLEM - cp201 Varnish Backends on cp201 is CRITICAL: 1 backends are down. mw183 [03:37:25] PROBLEM - mw183 HTTPS on mw183 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10001 milliseconds with 0 bytes received [03:37:33] PROBLEM - mw183 PowerDNS Recursor on mw183 is CRITICAL: CRITICAL - Plugin timed out while executing system call [03:37:47] PROBLEM - mw183 MediaWiki Rendering on mw183 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 8191 bytes in 5.366 second response time [03:37:55] PROBLEM - cp191 Varnish Backends on cp191 is CRITICAL: 1 backends are down. mw183 [03:38:16] PROBLEM - cp171 Varnish Backends on cp171 is CRITICAL: 1 backends are down. mw183 [03:38:33] PROBLEM - mw183 Current Load on mw183 is CRITICAL: LOAD CRITICAL - total load average: 36.45, 26.20, 12.79 [03:39:16] RECOVERY - cp201 Varnish Backends on cp201 is OK: All 31 backends are healthy [03:39:21] RECOVERY - mw183 HTTPS on mw183 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4308 bytes in 0.064 second response time [03:39:27] RECOVERY - mw183 PowerDNS Recursor on mw183 is OK: DNS OK: 0.168 seconds response time. mw183.fsslc.wtnet returns 10.0.18.155 [03:39:44] RECOVERY - mw183 MediaWiki Rendering on mw183 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.201 second response time [03:39:55] RECOVERY - cp191 Varnish Backends on cp191 is OK: All 31 backends are healthy [03:40:16] RECOVERY - cp171 Varnish Backends on cp171 is OK: All 31 backends are healthy [03:40:33] RECOVERY - mw183 Current Load on mw183 is OK: LOAD OK - total load average: 6.63, 18.23, 11.49 [05:00:35] [02ssl] 07WikiTideBot pushed 1 new commit to 03main 13https://github.com/miraheze/ssl/commit/643ea86485a6e4714b250b421f15c9f266bd9334 [05:00:35] 02ssl/03main 07WikiTideBot 03643ea86 Bot: Auto-update domain lists [09:00:32] [02ssl] 07WikiTideBot pushed 1 new commit to 03main 13https://github.com/miraheze/ssl/commit/b58eca16f0636b437710ee1872e51fe081182370 [09:00:32] 02ssl/03main 07WikiTideBot 03b58eca1 Bot: Auto-update domain lists [11:00:39] [02ssl] 07WikiTideBot pushed 1 new commit to 03main 13https://github.com/miraheze/ssl/commit/a7b4f2ccce818e8adf514d62651b289beb13768d [11:00:39] 02ssl/03main 07WikiTideBot 03a7b4f2c Bot: Auto-update domain lists [11:09:10] PROBLEM - swiftproxy161 Current Load on swiftproxy161 is WARNING: LOAD WARNING - total load average: 6.92, 5.99, 4.08 [11:11:10] RECOVERY - swiftproxy161 Current Load on swiftproxy161 is OK: LOAD OK - total load average: 5.22, 5.77, 4.24 [12:09:33] [02MirahezeMagic] 07translatewiki pushed 1 new commit to 03main 13https://github.com/miraheze/MirahezeMagic/commit/63b998baf02a887391ac0148887bd9e3e8bc4b4e [12:09:34] 02MirahezeMagic/03main 07translatewiki.net 0363b998b Localisation updates from https://translatewiki.net. [12:17:33] miraheze/MirahezeMagic - translatewiki the build passed. [13:15:45] PROBLEM - os162 Puppet on os162 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 19 minutes ago with 0 failures [13:15:48] PROBLEM - swiftproxy171 Puppet on swiftproxy171 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 6 minutes ago with 0 failures [13:15:53] PROBLEM - puppet181 Puppet on puppet181 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 3 minutes ago with 0 failures [13:15:55] PROBLEM - rdb191 Puppet on rdb191 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 6 minutes ago with 0 failures [13:15:57] PROBLEM - db171 Puppet on db171 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [13:16:01] PROBLEM - swiftobject171 Puppet on swiftobject171 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 20 minutes ago with 0 failures [13:16:03] PROBLEM - os202 Puppet on os202 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 18 minutes ago with 0 failures [13:16:06] PROBLEM - test151 Puppet on test151 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 13 minutes ago with 0 failures [13:16:09] PROBLEM - prometheus151 Puppet on prometheus151 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 10 minutes ago with 0 failures [13:16:13] PROBLEM - swiftac171 Puppet on swiftac171 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 2 minutes ago with 0 failures [13:16:17] PROBLEM - os191 Puppet on os191 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 15 minutes ago with 0 failures [13:16:21] PROBLEM - swiftobject201 Puppet on swiftobject201 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 4 minutes ago with 0 failures [13:16:32] PROBLEM - swiftobject181 Puppet on swiftobject181 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 3 minutes ago with 0 failures [13:16:35] PROBLEM - ns1 Puppet on ns1 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 11 seconds ago with 0 failures [13:16:41] PROBLEM - swiftobject191 Puppet on swiftobject191 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 4 minutes ago with 0 failures [13:16:58] PROBLEM - db172 Puppet on db172 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [13:16:58] PROBLEM - db151 Puppet on db151 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [13:17:03] PROBLEM - cloud21 Puppet on cloud21 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 2 minutes ago with 0 failures [13:17:05] PROBLEM - db161 Puppet on db161 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [13:17:09] PROBLEM - reports171 Puppet on reports171 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 12 seconds ago with 0 failures [13:17:09] PROBLEM - swiftobject161 Puppet on swiftobject161 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 9 seconds ago with 0 failures [13:17:09] PROBLEM - db192 Puppet on db192 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [13:17:11] PROBLEM - cloud18 Puppet on cloud18 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 2 minutes ago with 0 failures [13:17:18] PROBLEM - phorge171 Puppet on phorge171 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 20 seconds ago with 0 failures [13:17:18] PROBLEM - cloud20 Puppet on cloud20 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 2 minutes ago with 0 failures [13:17:18] PROBLEM - bast161 Puppet on bast161 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 2 minutes ago with 0 failures [13:17:22] PROBLEM - db181 Puppet on db181 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [13:18:06] RECOVERY - test151 Puppet on test151 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [13:18:07] PROBLEM - db182 Puppet on db182 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [13:18:08] PROBLEM - cp201 Puppet on cp201 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 3 minutes ago with 0 failures [13:18:16] PROBLEM - cloud15 Puppet on cloud15 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 3 minutes ago with 0 failures [13:18:25] PROBLEM - bots171 Puppet on bots171 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 3 minutes ago with 0 failures [13:18:31] PROBLEM - graylog161 Puppet on graylog161 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [13:18:34] PROBLEM - cp161 Puppet on cp161 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 3 minutes ago with 0 failures [13:18:37] PROBLEM - cloud16 Puppet on cloud16 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 3 minutes ago with 0 failures [13:18:37] PROBLEM - cloud19 Puppet on cloud19 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 3 minutes ago with 0 failures [13:18:38] PROBLEM - bast181 Puppet on bast181 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 3 minutes ago with 0 failures [13:18:39] PROBLEM - eventgate182 Puppet on eventgate182 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 3 minutes ago with 0 failures [13:18:39] PROBLEM - changeprop202 Puppet on changeprop202 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 3 minutes ago with 0 failures [13:18:41] PROBLEM - cloud17 Puppet on cloud17 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 3 minutes ago with 0 failures [13:18:42] PROBLEM - db201 Puppet on db201 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [13:18:42] PROBLEM - mem151 Puppet on mem151 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 3 minutes ago with 0 failures [13:18:44] PROBLEM - cp171 Puppet on cp171 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 3 minutes ago with 0 failures [13:18:45] PROBLEM - ldap171 Puppet on ldap171 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 3 minutes ago with 0 failures [13:18:51] PROBLEM - llm191 Puppet on llm191 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 3 minutes ago with 0 failures [13:18:51] PROBLEM - kafka182 Puppet on kafka182 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 3 minutes ago with 0 failures [13:18:53] PROBLEM - mem201 Puppet on mem201 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 3 minutes ago with 0 failures [13:18:57] PROBLEM - cp191 Puppet on cp191 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 4 minutes ago with 0 failures [13:18:59] PROBLEM - matomo151 Puppet on matomo151 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 3 minutes ago with 0 failures [13:19:02] PROBLEM - mem191 Puppet on mem191 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 3 minutes ago with 0 failures [13:19:03] PROBLEM - mw193 Puppet on mw193 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 2 minutes ago with 0 failures [13:19:05] PROBLEM - mw151 Puppet on mw151 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 2 minutes ago with 0 failures [13:19:05] PROBLEM - mw171 Puppet on mw171 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 2 minutes ago with 0 failures [13:19:08] PROBLEM - mw192 Puppet on mw192 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 2 minutes ago with 0 failures [13:19:10] PROBLEM - mw163 Puppet on mw163 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 3 minutes ago with 0 failures [13:19:11] PROBLEM - mw173 Puppet on mw173 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 3 minutes ago with 0 failures [13:19:16] PROBLEM - mw172 Puppet on mw172 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 3 minutes ago with 0 failures [13:19:21] PROBLEM - mw162 Puppet on mw162 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 3 minutes ago with 0 failures [13:19:23] PROBLEM - mattermost2 Puppet on mattermost2 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 3 minutes ago with 0 failures [13:19:23] PROBLEM - mw153 Puppet on mw153 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 3 minutes ago with 0 failures [13:19:28] PROBLEM - mw191 Puppet on mw191 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 3 minutes ago with 0 failures [13:19:32] PROBLEM - mw181 Puppet on mw181 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 3 minutes ago with 0 failures [13:19:37] PROBLEM - mw182 Puppet on mw182 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 3 minutes ago with 0 failures [13:19:38] PROBLEM - mw161 Puppet on mw161 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 3 minutes ago with 0 failures [13:19:39] PROBLEM - mw152 Puppet on mw152 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 3 minutes ago with 0 failures [13:19:42] PROBLEM - mem161 Puppet on mem161 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 4 minutes ago with 0 failures [13:19:45] PROBLEM - mw201 Puppet on mw201 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 2 minutes ago with 0 failures [13:19:51] PROBLEM - mw183 Puppet on mw183 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 3 minutes ago with 0 failures [13:19:54] PROBLEM - mwtask161 Puppet on mwtask161 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 3 minutes ago with 0 failures [13:20:00] PROBLEM - mwtask171 Puppet on mwtask171 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 3 minutes ago with 0 failures [13:20:15] PROBLEM - os161 Puppet on os161 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 3 minutes ago with 0 failures [13:20:17] PROBLEM - mwtask151 Puppet on mwtask151 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 3 minutes ago with 0 failures [13:20:38] PROBLEM - os201 Puppet on os201 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 3 minutes ago with 0 failures [13:20:41] PROBLEM - mw202 Puppet on mw202 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 3 minutes ago with 0 failures [13:20:42] PROBLEM - ns2 Puppet on ns2 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 3 minutes ago with 0 failures [13:20:43] PROBLEM - mwtask181 Puppet on mwtask181 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 3 minutes ago with 0 failures [13:20:44] PROBLEM - mw203 Puppet on mw203 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 3 minutes ago with 0 failures [13:20:46] PROBLEM - os151 Puppet on os151 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 3 minutes ago with 0 failures [13:20:54] PROBLEM - swiftobject151 Puppet on swiftobject151 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 3 minutes ago with 0 failures [13:21:14] PROBLEM - mon181 Puppet on mon181 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 3 minutes ago with 0 failures [13:21:26] PROBLEM - swiftproxy161 Puppet on swiftproxy161 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 4 minutes ago with 0 failures [13:22:07] PROBLEM - test151 Puppet on test151 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 3 minutes ago with 0 failures [13:35:07] [02puppet] 07paladox pushed 1 new commit to 03paladox-patch-1 13https://github.com/miraheze/puppet/commit/0d76c1a6d0e50d3c1e10dec1f579aaee802ca60f [13:35:07] 02puppet/03paladox-patch-1 07paladox 030d76c1a Update db.pp [13:36:09] [02puppet] 07paladox pushed 1 new commit to 03paladox-patch-1 13https://github.com/miraheze/puppet/commit/9c54d7a584c83000dff4678805ba4c01e12b0a63 [13:36:09] 02puppet/03paladox-patch-1 07paladox 039c54d7a Fix PQL subquery for role resources in graylog.pp [13:36:25] RECOVERY - bots171 Puppet on bots171 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [13:36:35] RECOVERY - cp161 Puppet on cp161 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [13:36:39] RECOVERY - changeprop202 Puppet on changeprop202 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [13:36:43] RECOVERY - cloud17 Puppet on cloud17 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [13:36:44] RECOVERY - cp171 Puppet on cp171 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [13:37:05] RECOVERY - cloud21 Puppet on cloud21 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [13:37:07] RECOVERY - cp191 Puppet on cp191 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [13:37:10] RECOVERY - cloud18 Puppet on cloud18 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [13:37:27] RECOVERY - mw191 Puppet on mw191 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [13:37:28] PROBLEM - db181 Puppet on db181 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 16 seconds ago with 0 failures [13:37:28] RECOVERY - mw181 Puppet on mw181 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [13:37:38] RECOVERY - mw161 Puppet on mw161 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [13:37:40] RECOVERY - mem161 Puppet on mem161 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [13:37:42] RECOVERY - mw152 Puppet on mw152 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [13:37:47] RECOVERY - mw182 Puppet on mw182 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [13:37:51] RECOVERY - mw183 Puppet on mw183 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [13:37:58] PROBLEM - swiftproxy161 Current Load on swiftproxy161 is CRITICAL: LOAD CRITICAL - total load average: 8.54, 7.06, 5.23 [13:37:58] RECOVERY - mem151 Puppet on mem151 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [13:38:08] PROBLEM - db182 Puppet on db182 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 58 seconds ago with 0 failures [13:38:10] PROBLEM - graylog161 Puppet on graylog161 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 59 seconds ago with 0 failures [13:38:12] PROBLEM - db171 Puppet on db171 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 1 minute ago with 0 failures [13:38:15] RECOVERY - kafka182 Puppet on kafka182 is OK: OK: Puppet is currently enabled, last run 0 seconds ago with 0 failures [13:38:35] RECOVERY - ns1 Puppet on ns1 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [13:38:39] RECOVERY - os201 Puppet on os201 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [13:38:41] RECOVERY - mw202 Puppet on mw202 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [13:38:42] RECOVERY - ns2 Puppet on ns2 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [13:38:44] RECOVERY - mw203 Puppet on mw203 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [13:38:46] RECOVERY - mon181 Puppet on mon181 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [13:38:52] RECOVERY - mwtask181 Puppet on mwtask181 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [13:38:54] RECOVERY - swiftobject151 Puppet on swiftobject151 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [13:38:54] PROBLEM - db201 Puppet on db201 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 1 minute ago with 0 failures [13:38:57] PROBLEM - db172 Puppet on db172 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 1 minute ago with 0 failures [13:38:58] PROBLEM - db151 Puppet on db151 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 1 minute ago with 0 failures [13:38:59] RECOVERY - os151 Puppet on os151 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [13:39:05] PROBLEM - db161 Puppet on db161 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 1 minute ago with 0 failures [13:39:09] RECOVERY - mw201 Puppet on mw201 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [13:39:09] RECOVERY - reports171 Puppet on reports171 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [13:39:09] RECOVERY - swiftobject161 Puppet on swiftobject161 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [13:39:09] PROBLEM - db192 Puppet on db192 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 1 minute ago with 0 failures [13:39:18] RECOVERY - phorge171 Puppet on phorge171 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [13:39:54] RECOVERY - swiftproxy171 Puppet on swiftproxy171 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [13:39:55] RECOVERY - swiftproxy161 Current Load on swiftproxy161 is OK: LOAD OK - total load average: 5.05, 6.37, 5.21 [13:40:09] RECOVERY - test151 Puppet on test151 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [13:40:26] PROBLEM - bots171 Puppet on bots171 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 3 minutes ago with 0 failures [13:40:34] PROBLEM - cp161 Puppet on cp161 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 3 minutes ago with 0 failures [13:40:39] PROBLEM - changeprop202 Puppet on changeprop202 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 3 minutes ago with 0 failures [13:40:44] PROBLEM - cloud17 Puppet on cloud17 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 3 minutes ago with 0 failures [13:40:44] PROBLEM - cp171 Puppet on cp171 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 3 minutes ago with 0 failures [13:41:06] PROBLEM - cloud21 Puppet on cloud21 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 3 minutes ago with 0 failures [13:41:11] PROBLEM - cloud18 Puppet on cloud18 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 3 minutes ago with 0 failures [13:41:11] PROBLEM - cp191 Puppet on cp191 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 3 minutes ago with 0 failures [13:41:28] PROBLEM - mw191 Puppet on mw191 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 2 minutes ago with 0 failures [13:41:31] PROBLEM - mw181 Puppet on mw181 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 3 minutes ago with 0 failures [13:41:38] PROBLEM - mw161 Puppet on mw161 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 3 minutes ago with 0 failures [13:41:42] PROBLEM - mem161 Puppet on mem161 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 3 minutes ago with 0 failures [13:41:43] PROBLEM - mw152 Puppet on mw152 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 3 minutes ago with 0 failures [13:41:50] PROBLEM - mem151 Puppet on mem151 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 3 minutes ago with 0 failures [13:41:50] PROBLEM - mw182 Puppet on mw182 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 3 minutes ago with 0 failures [13:41:51] PROBLEM - mw183 Puppet on mw183 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 3 minutes ago with 0 failures [13:42:09] PROBLEM - kafka182 Puppet on kafka182 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 3 minutes ago with 0 failures [13:42:35] PROBLEM - ns1 Puppet on ns1 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 3 minutes ago with 0 failures [13:42:38] PROBLEM - os201 Puppet on os201 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 3 minutes ago with 0 failures [13:42:41] PROBLEM - mw202 Puppet on mw202 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 3 minutes ago with 0 failures [13:42:42] PROBLEM - ns2 Puppet on ns2 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 3 minutes ago with 0 failures [13:42:43] PROBLEM - mon181 Puppet on mon181 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 2 minutes ago with 0 failures [13:42:44] PROBLEM - mw203 Puppet on mw203 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 3 minutes ago with 0 failures [13:42:54] PROBLEM - swiftobject151 Puppet on swiftobject151 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 3 minutes ago with 0 failures [13:42:55] PROBLEM - mwtask181 Puppet on mwtask181 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 2 minutes ago with 0 failures [13:43:02] PROBLEM - os151 Puppet on os151 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 3 minutes ago with 0 failures [13:43:03] [02puppet] 07paladox merged pull request #4804: Migrate from puppetdbquery to puppetdb::query_facts (03main...03paladox-patch-1) 13https://github.com/miraheze/puppet/pull/4804 [13:43:04] [02puppet] 07paladox pushed 1 new commit to 03main 13https://github.com/miraheze/puppet/commit/92e60846e211062c5327092e6153b24c1a647037 [13:43:05] 02puppet/03main 07paladox 0392e6084 Migrate from puppetdbquery to puppetdb::query_facts (#4804)… [13:43:05] PROBLEM - mw201 Puppet on mw201 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 3 minutes ago with 0 failures [13:43:07] [02puppet] 07paladox 04deleted 03paladox-patch-1 at 039c54d7a 13https://api.github.com/repos/miraheze/puppet/commit/9c54d7a [13:43:09] PROBLEM - reports171 Puppet on reports171 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 3 minutes ago with 0 failures [13:43:09] PROBLEM - swiftobject161 Puppet on swiftobject161 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 3 minutes ago with 0 failures [13:43:18] PROBLEM - phorge171 Puppet on phorge171 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 3 minutes ago with 0 failures [13:43:29] RECOVERY - db181 Puppet on db181 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [13:43:44] RECOVERY - cloud16 Puppet on cloud16 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [13:43:44] RECOVERY - cloud19 Puppet on cloud19 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [13:43:45] RECOVERY - bast181 Puppet on bast181 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [13:43:49] RECOVERY - eventgate182 Puppet on eventgate182 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [13:43:55] PROBLEM - swiftproxy171 Puppet on swiftproxy171 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 3 minutes ago with 0 failures [13:44:06] RECOVERY - graylog161 Puppet on graylog161 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [13:44:08] RECOVERY - db182 Puppet on db182 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [13:44:09] RECOVERY - cp201 Puppet on cp201 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [13:44:10] PROBLEM - test151 Puppet on test151 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 3 minutes ago with 0 failures [13:44:15] RECOVERY - db171 Puppet on db171 is OK: OK: Puppet is currently enabled, last run 3 seconds ago with 0 failures [13:44:17] RECOVERY - cloud15 Puppet on cloud15 is OK: OK: Puppet is currently enabled, last run 7 seconds ago with 0 failures [13:44:25] RECOVERY - bots171 Puppet on bots171 is OK: OK: Puppet is currently enabled, last run 10 seconds ago with 0 failures [13:44:32] RECOVERY - mw172 Puppet on mw172 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [13:44:33] RECOVERY - mw162 Puppet on mw162 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [13:44:34] RECOVERY - cp161 Puppet on cp161 is OK: OK: Puppet is currently enabled, last run 20 seconds ago with 0 failures [13:44:39] RECOVERY - mattermost2 Puppet on mattermost2 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [13:44:39] RECOVERY - changeprop202 Puppet on changeprop202 is OK: OK: Puppet is currently enabled, last run 25 seconds ago with 0 failures [13:44:39] RECOVERY - mon181 Puppet on mon181 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [13:44:44] RECOVERY - cp171 Puppet on cp171 is OK: OK: Puppet is currently enabled, last run 30 seconds ago with 0 failures [13:44:44] RECOVERY - cloud17 Puppet on cloud17 is OK: OK: Puppet is currently enabled, last run 33 seconds ago with 0 failures [13:44:51] RECOVERY - ldap171 Puppet on ldap171 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [13:44:52] RECOVERY - llm191 Puppet on llm191 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [13:44:55] RECOVERY - mem201 Puppet on mem201 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [13:44:57] RECOVERY - db172 Puppet on db172 is OK: OK: Puppet is currently enabled, last run 46 seconds ago with 0 failures [13:44:57] RECOVERY - mw173 Puppet on mw173 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [13:44:58] RECOVERY - db151 Puppet on db151 is OK: OK: Puppet is currently enabled, last run 45 seconds ago with 0 failures [13:44:58] RECOVERY - matomo151 Puppet on matomo151 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [13:44:58] RECOVERY - db201 Puppet on db201 is OK: OK: Puppet is currently enabled, last run 44 seconds ago with 0 failures [13:44:59] RECOVERY - mem191 Puppet on mem191 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [13:45:02] RECOVERY - mw193 Puppet on mw193 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [13:45:05] RECOVERY - mw171 Puppet on mw171 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [13:45:05] RECOVERY - db161 Puppet on db161 is OK: OK: Puppet is currently enabled, last run 53 seconds ago with 0 failures [13:45:06] RECOVERY - cloud21 Puppet on cloud21 is OK: OK: Puppet is currently enabled, last run 57 seconds ago with 0 failures [13:45:08] RECOVERY - mw192 Puppet on mw192 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [13:45:09] RECOVERY - db192 Puppet on db192 is OK: OK: Puppet is currently enabled, last run 53 seconds ago with 0 failures [13:45:09] RECOVERY - mw151 Puppet on mw151 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [13:45:09] RECOVERY - mw163 Puppet on mw163 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [13:45:10] RECOVERY - cloud18 Puppet on cloud18 is OK: OK: Puppet is currently enabled, last run 58 seconds ago with 0 failures [13:45:12] RECOVERY - mw153 Puppet on mw153 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [13:45:13] RECOVERY - cp191 Puppet on cp191 is OK: OK: Puppet is currently enabled, last run 58 seconds ago with 0 failures [13:45:16] RECOVERY - bast161 Puppet on bast161 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [13:45:18] RECOVERY - cloud20 Puppet on cloud20 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [13:45:28] RECOVERY - mw191 Puppet on mw191 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [13:45:29] RECOVERY - mw181 Puppet on mw181 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [13:45:38] RECOVERY - mw161 Puppet on mw161 is OK: OK: Puppet is currently enabled, last run 10 seconds ago with 0 failures [13:45:40] RECOVERY - mem161 Puppet on mem161 is OK: OK: Puppet is currently enabled, last run 29 seconds ago with 0 failures [13:45:41] RECOVERY - os161 Puppet on os161 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [13:45:44] RECOVERY - mw152 Puppet on mw152 is OK: OK: Puppet is currently enabled, last run 16 seconds ago with 0 failures [13:45:45] RECOVERY - os162 Puppet on os162 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [13:45:45] RECOVERY - mwtask161 Puppet on mwtask161 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [13:45:50] RECOVERY - mem151 Puppet on mem151 is OK: OK: Puppet is currently enabled, last run 38 seconds ago with 0 failures [13:45:51] RECOVERY - mw183 Puppet on mw183 is OK: OK: Puppet is currently enabled, last run 23 seconds ago with 0 failures [13:45:53] RECOVERY - mw182 Puppet on mw182 is OK: OK: Puppet is currently enabled, last run 24 seconds ago with 0 failures [13:45:54] RECOVERY - puppet181 Puppet on puppet181 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [13:45:54] RECOVERY - rdb191 Puppet on rdb191 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [13:45:59] RECOVERY - mwtask151 Puppet on mwtask151 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [13:46:00] RECOVERY - mwtask171 Puppet on mwtask171 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [13:46:01] RECOVERY - swiftobject171 Puppet on swiftobject171 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [13:46:02] RECOVERY - kafka182 Puppet on kafka182 is OK: OK: Puppet is currently enabled, last run 50 seconds ago with 0 failures [13:46:03] RECOVERY - os202 Puppet on os202 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [13:46:09] RECOVERY - prometheus151 Puppet on prometheus151 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [13:46:17] RECOVERY - swiftac171 Puppet on swiftac171 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [13:46:24] RECOVERY - os191 Puppet on os191 is OK: OK: Puppet is currently enabled, last run 0 seconds ago with 0 failures [13:46:35] RECOVERY - ns1 Puppet on ns1 is OK: OK: Puppet is currently enabled, last run 26 seconds ago with 0 failures [13:46:38] RECOVERY - os201 Puppet on os201 is OK: OK: Puppet is currently enabled, last run 15 seconds ago with 0 failures [13:46:41] RECOVERY - swiftobject181 Puppet on swiftobject181 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [13:46:41] RECOVERY - swiftobject191 Puppet on swiftobject191 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [13:46:41] RECOVERY - mw202 Puppet on mw202 is OK: OK: Puppet is currently enabled, last run 13 seconds ago with 0 failures [13:46:42] RECOVERY - swiftobject201 Puppet on swiftobject201 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [13:46:42] RECOVERY - ns2 Puppet on ns2 is OK: OK: Puppet is currently enabled, last run 16 seconds ago with 0 failures [13:46:44] RECOVERY - mw203 Puppet on mw203 is OK: OK: Puppet is currently enabled, last run 15 seconds ago with 0 failures [13:46:54] RECOVERY - swiftobject151 Puppet on swiftobject151 is OK: OK: Puppet is currently enabled, last run 25 seconds ago with 0 failures [13:46:57] RECOVERY - mwtask181 Puppet on mwtask181 is OK: OK: Puppet is currently enabled, last run 6 seconds ago with 0 failures [13:46:58] RECOVERY - mw201 Puppet on mw201 is OK: OK: Puppet is currently enabled, last run 30 seconds ago with 0 failures [13:47:04] RECOVERY - os151 Puppet on os151 is OK: OK: Puppet is currently enabled, last run 41 seconds ago with 0 failures [13:47:09] RECOVERY - reports171 Puppet on reports171 is OK: OK: Puppet is currently enabled, last run 42 seconds ago with 0 failures [13:47:09] RECOVERY - swiftobject161 Puppet on swiftobject161 is OK: OK: Puppet is currently enabled, last run 39 seconds ago with 0 failures [13:47:18] RECOVERY - phorge171 Puppet on phorge171 is OK: OK: Puppet is currently enabled, last run 56 seconds ago with 0 failures [13:47:32] RECOVERY - swiftproxy161 Puppet on swiftproxy161 is OK: OK: Puppet is currently enabled, last run 35 seconds ago with 0 failures [13:47:48] [02puppet] 07paladox created 03remove-puppetdbquery (+1 new commit) 13https://github.com/miraheze/puppet/commit/a095a917411d [13:47:48] 02puppet/03remove-puppetdbquery 07paladox 03a095a91 Get rid of puppetdbquery… [13:47:55] RECOVERY - swiftproxy171 Puppet on swiftproxy171 is OK: OK: Puppet is currently enabled, last run 57 seconds ago with 0 failures [13:47:57] [02puppet] 07paladox opened pull request #4805: Get rid of puppetdbquery (03main...03remove-puppetdbquery) 13https://github.com/miraheze/puppet/pull/4805 [13:48:10] RECOVERY - test151 Puppet on test151 is OK: OK: Puppet is currently enabled, last run 33 seconds ago with 0 failures [13:48:21] [02puppet] 07paladox merged pull request #4805: Get rid of puppetdbquery (03main...03remove-puppetdbquery) 13https://github.com/miraheze/puppet/pull/4805 [13:48:23] [02puppet] 07paladox pushed 1 new commit to 03main 13https://github.com/miraheze/puppet/commit/5c94fedeb4ca0d0a8f65aca86427a366bb6d5601 [13:48:25] [02puppet] 07paladox 04deleted 03remove-puppetdbquery at 03a095a91 13https://api.github.com/repos/miraheze/puppet/commit/a095a91 [13:48:26] 02puppet/03main 07paladox 035c94fed Get rid of puppetdbquery (#4805)… [13:53:53] PROBLEM - puppet181 Puppet on puppet181 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [13:54:09] PROBLEM - prometheus151 Puppet on prometheus151 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [14:03:05] PROBLEM - mwtask181 Puppet on mwtask181 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [14:03:44] PROBLEM - eventgate182 Puppet on eventgate182 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [14:06:21] PROBLEM - mon181 Puppet on mon181 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [15:27:36] [02puppet] 07paladox created 03paladox-patch-1 (+1 new commit) 13https://github.com/miraheze/puppet/commit/c666614e24e9 [15:27:36] 02puppet/03paladox-patch-1 07paladox 03c666614 varnish: Allow setting uds socket [15:27:43] [02puppet] 07paladox opened pull request #4806: varnish: Allow setting uds socket (03main...03paladox-patch-1) 13https://github.com/miraheze/puppet/pull/4806 [15:35:42] [02puppet] 07paladox created 03paladox-patch-2 (+1 new commit) 13https://github.com/miraheze/puppet/commit/4ee84c1cb9cf [15:35:44] 02puppet/03paladox-patch-2 07paladox 034ee84c1 monitoring: Fix to use puppetdb_query [15:35:46] [02puppet] 07paladox opened pull request #4807: monitoring: Fix to use puppetdb_query (03main...03paladox-patch-2) 13https://github.com/miraheze/puppet/pull/4807 [15:38:44] [02puppet] 07paladox pushed 1 new commit to 03paladox-patch-2 13https://github.com/miraheze/puppet/commit/c0bb7847ad5aa09e2570afc40443100c8740f2bd [15:38:46] 02puppet/03paladox-patch-2 07paladox 03c0bb784 Fix puppetdb_query function call in init.pp [15:42:21] RECOVERY - mon181 Puppet on mon181 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [15:45:22] [02puppet] 07paladox pushed 1 new commit to 03paladox-patch-2 13https://github.com/miraheze/puppet/commit/74ade33a9b2b05c1a76af5bd4b57ca73cc5be139 [15:45:23] 02puppet/03paladox-patch-2 07paladox 0374ade33 Update ssl.conf.erb [15:45:53] [02puppet] 07paladox merged pull request #4807: monitoring: Fix to use puppetdb_query (03main...03paladox-patch-2) 13https://github.com/miraheze/puppet/pull/4807 [15:45:54] [02puppet] 07paladox pushed 1 new commit to 03main 13https://github.com/miraheze/puppet/commit/c5d546a09f2e99f882b9569c2d125b7577522cea [15:45:55] 02puppet/03main 07paladox 03c5d546a monitoring: Fix to use puppetdb_query (#4807) [15:45:57] [02puppet] 07paladox 04deleted 03paladox-patch-2 at 0374ade33 13https://api.github.com/repos/miraheze/puppet/commit/74ade33 [15:49:06] [02puppet] 07paladox created 03paladox-patch-2 (+1 new commit) 13https://github.com/miraheze/puppet/commit/4f44173f4fc7 [15:49:06] 02puppet/03paladox-patch-2 07paladox 034f44173 salt: Fix to use puppetdb_query [15:49:10] [02puppet] 07paladox opened pull request #4808: salt: Fix to use puppetdb_query (03main...03paladox-patch-2) 13https://github.com/miraheze/puppet/pull/4808 [16:05:12] [02puppet] 07paladox pushed 1 new commit to 03paladox-patch-2 13https://github.com/miraheze/puppet/commit/4f0dce68fd54c7a2f1a165c035d07253d19b1fa8 [16:05:13] 02puppet/03paladox-patch-2 07paladox 034f0dce6 Refactor salt-user-known-hosts.erb for PuppetDB query [16:06:20] RECOVERY - puppet181 Puppet on puppet181 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [16:07:25] !log [skye@mwtask171] Starting import for pokemoninfinitywiki (XML: None; Images: .) (START) [16:07:26] !log [skye@mwtask171] sudo -u www-data php /srv/mediawiki/1.45/maintenance/run.php importImages --wiki=pokemoninfinitywiki --sleep=1 '--comment=Importing images from https://p-infinity.fandom.com/ ([[phorge:T15038|T15038]])' -- . (START) [16:07:27] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:07:29] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:07:36] [02puppet] 07paladox merged pull request #4808: salt: Fix to use puppetdb_query (03main...03paladox-patch-2) 13https://github.com/miraheze/puppet/pull/4808 [16:07:36] [02puppet] 07paladox pushed 1 new commit to 03main 13https://github.com/miraheze/puppet/commit/4186054de27bd2b666fa150539eeb58744dbeda0 [16:07:36] 02puppet/03main 07paladox 034186054 salt: Fix to use puppetdb_query (#4808) [16:07:36] [02puppet] 07paladox 04deleted 03paladox-patch-2 at 034f0dce6 13https://api.github.com/repos/miraheze/puppet/commit/4f0dce6 [16:12:16] [02puppet] 07paladox created 03paladox-patch-2 (+1 new commit) 13https://github.com/miraheze/puppet/commit/d4c6ec0dd560 [16:12:17] 02puppet/03paladox-patch-2 07paladox 03d4c6ec0 prometheus: Fix to use puppetdb_query [16:12:22] [02puppet] 07paladox opened pull request #4809: prometheus: Fix to use puppetdb_query (03main...03paladox-patch-2) 13https://github.com/miraheze/puppet/pull/4809 [16:12:36] [02puppet] 07paladox pushed 1 new commit to 03paladox-patch-2 13https://github.com/miraheze/puppet/commit/e5d04428e7db716af949c452809331055b65ba7e [16:12:37] 02puppet/03paladox-patch-2 07paladox 03e5d0442 Update class.pp [16:13:47] [02puppet] 07paladox pushed 1 new commit to 03paladox-patch-2 13https://github.com/miraheze/puppet/commit/2120a0d026db0cfdcda03b6d8eadd75d493e133b [16:13:47] 02puppet/03paladox-patch-2 07paladox 032120a0d Refactor server query to use PuppetDB PQL [16:16:20] PROBLEM - prometheus151 Puppet on prometheus151 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 1 minute ago with 0 failures [16:26:17] RECOVERY - prometheus151 Puppet on prometheus151 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [16:30:34] [02ssl] 07WikiTideBot pushed 1 new commit to 03main 13https://github.com/miraheze/ssl/commit/a51c58fee0630022cc1b1bb5898fd7cbbfe34445 [16:30:34] 02ssl/03main 07WikiTideBot 03a51c58f Bot: Auto-update domain lists [16:37:18] PROBLEM - prometheus151 Puppet on prometheus151 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 3 minutes ago with 0 failures [16:45:15] PROBLEM - prometheus151 Puppet on prometheus151 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [16:47:14] PROBLEM - prometheus151 Puppet on prometheus151 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 27 seconds ago with 0 failures [16:52:09] [02puppet] 07paladox created 03paladox-patch-4 (+1 new commit) 13https://github.com/miraheze/puppet/commit/96ac3e28e831 [16:52:10] 02puppet/03paladox-patch-4 07paladox 0396ac3e2 eventgate: Fix firewall [16:52:13] [02puppet] 07paladox opened pull request #4810: eventgate: Fix firewall (03main...03paladox-patch-4) 13https://github.com/miraheze/puppet/pull/4810 [16:52:22] [02puppet] 07paladox pushed 1 new commit to 03main 13https://github.com/miraheze/puppet/commit/aa075b78afb60a78e33ca392cb15899115bc6e3f [16:52:23] 02puppet/03main 07paladox 03aa075b7 eventgate: Fix firewall (#4810) [16:52:25] [02puppet] 07paladox 04deleted 03paladox-patch-4 at 0396ac3e2 13https://api.github.com/repos/miraheze/puppet/commit/96ac3e2 [16:52:28] [02puppet] 07paladox pushed 1 new commit to 03paladox-patch-2 13https://github.com/miraheze/puppet/commit/d54659845e3ca1626ab96692cef88033185c29cf [16:52:28] 02puppet/03paladox-patch-2 07paladox 03d546598 Merge branch 'main' into paladox-patch-2 [16:53:31] [02puppet] 07paladox pushed 1 new commit to 03main 13https://github.com/miraheze/puppet/commit/55d04b8dccd2a6efc3b6094f7066c5381f8a8b26 [16:53:31] 02puppet/03main 07paladox 0355d04b8 Fix [16:53:36] [02puppet] 07paladox pushed 1 new commit to 03paladox-patch-2 13https://github.com/miraheze/puppet/commit/654b6d3f8bc3158af87c72b36b01ea6073fca906 [16:53:36] 02puppet/03paladox-patch-2 07paladox 03654b6d3 Merge branch 'main' into paladox-patch-2 [16:55:16] RECOVERY - eventgate182 Puppet on eventgate182 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [16:55:20] [02puppet] 07paladox created 03paladox-patch-4 (+1 new commit) 13https://github.com/miraheze/puppet/commit/f9b193b6332f [16:55:20] 02puppet/03paladox-patch-4 07paladox 03f9b193b mediawiki: Fix [16:55:23] [02puppet] 07paladox opened pull request #4811: mediawiki: Fix (03main...03paladox-patch-4) 13https://github.com/miraheze/puppet/pull/4811 [16:55:29] [02puppet] 07paladox merged pull request #4811: mediawiki: Fix (03main...03paladox-patch-4) 13https://github.com/miraheze/puppet/pull/4811 [16:55:31] [02puppet] 07paladox pushed 1 new commit to 03main 13https://github.com/miraheze/puppet/commit/b0711828cec7ec2a96f6d36c64f2377a5108646e [16:55:33] [02puppet] 07paladox 04deleted 03paladox-patch-4 at 03f9b193b 13https://api.github.com/repos/miraheze/puppet/commit/f9b193b [16:55:34] 02puppet/03main 07paladox 03b071182 mediawiki: Fix (#4811) [16:55:36] [02puppet] 07paladox pushed 1 new commit to 03paladox-patch-2 13https://github.com/miraheze/puppet/commit/bce764720f5b5dc45b4a46015ca812c6f010104c [16:55:37] 02puppet/03paladox-patch-2 07paladox 03bce7647 Merge branch 'main' into paladox-patch-2 [16:56:40] [02puppet] 07paladox merged pull request #4810: eventgate: Fix firewall (03main...03paladox-patch-4) 13https://github.com/miraheze/puppet/pull/4810 [16:57:16] RECOVERY - mwtask181 Puppet on mwtask181 is OK: OK: Puppet is currently enabled, last run 23 seconds ago with 0 failures [17:04:21] [02puppet] 07paladox pushed 1 new commit to 03paladox-patch-2 13https://github.com/miraheze/puppet/commit/387213871d392d20e5327fbeb19634e1cb302b83 [17:04:21] 02puppet/03paladox-patch-2 07paladox 033872138 Fix PQL query to include module resources [17:07:07] PROBLEM - prometheus151 Puppet on prometheus151 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [17:13:05] PROBLEM - prometheus151 Puppet on prometheus151 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 1 minute ago with 0 failures [17:30:28] [02ssl] 07WikiTideBot pushed 1 new commit to 03main 13https://github.com/miraheze/ssl/commit/f5e2ea3d5305bec6ead6a74ced73dabace76f266 [17:30:29] 02ssl/03main 07WikiTideBot 03f5e2ea3 Bot: Auto-update domain lists [17:41:58] [02ssl] 07pskyechology pushed 1 new commit to 03main 13https://github.com/miraheze/ssl/commit/470b7324087241f6bed67b535ec9f26cfe8dd515 [17:41:58] 02ssl/03main 07Skye 03470b732 T14997: Redirect www.jrpstudios.wiki to jrpstudios.wiki [18:00:33] [02ssl] 07WikiTideBot pushed 1 new commit to 03main 13https://github.com/miraheze/ssl/commit/04e91316400c7a36fdb9db48aa11d05d0feb07dc [18:00:33] 02ssl/03main 07WikiTideBot 0304e9131 Bot: Auto-update domain lists [18:42:46] RECOVERY - prometheus151 Puppet on prometheus151 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [18:46:46] PROBLEM - prometheus151 Puppet on prometheus151 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 33 seconds ago with 0 failures [19:00:45] RECOVERY - prometheus151 Puppet on prometheus151 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [19:01:11] [02puppet] 07paladox pushed 1 new commit to 03paladox-patch-2 13https://github.com/miraheze/puppet/commit/a2b41049ee7da44330f91afd01323667991ea103 [19:01:11] 02puppet/03paladox-patch-2 07paladox 03a2b4104 Change include to class declaration for monitoring [19:06:45] PROBLEM - prometheus151 Puppet on prometheus151 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 1 minute ago with 0 failures [19:07:15] PROBLEM - mw151 Puppet on mw151 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [19:10:45] RECOVERY - prometheus151 Puppet on prometheus151 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [19:12:21] !log [skye@mwtask171] sudo -u www-data php /srv/mediawiki/1.45/maintenance/run.php importImages --wiki=pokemoninfinitywiki --sleep=1 '--comment=Importing images from https://p-infinity.fandom.com/ ([[phorge:T15038|T15038]])' -- . (END - exit=0) [19:12:22] !log [skye@mwtask171] sudo -u www-data php /srv/mediawiki/1.45/maintenance/run.php initSiteStats --wiki=pokemoninfinitywiki --update (START) [19:12:23] !log [skye@mwtask171] sudo -u www-data php /srv/mediawiki/1.45/maintenance/run.php initSiteStats --wiki=pokemoninfinitywiki --update (END - exit=0) [19:12:24] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:12:24] !log [skye@mwtask171] Finished import for pokemoninfinitywiki (XML: None; Images: .) (END - exit=0) [19:12:26] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:12:28] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:12:33] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:13:11] RECOVERY - mw151 Puppet on mw151 is OK: OK: Puppet is currently enabled, last run 18 seconds ago with 0 failures [19:16:02] PROBLEM - mw201 Current Load on mw201 is WARNING: LOAD WARNING - total load average: 21.63, 19.54, 17.54 [19:16:44] PROBLEM - prometheus151 Puppet on prometheus151 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 28 seconds ago with 0 failures [19:17:00] [02puppet] 07paladox pushed 1 new commit to 03paladox-patch-2 13https://github.com/miraheze/puppet/commit/9da62e8461a366300946b37075810aa2d025ece6 [19:17:00] 02puppet/03paladox-patch-2 07paladox 039da62e8 Update init.pp [19:18:02] RECOVERY - mw201 Current Load on mw201 is OK: LOAD OK - total load average: 20.35, 19.51, 17.76 [19:22:00] PROBLEM - puppet181 Check unit status of git_pull_puppet on puppet181 is CRITICAL: CRITICAL: Status of the systemd unit git_pull_puppet [19:23:43] [02puppet] 07paladox merged pull request #4809: prometheus: Fix to use puppetdb_query (03main...03paladox-patch-2) 13https://github.com/miraheze/puppet/pull/4809 [19:23:43] [02puppet] 07paladox pushed 1 new commit to 03main 13https://github.com/miraheze/puppet/commit/87f2a5259845d498e3c3ba97ccb1c706509256b8 [19:23:43] 02puppet/03main 07paladox 0387f2a52 prometheus: Fix to use puppetdb_query (#4809) [19:23:45] [02puppet] 07paladox 04deleted 03paladox-patch-2 at 039da62e8 13https://api.github.com/repos/miraheze/puppet/commit/9da62e8 [19:24:44] RECOVERY - prometheus151 Puppet on prometheus151 is OK: OK: Puppet is currently enabled, last run 22 seconds ago with 0 failures [19:27:44] [02TSPortal] 07Universal-Omega created 03renovate (+1 new commit) 13https://github.com/miraheze/TSPortal/commit/d1a8cfe2ff42 [19:27:44] 02TSPortal/03renovate 07CosmicAlpha 03d1a8cfe More reliable renovate configuration… [19:27:52] [02TSPortal] 07Universal-Omega opened pull request #59: More reliable renovate configuration (03main...03renovate) 13https://github.com/miraheze/TSPortal/pull/59 [19:28:59] [02TSPortal] 07Universal-Omega merged pull request #59: More reliable renovate configuration (03main...03renovate) 13https://github.com/miraheze/TSPortal/pull/59 [19:29:01] [02TSPortal] 07Universal-Omega pushed 1 new commit to 03main 13https://github.com/miraheze/TSPortal/commit/343f425ab6d13ae785f5f37958425f56ad8d9521 [19:29:02] 02TSPortal/03main 07CosmicAlpha 03343f425 More reliable renovate configuration… [19:29:04] [02TSPortal] 07Universal-Omega 04deleted 03renovate at 03d1a8cfe 13https://api.github.com/repos/miraheze/TSPortal/commit/d1a8cfe [19:29:31] [02TSPortal] 07renovate[bot] created 03renovate/dev-deps (+1 new commit) 13https://github.com/miraheze/TSPortal/commit/385ac8f77b5a [19:29:31] 02TSPortal/03renovate/dev-deps 07renovate[bot] 03385ac8f Update dependency nunomaduro/collision to ^8.9.1 [19:29:37] [02TSPortal] 07renovate[bot] created 03renovate/prod-deps (+1 new commit) 13https://github.com/miraheze/TSPortal/commit/784395362f30 [19:29:37] 02TSPortal/03renovate/prod-deps 07renovate[bot] 037843953 Update prod dependencies [19:29:40] [02TSPortal] 07renovate[bot] added the label 'composer' to pull request #60 (Update prod dependencies) 13https://github.com/miraheze/TSPortal/pull/60 [19:29:40] [02TSPortal] 07renovate[bot] added the label 'dependencies' to pull request #60 (Update prod dependencies) 13https://github.com/miraheze/TSPortal/pull/60 [19:29:42] [02TSPortal] 07renovate[bot] opened pull request #60: Update prod dependencies (03main...03renovate/prod-deps) 13https://github.com/miraheze/TSPortal/pull/60 [19:29:44] [02TSPortal] 07renovate[bot] added the label 'dependencies' to pull request #60 (Update prod dependencies) 13https://github.com/miraheze/TSPortal/pull/60 [19:29:46] [02TSPortal] 07renovate[bot] added the label 'composer' to pull request #60 (Update prod dependencies) 13https://github.com/miraheze/TSPortal/pull/60 [19:30:30] RECOVERY - puppet181 Check unit status of git_pull_puppet on puppet181 is OK: OK: Status of the systemd unit git_pull_puppet [19:31:24] [02TSPortal] 07Universal-Omega pushed 1 new commit to 03main 13https://github.com/miraheze/TSPortal/commit/9c736e9423947afcab2966c7605d046470b71e4a [19:31:24] 02TSPortal/03main 07CosmicAlpha 039c736e9 Don't auto bump php requirement [19:32:01] [02TSPortal] 07renovate[bot] 04force-pushed 03renovate/prod-deps from 037843953 to 034cdcde1 13https://api.github.com/repos/miraheze/TSPortal/commits/renovate/prod-deps [19:32:01] 02TSPortal/03renovate/prod-deps 07renovate[bot] 034cdcde1 Update dependency doctrine/dbal to ^4.4.2 [19:32:03] [02TSPortal] 07renovate[bot] retitled PR #60: "Update prod dependencies" ➜ "Update dependency doctrine/dbal to ^4.4.2" 13https://github.com/miraheze/TSPortal/pull/60 [19:33:23] [02TSPortal] 07Universal-Omega pushed 1 new commit to 03main 13https://github.com/miraheze/TSPortal/commit/7a66725b72e42197513edf09558849bff057d9c3 [19:33:23] 02TSPortal/03main 07CosmicAlpha 037a66725 Use replace strategy [19:34:03] [02TSPortal] 07renovate[bot] 04deleted 03renovate/dev-deps at 039debb64 13https://api.github.com/repos/miraheze/TSPortal/commit/9debb64 [19:34:03] [02TSPortal] 07renovate[bot] closed pull request #60: Update dependency doctrine/dbal to ^4.4.2 - autoclosed (03main...03renovate/prod-deps) 13https://github.com/miraheze/TSPortal/pull/60 [19:34:05] [02TSPortal] 07renovate[bot] retitled PR #60: "Update dependency doctrine/dbal to ^4.4.2" ➜ "Update dependency doctrine/dbal to ^4.4.2 - autoclosed" 13https://github.com/miraheze/TSPortal/pull/60 [19:34:07] [02TSPortal] 07renovate[bot] 04deleted 03renovate/prod-deps at 034cdcde1 13https://api.github.com/repos/miraheze/TSPortal/commit/4cdcde1 [19:37:17] [02TSPortal] 07renovate[bot] 04force-pushed 03renovate/dev-deps from 03385ac8f to 039debb64 13https://api.github.com/repos/miraheze/TSPortal/commits/renovate/dev-deps [19:37:17] 02TSPortal/03renovate/dev-deps 07renovate[bot] 039debb64 Update dependency nunomaduro/collision to ^8.9.1 [19:37:20] [Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.wikitide.net/d/GtxbP1Xnk?from=1772476610000&orgId=1&to=1772480240035 [19:37:25] PROBLEM - mw153 HTTPS on mw153 is WARNING: HTTP WARNING: HTTP/2 410 - 4308 bytes in 0.066 second response time [19:42:20] [Grafana] RESOLVED: PHP-FPM Worker Usage High https://grafana.wikitide.net/d/GtxbP1Xnk?from=1772476850000&orgId=1&to=1772480450000 [19:57:05] [02TSPortal] 07Universal-Omega created 03investigation-events (+1 new commit) 13https://github.com/miraheze/TSPortal/commit/4691637de210 [19:57:05] 02TSPortal/03investigation-events 07CosmicAlpha 034691637 Improvements/fixes to investigation events [19:57:12] [02TSPortal] 07Universal-Omega opened pull request #61: Improvements/fixes to investigation events (03main...03investigation-events) 13https://github.com/miraheze/TSPortal/pull/61 [19:58:08] [02TSPortal] 07Universal-Omega pushed 1 new commit to 03investigation-events 13https://github.com/miraheze/TSPortal/commit/86708ba6cdafc0389702b320959f38151d29791b [19:58:08] 02TSPortal/03investigation-events 07CosmicAlpha 0386708ba Only closed [19:58:59] [02TSPortal] 07Universal-Omega pushed 1 new commit to 03investigation-events 13https://github.com/miraheze/TSPortal/commit/2defeb3ad217c8891df7a7b3b6e20e9564c0c383 [19:59:00] 02TSPortal/03investigation-events 07CosmicAlpha 032defeb3 Reopened event [20:00:23] [02TSPortal] 07Universal-Omega pushed 1 new commit to 03investigation-events 13https://github.com/miraheze/TSPortal/commit/d566b0f653b122d20739b4bbdf05d698ff15e7ed [20:00:23] 02TSPortal/03investigation-events 07CosmicAlpha 03d566b0f Add event [20:07:14] RECOVERY - mw153 HTTPS on mw153 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4308 bytes in 0.077 second response time [20:09:07] [02TSPortal] 07Universal-Omega pushed 1 new commit to 03investigation-events 13https://github.com/miraheze/TSPortal/commit/625213cab4130461089affdea8f9617747e133b3 [20:09:07] 02TSPortal/03investigation-events 07CosmicAlpha 03625213c Not "New" if closed or reopened [20:32:55] PROBLEM - puppet181 Check unit status of listdomains_github_push on puppet181 is CRITICAL: CRITICAL: Status of the systemd unit listdomains_github_push [20:54:42] miraheze/mw-config - pskyechology the build passed. [20:57:18] !log [skye@mwtask181] starting deploy of {'pull': 'config', 'config': True} to all [20:57:20] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:57:51] !log [skye@mwtask181] finished deploy of {'pull': 'config', 'config': True} to all - SUCCESS in 33s [20:57:59] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:00:36] miraheze/mw-config - pskyechology the build passed. [21:01:39] RECOVERY - puppet181 Check unit status of listdomains_github_push on puppet181 is OK: OK: Status of the systemd unit listdomains_github_push [21:03:26] PROBLEM - mw201 Current Load on mw201 is WARNING: LOAD WARNING - total load average: 20.46, 20.70, 19.08 [21:05:21] RECOVERY - mw201 Current Load on mw201 is OK: LOAD OK - total load average: 18.88, 19.74, 18.91 [21:21:54] PROBLEM - mw191 MediaWiki Rendering on mw191 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 8191 bytes in 0.418 second response time [21:22:15] PROBLEM - mw191 Current Load on mw191 is CRITICAL: LOAD CRITICAL - total load average: 55.92, 33.34, 21.58 [21:22:16] PROBLEM - mw191 HTTPS on mw191 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 502 [21:22:20] PROBLEM - cp191 Varnish Backends on cp191 is CRITICAL: 1 backends are down. mw191 [21:22:53] PROBLEM - cp201 Varnish Backends on cp201 is CRITICAL: 1 backends are down. mw191 [21:23:08] PROBLEM - mw201 Current Load on mw201 is CRITICAL: LOAD CRITICAL - total load average: 27.88, 22.68, 20.39 [21:23:08] PROBLEM - cp171 Varnish Backends on cp171 is CRITICAL: 1 backends are down. mw191 [21:23:09] PROBLEM - cp161 Varnish Backends on cp161 is CRITICAL: 1 backends are down. mw191 [21:23:52] RECOVERY - mw191 MediaWiki Rendering on mw191 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.235 second response time [21:24:15] RECOVERY - mw191 HTTPS on mw191 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4308 bytes in 0.058 second response time [21:24:20] RECOVERY - cp191 Varnish Backends on cp191 is OK: All 31 backends are healthy [21:24:51] RECOVERY - cp201 Varnish Backends on cp201 is OK: All 31 backends are healthy [21:25:08] PROBLEM - mw201 Current Load on mw201 is WARNING: LOAD WARNING - total load average: 20.69, 21.60, 20.28 [21:25:08] RECOVERY - cp171 Varnish Backends on cp171 is OK: All 31 backends are healthy [21:25:09] RECOVERY - cp161 Varnish Backends on cp161 is OK: All 31 backends are healthy [21:28:07] PROBLEM - mw191 Current Load on mw191 is WARNING: LOAD WARNING - total load average: 13.05, 20.88, 19.90 [21:28:23] mw191 what are you cooking [21:31:19] itself? [21:32:04] RECOVERY - mw191 Current Load on mw191 is OK: LOAD OK - total load average: 12.42, 18.02, 19.07 [21:32:29] its ok only slightly cooking itself now [21:33:08] RECOVERY - mw201 Current Load on mw201 is OK: LOAD OK - total load average: 16.64, 19.65, 20.09 [21:45:08] PROBLEM - mw201 Current Load on mw201 is WARNING: LOAD WARNING - total load average: 21.97, 20.96, 20.45 [21:51:08] PROBLEM - mw201 Current Load on mw201 is CRITICAL: LOAD CRITICAL - total load average: 24.50, 22.56, 21.27 [21:51:30] @cosmicalpha you might want to see why mw* keep randomly going down [21:53:08] PROBLEM - mw201 Current Load on mw201 is WARNING: LOAD WARNING - total load average: 20.78, 21.78, 21.14 [21:59:08] RECOVERY - mw201 Current Load on mw201 is OK: LOAD OK - total load average: 18.80, 19.44, 20.32 [22:02:50] [Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.wikitide.net/d/GtxbP1Xnk?from=1772485340000&orgId=1&to=1772488970034 [22:05:08] PROBLEM - mw201 Current Load on mw201 is CRITICAL: LOAD CRITICAL - total load average: 25.45, 22.12, 21.10 [22:07:08] PROBLEM - mw201 Current Load on mw201 is WARNING: LOAD WARNING - total load average: 19.99, 20.67, 20.67 [22:12:50] [Grafana] RESOLVED: PHP-FPM Worker Usage High https://grafana.wikitide.net/d/GtxbP1Xnk?from=1772485340000&orgId=1&to=1772489510000 [22:13:08] RECOVERY - mw201 Current Load on mw201 is OK: LOAD OK - total load average: 19.97, 19.60, 20.17 [22:17:08] PROBLEM - mw201 Current Load on mw201 is CRITICAL: LOAD CRITICAL - total load average: 24.78, 21.28, 20.62 [22:19:08] PROBLEM - mw201 Current Load on mw201 is WARNING: LOAD WARNING - total load average: 19.99, 20.99, 20.62 [22:23:08] RECOVERY - mw201 Current Load on mw201 is OK: LOAD OK - total load average: 16.47, 19.46, 20.06 [22:33:19] !log [wwr@test151] sudo -u www-data php /srv/mediawiki/1.45/maintenance/run.php createAndPromote --wiki=metawikibeta TheWWRNerdGuy --interface-admin --force (END - exit=0) [22:33:21] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [22:37:53] PROBLEM - mw201 Current Load on mw201 is WARNING: LOAD WARNING - total load average: 20.79, 20.16, 19.80 [22:40:29] !log [wwr@test151] sudo -u www-data php /srv/mediawiki/1.45/maintenance/run.php createAndPromote --wiki=metawikibeta TheWWRNerdGuy --interface-admin --bureaucrat --sysop --force (END - exit=0) [22:40:32] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [22:41:45] RECOVERY - mw201 Current Load on mw201 is OK: LOAD OK - total load average: 19.73, 20.01, 19.86 [22:42:50] [Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.wikitide.net/d/GtxbP1Xnk?from=1772487740000&orgId=1&to=1772491370035 [22:46:46] !log [wwr@test151] starting deploy of {'config': True} to test151 [22:46:47] !log [wwr@test151] finished deploy of {'config': True} to test151 - SUCCESS in 0s [22:47:02] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [22:47:15] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [22:49:23] PROBLEM - ns1 Puppet on ns1 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_dns] [22:49:45] !log [wwr@test151] starting deploy of {'versions': '1.45', 'upgrade_extensions': 'Bucket'} to test151 [22:49:46] !log [wwr@test151] finished deploy of {'versions': '1.45', 'upgrade_extensions': 'Bucket'} to test151 - SUCCESS in 0s [22:49:59] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [22:50:03] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [22:51:19] !log [wwr@test151] starting deploy of {'l10n': True, 'extension_list': True, 'force_upgrade': True, 'versions': '1.45', 'upgrade_extensions': 'Bucket'} to test151 [22:51:21] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [22:54:06] !log [wwr@test151] DEPLOY ABORTED: Non-Zero Exit Code in prep, see output. [22:54:13] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [22:56:14] @thewwrnerdguy whats it screaming about [22:57:46] MintyDocs [22:58:32] what's the output, might beat it [22:59:09] He's in! [22:59:44] https://tenor.com/view/cover-page-hacker-man-gif-25175829 [23:04:19] currently teaching him how to explode us in 7.32 seconds or faster [23:05:09] already scaring him with concepts like "screen" and "mwdeploy" [23:08:20] !log [wwr@test151] starting deploy of {'l10n': True, 'extension_list': True, 'force_upgrade': True, 'versions': '1.45', 'upgrade_extensions': 'Bucket'} to test151 [23:08:29] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:12:23] !log [wwr@test151] finished deploy of {'l10n': True, 'extension_list': True, 'force_upgrade': True, 'versions': '1.45', 'upgrade_extensions': 'Bucket'} to test151 - SUCCESS in 242s [23:12:25] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:12:50] [Grafana] RESOLVED: PHP-FPM Worker Usage High https://grafana.wikitide.net/d/GtxbP1Xnk?from=1772487740000&orgId=1&to=1772493140000 [23:13:44] !log [wwr@test151] starting deploy of {'config': True} to test151 [23:13:45] !log [wwr@test151] finished deploy of {'config': True} to test151 - SUCCESS in 0s [23:13:50] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:13:54] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:17:22] RECOVERY - ns1 Puppet on ns1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [23:18:22] miraheze/dns - MacFan4000 the build passed. [23:19:00] let's go already blown up wikibeta once [23:25:22] PROBLEM - mw201 Current Load on mw201 is CRITICAL: LOAD CRITICAL - total load average: 26.32, 21.68, 20.19 [23:26:55] RECOVERY - graylog161 graylog tls port 12210 ssl cert check on graylog161 is OK: OK - Certificate 'wikitide.net' will expire on Sun May 31 22:19:44 2026 +0000.TCP OK - 0.183 second response time on localhost port 12210 [23:27:50] !log [wwr@test151] starting deploy of {'config': True} to test151 [23:27:51] !log [wwr@test151] finished deploy of {'config': True} to test151 - SUCCESS in 0s [23:27:54] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:27:56] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:28:47] !log [wwr@test151] starting deploy of {'config': True} to test151 [23:28:48] !log [wwr@test151] finished deploy of {'config': True} to test151 - SUCCESS in 0s [23:28:49] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:28:53] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:29:13] PROBLEM - mw201 Current Load on mw201 is WARNING: LOAD WARNING - total load average: 19.98, 20.66, 20.07 [23:30:06] !log [wwr@test151] starting deploy of {'config': True} to test151 [23:30:07] !log [wwr@test151] finished deploy of {'config': True} to test151 - SUCCESS in 0s [23:30:11] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:30:14] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:31:09] RECOVERY - mw201 Current Load on mw201 is OK: LOAD OK - total load average: 19.74, 20.06, 19.92 [23:34:18] !log [wwr@test151] starting deploy of {'config': True} to test151 [23:34:19] !log [wwr@test151] finished deploy of {'config': True} to test151 - SUCCESS in 0s [23:34:20] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:34:21] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:37:23] PROBLEM - test151 Puppet on test151 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_checkout_MediaWiki config] [23:38:01] !log [skye@test151] sudo -u www-data php /srv/mediawiki/1.45/maintenance/run.php Bucket:SetupDBPermission --wiki=skye2wikibeta (END - exit=0) [23:38:07] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:39:53] !log [wwr@test151] starting deploy of {'config': True} to test151 [23:39:54] !log [wwr@test151] finished deploy of {'config': True} to test151 - SUCCESS in 0s [23:40:11] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:40:15] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:42:02] PROBLEM - mw201 Current Load on mw201 is CRITICAL: LOAD CRITICAL - total load average: 24.48, 22.47, 20.85 [23:43:02] !log [wwr@test151] sudo -u www-data php /srv/mediawiki/1.45/maintenance/run.php Bucket:SetupDBPermission --wiki=exttestwikibeta (END - exit=0) [23:43:05] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:45:48] PROBLEM - mw183 Current Load on mw183 is WARNING: LOAD WARNING - total load average: 22.95, 20.00, 17.55 [23:47:42] RECOVERY - mw183 Current Load on mw183 is OK: LOAD OK - total load average: 15.46, 18.94, 17.50 [23:47:50] PROBLEM - mw201 Current Load on mw201 is WARNING: LOAD WARNING - total load average: 19.51, 23.30, 21.96 [23:55:33] RECOVERY - mw201 Current Load on mw201 is OK: LOAD OK - total load average: 16.25, 18.23, 19.96 [23:59:07] PROBLEM - mw183 Current Load on mw183 is CRITICAL: LOAD CRITICAL - total load average: 25.24, 20.91, 18.21