[00:00:16] [02TSPortal] 07Universal-Omega pushed 1 new commit to 03linttest 13https://github.com/miraheze/TSPortal/commit/634a1ff0a11e26cab1c30313081f7cdd63fae50c [00:00:16] 02TSPortal/03linttest 07CosmicAlpha 03634a1ff Update [00:01:09] [02statichelp] 07WikiTideBot pushed 1 new commit to 03main 13https://github.com/miraheze/statichelp/commit/fe6a922f95df055a65c8d866d7c876f0bcde6b02 [00:01:09] 02statichelp/03main 07WikiTideBot 03fe6a922 Bot: Auto-update Tech namespace pages 2026-01-27 00:01:06 [00:16:51] [02TSPortal] 07Universal-Omega pushed 1 new commit to 03linttest 13https://github.com/miraheze/TSPortal/commit/695541c3837dd89f5bc830897a57b827cbd15b79 [00:16:51] 02TSPortal/03linttest 07CosmicAlpha 03695541c Update [00:20:48] RECOVERY - cp171 Disk Space on cp171 is OK: DISK OK - free space: / 73024MiB (16.1% inode=100%); [00:21:29] [02TSPortal] 07Universal-Omega pushed 1 new commit to 03linttest 13https://github.com/miraheze/TSPortal/commit/c8e82c39a21f21bdd033608250279d8371048502 [00:21:29] 02TSPortal/03linttest 07CosmicAlpha 03c8e82c3 Update [00:21:52] [02TSPortal] 07github-actions[bot] pushed 1 new commit to 03linttest 13https://github.com/miraheze/TSPortal/commit/ca853428d301d4de537145a75c3ddde169b148bc [00:21:53] 02TSPortal/03linttest 07Universal-Omega 03ca85342 Fix styling [00:27:09] RECOVERY - cp191 Disk Space on cp191 is OK: DISK OK - free space: / 72784MiB (16.0% inode=100%); [00:33:22] [02TSPortal] 07Universal-Omega closed pull request #50: Add laravel pint workflow (03main...03pint) 13https://github.com/miraheze/TSPortal/pull/50 [00:33:23] [02TSPortal] 07Universal-Omega closed pull request #51: Test new lint workflow (03main...03linttest) 13https://github.com/miraheze/TSPortal/pull/51 [00:33:33] [02TSPortal] 07Universal-Omega 04deleted 03pint at 033a487d4 13https://api.github.com/repos/miraheze/TSPortal/commit/3a487d4 [00:33:39] [02TSPortal] 07Universal-Omega 04deleted 03linttest at 03ca85342 13https://api.github.com/repos/miraheze/TSPortal/commit/ca85342 [00:33:49] [02TSPortal] 07Universal-Omega closed pull request #48: Add lint workflow (03main...03lint) 13https://github.com/miraheze/TSPortal/pull/48 [00:33:52] [02TSPortal] 07Universal-Omega 04deleted 03lint at 03a7633c9 13https://api.github.com/repos/miraheze/TSPortal/commit/a7633c9 [00:35:42] RECOVERY - cp201 Disk Space on cp201 is OK: DISK OK - free space: / 72259MiB (15.9% inode=100%); [00:37:11] PROBLEM - bast161 NTP time on bast161 is UNKNOWN: check_ntp_time: Invalid hostname/address - time.cloudflare.comUsage: check_ntp_time -H [-4|-6] [-w ] [-c ] [-v verbose] [-o ] [00:37:18] PROBLEM - puppet181 Check unit status of listdomains_github_push on puppet181 is CRITICAL: CRITICAL: Status of the systemd unit listdomains_github_push [00:37:51] PROBLEM - mw183 Puppet on mw183 is CRITICAL: CRITICAL: Puppet has 2 failures. Last run 2 minutes ago with 2 failures. Failed resources (up to 3 shown): Exec[git_pull_femiwiki-deploy-1.44],Exec[git_pull_3d2png] [00:38:31] PROBLEM - mw182 Puppet on mw182 is CRITICAL: CRITICAL: Puppet has 2 failures. Last run 2 minutes ago with 2 failures. Failed resources (up to 3 shown): Exec[git_pull_femiwiki-deploy-1.44],Exec[git_pull_3d2png] [00:38:56] PROBLEM - mw152 Puppet on mw152 is CRITICAL: CRITICAL: Puppet has 2 failures. Last run 3 minutes ago with 2 failures. Failed resources (up to 3 shown): Exec[git_pull_femiwiki-deploy-1.44],Exec[git_pull_3d2png] [00:41:22] PROBLEM - puppet181 Check unit status of git_pull_ssl on puppet181 is CRITICAL: CRITICAL: Status of the systemd unit git_pull_ssl [00:41:23] PROBLEM - bast161 PowerDNS Recursor on bast161 is CRITICAL: Domain 'bast161.wikitide.net' was not found by the server [00:41:45] PROBLEM - mw162 Puppet on mw162 is CRITICAL: CRITICAL: Puppet has 2 failures. Last run 2 minutes ago with 2 failures. Failed resources (up to 3 shown): Exec[git_pull_femiwiki-deploy-1.44],Exec[git_pull_3d2png] [00:41:51] PROBLEM - puppet181 Check unit status of git_pull_puppet on puppet181 is CRITICAL: CRITICAL: Status of the systemd unit git_pull_puppet [00:42:48] PROBLEM - bast161 Puppet on bast161 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [00:42:53] PROBLEM - mw202 Puppet on mw202 is CRITICAL: CRITICAL: Puppet has 2 failures. Last run 3 minutes ago with 2 failures. Failed resources (up to 3 shown): Exec[git_pull_femiwiki-deploy-1.44],Exec[git_pull_3d2png] [00:43:06] PROBLEM - mw172 Puppet on mw172 is CRITICAL: CRITICAL: Puppet has 2 failures. Last run 2 minutes ago with 2 failures. Failed resources (up to 3 shown): Exec[git_pull_femiwiki-deploy-1.44],Exec[git_pull_3d2png] [00:43:39] PROBLEM - mw203 Puppet on mw203 is CRITICAL: CRITICAL: Puppet has 2 failures. Last run 2 minutes ago with 2 failures. Failed resources (up to 3 shown): Exec[git_pull_femiwiki-deploy-1.44],Exec[git_pull_3d2png] [00:44:16] PROBLEM - mw161 Puppet on mw161 is CRITICAL: CRITICAL: Puppet has 2 failures. Last run 3 minutes ago with 2 failures. Failed resources (up to 3 shown): Exec[git_pull_femiwiki-deploy-1.44],Exec[git_pull_3d2png] [00:44:53] PROBLEM - puppet181 Puppet on puppet181 is CRITICAL: CRITICAL: Puppet has 4 failures. Last run 2 minutes ago with 4 failures. Failed resources (up to 3 shown): Exec[git_pull_puppet],Exec[git_pull_ssl],Exec[git_pull_mediawiki-repos],Exec[git_pull_pywikibot-config] [00:45:02] PROBLEM - mwtask171 Puppet on mwtask171 is CRITICAL: CRITICAL: Puppet has 3 failures. Last run 3 minutes ago with 3 failures. Failed resources (up to 3 shown): Exec[git_pull_mathoid],Exec[git_pull_femiwiki-deploy-1.44],Exec[git_pull_3d2png] [00:45:32] PROBLEM - mw173 Puppet on mw173 is CRITICAL: CRITICAL: Puppet has 2 failures. Last run 2 minutes ago with 2 failures. Failed resources (up to 3 shown): Exec[git_pull_femiwiki-deploy-1.44],Exec[git_pull_3d2png] [00:45:44] PROBLEM - mwtask161 Puppet on mwtask161 is CRITICAL: CRITICAL: Puppet has 3 failures. Last run 2 minutes ago with 3 failures. Failed resources (up to 3 shown): Exec[git_pull_mathoid],Exec[git_pull_femiwiki-deploy-1.44],Exec[git_pull_3d2png] [00:46:17] PROBLEM - mwtask151 Puppet on mwtask151 is CRITICAL: CRITICAL: Puppet has 3 failures. Last run 2 minutes ago with 3 failures. Failed resources (up to 3 shown): Exec[git_pull_mathoid],Exec[git_pull_femiwiki-deploy-1.44],Exec[git_pull_3d2png] [00:46:55] PROBLEM - phorge171 Puppet on phorge171 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_phorge-extensions] [00:49:09] PROBLEM - mw193 Puppet on mw193 is CRITICAL: CRITICAL: Puppet has 2 failures. Last run 2 minutes ago with 2 failures. Failed resources (up to 3 shown): Exec[git_pull_femiwiki-deploy-1.44],Exec[git_pull_3d2png] [00:52:44] PROBLEM - mw153 Puppet on mw153 is CRITICAL: CRITICAL: Puppet has 2 failures. Last run 2 minutes ago with 2 failures. Failed resources (up to 3 shown): Exec[git_pull_femiwiki-deploy-1.44],Exec[git_pull_3d2png] [00:53:25] PROBLEM - mw191 Puppet on mw191 is CRITICAL: CRITICAL: Puppet has 2 failures. Last run 2 minutes ago with 2 failures. Failed resources (up to 3 shown): Exec[git_pull_femiwiki-deploy-1.44],Exec[git_pull_3d2png] [00:53:53] PROBLEM - mw171 Puppet on mw171 is CRITICAL: CRITICAL: Puppet has 2 failures. Last run 3 minutes ago with 2 failures. Failed resources (up to 3 shown): Exec[git_pull_femiwiki-deploy-1.44],Exec[git_pull_3d2png] [00:57:11] PROBLEM - bots171 Puppet on bots171 is CRITICAL: CRITICAL: Puppet has 2 failures. Last run 3 minutes ago with 2 failures. Failed resources (up to 3 shown): Exec[git_pull_Pywikibot-stable],Exec[git_pull_IRC-Discord-Relay-relaybot] [00:59:38] PROBLEM - mw192 Puppet on mw192 is CRITICAL: CRITICAL: Puppet has 2 failures. Last run 3 minutes ago with 2 failures. Failed resources (up to 3 shown): Exec[git_pull_femiwiki-deploy-1.44],Exec[git_pull_3d2png] [01:01:09] PROBLEM - mw151 Puppet on mw151 is CRITICAL: CRITICAL: Puppet has 2 failures. Last run 2 minutes ago with 2 failures. Failed resources (up to 3 shown): Exec[git_pull_femiwiki-deploy-1.44],Exec[git_pull_3d2png] [01:01:58] PROBLEM - mw163 Puppet on mw163 is CRITICAL: CRITICAL: Puppet has 2 failures. Last run 3 minutes ago with 2 failures. Failed resources (up to 3 shown): Exec[git_pull_femiwiki-deploy-1.44],Exec[git_pull_3d2png] [01:04:49] PROBLEM - mw181 Puppet on mw181 is CRITICAL: CRITICAL: Puppet has 2 failures. Last run 2 minutes ago with 2 failures. Failed resources (up to 3 shown): Exec[git_pull_femiwiki-deploy-1.44],Exec[git_pull_3d2png] [01:05:25] PROBLEM - mw201 Puppet on mw201 is CRITICAL: CRITICAL: Puppet has 2 failures. Last run 2 minutes ago with 2 failures. Failed resources (up to 3 shown): Exec[git_pull_femiwiki-deploy-1.44],Exec[git_pull_3d2png] [01:05:32] PROBLEM - test151 Puppet on test151 is CRITICAL: CRITICAL: Puppet has 4 failures. Last run 2 minutes ago with 4 failures. Failed resources (up to 3 shown): Exec[git_pull_mathoid],Exec[git_pull_femiwiki-deploy-1.44],Exec[git_pull_femiwiki-deploy-1.45],Exec[git_pull_3d2png] [01:07:08] PROBLEM - mwtask181 Puppet on mwtask181 is CRITICAL: CRITICAL: Puppet has 3 failures. Last run 3 minutes ago with 3 failures. Failed resources (up to 3 shown): Exec[git_pull_mathoid],Exec[git_pull_femiwiki-deploy-1.44],Exec[git_pull_3d2png] [01:28:13] PROBLEM - bast161 NTP time on bast161 is CRITICAL: connect to address 10.0.16.127 port 5666: No route to hostconnect to host 10.0.16.127 port 5666: No route to host [01:29:58] [02dns] 07Universal-Omega pushed 1 new commit to 03main 13https://github.com/miraheze/dns/commit/df75093d0283c2918e7e7b828834b322651e9c89 [01:29:58] 02dns/03main 07CosmicAlpha 03df75093 Switch bastion to bast181 for now [01:30:11] PROBLEM - bast161 NTP time on bast161 is UNKNOWN: check_ntp_time: Invalid hostname/address - time.cloudflare.comUsage: check_ntp_time -H [-4|-6] [-w ] [-c ] [-v verbose] [-o ] [01:30:50] miraheze/dns - Universal-Omega the build passed. [01:33:39] RECOVERY - mw201 Puppet on mw201 is OK: OK: Puppet is currently enabled, last run 15 seconds ago with 0 failures [01:34:18] RECOVERY - mw152 Puppet on mw152 is OK: OK: Puppet is currently enabled, last run 28 seconds ago with 0 failures [01:35:59] RECOVERY - puppet181 Check unit status of listdomains_github_push on puppet181 is OK: OK: Status of the systemd unit listdomains_github_push [01:36:00] RECOVERY - mw183 Puppet on mw183 is OK: OK: Puppet is currently enabled, last run 17 seconds ago with 0 failures [01:36:24] RECOVERY - mw182 Puppet on mw182 is OK: OK: Puppet is currently enabled, last run 52 seconds ago with 0 failures [01:37:31] PROBLEM - bast161 NTP time on bast161 is CRITICAL: connect to address 10.0.16.127 port 5666: Connection refusedconnect to host 10.0.16.127 port 5666: Connection refused [01:39:27] RECOVERY - bast161 NTP time on bast161 is OK: NTP OK: Offset -3.463029861e-05 secs [01:39:35] RECOVERY - bast161 PowerDNS Recursor on bast161 is OK: DNS OK: 0.046 seconds response time. bast161.wikitide.net returns 2602:294:0:b13::101,38.46.223.203 [01:39:44] RECOVERY - puppet181 Check unit status of git_pull_ssl on puppet181 is OK: OK: Status of the systemd unit git_pull_ssl [01:40:43] RECOVERY - mw172 Puppet on mw172 is OK: OK: Puppet is currently enabled, last run 31 seconds ago with 0 failures [01:41:00] RECOVERY - mw162 Puppet on mw162 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [01:41:15] RECOVERY - mw202 Puppet on mw202 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [01:41:19] RECOVERY - mw161 Puppet on mw161 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [01:41:27] RECOVERY - mwtask171 Puppet on mwtask171 is OK: OK: Puppet is currently enabled, last run 1 second ago with 0 failures [01:41:31] RECOVERY - bast161 Puppet on bast161 is OK: OK: Puppet is currently enabled, last run 10 seconds ago with 0 failures [01:42:28] RECOVERY - bots171 Puppet on bots171 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [01:43:15] RECOVERY - mw203 Puppet on mw203 is OK: OK: Puppet is currently enabled, last run 38 seconds ago with 0 failures [01:43:36] RECOVERY - mw181 Puppet on mw181 is OK: OK: Puppet is currently enabled, last run 47 seconds ago with 0 failures [01:43:40] RECOVERY - mw171 Puppet on mw171 is OK: OK: Puppet is currently enabled, last run 52 seconds ago with 0 failures [01:43:45] RECOVERY - mw163 Puppet on mw163 is OK: OK: Puppet is currently enabled, last run 56 seconds ago with 0 failures [01:43:49] RECOVERY - mw193 Puppet on mw193 is OK: OK: Puppet is currently enabled, last run 56 seconds ago with 0 failures [01:43:51] RECOVERY - puppet181 Puppet on puppet181 is OK: OK: Puppet is currently enabled, last run 20 seconds ago with 0 failures [01:43:59] RECOVERY - phorge171 Puppet on phorge171 is OK: OK: Puppet is currently enabled, last run 45 seconds ago with 0 failures [01:44:05] RECOVERY - mwtask161 Puppet on mwtask161 is OK: OK: Puppet is currently enabled, last run 46 seconds ago with 0 failures [01:44:17] RECOVERY - mw173 Puppet on mw173 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [01:44:23] RECOVERY - mw192 Puppet on mw192 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [01:44:30] RECOVERY - mwtask151 Puppet on mwtask151 is OK: OK: Puppet is currently enabled, last run 49 seconds ago with 0 failures [01:44:31] RECOVERY - mw151 Puppet on mw151 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [01:44:32] RECOVERY - mw153 Puppet on mw153 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [01:44:35] RECOVERY - mwtask181 Puppet on mwtask181 is OK: OK: Puppet is currently enabled, last run 50 seconds ago with 0 failures [01:44:39] RECOVERY - mw191 Puppet on mw191 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [01:44:40] RECOVERY - test151 Puppet on test151 is OK: OK: Puppet is currently enabled, last run 10 seconds ago with 0 failures [01:46:15] !log [universalomega@puppet181] Upgraded packages on mw152: libimage-magick-q16-perl, libimage-magick-perl, imagemagick-6-common, libmagickcore-6.q16-6, libmagickwand-6.q16-6, imagemagick-6.q16, imagemagick, libmagick++-6.q16-8, and libmagickcore-6.q16-6-extra [01:46:17] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:46:33] !log [universalomega@puppet181] Upgraded packages on mw171: libimage-magick-q16-perl, libimage-magick-perl, imagemagick-6-common, libmagickcore-6.q16-6, libmagickwand-6.q16-6, imagemagick-6.q16, imagemagick, libmagick++-6.q16-8, and libmagickcore-6.q16-6-extra [01:46:35] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:46:37] RECOVERY - mw152 APT on mw152 is OK: APT OK: 147 packages available for upgrade (0 critical updates). [01:46:53] !log [universalomega@puppet181] Upgraded packages on mw162: libimage-magick-q16-perl, libimage-magick-perl, imagemagick-6-common, libmagickcore-6.q16-6, libmagickwand-6.q16-6, imagemagick-6.q16, imagemagick, libmagick++-6.q16-8, and libmagickcore-6.q16-6-extra [01:46:55] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:47:12] !log [universalomega@puppet181] Upgraded packages on mw172: libimage-magick-q16-perl, libimage-magick-perl, imagemagick-6-common, libmagickcore-6.q16-6, libmagickwand-6.q16-6, imagemagick-6.q16, imagemagick, libmagick++-6.q16-8, and libmagickcore-6.q16-6-extra [01:47:14] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:47:31] !log [universalomega@puppet181] Upgraded packages on mw153: libimage-magick-q16-perl, libimage-magick-perl, imagemagick-6-common, libmagickcore-6.q16-6, libmagickwand-6.q16-6, imagemagick-6.q16, imagemagick, libmagick++-6.q16-8, and libmagickcore-6.q16-6-extra [01:47:33] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:47:51] !log [universalomega@puppet181] Upgraded packages on mw173: libimage-magick-q16-perl, libimage-magick-perl, imagemagick-6-common, libmagickcore-6.q16-6, libmagickwand-6.q16-6, imagemagick-6.q16, imagemagick, libmagick++-6.q16-8, and libmagickcore-6.q16-6-extra [01:47:53] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:47:57] RECOVERY - mw173 APT on mw173 is OK: APT OK: 148 packages available for upgrade (0 critical updates). [01:48:09] RECOVERY - mw162 APT on mw162 is OK: APT OK: 147 packages available for upgrade (0 critical updates). [01:48:10] RECOVERY - mw172 APT on mw172 is OK: APT OK: 147 packages available for upgrade (0 critical updates). [01:48:11] !log [universalomega@puppet181] Upgraded packages on mw163: libimage-magick-q16-perl, libimage-magick-perl, imagemagick-6-common, libmagickcore-6.q16-6, libmagickwand-6.q16-6, imagemagick-6.q16, imagemagick, libmagick++-6.q16-8, and libmagickcore-6.q16-6-extra [01:48:12] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:48:27] RECOVERY - mw171 APT on mw171 is OK: APT OK: 147 packages available for upgrade (0 critical updates). [01:48:30] !log [universalomega@puppet181] Upgraded packages on mw161: libimage-magick-q16-perl, libimage-magick-perl, imagemagick-6-common, libmagickcore-6.q16-6, libmagickwand-6.q16-6, imagemagick-6.q16, imagemagick, libmagick++-6.q16-8, and libmagickcore-6.q16-6-extra [01:48:31] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:48:45] RECOVERY - mw163 APT on mw163 is OK: APT OK: 148 packages available for upgrade (0 critical updates). [01:48:49] !log [universalomega@puppet181] Upgraded packages on mwtask151: libimage-magick-q16-perl, libimage-magick-perl, imagemagick-6-common, libmagickcore-6.q16-6, libmagickwand-6.q16-6, imagemagick-6.q16, imagemagick, libmagick++-6.q16-8, and libmagickcore-6.q16-6-extra [01:48:50] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:48:51] RECOVERY - mw153 APT on mw153 is OK: APT OK: 148 packages available for upgrade (0 critical updates). [01:49:09] !log [universalomega@puppet181] Upgraded packages on mwtask161: libimage-magick-q16-perl, libimage-magick-perl, imagemagick-6-common, libmagickcore-6.q16-6, libmagickwand-6.q16-6, imagemagick-6.q16, imagemagick, libmagick++-6.q16-8, and libmagickcore-6.q16-6-extra [01:49:11] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:49:30] !log [universalomega@puppet181] Upgraded packages on mw191: libimage-magick-q16-perl, libimage-magick-perl, imagemagick-6-common, libmagickcore-6.q16-6, libmagickwand-6.q16-6, imagemagick-6.q16, imagemagick, libmagick++-6.q16-8, and libmagickcore-6.q16-6-extra [01:49:32] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:49:33] RECOVERY - mw191 APT on mw191 is OK: APT OK: 148 packages available for upgrade (0 critical updates). [01:49:50] !log [universalomega@puppet181] Upgraded packages on mwtask171: libimage-magick-q16-perl, libimage-magick-perl, imagemagick-6-common, libmagickcore-6.q16-6, libmagickwand-6.q16-6, imagemagick-6.q16, imagemagick, libmagick++-6.q16-8, and libmagickcore-6.q16-6-extra [01:49:52] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:50:09] RECOVERY - mw161 APT on mw161 is OK: APT OK: 147 packages available for upgrade (0 critical updates). [01:50:10] RECOVERY - mwtask151 APT on mwtask151 is OK: APT OK: 152 packages available for upgrade (0 critical updates). [01:50:11] !log [universalomega@puppet181] Upgraded packages on mw151: libimage-magick-q16-perl, libimage-magick-perl, imagemagick-6-common, libmagickcore-6.q16-6, libmagickwand-6.q16-6, imagemagick-6.q16, imagemagick, libmagick++-6.q16-8, and libmagickcore-6.q16-6-extra [01:50:12] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:50:25] RECOVERY - mwtask161 APT on mwtask161 is OK: APT OK: 152 packages available for upgrade (0 critical updates). [01:50:31] !log [universalomega@puppet181] Upgraded packages on mw192: libimage-magick-q16-perl, libimage-magick-perl, imagemagick-6-common, libmagickcore-6.q16-6, libmagickwand-6.q16-6, imagemagick-6.q16, imagemagick, libmagick++-6.q16-8, and libmagickcore-6.q16-6-extra [01:50:32] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:50:41] RECOVERY - mwtask171 APT on mwtask171 is OK: APT OK: 158 packages available for upgrade (0 critical updates). [01:50:50] PROBLEM - test151 MediaWiki Rendering on test151 is CRITICAL: connect to address 10.0.15.118 and port 443: Connection refusedHTTP CRITICAL - Unable to open TCP socket [01:50:52] !log [universalomega@puppet181] Upgraded packages on mw182: libimage-magick-q16-perl, libimage-magick-perl, imagemagick-6-common, libmagickcore-6.q16-6, libmagickwand-6.q16-6, imagemagick-6.q16, imagemagick, libmagick++-6.q16-8, and libmagickcore-6.q16-6-extra [01:50:54] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:50:55] PROBLEM - test151 HTTPS on test151 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 7 - Failed to connect to test151.fsslc.wtnet port 443 after 0 ms: Could not connect to server [01:50:57] RECOVERY - mw151 APT on mw151 is OK: APT OK: 149 packages available for upgrade (0 critical updates). [01:51:13] !log [universalomega@puppet181] Upgraded packages on mw202: libimage-magick-q16-perl, libimage-magick-perl, imagemagick-6-common, libmagickcore-6.q16-6, libmagickwand-6.q16-6, imagemagick-6.q16, imagemagick, libmagick++-6.q16-8, and libmagickcore-6.q16-6-extra [01:51:14] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:51:34] !log [universalomega@puppet181] Upgraded packages on mw201: libimage-magick-q16-perl, libimage-magick-perl, imagemagick-6-common, libmagickcore-6.q16-6, libmagickwand-6.q16-6, imagemagick-6.q16, imagemagick, libmagick++-6.q16-8, and libmagickcore-6.q16-6-extra [01:51:36] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:51:45] RECOVERY - mw192 APT on mw192 is OK: APT OK: 148 packages available for upgrade (0 critical updates). [01:51:57] !log [universalomega@puppet181] Upgraded packages on mw181: libimage-magick-q16-perl, libimage-magick-perl, imagemagick-6-common, libmagickcore-6.q16-6, libmagickwand-6.q16-6, imagemagick-6.q16, imagemagick, libmagick++-6.q16-8, and libmagickcore-6.q16-6-extra [01:51:59] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:52:04] PROBLEM - test151 conntrack_table_size on test151 is CRITICAL: connect to address 10.0.15.118 port 5666: No route to hostconnect to host 10.0.15.118 port 5666: No route to host [01:52:17] RECOVERY - mw182 APT on mw182 is OK: APT OK: 148 packages available for upgrade (0 critical updates). [01:52:18] !log [universalomega@puppet181] Upgraded packages on mw203: libimage-magick-q16-perl, libimage-magick-perl, imagemagick-6-common, libmagickcore-6.q16-6, libmagickwand-6.q16-6, imagemagick-6.q16, imagemagick, libmagick++-6.q16-8, and libmagickcore-6.q16-6-extra [01:52:19] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:52:20] RECOVERY - mw201 APT on mw201 is OK: APT OK: 148 packages available for upgrade (0 critical updates). [01:52:27] RECOVERY - mw202 APT on mw202 is OK: APT OK: 148 packages available for upgrade (0 critical updates). [01:52:40] !log [universalomega@puppet181] Upgraded packages on mw193: libimage-magick-q16-perl, libimage-magick-perl, imagemagick-6-common, libmagickcore-6.q16-6, libmagickwand-6.q16-6, imagemagick-6.q16, imagemagick, libmagick++-6.q16-8, and libmagickcore-6.q16-6-extra [01:52:41] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:52:45] RECOVERY - mw181 APT on mw181 is OK: APT OK: 148 packages available for upgrade (0 critical updates). [01:52:58] !log [universalomega@puppet181] Upgraded packages on mwtask181: libimage-magick-q16-perl, libimage-magick-perl, imagemagick-6-common, libmagickcore-6.q16-6, libmagickwand-6.q16-6, imagemagick-6.q16, imagemagick, libmagick++-6.q16-8, and libmagickcore-6.q16-6-extra [01:53:00] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:53:07] PROBLEM - Host test151 is DOWN: CRITICAL - Host Unreachable (10.0.15.118) [01:53:21] !log [universalomega@puppet181] Upgraded packages on mw183: libimage-magick-q16-perl, libimage-magick-perl, imagemagick-6-common, libmagickcore-6.q16-6, libmagickwand-6.q16-6, imagemagick-6.q16, imagemagick, libmagick++-6.q16-8, and libmagickcore-6.q16-6-extra [01:53:22] RECOVERY - mw203 APT on mw203 is OK: APT OK: 148 packages available for upgrade (0 critical updates). [01:53:22] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:53:24] RECOVERY - mw193 APT on mw193 is OK: APT OK: 148 packages available for upgrade (0 critical updates). [01:53:28] RECOVERY - mwtask181 APT on mwtask181 is OK: APT OK: 66 packages available for upgrade (0 critical updates). [01:53:35] RECOVERY - mw183 APT on mw183 is OK: APT OK: 148 packages available for upgrade (0 critical updates). [01:54:23] !log [universalomega@puppet181] Upgraded packages on test151: connect [01:54:24] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:59:05] RECOVERY - Host test151 is UP: PING OK - Packet loss = 68%, RTA = 0.35 ms [01:59:17] RECOVERY - test151 HTTPS on test151 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4256 bytes in 0.198 second response time [01:59:23] RECOVERY - test151 MediaWiki Rendering on test151 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 3.630 second response time [02:00:48] RECOVERY - test151 conntrack_table_size on test151 is OK: OK: nf_conntrack is 0 % full [02:01:08] PROBLEM - test151 PowerDNS Recursor on test151 is CRITICAL: connect to address 10.0.15.118 port 5666: Connection refusedconnect to host 10.0.15.118 port 5666: Connection refused PROBLEM - test151 php-fpm on test151 is CRITICAL: connect to address 10.0.15.118 port 5666: Connection refusedconnect to host 10.0.15.118 port 5666: Connection refused [02:01:13] PROBLEM - test151 NTP time on test151 is WARNING: NTP WARNING: Offset 0.3135638237 secs [02:03:00] RECOVERY - test151 php-fpm on test151 is OK: PROCS OK: 25 processes with command name 'php-fpm8.2' [02:03:04] RECOVERY - test151 PowerDNS Recursor on test151 is OK: DNS OK: 0.040 seconds response time. test151.fsslc.wtnet returns 10.0.15.118 [02:03:08] RECOVERY - test151 NTP time on test151 is OK: NTP OK: Offset -1.201033592e-05 secs [02:04:00] !log [universalomega@puppet181] Upgraded packages on test151: libimage-magick-q16-perl, libimage-magick-perl, imagemagick-6-common, libmagickcore-6.q16-6, libmagickwand-6.q16-6, imagemagick-6.q16, imagemagick, libmagick++-6.q16-8, and libmagickcore-6.q16-6-extra [02:04:02] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:05:28] RECOVERY - test151 APT on test151 is OK: APT OK: 143 packages available for upgrade (0 critical updates). [02:37:59] PROBLEM - test151 MediaWiki Rendering on test151 is CRITICAL: Name or service not knownHTTP CRITICAL - Unable to open TCP socket [02:39:08] PROBLEM - test151 HTTPS on test151 is CRITICAL: HTTP CRITICAL - Unable to lookup IP address for '': getaddrinfo returned -2 - Name or service not known [06:43:18] PROBLEM - llm191 Puppet on llm191 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[openwebui] [07:11:11] RECOVERY - llm191 Puppet on llm191 is OK: OK: Puppet is currently enabled, last run 42 seconds ago with 0 failures [07:23:00] PROBLEM - kafka182 APT on kafka182 is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates). [07:29:08] PROBLEM - os162 APT on os162 is CRITICAL: APT CRITICAL: 4 packages available for upgrade (4 critical updates). [07:29:41] PROBLEM - graylog161 APT on graylog161 is CRITICAL: APT CRITICAL: 5 packages available for upgrade (4 critical updates). [07:30:21] PROBLEM - os202 APT on os202 is CRITICAL: APT CRITICAL: 4 packages available for upgrade (4 critical updates). [07:31:53] PROBLEM - os191 APT on os191 is CRITICAL: APT CRITICAL: 4 packages available for upgrade (4 critical updates). [07:33:25] PROBLEM - os161 APT on os161 is CRITICAL: APT CRITICAL: 4 packages available for upgrade (4 critical updates). [07:40:06] [1/4] https://cdn.discordapp.com/attachments/808001911868489748/1465612314123174050/image.png?ex=6979bd55&is=69786bd5&hm=6eb5b263dca2ea1e669243bd03b918eb4c365f8a60154c7b3e542087d05df6ae& [07:40:06] [2/4] https://cdn.discordapp.com/attachments/808001911868489748/1465612314681282601/image.png?ex=6979bd55&is=69786bd5&hm=300d5a38726fbfb5a6c30d0ce516612fc461fdc29ba4117b5ee1abfce7552a6e& [07:40:06] [3/4] https://cdn.discordapp.com/attachments/808001911868489748/1465612314983141506/image.png?ex=6979bd55&is=69786bd5&hm=15f43410048a2ee17ff34cf76942196a3daa0089c24664e2a776a75fbeeb49b6& [07:40:07] [4/4] https://cdn.discordapp.com/attachments/808001911868489748/1465612315557630084/image.png?ex=6979bd55&is=69786bd5&hm=5e2e59c19f8542a270452a0122780df8465f69fda952bc28027ac59ca6fcd94e& [07:43:59] PROBLEM - puppet181 APT on puppet181 is CRITICAL: APT CRITICAL: 7 packages available for upgrade (4 critical updates). [07:46:51] PROBLEM - os201 APT on os201 is CRITICAL: APT CRITICAL: 4 packages available for upgrade (4 critical updates). [07:49:47] PROBLEM - os151 APT on os151 is CRITICAL: APT CRITICAL: 4 packages available for upgrade (4 critical updates). [07:55:46] Again!? [07:58:49] !log [universalomega@puppet181] Upgraded packages on graylog161: openjdk-21-jdk, openjdk-21-jdk-headless, openjdk-21-jre, and openjdk-21-jre-headless [07:58:50] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [07:59:40] RECOVERY - graylog161 APT on graylog161 is OK: APT OK: 1 packages available for upgrade (0 critical updates). [07:59:47] !log [universalomega@puppet181] Upgraded packages on kafka182: openjdk-21-jre and openjdk-21-jre-headless [07:59:48] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [08:01:01] RECOVERY - kafka182 APT on kafka182 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [08:03:32] !log [universalomega@puppet181] Upgraded packages on os161: openjdk-21-jdk, openjdk-21-jdk-headless, openjdk-21-jre, and openjdk-21-jre-headless [08:03:34] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [08:04:24] !log [universalomega@puppet181] Upgraded packages on os151: openjdk-21-jdk, openjdk-21-jdk-headless, openjdk-21-jre, and openjdk-21-jre-headless [08:04:25] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [08:04:41] !log [universalomega@puppet181] Upgraded packages on puppet181: openjdk-21-jdk, openjdk-21-jdk-headless, openjdk-21-jre, and openjdk-21-jre-headless [08:04:43] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [08:05:07] !log [universalomega@puppet181] Upgraded packages on os201: openjdk-21-jdk, openjdk-21-jdk-headless, openjdk-21-jre, and openjdk-21-jre-headless [08:05:09] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [08:05:31] RECOVERY - os161 APT on os161 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [08:05:35] !log [universalomega@puppet181] Upgraded packages on os191: openjdk-21-jdk, openjdk-21-jdk-headless, openjdk-21-jre, and openjdk-21-jre-headless [08:05:36] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [08:05:45] RECOVERY - os151 APT on os151 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [08:05:52] !log [universalomega@puppet181] Upgraded packages on os162: openjdk-21-jdk, openjdk-21-jdk-headless, openjdk-21-jre, and openjdk-21-jre-headless [08:05:52] RECOVERY - os191 APT on os191 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [08:05:53] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [08:05:58] RECOVERY - puppet181 APT on puppet181 is OK: APT OK: 3 packages available for upgrade (0 critical updates). [08:07:02] RECOVERY - os201 APT on os201 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [08:07:04] RECOVERY - os162 APT on os162 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [08:07:23] PROBLEM - mattermost2 Puppet on mattermost2 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [08:07:31] !log [universalomega@puppet181] Upgraded packages on os202: openjdk-21-jdk, openjdk-21-jdk-headless, openjdk-21-jre, and openjdk-21-jre-headless [08:07:32] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [08:08:11] PROBLEM - matomo151 Puppet on matomo151 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [08:08:20] RECOVERY - os202 APT on os202 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [08:08:50] PROBLEM - prometheus151 Puppet on prometheus151 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [08:08:56] PROBLEM - swiftproxy161 Puppet on swiftproxy161 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [08:09:11] PROBLEM - mem201 Puppet on mem201 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [08:09:36] PROBLEM - cp201 Puppet on cp201 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [08:09:41] PROBLEM - changeprop202 Puppet on changeprop202 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [08:10:16] PROBLEM - db172 Puppet on db172 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [08:10:53] PROBLEM - mw161 Puppet on mw161 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [08:10:55] PROBLEM - mw202 Puppet on mw202 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [08:11:24] PROBLEM - swiftproxy171 Puppet on swiftproxy171 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [08:11:50] PROBLEM - rdb191 Puppet on rdb191 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [08:11:53] PROBLEM - mw162 Puppet on mw162 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [08:11:56] PROBLEM - bast161 Puppet on bast161 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [08:12:14] PROBLEM - mw172 Puppet on mw172 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [08:12:47] PROBLEM - mw203 Puppet on mw203 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [08:13:00] PROBLEM - llm191 Puppet on llm191 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [08:13:11] PROBLEM - mwtask171 Puppet on mwtask171 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [08:13:21] PROBLEM - cp171 Puppet on cp171 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [08:13:28] PROBLEM - mwtask161 Puppet on mwtask161 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [08:13:32] PROBLEM - puppet181 Puppet on puppet181 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [08:13:51] PROBLEM - mem161 Puppet on mem161 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [08:14:11] PROBLEM - swiftobject191 Puppet on swiftobject191 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [08:14:47] PROBLEM - swiftobject201 Puppet on swiftobject201 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [08:15:02] PROBLEM - mwtask151 Puppet on mwtask151 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [08:15:08] PROBLEM - phorge171 Puppet on phorge171 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [08:15:50] PROBLEM - swiftobject181 Puppet on swiftobject181 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [08:15:52] PROBLEM - swiftac171 Puppet on swiftac171 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [08:15:58] PROBLEM - mw173 Puppet on mw173 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [08:16:47] PROBLEM - db161 Puppet on db161 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [08:17:38] PROBLEM - cloud16 Puppet on cloud16 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [08:18:23] PROBLEM - os151 Puppet on os151 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [08:18:23] I think that will self resolve [08:18:25] PROBLEM - ns1 Puppet on ns1 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [08:18:44] PROBLEM - mem151 Puppet on mem151 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [08:18:53] PROBLEM - cloud15 Puppet on cloud15 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [08:19:00] PROBLEM - os201 Puppet on os201 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [08:19:25] PROBLEM - mw193 Puppet on mw193 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [08:21:03] PROBLEM - mw171 Puppet on mw171 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [08:21:08] PROBLEM - db171 Puppet on db171 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [08:22:36] PROBLEM - mw191 Puppet on mw191 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [08:22:36] PROBLEM - mw153 Puppet on mw153 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [08:22:43] PROBLEM - cloud17 Puppet on cloud17 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [08:22:48] PROBLEM - db182 Puppet on db182 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [08:22:53] PROBLEM - db201 Puppet on db201 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [08:23:24] PROBLEM - kafka182 Puppet on kafka182 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [08:24:50] PROBLEM - cloud19 Puppet on cloud19 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [08:24:54] PROBLEM - cloud18 Puppet on cloud18 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [08:26:05] PROBLEM - bots171 Puppet on bots171 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [08:26:14] PROBLEM - cp191 Puppet on cp191 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [08:27:15] PROBLEM - mw192 Puppet on mw192 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [08:27:39] PROBLEM - cp161 Puppet on cp161 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [08:27:52] PROBLEM - swiftobject171 Puppet on swiftobject171 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [08:28:50] PROBLEM - graylog161 Puppet on graylog161 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [08:28:56] PROBLEM - mw163 Puppet on mw163 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [08:29:09] PROBLEM - mem191 Puppet on mem191 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [08:29:35] PROBLEM - os162 Puppet on os162 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [08:30:23] PROBLEM - mw151 Puppet on mw151 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [08:30:48] PROBLEM - os202 Puppet on os202 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [08:31:34] PROBLEM - db181 Puppet on db181 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [08:32:39] PROBLEM - reports171 Puppet on reports171 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [08:32:51] PROBLEM - ns2 Puppet on ns2 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [08:32:53] PROBLEM - os161 Puppet on os161 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [08:33:17] PROBLEM - os191 Puppet on os191 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [08:33:35] PROBLEM - mw181 Puppet on mw181 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [08:33:36] PROBLEM - test151 Puppet on test151 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [08:34:02] PROBLEM - db151 Puppet on db151 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [08:34:14] PROBLEM - swiftobject151 Puppet on swiftobject151 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [08:34:22] PROBLEM - eventgate182 Puppet on eventgate182 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [08:34:26] PROBLEM - mw201 Puppet on mw201 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [08:34:28] PROBLEM - cloud20 Puppet on cloud20 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [08:34:40] PROBLEM - swiftobject161 Puppet on swiftobject161 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [08:34:44] PROBLEM - mwtask181 Puppet on mwtask181 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [08:35:57] PROBLEM - mw183 Puppet on mw183 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [08:36:16] PROBLEM - db192 Puppet on db192 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [08:36:20] PROBLEM - bast181 Puppet on bast181 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [08:36:22] PROBLEM - mon181 Puppet on mon181 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [08:36:28] PROBLEM - mw152 Puppet on mw152 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [08:37:13] PROBLEM - mw182 Puppet on mw182 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [08:37:19] PROBLEM - ldap171 Puppet on ldap171 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [08:43:02] PROBLEM - mw173 HTTPS on mw173 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10005 milliseconds with 0 bytes received [08:43:27] PROBLEM - cp161 Varnish Backends on cp161 is CRITICAL: 1 backends are down. mw191 [08:43:33] PROBLEM - mw163 HTTPS on mw163 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10005 milliseconds with 0 bytes received [08:43:49] PROBLEM - cp191 Varnish Backends on cp191 is CRITICAL: 3 backends are down. mw161 mw162 mw183 [08:43:55] PROBLEM - mw163 MediaWiki Rendering on mw163 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 8191 bytes in 0.010 second response time [08:44:15] PROBLEM - mw203 MediaWiki Rendering on mw203 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 8191 bytes in 0.010 second response time [08:44:16] PROBLEM - cp171 Varnish Backends on cp171 is CRITICAL: 8 backends are down. mw151 mw161 mw162 mw171 mw182 mw183 mw191 mw202 [08:44:23] PROBLEM - mw193 HTTPS on mw193 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10006 milliseconds with 0 bytes received [08:44:25] PROBLEM - mw181 MediaWiki Rendering on mw181 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:44:25] PROBLEM - cp161 HTTPS on cp161 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10006 milliseconds with 0 bytes received [08:44:33] PROBLEM - mw181 HTTPS on mw181 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10003 milliseconds with 0 bytes received [08:44:34] PROBLEM - cp161 HTTP 4xx/5xx ERROR Rate on cp161 is CRITICAL: CRITICAL - NGINX Error Rate is 61% [08:44:34] PROBLEM - mw152 MediaWiki Rendering on mw152 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:44:35] PROBLEM - cp191 HTTPS on cp191 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 502 [08:44:41] PROBLEM - mw201 HTTPS on mw201 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 502 [08:44:44] PROBLEM - mw173 MediaWiki Rendering on mw173 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 8191 bytes in 0.010 second response time [08:44:46] PROBLEM - cp201 Varnish Backends on cp201 is CRITICAL: 6 backends are down. mw151 mw171 mw182 mw191 mw192 mw202 [08:44:48] PROBLEM - mw153 MediaWiki Rendering on mw153 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 8191 bytes in 0.010 second response time [08:44:52] PROBLEM - mw203 HTTPS on mw203 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 502 [08:45:10] PROBLEM - mw152 HTTPS on mw152 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 502 [08:45:13] PROBLEM - mw172 HTTPS on mw172 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 502 [08:45:14] PROBLEM - mw172 MediaWiki Rendering on mw172 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 8191 bytes in 0.010 second response time [08:45:16] PROBLEM - mw201 MediaWiki Rendering on mw201 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 8191 bytes in 0.011 second response time [08:45:19] PROBLEM - mw193 MediaWiki Rendering on mw193 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:45:21] PROBLEM - cp171 HTTPS on cp171 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10003 milliseconds with 0 bytes received [08:45:38] PROBLEM - mw153 HTTPS on mw153 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10005 milliseconds with 0 bytes received [08:46:44] RECOVERY - cp191 HTTPS on cp191 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4303 bytes in 9.502 second response time [08:47:51] PROBLEM - puppet181 Check unit status of listdomains_github_push on puppet181 is CRITICAL: CRITICAL: Status of the systemd unit listdomains_github_push [08:49:08] RECOVERY - mw203 HTTPS on mw203 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4254 bytes in 8.694 second response time [08:49:20] RECOVERY - mw152 HTTPS on mw152 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4254 bytes in 5.640 second response time [08:49:20] RECOVERY - cp171 HTTPS on cp171 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4303 bytes in 9.493 second response time [08:50:12] RECOVERY - cp161 HTTPS on cp161 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4358 bytes in 3.145 second response time [08:50:13] RECOVERY - mw203 MediaWiki Rendering on mw203 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.855 second response time [08:50:19] RECOVERY - mw193 HTTPS on mw193 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4254 bytes in 0.061 second response time [08:50:21] RECOVERY - mw181 MediaWiki Rendering on mw181 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.784 second response time [08:50:27] RECOVERY - mw152 MediaWiki Rendering on mw152 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.184 second response time [08:50:28] RECOVERY - mw181 HTTPS on mw181 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4254 bytes in 0.052 second response time [08:50:34] PROBLEM - cp161 HTTP 4xx/5xx ERROR Rate on cp161 is WARNING: WARNING - NGINX Error Rate is 41% [08:50:42] RECOVERY - mw201 HTTPS on mw201 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4254 bytes in 0.063 second response time [08:50:46] RECOVERY - mw173 MediaWiki Rendering on mw173 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.175 second response time [08:50:46] RECOVERY - cp201 Varnish Backends on cp201 is OK: All 31 backends are healthy [08:50:48] RECOVERY - mw173 HTTPS on mw173 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4254 bytes in 0.061 second response time [08:50:54] RECOVERY - mw153 MediaWiki Rendering on mw153 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.172 second response time [08:51:11] RECOVERY - mw172 HTTPS on mw172 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4254 bytes in 0.058 second response time [08:51:12] RECOVERY - mw172 MediaWiki Rendering on mw172 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.170 second response time [08:51:19] RECOVERY - mw201 MediaWiki Rendering on mw201 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.187 second response time [08:51:27] RECOVERY - cp161 Varnish Backends on cp161 is OK: All 31 backends are healthy [08:51:28] RECOVERY - mw193 MediaWiki Rendering on mw193 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.192 second response time [08:51:38] RECOVERY - mw163 MediaWiki Rendering on mw163 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.169 second response time [08:51:48] RECOVERY - mw153 HTTPS on mw153 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4254 bytes in 0.053 second response time [08:51:49] RECOVERY - cp191 Varnish Backends on cp191 is OK: All 31 backends are healthy [08:51:49] RECOVERY - mw163 HTTPS on mw163 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4254 bytes in 0.075 second response time [08:52:16] RECOVERY - cp171 Varnish Backends on cp171 is OK: All 31 backends are healthy [08:52:34] RECOVERY - cp161 HTTP 4xx/5xx ERROR Rate on cp161 is OK: OK - NGINX Error Rate is 27% [08:55:51] RECOVERY - puppet181 Check unit status of listdomains_github_push on puppet181 is OK: OK: Status of the systemd unit listdomains_github_push [09:25:32] RECOVERY - puppet181 Puppet on puppet181 is OK: OK: Puppet is currently enabled, last run 17 seconds ago with 0 failures [09:25:51] RECOVERY - swiftobject171 Puppet on swiftobject171 is OK: OK: Puppet is currently enabled, last run 21 seconds ago with 0 failures [09:26:14] RECOVERY - cp191 Puppet on cp191 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [09:27:18] RECOVERY - mem191 Puppet on mem191 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [09:27:37] RECOVERY - cloud16 Puppet on cloud16 is OK: OK: Puppet is currently enabled, last run 3 seconds ago with 0 failures [09:27:39] RECOVERY - cp161 Puppet on cp161 is OK: OK: Puppet is currently enabled, last run 38 seconds ago with 0 failures [09:27:41] RECOVERY - changeprop202 Puppet on changeprop202 is OK: OK: Puppet is currently enabled, last run 7 seconds ago with 0 failures [09:27:52] RECOVERY - bast161 Puppet on bast161 is OK: OK: Puppet is currently enabled, last run 15 seconds ago with 0 failures [09:28:15] RECOVERY - db192 Puppet on db192 is OK: OK: Puppet is currently enabled, last run 36 seconds ago with 0 failures [09:28:16] RECOVERY - db172 Puppet on db172 is OK: OK: Puppet is currently enabled, last run 41 seconds ago with 0 failures [09:28:27] RECOVERY - cloud20 Puppet on cloud20 is OK: OK: Puppet is currently enabled, last run 52 seconds ago with 0 failures [09:28:29] RECOVERY - db151 Puppet on db151 is OK: OK: Puppet is currently enabled, last run 53 seconds ago with 0 failures [09:28:29] RECOVERY - eventgate182 Puppet on eventgate182 is OK: OK: Puppet is currently enabled, last run 54 seconds ago with 0 failures [09:28:32] RECOVERY - bots171 Puppet on bots171 is OK: OK: Puppet is currently enabled, last run 53 seconds ago with 0 failures [09:28:37] RECOVERY - bast181 Puppet on bast181 is OK: OK: Puppet is currently enabled, last run 57 seconds ago with 0 failures [09:28:40] RECOVERY - cloud17 Puppet on cloud17 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [09:28:46] RECOVERY - db161 Puppet on db161 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [09:28:48] RECOVERY - db182 Puppet on db182 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [09:28:50] RECOVERY - graylog161 Puppet on graylog161 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [09:28:50] RECOVERY - cloud19 Puppet on cloud19 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [09:28:52] RECOVERY - cloud18 Puppet on cloud18 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [09:28:53] RECOVERY - cloud15 Puppet on cloud15 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [09:29:00] RECOVERY - llm191 Puppet on llm191 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [09:29:08] RECOVERY - db171 Puppet on db171 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [09:29:16] RECOVERY - db201 Puppet on db201 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [09:29:18] !log reboot puppet181 [09:29:19] RECOVERY - ldap171 Puppet on ldap171 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [09:29:20] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [09:29:21] RECOVERY - cp171 Puppet on cp171 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [09:29:22] RECOVERY - kafka182 Puppet on kafka182 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [09:29:34] RECOVERY - db181 Puppet on db181 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [09:29:34] RECOVERY - os162 Puppet on os162 is OK: OK: Puppet is currently enabled, last run 20 seconds ago with 0 failures [09:29:34] RECOVERY - cp201 Puppet on cp201 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [09:29:51] RECOVERY - mem161 Puppet on mem161 is OK: OK: Puppet is currently enabled, last run 2 seconds ago with 0 failures [09:30:11]