[00:02:33] PROBLEM - cloud2 Current Load on cloud2 is CRITICAL: CRITICAL - load average: 38.40, 24.69, 18.93 [00:05:38] PROBLEM - mw7 APT on mw7 is CRITICAL: APT CRITICAL: 2 packages available for upgrade (1 critical updates). [00:06:24] PROBLEM - cloud2 Current Load on cloud2 is WARNING: WARNING - load average: 19.85, 23.66, 19.91 [00:18:24] RECOVERY - cloud2 Current Load on cloud2 is OK: OK - load average: 12.31, 17.65, 19.05 [00:52:37] PROBLEM - mw4 MediaWiki Rendering on mw4 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:52:42] PROBLEM - cp9 Stunnel Http for mw4 on cp9 is CRITICAL: HTTP CRITICAL - No data received from host [00:53:11] PROBLEM - cp3 Stunnel Http for mw4 on cp3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:53:30] PROBLEM - cp7 Stunnel Http for mw4 on cp7 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:53:43] PROBLEM - cp9 Varnish Backends on cp9 is CRITICAL: 1 backends are down. mw4 [00:53:46] PROBLEM - cp6 Stunnel Http for mw4 on cp6 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:54:11] PROBLEM - mw4 HTTPS on mw4 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:54:59] PROBLEM - cp3 Varnish Backends on cp3 is CRITICAL: 1 backends are down. mw4 [00:55:01] PROBLEM - cp7 Varnish Backends on cp7 is CRITICAL: 1 backends are down. mw4 [00:55:21] PROBLEM - cp6 Varnish Backends on cp6 is CRITICAL: 1 backends are down. mw4 [01:00:09] PROBLEM - mw6 Puppet on mw6 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki core] [01:00:26] PROBLEM - cloud2 Current Load on cloud2 is WARNING: WARNING - load average: 23.42, 18.96, 16.24 [01:06:25] RECOVERY - cloud2 Current Load on cloud2 is OK: OK - load average: 18.78, 19.78, 17.55 [01:10:28] PROBLEM - cloud2 Current Load on cloud2 is WARNING: WARNING - load average: 20.31, 20.73, 18.44 [01:12:26] RECOVERY - cloud2 Current Load on cloud2 is OK: OK - load average: 14.67, 19.32, 18.24 [01:13:51] PROBLEM - mw6 MediaWiki Rendering on mw6 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:14:01] PROBLEM - cp6 Stunnel Http for mw6 on cp6 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [01:14:15] PROBLEM - cp7 Stunnel Http for mw6 on cp7 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [01:14:30] PROBLEM - cp9 Stunnel Http for mw6 on cp9 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [01:14:50] PROBLEM - cp3 Stunnel Http for mw6 on cp3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [01:15:41] PROBLEM - mw6 HTTPS on mw6 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:48:03] PROBLEM - mw4 Puppet on mw4 is WARNING: WARNING: Puppet last ran 1 hour ago [02:17:31] PROBLEM - cp6 Current Load on cp6 is CRITICAL: CRITICAL - load average: 4.99, 11.83, 5.95 [02:19:27] PROBLEM - cp6 Current Load on cp6 is WARNING: WARNING - load average: 0.74, 7.94, 5.23 [02:21:28] RECOVERY - cp6 Current Load on cp6 is OK: OK - load average: 0.27, 5.38, 4.62 [03:08:17] PROBLEM - cloud2 APT on cloud2 is CRITICAL: APT CRITICAL: 84 packages available for upgrade (1 critical updates). [03:38:17] PROBLEM - cloud1 APT on cloud1 is CRITICAL: APT CRITICAL: 84 packages available for upgrade (1 critical updates). [03:43:10] PROBLEM - cloud3 APT on cloud3 is CRITICAL: APT CRITICAL: 47 packages available for upgrade (1 critical updates). [04:15:23] PROBLEM - jobrunner1 APT on jobrunner1 is CRITICAL: APT CRITICAL: 2 packages available for upgrade (1 critical updates). [05:05:49] [02mw-config] 07Reception123 closed pull request 03#3233: Add OrphanedTalkPages extension (T6149) - 13https://git.io/JUGdz [05:05:51] [02miraheze/mw-config] 07Reception123 pushed 031 commit to 03master [+0/-0/±4] 13https://git.io/JUnm1 [05:05:52] [02miraheze/mw-config] 07Universal-Omega 03d3b72b3 - Add OrphanedTalkPages extension (T6149) (#3233) * Add OrphanedTalkPages extension (T6149) * Add OrphanedTalkPages extension (T6149) * Add OrphanedTalkPages extension (T6149) * Add OrphanedTalkPages extension (T6149) [05:14:10] [02mw-config] 07Universal-Omega synchronize pull request 03#3237: Add more Cosmos configs into a new Cosmos tab - 13https://git.io/JUZx8 [05:20:35] PROBLEM - test2 APT on test2 is CRITICAL: APT CRITICAL: 2 packages available for upgrade (1 critical updates). [05:23:21] !log sudo -u www-data php /srv/mediawiki/w/maintenance/mergeMessageFileList.php --output /srv/mediawiki/config/ExtensionMessageFiles.php --wiki loginwiki on mw* and jbr* [05:23:25] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [05:23:39] !log sudo -u www-data php /srv/mediawiki/w/maintenance/rebuildLocalisationCache.php --wiki loginwiki on mw* and jbr* [05:23:45] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [05:25:16] [02mw-config] 07Universal-Omega edited pull request 03#3237: Add more Cosmos configs - 13https://git.io/JUZx8 [05:26:13] [02mw-config] 07Universal-Omega edited pull request 03#3237: Add more Cosmos configs - 13https://git.io/JUZx8 [05:29:40] [02mw-config] 07Reception123 reviewed pull request 03#3237 commit - 13https://git.io/JUnY0 [05:32:26] PROBLEM - cloud2 Current Load on cloud2 is CRITICAL: CRITICAL - load average: 24.88, 21.07, 17.96 [05:34:48] [02mw-config] 07Universal-Omega reviewed pull request 03#3237 commit - 13https://git.io/JUnYi [05:34:59] PROBLEM - mw7 Current Load on mw7 is WARNING: WARNING - load average: 7.04, 6.70, 5.73 [05:36:27] PROBLEM - cloud2 Current Load on cloud2 is WARNING: WARNING - load average: 22.45, 22.30, 19.26 [05:36:56] RECOVERY - mw7 Current Load on mw7 is OK: OK - load average: 6.77, 6.73, 5.86 [05:37:33] [02mw-config] 07Universal-Omega reviewed pull request 03#3237 commit - 13https://git.io/JUnYi [05:38:25] RECOVERY - cloud2 Current Load on cloud2 is OK: OK - load average: 16.50, 20.09, 18.81 [05:38:37] [02mw-config] 07Reception123 closed pull request 03#3237: Add more Cosmos configs - 13https://git.io/JUZx8 [05:38:39] [02miraheze/mw-config] 07Reception123 pushed 031 commit to 03master [+0/-0/±2] 13https://git.io/JUnYS [05:38:40] [02miraheze/mw-config] 07Universal-Omega 0359e30dd - Add more Cosmos configs (#3237) * Add more Cosmos configs; move Cosmos configs to own Cosmos skin tab * fix config name * Add Cosmos configs * Fix config * revert adding new cosmos tag, moving them to styling tab [05:39:37] PROBLEM - mw5 Puppet on mw5 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki core] [05:40:20] PROBLEM - jobrunner2 Puppet on jobrunner2 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki core] [05:41:24] PROBLEM - mw7 Puppet on mw7 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki core] [05:43:57] [02mw-config] 07Reception123 closed pull request 03#3235: extra settings for marionetworkwiki per req - 13https://git.io/JUZuO [05:43:58] [02miraheze/mw-config] 07Reception123 pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JUnYp [05:44:00] [02miraheze/mw-config] 07Reception123 03519bafb - extra settings for marionetworkwiki per req (#3235) [05:44:02] [02mw-config] 07Reception123 deleted branch 03Reception123-patch-2 - 13https://git.io/vbvb3 [05:44:03] [02miraheze/mw-config] 07Reception123 deleted branch 03Reception123-patch-2 [05:45:46] RECOVERY - mw5 Puppet on mw5 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [05:46:15] RECOVERY - jobrunner2 Puppet on jobrunner2 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [05:47:20] RECOVERY - mw7 Puppet on mw7 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [06:02:31] PROBLEM - jobrunner2 APT on jobrunner2 is CRITICAL: APT CRITICAL: 2 packages available for upgrade (1 critical updates). [06:13:46] RECOVERY - cp3 APT on cp3 is OK: APT OK: 1 packages available for upgrade (0 critical updates). [06:21:18] RECOVERY - jobrunner1 APT on jobrunner1 is OK: APT OK: 1 packages available for upgrade (0 critical updates). [06:21:40] RECOVERY - mw7 APT on mw7 is OK: APT OK: 1 packages available for upgrade (0 critical updates). [06:22:18] RECOVERY - cloud1 APT on cloud1 is OK: APT OK: 83 packages available for upgrade (0 critical updates). [06:32:26] RECOVERY - cloud3 APT on cloud3 is OK: APT OK: 46 packages available for upgrade (0 critical updates). [06:36:31] RECOVERY - cloud2 APT on cloud2 is OK: APT OK: 83 packages available for upgrade (0 critical updates). [06:41:52] RECOVERY - cp9 APT on cp9 is OK: APT OK: 1 packages available for upgrade (0 critical updates). [06:52:32] RECOVERY - test2 APT on test2 is OK: APT OK: 1 packages available for upgrade (0 critical updates). [06:58:32] RECOVERY - jobrunner2 APT on jobrunner2 is OK: APT OK: 1 packages available for upgrade (0 critical updates). [08:16:49] [02mw-config] 07JohnFLewis commented on pull request 03#3237: Add more Cosmos configs - 13https://git.io/JUncA [08:30:17] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JUnCo [08:30:19] [02miraheze/services] 07MirahezeSSLBot 037084a2d - BOT: Updating services config for wikis [08:31:20] [02mw-config] 07RhinosF1 commented on pull request 03#3221: Fix recently introduced configs - 13https://git.io/JUnCX [08:33:15] [02mw-config] 07JohnFLewis commented on pull request 03#3221: Fix recently introduced configs - 13https://git.io/JUnCQ [08:34:01] @Reception123 ^ [08:34:41] No access atm but did see [08:35:17] @Reception123: okay well if you have access before 10pm you'll beat me to it [08:35:48] Will not be the case [08:35:56] So I won't :) [08:43:22] looks at @paladox  [11:31:26] [02mw-config] 07Universal-Omega commented on pull request 03#3221: Fix recently introduced configs - 13https://git.io/JUnzu [11:31:37] [02mw-config] 07Universal-Omega commented on pull request 03#3237: Add more Cosmos configs - 13https://git.io/JUnzz [11:41:14] PROBLEM - mw5 APT on mw5 is CRITICAL: APT CRITICAL: 2 packages available for upgrade (1 critical updates). [12:00:54] [02mw-config] 07Universal-Omega opened pull request 03#3238: Fixing configs for Cosmos to match ManageWikiSettings - 13https://git.io/JUnga [12:02:24] [02mw-config] 07JohnFLewis closed pull request 03#3238: Fixing configs for Cosmos to match ManageWikiSettings - 13https://git.io/JUnga [12:02:26] [02miraheze/mw-config] 07JohnFLewis pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JUngM [12:02:27] [02miraheze/mw-config] 07Universal-Omega 03a2c1934 - Fixing configs per conversation with John (#3238) [12:37:12] PROBLEM - wiki.ricochetuniverse.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.ricochetuniverse.com' expires in 15 day(s) (Thu 24 Sep 2020 12:28:26 GMT +0000). [12:37:56] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JUna9 [12:37:58] [02miraheze/ssl] 07MirahezeSSLBot 03db6f76c - Bot: Update SSL cert for wiki.ricochetuniverse.com [12:42:45] [02miraheze/mediawiki] 07paladox pushed 031 commit to 03REL1_34 [+0/-0/±1] 13https://git.io/JUnVf [12:42:47] [02miraheze/mediawiki] 07paladox 036f07cbe - Update Cosmos [12:43:52] RECOVERY - wiki.ricochetuniverse.com - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.ricochetuniverse.com' will expire on Mon 07 Dec 2020 11:37:47 GMT +0000. [12:47:52] PROBLEM - mw4 Puppet on mw4 is CRITICAL: CRITICAL: Puppet last ran 12 hours ago [13:20:34] !log update puppetserver, puppetdb and puppet-agent on puppet2 [13:20:40] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [13:22:49] PROBLEM - gluster1 Puppet on gluster1 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [13:22:56] PROBLEM - cp3 Puppet on cp3 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [13:23:00] PROBLEM - db7 Puppet on db7 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [13:23:07] PROBLEM - mw7 Puppet on mw7 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [13:23:07] PROBLEM - mw5 Puppet on mw5 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [13:23:13] !log update libzmq5 puppet-agent on mw4 [13:23:18] PROBLEM - phab1 Puppet on phab1 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [13:23:29] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [13:24:00] PROBLEM - puppet2 Puppet on puppet2 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [13:24:02] PROBLEM - cp6 Puppet on cp6 is CRITICAL: CRITICAL: Puppet has 292 failures. Last run 2 minutes ago with 292 failures. Failed resources (up to 3 shown) [13:24:12] PROBLEM - jobrunner2 Puppet on jobrunner2 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [13:24:28] PROBLEM - mon1 Puppet on mon1 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [13:24:45] PROBLEM - gluster2 Puppet on gluster2 is CRITICAL: CRITICAL: Puppet has 7 failures. Last run 3 minutes ago with 7 failures. Failed resources (up to 3 shown) [13:25:19] PROBLEM - cp9 Puppet on cp9 is CRITICAL: CRITICAL: Puppet has 379 failures. Last run 2 minutes ago with 379 failures. Failed resources (up to 3 shown): File[lodge.jsnydr.com],File[lodge.jsnydr.com_private],File[wiki.worlduniversityandschool.org],File[wiki.worlduniversityandschool.org_private] [13:25:22] PROBLEM - cp7 Puppet on cp7 is CRITICAL: CRITICAL: Puppet has 300 failures. Last run 3 minutes ago with 300 failures. Failed resources (up to 3 shown) [13:25:53] PROBLEM - mw4 APT on mw4 is CRITICAL: APT CRITICAL: 2 packages available for upgrade (1 critical updates). [13:26:36] !log reboot mw4 - seems to have been stuck for 12h? [13:26:42] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [13:27:24] PROBLEM - jobrunner1 Puppet on jobrunner1 is CRITICAL: CRITICAL: Puppet has 357 failures. Last run 2 minutes ago with 357 failures. Failed resources (up to 3 shown): File[www.modesofdiscourse.com_private],File[wiki.pablojprieto.com.es],File[wiki.pablojprieto.com.es_private],File[wiki.nowchess.org] [13:28:48] PROBLEM - mw4 SSH on mw4 is CRITICAL: connect to address 51.89.160.128 and port 22: Connection refused [13:29:31] PROBLEM - mw4 php-fpm on mw4 is CRITICAL: connect to address 51.89.160.128 port 5666: Connection refusedconnect to host 51.89.160.128 port 5666: Connection refused [13:30:22] PROBLEM - mw4 Current Load on mw4 is CRITICAL: connect to address 51.89.160.128 port 5666: Connection refusedconnect to host 51.89.160.128 port 5666: Connection refused [13:30:40] PROBLEM - mw4 NTP time on mw4 is CRITICAL: connect to address 51.89.160.128 port 5666: Connection refusedconnect to host 51.89.160.128 port 5666: Connection refused [13:30:42] PROBLEM - mw4 Disk Space on mw4 is CRITICAL: connect to address 51.89.160.128 port 5666: Connection refusedconnect to host 51.89.160.128 port 5666: Connection refused [13:33:11] RECOVERY - db7 Puppet on db7 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [13:34:11] RECOVERY - puppet2 Puppet on puppet2 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [13:34:34] PROBLEM - ping4 on mw4 is CRITICAL: PING CRITICAL - Packet loss = 100% [13:35:02] PROBLEM - mail1 Puppet on mail1 is CRITICAL: CRITICAL: Puppet has 7 failures. Last run 3 minutes ago with 7 failures. Failed resources (up to 3 shown) [13:35:06] PROBLEM - ping6 on mw4 is CRITICAL: PING CRITICAL - Packet loss = 100% [13:35:39] PROBLEM - services2 Puppet on services2 is CRITICAL: CRITICAL: Puppet has 5 failures. Last run 3 minutes ago with 5 failures. Failed resources (up to 3 shown) [13:35:40] PROBLEM - misc1 Puppet on misc1 is CRITICAL: CRITICAL: Puppet has 3 failures. Last run 3 minutes ago with 3 failures. Failed resources (up to 3 shown) [13:35:47] RECOVERY - mw4 MediaWiki Rendering on mw4 is OK: HTTP OK: HTTP/1.1 200 OK - 17161 bytes in 4.550 second response time [13:35:57] PROBLEM - services1 Puppet on services1 is CRITICAL: CRITICAL: Puppet has 5 failures. Last run 3 minutes ago with 5 failures. Failed resources (up to 3 shown) [13:35:58] RECOVERY - mw4 HTTPS on mw4 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 557 bytes in 0.005 second response time [13:36:02] RECOVERY - mw4 php-fpm on mw4 is OK: PROCS OK: 27 processes with command name 'php-fpm7.3' [13:36:19] RECOVERY - mw4 APT on mw4 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [13:36:25] PROBLEM - cloud2 Puppet on cloud2 is CRITICAL: CRITICAL: Puppet has 5 failures. Last run 3 minutes ago with 5 failures. Failed resources (up to 3 shown) [13:36:32] RECOVERY - ping4 on mw4 is OK: PING OK - Packet loss = 0%, RTA = 0.22 ms [13:36:33] RECOVERY - cp3 Stunnel Http for mw4 on cp3 is OK: HTTP OK: HTTP/1.1 200 OK - 15653 bytes in 1.365 second response time [13:36:41] RECOVERY - cp7 Stunnel Http for mw4 on cp7 is OK: HTTP OK: HTTP/1.1 200 OK - 15653 bytes in 0.007 second response time [13:36:45] PROBLEM - bacula2 Puppet on bacula2 is CRITICAL: CRITICAL: Puppet has 3 failures. Last run 4 minutes ago with 3 failures. Failed resources (up to 3 shown) [13:37:01] RECOVERY - mw4 Current Load on mw4 is OK: OK - load average: 3.19, 0.91, 0.31 [13:37:07] RECOVERY - mw4 NTP time on mw4 is OK: NTP OK: Offset -0.002775758505 secs [13:37:08] RECOVERY - ping6 on mw4 is OK: PING OK - Packet loss = 0%, RTA = 1.63 ms [13:37:08] RECOVERY - cp6 Stunnel Http for mw4 on cp6 is OK: HTTP OK: HTTP/1.1 200 OK - 15653 bytes in 0.013 second response time [13:37:14] RECOVERY - mw4 Disk Space on mw4 is OK: DISK OK - free space: / 5945 MB (31% inode=66%); [13:37:29] RECOVERY - mw4 SSH on mw4 is OK: SSH OK - OpenSSH_7.9p1 Debian-10+deb10u2 (protocol 2.0) [13:37:30] RECOVERY - cp9 Stunnel Http for mw4 on cp9 is OK: HTTP OK: HTTP/1.1 200 OK - 15653 bytes in 0.337 second response time [13:42:22] RECOVERY - cloud2 Puppet on cloud2 is OK: OK: Puppet is currently enabled, last run 9 seconds ago with 0 failures [13:42:37] RECOVERY - mon1 Puppet on mon1 is OK: OK: Puppet is currently enabled, last run 13 seconds ago with 0 failures [13:42:41] RECOVERY - bacula2 Puppet on bacula2 is OK: OK: Puppet is currently enabled, last run 29 seconds ago with 0 failures [13:42:45] RECOVERY - mail1 Puppet on mail1 is OK: OK: Puppet is currently enabled, last run 27 seconds ago with 0 failures [13:43:06] RECOVERY - gluster2 Puppet on gluster2 is OK: OK: Puppet is currently enabled, last run 35 seconds ago with 0 failures [13:43:08] RECOVERY - gluster1 Puppet on gluster1 is OK: OK: Puppet is currently enabled, last run 47 seconds ago with 0 failures [13:43:21] RECOVERY - cp7 Puppet on cp7 is OK: OK: Puppet is currently enabled, last run 28 seconds ago with 0 failures [13:43:27] RECOVERY - misc1 Puppet on misc1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [13:43:28] RECOVERY - services2 Puppet on services2 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [13:43:42] RECOVERY - phab1 Puppet on phab1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [13:43:55] RECOVERY - services1 Puppet on services1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [13:43:59] RECOVERY - cp9 Puppet on cp9 is OK: OK: Puppet is currently enabled, last run 42 seconds ago with 0 failures [13:44:04] RECOVERY - cp6 Puppet on cp6 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [13:45:20] RECOVERY - cp3 Puppet on cp3 is OK: OK: Puppet is currently enabled, last run 22 seconds ago with 0 failures [13:45:52] RECOVERY - mw5 Puppet on mw5 is OK: OK: Puppet is currently enabled, last run 3 seconds ago with 0 failures [13:45:57] RECOVERY - jobrunner1 Puppet on jobrunner1 is OK: OK: Puppet is currently enabled, last run 18 seconds ago with 0 failures [13:46:33] RECOVERY - jobrunner2 Puppet on jobrunner2 is OK: OK: Puppet is currently enabled, last run 51 seconds ago with 0 failures [13:47:22] RECOVERY - mw7 Puppet on mw7 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [13:50:11] PROBLEM - db7 Check MariaDB Replication on db7 is CRITICAL: MariaDB replication - both - CRITICAL - Slave_IO_Running state : Yes, Slave_SQL_Running state : Yes, Seconds_Behind_Master : 110s [13:52:08] PROBLEM - db7 Check MariaDB Replication on db7 is WARNING: MariaDB replication - both - WARNING - Slave_IO_Running state : Yes, Slave_SQL_Running state : Yes, Seconds_Behind_Master : 15s [13:54:08] PROBLEM - db7 Check MariaDB Replication on db7 is CRITICAL: MariaDB replication - both - CRITICAL - Slave_IO_Running state : Yes, Slave_SQL_Running state : Yes, Seconds_Behind_Master : 135s [13:55:49] !log reboot mw6 - seems stuck [13:55:52] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [13:56:08] RECOVERY - db7 Check MariaDB Replication on db7 is OK: MariaDB replication - both - OK - Slave_IO_Running state : Yes, Slave_SQL_Running state : Yes, Seconds_Behind_Master : 2s [13:56:42] RECOVERY - mw4 Puppet on mw4 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [13:59:03] PROBLEM - mw6 SSH on mw6 is CRITICAL: connect to address 51.89.160.136 and port 22: Connection refused [13:59:04] PROBLEM - mw6 php-fpm on mw6 is CRITICAL: connect to address 51.89.160.136 port 5666: Connection refusedconnect to host 51.89.160.136 port 5666: Connection refused [13:59:17] PROBLEM - mw6 NTP time on mw6 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [13:59:19] PROBLEM - mw6 Current Load on mw6 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [14:00:04] PROBLEM - mw6 Disk Space on mw6 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [14:00:07] PROBLEM - mw6 APT on mw6 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [14:00:12] !log update libzmq5 puppet-agent on mw5 [14:00:21] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [14:00:49] !log update puppet-agent on mw5 [14:00:53] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [14:02:03] PROBLEM - ping6 on mw6 is CRITICAL: CRITICAL - Destination Unreachable (2001:41d0:800:105a::4) [14:02:43] RECOVERY - mw5 APT on mw5 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [14:03:19] RECOVERY - mw6 SSH on mw6 is OK: SSH OK - OpenSSH_7.9p1 Debian-10+deb10u2 (protocol 2.0) [14:03:23] RECOVERY - cp3 Stunnel Http for mw6 on cp3 is OK: HTTP OK: HTTP/1.1 200 OK - 15653 bytes in 1.340 second response time [14:03:30] RECOVERY - mw6 php-fpm on mw6 is OK: PROCS OK: 27 processes with command name 'php-fpm7.3' [14:03:36] RECOVERY - mw6 Current Load on mw6 is OK: OK - load average: 2.42, 0.70, 0.24 [14:03:39] RECOVERY - mw6 NTP time on mw6 is OK: NTP OK: Offset -0.002932399511 secs [14:03:41] RECOVERY - cp6 Stunnel Http for mw6 on cp6 is OK: HTTP OK: HTTP/1.1 200 OK - 15653 bytes in 0.003 second response time [14:03:42] RECOVERY - cp7 Stunnel Http for mw6 on cp7 is OK: HTTP OK: HTTP/1.1 200 OK - 15653 bytes in 0.003 second response time [14:03:48] PROBLEM - mw7 Puppet on mw7 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [14:04:01] RECOVERY - ping6 on mw6 is OK: PING OK - Packet loss = 0%, RTA = 0.23 ms [14:04:13] [02miraheze/dns] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JUnKw [14:04:14] [02miraheze/dns] 07paladox 036ecbec3 - Update orain.org [14:04:15] PROBLEM - cloud2 Current Load on cloud2 is CRITICAL: CRITICAL - load average: 23.11, 24.08, 19.81 [14:04:20] PROBLEM - mw5 Puppet on mw5 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [14:04:25] RECOVERY - mw6 Disk Space on mw6 is OK: DISK OK - free space: / 6175 MB (32% inode=66%); [14:04:54] RECOVERY - mw6 HTTPS on mw6 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 557 bytes in 0.111 second response time [14:05:04] RECOVERY - cp9 Stunnel Http for mw6 on cp9 is OK: HTTP OK: HTTP/1.1 200 OK - 15653 bytes in 0.306 second response time [14:05:09] RECOVERY - cp7 Varnish Backends on cp7 is OK: All 7 backends are healthy [14:05:31] RECOVERY - cp6 Varnish Backends on cp6 is OK: All 7 backends are healthy [14:05:45] RECOVERY - mw6 MediaWiki Rendering on mw6 is OK: HTTP OK: HTTP/1.1 200 OK - 17160 bytes in 0.250 second response time [14:05:52] RECOVERY - cp9 Varnish Backends on cp9 is OK: All 7 backends are healthy [14:05:58] RECOVERY - cp3 Varnish Backends on cp3 is OK: All 7 backends are healthy [14:06:08] [02miraheze/ssl] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JUnKX [14:06:09] [02miraheze/ssl] 07paladox 03a7c31f5 - Update orain.org.crt [14:07:12] [02miraheze/dns] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JUnKD [14:07:13] [02miraheze/dns] 07paladox 0337f998a - Update miraheze.com [14:07:40] PROBLEM - cp7 Current Load on cp7 is WARNING: WARNING - load average: 6.13, 7.88, 5.26 [14:08:11] [02miraheze/dns] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JUnKQ [14:08:13] [02miraheze/dns] 07paladox 034bc0b77 - Update miraheze.com [14:09:03] [02miraheze/ssl] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JUnKF [14:09:05] [02miraheze/ssl] 07paladox 03fe044f5 - Update miraheze.com.crt [14:09:37] RECOVERY - cp7 Current Load on cp7 is OK: OK - load average: 2.27, 5.82, 4.81 [14:11:57] PROBLEM - cloud2 Current Load on cloud2 is WARNING: WARNING - load average: 18.75, 23.12, 22.06 [14:12:30] [02miraheze/ssl] 07paladox pushed 031 commit to 03paladox-patch-3 [+0/-0/±1] 13https://git.io/JUn6t [14:12:32] [02miraheze/ssl] 07paladox 0382ad408 - Switch hololive.wiki to a LetsEncrypt certificate Bug: T6058 [14:12:33] [02ssl] 07paladox created branch 03paladox-patch-3 - 13https://git.io/vxP9L [14:12:43] RECOVERY - orain.org - LetsEncrypt on sslhost is OK: OK - Certificate 'orain.org' will expire on Mon 07 Dec 2020 13:05:45 GMT +0000. [14:12:46] [02ssl] 07paladox opened pull request 03#357: Switch hololive.wiki to a LetsEncrypt certificate - 13https://git.io/JUn6Y [14:13:13] [02miraheze/ssl] 07paladox pushed 031 commit to 03paladox-patch-3 [+0/-0/±1] 13https://git.io/JUn6s [14:13:15] [02miraheze/ssl] 07paladox 0381873ed - Update hololive.wiki.crt [14:13:16] [02ssl] 07paladox synchronize pull request 03#357: Switch hololive.wiki to a LetsEncrypt certificate - 13https://git.io/JUn6Y [14:13:44] [02ssl] 07paladox closed pull request 03#357: Switch hololive.wiki to a LetsEncrypt certificate - 13https://git.io/JUn6Y [14:13:46] [02miraheze/ssl] 07paladox pushed 031 commit to 03master [+0/-0/±2] 13https://git.io/JUn6Z [14:13:47] [02miraheze/ssl] 07paladox 03c8d1b52 - Switch hololive.wiki to a LetsEncrypt certificate (#357) * Switch hololive.wiki to a LetsEncrypt certificate Bug: T6058 * Update hololive.wiki.crt [14:13:49] [02miraheze/ssl] 07paladox deleted branch 03paladox-patch-3 [14:13:51] PROBLEM - cloud2 Current Load on cloud2 is CRITICAL: CRITICAL - load average: 24.04, 23.87, 22.45 [14:14:04] [02ssl] 07paladox deleted branch 03paladox-patch-3 - 13https://git.io/vxP9L [14:14:33] !log update libzmq5 puppet-agent on mw6 [14:14:37] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [14:15:08] !log update puppet-agent on test2 [14:15:12] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [14:16:04] !log update puppet-agent on jobrunner2 [14:16:08] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [14:16:15] RECOVERY - poserdazfreebies.orain.org - LetsEncrypt on sslhost is OK: OK - Certificate 'orain.org' will expire on Mon 07 Dec 2020 13:05:45 GMT +0000. [14:16:22] RECOVERY - mw5 Puppet on mw5 is OK: OK: Puppet is currently enabled, last run 53 seconds ago with 0 failures [14:16:38] RECOVERY - mw6 APT on mw6 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [14:16:45] RECOVERY - allthetropes.orain.org - LetsEncrypt on sslhost is OK: OK - Certificate 'orain.org' will expire on Mon 07 Dec 2020 13:05:45 GMT +0000. [14:16:54] !log update puppet-agent on jobrunner1 [14:16:57] RECOVERY - meta.orain.org - LetsEncrypt on sslhost is OK: OK - Certificate 'orain.org' will expire on Mon 07 Dec 2020 13:05:45 GMT +0000. [14:17:03] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [14:17:41] PROBLEM - cp6 Current Load on cp6 is WARNING: WARNING - load average: 7.28, 6.97, 3.38 [14:18:10] [02miraheze/dns] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JUn6V [14:18:12] [02miraheze/dns] 07paladox 0350d1105 - Update orain.org [14:19:38] RECOVERY - cp6 Current Load on cp6 is OK: OK - load average: 1.93, 5.05, 3.09 [14:19:42] PROBLEM - cloud2 Current Load on cloud2 is WARNING: WARNING - load average: 15.43, 23.95, 23.22 [14:19:51] [02miraheze/ssl] 07paladox pushed 031 commit to 03paladox-patch-3 [+0/-0/±1] 13https://git.io/JUn6X [14:19:53] [02miraheze/ssl] 07paladox 03173e4f9 - Enable event for some Miraheze domains We don't use wildcard certs for some Miraheze domains so we can enable the events. [14:19:54] [02ssl] 07paladox created branch 03paladox-patch-3 - 13https://git.io/vxP9L [14:19:56] [02ssl] 07paladox opened pull request 03#358: Enable event for some Miraheze domains - 13https://git.io/JUn61 [14:20:07] [02ssl] 07paladox closed pull request 03#358: Enable event for some Miraheze domains - 13https://git.io/JUn61 [14:20:09] [02miraheze/ssl] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JUn6M [14:20:10] [02miraheze/ssl] 07paladox 03e31eecf - Enable event for some Miraheze domains (#358) We don't use wildcard certs for some Miraheze domains so we can enable the events. [14:20:12] [02miraheze/ssl] 07paladox deleted branch 03paladox-patch-3 [14:20:13] [02ssl] 07paladox deleted branch 03paladox-patch-3 - 13https://git.io/vxP9L [14:21:39] PROBLEM - cloud2 Current Load on cloud2 is CRITICAL: CRITICAL - load average: 27.14, 25.22, 23.74 [14:27:12] RECOVERY - mw7 Puppet on mw7 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [14:27:17] RECOVERY - miraheze.com - LetsEncrypt on sslhost is OK: OK - Certificate 'miraheze.com' will expire on Mon 07 Dec 2020 13:08:38 GMT +0000. [14:27:30] PROBLEM - cloud2 Current Load on cloud2 is WARNING: WARNING - load average: 18.55, 23.34, 23.68 [14:27:37] RECOVERY - mw6 Puppet on mw6 is OK: OK: Puppet is currently enabled, last run 55 seconds ago with 0 failures [14:29:14] [02miraheze/ssl] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JUniT [14:29:15] [02miraheze/ssl] 07paladox 03f6deda3 - Update miraheze.tk.crt [14:29:28] PROBLEM - cloud2 Current Load on cloud2 is CRITICAL: CRITICAL - load average: 24.58, 23.43, 23.62 [14:29:45] [02miraheze/ssl] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JUnik [14:29:47] [02miraheze/ssl] 07paladox 03902af99 - Update miraheze.ml.crt [14:30:15] [02miraheze/ssl] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JUniI [14:30:16] [02miraheze/ssl] 07paladox 03b88ee35 - Update miraheze.ga.crt [14:30:38] [02miraheze/ssl] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JUniq [14:30:39] [02miraheze/ssl] 07paladox 035e643c4 - Update miraheze.gq.crt [14:31:25] PROBLEM - cloud2 Current Load on cloud2 is WARNING: WARNING - load average: 23.38, 23.00, 23.42 [14:32:12] [02miraheze/ssl] 07paladox pushed 031 commit to 03paladox-patch-3 [+0/-0/±1] 13https://git.io/JUnis [14:32:14] [02miraheze/ssl] 07paladox 03c3374df - Remove redirect for www.shkilev.online It's not pointing at us thus does not work. [14:32:15] [02ssl] 07paladox created branch 03paladox-patch-3 - 13https://git.io/vxP9L [14:32:17] [02mw-config] 07Universal-Omega opened pull request 03#3239: Re-add Extension:WikiHiero (T6048) - 13https://git.io/JUniG [14:32:18] [02ssl] 07paladox opened pull request 03#359: Remove redirect for www.shkilev.online - 13https://git.io/JUniZ [14:32:48] [02ssl] 07paladox closed pull request 03#359: Remove redirect for www.shkilev.online - 13https://git.io/JUniZ [14:32:49] [02miraheze/ssl] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JUnic [14:32:51] [02miraheze/ssl] 07paladox 035270bda - Remove redirect for www.shkilev.online (#359) It's not pointing at us thus does not work. [14:32:52] [02ssl] 07paladox deleted branch 03paladox-patch-3 - 13https://git.io/vxP9L [14:32:54] [02miraheze/ssl] 07paladox deleted branch 03paladox-patch-3 [14:34:05] [02miraheze/ssl] 07paladox pushed 031 commit to 03paladox-patch-3 [+0/-0/±1] 13https://git.io/JUnil [14:34:06] [02miraheze/ssl] 07paladox 039b6c880 - Remove wiki.archman.de No longer pointing at us [14:34:08] [02ssl] 07paladox created branch 03paladox-patch-3 - 13https://git.io/vxP9L [14:34:09] [02ssl] 07paladox opened pull request 03#360: Remove wiki.archman.de - 13https://git.io/JUni8 [14:34:17] [02ssl] 07paladox closed pull request 03#360: Remove wiki.archman.de - 13https://git.io/JUni8 [14:34:18] [02miraheze/ssl] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JUniB [14:34:20] [02miraheze/ssl] 07paladox 0320b14a5 - Remove wiki.archman.de (#360) No longer pointing at us [14:34:21] [02miraheze/ssl] 07paladox deleted branch 03paladox-patch-3 [14:34:23] [02ssl] 07paladox deleted branch 03paladox-patch-3 - 13https://git.io/vxP9L [14:34:28] [02miraheze/ssl] 07paladox pushed 031 commit to 03master [+0/-1/±0] 13https://git.io/JUniR [14:34:29] [02miraheze/ssl] 07paladox 034c68151 - Delete wiki.archman.de.crt [14:39:36] PROBLEM - wiki.erisly.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.erisly.com' expires in 15 day(s) (Thu 24 Sep 2020 14:31:48 GMT +0000). [14:40:46] PROBLEM - ao90.pinho.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'ao90.pinho.org' expires in 15 day(s) (Thu 24 Sep 2020 14:35:22 GMT +0000). [14:42:04] RECOVERY - miraheze.ml - LetsEncrypt on sslhost is OK: OK - Certificate 'miraheze.ml' will expire on Mon 07 Dec 2020 13:29:25 GMT +0000. [14:43:54] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JUnPk [14:43:55] [02miraheze/ssl] 07MirahezeSSLBot 03357c3aa - Bot: Update SSL cert for wiki.erisly.com [14:44:48] RECOVERY - miraheze.tk - LetsEncrypt on sslhost is OK: OK - Certificate 'miraheze.tk' will expire on Mon 07 Dec 2020 13:28:46 GMT +0000. [14:45:48] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JUnPY [14:45:50] [02miraheze/ssl] 07MirahezeSSLBot 039d5e25e - Bot: Update SSL cert for ao90.pinho.org [14:46:32] RECOVERY - miraheze.gq - LetsEncrypt on sslhost is OK: OK - Certificate 'miraheze.gq' will expire on Mon 07 Dec 2020 13:30:26 GMT +0000. [14:48:14] RECOVERY - miraheze.ga - LetsEncrypt on sslhost is OK: OK - Certificate 'miraheze.ga' will expire on Mon 07 Dec 2020 13:29:57 GMT +0000. [14:51:04] RECOVERY - cloud2 Current Load on cloud2 is OK: OK - load average: 18.90, 18.67, 20.26 [14:53:04] RECOVERY - wiki.erisly.com - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.erisly.com' will expire on Mon 07 Dec 2020 13:43:47 GMT +0000. [14:54:36] RECOVERY - ao90.pinho.org - LetsEncrypt on sslhost is OK: OK - Certificate 'ao90.pinho.org' will expire on Mon 07 Dec 2020 13:45:41 GMT +0000. [14:55:46] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JUnPD [14:55:47] [02miraheze/puppet] 07paladox 0352c6b18 - Update mount.pp [15:06:53] PROBLEM - mw5 Puppet on mw5 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 1 minute ago with 1 failures. Failed resources (up to 3 shown): Mount[/mnt/mediawiki-static] [15:08:29] PROBLEM - mw6 Puppet on mw6 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Mount[/mnt/mediawiki-static] [15:08:39] PROBLEM - mw4 Puppet on mw4 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Mount[/mnt/mediawiki-static] [15:12:54] PROBLEM - cloud2 Current Load on cloud2 is WARNING: WARNING - load average: 20.74, 20.71, 19.48 [15:16:11] RECOVERY - mw6 Puppet on mw6 is OK: OK: Puppet is currently enabled, last run 48 seconds ago with 0 failures [15:16:14] !log depool mw4 and repool (reboot) [15:16:19] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [15:16:37] RECOVERY - mw5 Puppet on mw5 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [15:16:38] RECOVERY - mw4 Puppet on mw4 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [15:18:53] PROBLEM - cloud2 Current Load on cloud2 is CRITICAL: CRITICAL - load average: 31.12, 23.33, 20.72 [15:19:20] !log depool mw5 and repool (reboot) [15:22:55] PROBLEM - cp6 Varnish Backends on cp6 is CRITICAL: 1 backends are down. mw5 [15:22:59] PROBLEM - cp7 Varnish Backends on cp7 is CRITICAL: 1 backends are down. mw5 [15:23:05] PROBLEM - cp9 Varnish Backends on cp9 is CRITICAL: 1 backends are down. mw5 [15:23:14] PROBLEM - cp3 Varnish Backends on cp3 is CRITICAL: 1 backends are down. mw5 [15:24:23] !log depool mw6 and repool (reboot) [15:24:54] RECOVERY - cp6 Varnish Backends on cp6 is OK: All 7 backends are healthy [15:24:54] RECOVERY - cp7 Varnish Backends on cp7 is OK: All 7 backends are healthy [15:25:02] RECOVERY - cp9 Varnish Backends on cp9 is OK: All 7 backends are healthy [15:25:13] RECOVERY - cp3 Varnish Backends on cp3 is OK: All 7 backends are healthy [15:28:48] PROBLEM - cp7 Varnish Backends on cp7 is CRITICAL: 1 backends are down. mw6 [15:28:51] PROBLEM - cp6 Varnish Backends on cp6 is CRITICAL: 1 backends are down. mw6 [15:28:56] PROBLEM - cp9 Varnish Backends on cp9 is CRITICAL: 1 backends are down. mw6 [15:29:13] PROBLEM - cp3 Varnish Backends on cp3 is CRITICAL: 1 backends are down. mw6 [15:34:05] PROBLEM - gluster1 Current Load on gluster1 is CRITICAL: CRITICAL - load average: 10.46, 7.89, 6.53 [15:36:38] RECOVERY - cp7 Varnish Backends on cp7 is OK: All 7 backends are healthy [15:36:44] PROBLEM - mw6 Puppet on mw6 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki core] [15:36:45] RECOVERY - cp9 Varnish Backends on cp9 is OK: All 7 backends are healthy [15:36:50] RECOVERY - cp6 Varnish Backends on cp6 is OK: All 7 backends are healthy [15:37:07] RECOVERY - cp3 Varnish Backends on cp3 is OK: All 7 backends are healthy [15:38:01] PROBLEM - gluster1 Current Load on gluster1 is WARNING: WARNING - load average: 6.25, 7.60, 6.76 [15:41:59] RECOVERY - gluster1 Current Load on gluster1 is OK: OK - load average: 5.87, 6.43, 6.46 [15:45:47] PROBLEM - mw7 Puppet on mw7 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki core] [15:46:20] RECOVERY - mw6 Puppet on mw6 is OK: OK: Puppet is currently enabled, last run 16 seconds ago with 0 failures [15:52:38] PROBLEM - cloud2 Current Load on cloud2 is WARNING: WARNING - load average: 20.73, 21.07, 23.92 [15:57:46] RECOVERY - mw7 Puppet on mw7 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [15:59:27] [02miraheze/mediawiki] 07paladox pushed 031 commit to 03REL1_34 [+0/-0/±1] 13https://git.io/JUnDk [15:59:28] [02miraheze/mediawiki] 07paladox 039075466 - Update Cosmos [16:01:38] !log gluster volume set mvol features.cache-invalidation-timeout 600 (set earlier) [16:01:44] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:12:35] PROBLEM - cloud2 Current Load on cloud2 is CRITICAL: CRITICAL - load average: 24.68, 21.22, 21.34 [16:18:34] PROBLEM - gluster1 Current Load on gluster1 is WARNING: WARNING - load average: 7.84, 6.58, 5.63 [16:20:33] RECOVERY - gluster1 Current Load on gluster1 is OK: OK - load average: 6.48, 6.68, 5.80 [16:21:12] PROBLEM - wiki.newfil.com.au - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.newfil.com.au could not be found [16:21:12] PROBLEM - www.sdiy.info - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for www.sdiy.info could not be found [16:21:13] PROBLEM - wiki.mxlinuxusers.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.mxlinuxusers.org could not be found [16:21:26] PROBLEM - wiki.cyberepo.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.cyberepo.org could not be found [16:24:31] PROBLEM - cloud2 Current Load on cloud2 is WARNING: WARNING - load average: 20.44, 23.32, 23.70 [16:27:57] RECOVERY - www.sdiy.info - reverse DNS on sslhost is OK: rDNS OK - www.sdiy.info reverse DNS resolves to cp7.miraheze.org [16:28:02] RECOVERY - wiki.newfil.com.au - reverse DNS on sslhost is OK: rDNS OK - wiki.newfil.com.au reverse DNS resolves to cp7.miraheze.org [16:28:08] RECOVERY - wiki.mxlinuxusers.org - reverse DNS on sslhost is OK: rDNS OK - wiki.mxlinuxusers.org reverse DNS resolves to cp7.miraheze.org [16:28:25] RECOVERY - wiki.cyberepo.org - reverse DNS on sslhost is OK: rDNS OK - wiki.cyberepo.org reverse DNS resolves to cp7.miraheze.org [16:30:27] PROBLEM - cloud2 Current Load on cloud2 is CRITICAL: CRITICAL - load average: 27.35, 22.13, 22.93 [16:32:25] PROBLEM - cloud2 Current Load on cloud2 is WARNING: WARNING - load average: 15.88, 19.56, 21.90 [16:38:20] PROBLEM - cloud2 Current Load on cloud2 is CRITICAL: CRITICAL - load average: 42.12, 26.74, 23.69 [16:42:17] PROBLEM - cloud2 Current Load on cloud2 is WARNING: WARNING - load average: 20.24, 23.98, 23.28 [16:44:12] @Doug I created a redirect that allows for lowercase p in policy now. [16:46:13] PROBLEM - cloud2 Current Load on cloud2 is CRITICAL: CRITICAL - load average: 31.75, 27.00, 24.48 [16:48:12] PROBLEM - cloud2 Current Load on cloud2 is WARNING: WARNING - load average: 16.75, 23.42, 23.50 [16:51:20] !log MariaDB [(none)]> DROP DATABASE miiwiki; [16:51:24] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:51:50] !log (on db12) [16:51:54] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:56:11] PROBLEM - cloud2 Current Load on cloud2 is CRITICAL: CRITICAL - load average: 37.81, 26.15, 23.87 [16:58:13] PROBLEM - cloud2 Current Load on cloud2 is WARNING: WARNING - load average: 17.64, 22.28, 22.73 [17:03:06] [02miraheze/ssl] 07Pix1234 pushed 031 commit to 03master [+0/-1/±1] 13https://git.io/JUnSu [17:03:08] [02miraheze/ssl] 07Pix1234 03083cbde - Remove wiki.arcolinuxclub.org as it no longer points to us (T6090) [17:10:09] RECOVERY - cloud2 Current Load on cloud2 is OK: OK - load average: 14.03, 17.63, 20.12 [17:17:35] [02miraheze/ssl] 07Pix1234 pushed 031 commit to 03master [+1/-0/±1] 13https://git.io/JUn9J [17:17:36] [02miraheze/ssl] 07Pix1234 03c16f0cc - Add privacy-wiki.0x.no (T6153) [17:23:19] PROBLEM - cloud2 Current Load on cloud2 is WARNING: WARNING - load average: 22.78, 22.92, 21.10 [17:23:38] PROBLEM - chouverse.cludisciples.cyou - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for chouverse.cludisciples.cyou could not be found [17:25:00] [02mw-config] 07Reception123 commented on pull request 03#3239: Re-add Extension:WikiHiero (T6048) - 13https://git.io/JUn9B [17:25:03] [02mw-config] 07Reception123 closed pull request 03#3239: Re-add Extension:WikiHiero (T6048) - 13https://git.io/JUniG [17:25:05] [02miraheze/mw-config] 07Reception123 pushed 031 commit to 03master [+0/-0/±2] 13https://git.io/JUn9R [17:25:06] [02miraheze/mw-config] 07Universal-Omega 030db8a1f - Re-add Extension:WikiHiero (T6048) (#3239) * Re-add Extension:WikiHiero (T6048) * Re-add Extension:WikiHiero (T6048) * Re-add Extension:WikiHiero (T6048) * Re-add Extension:WikiHiero (T6048) * Re-add Extension:WikiHiero (T6048) * Re-add Extension:WikiHiero (T6048) [17:25:11] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JUn9E [17:25:13] [02miraheze/services] 07MirahezeSSLBot 030555766 - BOT: Updating services config for wikis [17:25:19] PROBLEM - cloud2 Current Load on cloud2 is CRITICAL: CRITICAL - load average: 26.08, 23.27, 21.39 [17:30:33] RECOVERY - chouverse.cludisciples.cyou - reverse DNS on sslhost is OK: rDNS OK - chouverse.cludisciples.cyou reverse DNS resolves to cp7.miraheze.org [17:32:39] !log sudo -u www-data php /srv/mediawiki/w/maintenance/mergeMessageFileList.php --output /srv/mediawiki/config/ExtensionMessageFiles.php --wiki loginwiki on mw* and jbr* [17:32:46] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:32:54] PROBLEM - cp7 Current Load on cp7 is CRITICAL: CRITICAL - load average: 3.99, 8.33, 5.63 [17:33:15] [02mw-config] 07Pix1234 deleted branch 03RhinosF1-patch-1 - 13https://git.io/vbvb3 [17:33:16] [02miraheze/mw-config] 07Pix1234 deleted branch 03RhinosF1-patch-1 [17:33:18] [02mw-config] 07Pix1234 deleted branch 03Reception123-patch-1 - 13https://git.io/vbvb3 [17:33:19] [02miraheze/mw-config] 07Pix1234 deleted branch 03Reception123-patch-1 [17:33:23] [02ssl] 07paladox closed pull request 03#356: Remove museummiddelland.nl redirect - 13https://git.io/JJND1 [17:33:25] [02miraheze/ssl] 07paladox deleted branch 03paladox-patch-2 [17:33:26] [02ssl] 07paladox deleted branch 03paladox-patch-2 - 13https://git.io/vxP9L [17:33:44] !log sudo -u www-data php /srv/mediawiki/w/maintenance/rebuildLocalisationCache.php --wiki loginwiki on mw* and jbr* [17:33:48] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:34:54] RECOVERY - cp7 Current Load on cp7 is OK: OK - load average: 2.79, 6.64, 5.33 [17:39:18] PROBLEM - cloud2 Current Load on cloud2 is WARNING: WARNING - load average: 15.84, 22.89, 23.88 [17:41:19] PROBLEM - cloud2 Current Load on cloud2 is CRITICAL: CRITICAL - load average: 24.35, 22.85, 23.70 [17:51:12] PROBLEM - mw4 Puppet on mw4 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki core] [17:51:47] PROBLEM - mw7 Puppet on mw7 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki core] [17:51:55] [02puppet] 07paladox closed pull request 03#1489: add vimeo to CSP per T6129 - 13https://git.io/JUO42 [17:51:57] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JUnH0 [17:51:58] [02miraheze/puppet] 07Reception123 03a96c0fe - add vimeo to CSP per T6129 (#1489) second-largest video sharing service (after YouTube) [17:52:00] [02puppet] 07paladox deleted branch 03Reception123-patch-1 - 13https://git.io/vbiAS [17:52:01] [02miraheze/puppet] 07paladox deleted branch 03Reception123-patch-1 [17:52:43] PROBLEM - mw6 Puppet on mw6 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki core] [17:53:36] [02puppet] 07Pix1234 commented on pull request 03#1490: fix project name to "acl*security" from "Security" in README.md - 13https://git.io/JUnHg [17:56:57] [02miraheze/RottenLinks] 07JohnFLewis pushed 031 commit to 03master [+1/-0/±3] 13https://git.io/JUnH1 [17:56:58] [02miraheze/RottenLinks] 07JohnFLewis 0358c7cf9 - T6095: Add primary key to RottenLinks [17:57:00] RECOVERY - mw4 Puppet on mw4 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [17:57:42] RECOVERY - mw7 Puppet on mw7 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [17:58:03] PROBLEM - test2 Current Load on test2 is CRITICAL: CRITICAL - load average: 5.27, 3.39, 1.62 [17:58:34] RECOVERY - mw6 Puppet on mw6 is OK: OK: Puppet is currently enabled, last run 41 seconds ago with 0 failures [17:59:48] when the https://cdn.discordapp.com/attachments/334296645833326604/752914983158808620/image0.gif [17:59:59] BleepoBleapo: can we help you? [18:00:04] RECOVERY - test2 Current Load on test2 is OK: OK - load average: 0.84, 2.35, 1.45 [18:00:12] no thank you [18:03:37] [02puppet] 07Universal-Omega commented on pull request 03#1490: fix project name to "acl*security" from "Security" in README.md - 13https://git.io/JUnH7 [18:03:38] [02puppet] 07Universal-Omega closed pull request 03#1490: fix project name to "acl*security" from "Security" in README.md - 13https://git.io/JUssk [18:04:24] PROBLEM - gluster1 Current Load on gluster1 is CRITICAL: CRITICAL - load average: 10.46, 7.89, 6.36 [18:05:19] PROBLEM - wiki.climatechange.ai - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.climatechange.ai could not be found [18:05:30] PROBLEM - www.dariawiki.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for www.dariawiki.org could not be found [18:05:30] PROBLEM - eerstelijnszones.be - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for eerstelijnszones.be could not be found [18:10:24] PROBLEM - gluster1 Current Load on gluster1 is WARNING: WARNING - load average: 6.20, 7.51, 6.81 [18:12:08] RECOVERY - wiki.climatechange.ai - reverse DNS on sslhost is OK: rDNS OK - wiki.climatechange.ai reverse DNS resolves to cp6.miraheze.org [18:12:11] RECOVERY - www.dariawiki.org - reverse DNS on sslhost is OK: rDNS OK - www.dariawiki.org reverse DNS resolves to cp6.miraheze.org [18:12:13] RECOVERY - eerstelijnszones.be - reverse DNS on sslhost is OK: rDNS OK - eerstelijnszones.be reverse DNS resolves to cp7.miraheze.org [18:16:24] PROBLEM - gluster1 Current Load on gluster1 is CRITICAL: CRITICAL - load average: 10.37, 8.11, 7.18 [18:17:35] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 3 datacenters are down: 2400:6180:0:d0::403:f001/cpweb, 51.222.27.129/cpweb, 2607:5300:205:200::2ac4/cpweb [18:19:30] [02puppet] 07RhinosF1 commented on pull request 03#1490: fix project name to "acl*security" from "Security" in README.md - 13https://git.io/JUnQR [18:19:34] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [18:20:25] PROBLEM - gluster1 Current Load on gluster1 is WARNING: WARNING - load average: 5.57, 7.27, 7.14 [18:22:45] [02mediawiki] 07Universal-Omega closed pull request 03#152: Updating Cosmos - 13https://git.io/JUZbo [18:23:28] [02CreateWiki] 07Universal-Omega closed pull request 03#159: fix project name to "acl*security" from "Security" in README.md - 13https://git.io/JUs3d [18:23:42] [02miraheze/mediawiki] 07JohnFLewis pushed 031 commit to 03REL1_34 [+0/-0/±1] 13https://git.io/JUnQi [18:23:43] [02miraheze/mediawiki] 07JohnFLewis 0343a0c72 - update RL [18:24:29] [02ManageWiki] 07JohnFLewis closed pull request 03#179: Add $wgRevokePermissions to ManageWiki (T5991) - 13https://git.io/JULOg [18:24:34] [02miraheze/mw-config] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/JUnQX [18:24:35] [02miraheze/mw-config] 07paladox 038350805 - DataDump: Exclude some directories when zipping the images [18:24:37] [02mw-config] 07paladox created branch 03paladox-patch-1 - 13https://git.io/vbvb3 [18:24:38] [02mw-config] 07paladox opened pull request 03#3240: DataDump: Exclude some directories when zipping the images - 13https://git.io/JUnQ1 [18:26:09] miraheze/mw-config/paladox-patch-1/8350805 - paladox The build passed. https://travis-ci.org/miraheze/mw-config/builds/725336962 [18:28:25] PROBLEM - gluster1 Current Load on gluster1 is CRITICAL: CRITICAL - load average: 12.22, 8.59, 7.52 [18:34:25] PROBLEM - gluster1 Current Load on gluster1 is WARNING: WARNING - load average: 6.40, 7.49, 7.41 [18:35:55] !log root@gluster1:/home/paladox# gluster volume set mvol network.inode-lru-limit 5000 [18:36:04] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:36:24] PROBLEM - gluster1 Current Load on gluster1 is CRITICAL: CRITICAL - load average: 10.09, 8.71, 7.88 [18:40:24] PROBLEM - gluster1 Current Load on gluster1 is WARNING: WARNING - load average: 4.55, 7.41, 7.64 [18:41:18] PROBLEM - cloud2 Current Load on cloud2 is WARNING: WARNING - load average: 18.58, 21.08, 23.82 [18:44:37] [02mw-config] 07paladox synchronize pull request 03#3240: DataDump: Exclude some directories when zipping the images - 13https://git.io/JUnQ1 [18:44:39] [02miraheze/mw-config] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/JUn7o [18:44:40] [02miraheze/mw-config] 07paladox 03fd984f2 - Update LocalExtensions.php [18:45:19] PROBLEM - cloud2 Current Load on cloud2 is CRITICAL: CRITICAL - load average: 24.05, 22.23, 23.61 [18:46:24] PROBLEM - gluster1 Current Load on gluster1 is CRITICAL: CRITICAL - load average: 9.75, 7.49, 7.45 [18:47:18] PROBLEM - cloud2 Current Load on cloud2 is WARNING: WARNING - load average: 20.25, 22.14, 23.46 [18:48:26] PROBLEM - gluster1 Current Load on gluster1 is WARNING: WARNING - load average: 6.57, 7.19, 7.36 [18:52:24] PROBLEM - gluster1 Current Load on gluster1 is CRITICAL: CRITICAL - load average: 9.17, 7.71, 7.49 [18:52:36] [02miraheze/RottenLinks] 07JohnFLewis pushed 031 commit to 03master [+0/-0/±3] 13https://git.io/JUn7F [18:52:37] [02miraheze/RottenLinks] 07JohnFLewis 032964f28 - Increase key length + update extension.json [18:52:57] [02miraheze/mediawiki] 07JohnFLewis pushed 031 commit to 03REL1_34 [+0/-0/±1] 13https://git.io/JUn7N [18:52:59] [02miraheze/mediawiki] 07JohnFLewis 0321f5fd1 - update RL [18:53:22] PROBLEM - cloud2 Current Load on cloud2 is CRITICAL: CRITICAL - load average: 34.03, 24.79, 23.70 [18:57:18] PROBLEM - cloud2 Current Load on cloud2 is WARNING: WARNING - load average: 18.41, 23.00, 23.37 [18:59:23] PROBLEM - cloud2 Current Load on cloud2 is CRITICAL: CRITICAL - load average: 38.60, 29.50, 25.74 [19:00:54] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 8 datacenters are down: 128.199.139.216/cpweb, 2400:6180:0:d0::403:f001/cpweb, 51.77.107.210/cpweb, 51.89.160.142/cpweb, 2001:41d0:800:1056::2/cpweb, 2001:41d0:800:105a::10/cpweb, 51.222.27.129/cpweb, 2607:5300:205:200::2ac4/cpweb [19:01:16] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 5 datacenters are down: 128.199.139.216/cpweb, 2400:6180:0:d0::403:f001/cpweb, 51.89.160.142/cpweb, 51.222.27.129/cpweb, 2607:5300:205:200::2ac4/cpweb [19:01:21] PROBLEM - mw4 Current Load on mw4 is WARNING: WARNING - load average: 7.43, 6.05, 4.36 [19:01:31] PROBLEM - mw7 Current Load on mw7 is CRITICAL: CRITICAL - load average: 8.09, 7.10, 4.82 [19:01:32] PROBLEM - mw5 Current Load on mw5 is WARNING: WARNING - load average: 7.84, 6.90, 4.75 [19:02:53] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [19:03:16] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [19:03:21] RECOVERY - mw4 Current Load on mw4 is OK: OK - load average: 5.12, 5.82, 4.49 [19:03:26] RECOVERY - mw7 Current Load on mw7 is OK: OK - load average: 4.59, 6.12, 4.73 [19:03:30] RECOVERY - mw5 Current Load on mw5 is OK: OK - load average: 4.14, 5.83, 4.61 [19:07:51] PROBLEM - jobrunner1 Puppet on jobrunner1 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki core] [19:09:42] PROBLEM - mw7 Puppet on mw7 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki core] [19:10:15] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JUn5V [19:10:16] [02miraheze/services] 07MirahezeSSLBot 03637351a - BOT: Updating services config for wikis [19:10:33] PROBLEM - mw6 Puppet on mw6 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki core] [19:10:46] [02mw-config] 07paladox closed pull request 03#3240: DataDump: Exclude some directories when zipping the images - 13https://git.io/JUnQ1 [19:10:47] [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JUn5K [19:10:49] [02miraheze/mw-config] 07paladox 0343ace79 - DataDump: Exclude some directories when zipping the images (#3240) * DataDump: Exclude some directories when zipping the images * Update LocalExtensions.php [19:10:50] [02mw-config] 07paladox deleted branch 03paladox-patch-1 - 13https://git.io/vbvb3 [19:10:52] [02miraheze/mw-config] 07paladox deleted branch 03paladox-patch-1 [19:15:42] RECOVERY - mw7 Puppet on mw7 is OK: OK: Puppet is currently enabled, last run 8 seconds ago with 0 failures [19:16:26] RECOVERY - mw6 Puppet on mw6 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [19:38:26] PROBLEM - gluster1 Current Load on gluster1 is WARNING: WARNING - load average: 7.06, 7.36, 7.94 [19:39:20] PROBLEM - cloud2 Current Load on cloud2 is WARNING: WARNING - load average: 17.74, 20.80, 23.40 [19:48:24] RECOVERY - gluster1 Current Load on gluster1 is OK: OK - load average: 4.59, 5.41, 6.74 [19:51:18] RECOVERY - cloud2 Current Load on cloud2 is OK: OK - load average: 14.89, 16.59, 20.10 [19:55:19] PROBLEM - cloud2 Current Load on cloud2 is WARNING: WARNING - load average: 21.37, 20.09, 20.80 [20:00:50] [02miraheze/mediawiki] 07paladox pushed 032 commits to 03REL1_34 [+0/-0/±2] 13https://git.io/JUnFn [20:00:52] [02miraheze/mediawiki] 07paladox 036de4422 - Update Cosmos [20:00:53] [02miraheze/mediawiki] 07paladox 031e9ed3e - Merge branch 'REL1_34' of github.com:miraheze/mediawiki into REL1_34 [20:01:18] RECOVERY - cloud2 Current Load on cloud2 is OK: OK - load average: 15.90, 18.36, 19.88 [20:06:51] [02miraheze/CreateWiki] 07JohnFLewis pushed 031 commit to 03master [+0/-0/±18] 13https://git.io/JUnFz [20:06:52] [02miraheze/CreateWiki] 07JohnFLewis 0398d4a86 - T4394: Convert to Config Object over globals [20:07:18] PROBLEM - cloud2 Current Load on cloud2 is WARNING: WARNING - load average: 21.83, 19.95, 20.07 [20:09:18] PROBLEM - cloud2 Current Load on cloud2 is CRITICAL: CRITICAL - load average: 26.59, 21.66, 20.65 [20:11:18] PROBLEM - cloud2 Current Load on cloud2 is WARNING: WARNING - load average: 20.41, 21.73, 20.86 [20:11:58] PROBLEM - gluster1 Current Load on gluster1 is WARNING: WARNING - load average: 7.39, 7.29, 6.60 [20:13:58] PROBLEM - gluster1 Current Load on gluster1 is CRITICAL: CRITICAL - load average: 10.51, 7.90, 6.86 [20:15:18] PROBLEM - cloud2 Current Load on cloud2 is CRITICAL: CRITICAL - load average: 24.87, 22.34, 21.17 [20:15:51] PROBLEM - gluster1 Current Load on gluster1 is WARNING: WARNING - load average: 6.45, 7.19, 6.71 [20:17:46] RECOVERY - gluster1 Current Load on gluster1 is OK: OK - load average: 5.36, 6.63, 6.57 [20:19:18] PROBLEM - cloud2 Current Load on cloud2 is WARNING: WARNING - load average: 17.28, 21.37, 21.17 [20:21:35] PROBLEM - gluster1 Current Load on gluster1 is WARNING: WARNING - load average: 7.27, 7.03, 6.72 [20:23:29] RECOVERY - gluster1 Current Load on gluster1 is OK: OK - load average: 4.73, 6.31, 6.50 [20:25:18] RECOVERY - cloud2 Current Load on cloud2 is OK: OK - load average: 15.36, 18.70, 20.30 [21:06:07] PROBLEM - cloud2 Current Load on cloud2 is WARNING: WARNING - load average: 21.19, 21.23, 19.51 [21:08:01] RECOVERY - cloud2 Current Load on cloud2 is OK: OK - load average: 16.22, 19.43, 19.05 [21:11:50] PROBLEM - cloud2 Current Load on cloud2 is WARNING: WARNING - load average: 22.27, 21.25, 19.83 [21:13:54] PROBLEM - bacula2 Bacula Static 1 on bacula2 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [21:15:52] PROBLEM - bacula2 Bacula Static 1 on bacula2 is WARNING: WARNING: Full, 11404431 files, 584.4GB, 2020-08-21 15:26:00 (2.6 weeks ago) [21:17:35] RECOVERY - cloud2 Current Load on cloud2 is OK: OK - load average: 12.71, 18.37, 19.31 [21:20:17] PROBLEM - bacula2 Bacula Databases db13 on bacula2 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [21:22:15] PROBLEM - bacula2 Bacula Databases db13 on bacula2 is WARNING: WARNING: Diff, 2796 files, 95.05GB, 2020-08-16 02:29:00 (3.4 weeks ago) [21:26:53] PROBLEM - bacula2 Bacula Static 1 on bacula2 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [21:28:53] PROBLEM - bacula2 Bacula Static 1 on bacula2 is WARNING: WARNING: Full, 11404431 files, 584.4GB, 2020-08-21 15:26:00 (2.6 weeks ago) [21:35:11] PROBLEM - mw7 Current Load on mw7 is WARNING: WARNING - load average: 7.20, 5.93, 4.34 [21:37:06] RECOVERY - mw7 Current Load on mw7 is OK: OK - load average: 5.29, 5.81, 4.49 [21:38:03] PROBLEM - nonbinary.wiki - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for nonbinary.wiki could not be found [21:38:03] PROBLEM - wiki.xfbs.net - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.xfbs.net could not be found [21:44:03] [02miraheze/mediawiki] 07paladox pushed 031 commit to 03REL1_34 [+0/-0/±1] 13https://git.io/JUnpj [21:44:05] [02miraheze/mediawiki] 07paladox 03d964e2c - Update Cosmos [21:44:52] RECOVERY - nonbinary.wiki - reverse DNS on sslhost is OK: rDNS OK - nonbinary.wiki reverse DNS resolves to cp7.miraheze.org [21:44:52] RECOVERY - wiki.xfbs.net - reverse DNS on sslhost is OK: rDNS OK - wiki.xfbs.net reverse DNS resolves to cp7.miraheze.org [21:47:19] PROBLEM - cloud2 Current Load on cloud2 is WARNING: WARNING - load average: 21.71, 20.79, 18.69 [21:49:18] RECOVERY - cloud2 Current Load on cloud2 is OK: OK - load average: 19.31, 19.89, 18.59 [21:49:43] PROBLEM - mw7 Puppet on mw7 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki core] [21:53:18] PROBLEM - cloud2 Current Load on cloud2 is CRITICAL: CRITICAL - load average: 30.92, 24.52, 20.64 [21:55:30] [02miraheze/DataDump] 07paladox pushed 031 commit to 03paladox-patch-2 [+0/-0/±1] 13https://git.io/JUnhc [21:55:31] [02miraheze/DataDump] 07paladox 0388e6f09 - Switch dumps_size to BIGINT [21:55:33] [02DataDump] 07paladox created branch 03paladox-patch-2 - 13https://git.io/fhhKV [21:55:34] [02DataDump] 07paladox opened pull request 03#7: Switch dumps_size to BIGINT - 13https://git.io/JUnhC [21:57:18] PROBLEM - cloud2 Current Load on cloud2 is WARNING: WARNING - load average: 18.06, 22.24, 20.74 [21:57:42] RECOVERY - mw7 Puppet on mw7 is OK: OK: Puppet is currently enabled, last run 24 seconds ago with 0 failures [22:00:04] [02miraheze/DataDump] 07paladox pushed 031 commit to 03paladox-patch-2 [+1/-0/±0] 13https://git.io/JUnhr [22:00:05] [02miraheze/DataDump] 07paladox 03787d863 - Create patch-dumps_size-bigint.sql [22:00:07] [02DataDump] 07paladox synchronize pull request 03#7: Switch dumps_size to BIGINT - 13https://git.io/JUnhC [22:00:28] PROBLEM - mw6 Puppet on mw6 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki core] [22:00:59] [02miraheze/DataDump] 07paladox pushed 031 commit to 03paladox-patch-2 [+0/-0/±1] 13https://github.com/miraheze/DataDump/compare/787d86316cc2...805527b70f94 [22:01:00] [ Comparing 787d86316cc2...805527b70f94 · miraheze/DataDump · GitHub ] - github.com [22:01:01] [02miraheze/DataDump] 07paladox 03805527b - Update DataDumpHooks.php [22:01:02] [02DataDump] 07paladox synchronize pull request 03#7: Switch dumps_size to BIGINT - 13https://git.io/JUnhC [22:01:04] [02miraheze/DataDump] 07paladox pushed 031 commit to 03paladox-patch-2 [+0/-0/±1] 13https://git.io/JUnhi [22:01:05] [02miraheze/DataDump] 07paladox 036d6afe6 - Update DataDumpHooks.php [22:01:07] [02DataDump] 07paladox synchronize pull request 03#7: Switch dumps_size to BIGINT - 13https://git.io/JUnhC [22:02:18] [02DataDump] 07paladox closed pull request 03#7: Switch dumps_size to BIGINT - 13https://git.io/JUnhC [22:02:19] [02miraheze/DataDump] 07paladox pushed 031 commit to 03master [+1/-0/±2] 13https://git.io/JUnhM [22:02:21] [02miraheze/DataDump] 07paladox 03cffa178 - Switch dumps_size to BIGINT (#7) * Switch dumps_size to BIGINT * Create patch-dumps_size-bigint.sql * Update DataDumpHooks.php * Update DataDumpHooks.php [22:02:22] [02miraheze/DataDump] 07paladox deleted branch 03paladox-patch-2 [22:02:24] [02DataDump] 07paladox deleted branch 03paladox-patch-2 - 13https://git.io/fhhKV [22:02:34] PROBLEM - mw6 Current Load on mw6 is WARNING: WARNING - load average: 7.45, 6.80, 5.75 [22:02:56] [02miraheze/mediawiki] 07paladox pushed 031 commit to 03REL1_34 [+0/-0/±1] 13https://git.io/JUnhD [22:02:57] [02miraheze/mediawiki] 07paladox 03352469c - Update DataDump [22:04:07] miraheze/DataDump/paladox-patch-2/6d6afe6 - paladox The build has errored. https://travis-ci.com/miraheze/DataDump/builds/183450950 [22:06:27] RECOVERY - mw6 Puppet on mw6 is OK: OK: Puppet is currently enabled, last run 1 second ago with 0 failures [22:07:06] !log root@jobrunner1:/home/paladox# /usr/local/bin/foreachwikiindblist /srv/mediawiki/w/cache/databases.json /srv/mediawiki/w/maintenance/sql.php /srv/mediawiki/w/extensions/DataDump/sql/patches/patch-dumps_size-bigint.sql [22:07:11] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [22:07:18] RECOVERY - cloud2 Current Load on cloud2 is OK: OK - load average: 18.84, 20.27, 20.35 [22:08:23] RECOVERY - mw6 Current Load on mw6 is OK: OK - load average: 5.82, 6.58, 6.03 [22:15:19] PROBLEM - cloud2 Current Load on cloud2 is WARNING: WARNING - load average: 20.31, 21.96, 21.22 [22:19:42] PROBLEM - mw7 Puppet on mw7 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[/mnt/mediawiki-static] [22:21:20] PROBLEM - cloud2 Current Load on cloud2 is CRITICAL: CRITICAL - load average: 29.37, 21.65, 21.05 [22:23:18] PROBLEM - cloud2 Current Load on cloud2 is WARNING: WARNING - load average: 21.35, 21.75, 21.20 [22:27:18] RECOVERY - cloud2 Current Load on cloud2 is OK: OK - load average: 15.87, 19.22, 20.30 [22:28:27] PROBLEM - mw6 Puppet on mw6 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[/mnt/mediawiki-static] [22:33:18] PROBLEM - cloud2 Current Load on cloud2 is WARNING: WARNING - load average: 23.25, 20.67, 20.27 [22:35:18] PROBLEM - cloud2 Current Load on cloud2 is CRITICAL: CRITICAL - load average: 26.08, 22.57, 21.01 [22:35:45] RECOVERY - mw7 Puppet on mw7 is OK: OK: Puppet is currently enabled, last run 28 seconds ago with 0 failures [22:36:26] RECOVERY - mw6 Puppet on mw6 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [22:36:41] !log apply https://github.com/miraheze/DataDump/blob/master/sql/data_dump.sql to zymonicwiki [22:36:42] [ DataDump/data_dump.sql at master · miraheze/DataDump · GitHub ] - github.com [22:36:46] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [22:37:19] PROBLEM - cloud2 Current Load on cloud2 is WARNING: WARNING - load average: 22.96, 23.20, 21.47 [22:38:24] PROBLEM - gluster1 Current Load on gluster1 is CRITICAL: CRITICAL - load average: 8.57, 8.11, 6.66 [22:40:24] PROBLEM - gluster1 Current Load on gluster1 is WARNING: WARNING - load average: 6.25, 7.25, 6.51 [22:45:18] PROBLEM - cloud2 Current Load on cloud2 is CRITICAL: CRITICAL - load average: 26.76, 22.86, 21.67 [22:47:18] PROBLEM - cloud2 Current Load on cloud2 is WARNING: WARNING - load average: 15.92, 20.74, 21.09 [22:48:24] RECOVERY - gluster1 Current Load on gluster1 is OK: OK - load average: 4.18, 5.95, 6.34 [22:49:18] RECOVERY - cloud2 Current Load on cloud2 is OK: OK - load average: 14.14, 18.52, 20.22 [22:54:24] PROBLEM - gluster1 Current Load on gluster1 is WARNING: WARNING - load average: 6.95, 6.23, 6.26 [22:56:24] RECOVERY - gluster1 Current Load on gluster1 is OK: OK - load average: 4.50, 5.59, 6.02 [23:05:56] !log update puppet-agent on phab1 [23:05:59] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:11:32] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/JUcvP [23:11:34] [02miraheze/puppet] 07paladox 03a511b7c - Catch resolver.NXDOMAIN Bug: T6093 [23:11:36] [02puppet] 07paladox created branch 03paladox-patch-1 - 13https://git.io/vbiAS [23:11:37] [02puppet] 07paladox opened pull request 03#1491: Catch resolver.NXDOMAIN - 13https://git.io/JUcvX [23:11:49] [02puppet] 07paladox synchronize pull request 03#1491: Catch resolver.NXDOMAIN - 13https://git.io/JUcvX [23:11:51] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/JUcv1 [23:11:52] [02miraheze/puppet] 07paladox 0381d41db - Update check_reverse_dns.py [23:12:07] [02puppet] 07paladox edited pull request 03#1491: Catch resolver.NoAnswer - 13https://git.io/JUcvX [23:13:14] [02puppet] 07paladox closed pull request 03#1491: Catch resolver.NoAnswer - 13https://git.io/JUcvX [23:13:15] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JUcvD [23:13:17] [02miraheze/puppet] 07paladox 03962e217 - Catch resolver.NoAnswer (#1491) * Catch resolver.NXDOMAIN Bug: T6093 * Update check_reverse_dns.py [23:13:18] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-1 [23:13:20] [02puppet] 07paladox deleted branch 03paladox-patch-1 - 13https://git.io/vbiAS [23:13:41] PROBLEM - cloud2 Current Load on cloud2 is WARNING: WARNING - load average: 20.42, 18.69, 18.01 [23:15:35] RECOVERY - cloud2 Current Load on cloud2 is OK: OK - load average: 18.60, 19.09, 18.26 [23:21:30] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JUcvp [23:21:31] [02miraheze/puppet] 07paladox 038ea25cf - Phabricator: Set fastcgi_buffers and fastcgi_buffer_size [23:25:18] !log MariaDB [doctorwhofanfilmsdatabasewiki]> INSERT INTO slot_roles (role_id, role_name) VALUES (1, 'main'); [23:25:24] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:32:33] !log deleted some pages on pseudocienciawiki using deleteBatch script [23:32:37] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:40:29] [02mw-config] 07Universal-Omega opened pull request 03#3241: add meta namespace for privacywiki (T6157) - 13https://git.io/JUcfD [23:46:14] [02mw-config] 07Universal-Omega synchronize pull request 03#3241: add meta namespace for privacywiki (T6157) - 13https://git.io/JUcfD [23:48:45] [02mw-config] 07Universal-Omega synchronize pull request 03#3241: add meta namespace for privacywiki (T6157) - 13https://git.io/JUcfD [23:51:03] [02mw-config] 07Universal-Omega synchronize pull request 03#3241: add meta namespace for privacywiki (T6157) - 13https://git.io/JUcfD [23:52:06] [02mw-config] 07Universal-Omega edited pull request 03#3241: add wgMetaNamespace and wgMetaNamespaceTalk to ManageWikiSettings (T6157) - 13https://git.io/JUcfD