[00:18:30] !log downtimed load alerts for dbbackup[12] till March 31, per https://phabricator.miraheze.org/T6984 [00:18:31] [ ⚓ T6984 High load on dbbackup servers ] - phabricator.miraheze.org [00:18:33] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:18:46] that will reduce the spam in this channel [00:20:00] night [01:01:48] [02miraheze/MirahezeMagic] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://git.io/JqjgQ [01:01:49] [02miraheze/MirahezeMagic] 07Universal-Omega 0316ee76f - removePII: only remove logs from after user registration (T6958) [01:01:51] [02MirahezeMagic] 07Universal-Omega created branch 03Universal-Omega-patch-1 - 13https://git.io/fQRGX [01:01:52] [02MirahezeMagic] 07Universal-Omega opened pull request 03#233: removePII: only remove logs from after user registration (T6958) - 13https://git.io/Jqjg5 [01:03:03] miraheze/MirahezeMagic - Universal-Omega the build passed. [01:24:59] PROBLEM - dbbackup2 Check MariaDB Replication c3 on dbbackup2 is CRITICAL: MariaDB replication - both - CRITICAL - Slave_IO_Running state : Yes, Slave_SQL_Running state : Yes, Seconds_Behind_Master : 302s [01:31:00] RECOVERY - dbbackup2 Check MariaDB Replication c3 on dbbackup2 is OK: MariaDB replication - both - OK - Slave_IO_Running state : Yes, Slave_SQL_Running state : Yes, Seconds_Behind_Master : 1s [02:39:10] PROBLEM - jobrunner4 Current Load on jobrunner4 is CRITICAL: CRITICAL - load average: 6.18, 4.70, 3.18 [02:39:46] PROBLEM - dbbackup2 Check MariaDB Replication c3 on dbbackup2 is CRITICAL: MariaDB replication - both - CRITICAL - Slave_IO_Running state : Yes, Slave_SQL_Running state : Yes, Seconds_Behind_Master : 253s [02:41:10] PROBLEM - jobrunner4 Current Load on jobrunner4 is WARNING: WARNING - load average: 5.92, 5.08, 3.50 [02:43:43] RECOVERY - dbbackup2 Check MariaDB Replication c3 on dbbackup2 is OK: MariaDB replication - both - OK - Slave_IO_Running state : Yes, Slave_SQL_Running state : Yes, Seconds_Behind_Master : 0s [02:45:10] RECOVERY - jobrunner4 Current Load on jobrunner4 is OK: OK - load average: 4.01, 4.76, 3.77 [03:25:12] PROBLEM - jobrunner4 Current Load on jobrunner4 is CRITICAL: CRITICAL - load average: 6.69, 5.03, 3.56 [03:30:28] PROBLEM - jobrunner3 Current Load on jobrunner3 is WARNING: WARNING - load average: 5.58, 4.65, 3.28 [03:33:12] PROBLEM - jobrunner4 Current Load on jobrunner4 is WARNING: WARNING - load average: 5.46, 5.70, 4.55 [03:34:28] PROBLEM - jobrunner3 Current Load on jobrunner3 is CRITICAL: CRITICAL - load average: 6.56, 5.50, 3.92 [03:39:10] RECOVERY - jobrunner4 Current Load on jobrunner4 is OK: OK - load average: 3.84, 5.08, 4.70 [04:02:29] PROBLEM - jobrunner3 Current Load on jobrunner3 is WARNING: WARNING - load average: 4.95, 5.75, 5.88 [04:14:28] RECOVERY - jobrunner3 Current Load on jobrunner3 is OK: OK - load average: 3.18, 4.23, 5.07 [04:58:05] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-2 [+0/-0/±1] 13https://git.io/Jmenm [04:58:07] [02miraheze/mw-config] 07Universal-Omega 036d228c0 - Add simulatorwiki configs [04:58:08] [02mw-config] 07Universal-Omega created branch 03Universal-Omega-patch-2 - 13https://git.io/vbvb3 [04:58:22] [02mw-config] 07Universal-Omega opened pull request 03#3791: Add simulatorwiki configs (T6986) - 13https://git.io/Jmens [04:58:57] [02mw-config] 07Universal-Omega closed pull request 03#3791: Add simulatorwiki configs (T6986) - 13https://git.io/Jmens [04:58:59] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JmenW [04:59:00] [02miraheze/mw-config] 07Universal-Omega 035ca5467 - Add simulatorwiki configs (#3791) [04:59:02] [02miraheze/mw-config] 07Universal-Omega deleted branch 03Universal-Omega-patch-2 [04:59:03] [02mw-config] 07Universal-Omega deleted branch 03Universal-Omega-patch-2 - 13https://git.io/vbvb3 [04:59:30] miraheze/mw-config - Universal-Omega the build passed. [04:59:37] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-2 [+0/-0/±1] 13https://git.io/Jmena [04:59:39] [02miraheze/mw-config] 07Universal-Omega 03e41ea87 - Update LocalSettings.php [04:59:40] [02mw-config] 07Universal-Omega created branch 03Universal-Omega-patch-2 - 13https://git.io/vbvb3 [05:00:00] miraheze/mw-config - Universal-Omega the build passed. [05:00:02] [02mw-config] 07Universal-Omega opened pull request 03#3792: Add 2 moderation settings to ManageWiki (T6986) - 13https://git.io/Jmeno [05:01:25] miraheze/mw-config - Universal-Omega the build passed. [05:08:00] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-2 [+0/-0/±1] 13https://git.io/JmeC2 [05:08:01] [02miraheze/mw-config] 07Universal-Omega 03cd7e5b7 - Update ManageWikiSettings.php [05:08:03] [02mw-config] 07Universal-Omega synchronize pull request 03#3792: Add 2 moderation settings to ManageWiki (T6986) - 13https://git.io/Jmeno [05:09:05] miraheze/mw-config - Universal-Omega the build passed. [05:09:24] [02mw-config] 07Universal-Omega closed pull request 03#3792: Add 2 moderation settings to ManageWiki (T6986) - 13https://git.io/Jmeno [05:09:25] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±2] 13https://git.io/JmeCd [05:09:27] [02miraheze/mw-config] 07Universal-Omega 0391bc9b8 - Add 2 moderation settings to ManageWiki (T6986) (#3792) [05:09:28] [02mw-config] 07Universal-Omega deleted branch 03Universal-Omega-patch-2 - 13https://git.io/vbvb3 [05:09:30] [02miraheze/mw-config] 07Universal-Omega deleted branch 03Universal-Omega-patch-2 [05:10:35] miraheze/mw-config - Universal-Omega the build passed. [05:20:28] PROBLEM - jobrunner3 Current Load on jobrunner3 is WARNING: WARNING - load average: 5.34, 4.58, 3.47 [05:22:28] RECOVERY - jobrunner3 Current Load on jobrunner3 is OK: OK - load average: 4.80, 4.69, 3.65 [07:12:37] !log renamed outlawsrefugewiki to outlawshavenwiki [07:12:40] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [08:03:11] [02mw-config] 07R4356th synchronize pull request 03#3789: Add new "Song Contest" category for CreateWiki - 13https://git.io/JqAlE [08:03:25] [02mw-config] 07R4356th reviewed pull request 03#3789 commit - 13https://git.io/Jmexa [08:04:31] miraheze/mw-config - R4356th the build passed. [08:52:45] [02mediawiki] 07R4356th opened pull request 03#1333: Install VariablesLua (T6959) - 13https://git.io/JmJs3 [09:17:12] PROBLEM - ping4 on cp3 is WARNING: PING WARNING - Packet loss = 0%, RTA = 349.44 ms [09:19:12] RECOVERY - ping4 on cp3 is OK: PING OK - Packet loss = 0%, RTA = 253.09 ms [09:23:19] PROBLEM - ping4 on cp3 is WARNING: PING WARNING - Packet loss = 0%, RTA = 349.78 ms [09:25:19] RECOVERY - ping4 on cp3 is OK: PING OK - Packet loss = 0%, RTA = 253.44 ms [10:06:59] PROBLEM - dbbackup2 Check MariaDB Replication c3 on dbbackup2 is WARNING: MariaDB replication - both - WARNING - Slave_IO_Running state : Yes, Slave_SQL_Running state : Yes, Seconds_Behind_Master : 168s [10:08:59] RECOVERY - dbbackup2 Check MariaDB Replication c3 on dbbackup2 is OK: MariaDB replication - both - OK - Slave_IO_Running state : Yes, Slave_SQL_Running state : Yes, Seconds_Behind_Master : 92s [10:12:59] PROBLEM - dbbackup2 Check MariaDB Replication c3 on dbbackup2 is WARNING: MariaDB replication - both - WARNING - Slave_IO_Running state : Yes, Slave_SQL_Running state : Yes, Seconds_Behind_Master : 192s [10:16:59] RECOVERY - dbbackup2 Check MariaDB Replication c3 on dbbackup2 is OK: MariaDB replication - both - OK - Slave_IO_Running state : Yes, Slave_SQL_Running state : Yes, Seconds_Behind_Master : 0s [12:11:06] [02mw-config] 07R4356th opened pull request 03#3793: Comments conflicts with ProtectionIndicator (T6987) - 13https://git.io/JmtI4 [12:12:16] miraheze/mw-config - R4356th the build passed. [14:10:59] PROBLEM - dbbackup2 Check MariaDB Replication c3 on dbbackup2 is CRITICAL: MariaDB replication - both - CRITICAL - Slave_IO_Running state : Yes, Slave_SQL_Running state : Yes, Seconds_Behind_Master : 272s [14:12:25] [02miraheze/puppet] 07JohnFLewis pushed 031 commit to 03master [+1/-0/±2] 13https://git.io/JmY4s [14:12:27] [02miraheze/puppet] 07JohnFLewis 03392cd05 - T6973: monitor disk health [14:14:57] [02miraheze/puppet] 07JohnFLewis pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JmYEL [14:14:58] [02miraheze/puppet] 07JohnFLewis 03db518ef - fact -> facts [14:20:12] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JmYoE [14:20:13] [02miraheze/services] 07MirahezeSSLBot 03e282b0e - BOT: Updating services config for wikis [14:34:41] PROBLEM - dbbackup2 Check MariaDB Replication c3 on dbbackup2 is WARNING: MariaDB replication - both - WARNING - Slave_IO_Running state : Yes, Slave_SQL_Running state : Yes, Seconds_Behind_Master : 177s [14:36:39] RECOVERY - dbbackup2 Check MariaDB Replication c3 on dbbackup2 is OK: MariaDB replication - both - OK - Slave_IO_Running state : Yes, Slave_SQL_Running state : Yes, Seconds_Behind_Master : 0s [14:43:32] PROBLEM - wiki.thesimswiki.com - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.thesimswiki.com All nameservers failed to answer the query. [14:50:14] RECOVERY - wiki.thesimswiki.com - reverse DNS on sslhost is OK: rDNS OK - wiki.thesimswiki.com reverse DNS resolves to cp10.miraheze.org [15:04:18] [02miraheze/mediawiki] 07Universal-Omega pushed 031 commit to 03REL1_35 [+1/-0/±1] 13https://git.io/JmOHc [15:04:19] [02miraheze/mediawiki] 07R4356th 031e40de3 - Install VariablesLua (T6959) (#1333) [15:05:26] [02mediawiki] 07Universal-Omega closed pull request 03#1333: Install VariablesLua (T6959) - 13https://git.io/JmJs3 [15:07:09] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JmOdx [15:07:10] [02miraheze/mw-config] 07R4356th 032883546 - Add new "Song Contest" category for CreateWiki (#3789) [15:07:18] [02mw-config] 07Universal-Omega reviewed pull request 03#3789 commit - 13https://git.io/JmOFq [15:07:55] [02mw-config] 07Universal-Omega closed pull request 03#3789: Add new "Song Contest" category for CreateWiki - 13https://git.io/JqAlE [15:08:02] [02mw-config] 07Universal-Omega commented on pull request 03#3793: Comments conflicts with ProtectionIndicator (T6987) - 13https://git.io/JmOb0 [15:08:46] [02mw-config] 07Universal-Omega closed pull request 03#3793: Comments conflicts with ProtectionIndicator (T6987) - 13https://git.io/JmtI4 [15:08:48] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JmONS [15:08:49] [02miraheze/mw-config] 07R4356th 032500813 - Comments conflicts with ProtectionIndicator (#3793) [15:08:50] miraheze/mw-config - Universal-Omega the build passed. [15:10:00] miraheze/mw-config - Universal-Omega the build passed. [15:40:40] PROBLEM - mw10 Puppet on mw10 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki core] [15:41:09] PROBLEM - mw11 Puppet on mw11 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki core] [15:42:25] [02mw-config] 07Universal-Omega closed pull request 03#3786: Add VariablesLua extension (T6959) - 13https://git.io/JqdI1 [15:42:26] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±4] 13https://git.io/Jm3Mn [15:42:27] PROBLEM - jobrunner4 Puppet on jobrunner4 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 4 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki core] [15:42:28] [02miraheze/mw-config] 07Universal-Omega 0385088ec - Add VariablesLua extension (T6959) (#3786) [15:42:29] [02mw-config] 07Universal-Omega deleted branch 03Universal-Omega-patch-1 - 13https://git.io/vbvb3 [15:42:31] [02miraheze/mw-config] 07Universal-Omega deleted branch 03Universal-Omega-patch-1 [15:43:28] miraheze/mw-config - Universal-Omega the build passed. [15:50:39] [02MirahezeMagic] 07Universal-Omega closed pull request 03#233: removePII: only remove logs from after user registration (T6958) - 13https://git.io/Jqjg5 [15:50:41] [02MirahezeMagic] 07Universal-Omega deleted branch 03Universal-Omega-patch-1 - 13https://git.io/fQRGX [15:50:42] [02miraheze/MirahezeMagic] 07Universal-Omega deleted branch 03Universal-Omega-patch-1 [15:58:58] [02miraheze/MirahezeMagic] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://git.io/JmsTX [15:58:59] [02miraheze/MirahezeMagic] 07Universal-Omega 03d3b8fe9 - remove 2 abusefilter tables from removePII [15:59:00] [02MirahezeMagic] 07Universal-Omega created branch 03Universal-Omega-patch-1 - 13https://git.io/fQRGX [15:59:02] [02MirahezeMagic] 07Universal-Omega opened pull request 03#234: remove 2 abusefilter tables from removePII - 13https://git.io/JmsTS [16:00:02] [02MirahezeMagic] 07Universal-Omega closed pull request 03#234: remove 2 abusefilter tables from removePII - 13https://git.io/JmsTS [16:00:03] [02miraheze/MirahezeMagic] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JmsIC [16:00:04] [02miraheze/MirahezeMagic] 07Universal-Omega 0360654d5 - remove 2 abusefilter tables from removePII (#234) [16:00:06] [02miraheze/MirahezeMagic] 07Universal-Omega deleted branch 03Universal-Omega-patch-1 [16:00:08] [02MirahezeMagic] 07Universal-Omega deleted branch 03Universal-Omega-patch-1 - 13https://git.io/fQRGX [16:00:11] miraheze/MirahezeMagic - Universal-Omega the build passed. [16:01:01] miraheze/MirahezeMagic - Universal-Omega the build passed. [16:04:24] RECOVERY - jobrunner4 Puppet on jobrunner4 is OK: OK: Puppet is currently enabled, last run 29 seconds ago with 0 failures [16:05:08] RECOVERY - mw11 Puppet on mw11 is OK: OK: Puppet is currently enabled, last run 33 seconds ago with 0 failures [16:06:25] RECOVERY - mw10 Puppet on mw10 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [16:06:44] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 6.88, 7.20, 5.64 [16:08:45] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 4.18, 6.36, 5.55 [16:10:38] PROBLEM - ping6 on dbbackup2 is WARNING: PING WARNING - Packet loss = 0%, RTA = 214.40 ms [16:11:06] PROBLEM - ping4 on bacula2 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 155.68 ms [16:11:34] PROBLEM - ping6 on dbbackup1 is WARNING: PING WARNING - Packet loss = 0%, RTA = 190.20 ms [16:12:10] PROBLEM - ping6 on ns1 is WARNING: PING WARNING - Packet loss = 0%, RTA = 221.37 ms [16:20:19] PROBLEM - ping6 on ns1 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 229.62 ms [16:21:07] PROBLEM - ping4 on bacula2 is WARNING: PING WARNING - Packet loss = 0%, RTA = 153.53 ms [16:22:21] PROBLEM - ping6 on ns1 is WARNING: PING WARNING - Packet loss = 0%, RTA = 209.17 ms [16:23:08] PROBLEM - ping4 on bacula2 is CRITICAL: PING CRITICAL - Packet loss = 28%, RTA = 157.31 ms [16:28:39] PROBLEM - db11 Disk Space on db11 is WARNING: DISK WARNING - free space: / 41918 MB (10% inode=97%); [16:29:07] PROBLEM - ping4 on bacula2 is WARNING: PING WARNING - Packet loss = 0%, RTA = 153.98 ms [16:31:06] PROBLEM - ping4 on bacula2 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 155.46 ms [16:32:30] RECOVERY - ping6 on ns1 is OK: PING OK - Packet loss = 0%, RTA = 103.74 ms [16:33:01] RECOVERY - ping6 on dbbackup2 is OK: PING OK - Packet loss = 0%, RTA = 100.01 ms [16:33:04] PROBLEM - ping4 on bacula2 is WARNING: PING WARNING - Packet loss = 0%, RTA = 157.35 ms [16:33:25] RECOVERY - ping6 on dbbackup1 is OK: PING OK - Packet loss = 0%, RTA = 102.40 ms [16:33:59] db11 alert: we have about 50GB of binlogs [16:34:20] however, dbbackup1 has 29 hours of replication lag, so we cannot delete all previous binlogs [16:35:05] PROBLEM - ping4 on bacula2 is CRITICAL: PING CRITICAL - Packet loss = 28%, RTA = 156.46 ms [16:35:21] !log db11: MariaDB [(none)]> purge binary logs to 'mysql-bin.000973'; [16:35:25] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:35:51] binlogs total size 52G -> 29G [16:36:39] RECOVERY - db11 Disk Space on db11 is OK: DISK OK - free space: / 65393 MB (17% inode=97%); [16:37:04] PROBLEM - ping4 on bacula2 is WARNING: PING WARNING - Packet loss = 0%, RTA = 129.97 ms [16:37:14] * SPF|Cloud goes back to work [16:39:04] PROBLEM - ping4 on bacula2 is CRITICAL: PING CRITICAL - Packet loss = 28%, RTA = 146.95 ms [16:41:03] RECOVERY - ping4 on bacula2 is OK: PING OK - Packet loss = 0%, RTA = 76.22 ms [16:59:03] PROBLEM - ping4 on bacula2 is WARNING: PING WARNING - Packet loss = 0%, RTA = 141.99 ms [17:03:04] PROBLEM - ping4 on bacula2 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 157.92 ms [17:03:12] PROBLEM - ping6 on dbbackup1 is WARNING: PING WARNING - Packet loss = 0%, RTA = 196.79 ms [17:05:12] RECOVERY - ping6 on dbbackup1 is OK: PING OK - Packet loss = 0%, RTA = 119.36 ms [17:09:14] PROBLEM - ping6 on dbbackup1 is WARNING: PING WARNING - Packet loss = 0%, RTA = 166.24 ms [17:11:02] RECOVERY - ping4 on bacula2 is OK: PING OK - Packet loss = 0%, RTA = 76.96 ms [17:11:12] RECOVERY - ping6 on dbbackup1 is OK: PING OK - Packet loss = 0%, RTA = 119.70 ms [17:11:41] PROBLEM - ping6 on dbbackup2 is WARNING: PING WARNING - Packet loss = 0%, RTA = 129.21 ms [17:12:53] !log RebuildLocalisationCache and mergeMessageFileList on Jbr* and mw* [17:12:57] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:13:43] RECOVERY - ping6 on dbbackup2 is OK: PING OK - Packet loss = 0%, RTA = 114.15 ms [17:22:04] [02miraheze/CreateWiki] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://git.io/JmZW6 [17:22:05] [02miraheze/CreateWiki] 07Universal-Omega 03be98e36 - Fix T6990 [17:22:07] [02CreateWiki] 07Universal-Omega created branch 03Universal-Omega-patch-1 - 13https://git.io/vpJTL [17:22:08] [02CreateWiki] 07Universal-Omega opened pull request 03#206: Fix T6990 - 13https://git.io/JmZWH [17:23:17] miraheze/CreateWiki - Universal-Omega the build passed. [17:31:19] PROBLEM - ping6 on dbbackup1 is WARNING: PING WARNING - Packet loss = 0%, RTA = 146.61 ms [17:32:08] PROBLEM - ping6 on dbbackup2 is WARNING: PING WARNING - Packet loss = 0%, RTA = 197.36 ms [17:34:34] [02CreateWiki] 07Universal-Omega closed pull request 03#206: Fix T6990 - 13https://git.io/JmZWH [17:34:35] [02miraheze/CreateWiki] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JmZ64 [17:34:36] [02miraheze/CreateWiki] 07Universal-Omega 03b6e0f3e - Fix T6990 (#206) [17:34:38] [02miraheze/CreateWiki] 07Universal-Omega deleted branch 03Universal-Omega-patch-1 [17:34:40] [02CreateWiki] 07Universal-Omega deleted branch 03Universal-Omega-patch-1 - 13https://git.io/vpJTL [17:34:54] PROBLEM - ping6 on ns1 is WARNING: PING WARNING - Packet loss = 0%, RTA = 122.00 ms [17:35:51] miraheze/CreateWiki - Universal-Omega the build passed. [17:38:58] RECOVERY - ping6 on ns1 is OK: PING OK - Packet loss = 0%, RTA = 111.43 ms [17:44:22] RECOVERY - ping6 on dbbackup2 is OK: PING OK - Packet loss = 0%, RTA = 101.01 ms [17:45:12] RECOVERY - ping6 on dbbackup1 is OK: PING OK - Packet loss = 0%, RTA = 98.55 ms [17:45:15] [02miraheze/mediawiki] 07Universal-Omega pushed 031 commit to 03REL1_35 [+0/-0/±1] 13https://git.io/JmZ5H [17:45:16] [02miraheze/mediawiki] 07Universal-Omega 037db5458 - Update CreateWiki [18:28:15] [02miraheze/mw-config] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/JmnaI [18:28:16] [02miraheze/mw-config] 07paladox 030e15b27 - Make values in wmgCacheSettings array [18:28:18] [02mw-config] 07paladox created branch 03paladox-patch-1 - 13https://git.io/vbvb3 [18:28:19] [02mw-config] 07paladox opened pull request 03#3794: Make values in wmgCacheSettings array - 13https://git.io/Jmnat [18:28:30] [02mw-config] 07paladox closed pull request 03#3794: Make values in wmgCacheSettings array - 13https://git.io/Jmnat [18:28:31] [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Jmna0 [18:28:33] [02miraheze/mw-config] 07paladox 030217d6d - Make values in wmgCacheSettings array (#3794) [18:28:50] [02miraheze/MirahezeMagic] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JmnaH [18:28:52] [02miraheze/MirahezeMagic] 07paladox 039068630 - removeMemcachedKey: Pick the first option in the array in $wmgCacheSettings [18:29:23] [02miraheze/mediawiki] 07paladox pushed 031 commit to 03REL1_35 [+0/-0/±1] 13https://git.io/JmnVn [18:29:25] [02miraheze/mediawiki] 07paladox 03eab0d2f - Update MirahezeMagic [18:29:30] miraheze/mw-config - paladox the build passed. [18:30:00] miraheze/mw-config - paladox the build passed. [18:30:01] miraheze/MirahezeMagic - paladox the build passed. [18:31:08] [02mw-config] 07paladox deleted branch 03paladox-patch-1 - 13https://git.io/vbvb3 [18:31:10] [02miraheze/mw-config] 07paladox deleted branch 03paladox-patch-1 [18:32:54] PROBLEM - graylog2 Current Load on graylog2 is CRITICAL: CRITICAL - load average: 14.22, 7.17, 3.11 [18:33:28] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 18.39, 11.38, 6.87 [18:33:47] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 17.98, 12.15, 6.92 [18:33:55] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 18.52, 12.06, 6.82 [18:34:06] [02miraheze/mw-config] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/JmnoA [18:34:07] [02miraheze/mw-config] 07paladox 0385d621f - memcached: Send parser and message cache to mem1 [18:34:07] paladox: ^^ fyi [18:34:09] [02mw-config] 07paladox created branch 03paladox-patch-1 - 13https://git.io/vbvb3 [18:34:10] [02mw-config] 07paladox opened pull request 03#3795: memcached: Send parser and message cache to mem1 - 13https://git.io/Jmnox [18:34:10] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 18.93, 11.81, 6.93 [18:34:20] oh [18:34:26] Graylog has almost 1 million messages unprocessed [18:34:28] PROBLEM - cp11 Stunnel Http for mw8 on cp11 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [18:34:36] [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JmnKn [18:34:38] [02miraheze/mw-config] 07paladox 0316ed645 - Fix [18:34:41] PROBLEM - cp12 Stunnel Http for mw8 on cp12 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [18:34:42] and graylog crashed [18:35:02] PROBLEM - cp12 HTTPS on cp12 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:35:05] PROBLEM - cp3 Stunnel Http for mw9 on cp3 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 344 bytes in 0.830 second response time [18:35:05] PROBLEM - cp3 Varnish Backends on cp3 is CRITICAL: 2 backends are down. mw10 mw11 [18:35:09] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 8 datacenters are down: 128.199.139.216/cpweb, 2400:6180:0:d0::403:f001/cpweb, 51.195.236.219/cpweb, 51.195.236.250/cpweb, 2001:41d0:800:178a::5/cpweb, 2001:41d0:800:1bbd::4/cpweb, 51.222.25.132/cpweb, 2607:5300:205:200::1c30/cpweb [18:35:11] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 8 datacenters are down: 128.199.139.216/cpweb, 2400:6180:0:d0::403:f001/cpweb, 51.195.236.219/cpweb, 51.195.236.250/cpweb, 2001:41d0:800:178a::5/cpweb, 2001:41d0:800:1bbd::4/cpweb, 51.222.25.132/cpweb, 2607:5300:205:200::1c30/cpweb [18:35:11] PROBLEM - cp10 Varnish Backends on cp10 is CRITICAL: 3 backends are down. mw8 mw10 mw11 [18:35:15] miraheze/mw-config - paladox the build passed. [18:35:32] RECOVERY - test3 Puppet on test3 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [18:35:43] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 23.84, 19.61, 13.92 [18:35:44] PROBLEM - mw8 MediaWiki Rendering on mw8 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:35:55] PROBLEM - cp12 Stunnel Http for mw9 on cp12 is CRITICAL: HTTP CRITICAL - No data received from host [18:35:55] miraheze/mw-config - paladox the build passed. [18:36:05] PROBLEM - cp12 Varnish Backends on cp12 is CRITICAL: 2 backends are down. mw8 mw9 [18:36:24] PROBLEM - cp11 Varnish Backends on cp11 is CRITICAL: 3 backends are down. mw8 mw9 mw11 [18:36:25] fixed [18:36:28] RECOVERY - cp11 Stunnel Http for mw8 on cp11 is OK: HTTP OK: HTTP/1.1 200 OK - 15193 bytes in 0.006 second response time [18:36:35] You caused it? [18:36:39] RECOVERY - cp12 Stunnel Http for mw8 on cp12 is OK: HTTP OK: HTTP/1.1 200 OK - 15193 bytes in 0.352 second response time [18:36:48] Yeh, i missed fixing a variable [18:36:57] RECOVERY - cp12 HTTPS on cp12 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 2229 bytes in 0.343 second response time [18:36:59] You know what to do next then :) [18:37:03] RECOVERY - cp3 Varnish Backends on cp3 is OK: All 7 backends are healthy [18:37:04] I turned a variable value into an array but missed [0] [18:37:04] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [18:37:05] RECOVERY - cp10 Varnish Backends on cp10 is OK: All 7 backends are healthy [18:37:06] RECOVERY - cp3 Stunnel Http for mw9 on cp3 is OK: HTTP OK: HTTP/1.1 200 OK - 15193 bytes in 1.045 second response time [18:37:11] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [18:37:11] yeh [18:37:40] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 16.45, 18.17, 14.07 [18:37:43] RECOVERY - mw8 MediaWiki Rendering on mw8 is OK: HTTP OK: HTTP/1.1 200 OK - 20708 bytes in 0.334 second response time [18:37:51] RECOVERY - cp12 Stunnel Http for mw9 on cp12 is OK: HTTP OK: HTTP/1.1 200 OK - 15193 bytes in 0.343 second response time [18:38:05] RECOVERY - cp12 Varnish Backends on cp12 is OK: All 7 backends are healthy [18:38:13] 2 million messages in Graylog and increasing as well [18:38:24] RECOVERY - cp11 Varnish Backends on cp11 is OK: All 7 backends are healthy [18:39:37] Going down now but peaked at 2.3 million [18:41:11] :O [18:41:26] what are the message that is causing that JohnLewis ? [18:41:28] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 3.06, 6.82, 6.86 [18:41:48] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 4.39, 7.87, 7.31 [18:41:48] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 3.89, 7.18, 6.89 [18:43:26] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 4.17, 6.00, 6.56 [18:43:45] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 2.89, 5.69, 6.38 [18:43:49] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 5.70, 7.22, 7.08 [18:44:41] paladox: all the memached failures [18:44:47] ah ok [18:45:46] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 4.43, 5.80, 6.57 [18:45:48] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 5.07, 6.35, 6.77 [18:46:09] [02miraheze/mw-config] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/Jmnyx [18:46:10] [02miraheze/mw-config] 07paladox 0327a7341 - Update GlobalCache.php [18:46:12] [02mw-config] 07paladox synchronize pull request 03#3795: memcached: Send parser and message cache to mem1 - 13https://git.io/Jmnox [18:46:30] [02miraheze/mw-config] 07paladox pushed 032 commits to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/JmnSO [18:46:31] [02miraheze/mw-config] 07paladox 038832003 - Merge branch 'master' into paladox-patch-1 [18:46:33] [02mw-config] 07paladox synchronize pull request 03#3795: memcached: Send parser and message cache to mem1 - 13https://git.io/Jmnox [18:46:43] [02miraheze/mw-config] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/JmnS4 [18:46:45] [02miraheze/mw-config] 07paladox 0395a2c00 - Update GlobalCache.php [18:46:46] [02mw-config] 07paladox synchronize pull request 03#3795: memcached: Send parser and message cache to mem1 - 13https://git.io/Jmnox [18:47:16] paladox: quick logical question here [18:47:38] miraheze/mw-config - paladox the build passed. [18:47:38] Why are we swapped mem2 from obj to mixed, and introducing mem1 as obj rather than mixed? [18:47:53] miraheze/mw-config - paladox the build passed. [18:49:04] JohnLewis we're not. [18:49:17] That's not what the change suggests [18:49:55] any idea where https://phabricator.miraheze.org/T6992 is coming from? I can't reproduce [18:49:57] [ ⚓ T6992 502/503s ~18:30-18:45 17 March ] - phabricator.miraheze.org [18:50:26] $wgSessionCacheType = 'memcached-mem-1'; is deploying mem1 as obj [18:50:33] Reception123: see above, Incident report already up [18:51:13] uh, memcached-mem-1 is mem2 [18:51:27] it does $wmgCacheSettings['memcached']['server'][0] [18:51:31] which is mem2 [18:51:41] i'll rename it [18:51:47] So, mem-1 = mem2 and mem-2 = mem1 :) [18:51:57] Please [18:52:08] ah, thanks [18:52:43] [02mw-config] 07paladox synchronize pull request 03#3795: memcached: Send parser and message cache to mem1 - 13https://git.io/Jmnox [18:52:45] [02miraheze/mw-config] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/Jmn7o [18:52:46] [02miraheze/mw-config] 07paladox 0351113a4 - Update GlobalCache.php [18:53:20] JohnLewis done :) [18:53:53] miraheze/mw-config - paladox the build passed. [18:55:31] paladox: now it's clearer, looks good. Deploy, monitor and if you can, find some data to show an improvement :) [19:05:20] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 8.17, 6.65, 6.12 [19:07:17] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 4.71, 5.80, 5.86 [19:08:40] JohnLewis: I have seen your tasks regarding getting data from logs into grafana. I cannot comment in detail right now, but my proposal is to use prometheus-es-exporter for exporting ES data to prometheus [19:09:11] Wikimedia uses that, so we can steal their configuration if we want [19:11:31] PROBLEM - ping4 on bacula2 is WARNING: PING WARNING - Packet loss = 0%, RTA = 129.35 ms [19:15:12] SPF|Cloud: that sounds good, Reception123 is the EM for those tasks getting completed though so discuss with him so he can disseminate it down to whoever works on it :) [19:16:30] Universal_Omega was looking into the API task [19:17:31] PROBLEM - ping4 on bacula2 is CRITICAL: PING CRITICAL - Packet loss = 44%, RTA = 160.65 ms [19:17:53] [02mw-config] 07paladox closed pull request 03#3795: memcached: Send parser and message cache to mem1 - 13https://git.io/Jmnox [19:17:55] [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±2] 13https://git.io/JmcIv [19:17:56] [02miraheze/mw-config] 07paladox 03347ea3a - memcached: Send parser and message cache to mem1 (#3795) [19:17:58] [02mw-config] 07paladox deleted branch 03paladox-patch-1 - 13https://git.io/vbvb3 [19:17:59] [02miraheze/mw-config] 07paladox deleted branch 03paladox-patch-1 [19:18:50] Graylog has done well, only 500k messages left to go [19:18:57] miraheze/mw-config - paladox the build passed. [19:19:30] PROBLEM - ping4 on bacula2 is WARNING: PING WARNING - Packet loss = 0%, RTA = 156.48 ms [19:23:27] RECOVERY - ping4 on bacula2 is OK: PING OK - Packet loss = 0%, RTA = 76.15 ms [19:26:45] [02miraheze/IncidentReporting] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://git.io/JmcOu [19:26:47] [02miraheze/IncidentReporting] 07Universal-Omega 038c94a99 - Update SpecialIncidentReports.php [19:26:48] [02IncidentReporting] 07Universal-Omega synchronize pull request 03#17: T4420: Introduce stats for IncidentReports (WIP) - 13https://git.io/Jq1rG [19:27:03] JohnLewis: ack, but the MW task is limited to ensuring he necessary logs are sent 5 to the syslog-ng, getting the data from ES to Prom requires Infra's expertise :) [19:27:53] miraheze/IncidentReporting - Universal-Omega the build passed. [19:28:11] SPF|Cloud: Reception is an SRE though, so has the technical ability and experience to act in a capacity to minimise Infra's dependency on ensuring the task is done [19:28:12] Also, if bacula2 is still short on space, please open an upgrade task, I'll put some ££ in the RamNode account [19:29:26] bacula2 is short on space because we're still backing up db* via the old method waiting for the new method to be in use [19:31:34] Right, if the new method is inplemented, there are no space issues on bacula2? Since we have procured the new cloud servers, the gluster VMs got a boost in disk space [19:31:55] paladox: ^ [19:32:20] SPF|Cloud if you feel like its safe to do i can remove the db backups from bacula [19:36:18] Not 'safe' yet, since there are no logical dumps on dbbackupX, but neither is the current way of doing backups 'safe', since we are constantly out of space [19:37:16] Ok, so i should go ahead and remove db* from bacula2 SPF|Cloud [19:37:27] I have no issues with removing c3 from bacula, that cluster is currently being replicated to dbbackup2 without issues [19:37:27] 426931 [19:37:31] oop [19:38:04] Removing c3 will help fixing the backup process for now [19:38:37] ok [19:38:52] dbbackup1 has 24 hrs+ replication lag, because there is too much I/O on that VM [19:43:15] I don't know (yet) what wiki/software causes so much writing I/O. I presume matomo is the culprit, so we could try excluding that database from the replication process. In my opinion, matomo data is not very critical [19:44:25] PROBLEM - graylog2 Current Load on graylog2 is WARNING: WARNING - load average: 0.35, 0.90, 3.87 [19:44:55] Measuring the callers of lots of insert/update/delete queries is the best way of finding out [19:46:02] Brb [19:46:41] "aoc2wiki: Void triggered [[Special:AbuseFilter/8|filter 8]], performing the action "autocreateaccount" on [[Special:UserLogin]]." [19:46:42] hmm [19:48:25] RECOVERY - graylog2 Current Load on graylog2 is OK: OK - load average: 0.53, 0.69, 3.09 [19:50:27] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-3 [+0/-0/±1] 13https://git.io/JmcEy [19:50:28] [02miraheze/puppet] 07paladox 038cd6ddf - Remove db13 from bacula [19:50:30] [02puppet] 07paladox created branch 03paladox-patch-3 - 13https://git.io/vbiAS [19:50:31] [02puppet] 07paladox opened pull request 03#1715: Remove db13 from bacula - 13https://git.io/JmcEH [19:50:59] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-3 [+0/-0/±1] 13https://git.io/Jmcun [19:51:01] [02miraheze/puppet] 07paladox 03111b9e7 - Update director.pp [19:51:03] [02puppet] 07paladox synchronize pull request 03#1715: Remove db13 from bacula - 13https://git.io/JmcEH [19:51:11] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-3 [+0/-0/±1] 13https://git.io/Jmcu0 [19:51:12] [02miraheze/puppet] 07paladox 03a3cde02 - Update nrpe.cfg.erb [19:51:14] [02puppet] 07paladox synchronize pull request 03#1715: Remove db13 from bacula - 13https://git.io/JmcEH [19:51:27] db13 is c4, not c3 [19:53:07] oh damn you are right [19:53:13] i confused them sorry [19:54:57] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-4 [+0/-0/±1] 13https://git.io/JmcgM [19:54:58] [02miraheze/puppet] 07paladox 034cf7a14 - Remove db12 from bacula [19:55:00] [02puppet] 07paladox created branch 03paladox-patch-4 - 13https://git.io/vbiAS [19:55:01] [02puppet] 07paladox opened pull request 03#1716: Remove db12 from bacula - 13https://git.io/JmcgD [19:55:24] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-4 [+0/-0/±1] 13https://git.io/JmcgA [19:55:26] [02miraheze/puppet] 07paladox 03b77f15b - Update director.pp [19:55:27] [02puppet] 07paladox synchronize pull request 03#1716: Remove db12 from bacula - 13https://git.io/JmcgD [19:55:35] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-4 [+0/-0/±1] 13https://git.io/Jmc2I [19:55:37] [02miraheze/puppet] 07paladox 03949e1b5 - Update nrpe.cfg.erb [19:55:38] [02puppet] 07paladox synchronize pull request 03#1716: Remove db12 from bacula - 13https://git.io/JmcgD [19:55:51] [02puppet] 07paladox closed pull request 03#1716: Remove db12 from bacula - 13https://git.io/JmcgD [19:55:52] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±3] 13https://git.io/Jmc2O [19:55:53] [02miraheze/puppet] 07paladox 03940159a - Remove db12 from bacula (#1716) [19:55:55] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-4 [19:55:57] [02puppet] 07paladox deleted branch 03paladox-patch-4 - 13https://git.io/vbiAS [19:55:58] [02puppet] 07paladox closed pull request 03#1715: Remove db13 from bacula - 13https://git.io/JmcEH [19:56:00] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-3 [19:56:01] [02puppet] 07paladox deleted branch 03paladox-patch-3 - 13https://git.io/vbiAS [19:56:57] !log remove db12* backup files from bacula2 [19:57:01] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:58:28] made T6994 regarding that abusefilter [20:00:01] Voidwalker would you like me to attach your account? [20:00:20] PROBLEM - bacula2 Bacula Databases db12 on bacula2 is UNKNOWN: NRPE: Command 'check_bacula_databasesdb12' not defined [20:00:54] As a solution for this particular occurrence, sure, would like to know if we could make abusefilters not apply to this sort of thing though [20:00:59] ah it doesn't say which wiki. [20:01:09] oh aoc2wiki [20:01:13] oh wait nvm [20:01:14] yeh [20:01:38] apparently fails on the cli too [20:02:08] oh damn, well that's definitely a problem then [20:02:59] Voidwalker i guess we can use wgAbuseFilterActions? [20:03:27] I don't think so, but let me check [20:06:02] I think that only controls what options are available on the filter creation/edit form [20:12:41] There's apparently a hook for onAbuseFilterShouldFilterAction [20:14:02] Voidwalker https://github.com/wikimedia/mediawiki-extensions-CentralAuth/blob/91c35f2f341b7d8b624473ac91621228a72ca469/includes/CentralAuthHooks.php#L1477 [20:14:03] [ mediawiki-extensions-CentralAuth/CentralAuthHooks.php at 91c35f2f341b7d8b624473ac91621228a72ca469 · wikimedia/mediawiki-extensions-CentralAuth · GitHub ] - github.com [20:14:57] yup, probably a hook to add to MirahezeMagic [20:15:56] if ( $action === 'autocreateaccount' ) skip because automatic account creation should always be possible [20:18:57] [02miraheze/IncidentReporting] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://git.io/JmcyK [20:18:58] [02miraheze/IncidentReporting] 07Universal-Omega 039157df8 - Update SpecialIncidentReports.php [20:19:00] [02IncidentReporting] 07Universal-Omega synchronize pull request 03#17: T4420: Introduce stats for IncidentReports (WIP) - 13https://git.io/Jq1rG [20:20:01] miraheze/IncidentReporting - Universal-Omega the build passed. [20:24:36] now I'm getting another error with Cargo, oh well [20:25:04] [daeed793c2ebb9e39e1b3f9d] 2021-03-17 20:24:04: Fatal exception of type “JobQueueConnectionError” [20:25:33] (when trying to access Special:CargoTables) [20:27:13] paladox, https://phabricator.miraheze.org/T6994#138192 [20:27:15] [ ⚓ T6994 Abuse Filters can apply to action "autocreateaccount" ] - phabricator.miraheze.org [20:27:40] thanks! [20:27:57] Voidwalker if you want to add the hook to MirahezeMagics you can do so or do you want me to do it? [20:28:39] I could, but I'll have to do it through the browser, since I don't think I have a local repo (will take a few commits to do) [20:29:01] ah ok, if you want i can do it, i was just thinking about attribution [20:29:16] @Lake "JobQueueConnectionError from line 764 of /srv/mediawiki/w/includes/jobqueue/JobQueueRedis.php: Unable to connect to redis server Array." [20:29:44] 🤔 [20:29:54]