[00:00:03] PROBLEM - wiki.nowchess.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:02:35] RECOVERY - cp12 Disk Space on cp12 is OK: DISK OK - free space: / 6831 MB (17% inode=96%); [00:25:32] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.61, 3.15, 2.58 [00:27:32] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.30, 2.74, 2.49 [00:40:05] RECOVERY - wiki.nowchess.org - reverse DNS on sslhost is OK: rDNS OK - wiki.nowchess.org reverse DNS resolves to cp11.miraheze.org [00:42:41] RECOVERY - wiki.nowchess.org - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.nowchess.org' will expire on Tue 25 May 2021 01:11:57 GMT +0000. [01:06:17] [02puppet] 07Universal-Omega opened pull request 03#1703: Allow wikis to control Robots.txt from MediaWiki:Robots.txt (T6881) - 13https://git.io/JqZjt [01:23:18] PROBLEM - graylog2 Current Load on graylog2 is CRITICAL: CRITICAL - load average: 5.51, 4.93, 2.47 [01:25:18] RECOVERY - graylog2 Current Load on graylog2 is OK: OK - load average: 1.04, 3.40, 2.20 [01:28:25] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JqZjD [01:28:27] [02miraheze/puppet] 07paladox 03dc4bb77 - mediawiki: Fix Nginx config part 3 [01:28:28] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JqZjD [01:28:30] [02miraheze/puppet] 07paladox 03dc4bb77 - mediawiki: Fix Nginx config part 3 [01:30:58] RECOVERY - dbbackup1 MariaDB c4 on dbbackup1 is OK: Uptime: 44 Threads: 8 Questions: 10 Slow queries: 1 Opens: 16 Flush tables: 1 Open tables: 10 Queries per second avg: 0.227 [01:32:16] PROBLEM - dbbackup1 Check MariaDB Replication c4 on dbbackup1 is CRITICAL: MariaDB replication - both - CRITICAL - Slave IO State not correct, slave stopped or replication broken! [01:33:26] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Jqnem [01:33:27] [02miraheze/puppet] 07paladox 0313fd763 - Fix nginx config part 4 [01:33:29] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Jqnem [01:33:30] [02miraheze/puppet] 07paladox 0313fd763 - Fix nginx config part 4 [02:11:38] [02puppet] 07Universal-Omega synchronize pull request 03#1703: Allow wikis to control Robots.txt from MediaWiki:Robots.txt (T6881) - 13https://git.io/JqZjt [02:28:00] PROBLEM - dbbackup1 Current Load on dbbackup1 is CRITICAL: CRITICAL - load average: 4.40, 3.74, 3.24 [02:29:58] PROBLEM - dbbackup1 Current Load on dbbackup1 is WARNING: WARNING - load average: 2.69, 3.41, 3.19 [02:35:57] RECOVERY - dbbackup1 Current Load on dbbackup1 is OK: OK - load average: 2.73, 3.19, 3.19 [02:42:16] PROBLEM - dbbackup1 Check MariaDB Replication c4 on dbbackup1 is WARNING: MariaDB replication - both - WARNING - Slave_IO_Running state : Yes, Slave_SQL_Running state : Yes, Seconds_Behind_Master : 164s [02:44:16] RECOVERY - dbbackup1 Check MariaDB Replication c4 on dbbackup1 is OK: MariaDB replication - both - OK - Slave_IO_Running state : Yes, Slave_SQL_Running state : Yes, Seconds_Behind_Master : 75s [04:05:09] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JqnUA [04:05:11] [02miraheze/mw-config] 07Universal-Omega 0384427b1 - Add wikimediacommons to wgImportSources for ahinfoboxeswiki (T6945)( [04:06:11] miraheze/mw-config - Universal-Omega the build passed. [04:21:58] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://git.io/JqnTv [04:22:00] [02miraheze/mw-config] 07Universal-Omega 030dfe3d6 - Use in_array rather then multiple || on $_SERVER['REMOTE_ADDR'] [04:22:01] [02mw-config] 07Universal-Omega created branch 03Universal-Omega-patch-1 - 13https://git.io/vbvb3 [04:22:03] [02mw-config] 07Universal-Omega opened pull request 03#3771: Use in_array rather then multiple || on $_SERVER['REMOTE_ADDR'] - 13https://git.io/JqnTf [04:23:07] miraheze/mw-config - Universal-Omega the build passed. [04:37:52] PROBLEM - dbbackup1 Current Load on dbbackup1 is CRITICAL: CRITICAL - load average: 4.30, 3.25, 2.51 [04:39:51] RECOVERY - dbbackup1 Current Load on dbbackup1 is OK: OK - load average: 3.13, 3.32, 2.64 [04:43:47] PROBLEM - dbbackup1 Current Load on dbbackup1 is WARNING: WARNING - load average: 3.44, 3.57, 2.92 [04:45:45] RECOVERY - dbbackup1 Current Load on dbbackup1 is OK: OK - load average: 2.95, 3.27, 2.88 [04:48:17] [02mw-config] 07Universal-Omega commented on pull request 03#3761: Better canned responses - 13https://git.io/JqnTo [04:57:34] [02mw-config] 07Universal-Omega edited a comment on pull request 03#3761: Better canned responses - 13https://git.io/JqnTo [04:59:42] PROBLEM - dbbackup1 Current Load on dbbackup1 is CRITICAL: CRITICAL - load average: 4.85, 3.66, 3.18 [05:03:41] PROBLEM - dbbackup1 Current Load on dbbackup1 is WARNING: WARNING - load average: 3.74, 3.81, 3.37 [05:05:05] [02mw-config] 07dmehus commented on pull request 03#3761: Better canned responses - 13https://git.io/JqnTN [05:05:41] PROBLEM - dbbackup1 Current Load on dbbackup1 is CRITICAL: CRITICAL - load average: 4.64, 4.01, 3.49 [05:06:37] [02mw-config] 07dmehus edited a comment on pull request 03#3761: Better canned responses - 13https://git.io/JqnTN [05:06:52] [02mw-config] 07dmehus edited a comment on pull request 03#3761: Better canned responses - 13https://git.io/JqnTN [05:10:16] PROBLEM - dbbackup1 Check MariaDB Replication c4 on dbbackup1 is WARNING: MariaDB replication - both - WARNING - Slave_IO_Running state : Yes, Slave_SQL_Running state : Yes, Seconds_Behind_Master : 199s [05:11:32] [02mw-config] 07Universal-Omega edited a comment on pull request 03#3761: Better canned responses - 13https://git.io/JqnTo [05:11:39] PROBLEM - dbbackup1 Current Load on dbbackup1 is WARNING: WARNING - load average: 3.98, 3.98, 3.64 [05:12:18] PROBLEM - dbbackup1 Check MariaDB Replication c4 on dbbackup1 is CRITICAL: MariaDB replication - both - CRITICAL - Slave_IO_Running state : Yes, Slave_SQL_Running state : Yes, Seconds_Behind_Master : 321s [05:13:39] PROBLEM - dbbackup1 Current Load on dbbackup1 is CRITICAL: CRITICAL - load average: 4.21, 4.04, 3.70 [05:14:12] [02mw-config] 07Universal-Omega edited a comment on pull request 03#3761: Better canned responses - 13https://git.io/JqnTo [05:15:39] PROBLEM - dbbackup1 Current Load on dbbackup1 is WARNING: WARNING - load average: 3.33, 3.74, 3.63 [05:19:22] [02mw-config] 07Universal-Omega edited a comment on pull request 03#3761: Better canned responses - 13https://git.io/JqnTo [05:21:36] PROBLEM - dbbackup1 Current Load on dbbackup1 is CRITICAL: CRITICAL - load average: 4.73, 3.85, 3.65 [05:33:53] [02mw-config] 07Universal-Omega edited a comment on pull request 03#3761: Better canned responses - 13https://git.io/JqnTo [05:39:21] PROBLEM - dbbackup1 Current Load on dbbackup1 is WARNING: WARNING - load average: 2.64, 3.52, 3.91 [05:40:16] PROBLEM - dbbackup1 Check MariaDB Replication c4 on dbbackup1 is WARNING: MariaDB replication - both - WARNING - Slave_IO_Running state : Yes, Slave_SQL_Running state : Yes, Seconds_Behind_Master : 149s [05:44:16] RECOVERY - dbbackup1 Check MariaDB Replication c4 on dbbackup1 is OK: MariaDB replication - both - OK - Slave_IO_Running state : Yes, Slave_SQL_Running state : Yes, Seconds_Behind_Master : 5s [05:47:15] RECOVERY - dbbackup1 Current Load on dbbackup1 is OK: OK - load average: 1.75, 2.66, 3.40 [06:06:11] RECOVERY - bacula2 APT on bacula2 is OK: APT OK: 1 packages available for upgrade (0 critical updates). [06:19:14] PROBLEM - dbbackup1 Current Load on dbbackup1 is WARNING: WARNING - load average: 3.38, 3.52, 3.09 [06:20:16] PROBLEM - dbbackup1 Check MariaDB Replication c4 on dbbackup1 is CRITICAL: MariaDB replication - both - CRITICAL - Slave_IO_Running state : Yes, Slave_SQL_Running state : Yes, Seconds_Behind_Master : 230s [06:21:13] RECOVERY - dbbackup1 Current Load on dbbackup1 is OK: OK - load average: 2.94, 3.34, 3.08 [06:21:44] PROBLEM - dbbackup1 Check MariaDB Replication c2 on dbbackup1 is WARNING: MariaDB replication - both - WARNING - Slave_IO_Running state : Yes, Slave_SQL_Running state : Yes, Seconds_Behind_Master : 118s [06:25:10] PROBLEM - dbbackup1 Current Load on dbbackup1 is CRITICAL: CRITICAL - load average: 5.80, 4.47, 3.57 [06:25:45] PROBLEM - dbbackup1 Check MariaDB Replication c2 on dbbackup1 is CRITICAL: MariaDB replication - both - CRITICAL - Slave_IO_Running state : Yes, Slave_SQL_Running state : Yes, Seconds_Behind_Master : 244s [06:37:05] PROBLEM - dbbackup1 Current Load on dbbackup1 is WARNING: WARNING - load average: 3.40, 3.79, 3.80 [06:39:04] PROBLEM - dbbackup1 Current Load on dbbackup1 is CRITICAL: CRITICAL - load average: 5.02, 4.12, 3.91 [06:44:00] !log sudo service ircrcbot restart [06:44:02] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [06:45:04] PROBLEM - dbbackup1 Current Load on dbbackup1 is WARNING: WARNING - load average: 3.40, 3.93, 3.94 [06:47:26] [02mw-config] 07R4356th commented on pull request 03#3761: Better canned responses - 13https://git.io/JqnqM [06:47:43] PROBLEM - dbbackup1 Check MariaDB Replication c2 on dbbackup1 is WARNING: MariaDB replication - both - WARNING - Slave_IO_Running state : Yes, Slave_SQL_Running state : Yes, Seconds_Behind_Master : 197s [06:49:43] RECOVERY - dbbackup1 Check MariaDB Replication c2 on dbbackup1 is OK: MariaDB replication - both - OK - Slave_IO_Running state : Yes, Slave_SQL_Running state : Yes, Seconds_Behind_Master : 95s [06:51:07] PROBLEM - dbbackup1 Current Load on dbbackup1 is CRITICAL: CRITICAL - load average: 4.18, 3.70, 3.80 [06:53:05] PROBLEM - dbbackup1 Current Load on dbbackup1 is WARNING: WARNING - load average: 3.63, 3.86, 3.87 [06:54:14] !log reception@jobrunner4:~$ sudo -u www-data php /srv/mediawiki/w/maintenance/importDump.php --wiki minecraftwiki /home/reception/minecraft_pages_full.xml --username-prefix="wikia:minecraftwiki" [06:54:17] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [06:56:31] [02mw-config] 07dmehus commented on pull request 03#3761: Better canned responses - 13https://git.io/Jqnmm [06:59:05] PROBLEM - dbbackup1 Current Load on dbbackup1 is CRITICAL: CRITICAL - load average: 4.30, 3.86, 3.84 [07:01:05] PROBLEM - dbbackup1 Current Load on dbbackup1 is WARNING: WARNING - load average: 3.47, 3.74, 3.80 [07:04:03] Reception123, was the Fandom subdomain ^ minecraftwiki.fandom.com, or did you use the Miraheze `minecraftwiki` database name by mistake? [07:04:15] oh it was lol [07:04:28] there is a Fandom minecraftwiki :D [07:04:32] yes heh [07:04:47] heh cool [07:07:04] PROBLEM - dbbackup1 Current Load on dbbackup1 is CRITICAL: CRITICAL - load average: 4.81, 4.15, 3.92 [07:11:04] PROBLEM - dbbackup1 Current Load on dbbackup1 is WARNING: WARNING - load average: 3.42, 3.76, 3.81 [07:13:05] PROBLEM - dbbackup1 Current Load on dbbackup1 is CRITICAL: CRITICAL - load average: 4.62, 4.07, 3.92 [07:19:04] PROBLEM - dbbackup1 Current Load on dbbackup1 is WARNING: WARNING - load average: 3.63, 3.95, 3.93 [07:27:04] PROBLEM - dbbackup1 Current Load on dbbackup1 is CRITICAL: CRITICAL - load average: 4.07, 3.72, 3.75 [07:27:44] PROBLEM - dbbackup1 Check MariaDB Replication c2 on dbbackup1 is WARNING: MariaDB replication - both - WARNING - Slave_IO_Running state : Yes, Slave_SQL_Running state : Yes, Seconds_Behind_Master : 145s [07:29:45] PROBLEM - dbbackup1 Check MariaDB Replication c2 on dbbackup1 is CRITICAL: MariaDB replication - both - CRITICAL - Slave_IO_Running state : Yes, Slave_SQL_Running state : Yes, Seconds_Behind_Master : 205s [07:39:05] PROBLEM - dbbackup1 Current Load on dbbackup1 is WARNING: WARNING - load average: 3.16, 3.77, 3.89 [07:41:41] [02mw-config] 07Reception123 commented on pull request 03#3761: Better canned responses - 13https://git.io/Jqn3R [07:45:05] PROBLEM - dbbackup1 Current Load on dbbackup1 is CRITICAL: CRITICAL - load average: 4.18, 3.81, 3.83 [07:47:04] PROBLEM - dbbackup1 Current Load on dbbackup1 is WARNING: WARNING - load average: 3.86, 3.85, 3.85 [07:59:05] PROBLEM - dbbackup1 Current Load on dbbackup1 is CRITICAL: CRITICAL - load average: 4.24, 3.87, 3.79 [08:10:51] [02mw-config] 07dmehus commented on pull request 03#3761: Better canned responses - 13https://git.io/Jqnsp [08:13:04] PROBLEM - dbbackup1 Current Load on dbbackup1 is WARNING: WARNING - load average: 3.46, 3.94, 3.95 [08:29:04] PROBLEM - dbbackup1 Current Load on dbbackup1 is CRITICAL: CRITICAL - load average: 4.12, 3.72, 3.72 [08:35:05] PROBLEM - dbbackup1 Current Load on dbbackup1 is WARNING: WARNING - load average: 3.40, 3.85, 3.82 [08:43:05] PROBLEM - dbbackup1 Current Load on dbbackup1 is CRITICAL: CRITICAL - load average: 4.04, 3.87, 3.80 [08:45:05] PROBLEM - dbbackup1 Current Load on dbbackup1 is WARNING: WARNING - load average: 3.94, 3.87, 3.81 [08:51:05] PROBLEM - dbbackup1 Current Load on dbbackup1 is CRITICAL: CRITICAL - load average: 4.09, 3.86, 3.81 [09:42:22] PROBLEM - dbbackup2 Check MariaDB Replication c3 on dbbackup2 is CRITICAL: MariaDB replication - both - CRITICAL - Slave IO State not correct, slave stopped or replication broken! [09:44:21] PROBLEM - dbbackup2 Check MariaDB Replication c3 on dbbackup2 is UNKNOWN: NRPE: Unable to read output [09:46:21] PROBLEM - dbbackup2 Check MariaDB Replication c3 on dbbackup2 is CRITICAL: MariaDB replication - both - CRITICAL - Slave IO State not correct, slave stopped or replication broken! [09:46:48] RECOVERY - dbbackup2 MariaDB c3 on dbbackup2 is OK: Uptime: 87 Threads: 11 Questions: 534 Slow queries: 1 Opens: 30 Flush tables: 1 Open tables: 24 Queries per second avg: 6.137 [09:53:05] PROBLEM - dbbackup1 Current Load on dbbackup1 is WARNING: WARNING - load average: 3.02, 3.64, 3.94 [09:56:28] [02mw-config] 07R4356th commented on pull request 03#3761: Better canned responses - 13https://git.io/JqnWn [09:59:05] PROBLEM - dbbackup1 Current Load on dbbackup1 is CRITICAL: CRITICAL - load average: 4.80, 3.89, 3.91 [10:09:04] PROBLEM - dbbackup1 Current Load on dbbackup1 is WARNING: WARNING - load average: 3.79, 3.93, 3.98 [10:11:05] PROBLEM - dbbackup1 Current Load on dbbackup1 is CRITICAL: CRITICAL - load average: 4.40, 4.00, 3.99 [10:22:11] PROBLEM - dbbackup2 Current Load on dbbackup2 is WARNING: WARNING - load average: 3.50, 3.60, 2.87 [10:24:11] RECOVERY - dbbackup2 Current Load on dbbackup2 is OK: OK - load average: 1.83, 2.99, 2.74 [10:30:22] RECOVERY - dbbackup2 Check MariaDB Replication c3 on dbbackup2 is OK: MariaDB replication - both - OK - Slave_IO_Running state : Yes, Slave_SQL_Running state : Yes, Seconds_Behind_Master : 0s [10:51:05] PROBLEM - dbbackup1 Current Load on dbbackup1 is WARNING: WARNING - load average: 3.48, 3.57, 3.92 [10:51:19] PROBLEM - graylog2 Current Load on graylog2 is CRITICAL: CRITICAL - load average: 14.56, 7.42, 3.18 [10:57:04] PROBLEM - dbbackup1 Current Load on dbbackup1 is CRITICAL: CRITICAL - load average: 4.03, 3.84, 3.91 [10:59:05] PROBLEM - dbbackup1 Current Load on dbbackup1 is WARNING: WARNING - load average: 3.69, 3.78, 3.88 [11:01:06] PROBLEM - dbbackup1 Current Load on dbbackup1 is CRITICAL: CRITICAL - load average: 4.11, 3.98, 3.95 [11:03:05] PROBLEM - dbbackup1 Current Load on dbbackup1 is WARNING: WARNING - load average: 3.70, 3.89, 3.92 [11:05:04] PROBLEM - dbbackup1 Current Load on dbbackup1 is CRITICAL: CRITICAL - load average: 4.57, 4.07, 3.98 [11:11:11] [02mw-config] 07R4356th synchronize pull request 03#3761: Better canned responses - 13https://git.io/JqIWl [11:11:20] PROBLEM - graylog2 Current Load on graylog2 is WARNING: WARNING - load average: 0.60, 1.81, 3.89 [11:12:20] miraheze/mw-config - R4356th the build passed. [11:15:04] PROBLEM - dbbackup1 Current Load on dbbackup1 is WARNING: WARNING - load average: 3.38, 3.84, 3.97 [11:15:18] RECOVERY - graylog2 Current Load on graylog2 is OK: OK - load average: 0.36, 1.06, 3.11 [11:21:05] PROBLEM - dbbackup1 Current Load on dbbackup1 is CRITICAL: CRITICAL - load average: 4.22, 3.86, 3.90 [11:23:04] PROBLEM - dbbackup1 Current Load on dbbackup1 is WARNING: WARNING - load average: 3.94, 3.95, 3.94 [11:29:05] PROBLEM - dbbackup1 Current Load on dbbackup1 is CRITICAL: CRITICAL - load average: 4.94, 4.09, 3.97 [11:39:05] PROBLEM - dbbackup1 Current Load on dbbackup1 is WARNING: WARNING - load average: 3.59, 3.90, 3.99 [11:39:58] PROBLEM - dbbackup2 Current Load on dbbackup2 is CRITICAL: CRITICAL - load average: 4.49, 3.28, 2.28 [11:41:58] RECOVERY - dbbackup2 Current Load on dbbackup2 is OK: OK - load average: 1.01, 2.37, 2.06 [11:43:04] PROBLEM - dbbackup1 Current Load on dbbackup1 is CRITICAL: CRITICAL - load average: 4.44, 4.07, 4.03 [11:51:50] PROBLEM - dbbackup2 Check MariaDB Replication c3 on dbbackup2 is CRITICAL: MariaDB replication - both - CRITICAL - Slave_IO_Running state : Yes, Slave_SQL_Running state : Yes, Seconds_Behind_Master : 215s [11:51:51] PROBLEM - dbbackup2 Current Load on dbbackup2 is CRITICAL: CRITICAL - load average: 5.87, 3.69, 2.63 [11:53:46] RECOVERY - dbbackup2 Check MariaDB Replication c3 on dbbackup2 is OK: MariaDB replication - both - OK - Slave_IO_Running state : Yes, Slave_SQL_Running state : Yes, Seconds_Behind_Master : 31s [11:53:50] PROBLEM - dbbackup2 Current Load on dbbackup2 is WARNING: WARNING - load average: 3.47, 3.72, 2.79 [11:55:49] RECOVERY - dbbackup2 Current Load on dbbackup2 is OK: OK - load average: 1.69, 3.06, 2.67 [11:57:05] PROBLEM - dbbackup1 Current Load on dbbackup1 is WARNING: WARNING - load average: 3.67, 3.76, 3.96 [11:59:05] PROBLEM - dbbackup1 Current Load on dbbackup1 is CRITICAL: CRITICAL - load average: 4.03, 3.92, 4.00 [12:09:05] PROBLEM - dbbackup1 Current Load on dbbackup1 is WARNING: WARNING - load average: 3.21, 3.81, 3.96 [12:13:05] PROBLEM - dbbackup1 Current Load on dbbackup1 is CRITICAL: CRITICAL - load average: 4.13, 3.91, 3.96 [12:15:04] PROBLEM - dbbackup1 Current Load on dbbackup1 is WARNING: WARNING - load average: 3.87, 3.94, 3.97 [12:21:05] PROBLEM - dbbackup1 Current Load on dbbackup1 is CRITICAL: CRITICAL - load average: 4.39, 3.92, 3.93 [12:37:04] PROBLEM - dbbackup1 Current Load on dbbackup1 is WARNING: WARNING - load average: 2.85, 3.60, 3.93 [12:39:04] PROBLEM - dbbackup1 Current Load on dbbackup1 is CRITICAL: CRITICAL - load average: 4.12, 3.78, 3.96 [12:43:42] PROBLEM - dbbackup2 Check MariaDB Replication c3 on dbbackup2 is WARNING: MariaDB replication - both - WARNING - Slave_IO_Running state : Yes, Slave_SQL_Running state : Yes, Seconds_Behind_Master : 168s [12:44:35] PROBLEM - dbbackup2 Current Load on dbbackup2 is CRITICAL: CRITICAL - load average: 5.32, 3.35, 2.25 [12:45:04] PROBLEM - dbbackup1 Current Load on dbbackup1 is WARNING: WARNING - load average: 3.90, 3.95, 4.00 [12:46:35] RECOVERY - dbbackup2 Current Load on dbbackup2 is OK: OK - load average: 3.36, 3.19, 2.33 [12:47:04] PROBLEM - dbbackup1 Current Load on dbbackup1 is CRITICAL: CRITICAL - load average: 4.31, 4.11, 4.05 [12:47:36] PROBLEM - dbbackup2 Check MariaDB Replication c3 on dbbackup2 is CRITICAL: MariaDB replication - both - CRITICAL - Slave_IO_Running state : Yes, Slave_SQL_Running state : Yes, Seconds_Behind_Master : 251s [12:52:35] PROBLEM - dbbackup2 Current Load on dbbackup2 is WARNING: WARNING - load average: 3.43, 3.84, 2.94 [12:54:35] RECOVERY - dbbackup2 Current Load on dbbackup2 is OK: OK - load average: 2.44, 3.26, 2.83 [12:55:26] PROBLEM - dbbackup2 Check MariaDB Replication c3 on dbbackup2 is WARNING: MariaDB replication - both - WARNING - Slave_IO_Running state : Yes, Slave_SQL_Running state : Yes, Seconds_Behind_Master : 147s [12:57:23] RECOVERY - dbbackup2 Check MariaDB Replication c3 on dbbackup2 is OK: MariaDB replication - both - OK - Slave_IO_Running state : Yes, Slave_SQL_Running state : Yes, Seconds_Behind_Master : 57s [13:22:35] PROBLEM - dbbackup2 Current Load on dbbackup2 is WARNING: WARNING - load average: 3.82, 2.82, 2.28 [13:22:59] PROBLEM - dbbackup2 Check MariaDB Replication c3 on dbbackup2 is CRITICAL: MariaDB replication - both - CRITICAL - Slave_IO_Running state : Yes, Slave_SQL_Running state : Yes, Seconds_Behind_Master : 211s [13:24:35] PROBLEM - dbbackup2 Current Load on dbbackup2 is CRITICAL: CRITICAL - load average: 4.30, 3.62, 2.66 [13:24:57] RECOVERY - dbbackup2 Check MariaDB Replication c3 on dbbackup2 is OK: MariaDB replication - both - OK - Slave_IO_Running state : Yes, Slave_SQL_Running state : Yes, Seconds_Behind_Master : 46s [13:26:36] RECOVERY - dbbackup2 Current Load on dbbackup2 is OK: OK - load average: 0.92, 2.59, 2.39 [14:24:22] PROBLEM - mail2 IMAP on mail2 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:39:00] RECOVERY - mail2 IMAP on mail2 is OK: IMAP OK - 0.006 second response time on 51.195.236.253 port 143 [* OK [CAPABILITY IMAP4rev1 SASL-IR LOGIN-REFERRALS ID ENABLE IDLE LITERAL+ STARTTLS LOGINDISABLED] Dovecot (Debian) ready.] [15:13:05] PROBLEM - dbbackup1 Current Load on dbbackup1 is WARNING: WARNING - load average: 3.57, 3.69, 3.93 [15:15:05] PROBLEM - dbbackup1 Current Load on dbbackup1 is CRITICAL: CRITICAL - load average: 4.41, 3.94, 3.99 [15:54:11] [02mw-config] 07paladox closed pull request 03#3771: Use in_array rather then multiple || on $_SERVER['REMOTE_ADDR'] - 13https://git.io/JqnTf [15:54:12] [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Jqn18 [15:54:14] [02miraheze/mw-config] 07Universal-Omega 03f69913e - Use in_array rather then multiple || on $_SERVER['REMOTE_ADDR'] (#3771) [15:54:18] [02mw-config] 07paladox deleted branch 03Universal-Omega-patch-1 - 13https://git.io/vbvb3 [15:54:19] [02miraheze/mw-config] 07paladox deleted branch 03Universal-Omega-patch-1 [15:55:34] miraheze/mw-config - paladox the build passed. [15:56:41] PROBLEM - mail2 IMAP on mail2 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:58:38] RECOVERY - mail2 IMAP on mail2 is OK: IMAP OK - 0.016 second response time on 51.195.236.253 port 143 [* OK [CAPABILITY IMAP4rev1 SASL-IR LOGIN-REFERRALS ID ENABLE IDLE LITERAL+ STARTTLS LOGINDISABLED] Dovecot (Debian) ready.] [16:03:36] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JqnME [16:03:37] [02miraheze/mw-config] 07Universal-Omega 0383d710c - Prepare comment for more types [16:04:04] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JqnM2 [16:04:05] [02miraheze/mw-config] 07Universal-Omega 035fac6ff - Fix comment [16:05:13] miraheze/mw-config - Universal-Omega the build passed. [16:05:14] miraheze/mw-config - Universal-Omega the build passed. [16:05:54] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JqnMH [16:05:55] [02miraheze/mw-config] 07Universal-Omega 033a33e15 - Add other types to comment [16:06:25] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JqnMA [16:06:27] [02miraheze/mw-config] 07Universal-Omega 039e8475b - Fix comment [16:07:00] miraheze/mw-config - Universal-Omega the build passed. [16:07:33] miraheze/mw-config - Universal-Omega the build passed. [16:07:39] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JqnDf [16:07:40] [02miraheze/mw-config] 07Universal-Omega 037a7cabf - Fix documentation for database type [16:08:33] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JqnDt [16:08:35] [02miraheze/mw-config] 07Universal-Omega 0371c2793 - Fix documentation for database type [16:08:43] miraheze/mw-config - Universal-Omega the build passed. [16:09:38] miraheze/mw-config - Universal-Omega the build passed. [16:12:15] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JqnDg [16:12:17] [02miraheze/mw-config] 07Universal-Omega 03ed00dee - Add documentation for 'constant' [16:13:25] miraheze/mw-config - Universal-Omega the build passed. [16:13:54] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JqnDX [16:13:55] [02miraheze/mw-config] 07Universal-Omega 039747c4e - Fix documentation for 'constant' [16:15:00] miraheze/mw-config - Universal-Omega the build passed. [16:25:40] PROBLEM - mail2 IMAP on mail2 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:44:36] RECOVERY - mail2 IMAP on mail2 is OK: IMAP OK - 0.008 second response time on 51.195.236.253 port 143 [* OK [CAPABILITY IMAP4rev1 SASL-IR LOGIN-REFERRALS ID ENABLE IDLE LITERAL+ STARTTLS LOGINDISABLED] Dovecot (Debian) ready.] [16:53:05] PROBLEM - dbbackup1 Current Load on dbbackup1 is WARNING: WARNING - load average: 3.44, 3.76, 3.98 [16:59:04] PROBLEM - dbbackup1 Current Load on dbbackup1 is CRITICAL: CRITICAL - load average: 4.31, 3.96, 3.99 [17:23:05] PROBLEM - dbbackup1 Current Load on dbbackup1 is WARNING: WARNING - load average: 3.38, 3.78, 3.98 [17:24:32] PROBLEM - ping6 on dbbackup1 is WARNING: PING WARNING - Packet loss = 0%, RTA = 226.28 ms [17:24:57] PROBLEM - ping6 on ns1 is CRITICAL: PING CRITICAL - Packet loss = 28%, RTA = 215.98 ms [17:25:08] PROBLEM - ping6 on dbbackup2 is WARNING: PING WARNING - Packet loss = 0%, RTA = 218.82 ms [17:26:55] PROBLEM - ping6 on ns1 is WARNING: PING WARNING - Packet loss = 0%, RTA = 218.01 ms [17:27:05] PROBLEM - dbbackup1 Current Load on dbbackup1 is CRITICAL: CRITICAL - load average: 4.57, 3.98, 3.99 [17:35:09] PROBLEM - ping6 on dbbackup2 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 222.93 ms [17:37:09] PROBLEM - ping6 on dbbackup2 is WARNING: PING WARNING - Packet loss = 0%, RTA = 217.78 ms [17:59:11] RECOVERY - ping6 on dbbackup2 is OK: PING OK - Packet loss = 0%, RTA = 119.77 ms [18:03:20] PROBLEM - ping6 on dbbackup2 is WARNING: PING WARNING - Packet loss = 0%, RTA = 156.50 ms [18:08:34] PROBLEM - mail2 IMAP on mail2 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:12:11] [02mw-config] 07R4356th synchronize pull request 03#3761: Better canned responses - 13https://git.io/JqIWl [18:13:16] miraheze/mw-config - R4356th the build passed. [18:16:38] [02mw-config] 07R4356th synchronize pull request 03#3761: Better canned responses - 13https://git.io/JqIWl [18:16:52] RECOVERY - mail2 IMAP on mail2 is OK: IMAP OK - 0.038 second response time on 51.195.236.253 port 143 [* OK [CAPABILITY IMAP4rev1 SASL-IR LOGIN-REFERRALS ID ENABLE IDLE LITERAL+ STARTTLS LOGINDISABLED] Dovecot (Debian) ready.] [18:17:05] PROBLEM - dbbackup1 Current Load on dbbackup1 is WARNING: WARNING - load average: 3.29, 3.45, 3.91 [18:17:22] RECOVERY - ping6 on dbbackup2 is OK: PING OK - Packet loss = 0%, RTA = 107.21 ms [18:17:41] miraheze/mw-config - R4356th the build passed. [18:18:18] [02mw-config] 07R4356th synchronize pull request 03#3761: Better canned responses - 13https://git.io/JqIWl [18:19:04] PROBLEM - dbbackup1 Current Load on dbbackup1 is CRITICAL: CRITICAL - load average: 4.03, 3.61, 3.91 [18:19:24] miraheze/mw-config - R4356th the build passed. [18:21:04] PROBLEM - dbbackup1 Current Load on dbbackup1 is WARNING: WARNING - load average: 3.12, 3.42, 3.81 [18:25:05] PROBLEM - dbbackup1 Current Load on dbbackup1 is CRITICAL: CRITICAL - load average: 5.19, 4.11, 3.98 [18:25:11] PROBLEM - mail2 IMAP on mail2 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:26:34] dmehus: that's not the password wrong message [18:26:50] Unless they've set their wiki to non English and got a translate plugin on [18:27:04] PROBLEM - dbbackup1 Current Load on dbbackup1 is WARNING: WARNING - load average: 3.57, 3.86, 3.91 [18:29:04] PROBLEM - dbbackup1 Current Load on dbbackup1 is CRITICAL: CRITICAL - load average: 5.04, 4.24, 4.03 [18:32:09] dmehus: i found the message [18:32:28] RECOVERY - ping6 on ns1 is OK: PING OK - Packet loss = 0%, RTA = 116.18 ms [18:32:51] RhinosF1, ack, oh okay, what's wrong then? [18:33:09] That's my next step [18:33:13] authmanager-authn-no-primary [18:33:17] ah ok [18:35:35] dmehus: Login failed in primary authentication because no provider accepted [18:37:00] Universal_Omega: https://github.com/wikimedia/mediawiki/blob/600f75010c27a53c967539c2622ce2e497360ed4/includes/auth/AuthManager.php#L535 [18:37:01] [ mediawiki/AuthManager.php at 600f75010c27a53c967539c2622ce2e497360ed4 · wikimedia/mediawiki · GitHub ] - github.com [18:37:12] That's the only place that message is used [18:38:10] paladox: https://grafana.miraheze.org/d/LaSguOMZz/varnish?from=now-24h&to=now that's looking much better now :) [18:38:11] [ Grafana ] - grafana.miraheze.org [18:38:47] RhinosF1: this is related to https://phabricator.miraheze.org/T6946 ? [18:38:48] [ ⚓ T6946 Login problem in specified wiki. ] - phabricator.miraheze.org [18:39:23] Universal_Omega: yep [18:39:31] I'm asking in #mediawiki [18:41:42] RhinosF1: hmm, I think that is some sort of password wrong error though not 100% sure. If that's not it, I have no idea. [18:42:11] Universal_Omega: tgr says CA gave a PASS response [18:42:28] RECOVERY - ping6 on dbbackup1 is OK: PING OK - Packet loss = 0%, RTA = 113.39 ms [18:42:43] Password wrong should be FAILS [18:42:46] FAIL* [18:43:49] Well that's interesting. I wonder why that'd happen then. [18:43:54] Which getting my password wrong gives Incorrect username or password entered. Please try again. [18:44:07] So it's not that which I didn't think it would be [18:45:21] RhinosF1, hrm, strange [18:47:50] dmehus: CentralAuth is refusing to accept the request to authenticate. That's all we know so far. [18:47:56] It's not failing [18:48:00] It's skipping it [18:48:06] Oh [18:48:08] Then there's no providers left [18:48:37] so yeah, seems to be some sort of possible upstream bug related to non-English wikis and CentralAuth? [18:49:40] dmehus: no English has nothing to do with it or it'd happen to every non English wiki I think. [18:50:04] Also, it works for me when I login through that wiki. [18:50:11] dmehus: there's nothing to indicate that whatsoever [18:50:18] Universal_Omega, yeah, true. weird [18:50:21] It's as far as I know 1 user on 1 wiki [18:51:17] Very much doubt it's because it's a private wiki, but just noting that is a private wiki [18:52:22] I am logged in fine on that wiki though [18:52:38] And they are logged into every other wiki from what the ticket suggests [18:54:04] RhinosF1, yeah I'm logged in fine as well, but did you try logging out and logging in through that wiki? [18:54:30] dmehus: no but feel free to try [18:54:55] I did try that. [18:55:04] RhinosF1, wouldn't mind if you tried :P [18:55:13] Universal_Omega, oh okay then, thanks! :) [18:55:21] Ty Universal_Omega [18:56:07] I logged in through incognito on my main account as well as my alternative test account, and both works. [18:56:38] This is a very weird issue. [18:56:53] Universal_Omega, ah, okay, that sounds like a reasonable test then. I wonder if it could be a case of the user just having the wrong password saved in their saved passwords? [18:57:33] Well I tried to enter a wrong password for me and it just gave the normal error. [18:57:45] dmehus: no [18:58:01] I tried entering my wrong password and got told incorrect password [18:59:06] RhinosF1, ack, okay. Hrm, this is very strange, as his username is all Latin alphabetic characters like you, I, and Universal_Omega [18:59:15] dmehus: the suggestion is it could be corrupt data somewhere [18:59:21] Have they been renamed recently? [18:59:27] RhinosF1, yes [18:59:59] RhinosF1, https://meta.miraheze.org/wiki/Special:GlobalRenameProgress?username=SHEIKH [19:00:00] [ Global rename progress - Miraheze Meta ] - meta.miraheze.org [19:00:05] That's likely the cause then. I'd think. Something must've went wrong with rename. [19:00:09] What might the proposed remedy be in that case? [19:00:34] dmehus: I work out what data is corrupt [19:00:52] Universal_Omega, yeah, except they were able to use that wiki after they were renamed, so it is strange [19:01:06] RhinosF1, ack, okay [19:01:16] Maybe dmehus, but maybe they just got logged out now. Then the issue was caused. [19:02:14] Universal_Omega, yeah, so likely/possibly corrupt data on that wiki; just a matter of determining which table(s) are affected [19:04:57] dmehus: I can do some poking this evening I guess [19:07:51] My guess would be local* [19:07:54] tables [19:08:22] RhinosF1, ack, okay. Thanks! The user does have their alternate account, SHEKH, which they might be able to use. Maybe we should ask them if they can login to that account on that wiki okay? [19:08:39] then we can know if it's localized to only one user account, potentially [19:08:50] dmehus: given 3 people have said they can then I'm gonna assume the answer is yes to that [19:08:56] okay [19:09:43] Universal_Omega: do you know CA schema well enough to look for corrupt data or do you want to wait for me? [19:10:07] I can try like sometime between 9:30-10:30PM UK time [19:12:38] RhinosF1, that's 1:30-2:30 PM Pacific time, right? [19:12:57] It's just over 2 hours away [19:13:01] 2-3 hours [19:13:04] It's 19:13 now [19:13:07] oh yeah [19:13:13] that's what I meant [19:13:16] thanks :) [19:15:43] paladox: can we change log level to debug for that wiki? [19:15:56] Back in an hour [19:21:23] RECOVERY - mail2 IMAP on mail2 is OK: IMAP OK - 0.010 second response time on 51.195.236.253 port 143 [* OK [CAPABILITY IMAP4rev1 SASL-IR LOGIN-REFERRALS ID ENABLE IDLE LITERAL+ STARTTLS LOGINDISABLED] Dovecot (Debian) ready.] [19:35:33] RhinosF1: currently away, it'd be about that time before I could also. [19:37:38] But I'd check `localuser` table first. As that's what I think it'd likely be. [19:41:42] PROBLEM - mail2 IMAP on mail2 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:48:07] Universal_Omega: yeah I agree [19:59:05] [02MirahezeMagic] 07paladox reviewed pull request 03#214 commit - 13https://git.io/Jqnh0 [20:05:53] [02MirahezeMagic] 07paladox reviewed pull request 03#214 commit - 13https://git.io/JqnhF [20:07:26] [02MirahezeMagic] 07Universal-Omega reviewed pull request 03#214 commit - 13https://git.io/Jqnjf [20:08:02] [02MirahezeMagic] 07paladox reviewed pull request 03#214 commit - 13https://git.io/JqnjJ [20:08:36] [02MirahezeMagic] 07Universal-Omega reviewed pull request 03#214 commit - 13https://git.io/Jqnjk [20:23:06] PROBLEM - dbbackup1 Current Load on dbbackup1 is WARNING: WARNING - load average: 3.31, 3.66, 3.95 [20:24:42] [02miraheze/MirahezeMagic] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://git.io/JqceL [20:24:43] [02miraheze/MirahezeMagic] 07Universal-Omega 033502f2a - Add sitenotice messages to qqq.json [20:24:45] [02MirahezeMagic] 07Universal-Omega synchronize pull request 03#214: Move SiteNoticeAfter hooks from mw-config to MirahezeMagic - 13https://git.io/JtbsK [20:25:04] PROBLEM - dbbackup1 Current Load on dbbackup1 is CRITICAL: CRITICAL - load average: 5.19, 4.17, 4.10 [20:25:45] miraheze/MirahezeMagic - Universal-Omega the build passed. [20:33:11] [02miraheze/MirahezeMagic] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±4] 13https://git.io/Jqceo [20:33:12] [02miraheze/MirahezeMagic] 07Universal-Omega 03d128e32 - Move SiteNoticeAfter hooks from mw-config to MirahezeMagic (#214) [20:33:14] [02MirahezeMagic] 07Universal-Omega closed pull request 03#214: Move SiteNoticeAfter hooks from mw-config to MirahezeMagic - 13https://git.io/JtbsK [20:33:15] [02MirahezeMagic] 07Universal-Omega deleted branch 03Universal-Omega-patch-1 - 13https://git.io/fQRGX [20:33:17] [02miraheze/MirahezeMagic] 07Universal-Omega deleted branch 03Universal-Omega-patch-1 [20:34:13] miraheze/MirahezeMagic - Universal-Omega the build passed. [20:59:04] PROBLEM - dbbackup1 Current Load on dbbackup1 is WARNING: WARNING - load average: 3.73, 3.75, 3.99 [21:01:05] PROBLEM - dbbackup1 Current Load on dbbackup1 is CRITICAL: CRITICAL - load average: 4.62, 4.07, 4.07 [21:35:54] SPF|Cloud all backups are running on dbbbackups [21:36:30] that is awesome, thanks :) [21:36:54] the clusters on dbbackup1 are lagged, though [21:39:12] even though dbb1 and dbb2 have the same number of IOPS, hmm [21:39:21] Paladox: https://phabricator.miraheze.org/T4425#137251 what did Rhinos say? [21:39:22] [ ⚓ T4425 Fix all mysql tables that are using latin rather then utf8mb4 ] - phabricator.miraheze.org [21:39:36] He said that they use the mediawiki default [21:40:03] Why didn’t you say that then? :) [21:40:44] Oh, adding. [21:40:48] done [21:41:16] So they use latin? [21:44:49] it uses https://github.com/wikimedia/mediawiki/blob/d48d5292a61c1341f86e1f7611e5df169cb4cf11/includes/DefaultSettings.php#L2143, so i'm presuming so. (if that's the correct config) [21:44:50] [ mediawiki/DefaultSettings.php at d48d5292a61c1341f86e1f7611e5df169cb4cf11 · wikimedia/mediawiki · GitHub ] - github.com [21:46:27] Mkay [21:46:53] Seems odd that WMF would use Latin on non Latin text though but okay [21:49:37] https://www.irccloud.com/pastebin/ztKXC8TN/ [21:49:37] [ Snippet | IRCCloud ] - www.irccloud.com [21:51:45] they use utf8? [21:51:51] >utf8_general_ci [21:52:02] in hindsight, I'm not sure that is the right variable [21:52:05] do they use unicode on any of the tables [21:53:00] when running show variables;, we have latin1_swedish_ci on collation_(database|server), but their replicas have binary [21:54:42] https://www.mediawiki.org/wiki/Manual:$wgDBTableOptions the default is binary [21:54:42] [ Manual:$wgDBTableOptions - MediaWiki ] - www.mediawiki.org [21:54:53] with a reason: "No matter which value is selected here, MediaWiki internally always produces data in UTF-8 encoding to be saved in the database. Thus a conversion is never needed. Since binary charsets prevent MySQL from doing charset conversions, which otherwise unnecessarily might take place, the recommended setting is the default setting." [21:58:43] hmm [21:58:52] SPF|Cloud what should we do? [22:05:34] eval says wgDBTableOptions has the binary charset [22:05:48] so we have a mix of latin1 and binary? [22:09:10] i think the db server has latin1 set as the default. Though don't we apply tables.sql manually plus other sql files? Could it be that wgDBTableOptions isn't being set for the sql? By that i mean "binary" not being applied by replacing wgDBTableOptions within the sql [22:09:57] latin1 should be the default if no charset is specified, but for mediawiki, the default should be overridden usign wgDBTableOptions [22:10:33] just checked the most recently created wiki, it's using latin1 [22:10:49] so the default options are not overridden by CreateWiki [22:11:34] https://github.com/wikimedia/mediawiki/blob/467a4f32b0475fd2840264ada94e3ce87518d22a/includes/installer/MysqlInstaller.php#L622 and that's why [22:11:34] [ mediawiki/MysqlInstaller.php at 467a4f32b0475fd2840264ada94e3ce87518d22a · wikimedia/mediawiki · GitHub ] - github.com [22:12:28] PROBLEM - cp12 Disk Space on cp12 is WARNING: DISK WARNING - free space: / 4228 MB (10% inode=96%); [22:15:32] SPF|Cloud hmm? [22:16:01] we do not substitute the wgDBTableOptions comment that's inside each CREATE TABLE query [22:16:16] hence tables are created using whatever mariadb's default charset is, not mediawiki's [22:16:51] oh [22:19:29] that's the real bug :) [22:20:25] SPF|Cloud do you know the fix? :) [22:21:26] not at first glance [22:22:06] and keep in mind altering database tables in production is a dangerous operation [22:23:47] but until 1.33 the default charset was not set in mediawiki, so I presume they have a migration checklist? [22:24:50] https://github.com/wikimedia/mediawiki/commit/9bea45eab77f731aaba56b69aa9432f41d199ef8 [22:24:51] [ Use binary charset in default table options · wikimedia/mediawiki@9bea45e · GitHub ] - github.com [22:25:16] "and is always set to binary now that the installer UI" -> when was that added? [22:27:08] SPF|Cloud so we have to set https://github.com/wikimedia/mediawiki/blob/85754996e13c7498673a7c4d81db50bf1026843b/includes/libs/rdbms/database/Database.php#L5088 [22:27:09] [ mediawiki/Database.php at 85754996e13c7498673a7c4d81db50bf1026843b · wikimedia/mediawiki · GitHub ] - github.com [22:27:16] mediawiki only supports utf-8 or binary,so we've been running miraheze on unsupported charsets since day 1? [22:27:23] here https://github.com/miraheze/CreateWiki/blob/master/includes/WikiManager.php#L100 [22:27:23] [ CreateWiki/WikiManager.php at master · miraheze/CreateWiki · GitHub ] - github.com [22:29:53] possibly, then use replaceVars() to replace the vars [22:30:42] please document this on the task and only try it when creating test wikis [22:31:35] running these ALTERs on all wikis with a thorough review is a bad idea, with my lack of knowledge regarding character sets / collations, I do not feel comfortable performing such changes [22:32:33] SPF|Cloud: do we keep logs of inserts & updates ? [22:32:46] no [22:33:22] technically, we have the binary logs, which stores all database updates, but we only store them for four days [22:34:13] SPF|Cloud: so we'd have them for 8 March [22:34:19] yes [22:34:33] what is your request? [22:35:16]