[00:03:22] PROBLEM - wiki.jafaron.net - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.jafaron.net could not be found [00:03:23] PROBLEM - towarzystwo.prawa.kredytowego.pl - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for towarzystwo.prawa.kredytowego.pl could not be found [00:03:24] PROBLEM - wiki.shaazzz.ir - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.shaazzz.ir could not be found [00:03:27] PROBLEM - cloud2 Current Load on cloud2 is CRITICAL: CRITICAL - load average: 32.02, 27.02, 19.48 [00:03:31] PROBLEM - electowiki.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for electowiki.org could not be found [00:03:36] PROBLEM - wiki.minecraftathome.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.minecraftathome.com could not be found [00:05:27] PROBLEM - cloud2 Current Load on cloud2 is WARNING: WARNING - load average: 16.00, 22.47, 18.72 [00:07:28] RECOVERY - cloud2 Current Load on cloud2 is OK: OK - load average: 13.78, 19.79, 18.20 [00:10:10] RECOVERY - electowiki.org - reverse DNS on sslhost is OK: rDNS OK - electowiki.org reverse DNS resolves to cp6.miraheze.org [00:10:18] RECOVERY - towarzystwo.prawa.kredytowego.pl - reverse DNS on sslhost is OK: rDNS OK - towarzystwo.prawa.kredytowego.pl reverse DNS resolves to cp7.miraheze.org [00:10:19] RECOVERY - wiki.shaazzz.ir - reverse DNS on sslhost is OK: rDNS OK - wiki.shaazzz.ir reverse DNS resolves to cp7.miraheze.org [00:10:21] RECOVERY - wiki.jafaron.net - reverse DNS on sslhost is OK: rDNS OK - wiki.jafaron.net reverse DNS resolves to cp6.miraheze.org [00:10:24] RECOVERY - wiki.minecraftathome.com - reverse DNS on sslhost is OK: rDNS OK - wiki.minecraftathome.com reverse DNS resolves to cp7.miraheze.org [00:18:51] Hello limpd! If you have any questions, feel free to ask and someone should answer soon. [00:21:09] My wiki in operation suddenly shows the message below today and cannot enter. What is the problem and what can I do? [00:21:22] MediaWiki internal error. [00:21:24] What wiki? [00:22:26] https://limpd.miraheze.org/ [00:23:08] Looking [00:23:26] Thank you [00:31:03] miraheze/mw-config/master/58df7a4 - paladox The build passed. https://travis-ci.org/miraheze/mw-config/builds/748438788 [00:37:14] !log INSERT INTO slot_roles (role_id, role_name) VALUES (1, 'main'); on limpdwiki [00:37:19] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:37:31] limpd: fixed [00:37:48] Oh, Thank you! [00:38:07] What was the problem? [00:39:48] limpd: Some database issue [00:40:45] Oh, OK! Thank you! Have a good day [00:40:57] No problem. [00:59:52] !log sudo -u www-data rm /srv/mediawiki/w/cache/lggyaanipediawiki.json on mw* and jobrunner* [00:59:55] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:29:51] miraheze/MirahezeMagic/Universal-Omega-patch-1/ae561ee - Universal Omega The build passed. https://travis-ci.org/miraheze/MirahezeMagic/builds/748440507 [02:39:14] PROBLEM - db7 Check MariaDB Replication on db7 is CRITICAL: MariaDB replication - both - CRITICAL - Slave_IO_Running state : Yes, Slave_SQL_Running state : Yes, Seconds_Behind_Master : 245s [02:43:14] RECOVERY - db7 Check MariaDB Replication on db7 is OK: MariaDB replication - both - OK - Slave_IO_Running state : Yes, Slave_SQL_Running state : Yes, Seconds_Behind_Master : 0s [06:02:35] RECOVERY - bacula2 APT on bacula2 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [06:12:08] RECOVERY - cloud2 APT on cloud2 is OK: APT OK: 99 packages available for upgrade (0 critical updates). [06:12:28] RECOVERY - db7 APT on db7 is OK: APT OK: 50 packages available for upgrade (0 critical updates). [06:13:38] RECOVERY - rdb1 APT on rdb1 is OK: APT OK: 24 packages available for upgrade (0 critical updates). [06:13:47] RECOVERY - rdb2 APT on rdb2 is OK: APT OK: 24 packages available for upgrade (0 critical updates). [06:26:12] RECOVERY - cloud1 APT on cloud1 is OK: APT OK: 98 packages available for upgrade (0 critical updates). [06:26:53] RECOVERY - gluster1 APT on gluster1 is OK: APT OK: 28 packages available for upgrade (0 critical updates). [06:40:13] RECOVERY - db13 APT on db13 is OK: APT OK: 50 packages available for upgrade (0 critical updates). [06:47:28] RECOVERY - gluster2 APT on gluster2 is OK: APT OK: 32 packages available for upgrade (0 critical updates). [06:49:07] RECOVERY - db12 APT on db12 is OK: APT OK: 50 packages available for upgrade (0 critical updates). [06:53:15] RECOVERY - cloud3 APT on cloud3 is OK: APT OK: 78 packages available for upgrade (0 critical updates). [07:01:40] RECOVERY - db11 APT on db11 is OK: APT OK: 50 packages available for upgrade (0 critical updates). [08:03:13] PROBLEM - wooriwiki.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wooriwiki.com could not be found [08:35:15] PROBLEM - wooriwiki.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:15:41] [02miraheze/ssl] 07Reception123 pushed 031 commit to 03master [+0/-1/±1] 13https://git.io/JIzOY [09:15:42] [02miraheze/ssl] 07Reception123 03baa9f40 - rm wooriwiki.com cert per request [09:21:44] [02miraheze/ssl] 07Reception123 pushed 031 commit to 03master [+1/-0/±1] 13https://git.io/JIzOp [09:21:46] [02miraheze/ssl] 07Reception123 037cc9029 - add wiki.cyberfurs.org cert [09:49:57] [02miraheze/ssl] 07Reception123 pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JIzny [09:49:59] [02miraheze/ssl] 07Reception123 039bafd40 - regen wiki.cyberfurs.org cert [09:56:17] [02miraheze/dns] 07Reception123 pushed 031 commit to 03master [+0/-1/±0] 13https://git.io/JIzcu [09:56:19] [02miraheze/dns] 07Reception123 034e505fb - remove wooriwiki.com zone requested, SSL no longer used [10:01:38] PROBLEM - cp7 Puppet on cp7 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[nginx-syntax] [10:29:40] RECOVERY - cp7 Puppet on cp7 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [14:14:32] I wonder if it had anything to do with Cp9 then. I would be a little surprised if that appears to be the case though. [15:31:31] Could be related to the general load/traffic spike issues with Cp9 (it's our only North American cache proxy, and it's located in a suburb of Montreal, Quebec, Canada), but that time of day doesn't normally see traffic spikes, afaik, so I tend to think as there were Debian updates being performed on all CPs at the time, it could've been a temporary issue related to certain CPs being rebooted and traffic being rerouted and "pooled" (I [15:31:32] think is the correct terminology) to the nearest CPs, at least I'm hoping that's all it was. 😉 [16:38:06] @dmehus Do you have a moment? [16:38:13] @Reception123 as well [16:38:30] DM me on Discord when you have the chance [16:41:32] PROBLEM - phab1 APT on phab1 is CRITICAL: APT CRITICAL: 7 packages available for upgrade (7 critical updates). [16:42:17] PROBLEM - gluster2 APT on gluster2 is CRITICAL: APT CRITICAL: 39 packages available for upgrade (7 critical updates). [16:43:09] PROBLEM - cp3 APT on cp3 is CRITICAL: APT CRITICAL: 7 packages available for upgrade (7 critical updates). [16:44:03] PROBLEM - puppet2 APT on puppet2 is CRITICAL: APT CRITICAL: 7 packages available for upgrade (7 critical updates). [16:46:47] PROBLEM - rdb2 APT on rdb2 is CRITICAL: APT CRITICAL: 31 packages available for upgrade (7 critical updates). [16:49:56] PROBLEM - graylog1 APT on graylog1 is CRITICAL: APT CRITICAL: 8 packages available for upgrade (7 critical updates). [16:50:27] PROBLEM - mon1 APT on mon1 is CRITICAL: APT CRITICAL: 7 packages available for upgrade (7 critical updates). [16:50:40] PROBLEM - bacula2 APT on bacula2 is CRITICAL: APT CRITICAL: 7 packages available for upgrade (7 critical updates). [16:50:41] PROBLEM - cloud3 APT on cloud3 is CRITICAL: APT CRITICAL: 85 packages available for upgrade (7 critical updates). [16:50:44] PROBLEM - ns2 APT on ns2 is CRITICAL: APT CRITICAL: 7 packages available for upgrade (7 critical updates). [16:50:51] PROBLEM - db11 APT on db11 is CRITICAL: APT CRITICAL: 57 packages available for upgrade (7 critical updates). [16:51:51] PROBLEM - db12 APT on db12 is CRITICAL: APT CRITICAL: 57 packages available for upgrade (7 critical updates). [16:52:02] PROBLEM - rdb1 APT on rdb1 is CRITICAL: APT CRITICAL: 31 packages available for upgrade (7 critical updates). [16:53:41] PROBLEM - cloud2 APT on cloud2 is CRITICAL: APT CRITICAL: 106 packages available for upgrade (7 critical updates). [16:53:43] PROBLEM - ldap1 APT on ldap1 is CRITICAL: APT CRITICAL: 7 packages available for upgrade (7 critical updates). [16:53:47] PROBLEM - db13 APT on db13 is CRITICAL: APT CRITICAL: 57 packages available for upgrade (7 critical updates). [16:56:52] PROBLEM - db7 APT on db7 is CRITICAL: APT CRITICAL: 57 packages available for upgrade (7 critical updates). [16:57:08] PROBLEM - ns1 APT on ns1 is CRITICAL: APT CRITICAL: 7 packages available for upgrade (7 critical updates). [16:57:52] PROBLEM - gluster1 APT on gluster1 is CRITICAL: APT CRITICAL: 35 packages available for upgrade (7 critical updates). [17:01:05] PROBLEM - cp7 APT on cp7 is CRITICAL: APT CRITICAL: 7 packages available for upgrade (7 critical updates). [17:03:06] PROBLEM - services1 APT on services1 is CRITICAL: APT CRITICAL: 7 packages available for upgrade (7 critical updates). [17:03:07] PROBLEM - services2 APT on services2 is CRITICAL: APT CRITICAL: 7 packages available for upgrade (7 critical updates). [17:04:11] PROBLEM - cp6 APT on cp6 is CRITICAL: APT CRITICAL: 7 packages available for upgrade (7 critical updates). [17:04:30] PROBLEM - jobrunner2 APT on jobrunner2 is CRITICAL: APT CRITICAL: 7 packages available for upgrade (7 critical updates). [17:04:42] PROBLEM - mw7 APT on mw7 is CRITICAL: APT CRITICAL: 7 packages available for upgrade (7 critical updates). [17:04:47] PROBLEM - test2 APT on test2 is CRITICAL: APT CRITICAL: 7 packages available for upgrade (7 critical updates). [17:05:08] ^ paladox oh wow what's up with all these updates [17:05:20] PROBLEM - mw5 APT on mw5 is CRITICAL: APT CRITICAL: 7 packages available for upgrade (7 critical updates). [17:05:23] PROBLEM - jobrunner1 APT on jobrunner1 is CRITICAL: APT CRITICAL: 7 packages available for upgrade (7 critical updates). [17:06:04] PROBLEM - mw4 APT on mw4 is CRITICAL: APT CRITICAL: 7 packages available for upgrade (7 critical updates). [17:06:17] PROBLEM - cloud1 APT on cloud1 is CRITICAL: APT CRITICAL: 105 packages available for upgrade (7 critical updates). [17:06:28] PROBLEM - mw6 APT on mw6 is CRITICAL: APT CRITICAL: 7 packages available for upgrade (7 critical updates). [17:06:30] PROBLEM - mail1 APT on mail1 is CRITICAL: APT CRITICAL: 7 packages available for upgrade (7 critical updates). [17:07:29] PROBLEM - cp9 APT on cp9 is CRITICAL: APT CRITICAL: 7 packages available for upgrade (7 critical updates). [17:22:22] PROBLEM - ping6 on bacula2 is CRITICAL: PING CRITICAL - Packet loss = 100% [17:23:50] Hello nawis! If you have any questions, feel free to ask and someone should answer soon. [17:24:23] RECOVERY - ping6 on bacula2 is OK: PING OK - Packet loss = 0%, RTA = 74.04 ms [17:40:30] !log deleted emails from PhabricatorManiphestApplication, no longer in use [17:40:33] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:41:43] PROBLEM - test2 Puppet on test2 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki core] [17:42:41] !log removed bugs@ mail redirect from sre@ [17:42:44] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:07:43] RECOVERY - test2 Puppet on test2 is OK: OK: Puppet is currently enabled, last run 45 seconds ago with 0 failures [18:25:16] * blackwidowmovie0 yelps for help [18:33:15] .op [18:33:25] that was a test for RhinosF1, btw [18:33:30] Good [18:33:32] That worked [19:07:41] Hello quarter4vcs! If you have any questions, feel free to ask and someone should answer soon. [20:29:11] @Reception123 requesting unban from discord [20:29:58] You can be unbanned on one condition. You follow the rules. We've overhauled the way we mod. [20:30:29] ok [20:30:52] what do you mean you've overhauled? [20:31:26] @paladox I agree [20:34:04] blackwidowmovie0: we've decided it would be best if the moderation team discussed this a bit more before agreeing to an unban [20:34:19] ok [20:51:52] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-6 [+0/-0/±1] 13https://git.io/JI2JC [20:51:54] [02miraheze/mw-config] 07Universal-Omega 03b8c506e - Update configs to modern wikibase; re-format file [20:51:56] [02mw-config] 07Universal-Omega created branch 03Universal-Omega-patch-6 - 13https://git.io/vbvb3 [20:52:13] [02mw-config] 07Universal-Omega opened pull request 03#3553: Update configs to modern wikibase; re-format Wikibase.php - 13https://git.io/JI2Jl [21:01:34] [02mw-config] 07Universal-Omega edited pull request 03#3553: Update configs to modern wikibase; re-format Wikibase.php - 13https://git.io/JI2Jl [21:19:17] Hello user21! If you have any questions, feel free to ask and someone should answer soon. [21:20:53] Is it possible to edit or change the interface language on a Miraheze wiki? [21:21:09] yes [21:21:18] Special:ManageWiki/namespaces [21:21:36] thank you [21:24:30] of course [21:27:55] Hello mpamies_giaxni! If you have any questions, feel free to ask and someone should answer soon. [23:07:35] PROBLEM - mon1 Disk Space on mon1 is WARNING: DISK WARNING - free space: / 4897 MB (10% inode=92%);