[00:49:12] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 7.77, 5.75, 4.25 [00:50:11] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 8.74, 7.05, 5.19 [00:51:12] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 5.95, 5.84, 4.47 [00:52:10] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 5.36, 6.43, 5.20 [01:15:35] RECOVERY - wiki.mlpwiki.net - reverse DNS on sslhost is OK: rDNS OK - wiki.mlpwiki.net reverse DNS resolves to cp11.miraheze.org [01:24:39] PROBLEM - wiki.mlpwiki.net - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.mlpwiki.net reverse DNS resolves to 192-185-16-85.unifiedlayer.com [02:11:53] PROBLEM - services4 APT on services4 is CRITICAL: APT CRITICAL: 31 packages available for upgrade (3 critical updates). [02:14:34] PROBLEM - services3 APT on services3 is CRITICAL: APT CRITICAL: 31 packages available for upgrade (3 critical updates). [04:24:35] RECOVERY - wiki.mlpwiki.net - reverse DNS on sslhost is OK: rDNS OK - wiki.mlpwiki.net reverse DNS resolves to cp11.miraheze.org [04:52:31] [02miraheze/dns] 07Reception123 pushed 031 commit to 03master [+0/-1/±0] 13https://git.io/J3fOH [04:52:33] [02miraheze/dns] 07Reception123 030a9f882 - remove inourownwords.online (redoing; cert generation doesn't work) [05:04:29] !log reception@jobrunner3:~$ sudo -u www-data php /srv/mediawiki/w/maintenance/deleteBatch.php --wiki=awfulmovieswiki --r "[[phab:T7200|Requested[[" /home/reception/awfuldel.txt [05:04:32] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [05:33:06] !log reception@jobrunner3:~$ sudo -u www-data php /srv/mediawiki/w/maintenance/deleteBatch.php --wiki=greatestmovieswiki --r "[[phab:T7201|Requested]]" /home/reception/greatdel.txt [05:33:10] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [05:36:53] !log reception@jobrunner4:~$ sudo -u www-data php /srv/mediawiki/w/maintenance/deleteBatch.php --wiki=cancelledmovieswiki --r "[[phab:T7202|Requested]]" /home/reception/Cancelled_Movies_Wiki_leftovers.txt [05:36:56] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [05:44:34] [02miraheze/mw-config] 07Reception123 pushed 031 commit to 03Reception123-patch-4 [+0/-0/±1] 13https://git.io/J3fz3 [05:44:36] [02miraheze/mw-config] 07Reception123 036fe24a5 - set usenewrc to 0 for bioencyclopediawiki per T7198 [05:44:38] [02mw-config] 07Reception123 created branch 03Reception123-patch-4 - 13https://git.io/vbvb3 [05:44:39] [02mw-config] 07Reception123 opened pull request 03#3857: set usenewrc to 0 for bioencyclopediawiki per T7198 - 13https://git.io/J3fzG [05:45:46] miraheze/mw-config - Reception123 the build passed. [05:45:59] [02mw-config] 07Reception123 closed pull request 03#3857: set usenewrc to 0 for bioencyclopediawiki per T7198 - 13https://git.io/J3fzG [05:46:00] [02miraheze/mw-config] 07Reception123 pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/J3fzD [05:46:01] [02miraheze/mw-config] 07Reception123 030259e31 - set usenewrc to 0 for bioencyclopediawiki per T7198 (#3857) [05:46:03] [02mw-config] 07Reception123 synchronize pull request 03#3845: Merge 'master' into REL1_36 - 13https://git.io/JOKA0 [05:46:05] [02miraheze/mw-config] 07Reception123 deleted branch 03Reception123-patch-4 [05:46:06] [02mw-config] 07Reception123 deleted branch 03Reception123-patch-4 - 13https://git.io/vbvb3 [05:48:22] miraheze/mw-config - Reception123 the build passed. [05:55:31] [02miraheze/dns] 07Reception123 pushed 031 commit to 03master [+1/-0/±0] 13https://git.io/J3f29 [05:55:32] [02miraheze/dns] 07Reception123 038f75b41 - re-add inourownwords.online zone [05:59:20] !log reception@jobrunner3:~$ sudo -u www-data php /srv/mediawiki/w/maintenance/importDump.php --wiki particracyclassicwiki --username-prefix="wikia:particracy" /home/reception/particracy_pages_full.xml [05:59:25] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [05:59:43] PROBLEM - jobrunner3 Disk Space on jobrunner3 is WARNING: DISK WARNING - free space: / 3866 MB (8% inode=83%); [06:01:44] RECOVERY - jobrunner3 Disk Space on jobrunner3 is OK: DISK OK - free space: / 14416 MB (32% inode=83%); [06:05:53] RECOVERY - services4 APT on services4 is OK: APT OK: 28 packages available for upgrade (0 critical updates). [06:16:33] RECOVERY - services3 APT on services3 is OK: APT OK: 28 packages available for upgrade (0 critical updates). [07:28:35] PROBLEM - wiki.mlpwiki.net - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.mlpwiki.net reverse DNS resolves to 192-185-16-85.unifiedlayer.com [09:31:52] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.05, 5.83, 4.25 [09:33:51] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 4.14, 5.11, 4.17 [09:47:58] !log MariaDB [metawiki]> DELETE FROM echo_unread_wikis WHERE euw_user = '211266' AND euw_wiki = 'naasgamelandwiki'; [09:48:00] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [10:01:03] PROBLEM - wiki.bikinibottomdiscord.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.bikinibottomdiscord.org' expires in 15 day(s) (Fri 14 May 2021 09:55:46 GMT +0000). [10:03:43] PROBLEM - puardian.xyz - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'puardian.xyz' expires in 15 day(s) (Fri 14 May 2021 09:57:51 GMT +0000). [10:07:53] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/J3JwE [10:07:55] [02miraheze/ssl] 07MirahezeSSLBot 03d20d4fd - Bot: Update SSL cert for wiki.bikinibottomdiscord.org [10:14:36] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/J3Jom [10:14:38] [02miraheze/ssl] 07MirahezeSSLBot 034550b68 - Bot: Update SSL cert for puardian.xyz [10:38:27] RECOVERY - puardian.xyz - LetsEncrypt on sslhost is OK: OK - Certificate 'puardian.xyz' will expire on Tue 27 Jul 2021 09:14:30 GMT +0000. [10:41:36] RECOVERY - wiki.bikinibottomdiscord.org - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.bikinibottomdiscord.org' will expire on Tue 27 Jul 2021 09:07:46 GMT +0000. [13:19:48] paladox: any idea why I can't generate a cert for inourownwords.online? I added the zone and it says it's pointed to our NS https://github.com/miraheze/dns/commit/8f75b4180dd717e313021363198065f218d15975 [13:19:48] [ re-add inourownwords.online zone · miraheze/dns@8f75b41 · GitHub ] - github.com [13:19:59] https://www.irccloud.com/pastebin/Lt7O1wHc/ [13:19:59] [ Snippet | IRCCloud ] - www.irccloud.com [13:24:19] [02miraheze/ssl] 07Reception123 pushed 032 commits to 03master [+2/-0/±2] 13https://git.io/J3UVt [13:24:20] [02miraheze/ssl] 07Reception123 037691340 - add www.arru.xyz cert [13:24:22] [02miraheze/ssl] 07Reception123 036157683 - add wiki.beergeeks.co.il cert [13:30:35] RECOVERY - wiki.mlpwiki.net - reverse DNS on sslhost is OK: rDNS OK - wiki.mlpwiki.net reverse DNS resolves to cp10.miraheze.org [13:34:31] [02mediawiki] 07R4356th opened pull request 03#1527: Update DarkMode - 13https://git.io/J3UoM [13:39:36] !log renamed wikiaskwiki to wikianswerswiki [13:39:40] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [13:45:32] [02mw-config] 07R4356th opened pull request 03#3858: Add wgScratchBlocks4BlockVersion to ManageWiki - 13https://git.io/J3UPK [13:46:18] [02mw-config] 07Reception123 closed pull request 03#3844: Fix category collation for CategorySortHeaders - 13https://git.io/JOKHg [13:46:20] [02miraheze/mw-config] 07Reception123 pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/J3UPx [13:46:21] [02miraheze/mw-config] 07Universal-Omega 03b5e0790 - Fix category collation for CategorySortHeaders (#3844) [13:46:23] [02mw-config] 07Reception123 synchronize pull request 03#3845: Merge 'master' into REL1_36 - 13https://git.io/JOKA0 [13:46:35] miraheze/mw-config - R4356th the build passed. [13:47:21] miraheze/mw-config - Reception123 the build passed. [13:51:33] RECOVERY - techwiki.techboyg5blog.com - reverse DNS on sslhost is OK: rDNS OK - techwiki.techboyg5blog.com reverse DNS resolves to cp11.miraheze.org [13:57:31] Reception123: did you run puppet on the nameservers? [13:57:53] [02mw-config] 07R4356th synchronize pull request 03#3858: Add wgScratchBlocks4BlockVersion to ManageWiki - 13https://git.io/J3UPK [13:58:21] paladox: well not manually but it would've ran since then it's been a few hours [13:58:29] I added the zone at 5:55 AM UTC [13:59:19] miraheze/mw-config - R4356th the build has errored. [13:59:37] [02mw-config] 07R4356th synchronize pull request 03#3858: Add wgScratchBlocks4BlockVersion to ManageWiki - 13https://git.io/J3UPK [14:00:44] miraheze/mw-config - R4356th the build passed. [14:26:19] Oh [14:26:27] What did it say? [14:29:08] https://www.irccloud.com/pastebin/9Si1SVYw/ [14:29:08] [ Snippet | IRCCloud ] - www.irccloud.com [14:29:10] ^ paladox [14:29:58] Hmm [14:31:33] Is the domain even pointing at our nameservers? [14:31:49] yes, see the dig above [14:31:54] https://www.irccloud.com/pastebin/Lt7O1wHc/ [14:31:54] [ Snippet | IRCCloud ] - www.irccloud.com [14:31:59] and a WHOIS search also says it is [14:32:19] Weird [14:32:32] yeah, I don't get why it's not working [14:32:39] I even tried to remove it let puppet run and add it back to DNS config [14:32:41] and that didn't fix it [14:34:22] https://www.irccloud.com/pastebin/YYWDt9lY/ [14:34:22] [ Snippet | IRCCloud ] - www.irccloud.com [14:35:29] !log restart gdnsd on ns1 [14:35:32] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [14:35:35] Reception123: try now [14:37:46] paladox: same error [14:37:56] hmm, i'm not sure then [14:38:03] paladox: ah maybe this extra line might be useful to you [14:38:04] Failed authorization procedure. inourownwords.online (http-01): urn:ietf:params:acme:error:dns :: DNS problem: SERVFAIL looking up A for inourownwords.online - the domain's nameservers may be malfunctioning [14:38:25] it's failing to find the dns? [14:38:54] I guess so yeah [14:40:49] [02miraheze/dns] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/J3Tel [14:40:50] [02miraheze/dns] 07paladox 038e0bdd8 - Update inourownwords.online [14:41:14] found it [14:41:22] Apr 28 13:19:17 ns2 gdnsd[11085]: UDP sendmmsg() failed: Permission denied [14:41:27] JohnLewis: any ideas ^? [14:41:40] ? [14:42:40] I saw "Apr 28 13:19:17 ns2 gdnsd[11085]: UDP sendmmsg() failed: Permission denied" in ns2 logs [14:42:55] though restarting gdnsd doesn't seem to fix the issue Reception123 has [14:43:00] !log restarting gdnsd on ns2 [14:43:03] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [14:43:53] No monitoring has gone off to suggest ns2 is down, so I'm unsure what the problem with ns2 is [14:43:58] * dmehus is surprised we don't use BIND as our DNS server [14:44:33] i'm out of ideas on what else to do to fix the "DNS problem: SERVFAIL looking up A for inourownwords.online" error [14:45:00] yeah, that's not happened before iirc, first time I have this issue where everything is pointed correctly but I can't generate a cert [14:45:10] either a) there's no DNS servers configured or b) the DNS server aren't reachable/returning that domain when requested [14:45:29] Restarting DNS servers -> wontfix it and would cause more trouble than it could fix [14:46:54] dig shows ns1 and ns2 return results for that domain, and a host shows results [14:47:34] Reception123 seems to have shown above that dig also returns results, so why were ns* restarted? [14:47:47] yeah, do we know the user has set up their DNS servers correctly? [14:47:59] I'm showing they haven't set up a ns1 for their domain [14:48:03] ```{ [14:48:03] "objectClassName": "nameserver", [14:48:03] "ldhName": "ns2.miraheze.org", [14:48:03] "status": [ [14:48:03] "removed" [14:48:03] ], [14:48:03] "links": [ [14:48:04] { [14:48:04] "title": "Authoritative URL for this object", [14:48:05] "type": "application/rdap+json", [14:48:05] "rel": "self", [14:48:06] "href": "https://rdap.centralnic.com/online/nameserver/ns2.miraheze.org" [14:48:56] dig on 8.8.8.8, 8.8.4.4 and 1.1.1.1 shows no NS entries for the domain [14:48:56] nvm the above [14:49:09] hrm, interesting [14:50:58] DotOnline registry shows our two nameservers, but status indicates as "removed" [14:51:08] does that mean they removed our nameservers from their domain? [14:51:21] https://rdap.centralnic.com/online/ [14:53:20] I checked with https://whois.domaintools.com/inourownwords.online and it also indicates that they're pointed [14:53:20] [ Whois Lookup Captcha ] - whois.domaintools.com [14:55:08] Reception123, yeah have we double-checked the code for that domain's zone file? [14:55:21] I wonder if it's a syntax issue with our zone [14:55:27] yes, I did myself, Paladox did also as he made some changes [14:55:45] strange [14:56:49] I stated above it's nothing on our end, and Reception123 posted results to show that [14:57:14] JohnLewis: what could I suggest to the user then? As if they're not very tech savvy they might not really know what to do [14:57:23] If it was a problem with us, dig @ns[12].miraheze.org wouldn't return any results, if it was a syntax error, gdnsd wouldn't be running now [14:57:28] Reception123: wait? [14:57:49] There is a notice provided by all registrars that changes can take up to 48 hours to show [14:58:12] and dig tells me neither Google or Cloudflare know there are any NS entries for the domain [14:58:13] the request was made on the 25th though [14:58:56] They should probably contact their registrar in that case then [15:00:57] ok, will advise them to do that then [15:02:02] Yeah, I was wondering if this could be a DNS propagation issue. Seems maybe either this registrar or the Radix Registry are quite slow [15:05:00] I would've thought that too but since the request was done 3 days ago that's more than the 48 hours [15:05:04] so I'd think there's something else wrong [15:05:48] yeah [15:06:20] The problem is between the Registrar and Root servers though, as the big public caches don't have any records for the domain [15:06:36] We have local records for the domain, which work [15:07:32] heh maybe Miraheze Limited should become a domain reseller on a wholesale registrar; could provide some additional revenue for Miraheze, and users have to pay for their domains anyway [15:14:52] PROBLEM - cp10 Current Load on cp10 is CRITICAL: CRITICAL - load average: 4.42, 5.57, 2.87 [15:16:52] PROBLEM - cp10 Current Load on cp10 is WARNING: WARNING - load average: 1.08, 3.88, 2.57 [15:18:51] RECOVERY - cp10 Current Load on cp10 is OK: OK - load average: 0.80, 3.06, 2.44 [15:35:08] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 8.59, 6.05, 4.94 [15:37:09] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 5.02, 5.58, 4.90 [15:56:02] [02puppet] 07MacFan4000 opened pull request 03#1748: users: remove usage of the deprecated function is_array - 13https://git.io/J3TE8 [15:59:08] [02puppet] 07paladox closed pull request 03#1748: users: remove usage of the deprecated function is_array - 13https://git.io/J3TE8 [15:59:10] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/J3TuB [15:59:11] [02miraheze/puppet] 07MacFan4000 0322e4942 - users: remove usage of the deprecated function is_array (#1748) [15:59:14] [02puppet] 07paladox commented on pull request 03#1748: users: remove usage of the deprecated function is_array - 13https://git.io/J3TuV [16:04:36] Ty paladox [16:13:38] [02mw-config] 07MacFan4000 commented on pull request 03#3854: Remove wmgUseYandexTranslate - 13https://git.io/J3TV6 [16:16:35] [02mw-config] 07Universal-Omega commented on pull request 03#3854: Remove wmgUseYandexTranslate - 13https://git.io/J3Tw1 [16:18:54] [02mw-config] 07R4356th commented on pull request 03#3854: Remove wmgUseYandexTranslate - 13https://git.io/J3Tr0 [16:22:54] [02mw-config] 07dmehus commented on pull request 03#3854: Remove wmgUseYandexTranslate - 13https://git.io/J3TKl [16:24:07] [02mw-config] 07R4356th commented on pull request 03#3854: Remove wmgUseYandexTranslate - 13https://git.io/J3TKd [16:24:17] paladox: do you know how to do https://phabricator.miraheze.org/T7186 ? It's installed with a 'composer require' and I'm not sure how we'd do that in puppet [16:24:18] [ ⚓ T7186 PHPSpreadsheet for Data Transfer ] - phabricator.miraheze.org [16:24:40] [02mw-config] 07dmehus commented on pull request 03#3854: Remove wmgUseYandexTranslate - 13https://git.io/J3T6J [16:24:41] `composer install`? [16:25:23] https://github.com/miraheze/puppet/blob/master/modules/mediawiki/manifests/extensionsetup.pp [16:25:24] [ puppet/extensionsetup.pp at master · miraheze/puppet · GitHub ] - github.com [16:25:56] paladox: but would that work if the docs want `composer require`? [16:26:14] yes but don't they provide a composer.json file? [16:26:39] PROBLEM - cp11 Current Load on cp11 is CRITICAL: CRITICAL - load average: 1.83, 4.60, 2.88 [16:26:42] oh yeah there's https://github.com/PHPOffice/PhpSpreadsheet/blob/master/composer.json [16:26:43] [ PhpSpreadsheet/composer.json at master · PHPOffice/PhpSpreadsheet · GitHub ] - github.com [16:27:42] so you could just do composer install [16:27:58] ok then, thanks :) [16:28:40] RECOVERY - cp11 Current Load on cp11 is OK: OK - load average: 0.82, 3.36, 2.63 [16:41:36] PROBLEM - dbbackup2 Check MariaDB Replication c3 on dbbackup2 is CRITICAL: MariaDB replication - both - CRITICAL - Slave_IO_Running state : Yes, Slave_SQL_Running state : Yes, Seconds_Behind_Master : 220s [16:45:32] [02mw-config] 07Universal-Omega reviewed pull request 03#3858 commit - 13https://git.io/J3TMh [16:47:24] [02mw-config] 07R4356th reviewed pull request 03#3858 commit - 13https://git.io/J3TDV [16:49:35] PROBLEM - dbbackup2 Check MariaDB Replication c3 on dbbackup2 is WARNING: MariaDB replication - both - WARNING - Slave_IO_Running state : Yes, Slave_SQL_Running state : Yes, Seconds_Behind_Master : 190s [16:55:43] PROBLEM - wiki.mlpwiki.net - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.mlpwiki.net reverse DNS resolves to 192-185-16-85.unifiedlayer.com [16:57:35] RECOVERY - dbbackup2 Check MariaDB Replication c3 on dbbackup2 is OK: MariaDB replication - both - OK - Slave_IO_Running state : Yes, Slave_SQL_Running state : Yes, Seconds_Behind_Master : 68s [17:05:13] [02mw-config] 07Universal-Omega reviewed pull request 03#3858 commit - 13https://git.io/J3TQB [17:34:56] !log sudo -u www-data /usr/local/bin/foreachwikiindblist /home/reception/categorysortheaders.json /srv/mediawiki/w/maintenance/updateCollation.php [17:34:59] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:38:54] [02miraheze/WikiDiscover] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/J3Tpk [17:38:56] [02miraheze/WikiDiscover] 07paladox 03087c105 - Temporarily remove the api [17:38:57] [02WikiDiscover] 07paladox created branch 03paladox-patch-1 - 13https://git.io/vhUAp [17:38:59] [02WikiDiscover] 07paladox opened pull request 03#43: Temporarily remove the api - 13https://git.io/J3TpI [17:39:28] [02WikiDiscover] 07paladox closed pull request 03#43: Temporarily remove the api - 13https://git.io/J3TpI [17:39:30] [02miraheze/WikiDiscover] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/J3Tp3 [17:39:31] [02miraheze/WikiDiscover] 07paladox 03e3facd8 - Temporarily remove the api (#43) [17:39:32] [02WikiDiscover] 07paladox deleted branch 03paladox-patch-1 - 13https://git.io/vhUAp [17:39:34] [02miraheze/WikiDiscover] 07paladox deleted branch 03paladox-patch-1 [17:39:59] miraheze/WikiDiscover - paladox the build passed. [17:40:26] miraheze/WikiDiscover - paladox the build passed. [17:40:32] ohh [17:41:03] [02miraheze/WikiDiscover] 07paladox pushed 031 commit to 03revert-43-paladox-patch-1 [+0/-0/±1] 13https://git.io/J3TpK [17:41:04] [02miraheze/WikiDiscover] 07paladox 033778e1d - Revert "Temporarily remove the api (#43)" [17:41:06] [02WikiDiscover] 07paladox created branch 03revert-43-paladox-patch-1 - 13https://git.io/vhUAp [17:41:07] [02WikiDiscover] 07paladox opened pull request 03#44: Revert "Temporarily remove the api" - 13https://git.io/J3Tp6 [17:41:11] [02WikiDiscover] 07paladox closed pull request 03#44: Revert "Temporarily remove the api" - 13https://git.io/J3Tp6 [17:41:12] [02miraheze/WikiDiscover] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/J3Tpi [17:41:13] [02miraheze/WikiDiscover] 07paladox 037905107 - Revert "Temporarily remove the api (#43)" (#44) [17:41:26] [02WikiDiscover] 07paladox deleted branch 03revert-43-paladox-patch-1 - 13https://git.io/vhUAp [17:41:28] [02miraheze/WikiDiscover] 07paladox deleted branch 03revert-43-paladox-patch-1 [17:42:03] [02miraheze/ManageWiki] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/J3TpQ [17:42:04] [02miraheze/ManageWiki] 07paladox 03bb0bf7b - Temporarily remove api [17:42:06] [02ManageWiki] 07paladox created branch 03paladox-patch-1 - 13https://git.io/vpSns [17:42:06] miraheze/WikiDiscover - paladox the build passed. [17:42:07] [02ManageWiki] 07paladox opened pull request 03#269: Temporarily remove api - 13https://git.io/J3Tpd [17:42:13] miraheze/WikiDiscover - paladox the build passed. [17:43:06] miraheze/ManageWiki - paladox the build passed. [17:44:11] [02ManageWiki] 07paladox closed pull request 03#269: Temporarily remove api - 13https://git.io/J3Tpd [17:44:16] [02miraheze/ManageWiki] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/J3ThC [17:44:17] [02miraheze/ManageWiki] 07paladox 0330dd8b4 - Temporarily remove api (#269) [17:44:19] [02ManageWiki] 07paladox deleted branch 03paladox-patch-1 - 13https://git.io/vpSns [17:44:20] [02miraheze/ManageWiki] 07paladox deleted branch 03paladox-patch-1 [17:45:06] miraheze/ManageWiki - paladox the build passed. [17:45:33] [02miraheze/mw-config] 07Reception123 pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/J3ThM [17:45:34] [02miraheze/mw-config] 07Reception123 0371ea281 - disable APIquery for wikiconfig [17:45:36] [02mw-config] 07Reception123 synchronize pull request 03#3845: Merge 'master' into REL1_36 - 13https://git.io/JOKA0 [17:46:41] miraheze/mw-config - Reception123 the build passed. [17:47:40] [02miraheze/mediawiki] 07paladox pushed 031 commit to 03REL1_35 [+0/-0/±1] 13https://git.io/J3Tjt [17:47:42] [02miraheze/mediawiki] 07paladox 03b828ef2 - Update ManageWiki [17:54:30] [02miraheze/ManageWiki] 07paladox pushed 031 commit to 03revert-269-paladox-patch-1 [+0/-0/±1] 13https://git.io/J3kea [17:54:31] [02miraheze/ManageWiki] 07paladox 03e07ae3f - Revert "Temporarily remove api (#269)" [17:54:33] [02ManageWiki] 07paladox created branch 03revert-269-paladox-patch-1 - 13https://git.io/vpSns [17:54:34] [02ManageWiki] 07paladox opened pull request 03#270: Revert "Temporarily remove api" - 13https://git.io/J3keV [17:54:39] [02ManageWiki] 07paladox closed pull request 03#270: Revert "Temporarily remove api" - 13https://git.io/J3keV [17:54:40] [02miraheze/ManageWiki] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/J3kew [17:54:42] [02miraheze/ManageWiki] 07paladox 0399f3b2c - Revert "Temporarily remove api (#269)" (#270) [17:54:43] [02ManageWiki] 07paladox deleted branch 03revert-269-paladox-patch-1 - 13https://git.io/vpSns [17:54:45] [02miraheze/ManageWiki] 07paladox deleted branch 03revert-269-paladox-patch-1 [17:55:14] [02miraheze/mediawiki] 07paladox pushed 031 commit to 03REL1_35 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/compare/b828ef27770f...a41368915301 [17:55:15] [ Comparing b828ef27770f...a41368915301 · miraheze/mediawiki · GitHub ] - github.com [17:55:15] [02miraheze/mediawiki] 07paladox 03a413689 - Update ManageWiki [17:55:36] miraheze/ManageWiki - paladox the build passed. [17:55:38] miraheze/ManageWiki - paladox the build passed. [18:10:00] PROBLEM - jobrunner4 Puppet on jobrunner4 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki core] [18:11:09] PROBLEM - mw9 Puppet on mw9 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki core] [18:11:47] PROBLEM - jobrunner3 Puppet on jobrunner3 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki core] [18:12:15] PROBLEM - mw11 Puppet on mw11 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki core] [18:12:31] PROBLEM - mw8 Puppet on mw8 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki core] [18:12:32] PROBLEM - mw10 Puppet on mw10 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki core] [18:12:33] PROBLEM - sdiy.info - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for sdiy.info could not be found [18:12:34] PROBLEM - wiki.pablojprieto.com.es - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.pablojprieto.com.es could not be found [18:12:36] PROBLEM - phab.bots.miraheze.wiki - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for phab.bots.miraheze.wiki could not be found [18:12:36] PROBLEM - history.sdtef.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for history.sdtef.org could not be found [18:12:43] PROBLEM - wiki.cyberfurs.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.cyberfurs.org could not be found [18:12:47] PROBLEM - www.zenbuddhism.info - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for www.zenbuddhism.info could not be found [18:12:47] PROBLEM - wiki.counterculturelabs.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.counterculturelabs.org could not be found [18:19:30] RECOVERY - history.sdtef.org - reverse DNS on sslhost is OK: rDNS OK - history.sdtef.org reverse DNS resolves to cp10.miraheze.org [18:19:31] RECOVERY - phab.bots.miraheze.wiki - reverse DNS on sslhost is OK: rDNS OK - phab.bots.miraheze.wiki reverse DNS resolves to cp11.miraheze.org [18:19:32] RECOVERY - sdiy.info - reverse DNS on sslhost is OK: rDNS OK - sdiy.info reverse DNS resolves to cp10.miraheze.org [18:19:33] RECOVERY - wiki.counterculturelabs.org - reverse DNS on sslhost is OK: rDNS OK - wiki.counterculturelabs.org reverse DNS resolves to cp11.miraheze.org [18:19:33] RECOVERY - wiki.pablojprieto.com.es - reverse DNS on sslhost is OK: rDNS OK - wiki.pablojprieto.com.es reverse DNS resolves to cp11.miraheze.org [18:19:40] RECOVERY - wiki.cyberfurs.org - reverse DNS on sslhost is OK: rDNS OK - wiki.cyberfurs.org reverse DNS resolves to cp10.miraheze.org [18:19:46] RECOVERY - www.zenbuddhism.info - reverse DNS on sslhost is OK: rDNS OK - www.zenbuddhism.info reverse DNS resolves to cp10.miraheze.org [18:24:36] [02ManageWiki] 07paladox opened pull request 03#271: Patch 9 - 13https://git.io/J3kLg [18:25:43] miraheze/ManageWiki - paladox the build passed. [18:33:44] RECOVERY - jobrunner3 Puppet on jobrunner3 is OK: OK: Puppet is currently enabled, last run 6 seconds ago with 0 failures [18:33:58] RECOVERY - jobrunner4 Puppet on jobrunner4 is OK: OK: Puppet is currently enabled, last run 25 seconds ago with 0 failures [18:34:15] RECOVERY - mw11 Puppet on mw11 is OK: OK: Puppet is currently enabled, last run 42 seconds ago with 0 failures [18:34:26] RECOVERY - mw8 Puppet on mw8 is OK: OK: Puppet is currently enabled, last run 15 seconds ago with 0 failures [18:34:32] RECOVERY - mw10 Puppet on mw10 is OK: OK: Puppet is currently enabled, last run 46 seconds ago with 0 failures [18:35:08] RECOVERY - mw9 Puppet on mw9 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [18:45:48] Be back later [18:50:13] [02ManageWiki] 07Universal-Omega opened pull request 03#272: Update ApiQueryWikiConfig.php - 13https://git.io/J3k3v [18:51:05] [02ManageWiki] 07paladox closed pull request 03#271: Patch 9 - 13https://git.io/J3kLg [18:51:08] miraheze/ManageWiki - Universal-Omega the build passed. [18:59:35] PROBLEM - dbbackup2 Check MariaDB Replication c3 on dbbackup2 is WARNING: MariaDB replication - both - WARNING - Slave_IO_Running state : Yes, Slave_SQL_Running state : Yes, Seconds_Behind_Master : 172s [19:01:35] PROBLEM - dbbackup2 Check MariaDB Replication c3 on dbbackup2 is CRITICAL: MariaDB replication - both - CRITICAL - Slave_IO_Running state : Yes, Slave_SQL_Running state : Yes, Seconds_Behind_Master : 200s [19:03:33] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 7.00, 5.56, 5.13 [19:05:11] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/J3kGV [19:05:13] [02miraheze/services] 07MirahezeSSLBot 03790cc55 - BOT: Updating services config for wikis [19:05:29] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 6.48, 5.96, 5.34 [19:07:35] RECOVERY - dbbackup2 Check MariaDB Replication c3 on dbbackup2 is OK: MariaDB replication - both - OK - Slave_IO_Running state : Yes, Slave_SQL_Running state : Yes, Seconds_Behind_Master : 47s [19:30:55] [02ManageWiki] 07Universal-Omega closed pull request 03#272: Update ApiQueryWikiConfig.php - 13https://git.io/J3k3v [19:33:38] [02miraheze/ManageWiki] 07Reception123 pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/J3klv [19:33:40] [02miraheze/ManageWiki] 07Universal-Omega 03befb83c - Merge pull request from GHSA-jmc9-rv2f-g8vv [19:34:39] miraheze/ManageWiki - Reception123 the build passed. [19:36:35] [02miraheze/mediawiki] 07Reception123 pushed 032 commits to 03REL1_35 [+0/-0/±1] 13https://git.io/J3klu [19:36:36] [02miraheze/mediawiki] 07Reception123 0315f1fc6 - Update ManageWiki [19:36:38] [02miraheze/mediawiki] 07Reception123 0338074cb - Merge branch 'REL1_35' of https://github.com/miraheze/mediawiki into REL1_35 [19:36:39] [ GitHub - miraheze/mediawiki: The collaborative editing software that runs Wikipedia. ] - github.com [19:41:16] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 6.26, 6.85, 5.61 [19:43:11] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 4.95, 6.22, 5.53 [20:58:09] [02miraheze/mw-config] 07RhinosF1 pushed 031 commit to 03RhinosF1-patch-1 [+0/-0/±1] 13https://git.io/J3kV1 [20:58:10] [02miraheze/mw-config] 07RhinosF1 03b832639 - Update Sitenotice.php [20:58:12] [02mw-config] 07RhinosF1 created branch 03RhinosF1-patch-1 - 13https://git.io/vbvb3 [20:58:13] [02mw-config] 07RhinosF1 opened pull request 03#3859: Update Sitenotice.php - 13https://git.io/J3kVM [20:59:28] miraheze/mw-config - RhinosF1 the build passed. [20:59:43] [02miraheze/mw-config] 07RhinosF1 pushed 031 commit to 03RhinosF1-patch-2 [+0/-0/±1] 13https://git.io/J3kV7 [20:59:43] [02miraheze/mw-config] 07RhinosF1 03539dd08 - Update LocalWiki.php [20:59:43] [02mw-config] 07RhinosF1 created branch 03RhinosF1-patch-2 - 13https://git.io/vbvb3 [20:59:43] [02mw-config] 07RhinosF1 opened pull request 03#3860: Update LocalWiki.php - 13https://git.io/J3kV5 [20:59:44] [02mw-config] 07Universal-Omega reviewed pull request 03#3859 commit - 13https://git.io/J3kVb [21:00:16] miraheze/mw-config - RhinosF1 the build passed. [21:00:19] [02miraheze/mw-config] 07RhinosF1 pushed 031 commit to 03RhinosF1-patch-1 [+0/-0/±1] 13https://git.io/J3kwf [21:00:20] [02miraheze/mw-config] 07RhinosF1 03bdfb497 - Update Sitenotice.php [21:00:21] [02mw-config] 07RhinosF1 synchronize pull request 03#3859: Update Sitenotice.php - 13https://git.io/J3kVM [21:00:23] JohnLewis: clear to merge ? [21:01:23] miraheze/mw-config - RhinosF1 the build passed. [21:10:16] paladox: ^ can you sense check sitenotice and LW revert [21:13:14] seems fine, though pls give them a better tittle [21:13:18] *title [21:15:06] [02mw-config] 07RhinosF1 edited pull request 03#3860: Re-enable ManageWiki API - 13https://git.io/J3kV5 [21:15:15] [02mw-config] 07RhinosF1 closed pull request 03#3860: Re-enable ManageWiki API - 13https://git.io/J3kV5 [21:15:16] [02miraheze/mw-config] 07RhinosF1 pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/J3koc [21:15:18] [02miraheze/mw-config] 07RhinosF1 0352130ea - Re-enable ManageWiki API (#3860) [21:15:19] [02mw-config] 07RhinosF1 synchronize pull request 03#3845: Merge 'master' into REL1_36 - 13https://git.io/JOKA0 [21:15:41] [02mw-config] 07RhinosF1 edited pull request 03#3859: Add Sitenotice about Security Incident - 13https://github.com/miraheze/mw-config/pull/3859 [21:15:50] [02mw-config] 07RhinosF1 closed pull request 03#3859: Add Sitenotice about Security Incident - 13https://git.io/J3kVM [21:15:51] [02miraheze/mw-config] 07RhinosF1 pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/J3koE [21:15:53] [02miraheze/mw-config] 07RhinosF1 030bd604f - Add Sitenotice about Security Incident (#3859) [21:15:54] [02mw-config] 07RhinosF1 synchronize pull request 03#3845: Merge 'master' into REL1_36 - 13https://git.io/JOKA0 [21:15:56] [02miraheze/mw-config] 07RhinosF1 deleted branch 03RhinosF1-patch-1 [21:15:57] [02mw-config] 07RhinosF1 deleted branch 03RhinosF1-patch-1 - 13https://git.io/vbvb3 [21:16:18] miraheze/mw-config - RhinosF1 the build passed. [21:16:21] [02miraheze/mw-config] 07RhinosF1 deleted branch 03RhinosF1-patch-2 [21:16:22] [02mw-config] 07RhinosF1 deleted branch 03RhinosF1-patch-2 - 13https://git.io/vbvb3 [21:16:30] paladox: can you mass run puppet [21:16:53] done [21:16:53] miraheze/mw-config - RhinosF1 the build passed. [21:17:16] mw9 is me [21:18:24] [02miraheze/mw-config] 07RhinosF1 pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/J3koS [21:18:26] [02miraheze/mw-config] 07RhinosF1 039d79f54 - Update Sitenotice.php [21:18:27] [02mw-config] 07RhinosF1 synchronize pull request 03#3845: Merge 'master' into REL1_36 - 13https://git.io/JOKA0 [21:18:35] paladox: ^ grammar [21:18:43] done [21:19:25] miraheze/mw-config - RhinosF1 the build passed. [21:19:55] I'm off to sleep [21:20:24] Night RhinosF1, and thanks for handling all that stuff today. [21:20:33] np [21:20:47] i'd already lost track of what day it is this week [21:21:53] CVE-2021-29483 [21:23:20] https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2021-29483 [21:23:21] [ CVE - CVE-2021-29483 ] - cve.mitre.org [21:24:00] [02mw-config] 07dmehus opened pull request 03#3861: Fix typo and correct grammar - 13https://git.io/J3k6r [21:24:50] paladox: ^ [21:25:09] miraheze/mw-config - dmehus the build passed. [21:25:12] [02mw-config] 07paladox closed pull request 03#3861: Fix typo and correct grammar - 13https://git.io/J3k6r [21:25:14] [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/J3k65 [21:25:15] [02miraheze/mw-config] 07dmehus 037ed2f1b - Fix typo and correct grammar (#3861) [21:25:17] [02mw-config] 07paladox synchronize pull request 03#3845: Merge 'master' into REL1_36 - 13https://git.io/JOKA0 [21:25:33] ty [21:26:15] miraheze/mw-config - paladox the build passed. [21:27:39] PROBLEM - dbbackup2 Check MariaDB Replication c3 on dbbackup2 is WARNING: MariaDB replication - both - WARNING - Slave_IO_Running state : Yes, Slave_SQL_Running state : Yes, Seconds_Behind_Master : 123s [21:29:37] RECOVERY - dbbackup2 Check MariaDB Replication c3 on dbbackup2 is OK: MariaDB replication - both - OK - Slave_IO_Running state : Yes, Slave_SQL_Running state : Yes, Seconds_Behind_Master : 0s [21:36:27] PROBLEM - test3 Puppet on test3 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki config] [21:47:26] [02mw-config] 07Universal-Omega commented on pull request 03#3845: Merge 'master' into REL1_36 - 13https://git.io/J3kMJ [21:48:45] PROBLEM - cp10 Current Load on cp10 is CRITICAL: CRITICAL - load average: 4.60, 4.71, 2.40 [21:50:46] PROBLEM - cp10 Current Load on cp10 is WARNING: WARNING - load average: 1.24, 3.44, 2.22 [21:52:45] RECOVERY - cp10 Current Load on cp10 is OK: OK - load average: 0.47, 2.43, 1.99 [22:05:30] [02mw-config] 07dmehus commented on pull request 03#3845: Merge 'master' into REL1_36 - 13https://git.io/J3kyP [22:07:13] Regarding the recent security incident, are there any plans to audit the access logs to check what was compromised, if anything? [22:09:44] security incident? what happened? [22:09:54] sorry, I was very busy today [22:11:11] https://phabricator.miraheze.org/T7213 [22:11:12] [ ⚓ T7213 ManageWiki API allows viewing configs that shouldn't be viewed publicly ] - phabricator.miraheze.org [22:16:59] @R4356th, what would potentially be compromised that return something in the access logs? That's related to the user's Discord server or Slack workspace, no? [22:19:55] The API URLs? [22:20:18] Any API request should be stored in the access logs. [22:21:31] Ah, true, yeah. That's right. I mean, at least someone wouldn't have been able to read the page content, I think, but not sure exactly. Maybe paladox can answer? [22:21:50] Additionally, could someone merge https://github.com/miraheze/mediawiki/pull/1527 when possible, please? [22:21:51] [ Update DarkMode by R4356th · Pull Request #1527 · miraheze/mediawiki · GitHub ] - github.com [22:25:49] oh, so this is the persistence fix for Dark Mode? [22:26:52] Lake, yeah [22:26:59] it'll work for anonymous users now [22:27:21] hmm, good. Well, it's alright, but I still would like if anyone used my idea for this extension :P [22:30:44] Lake, what as your idea again? [22:31:33] when using the extension, it creates an alternate .css page (like "MediaWiki:DarkMode.css") and you can put your own dark mode styles there [22:32:00] so this CSS page is "turned on" when you turn Dark Mode on, if it's off, it uses just Common/skin.css [22:32:42]