[00:00:02] the problem was not "i did not receive email" [00:00:24] the main problem is that you get email with a password that does not work [00:00:33] and then Zppix has to manually reset it [00:00:58] so given that and that nothing was changed... i dunno :p [00:01:21] also why it had to be reset in the first place is unknown [00:01:29] since my password is stored in my password manager [00:01:33] RECOVERY - cp3 Disk Space on cp3 is OK: DISK OK - free space: / 3865 MB (16% inode=93%); [00:01:38] who knows [00:02:05] lol [00:16:43] !log root@jobrunner1:/srv/mediawiki/w/maintenance# sudo -u www-data php emptyUserGroup.php --wiki moviepediawiki Wiki_Staff [00:16:46] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:18:17] weird as it is .. i can NOT reproduce the issue now [00:18:30] so you guys are right with the "who knows" approach, haha [00:43:51] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 2 datacenters are down: 2400:6180:0:d0::403:f001/cpweb, 2607:5300:205:200::2ac4/cpweb [00:44:28] PROBLEM - cloud2 Current Load on cloud2 is CRITICAL: CRITICAL - load average: 38.58, 26.20, 18.32 [00:45:12] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JJ8MY [00:45:14] [02miraheze/services] 07MirahezeSSLBot 0314c157e - BOT: Updating services config for wikis [00:45:47] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [00:48:27] PROBLEM - cloud2 Current Load on cloud2 is WARNING: WARNING - load average: 13.82, 20.66, 18.05 [00:50:25] RECOVERY - cloud2 Current Load on cloud2 is OK: OK - load average: 10.83, 17.28, 17.12 [01:51:07] PROBLEM - ping4 on cp3 is WARNING: PING WARNING - Packet loss = 37%, RTA = 256.27 ms [01:51:46] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 2 datacenters are down: 128.199.139.216/cpweb, 2400:6180:0:d0::403:f001/cpweb [01:53:06] RECOVERY - ping4 on cp3 is OK: PING OK - Packet loss = 0%, RTA = 256.40 ms [01:57:05] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 1 datacenter is down: 128.199.139.216/cpweb [01:59:04] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [02:02:57] PROBLEM - cp3 HTTP 4xx/5xx ERROR Rate on cp3 is WARNING: WARNING - NGINX Error Rate is 44% [02:03:06] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 1 datacenter is down: 128.199.139.216/cpweb [02:04:33] PROBLEM - ping4 on cp3 is WARNING: PING WARNING - Packet loss = 28%, RTA = 255.93 ms [02:04:54] RECOVERY - cp3 HTTP 4xx/5xx ERROR Rate on cp3 is OK: OK - NGINX Error Rate is 1% [02:08:44] RECOVERY - ping4 on cp3 is OK: PING OK - Packet loss = 0%, RTA = 247.89 ms [02:09:03] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [02:09:44] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [05:21:11] [02mw-config] 07Reception123 commented on pull request 03#3185: Add discord to $wgURLProtocols for vtwiki T5947 - 13https://git.io/JJ85M [05:22:13] [02mw-config] 07Reception123 closed pull request 03#3179: Remove deleted two wikis - 13https://git.io/JJZa1 [05:22:15] [02miraheze/mw-config] 07Reception123 pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JJ85y [05:22:16] [02miraheze/mw-config] 07soratako 0345a7ed7 - Remove deleted two wikis (#3179) [05:31:55] [02miraheze/mw-config] 07Reception123 pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JJ8d3 [05:31:57] [02miraheze/mw-config] 07Reception123 03a4d1995 - whitelist vedopedia.witches-empire.com for short urls requested at CN [08:11:23] RhinosF1: I guess that PR can be merged (regarding discord) then. I've got no access till the afternoon so if you want you can merge and resolve the task [08:11:39] I'm mobile atm [08:12:49] Ok then I'll do it later [09:40:46] [02mw-config] 07Reception123 closed pull request 03#3185: Add discord to $wgURLProtocols for vtwiki T5947 - 13https://git.io/JJWGW [09:40:47] [02miraheze/mw-config] 07Reception123 pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JJ4fN [09:40:49] [02miraheze/mw-config] 07Amanda-Catherine 036e9104e - Add discord to $wgURLProtocols for vtwiki T5947 (#3185) [10:20:54] !log added a discord webhook for erislywiki [10:21:00] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [11:20:19] [02miraheze/mw-config] 07Reception123 pushed 031 commit to 03Reception123-patch-1 [+0/-0/±1] 13https://git.io/JJ4tZ [11:20:20] [02miraheze/mw-config] 07Reception123 03a641f79 - dcmultiversewiki settings per T5921 [11:20:22] [02mw-config] 07Reception123 created branch 03Reception123-patch-1 - 13https://git.io/vbvb3 [11:20:30] [02mw-config] 07Reception123 opened pull request 03#3187: dcmultiversewiki settings per T5921 - 13https://git.io/JJ4tn [11:30:44] [02miraheze/mw-config] 07Reception123 pushed 031 commit to 03Reception123-patch-1 [+0/-0/±1] 13https://git.io/JJ4tz [11:30:46] [02miraheze/mw-config] 07Reception123 03d336b22 - Update LocalWiki.php [11:30:47] [02mw-config] 07Reception123 synchronize pull request 03#3187: dcmultiversewiki settings per T5921 - 13https://git.io/JJ4tn [11:36:16] [02mw-config] 07Reception123 closed pull request 03#3187: dcmultiversewiki settings per T5921 - 13https://git.io/JJ4tn [11:36:18] [02miraheze/mw-config] 07Reception123 pushed 031 commit to 03master [+0/-0/±2] 13https://git.io/JJ4tS [11:36:19] [02miraheze/mw-config] 07Reception123 03697a968 - dcmultiversewiki settings per T5921 (#3187) * dcmultiversewiki settings per T5921 * Update LocalWiki.php [11:36:21] [02mw-config] 07Reception123 deleted branch 03Reception123-patch-1 - 13https://git.io/vbvb3 [11:36:23] [02miraheze/mw-config] 07Reception123 deleted branch 03Reception123-patch-1 [12:41:03] PROBLEM - db7 Check MariaDB Replication on db7 is CRITICAL: MariaDB replication - both - CRITICAL - Slave_IO_Running state : Yes, Slave_SQL_Running state : Yes, Seconds_Behind_Master : 129s [12:43:03] RECOVERY - db7 Check MariaDB Replication on db7 is OK: MariaDB replication - both - OK - Slave_IO_Running state : Yes, Slave_SQL_Running state : Yes, Seconds_Behind_Master : 0s [12:45:19] Is this acceptable design for userboxes? https://meta.miraheze.org/wiki/Template:User_administrator [12:45:20] [ Template:User administrator - Miraheze Meta ] - meta.miraheze.org [13:10:23] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JJ4sN [13:10:24] [02miraheze/services] 07MirahezeSSLBot 03ed5d416 - BOT: Updating services config for wikis [13:15:11] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JJ4GZ [13:15:13] [02miraheze/services] 07MirahezeSSLBot 036a9289b - BOT: Updating services config for wikis [13:50:40] Hi everyone [14:15:09] > hi JohnBishop [14:16:10] :-) [14:43:46] PROBLEM - kunwok.org - DNS on sslhost is CRITICAL: DNS CRITICAL - expected '2001:41d0:800:1056::2,2001:41d0:800:105a::10,51.77.107.210,51.89.160.142' but got '2001:41d0:800:1056::2,2001:41d0:800:105a::10,51.89.160.142' [14:43:58] PROBLEM - en.gyaanipedia.co.in - DNS on sslhost is CRITICAL: DNS CRITICAL - expected '2001:41d0:800:1056::2,2001:41d0:800:105a::10,51.77.107.210,51.89.160.142' but got '2001:41d0:800:1056::2,2001:41d0:800:105a::10,51.89.160.142' [14:43:58] PROBLEM - bebaskanpengetahuan.id - DNS on sslhost is CRITICAL: DNS CRITICAL - expected '2001:41d0:800:1056::2,2001:41d0:800:105a::10,51.77.107.210,51.89.160.142' but got '2001:41d0:800:1056::2,2001:41d0:800:105a::10,51.89.160.142' [14:44:05] PROBLEM - wiki.thesimswiki.com - DNS on sslhost is CRITICAL: DNS CRITICAL - expected '2001:41d0:800:1056::2,2001:41d0:800:105a::10,51.77.107.210,51.89.160.142' but got '2001:41d0:800:1056::2,2001:41d0:800:105a::10,51.89.160.142' [14:44:18] PROBLEM - wiki.lbcomms.co.za - DNS on sslhost is CRITICAL: DNS CRITICAL - expected '2001:41d0:800:1056::2,2001:41d0:800:105a::10,51.77.107.210,51.89.160.142' but got '2001:41d0:800:1056::2,2001:41d0:800:105a::10,51.89.160.142' [14:45:34] PROBLEM - wiki.archman.de - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.archman.de' expires in 15 day(s) (Sun 09 Aug 2020 14:41:20 GMT +0000). [14:47:22] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JJ4lR [14:47:24] [02miraheze/ssl] 07MirahezeSSLBot 03e6b1d13 - Bot: Update SSL cert for wiki.archman.de [14:50:37] RECOVERY - kunwok.org - DNS on sslhost is OK: DNS OK: 0.045 seconds response time. sslhost returns 2001:41d0:800:1056::2,2001:41d0:800:105a::10,51.77.107.210,51.89.160.142 [14:50:42] RECOVERY - en.gyaanipedia.co.in - DNS on sslhost is OK: DNS OK: 0.031 seconds response time. sslhost returns 2001:41d0:800:1056::2,2001:41d0:800:105a::10,51.77.107.210,51.89.160.142 [14:50:43] RECOVERY - bebaskanpengetahuan.id - DNS on sslhost is OK: DNS OK: 0.055 seconds response time. sslhost returns 2001:41d0:800:1056::2,2001:41d0:800:105a::10,51.77.107.210,51.89.160.142 [14:50:54] RECOVERY - wiki.thesimswiki.com - DNS on sslhost is OK: DNS OK: 0.050 seconds response time. sslhost returns 2001:41d0:800:1056::2,2001:41d0:800:105a::10,51.77.107.210,51.89.160.142 [14:50:59] RECOVERY - wiki.lbcomms.co.za - DNS on sslhost is OK: DNS OK: 0.043 seconds response time. sslhost returns 2001:41d0:800:1056::2,2001:41d0:800:105a::10,51.77.107.210,51.89.160.142 [14:52:17] * hispano76 greetings [14:52:27] RECOVERY - wiki.archman.de - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.archman.de' will expire on Thu 22 Oct 2020 13:47:15 GMT +0000. [14:53:24] PROBLEM - hentailolicon.org - DNS on sslhost is CRITICAL: DNS CRITICAL - expected '2001:41d0:800:1056::2,2001:41d0:800:105a::10,51.77.107.210,51.89.160.142' but got '2001:41d0:800:1056::2,2001:41d0:800:105a::10,51.89.160.142' [15:00:17] RECOVERY - hentailolicon.org - DNS on sslhost is OK: DNS OK: 0.039 seconds response time. sslhost returns 2001:41d0:800:1056::2,2001:41d0:800:105a::10,51.77.107.210,51.89.160.142 [15:33:41] PROBLEM - cloud2 Current Load on cloud2 is CRITICAL: CRITICAL - load average: 42.24, 29.82, 20.05 [15:33:53] Hey Voidwalker [15:34:40] PROBLEM - cp7 Current Load on cp7 is CRITICAL: CRITICAL - load average: 7.89, 10.46, 5.98 [15:36:37] PROBLEM - cp7 Current Load on cp7 is WARNING: WARNING - load average: 2.50, 7.73, 5.52 [15:37:45] PROBLEM - cloud2 Current Load on cloud2 is WARNING: WARNING - load average: 14.38, 21.23, 18.82 [15:38:28] Network just died [15:38:33] RECOVERY - cp7 Current Load on cp7 is OK: OK - load average: 1.60, 5.74, 5.05 [15:39:44] RECOVERY - cloud2 Current Load on cloud2 is OK: OK - load average: 15.35, 18.95, 18.25 [16:21:32] PROBLEM - cloud2 Current Load on cloud2 is WARNING: WARNING - load average: 23.08, 19.97, 15.44 [16:22:21] PROBLEM - cp7 Current Load on cp7 is WARNING: WARNING - load average: 4.41, 7.84, 4.77 [16:23:30] RECOVERY - cloud2 Current Load on cloud2 is OK: OK - load average: 10.23, 16.66, 14.82 [16:24:20] RECOVERY - cp7 Current Load on cp7 is OK: OK - load average: 1.05, 5.48, 4.28 [16:41:36] mutante: have you had any further issues with your account since? [16:43:21] Zppix: no, but i just woke up so there was nothing i did with it. i can confirm i can still login, it just showed one warning "action has been canceled as a precaution against session hijacking. Please resubmit the form" and then it worked [16:43:32] thanks for checking [16:43:42] mutante: ah that error is "normal" in the since that its known [16:43:52] sense rather [16:44:22] no problem its a goal of mine to increase follow through from MH staff mutante [17:18:07] Zppix: good job :) [17:33:32] PROBLEM - cp3 Disk Space on cp3 is WARNING: DISK WARNING - free space: / 2648 MB (10% inode=93%); [18:00:12] !log disabled a user on phab [18:00:19] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:28:20] PROBLEM - madgenderscience.wiki - DNS on sslhost is CRITICAL: DNS CRITICAL - expected '2001:41d0:800:1056::2,2001:41d0:800:105a::10,51.77.107.210,51.89.160.142' but got '2001:41d0:800:1056::2,51.77.107.210,51.89.160.142' [18:35:03] RECOVERY - madgenderscience.wiki - DNS on sslhost is OK: DNS OK: 0.033 seconds response time. sslhost returns 2001:41d0:800:1056::2,2001:41d0:800:105a::10,51.77.107.210,51.89.160.142 [19:02:05] [02miraheze/CreateWiki] 07paladox pushed 031 commit to 03paladox-patch-3 [+0/-0/±1] 13https://git.io/JJ4wC [19:02:07] [02miraheze/CreateWiki] 07paladox 03372de37 - Fix wikis being deleted [19:02:08] [02CreateWiki] 07paladox created branch 03paladox-patch-3 - 13https://git.io/vpJTL [19:02:10] [02CreateWiki] 07paladox opened pull request 03#155: Fix wikis being deleted - 13https://git.io/JJ4wl [19:02:42] [02CreateWiki] 07paladox synchronize pull request 03#155: Fix wikis being deleted - 13https://git.io/JJ4wl [19:02:44] [02miraheze/CreateWiki] 07paladox pushed 031 commit to 03paladox-patch-3 [+0/-0/±1] 13https://git.io/JJ4w4 [19:02:45] [02miraheze/CreateWiki] 07paladox 038d435e9 - Update CreateWikiJson.php [19:04:34] miraheze/CreateWiki/paladox-patch-3/372de37 - paladox The build passed. https://travis-ci.com/miraheze/CreateWiki/builds/177074043 [19:06:42] miraheze/CreateWiki/paladox-patch-3/8d435e9 - paladox The build passed. https://travis-ci.com/miraheze/CreateWiki/builds/177074097 [19:09:05] [02miraheze/CreateWiki] 07paladox pushed 031 commit to 03paladox-patch-3 [+0/-0/±1] 13https://git.io/JJ4wy [19:09:06] [02miraheze/CreateWiki] 07paladox 03faa6949 - Update WikiInitialise.php [19:09:08] [02CreateWiki] 07paladox synchronize pull request 03#155: Fix wikis being deleted - 13https://git.io/JJ4wl [19:13:06] [02CreateWiki] 07paladox synchronize pull request 03#155: Fix wikis being deleted - 13https://git.io/JJ4wl [19:13:08] [02miraheze/CreateWiki] 07paladox pushed 031 commit to 03paladox-patch-3 [+0/-0/±1] 13https://git.io/JJ4w5 [19:13:09] [02miraheze/CreateWiki] 07paladox 033d56d4f - Update CreateWikiJson.php [19:13:36] [02CreateWiki] 07paladox synchronize pull request 03#155: Fix wikis being deleted - 13https://git.io/JJ4wl [19:13:37] [02miraheze/CreateWiki] 07paladox pushed 031 commit to 03paladox-patch-3 [+0/-0/±1] 13https://git.io/JJ4wb [19:13:39] [02miraheze/CreateWiki] 07paladox 03f2714a4 - Update CreateWikiJson.php [19:14:18] [02CreateWiki] 07paladox synchronize pull request 03#155: Fix wikis being deleted - 13https://git.io/JJ4wl [19:14:20] [02miraheze/CreateWiki] 07paladox pushed 031 commit to 03paladox-patch-3 [+0/-0/±1] 13https://git.io/JJ4wx [19:14:22] [02miraheze/CreateWiki] 07paladox 03ec24593 - Update WikiInitialise.php [19:14:31] [02CreateWiki] 07paladox synchronize pull request 03#155: Fix wikis being deleted - 13https://git.io/JJ4wl [19:14:32] [02miraheze/CreateWiki] 07paladox pushed 031 commit to 03paladox-patch-3 [+0/-0/±1] 13https://git.io/JJ4wp [19:14:34] [02miraheze/CreateWiki] 07paladox 0308fc4ad - Update WikiInitialise.php [19:14:51] [02CreateWiki] 07paladox synchronize pull request 03#155: Fix wikis being deleted - 13https://git.io/JJ4wl [19:14:53] [02miraheze/CreateWiki] 07paladox pushed 031 commit to 03paladox-patch-3 [+0/-0/±1] 13https://git.io/JJ4wj [19:14:54] [02miraheze/CreateWiki] 07paladox 03450cdcd - Update CreateWikiJson.php [19:15:08] [02miraheze/CreateWiki] 07paladox pushed 031 commit to 03paladox-patch-3 [+0/-0/±1] 13https://git.io/JJ4rv [19:15:10] [02miraheze/CreateWiki] 07paladox 03ff53ea1 - Update WikiInitialise.php [19:15:11] [02CreateWiki] 07paladox synchronize pull request 03#155: Fix wikis being deleted - 13https://git.io/JJ4wl [19:18:42] [02CreateWiki] 07paladox edited pull request 03#155: Fix wikis being deleted - 13https://git.io/JJ4wl [19:20:12] Zppix: what is "eir" ? [19:20:41] mutante: manages our bans [19:20:52] mutante: we can set them to expire at x and eir will auto unban [19:21:04] its a freenode provided bot [19:21:17] Zppix: interesting. i just saw it remove a ban in -operations [19:21:21] and then i did a whois on it [19:21:31] and i noticed it is in wikimedia, miraheze and ..dogecoin [19:21:34] mutante: im more shocked you didnt know what it was xD [19:21:41] that immediately made me think "it must be Zppix" [19:21:46] because of that combo [19:21:51] mutante: are you implying im a bot xD [19:22:10] no, i am implying nobody else is Wikimedia, Miraheze and Dogecoin at the same time [19:22:16] [02CreateWiki] 07paladox closed pull request 03#155: Fix wikis being deleted - 13https://git.io/JJ4wl [19:22:17] except ..well.. you and me. lol [19:22:18] [02miraheze/CreateWiki] 07paladox pushed 031 commit to 03master [+0/-0/±2] 13https://git.io/JJ4rW [19:22:20] [02miraheze/CreateWiki] 07paladox 0392879eb - Fix wikis being deleted (#155) Bug: T5961 [19:22:21] [02miraheze/CreateWiki] 07paladox deleted branch 03paladox-patch-3 [19:22:23] [02CreateWiki] 07paladox deleted branch 03paladox-patch-3 - 13https://git.io/vpJTL [19:22:25] mutante: actually not true [19:22:29] heh, ok [19:22:41] mutante: Vermont is in all those [19:22:58] so all it does is remove expired bans? [19:23:02] [02miraheze/mediawiki] 07paladox pushed 031 commit to 03REL1_34 [+0/-0/±1] 13https://git.io/JJ4r4 [19:23:03] [02miraheze/mediawiki] 07paladox 03d5a2423 - Update CreateWiki [19:23:10] mutante: yes [19:23:21] i see https://github.com/freenode/eir [19:23:22] [ GitHub - freenode/eir ] - github.com [19:23:34] https://github.com/freenode/eir/blob/master/doc/bantracker/operators-guide.txt [19:23:34] [ eir/operators-guide.txt at master · freenode/eir · GitHub ] - github.com [19:23:39] Zppix: ok, thanks. i probably just forgot about it because i haven't been chanop in many years [19:23:57] in Efnet we had to run our own bots :p [19:24:29] That ban is on #wikimedia-bans anyway [19:25:04] miraheze/CreateWiki/paladox-patch-3/450cdcd - paladox The build has errored. https://travis-ci.com/miraheze/CreateWiki/builds/177075317 [19:26:17] miraheze/CreateWiki/paladox-patch-3/ff53ea1 - paladox The build has errored. https://travis-ci.com/miraheze/CreateWiki/builds/177075346 [19:32:17] [02miraheze/CreateWiki] 07paladox pushed 031 commit to 03paladox-patch-3 [+0/-0/±1] 13https://git.io/JJ4r9 [19:32:19] [02miraheze/CreateWiki] 07paladox 035bc9bb3 - Fix "Warning: Illegal string offset 'c'" if deleted.json doesn't exist on first run [19:32:20] [02CreateWiki] 07paladox created branch 03paladox-patch-3 - 13https://git.io/vpJTL [19:32:28] [02CreateWiki] 07paladox opened pull request 03#156: Fix "Warning: Illegal string offset 'c'" if deleted.json doesn't exist on first run - 13https://git.io/JJ4rH [19:33:35] [02CreateWiki] 07paladox synchronize pull request 03#156: Fix "Warning: Illegal string offset 'c'" if deleted.json doesn't exist on first run - 13https://git.io/JJ4rH [19:33:37] [02miraheze/CreateWiki] 07paladox pushed 031 commit to 03paladox-patch-3 [+0/-0/±1] 13https://git.io/JJ4rF [19:33:38] [02miraheze/CreateWiki] 07paladox 030643a35 - Update WikiInitialise.php [19:34:06] [02CreateWiki] 07paladox edited pull request 03#156: Check if deleted.json exists otherwise create an array - 13https://git.io/JJ4rH [19:35:35] miraheze/CreateWiki/paladox-patch-3/5bc9bb3 - paladox The build passed. https://travis-ci.com/miraheze/CreateWiki/builds/177077027 [19:36:36] !log root@jobrunner1:/var/log/mediawiki/debuglogs# rm /tmp/mw-UID* [19:36:47] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:37:52] miraheze/CreateWiki/paladox-patch-3/0643a35 - paladox The build passed. https://travis-ci.com/miraheze/CreateWiki/builds/177077180 [19:39:25] [02miraheze/MirahezeMagic] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/JJ4oU [19:39:27] [02miraheze/MirahezeMagic] 07paladox 0354e358c - Fix onGlobalUserPageWikis to support new 'combi' [19:39:28] [02MirahezeMagic] 07paladox created branch 03paladox-patch-1 - 13https://git.io/fQRGX [19:39:30] [02MirahezeMagic] 07paladox opened pull request 03#138: Fix onGlobalUserPageWikis to support new 'combi' - 13https://git.io/JJ4oT [19:40:49] [02MirahezeMagic] 07paladox closed pull request 03#138: Fix onGlobalUserPageWikis to support new 'combi' - 13https://git.io/JJ4oT [19:40:51] [02miraheze/MirahezeMagic] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JJ4ok [19:40:52] [02miraheze/MirahezeMagic] 07paladox 03d3f9085 - Fix onGlobalUserPageWikis to support new 'combi' (#138) [19:40:54] [02MirahezeMagic] 07paladox deleted branch 03paladox-patch-1 - 13https://git.io/fQRGX [19:40:55] [02miraheze/MirahezeMagic] 07paladox deleted branch 03paladox-patch-1 [19:41:25] [02miraheze/mediawiki] 07paladox pushed 031 commit to 03REL1_34 [+0/-0/±1] 13https://git.io/JJ4ot [19:41:26] [02miraheze/mediawiki] 07paladox 038aba6ce - Update MirahezeMagic [19:46:12] !log rebuild lc on test2, mw* and jobrunner1 [19:46:15] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:46:27] can someone look into [092d3e5321fd344e54080671] 2020-07-24 19:46:12: Fatal exception of type "Wikimedia\Rdbms\DBQueryError" for me? [19:46:49] getting that when trying to look at some specific GlobalRenameQueue entries [19:47:52] looking [19:48:01] Voidwalker: any idea what mw you are being served by? [19:48:26] check [f54864d206faf2700747e578] 2020-07-24 19:48:10: Fatal exception of type "Wikimedia\Rdbms\DBQueryError" on mw6 [19:49:32] lookin [19:50:35] Query: USE `admintrainingwiki` [19:50:35] Function: Wikimedia\Rdbms\DatabaseMysqlBase::doSelectDomain [19:50:35] Error: 1049 Unknown database 'admintrainingwiki' (db11.miraheze.org) [19:50:38] Voidwalker: [19:52:01] huh, that's odd [19:52:05] Voidwalker: so it looks like the wiki may of been deleted? [19:52:59] yup, paladox ^ [19:53:25] looking [19:53:36] maybe it was used as training for admins [19:53:57] mutante: i thought it was training monkeys [19:53:58] looks like the change we made to the handling of deleted wikis may have broken CentralAuth [19:54:30] Zppix: a reference to https://en.wikipedia.org/wiki/Infinite_monkey_theorem ? [19:54:30] [WIKIPEDIA] Infinite monkey theorem | "The infinite monkey theorem states that a monkey hitting keys at random on a typewriter keyboard for an infinite amount of time will almost surely type any given text, such as the complete works of William Shakespeare. In fact, the monkey would almost surely type every possible finite text an infinite..." [19:54:32] oh [19:54:33] you just looking at the special page for rename progress right? Voidwalker [19:54:43] mutante: i didnt even know that was a thing [19:54:52] Zppix: :) check it out [19:55:14] also check out https://meta.miraheze.org/wiki/Special:CentralAuth/Void [19:55:15] [ Database error - Miraheze Meta ] - meta.miraheze.org [19:55:52] noice [19:56:24] mutante: first you break your account and now you're breaking our central auth tsk tsk [19:56:57] [02miraheze/CreateWiki] 07paladox pushed 031 commit to 03paladox-patch-4 [+0/-0/±1] 13https://git.io/JJ4oi [19:56:58] [02miraheze/CreateWiki] 07paladox 03541ee2f - Always add deleted wikis to wikiDBClusters [19:57:00] [02CreateWiki] 07paladox created branch 03paladox-patch-4 - 13https://git.io/vpJTL [19:57:01] [02CreateWiki] 07paladox opened pull request 03#157: Always add deleted wikis to wikiDBClusters - 13https://git.io/JJ4oP [19:57:16] Voidwalker would something like https://github.com/miraheze/CreateWiki/pull/157/files work for you? [19:57:17] [ Always add deleted wikis to wikiDBClusters by paladox · Pull Request #157 · miraheze/CreateWiki · GitHub ] - github.com [19:57:42] Zppix: shooting the messenger much :p [19:57:54] heh [19:58:40] it probably will, should we test? [19:59:14] [02CreateWiki] 07paladox closed pull request 03#157: Always add deleted wikis to wikiDBClusters - 13https://git.io/JJ4oP [19:59:15] [02miraheze/CreateWiki] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JJ4o9 [19:59:17] [02miraheze/CreateWiki] 07paladox 033b3e9a3 - Always add deleted wikis to wikiDBClusters (#157) [19:59:18] [02CreateWiki] 07paladox deleted branch 03paladox-patch-4 - 13https://git.io/vpJTL [19:59:20] [02miraheze/CreateWiki] 07paladox deleted branch 03paladox-patch-4 [19:59:44] [02miraheze/mediawiki] 07paladox pushed 031 commit to 03REL1_34 [+0/-0/±1] 13https://git.io/JJ4o7 [19:59:46] [02miraheze/mediawiki] 07paladox 037f4894a - Update CreateWiki [20:00:50] Voidwalker works [20:01:25] yup, seems to be [20:03:50] now we shall see what happens to the rename [20:05:54] seems to have handled the deleted wiki just fine [20:08:46] Can you also delete local cvt group on Meta? [20:09:57] @MrJaroslavik I asked @Reception123 about that and he said it probably should be a steward that does that. [20:10:05] but yeah, I would support that as well. 🙂 [20:10:14] i think the local cvt group is there [20:10:15] for a reason [20:10:34] Void is not a steward? [20:10:48] since the GS global group doesnt actually have permissons on meta [20:10:50] Sorry, didn't know who you were addressing it to. [20:11:31] There's no cvt members, though, @Zppix, and that wouldn't help anyway. All GSes are local Meta sysops so they can act on Meta as local Meta sysops. [20:11:53] not really when i first joined CVT i wasnt a sysadmin nor a local meta sysop [20:12:29] the only reason i dont have the local group is cause im both xD [20:12:34] yeah, I know, but I don't think there's any plan for CVT to have any members, so to me, it just seems like needless clutter. Harmless clutter, to be sure. But clutter nonetheless. [20:12:45] You're a Meta sysop, though. [20:12:50] if someone were to become cvt that isnt either they would be given that group [20:13:21] it could probably use a rename though [20:13:23] doesn't seem to want to be deleted, it has an empty right clobbered on that is preventing deletion [20:13:29] I don't think there's technically a policy basis for CVT, though...it exists as an umbrella name for Stewards + GS, as I understand it. [20:13:35] Voidwalker: i think it may be defined in LS [20:13:48] doubtful, this was a bug [20:14:58] thing is you have to have a local group on meta or you wouldnt be able to access global (b)lock pages [20:15:22] if you didnt have access from another usergroup that is [20:16:05] nah, the specific thing with the empty user right was a bug from somewhere, but I forgot what specifically it was [20:17:03] And anything about requested deletion of (unused) 'translator' group? [20:17:40] probably gonna hit the same issue, so I'm gonna leave it [20:18:08] Ok [20:23:37] Sorry, for bother you, but when i see you, can you check https://meta.miraheze.org/wiki/Stewards%27_noticeboard/Archive_11#Request_for_inactivity_exempt? [20:23:37] [ Stewards' noticeboard/Archive 11 - Miraheze Meta ] - meta.miraheze.org [20:27:42] PROBLEM - storytime.jdstroy.cf - DNS on sslhost is CRITICAL: DNS CRITICAL - expected '2001:41d0:800:1056::2,2001:41d0:800:105a::10,51.77.107.210,51.89.160.142' but got '2001:41d0:800:1056::2,51.77.107.210,51.89.160.142' [20:29:32] PROBLEM - bacula2 Puppet on bacula2 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): File[/usr/local/bin/puppet-enabled] [20:30:48] paladox: whats up with the DNS alert? [20:31:16] looks like a cp got depooled. [20:31:22] 2001:41d0:800:105a::10 [20:31:30] cp7 [20:33:34] RECOVERY - bacula2 Puppet on bacula2 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [20:34:24] RECOVERY - storytime.jdstroy.cf - DNS on sslhost is OK: DNS OK: 0.032 seconds response time. sslhost returns 2001:41d0:800:1056::2,2001:41d0:800:105a::10,51.77.107.210,51.89.160.142 [20:47:40] @System Administrators I just purged https://meta.miraheze.org/wiki/Miraheze after creating two more wikis and the number of wikis dropped to 3,751. [20:47:41] [ Meta ] - meta.miraheze.org [20:47:46] Caching issue somewhere? [20:49:26] Sounds about right after our deleted wiki fix [20:49:36] select * from cw_wikis where wiki_deleted = 0; [20:49:39] shows 3751 [20:49:42] heh paladox beat me to the answer [20:52:03] You just deleted roughly 1,000 wikis? [20:52:30] well yeh, over 1k wikis are in deleted state [20:52:42] ah, okay, that makes sense then. cool [20:52:43] but not permanently deleted yet [20:52:54] glad to know the CW fix was implemented [20:53:12] checks to see if cwtestwiki is now deleted [20:53:35] yep! 🙂 [20:54:12] Cool. Were any of them wikis that had been in a closed state for > 180 days? [20:54:23] wonders if subdomain is deleted too [20:57:46] !log root@jobrunner1:/srv/mediawiki/w/extensions/CreateWiki/maintenance# sudo -u www-data php deleteWikis.php --wiki loginwiki --delete paladox [20:57:49] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:58:19] !log DROP DATABASE abominablememefandomswiki; [20:58:22] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:58:57] paladox or Zppix, so by not permanently deleted, do you mean the wiki dbnames haven't been removed from all the configuration files yet, or just that the actual database files still exist in a secure, private location on the server? [20:59:18] !log DROP DATABASE bigbrotherwikiwiki; [20:59:28] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:59:29] !log DROP DATABASE lousyreceptionwikiswiki; [20:59:34] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:59:38] Big database purge day today. 🙂 [20:59:52] !log DROP DATABASE wikicubiawiki; [20:59:56] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:59:57] looks at my [[Special:CentralAuth]] to see if my attached accounts dropped [21:00:04] only like 4 dbs out of a 1,000 plus :( [21:00:08] @doug perm deletion is when we actually delete the database from our servers [21:00:21] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JJ46H [21:00:23] [02miraheze/services] 07MirahezeSSLBot 03ed36b39 - BOT: Updating services config for wikis [21:01:00] ah, thanks @Zppix and @paladox. I imagine permanent deletion, then, probably doesn't occur for some time, then, in case the owner wishes to recover their data, eh? At least 6 more months, or more or less? [21:01:22] yup [21:01:43] Sounds good. The main thing is to purge it from CreateWiki and the DNS zone file. 🙂 [21:02:14] only if the wiki has a custom domain and decided to use our namespace over mw-lb [21:03:43] ah, thanks @paladox, I should check the Miraheze zone file. Is it wildcarded and then the subdomains are defined in LocalSettings.php and some other file? [21:04:02] it's wildcarded, and then individual subdomains (for the servers) [21:04:28] ah, okay, thanks, that makes sense. Less updates to the zone file for every wiki creation. Can you imagine? 😛 [21:04:53] yup [21:05:31] @doug if i had to add zones to the dns for every new wiki I may just have to check into a insane asylum [21:06:10] Voidwalker: i see renames are going ok [21:06:33] Seems so [21:06:40] miraheze.org 21:02, 15 July 2020 created on login(?) — 0 bureaucrat, sysop Is that a bug? [21:06:48] Voidwalker: you purposefully doing one at a time? [21:06:58] @doug is what a bug [21:07:07] I'm showing as a "bureaucrat, sysop" on the root domain miraheze.org. [21:07:07] yeah, if something goes wrong, I don't wanna lock a whole bunch of users out of their accounts [21:07:11] which isn't a wiki [21:07:25] also, I think the jobs will run better if I don't have a bunch going at the same time [21:07:30] Voidwalker: hmm that would make me want to do it more xD jk [21:07:52] @Doug that's an artefact of a deleted wiki [21:07:58] .mhca Dmehus [21:07:58] https://meta.miraheze.org/wiki/Special:CentralAuth/Dmehus [21:08:28] it shows miraheze.org instead of the actual subdomain because the wiki doesn't exist but hasn't been permanently deleted (the DB still exists) [21:08:32] @Doug doesnt show any usergroups for you on my end https://usercontent.irccloud-cdn.com/file/m9jgRcUd/image.png [21:09:31] thanks, @Void and @Zppix...weird because I do see those user rights on my end. Must be a local only thing for the user in question. Zppix, look yourself up. I'm betting it'll show as bureaucrat, sysop for you. [21:10:08] I'm seeing it too Zppix, there are about a dozen of those entries for his account [21:10:32] try clearing your caches [21:10:41] if that doesnt do anything try cookies maybe? [21:10:45] and no, it won't show up as bureaucrat, sysop for all wikis where this is the case, just whatever deleted wiki you were on that you had advanced rights [21:10:55] @Void You mean I have more than one "miraheze.org" entry? Interesting. [21:10:59] ah, okay [21:11:08] in my case, it's because of cwwikitest then [21:11:26] as that's the only deleted wiki I held anything more than autopatrolled on [21:11:28] https://cdn.discordapp.com/attachments/435711390544560128/736329728352583820/unknown.png [21:11:28] cause im looking at the CA for Doug from meta and all looks normal to me [21:11:51] i dont see any of that on my end Voidwalker [21:11:56] perfectly normal though [21:12:03] Zppix, sort the list by wiki [21:12:09] that should clump them together [21:12:21] weird now i do... [21:12:40] i wonder if it is anything to do with the change paladox made with CreateWiki? [21:12:51] Would it sold it, Void and Zppix, if we moved the database to a different folder? [21:13:05] The system would think, then, the db is deleted. [21:13:19] no, this is old, deleted wikis have shown up like this in CentralAuth for a long time [21:13:37]