[00:03:16] PROBLEM - cp8 Current Load on cp8 is WARNING: WARNING - load average: 1.77, 1.28, 0.90 [00:05:19] RECOVERY - cp8 Current Load on cp8 is OK: OK - load average: 1.09, 1.12, 0.89 [05:02:45] PROBLEM - en.wikireserve.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'en.wikireserve.com' expires in 15 day(s) (Sun 05 Jul 2020 05:00:27 GMT +0000). [05:04:06] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JfFPN [05:04:08] [02miraheze/ssl] 07MirahezeSSLBot 03abbc921 - Bot: Update SSL cert for en.wikireserve.com [05:12:49] RECOVERY - en.wikireserve.com - LetsEncrypt on sslhost is OK: OK - Certificate 'en.wikireserve.com' will expire on Thu 17 Sep 2020 04:03:59 GMT +0000. [05:33:42] [02miraheze/ssl] 07Reception123 pushed 031 commit to 03master [+1/-0/±1] 13https://git.io/JfFX8 [05:33:43] [02miraheze/ssl] 07Reception123 03b7de6fb - add wikiverte.pl cert [05:36:37] [02miraheze/ssl] 07Reception123 pushed 031 commit to 03master [+1/-0/±1] 13https://git.io/JfFXE [05:36:38] [02miraheze/ssl] 07Reception123 0318867e1 - add wiki.ssij.se cert [05:38:26] [02miraheze/ssl] 07Reception123 pushed 031 commit to 03master [+1/-0/±1] 13https://git.io/JfFXa [05:38:27] [02miraheze/ssl] 07Reception123 03500cfdd - add aman.awiki.org cert [05:45:12] [02puppet] 07Reception123 commented on pull request 03#1388: Add Reddit and Dropbox to CSP whitelist T5685 - 13https://git.io/JfFXD [05:45:49] [02puppet] 07Reception123 edited a comment on pull request 03#1388: Add Reddit and Dropbox to CSP whitelist T5685 - 13https://git.io/JfFXD [09:20:23] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JfFQN [09:20:24] [02miraheze/services] 07MirahezeSSLBot 031a752e5 - BOT: Updating services config for wikis [09:47:23] Unexpected error (list index out of range) from catbeard at 2020-06-19 09:47:23.026655. Message was: nope [10:28:15] .mh sal [10:28:15] https://meta.miraheze.org/wiki/sal [10:28:29] morning JohnLewis [10:28:43] Morning [10:30:42] JohnLewis: how you doing today? [10:31:02] JohnLewis: do you have access as well? [10:32:57] I do [10:33:08] JohnLewis: see PM [10:38:35] !log MariaDB [factoriomodswiki]> INSERT INTO content_models (model_id, model_name) VALUES (1, 'wikitext'); [10:38:39] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [10:38:40] !log MariaDB [factoriomodswiki]> INSERT INTO content_models (model_id, model_name) VALUES (5, 'json'); [10:38:44] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [10:38:48] !log MariaDB [factoriomodswiki]> INSERT INTO content_models (model_id, model_name) VALUES (3, 'javascript'); [10:38:52] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [10:38:53] !log MariaDB [factoriomodswiki]> INSERT INTO content_models (model_id, model_name) VALUES (2, 'css'); [10:38:54] !log MariaDB [factoriomodswiki]> INSERT INTO content_models (model_id, model_name) VALUES (4, 'Scribunto'); [10:38:56] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [10:38:59] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [10:40:15] paladox: ^ why does that happen? [11:14:28] !log running wikibackups on jobrunner1 (bash ./wikibackups.sh /home/reception/backupwikis.dblist /srv/mediawiki/w/maintenance/dumpBackup.php) [11:14:32] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [12:06:19] Looking at the wrong person :). I only know how to fix it, I’m not sure why it breaks. [12:08:06] paladox: same here, It seems random. [12:13:17] PROBLEM - cloud1 Current Load on cloud1 is CRITICAL: CRITICAL - load average: 24.57, 23.71, 18.46 [12:15:12] RECOVERY - cloud1 Current Load on cloud1 is OK: OK - load average: 14.50, 20.30, 17.83 [12:15:27] goddamn [12:19:04] hi noteness [12:19:42] That's some load average! [12:19:48] Hello RhinosF1 [12:20:29] noteness: it'll be fine [12:21:12] RhinosF1, is it like a 16c machine? [12:21:56] hi PuppyKun [12:22:18] noteness: just a very random fluke lasting a minute or 2 [12:22:35] :O [12:23:14] * RhinosF1 checks the service history [12:24:09] paladox: that's 1st time today it's lasted long enough to trigger an alert but it's on and off. known? [12:32:54] PuppyKun: please don't forget to check my access request [12:56:27] Yup, i'm aware now. [12:56:29] https://grafana.miraheze.org/d/W9MIkA7iz/miraheze-cluster?orgId=1&var-job=node&var-node=cloud1.miraheze.org&var-port=9100 [12:56:29] [ Grafana ] - grafana.miraheze.org [12:58:18] thanks paladox [13:29:45] Good morning [13:34:01] hi AmandaCath [13:34:46] In case anyone who cares missed it, breaking news out of Wikipedia https://en.wikipedia.org/wiki/Wikipedia:Arbitration_Committee/Noticeboard#Revocation_of_CheckUser_access_for_Bbb23 [13:34:47] [WIKIPEDIA] Wikipedia:Arbitration Committee/Noticeboard#Revocation of CheckUser access for Bbb23 | "..." [13:35:32] AmandaCath: most here dont care [13:35:42] On a somewhat related note, reading through the global RFC, are we essentially proposing turning the code of conduct commission into an ArbCom-like body? [13:36:03] Yes [13:36:18] Huh [13:36:27] I belive proposal 1 even uses the word arbitration [13:37:45] It's job is to arbitrate conduct disputes [13:38:19] Which is essentially ArbCom [13:38:33] * AmandaCath is not sure if Miraheze really needs that at this point [13:40:55] We need something other than the CoCC [13:41:08] The CoCC are not fit for purpose [13:50:24] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JfbeP [13:50:25] [02miraheze/services] 07MirahezeSSLBot 0370c3b8a - BOT: Updating services config for wikis [14:25:24] PROBLEM - ns1 Current Load on ns1 is CRITICAL: CRITICAL - load average: 2.89, 2.17, 1.27 [14:27:26] RECOVERY - ns1 Current Load on ns1 is OK: OK - load average: 0.38, 1.50, 1.24 [14:55:03] * hispano76 greetings [15:03:27] hi hispano76 [15:06:11] :) [15:08:20] So if services is going offline, does that mean the channel will go offline too? [15:09:02] AmandaCath: no [15:09:26] it won't affect us at all [15:10:29] we have numerous bots that for the time they're down can handle the basics [15:21:53] RhinosF1: are your anti-troll rules sensitive to freenode user cloaks ? Seems like I should be able to get an unaffiliated cloak for my bouncer [15:29:34] * AmandaCath thinks that we need to add a note on Special:RequestWiki telling people to provide a description of their wiki's purpose [15:29:44] I've written "Please provide more information about the purpose of your wiki" so many times that it has become second nature to me :/ [15:31:37] Also what's been holding up the PRs lately? I count 7 open requests in mw-config and 10 open requests in puppet, many from several days/weeks ago [15:32:26] Unexpected error (list index out of range) from catbeard at 2020-06-19 15:32:25.446394. Message was: nope [15:33:08] Meh, net splits [15:33:18] Are so annoying [15:40:54] lastmikoi: it won't be able to tell then [15:41:24] .in 90mins fix rw i18n for AmandaCath [15:41:25] RhinosF1: Okay, will remind at 2020-06-19 - 18:11:25BST [15:41:44] ? [15:42:33] Oh, "rw i18n" = RequestWiki interface [15:42:36] Duh [15:42:40] AmandaCath: Yep :) [15:42:45] * AmandaCath slaps herself with a wet trout [15:42:49] And I think you just got a shiny cloak [15:42:56] Yep [15:43:18] i'll guess JohnLewis sorted it [15:45:27] Hello nostrooo! If you have any questions, feel free to ask and someone should answer soon. [15:45:28] AmandaCath: interesting case [15:45:53] RhinosF1: heh another commission is hard, it would mean the same users would have tons of responsibilities and tons of commissions [15:47:17] Hi nostrooo. May we help you with anything? [15:48:41] Reception123: yeah, it was definitely a long time coming [15:55:46] AmandaCath, Reception123: Chamelon is under active development [15:56:40] lastmikoi: Hm? [15:57:01] Sygnal and SigmaBot should be lenient on unaffiliated cloaks [15:57:46] Texas: it kicked him for globally blocked on mh last night [15:57:55] Oh [15:58:03] *user lastmikoi identify [15:58:04] I recognize lastmikoi as lastmikoi. [15:58:04] Can we add exemptions to sygnal [15:58:14] RhinosF1: Cloaks are exempt [15:58:27] Texas: he doesnt at moment though [15:58:45] RhinosF1: hmm, well shouldn't it at least be functional after 2 years? [15:59:02] Reception123: given the WMF want to use it, I'd hope so. [15:59:10] RhinosF1: Sygnal should only get him if he says something nasty about someone or if he is using an open proxy [15:59:18] RhinosF1: WMF wants what now? [15:59:39] Texas: it is a webhost so globally blocked [15:59:58] A new skin for a new wik [16:00:39] Oh [16:00:55] RhinosF1: I can exempt his nickserv account from Sygnal [16:01:06] Good idea [16:03:36] RhinosF1: This reset should exempt lastmikoi from any proxy or global block checks [16:03:40] !reset [16:03:56] Thanks Texas [16:04:01] np [16:04:27] thanks ! [16:05:05] np :) [16:06:04] AmandaCath: please just ask for revdel on irc rather than stating clearly that PII is visible on wiki [16:06:22] lastmikoi: You may have noticed that we have been having some serious troll issues lately and we have had to take somewhat drastic measures to put a stop to it. That said, we certainly don't want to keep good faith users out. If you run into anymore issues, please let me or an operator know. [16:15:14] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Jfbkp [16:15:15] [02miraheze/services] 07MirahezeSSLBot 032f2a71f - BOT: Updating services config for wikis [16:15:33] RhinosF1: I mean, some people will choose to out themselves (Robert Sterbal for example) [16:15:43] So I was just advising them that it's not recommended [16:15:48] I didn't see it as an emergency [16:16:15] AmandaCath: that wasn't optional. If you're removing it, don't draw attention to it. [16:16:37] PROBLEM - civclassic.civwiki.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'civclassic.civwiki.org' expires in 15 day(s) (Sun 05 Jul 2020 16:13:47 GMT +0000). [16:17:33] But I was removing it solely out of an abundance of caution. If they reinstated it it's not something that I would have edit warred over [16:17:37] Texas: yup, I've noticed. I'm not a user of miraheze's services but I'm keeping in touch in this channel regarding ongoing business ( SPF|Cloud being my contact ) so it's not that much of an issue if I do get kicked out for a few days [16:17:51] good luck with those trolls [16:18:04] Ah [16:18:10] Yes we will need it. [16:18:32] Apparently we are dealing with two different ones at the same time [16:21:37] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JfbIG [16:21:39] [02miraheze/ssl] 07MirahezeSSLBot 03c74ed44 - Bot: Update SSL cert for civclassic.civwiki.org [16:31:48] If someone has a minute I'd like a second pair of eyes to review https://meta.miraheze.org/wiki/Special:RequestWikiQueue/12616 [16:31:49] [ Wiki requests queue - Miraheze Meta ] - meta.miraheze.org [16:32:39] RECOVERY - civclassic.civwiki.org - LetsEncrypt on sslhost is OK: OK - Certificate 'civclassic.civwiki.org' will expire on Thu 17 Sep 2020 15:21:31 GMT +0000. [16:34:46] Also I don't know what's holding this up but if this could please be merged sooner rather than later https://github.com/miraheze/mw-config/pull/3114 [16:34:47] [ fix simcitywiki configuration by Amanda-Catherine · Pull Request #3114 · miraheze/mw-config · GitHub ] - github.com [16:34:58] Reception123 paladox ^ [16:35:53] can someone add +e to hd1 [16:46:49] RhinosF1: Is he getting caught by a bot or something else? [16:47:00] He should consider getting a bouncer and a cloak [16:47:15] Texas: an actual ban [16:48:19] Oh [16:48:32] RhinosF1: Is it the kiwi ban? [16:48:51] Texas: check the ban list [16:49:08] RhinosF1: I think it is https://873gear.com/irc/uploads/71209aa9981c4962/image.png [16:49:12] yep [16:49:18] paladox, Reception123: ^ [16:50:37] !ops We need to unban *!*@gateway/web/cgi-irc/kiwiirc.com/* or set +e on hd1!*@* [16:50:38] * SigmaBot yelps for help [16:51:24] * RhinosF1 here but this is one of 3 channels I don't have local access on [16:52:24] RhinosF1: Technically we are global chanops but I don't think this is really in scope [16:52:43] Texas: that's why I said local access :) [16:52:49] :) [16:53:07] * RhinosF1 will only use it when he's fixing or applying global bans [16:53:54] RhinosF1: I only use it when there is a known LTA present. Otherwise I leave it to the locals. [16:54:15] Texas: that's the only reason I'd be dealing with a global ban [16:54:22] true [16:54:37] * RhinosF1 has used *remove to be quick before until he can find decent global bans [16:56:54] same [16:57:04] PROBLEM - cp8 Current Load on cp8 is CRITICAL: CRITICAL - load average: 2.49, 2.04, 1.22 [16:57:10] I need to make a way to automatically gban a host [16:57:32] I don't have time to today [16:57:35] Texas: make it do a decent range [16:57:46] So xxx.xxx.xxx.* [16:58:11] * RhinosF1 gban'd 3 ISPs and most of spain earlier [16:58:47] PROBLEM - cp8 Disk Space on cp8 is CRITICAL: DISK CRITICAL - free space: / 896 MB (4% inode=93%); [16:58:59] RECOVERY - cp8 Current Load on cp8 is OK: OK - load average: 0.91, 1.67, 1.18 [16:59:12] You seem happy icinga-miraheze [16:59:24] RhinosF1: Most of Spain??? [16:59:43] Texas: Joaquinito's main range [16:59:47] Oh [16:59:54] Welp there goes ChanServ [16:59:59] It's a very dynamic one [17:00:02] I saw there was some planned maintenance today [17:00:12] * RhinosF1 plans to remove it later [17:00:15] k [17:00:30] so that maintenance was cancelled? [17:00:37] Hm? [17:00:39] No [17:00:43] No we are talking about on IRC [17:00:44] It's happening now [17:00:57] Network Services just went down [17:01:28] 10:05:24 Notice: tomaw [Global Notice] In a little under two hours we will be taking services (ChanServ, NickServ etc) offline for a quick reboot. Further updates will be via WALLOPS, /mode yourNick +w to receive them. [17:01:28] 10:31:47 Notice: tomaw Ahead of services maintenance later we need to move a few services to a different hub. This will be a little noisey but should be quick. [17:07:47] Hello e! If you have any questions, feel free to ask and someone should answer soon. [17:11:26] RhinosF1: fix rw i18n for AmandaCath [17:11:45] What is that issue? Maybe I can fix. [17:17:03] Texas: add a note to get people to write a description [17:17:30] RhinosF1: It said a fix... [17:17:51] Texas: I know what it sais [17:30:10] !sre just taken 15099ms to load (PHP7 at mw7) [17:30:26] paladox, Reception123, SPF|Cloud, JohnLewis: ^ [17:30:28] RhinosF1: I think we are the only people that stalk that :P [17:30:43] 255ms (PHP7 at mw6) [17:31:19] paladox: mw5 higher on meta [17:31:26] Testwiki fine [17:32:08] 384ms (PHP7 at mw5) [17:32:26] paladox: Them times are consistent for me [17:32:35] everything seems fine for me [17:32:47] I just loaded meta's main page and got similar times on 3 servers [17:33:00] Also cp8 needs its disk space clearing [17:33:26] Minimum = 116ms, Maximum = 140ms, Average = 122ms On mw7 [17:33:47] paladox: Try https://meta.miraheze.org/wiki/Special:CentralAuth [17:33:50] [ Global account information - Miraheze Meta ] - meta.miraheze.org [17:33:58] uh [17:33:58] When I search for Zalgo [17:34:02] that will take long [17:34:07] It's really slow [17:34:12] Every other page fine [17:34:15] That slow? [17:34:18] RhinosF1: To load in a browser? [17:34:25] I don't think it is slow [17:34:52] Oh yeah [17:34:52] paladox: I think I found the issue [17:35:03] When I search for Zalgo it is slow [17:35:29] but Sourav Halder loads fine [17:35:48] and https://meta.miraheze.org/wiki/Special:CentralAuth?target=Sourav+bot [17:35:49] [ Global account information for Sourav bot - Miraheze Meta ] - meta.miraheze.org [17:35:51] It's any account that doesn't exist [17:36:03] Lol I had no idea Sourav made a bot [17:36:03] But it's not spiking that high on wmf [17:36:21] Yeah wierd [17:36:23] Load times go from ~250 - ~750 on wmf [17:36:39] RhinosF1: I notice hardly any difference on WMF [17:37:16] Miraheze is ~250 - > 1500-15000 [17:37:38] Texas: ~.5s so not noticeable [17:37:47] yeah [17:38:06] why is sygnal still opped in here? [17:38:17] did someone trigger her? [17:38:41] paladox: something isn't right there [17:39:00] hmm? [17:39:06] RhinosF1: Do you have any idea what it would be? [17:39:31] paladox: load times shouldn't surge that much [17:40:00] Texas: sygnal or central auth? [17:40:05] tbh i would agree if we had more infra + used ssds [17:40:23] i'm not sure sure about it now because we have only 2 servers & also they are hdds [17:40:47] paladox: how long does a direct db query take? [17:41:07] i'm not sure [17:41:11] https://grafana.miraheze.org/d/MQWgroiiz/mysql-overview?orgId=1&refresh=1m may help you [17:41:12] [ Grafana ] - grafana.miraheze.org [17:41:57] paladox: I meant sampling some of the queires central auth runs [17:42:15] how would i find the queries CA runs? [17:42:52] PROBLEM - cp8 Current Load on cp8 is WARNING: WARNING - load average: 1.89, 1.81, 1.31 [17:44:40] paladox: https://github.com/wikimedia/mediawiki-extensions-CentralAuth/blob/master/includes/specials/SpecialCentralAuth.php maybe? [17:44:41] [ mediawiki-extensions-CentralAuth/SpecialCentralAuth.php at master · wikimedia/mediawiki-extensions-CentralAuth · GitHub ] - github.com [17:44:49] what line? [17:44:51] PROBLEM - cp8 Current Load on cp8 is CRITICAL: CRITICAL - load average: 2.23, 1.82, 1.36 [17:45:33] [02mw-config] 07paladox deleted branch 03Reception123-patch-1 - 13https://git.io/vbvb3 [17:45:35] [02miraheze/mw-config] 07paladox deleted branch 03Reception123-patch-1 [17:46:06] paladox: thanks [17:46:25] paladox: I'm not sure exactly. I'll try and look at the tables. [17:46:43] [02puppet] 07paladox commented on pull request 03#1388: Add Reddit and Dropbox to CSP whitelist T5685 - 13https://git.io/JfbmO [17:46:52] RECOVERY - cp8 Current Load on cp8 is OK: OK - load average: 0.99, 1.46, 1.28 [17:47:07] paladox: https://www.mediawiki.org/wiki/Extension:CentralAuth/localuser_table [17:47:07] [ Extension:CentralAuth/localuser table - MediaWiki ] - www.mediawiki.org [17:47:14] Most of the data is in there [17:47:54] So SELECT * from localuser where lu_name = 'Zalgo' [17:48:36] RhinosF1: But how would that help us solve the slow loading problem? It may not be an issue with the database. [17:48:38] It'll start there [17:48:44] I mean I guess it could but... [17:48:50] It is a little bit of a long shot [17:48:50] [02puppet] 07Reception123 commented on pull request 03#1388: Add Reddit and Dropbox to CSP whitelist T5685 - 13https://git.io/JfbmC [17:49:34] Texas: it is a long shot. I'm hoping if it's the database we can benchmark. [17:49:45] Yeah [17:50:13] If it's the database, that query and the same on the globaluser table will be slow [17:50:38] Which is lu_ -> gu_ [17:51:30] paladox: can you run the sql on both localuser and globaluser? [17:51:52] running select * from localuser; would defo be slow, but it doesn't do that. [17:52:35] 697 rows in set (0.016 sec) [17:52:37] when i did: [17:52:45] select * from localuser where lu_name = 'Paladox'; [17:53:07] select * from globaluser where gu_name = 'Paladox'; [17:53:11] 1 row in set (0.001 sec) [17:53:15] paladox: do it for = zalgo [17:53:27] MariaDB [mhglobal]> select * from globaluser where gu_name = 'zalgo'; [17:53:27] Empty set (0.001 sec) [17:53:39] MariaDB [mhglobal]> select * from localuser where lu_name = 'zalgo'; [17:53:39] Empty set (0.001 sec) [17:53:50] paladox: it's not the database then [17:53:58] nope [17:54:04] * Texas knew it all along [17:54:05] :P [17:54:11] Why's it taking so long then [17:54:24] Texas: I'm ruling stuff out [17:54:28] RhinosF1: Probably the MediaWiki server [17:54:34] That is my guess [17:54:47] * Texas is often wrong though [17:55:19] Texas: i wouldn't expect it to be on such a spefic case then [17:55:36] Yeah it is strange [17:55:36] It has to be something code wise [17:56:18] Yeah [17:56:22] But what though [17:56:34] We do not hack core or extension files [17:57:02] paladox: can you dump them times on a phab paste and the query for both you and zalgo [17:57:45] Texas: then it's something upstream on REL1_34 [17:58:01] Yeah [17:58:10] Or the wmf just have better servers so it's less impacting [17:58:13] RhinosF1: Technically WMF is running 1.35 [17:58:15] I think [17:58:20] https://phabricator.miraheze.org/P319 [17:58:21] [ ✎ P319 (An Untitled Masterwork) ] - phabricator.miraheze.org [17:58:23] Texas: I kmow that [17:58:31] though this page has took ages to load for years [17:58:37] 1.35.0-wmf.37 (0a00c54) on meta [17:58:43] so it's not a new issue [17:59:23] paladox: Then we should be more proactive but I don't have a time machine [17:59:40] * Texas has a time machine [18:00:15] Texas: Can I use it? [18:00:41] RhinosF1: Hmm, for 300,000 USD per trip sure [18:01:01] * Texas was just being an idiot and has no time machine [18:01:29] * RhinosF1 does wish he had one but that won't solve the problem [18:01:53] I guess I should moan upstream once I gather stats [18:03:42] .in 3hours file bug about CA [18:03:43] RhinosF1: Okay, will remind at 2020-06-19 - 22:03:42BST [18:04:00] .in 3hours https://phabricator.miraheze.org/P319 [18:04:01] RhinosF1: Okay, will remind at 2020-06-19 - 22:04:01BST [18:04:01] [ ✎ P319 (An Untitled Masterwork) ] - phabricator.miraheze.org [18:28:45] * RhinosF1 will fix a task then ask a DBA [18:30:24] Ok [19:54:15] *ungban *[m]!*@* [19:54:15] 03*[m]!*@* is no longer globally banned and may now join channels03 [19:54:42] Reception123, paladox: Can you guys remove that ban and invite eth01[m] back? [20:10:36] \o/ [20:11:18] Sorry about that eth01[m]. We have been having a ton of trolls lately. [20:11:23] *user eth01[m] add [20:11:23] 03Success!03 eth01[m] was added to userfile as handle: eth01[m] using hostmask: eth01[m]!sid271749@fosshost/founder/eth01 [20:11:24] eth01[m] granted standard access. [20:11:32] *user eth01[m] identify [20:11:32] I recognize eth01[m] as eth01[m]. [20:11:51] Texas: ^ add it using the hostmask in telnet [20:12:43] RhinosF1: Okay. I really need to add a frontend command for that [20:12:48] * eth01 has now a vhost for his project :) [20:12:52] * Texas may start doing that now [20:13:18] eth01: makes it very easy to whitelist you [20:13:22] how you guys doing? [20:13:26] Busy [20:13:48] sure, I use irccloud but a project vhost overides a normal vhost, I was surprised tbh I didn't know, so thats pretty cool [20:13:54] Yeah, I know that feeling! [20:14:00] How did you guys get on with testing your VM/s? [20:20:22] paladox: ^ [20:21:53] *rehash [20:21:58] Rehashing...... [20:22:31] *addhost eth01[m] *!*@fosshost/founder/eth01 [20:22:55] *user eth01 identify [20:22:55] 04Error!04 RhinosF1, you must first add eth01 to the userfile. [20:23:04] RhinosF1: Hang on [20:23:08] there was an error [20:23:14] Ah [20:23:24] *user Naleksuh add [20:23:25] 04Error!04 RhinosF1, Naleksuh already exists on userfile. [20:23:36] Good [20:23:48] eth01: we'll have you in the userfile soon [20:24:10] * Texas is just tinkering with his silly bto [20:24:13] bot [20:24:33] ok ;) [20:24:42] *rehash [20:24:44] Reception123 was playing around with the vms [20:24:50] fuck [20:24:53] goodbye [20:25:12] paladox: have you seen reedy in -operations [20:25:21] i just saw [20:25:52] paladox: let me know what we can do / if you need help [20:29:03] oh good SigmaBot is back [20:29:15] *addhost eth01[m] *!*@fosshost/founder/eth01 [20:29:50] *addhost eth01[m] *!*@fosshost/founder/eth01 [20:29:50] *!*@fosshost/founder/eth01 has been added to eth01[m] [20:29:53] yayy [20:30:09] it worked! [20:30:14] *delhost eth01[m] *!*@fosshost/founder/eth01 [20:30:39] *delhost eth01[m] *!*@fosshost/founder/eth01 [20:30:39] *!*@fosshost/founder/eth01 has been removed from eth01[m] [20:31:24] [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JfbZS [20:31:26] [02miraheze/mw-config] 07paladox 03f65dd1b - Update LocalSettings.php [20:31:33] *addhost eth01 *!*@fosshost/founder/eth01 [20:31:34] *!*@fosshost/founder/eth01 has been added to eth01 [20:31:39] *user eth01 identify [20:31:40] I recognize eth01 as eth01. [20:31:57] eth01: Sorry about the ping spam. You were just my test subject :P [20:32:44] PROBLEM - cloud1 Current Load on cloud1 is CRITICAL: CRITICAL - load average: 25.68, 17.18, 13.99 [20:34:42] RECOVERY - cloud1 Current Load on cloud1 is OK: OK - load average: 11.06, 14.56, 13.42 [20:36:53] PROBLEM - cp8 Current Load on cp8 is CRITICAL: CRITICAL - load average: 2.73, 2.12, 1.39 [20:38:49] PROBLEM - cp8 Current Load on cp8 is WARNING: WARNING - load average: 1.41, 1.80, 1.36 [20:40:55] RECOVERY - cp8 Current Load on cp8 is OK: OK - load average: 0.54, 1.41, 1.27 [20:41:30] Texas np [20:48:27] RhinosF1 [20:48:29] 97.81% 4220.354 1 - SpecialCentralAuth::execute [20:48:29] 87.76% 3786.793 1 - CentralAuthUser::queryAttached [20:48:29] 86.78% 3744.417 697 - CentralAuthUser::localUserData [20:48:29] 80.69% 3481.750 3498 - Wikimedia\Rdbms\Database::executeQuery [20:48:29] 80.18% 3459.786 3498 - Wikimedia\Rdbms\Database::executeQueryAttempt [20:48:30] 75.95% 3277.207 3498 - Wikimedia\Rdbms\DatabaseMysqli::doQuery [20:48:31] 75.47% 3256.660 3498 - mysqli::query [20:48:48] https://github.com/wikimedia/mediawiki-extensions-CentralAuth/blob/master/includes/CentralAuthUser.php#L2362 [20:48:49] and [20:48:49] [ mediawiki-extensions-CentralAuth/CentralAuthUser.php at master · wikimedia/mediawiki-extensions-CentralAuth · GitHub ] - github.com [20:48:54] https://github.com/wikimedia/mediawiki-extensions-CentralAuth/blob/043514ea3b2c894d98ed84b7ef2c305f5833bb31/includes/CentralAuthUser.php#L2401 [20:48:55] [ mediawiki-extensions-CentralAuth/CentralAuthUser.php at 043514ea3b2c894d98ed84b7ef2c305f5833bb31 · wikimedia/mediawiki-extensions-CentralAuth · GitHub ] - github.com [20:54:14] paladox: can you put it on the task [20:54:23] what task? [20:54:41] https://phabricator.wikimedia.org/T255891 [20:54:42] [ ⚓ T255891 Load time increases on Special:CentralAuth for unregistered accounts ] - phabricator.wikimedia.org [20:54:47] paladox: ^ [20:56:13] ty [21:03:42] RhinosF1: file bug about CA [21:04:02] RhinosF1: https://phabricator.miraheze.org/P319 [21:25:43] [02miraheze/MirahezeMagic] 07JohnFLewis pushed 031 commit to 03master [+0/-0/±3] 13https://git.io/JfbcK [21:25:45] [02miraheze/MirahezeMagic] 07JohnFLewis 03fdaca94 - Add Miraheze Survey 2020 [21:27:07] [02miraheze/mediawiki] 07JohnFLewis pushed 031 commit to 03REL1_34 [+0/-0/±1] 13https://git.io/Jfbc1 [21:27:09] [02miraheze/mediawiki] 07JohnFLewis 03a5abece - update MM [21:39:46] PROBLEM - mw4 Puppet on mw4 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki core] [21:39:46] PROBLEM - mw6 Puppet on mw6 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki core] [21:39:51] PROBLEM - mw5 Puppet on mw5 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki core] [21:40:51] PROBLEM - mw7 Puppet on mw7 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki core] [21:40:57] PROBLEM - jobrunner1 Puppet on jobrunner1 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki core] [21:44:49] RECOVERY - mw7 Puppet on mw7 is OK: OK: Puppet is currently enabled, last run 9 seconds ago with 0 failures [21:44:53] RECOVERY - jobrunner1 Puppet on jobrunner1 is OK: OK: Puppet is currently enabled, last run 13 seconds ago with 0 failures [21:45:51] RECOVERY - mw4 Puppet on mw4 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [21:45:55] RECOVERY - mw6 Puppet on mw6 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [21:46:02] RECOVERY - mw5 Puppet on mw5 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [22:19:38] PROBLEM - public.revi.wiki - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'public.revi.wiki' expires in 15 day(s) (Sun 05 Jul 2020 22:17:27 GMT +0000). [22:20:13] revi: ^ [22:20:30] paladox: where are we with the central auth issue? [22:20:43] It's LE so your problem [22:20:48] not mine [22:20:51] you can see the task [22:21:40] paladox: I saw -operations. Reedy asked if it was a specific query it was stuck on, is it? [22:21:55] yes i cannot get that [22:21:58] as it won't log it [22:22:00] it seems [22:22:03] revi: I guess give it half an hour then [22:22:18] paladox: what exact config have we had set? [22:22:29] i set wgDebugLogGroups [22:22:33] to DBQuery [22:23:17] paladox: and a db log file? [22:23:22] yes [22:23:31] it's not logging to it [22:23:38] https://www.mediawiki.org/wiki/Manual:$wgDebugLogFile [22:23:38] [ Manual:$wgDebugLogFile - MediaWiki ] - www.mediawiki.org [22:23:51] If the file isn't generated after doing some requests to the server, check if the webserver has permissions to write on that location, and if PHP can access that file (normally restricted by the open_basedir directive [22:24:01] paladox: can it ^ [22:24:15] i've looked at that page already [22:24:20] i've set it correctly [22:24:30] 'DBQueryError' => "$wmgLogDir/debuglogs/DBQuery.log", [22:24:40] root@test2:/srv/mediawiki/config# cat /var/log/mediawiki/debuglogs/DBQuery.log [22:24:41] empty [22:24:59] And it's definately writeable for www-data and php [22:25:09] Hd1: public logs are in the topic [22:25:20] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JfbWc [22:25:22] [02miraheze/ssl] 07MirahezeSSLBot 03886bafe - Bot: Update SSL cert for public.revi.wiki [22:26:44] first things first... I want to understand what you're trying to do [22:26:52] PROBLEM - zw.fontainebleau-avon.fr - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'zw.fontainebleau-avon.fr' expires in 15 day(s) (Sun 05 Jul 2020 22:23:56 GMT +0000). [22:27:23] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JfbW0 [22:27:24] hd1: it's central auth trying to work out whether an account exists [22:27:25] [02miraheze/ssl] 07MirahezeSSLBot 03f42b9ff - Bot: Update SSL cert for zw.fontainebleau-avon.fr [22:28:38] what's the query to do that? [22:28:59] hd1: I have no logs to know exactly as it won't save them [22:29:16] but https://github.com/wikimedia/mediawiki-extensions-CentralAuth/blob/master/includes/specials/SpecialCentralAuth.php is the code for the page [22:29:17] [ mediawiki-extensions-CentralAuth/SpecialCentralAuth.php at master · wikimedia/mediawiki-extensions-CentralAuth · GitHub ] - github.com [22:30:16] * hd1 has never done PHP [22:30:24] hd1: neither have I [22:30:29] paladox: anything to add [22:30:39] nope [22:30:49] i've done it correctly, even had John look. [22:31:05] Strange [22:31:40] paladox: where's the query you need to optimize? [22:31:57] https://github.com/wikimedia/mediawiki-extensions-CentralAuth/blob/043514ea3b2c894d98ed84b7ef2c305f5833bb31/includes/CentralAuthUser.php#L2479 [22:31:58] [ mediawiki-extensions-CentralAuth/CentralAuthUser.php at 043514ea3b2c894d98ed84b7ef2c305f5833bb31 · wikimedia/mediawiki-extensions-CentralAuth · GitHub ] - github.com [22:32:05] https://phabricator.wikimedia.org/T255891#6241516 [22:32:06] [ ⚓ T255891 Load time increases on Special:CentralAuth for unregistered accounts ] - phabricator.wikimedia.org [22:32:23] RhinosF1: . SET global general_log = 1; SET global log_output = 'table'; [22:32:35] enable that on mysql and run the query again [22:32:43] that will log it and we can look [22:32:52] paladox: ^ [22:32:57] RECOVERY - zw.fontainebleau-avon.fr - LetsEncrypt on sslhost is OK: OK - Certificate 'zw.fontainebleau-avon.fr' will expire on Thu 17 Sep 2020 21:27:17 GMT +0000. [22:33:33] oh [22:33:42] but that would be one hell of a query [22:33:47] RECOVERY - public.revi.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'public.revi.wiki' will expire on Thu 17 Sep 2020 21:25:14 GMT +0000. [22:33:50] by that i mean every query would be logged [22:34:00] not just for the one wiki i'll try it against [22:34:50] so it would be hard to know what i would be looking for [22:34:54] paladox: if you enable it try your query then disable it, it'll log a few queries [22:35:02] oh ok [22:35:04] paladox: I'll look for the query [22:35:04] Just be very quick [22:35:25] hd1: we might have to be careful to redact stuff first [22:38:20] i wonder if this could be https://github.com/wikimedia/mediawiki-extensions-CentralAuth/blob/043514ea3b2c894d98ed84b7ef2c305f5833bb31/includes/CentralAuthUser.php#L2535 [22:38:21] [ mediawiki-extensions-CentralAuth/CentralAuthUser.php at 043514ea3b2c894d98ed84b7ef2c305f5833bb31 · wikimedia/mediawiki-extensions-CentralAuth · GitHub ] - github.com [22:39:22] paladox: that shouldn't run for non existent though. Only existent. [22:40:06] well i mean i guess [22:40:22] paladox: if it is, that's a whole lot of wasted sql. [22:40:44] You could tell if a user exists in 1 query on the local wiki [22:40:48] And then halt [22:41:36] if by https://github.com/wikimedia/mediawiki-extensions-CentralAuth/blob/043514ea3b2c894d98ed84b7ef2c305f5833bb31/includes/CentralAuthUser.php#L2498 you can't get the user for the local wiki. Give up [22:41:37] [ mediawiki-extensions-CentralAuth/CentralAuthUser.php at 043514ea3b2c894d98ed84b7ef2c305f5833bb31 · wikimedia/mediawiki-extensions-CentralAuth · GitHub ] - github.com [22:42:10] actually [22:42:12] RhinosF1 i think that runs for non exiting users? [22:42:20] $row->user_editcount === null [22:42:35] paladox: that sounds insane [22:42:46] And would probably be half the issue [22:43:00] There's no need to touch actor if they're not in user [22:47:55] count(id) !!! [22:48:05] count(*) is inefficient AF [22:51:41] paladox: ^ [22:51:53] You wanna patch that [22:53:00] RhinosF1 actually https://phabricator.wikimedia.org/T255891#6241516 [22:53:01] [ ⚓ T255891 Load time increases on Special:CentralAuth for unregistered accounts ] - phabricator.wikimedia.org [22:53:03] doesn't hit that [22:53:05] if ( !$row ) { [22:53:11] if you use a non exiting user. [22:53:17] # abort the method [22:53:18] } [22:53:24] paladox: good, why the hell would it [22:54:34] https://github.com/wikimedia/mediawiki-extensions-CentralAuth/blob/043514ea3b2c894d98ed84b7ef2c305f5833bb31/includes/CentralAuthUser.php#L2208 [22:54:35] [ mediawiki-extensions-CentralAuth/CentralAuthUser.php at 043514ea3b2c894d98ed84b7ef2c305f5833bb31 · wikimedia/mediawiki-extensions-CentralAuth · GitHub ] - github.com [22:54:40] now that could be why [22:55:12] is there an index on user_name? [22:56:07] i'm not sure [22:56:39] yes [22:56:40] https://github.com/wikimedia/mediawiki/blob/master/maintenance/tables.sql#L138 [22:56:41] [ mediawiki/tables.sql at master · wikimedia/mediawiki · GitHub ] - github.com [22:57:47] can you get onto the database and gist the query plan for me? [23:00:17] hd1 it will be https://github.com/wikimedia/mediawiki-extensions-CentralAuth/blob/043514ea3b2c894d98ed84b7ef2c305f5833bb31/includes/CentralAuthUser.php#L2211 [23:00:18] [ mediawiki-extensions-CentralAuth/CentralAuthUser.php at 043514ea3b2c894d98ed84b7ef2c305f5833bb31 · wikimedia/mediawiki-extensions-CentralAuth · GitHub ] - github.com [23:00:25] each time opening a connection to the db [23:00:29] and then selecting [23:03:05] paladox: you aren't using connection pooling?!? [23:03:41] No [23:03:45] https://github.com/miraheze/mw-config/blob/master/Database.php [23:03:46] [ mw-config/Database.php at master · miraheze/mw-config · GitHub ] - github.com [23:04:39] i guess you meant persistent? [23:05:15] paladox: no, I meant pooling [23:05:32] oh k [23:06:03] paladox: connection pooling is establishing (and maintaining) a pool of connections to the database and giving them out on request [23:06:15] (and restoring the pool size after) [23:06:26] oh [23:07:16] https://stackoverflow.com/questions/24902615/connection-pooling-vs-persist-connection-mysqli#:~:text=Connection%20pooling%20means%20that%20there,to%20pool%20when%20script%20finishes. [23:07:17] [ php - Connection pooling vs persist connection mysqli - Stack Overflow ] - stackoverflow.com [23:08:07] hd1: a mediawiki dev just uploaded https://gerrit.wikimedia.org/r/#/c/mediawiki/extensions/CentralAuth/+/606764/1/includes/CentralAuthUser.php [23:08:08] [ Gerrit Code Review ] - gerrit.wikimedia.org [23:08:28] They're talking in #wikimedia-operations about it [23:09:00] good [23:09:41] hd1: http://wm-bot.wmflabs.org/browser/index.php?start=06%2F19%2F2020&end=06%2F19%2F2020&data=true&display=%23wikimedia-operations [23:09:42] [ Wikimedia IRC logs browser ] - wm-bot.wmflabs.org [23:15:45] * RhinosF1 sleep [23:25:13] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Jfb8R [23:25:14] [02miraheze/services] 07MirahezeSSLBot 0367f0d6b - BOT: Updating services config for wikis