[00:08:28] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 26.66, 20.52, 14.39 [00:12:23] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 23.37, 22.28, 16.53 [00:14:21] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 17.94, 20.11, 16.42 [00:40:10] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JqJ4n [00:40:12] [02miraheze/services] 07MirahezeSSLBot 03d3d4655 - BOT: Updating services config for wikis [03:00:56] !ops 528 jobs outstanding on `metawiki` [03:02:39] Universal_Omega, around? 531 jobs now. Might be worth taking a peek at how many jobs O/S across the farm [03:09:51] 550 jobs outstanding on `metawiki` now. a relatively high number of jobs exist on `loginwiki`, too [04:30:24] dmehus: unavailable for tonight. Sorry. If no one else does I can do it early tomorrow morning, probably. [05:20:37] it should eventually catch up on its own if anything [05:20:38] in theory [05:30:25] I'd hope. But over 500 outstanding jobs is a bit excessive... [05:40:13] Probably mostly wikis [08:18:20] [02dns] 07RhinosF1 closed pull request 03#190: [DO NOT MERGE] miraheze.wiki: fail services over to new infra - 13https://git.io/Jt2Bn [08:21:12] Reception123: on Monday, we're just gonna go with using redirects.yaml, cert should already exist so I assume I can just ping you to add that and once the redirects are there you can remove old DNS? [08:22:14] I'll do PRs on Sunday evening ready [08:22:26] yeah, that should be fine [09:39:05] PROBLEM - ping4 on cp3 is WARNING: PING WARNING - Packet loss = 0%, RTA = 336.49 ms [09:45:10] RECOVERY - ping4 on cp3 is OK: PING OK - Packet loss = 0%, RTA = 248.80 ms [10:45:56] Reception123: you will need to generate a *.bots.miraheze.wiki cert [10:46:34] Or individual for each [10:46:59] I'd just do one for each [10:49:59] You can do one cert with multiple domains I suppose [10:50:05] I wouldn't do a wildcard though [10:50:18] I'll create a PR based on individual though [10:52:53] [02ssl] 07RhinosF1 opened pull request 03#393: [MERGE ON MONDAY] Redirect old domains to new for bots - 13https://git.io/JqJba [10:53:37] [02dns] 07RhinosF1 opened pull request 03#192: miraheze.wiki: [MERGE ON MONDAY] fall back to geoip!cp - 13https://git.io/JqJbK [10:58:16] [02dns] 07RhinosF1 opened pull request 03#193: miraheze.wiki: add GSC TXT records - 13https://git.io/JqJbj [10:58:27] Reception123: merge and deploy the TXT records pls [11:11:03] [02ssl] 07RhinosF1 synchronize pull request 03#393: [MERGE ON MONDAY] Redirect old domains to new for bots - 13https://git.io/JqJba [11:12:22] [02ssl] 07RhinosF1 synchronize pull request 03#393: [MERGE ON MONDAY] Redirect old domains to new for bots - 13https://git.io/JqJba [11:17:15] perhaps you could do the certs then and I can deal with the private keys [11:17:21] [02dns] 07Reception123 closed pull request 03#193: miraheze.wiki: add GSC TXT records - 13https://git.io/JqJbj [11:17:23] [02miraheze/dns] 07Reception123 pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JqJAr [11:17:24] [02miraheze/dns] 07RhinosF1 03a8587c7 - miraheze.wiki: add GSC TXT records (#193) [11:17:52] Reception123: I can yeah but we can't do them while Monday anyway [11:17:58] As we have to be quick [11:18:10] I'll ping you when I'm back from covid testing so maybe like 10:30 [11:18:16] UK time [11:18:33] Can you run puppet on ns1/2 to deploy that PR [11:20:51] puppet should run in 10 mins, is it really urgent? [11:21:26] Reception123: oh ye [11:21:30] It's half past [11:22:17] [02dns] 07RhinosF1 opened pull request 03#194: miraheze.wiki: add a 2nd record for phab - 13https://git.io/JqJAb [11:22:24] Reception123: merge ^ too [11:23:44] [02dns] 07Reception123 closed pull request 03#194: miraheze.wiki: add a 2nd record for phab - 13https://git.io/JqJAb [11:23:45] [02miraheze/dns] 07Reception123 pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JqJAp [11:23:47] [02miraheze/dns] 07RhinosF1 03b832cc0 - miraheze.wiki: add a 2nd record for phab (#194) [11:27:28] Reception123: we are ready for Monday now [11:27:31] Let's set 10:30 [11:27:44] I'll schedule a downtime on hund & zenduty [11:29:00] ok [11:33:38] https://status.mirahezebots.org/issues/604216d410bdfb68587cf6b3 [11:33:39] [ Switchover of miraheze.wiki -> mirahezebots.org - Status Page - MirahezeBots status ] - status.mirahezebots.org [12:32:43] paladox, RhinosF1, Universal_Omega, either of you around to run the `createLocalAccount.php` maintenance script for me? [12:36:50] dmehus: not until ~13:40 [12:41:51] RhinosF1, ack. That's in about an hour or so, right? (If my math's right) [12:44:34] Yep [12:46:33] ok, cool [12:50:14] !sre ```[d8216ff4cdc92acd5c4a538d] 2021-03-05 12:49:41: Fatal exception of type "Wikimedia\Rdbms\DBQueryError"``` [12:50:14] on `metawiki` [12:50:36] Where on meta [12:51:25] while trying to search for `Special:CentralAuth/-SpringFreddy-` in the search bar [12:51:49] I bet it's the - [12:51:52] that's the user whose account didn't locally attach to the wiki, so I'm wondering if there's possibly an upstream bug or a configuration issue [12:52:05] Are you logged into graylog right now? [12:52:33] No [12:52:37] okay [12:52:43] I can login in like 50 minutes [12:52:53] yeah, that's fine [12:53:09] I can't see a fatal on CA though just browsing it [12:53:25] What exactly did you do? Link to the page you were on? [12:53:28] yeah, I can bring up his CA and search directly [12:53:34] https://meta.miraheze.org/wiki/Special:CentralAuth?target=-SpringFreddy- loads fine [12:53:35] [ Global account information for -SpringFreddy- - Miraheze Meta ] - meta.miraheze.org [12:53:45] Any page; not page specific. [12:53:46] So what page did that fatal happen on [12:54:00] Just enter the given query ^ into the search bar and hit enter [12:54:38] You cant search with a - in [12:54:38] `[feedf9ee34d4657b99f81fb0] 2021-03-05 12:54:10: Fatal exception of type "Wikimedia\Rdbms\DBQueryError"` [12:54:38] Just did it again from `Miraheze`; will reproduce a third time on a different page [12:54:46] That's been said a thousand times [12:54:54] yeah you can [12:55:27] No you can't [12:55:40] The top left box redirects to CA for me [12:55:49] Special:Search directly doesn't [12:55:52] So there is a bug [12:56:04] I just searched for `–revi` in the search bar. Let me try `-revi` [12:56:09] But any search with a - in will fail [12:56:36] Searching for `-revi` works [12:56:43] so I don't think that's it [12:56:48] It is [12:56:52] We should run a full trace to be sure [12:57:18] but how so? and besides, how does "that's been said a thousand times" help me when when this is the first time I'm hearing of it? [12:58:12] It's been a known bug since before Miraheze existed [12:58:20] Searches with - in don't work properly [12:58:44] okay, but still saying it's been said a thousand times doesn't really help me [12:58:56] I can moan that CentralAuth should be consistent in redirecting to CA when using Search [12:59:06] As I can only reproduce on Special:Search [12:59:21] The top right box sends me straight to the page [12:59:26] but that's not quite right. to be more accurate, searches that begin AND end in `-` will fail. Searches that only BEGIN in `-` will be successful [13:00:31] I haven't tried Special:Search; only top right search box. `-revi` was fine; `-revi-` failed [13:00:41] ```[ff581d2a82584552e67c5ff1] 2021-03-05 12:58:15: Fatal exception of type "Wikimedia\Rdbms\DBQueryError"``` [13:01:28] The fact is that error has been known since pre Miraheze [13:01:51] But typing a valid username into the top right box sends me to CentralAuth [13:02:03] It's only Special:Search directly that's weird [13:02:12] Yeah, anyway, I can reproduce in both the top right box and Special:Search. So it's not just `-`. It's when the query begins and ends with `-` [13:02:38] Right well it's not an issue I'm gonna look into further [13:02:56] > It's only Special:Search directly that's weird [13:02:56] Not true. You're also on mobile and a Mac environment, so there could be a different result here. [13:03:41] I'm not investigating the bug further [13:03:47] > Right well it's not an issue I'm gonna look into further [13:03:47] That's fine, I still wouldn't mind having a trace done by another MWE later, as time permits, though. Maybe they'll be upstream a patch. :) [13:03:50] It's a known issue and a long known one [13:03:57] It's not going to get a fix soon [13:04:07] Universal_Omega has tried before [13:04:09] It didn't work [13:04:46] > It's not going to get a fix soon [13:04:46] That's kind of defeatest. [13:04:46] > Universal_Omega has tried before [13:04:46] Hrm, I must've missed that, as this is the first I'm hearing of it [13:05:27] https://phabricator.wikimedia.org/T221560 [13:05:28] [ ⚓ T221560 Searches with hyphens yield a database query error ] - phabricator.wikimedia.org [13:06:12] RhinosF1, right, but just to be clear, it's not only one hyphen. It's when it begins and ends in a hyphen [13:06:18] but thanks :) [13:06:23] It's not as old as i thought [13:06:30] But it's still well known [13:06:44] And it's not deaftist it's MediaWiki development priorities [13:07:33] It is a problem from pre Miraheze [13:07:57] It's been a problem since day 1 of MediaWiki potentially and arguably [13:08:43] JohnLewis, interesting, yeah. I'm kind of surprised this is the first time I've encountered it, though [13:09:02] It's the reason we don't allow - in database names [13:10:38] JohnLewis, ah. Didn't realize that actually. I mean, I figured it was related to a problem with querying the database names, but didn't realize it was the same issue. Makes sense. That's probably what prompted Universal_Omega to take a look at trying to fix it then in January, to try and see if it could be fixed to allow `-` in database names [13:11:38] Tbf, it's a problem that won't get fixed - workarounds exist and the effort needed to resolve it heavily outweigh any possible benefits to external parties [13:12:35] oh :( [13:12:37] Such as for us, it would probably day hours or days for us to resolve it, then potentially weeks to get it deployed and then months for it to go live - for no actual benefit to us besides potentially allowing - in a database name, when alternatively we can spend 5 minutes and achieve the same result [13:13:12] I think we've said before if it mattered we could just issue a custom domain [13:13:21] Yep [13:13:41] true, and the fact I haven't noticed it until now shows me it's rare to run a search bar search for `Special:CentralAuth/-someUser-` heh [13:20:48] JohnLewis, apparently there's a Special:CreateLocalAccount special page that comes with CentralAuth, but we don't seem to have that enabled on Miraheze. Any idea why? [13:21:20] oh nevermind [13:21:25] it's MW 1.36 :( [13:21:31] https://www.mediawiki.org/wiki/Help:Extension:CentralAuth/CreateLocalAccount [13:21:32] [ Help:Extension:CentralAuth/CreateLocalAccount - MediaWiki ] - www.mediawiki.org [13:23:09] dmehus: link to the request to run the script [13:23:19] Should be able to look soon [13:25:27] RhinosF1, I didn't create a task, can I just paste the pertinent details here? [13:25:59] dmehus: I just need to know where the request came from, which user and which wiki [13:28:14] RhinosF1, ack. Request came from me. User is `-SpringFreddy-` and wiki is `sfreddys1993pizzafunworldwiki` [13:28:57] Looking [13:29:39] thanks :) [13:30:20] It likely didn't get attached by the wiki creation script because of the opening and closing `-` in the username [13:32:40] Here's the command from the last time Reception123 had to run it a few days ago... [13:32:40] ```sudo -u www-data php createLocalAccount.php --wiki robotiawiki Robotia``` [13:32:40] Modified for you: [13:32:40] ```sudo -u www-data php createLocalAccount.php --wiki sfreddys1993pizzafunworldwiki -SpringFreddy-``` [13:33:07] The script ain't liking it [13:33:11] ah [13:33:25] Might have to globally rename the user then? [13:34:10] can you try preceding the hypens with a `\` character or whatever regex that script uses? [13:34:32] I've put it in ""'s [13:34:46] And tried every location [13:34:51] oh [13:34:54] But apparently im giving a parameter twice [13:35:04] hrm [13:35:04] I might have to use eval.php [13:35:12] oh [13:35:21] to manually attach it to the wiki using `eval.php`? [13:35:49] Yep [13:36:18] Could also rename the user to SpringFreddy, as it's likely the user is going to have nothing but technical problems with that username [13:36:22] with the hyphens [13:38:26] That did it [13:38:39] oh [13:38:40] Nah no reason to force a rename [13:38:41] that's good [13:38:42] thanks [13:39:07] !log used eval.php to fix a failed wiki creation auto create user as the script wanted to be awkward [13:39:10] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [13:39:13] dmehus: can you assign rights [13:39:42] yep [13:39:44] doing now [13:39:57] that's why I needed the script run [13:40:26] Yeah! [13:41:13] Done. And thank goodness Special:UserRights still worked [13:41:14] heh [13:41:53] :) [13:46:04] thanks :) [14:32:02] paladox: will my dns PR work or do I have to manually add each domain and then DYNA geoip!cp [14:32:08] Because we have a wildcards entry [14:52:15] what dns pr? [14:52:26] the only one i see is https://github.com/miraheze/dns/pull/192 [14:52:27] [ miraheze.wiki: [MERGE ON MONDAY] fall back to geoip!cp by RhinosF1 · Pull Request #192 · miraheze/dns · GitHub ] - github.com [14:53:05] paladox: yes, will that work [14:53:33] Based on my ssl pr to add redirects. Well issue certs before merging ssl pr from jobrunner3 [14:53:43] That's what me + Reception123 are doing Monday morning [14:53:52] yes. [14:54:04] though i think that would just wildcard to our servers [14:54:57] host meta.mirahezr.wiki [14:54:57] Host meta.mirahezr.wiki not found: 3(NXDOMAIN) [14:54:58] hmm [14:55:09] paladox: shouldn't it follow certs.yaml and redirects.yaml [14:55:37] only if we're hosting it. [14:56:02] You are from 10:30 on Monday morning going to in charge of the redirects [15:18:02] !log reception@jobrunner4:~$ sudo -u www-data php /srv/mediawiki/w/maintenance/importDump.php --wiki mylittlefandompolskawiki /home/reception/MLF.xml [15:18:07] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [15:26:21] !log reception@jobrunner3:~$ sudo -u www-data php /srv/mediawiki/w/maintenance/importDump.php --wiki chadsofawiki /home/reception/officialdragonballsuper_pages_full.xml [15:26:24] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [15:27:11] paladox: any objections to https://phabricator.miraheze.org/T6916 ? [15:27:12] [ ⚓ T6916 embed external images ] - phabricator.miraheze.org [15:27:31] paladox: did you ever get rid of rdb3? [15:31:08] !log reception@jobrunner3:~$ sudo -u www-data php /srv/mediawiki/w/maintenance/importDump.php --wiki santrovelowiki /home/reception/delbackups4/santrovelowiki.xml --username-prefix="santrovelo" [15:31:11] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [15:35:06] [02ManageWiki] 07JohnFLewis closed pull request 03#252: Create new ManageWikiTypes helper class - 13https://git.io/Jtxof [15:35:07] [02miraheze/ManageWiki] 07JohnFLewis pushed 031 commit to 03master [+1/-0/±2] 13https://git.io/JqU3j [15:35:09] [02miraheze/ManageWiki] 07Universal-Omega 03eaa2219 - Create new ManageWikiTypes helper class (#252) [15:36:09] miraheze/ManageWiki - JohnFLewis the build passed. [15:39:47] JohnLewis: I need to do that. I left it just in case. Thanks for reminding! I’ll do that later today. [15:40:03] paladox: and also docs please for it :D [15:40:27] * JohnLewis is on a backlog clearing exercise seeing as he's not a lot of time to give the past week [15:45:54] [02miraheze/ManageWiki] 07JohnFLewis pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JqUsV [15:45:55] [02miraheze/ManageWiki] 07JohnFLewis 03d848eb9 - T6899: Log only the master namespace in the log [15:46:54] miraheze/ManageWiki - JohnFLewis the build passed. [15:49:47] JohnLewis: for that, wouldn't that not log at all if talk page was last selected? I could be totally wrong but just looked like that to me. [15:50:20] What is last selected irrelevant? [15:50:32] *is irrelevant [16:09:46] JohnLewis: last selected is irrelevant. But what that'll do is cause logging to never happen if I make a change to main namespace, then click on talk page tab, then submit the form. What that should do it log if main namespace is changed, log main. If talk it changed, log talk namespace, if both are changed, log both. Again, I could be wrong about what that'll do, but I do think that's what it'll do. [16:21:16] [02miraheze/WikiDiscover] 07JohnFLewis pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JqUnM [16:21:17] [02miraheze/WikiDiscover] 07JohnFLewis 035873954 - Simplify API code and respect limit/siteprop [16:22:15] miraheze/WikiDiscover - JohnFLewis the build passed. [16:35:12] It’ll log the main namespace only [16:35:30] theres no pointing in generating two log actions for one change [16:36:38] I’m unsure where the ‘last tab to be clicked on’ even comes from - that’s never been the case and I don’t know how the code could even know that information [16:38:15] JohnLewis: whatever tab was last clicked on was logged. That's what the task was saying the issue was. It's been like that for awhile but I never got around to looking at it or asking you. And also I'm not saying to log twice, but 1 log action with changes from both. [16:38:38] No, talk was always logged [16:38:44] Not ‘the last tab’ [16:39:42] And the one log does enable both to be logged because you can’t modify the talk namespace directly, you go to modify the main and it’s associated one [16:45:12] JohnLewis: https://test3.miraheze.org/wiki/Special:RecentChanges first time in that ManageWiki change, I first clicked on talk tab before saving, second time I never did, both times the exact same config modified (first on, second time off) first it logged as talk, second it logged as main. [16:45:14] [ Recent changes - Test3 ] - test3.miraheze.org [16:48:12] And my changes fixes it? [16:48:25] because now it’s only the main namespace that is logged... [16:50:04] JohnLewis: I don't think so. What I'm saying is that if it is adding a check to not log if the ID is a talk page, then it won't log at all if it tries to (if the talk tab was last selected), again could be wrong but based on what I saw that's what it looks like. [16:50:09] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JqUCQ [16:50:11] [02miraheze/services] 07MirahezeSSLBot 030db4971 - BOT: Updating services config for wikis [16:50:22] You are wrong [16:50:53] Which is what I've been trying to state, it seems which is added to the log is random and based on whichever is listed last in changes [16:51:05] So now only the main namespace is logged [16:59:20] Regarding ManageWiki/namespaces, I think one has to set the settings in the Main tab and the settings in the Talk tab before they can "submit" their changes. So really, only one change needs to be logged in the ManageWiki log, I think? [17:01:01] [02miraheze/WikiDiscover] 07JohnFLewis pushed 031 commit to 03master [+0/-0/±2] 13https://git.io/JqUWE [17:01:02] [02miraheze/WikiDiscover] 07JohnFLewis 03dafdb15 - Add support for deleted/locked wikis to API [17:02:01] miraheze/WikiDiscover - JohnFLewis the build passed. [17:33:32] [02miraheze/dns] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JqU8Q [17:33:33] [02miraheze/dns] 07paladox 038ad1749 - rename rdb3 to mem1 [17:37:15] !log reinstall rdb3 as mem1 [17:37:17] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:40:07] PROBLEM - rdb3 NTP time on rdb3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [17:40:28] PROBLEM - ping4 on rdb3 is CRITICAL: PING CRITICAL - Packet loss = 100% [17:40:34] PROBLEM - rdb3 Disk Space on rdb3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [17:40:36] PROBLEM - rdb3 APT on rdb3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [17:40:54] PROBLEM - rdb3 Puppet on rdb3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [17:41:07] PROBLEM - rdb3 PowerDNS Recursor on rdb3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [17:41:18] Downtimed ^ [17:47:22] PROBLEM - bacula2 APT on bacula2 is WARNING: APT WARNING: 0 packages available for upgrade (0 critical updates). warnings detected, errors detected. [17:49:27] PROBLEM - bacula2 APT on bacula2 is CRITICAL: APT CRITICAL: 4 packages available for upgrade (3 critical updates). [17:56:02] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-6 [+0/-0/±1] 13https://git.io/JqUBZ [17:56:03] [02miraheze/puppet] 07paladox 038da59f3 - Remove rdb3 [17:56:05] [02puppet] 07paladox created branch 03paladox-patch-6 - 13https://git.io/vbiAS [17:56:06] [02puppet] 07paladox opened pull request 03#1675: Remove rdb3 - 13https://git.io/JqUBn [17:56:24] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-6 [+0/-1/±0] 13https://git.io/JqUBC [17:56:25] [02miraheze/puppet] 07paladox 031d007b9 - Delete rdb3.yaml [17:56:27] [02puppet] 07paladox synchronize pull request 03#1675: Remove rdb3 - 13https://git.io/JqUBn [17:56:37] [02puppet] 07paladox closed pull request 03#1675: Remove rdb3 - 13https://git.io/JqUBn [17:56:38] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-1/±1] 13https://git.io/JqUBW [17:56:40] [02miraheze/puppet] 07paladox 032d5fa90 - Remove rdb3 (#1675) [17:56:41] [02puppet] 07paladox deleted branch 03paladox-patch-6 - 13https://git.io/vbiAS [17:56:43] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-6 [17:57:08] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-6 [+0/-0/±1] 13https://git.io/JqUBB [17:57:10] [02miraheze/puppet] 07paladox 0365630ba - Add mem1 [17:57:11] [02puppet] 07paladox created branch 03paladox-patch-6 - 13https://git.io/vbiAS [17:57:13] [02puppet] 07paladox opened pull request 03#1676: Add mem1 - 13https://git.io/JqUBR [17:57:40] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-6 [+1/-0/±0] 13https://git.io/JqUB2 [17:57:41] [02miraheze/puppet] 07paladox 032f59cd0 - Create mem1.yaml [17:57:43] [02puppet] 07paladox synchronize pull request 03#1676: Add mem1 - 13https://git.io/JqUBR [17:58:14] [02puppet] 07paladox closed pull request 03#1676: Add mem1 - 13https://git.io/JqUBR [17:58:15] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+1/-0/±1] 13https://git.io/JqUBr [17:58:17] [02miraheze/puppet] 07paladox 030ba587b - Add mem1 (#1676) [17:58:18] [02puppet] 07paladox deleted branch 03paladox-patch-6 - 13https://git.io/vbiAS [17:58:20] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-6 [18:00:06] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JqUBd [18:00:07] [02miraheze/puppet] 07paladox 03f33a359 - memcached: Fix ufw rule name [18:02:20] PROBLEM - mem1 memcached on mem1 is CRITICAL: connect to address 51.195.236.223 and port 11211: Connection refused [18:03:40] RECOVERY - mem1 memcached on mem1 is OK: TCP OK - 0.020 second response time on 51.195.236.223 port 11211 [19:05:09] [02CreateWiki] 07JohnFLewis closed pull request 03#195: Allow $wgCreateWikiBlacklistedSubdomains to be an array - 13https://git.io/JtpFP [19:05:11] [02miraheze/CreateWiki] 07JohnFLewis pushed 031 commit to 03master [+0/-0/±2] 13https://git.io/JqUzo [19:05:12] [02miraheze/CreateWiki] 07Universal-Omega 03b4fe021 - Allow $wgCreateWikiBlacklistedSubdomains to be an array (#195) [19:06:10] miraheze/CreateWiki - JohnFLewis the build passed. [19:18:29] SPF|Cloud hi, around? [19:20:39] [02puppet] 07paladox closed pull request 03#1671: varnish: Fix ExecReload script - 13https://git.io/JtxlV [19:20:41] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JqUgH [19:20:42] [02miraheze/puppet] 07paladox 03e6421dc - varnish: Fix ExecReload script (#1671) [19:20:44] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-1 [19:20:45] [02puppet] 07paladox deleted branch 03paladox-patch-1 - 13https://git.io/vbiAS [19:22:27] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-6 [+0/-0/±1] 13https://git.io/JqUgb [19:22:29] [02miraheze/mw-config] 07Universal-Omega 03c1cea60 - Convert wgCreateWikiBlacklistedSubdomains to array [19:22:30] [02mw-config] 07Universal-Omega created branch 03Universal-Omega-patch-6 - 13https://git.io/vbvb3 [19:22:41] [02mw-config] 07Universal-Omega opened pull request 03#3753: Convert wgCreateWikiBlacklistedSubdomains to array - 13https://git.io/JqUgp [19:23:04] [02miraheze/CreateWiki] 07Universal-Omega deleted branch 03Universal-Omega-patch-2 [19:23:06] [02CreateWiki] 07Universal-Omega deleted branch 03Universal-Omega-patch-2 - 13https://git.io/vpJTL [19:23:28] [02miraheze/CreateWiki] 07Universal-Omega deleted branch 03Universal-Omega-patch-1 [19:23:29] [02CreateWiki] 07Universal-Omega deleted branch 03Universal-Omega-patch-1 - 13https://git.io/vpJTL [19:23:45] miraheze/mw-config - Universal-Omega the build passed. [19:24:29] PROBLEM - cp11 Puppet on cp11 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[varnish] [19:36:28] RECOVERY - cp11 Puppet on cp11 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [20:02:34] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 11.82, 6.99, 4.82 [20:03:28] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 12.06, 7.33, 5.27 [20:06:26] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 5.46, 7.14, 5.47 [20:06:59] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/JqUVg [20:07:01] [02miraheze/puppet] 07paladox 0341760dd - varnish: Convert to systemd::service and also fix init script [20:07:02] [02puppet] 07paladox created branch 03paladox-patch-1 - 13https://git.io/vbiAS [20:07:04] [02puppet] 07paladox opened pull request 03#1677: varnish: Convert to systemd::service and also fix init script - 13https://git.io/JqUV2 [20:07:25] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 4.17, 6.79, 5.64 [20:08:23] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 4.07, 6.06, 5.28 [20:33:39] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/JqUrK [20:33:40] [02miraheze/puppet] 07paladox 03f022715 - Update init.pp [20:33:42] [02puppet] 07paladox synchronize pull request 03#1677: varnish: Convert to systemd::service and also fix init script - 13https://git.io/JqUV2 [20:34:27] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-1 [+1/-1/±0] 13https://git.io/JqUrP [20:34:29] [02miraheze/puppet] 07paladox 03c81f501 - Rename modules/varnish/files/varnish/varnish.service to modules/varnish/templates/initscripts/varnish.systemd.erb [20:34:30] [02puppet] 07paladox synchronize pull request 03#1677: varnish: Convert to systemd::service and also fix init script - 13https://git.io/JqUV2 [20:54:50] [02miraheze/ManageWiki] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://git.io/JqUK3 [20:54:52] [02miraheze/ManageWiki] 07Universal-Omega 0383c997d - Add what configs names are for ManageWikiNamespaces in labels [20:54:53] [02ManageWiki] 07Universal-Omega created branch 03Universal-Omega-patch-1 - 13https://git.io/vpSns [20:54:55] [02ManageWiki] 07Universal-Omega opened pull request 03#254: Add what configs names are for ManageWikiNamespaces in labels - 13https://git.io/JqUKZ [20:55:56] miraheze/ManageWiki - Universal-Omega the build passed. [21:25:58] [02miraheze/ManageWiki] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://git.io/JqUiI [21:25:59] [02miraheze/ManageWiki] 07Universal-Omega 03907fc6b - Update ManageWikiFormFactoryBuilder.php [21:26:01] [02ManageWiki] 07Universal-Omega synchronize pull request 03#254: Add what configs names are for ManageWikiNamespaces in labels - 13https://git.io/JqUKZ [21:27:00] miraheze/ManageWiki - Universal-Omega the build passed. [21:28:26] [02miraheze/ManageWiki] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://git.io/JqUim [21:28:28] [02miraheze/ManageWiki] 07Universal-Omega 034672e68 - Update ManageWikiFormFactoryBuilder.php [21:28:29] [02ManageWiki] 07Universal-Omega synchronize pull request 03#254: Add what configs names are for ManageWikiNamespaces in labels - 13https://git.io/JqUKZ [21:29:30] miraheze/ManageWiki - Universal-Omega the build passed. [21:33:02] [02ManageWiki] 07Universal-Omega edited pull request 03#254: Add config variables to ManageWikiNamespaces labels - 13https://git.io/JqUKZ [21:42:10] PROBLEM - ping6 on dbbackup1 is WARNING: PING WARNING - Packet loss = 0%, RTA = 128.77 ms [21:44:13] RECOVERY - ping6 on dbbackup1 is OK: PING OK - Packet loss = 0%, RTA = 116.56 ms [21:47:19] paladox: https://github.com/miraheze/puppet/blob/9b1019c7ce178079ac4ddb597460a992156072e6/modules/phabricator/files/phab.miraheze.wiki.conf#L8 [21:47:20] [ puppet/phab.miraheze.wiki.conf at 9b1019c7ce178079ac4ddb597460a992156072e6 · miraheze/puppet · GitHub ] - github.com [21:47:23] Need fixing [21:47:41] https://github.com/miraheze/puppet/blob/master/modules/phabricator/files/phab.miraheze.wiki.conf#L8 [21:47:42] [ puppet/phab.miraheze.wiki.conf at master · miraheze/puppet · GitHub ] - github.com [21:47:42] no? [21:48:44] Why that pick an old commit [21:49:33] paladox: also reviewing the initially issuing ssl certificates for Monday. I'm think that issuing a wildcard on *.bots.miraheze.wiki might actually be best [21:49:44] Because then we can use DNS based authentication [21:49:56] Verification for Le [21:50:32] And I'm not sure that acme-challenge will work without setting up a temp nginx site [21:50:42] And we need to add dyna records for each domain [21:50:51] As there's no wildcard dns for on .wiki [21:51:35] I know that needs manual renewal though [21:51:40] Which isn't fun [21:57:42] Reception123: FYI ^ [22:01:03] But that might mean I can pre issue the certificate [22:05:11] I'm gonna create the public key now ready for *.bots. [22:05:22] if you use wildcard, it doesn't need nginx. [22:05:31] paladox: that's what I'm saying [22:05:34] well if it uses a dns record. [22:05:55] It'll only be affected by redirects.yaml [22:06:03] Which uses its own system [22:06:12] I'm not sure the acme overrides are there [22:06:54] We probably should double check redirects.yaml will work [22:07:38] The only one we have set is for miraheze.wiki itself [22:07:56] Are we sure that the new sub domains and nested subdomains will work [22:10:33] [02dns] 07RhinosF1 synchronize pull request 03#192: miraheze.wiki: [MERGE ON MONDAY] fall back to geoip!cp - 13https://git.io/JqJbK [22:13:30] [02mw-config] 07Kelvs599 opened pull request 03#3754: Add wgVectorDefaultSkinVersion to ManageWikiSettings - 13https://git.io/JqUXw [22:14:34] miraheze/mw-config - Kelvs599 the build passed. [22:15:06] [02dns] 07RhinosF1 opened pull request 03#195: add ACME for *.bots.miraheze.wiki - 13https://git.io/JqUXP [22:15:14] paladox: can you merge ^ [22:15:19] That's from jobrunner3 [22:15:43] [02dns] 07paladox closed pull request 03#195: add ACME for *.bots.miraheze.wiki - 13https://git.io/JqUXP [22:15:44] [02miraheze/dns] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JqUXD [22:15:46] [02miraheze/dns] 07RhinosF1 03e4bb69c - add ACME for *.bots.miraheze.wiki (#195) [22:15:59] Let me know when it's live [22:16:02] paladox: ^ [22:16:29] it is live [22:17:39] paladox: certificate is on jobrunner3 but cat won't let me access it [22:17:47] Can you paste the public key? [22:17:59] So I can add to my PR [22:18:41] [02ssl] 07RhinosF1 synchronize pull request 03#393: [MERGE ON MONDAY] Redirect old domains to new for bots - 13https://git.io/JqJba [22:19:08] https://phabricator.miraheze.org/P393 [22:19:09] [ ✎ P393 (An Untitled Masterwork) ] - phabricator.miraheze.org [22:20:40] [02ssl] 07RhinosF1 synchronize pull request 03#393: [MERGE ON MONDAY] Redirect old domains to new for bots - 13https://git.io/JqJba [22:21:48] [02ssl] 07RhinosF1 synchronize pull request 03#393: [MERGE ON MONDAY] Redirect old domains to new for bots - 13https://git.io/JqJba [22:22:03] [02ssl] 07RhinosF1 edited pull request 03#393: Redirect old domains to new for bots - 13https://git.io/JqJba [22:22:51] paladox: if you add the private key then the ssl pr can go now and then we know in advance what nginx config it'll make [22:22:55] Before we switch the dns [22:24:29] done [22:24:37] [02ssl] 07paladox closed pull request 03#393: Redirect old domains to new for bots - 13https://git.io/JqJba [22:24:38] [02miraheze/ssl] 07paladox pushed 031 commit to 03master [+1/-0/±2] 13https://git.io/JqU1I [22:24:40] [02miraheze/ssl] 07RhinosF1 03e49f32a - Redirect old domains to new for bots (#393) [22:25:13] why are some of them using sslname: 'miraheze.wiki' [22:25:28] RhinosF1 [22:25:45] paladox: that's the 3 that are server.miraheze.wiki [22:25:53] oh ok [22:26:44]