[18:10:00] @kockaadmiralac https://github.com/miraheze/mw-config/commit/769ae613b407c19ba1fd5f2ef97d04a17de728fb will need to be reverted, it seems to be getting /{dbName}/w/rest.php at some points [18:10:10] Which makes me feel something is tripping over SUL3 stuff [19:04:44] [1/2] Yeah Wikimedia has the exact same problem [19:04:44] [2/2] https://cdn.discordapp.com/attachments/1006789349498699827/1500211458942963782/Screenshot_20260502_210419_Firefox.png?ex=69f79c4c&is=69f64acc&hm=25d80c47aa2f5749dcd95fca2a77a6d71cf15a84926e33ce0240922a3dac7906& [19:05:06] Except their auth wiki is never used for anything so they never noticed [19:06:07] except it was happening on metawiki as well and not just the pseudo authwiki via metawiki [19:06:31] realistically if it happens on the authwiki is it that big of a deal, if it happens outside of the authwiki then it becomes a problem [19:06:34] Yes but that's because you set $wgScriptPath across the entire metawiki [19:06:43] Not just in pseudo mode [19:06:57] the number of people affected by search being broken specifically on the authwiki is fairly minimal [19:08:55] this line in particular sets metawiki scriptpath to /metawiki/w [19:09:14] can't you just check for wmgSharedDomainPath [19:09:16] or whatever the variable is [19:09:39] and then if so set scriptpath back to /w [19:11:03] $wmgSharedDomainPathPrefix is one of the variables being set inside that block so it's not something we can check [19:12:05] And it's set unconditionally whenever the host is Meta [19:13:42] check if `$_SERVER['HTTP_HOST'] === $wi->getSharedDomain()` then? [19:13:52] and if it's not then set back to /w [19:13:55] That just checks if you're on Meta [19:14:05] auth* [19:14:10] https://github.com/miraheze/mw-config/blob/ac126b0d2d5b92a464b0a9d2675dd6604de104fb/initialise/MirahezeFunctions.php#L54 [19:14:23] Oh right [19:14:30] script path shouldn't be /$wgDBname unless it's on auth, so just forcefully set it to /w if it's on auth. [19:15:10] Okay I thought you were using Meta as auth hmm there's something fishy there [19:15:32] using meta as auth would break things [19:15:43] I don't think there's any support for having a wiki running on the shared domain [19:15:53] plus subdomains are cheap anyway [19:18:30] okay Wikimedia [19:18:34] I see how it is [19:19:50] I'll be so honest the wmf config is something I never want to look at [19:20:04] it's so confusing [19:22:52] free [19:22:59] that's what I meant lol [19:23:12] they are basically nothing to maintain [19:23:25] compared to shoving it onto meta [19:23:39] what purpose does loginwiki even serve in the sul3 days beyond checkusers [19:23:54] GUP [19:24:05] Does wgConf get cached anywhere btw [19:24:14] I think that might be the root cause here [19:25:22] WO doesn't use loginwiki for GUP lol so for us it serves absolutely no purpose [19:25:38] you guys are really smoking it there [19:25:41] I don't see the point in forcing people to have to go to a random wiki where its only real purpose is for GUP to edit their user page [19:26:00] I did stuff the quick way and don't want to fix the problems it causes for me later on [19:26:19] it works and until I can't defer fixing it any longer I won't bother fixing stuff [19:26:28] burnout is real [19:27:46] If it does, then that means it's likely that authwiki's change to $wgConf polluted the main wiki's $wgConf [19:28:14] I don't think that wgConf would be cached outside of opcache [19:30:07] Okay anyways I think the solution is clear I'm just unsure why it happened in the first place [20:08:15] That's probably one of the few sensible things you guys do [20:08:47] There is a config cache ye, I'm not sure how much it caches [20:10:05] ;) what can I say [20:10:55] things grew faster than the budget so networking is a fucked up tailscale mess and then most stuff runs on hard drives and changing it is becoming more and more difficult [20:12:05] hopefully some stuff will be cooked up in the future that brings things into a slightly more normal state of operations where I don't have to hop on to triage some obscure poorly documented issue every 12-18 hours [20:13:01] plus I made some very poor decisions that turned out to be very bad in the long run [20:13:22] one of them was the choice not to use swift for some reason [20:13:28] People often make them [20:13:32] Your images look broken [20:13:37] Btw [20:14:00] yeah turns out if any of the garage hosts die then the mediawiki servers die trying to pull images so it takes down the site [20:14:25] fun [20:14:39] uhh probably something to do with the fact that it relies on some really weird nginx rewrites [20:14:48] Hmm [20:15:04] genuinely the worst part of it is the cp* servers [20:15:22] puppet applies haproxy and varnish rules for prod to both of them and then the garage rewrite stuff to one of them I think [20:15:32] your architecture sounds a mess [20:15:40] realistically the garage logic should be moved onto nginx on the file servers and then it should be proxied through haproxy [20:15:52] although tbh I'm not sure our use of both cf and cps is that sane [20:16:15] cloudflare cache images and load.php and then varnish does /wiki/ [20:16:18] We have cf -> haproxy -> varnish [20:16:30] A little bit more complex [20:16:36] it relies on some dodgy extension that I don't entirely think works but I didn't write and last I checked it sort of works [20:16:42] we have varnish in front of haproxy [20:17:03] again not sure why but it sounded like a good idea at the time [20:17:13] Haproxy is only there for us as Cloudflare don't support host based routing [20:17:55] cloudflare load balancing is too expensive for our use case [20:18:01] I haven't looked into it too deeply but from what I can tell it is [20:18:15] We have CF Enterprise [20:18:19] From Gallieo [20:18:34] I think we did get approved for galileo but that was a justa thing so [20:18:43] I have no idea if it actually was and I don't care enough to look into it further [20:18:44] I'm not very impressed with CF [20:18:58] there are bigger issues and the only thing we use cf for really is images and ssl [20:19:05] the use of cf was a waki thing but it stuck [20:19:27] the rest was me doing stuff quickly because it wasn't working particularly great to begin with [20:19:46] the mediawiki side of things is usually fine it's just the architecture that backs it is so shit that it will break for reasons I have no idea why [20:23:12] Tbh Wikimedia's architecture is shit at times too [20:23:15] And so is some of ours [20:23:28] And don't even start me on things like banks and governments [20:23:35] the nhs [20:23:40] NASA made it all the way to the moon and outlook still didn't work [20:23:59] true true [20:24:05] their use of k8s is questionable [20:24:10] I'm not sure what runs on k8s and what doesn't [20:24:27] and I'm also surprised that at their scale they don't use some form of cloud [20:24:36] Most web frontend things [20:24:47] Clouds are expensive [20:24:49] at minimum s3 would probably do them some good because it's so much cheaper [20:24:58] It's probably cheap to just keep paying their DC staff [20:25:05] true [20:25:21] I don't actually know how many people they have as DC staff [20:25:24] I wouldn't assume it's many [20:25:35] probably mostly remote hands [20:25:43] Half a dozen I think [20:26:04] It's only codfw & eqiad I think with actual staff [20:26:16] makes sense [20:26:17] Rest are remote hands I believ [20:26:24] I know they fly them out to do buildouts [20:26:52] but edge caching is mostly redundant anyway so having actual DC staff with presence isn't a must [20:26:54] Rob, Jenn, J Clark, V Riley, Papaul [20:26:56] I think [20:26:58] So 5 [20:27:10] Across 2 DCs [20:27:14] does mira still have edge caching outside of SLC [20:27:44] They tried to get remote hands to do one of their build outs and it did not go well [20:27:49] Just CF [20:27:58] makes sense [20:28:20] does CF cache articles or just static stuff? [20:28:44] Mostly static stuff with some exceptions [20:28:49] help how do i locally test miraheze config changes [20:29:02] realistically there's not an easy way [20:29:02] the mw-config repo seems very hard to use with a local installation [20:29:05] @tbodt you don't [20:29:09] okay [20:29:17] I would just use WO to test any PRs if I was going to make them [20:29:20] WO? [20:29:26] "my" farm [20:29:27] His own wiki farm [20:29:32] ah [20:29:39] We can deploy it to beta if it needs testing [20:29:40] I use my loosely but as far as I'm concerned it is [20:29:58] that reminds me I need to fix betaoasis [20:30:08] shoddy redis stuff [20:30:19] Never fix beta [20:30:25] Beta is not something that works [20:30:27] well yeah but login doesn't work 💀 [20:30:29] I love beta [20:30:32] I'm sure beyond that it works but [20:30:42] We've never actually finished the deployment of beta [20:30:42] beta is more of the test for does this change take down the site [20:30:50] Because it's still not fully isolated from prod [20:31:04] at one point I had the beta setup where it was just testwiki would point at the staging server [20:31:10] but for major database changes that was unwise [20:31:13] That would require reproducing Miraheze's entire stack locally which is a lot of work. Right now the only way to test mw-config changes is by using access to the beta cluster. [20:31:36] We used to just have a wiki with the same db name as the test server [20:31:46] So test1wiki went to the server called test1 [20:31:50] smart [20:31:50] https://cdn.discordapp.com/attachments/1006789349498699827/1500233378748829746/4975ba20b055417bcdcbd421c758cd47.png?ex=69f7b0b6&is=69f65f36&hm=e2cebdbfc49d792d3c74d5f76b787cf0cb1263f9a6de4744f47c7eed2de56236& [20:31:55] It was a single wiki and it shared central auth etc [20:32:02] I mean it works mostly [20:32:14] We eventually decided we wanted something a bit more complex [20:32:22] now I think I force pushed away this thing I was testing [20:32:30] yeah I definitely did [20:32:30] oh well [20:32:32] Partly because mediawiki upgrades being done to beta kept breaking stuff [20:33:01] mediawiki upgrades with database changes pmo [20:33:13] I don't think update.php is the way to do it but [20:33:23] I think we are nearly done with the nightmare migrations [20:33:28] We need some new ideas [20:33:33] it works and it saves me having to pull out datagrip [20:33:33] We've had actor, links [20:33:57] I guess we've got the file migration to do [20:34:06] is this for 1.46 or 1.45 [20:34:45] Not sure if it made it in 1.46 or maybe even 1.47 [20:35:05] They are replacing image/oldimage with file/filerevision [20:35:05] fun fun [20:35:13] seems slightly stupid [20:35:27] changing terminology from image to file is very semantic [20:35:30] Nah it makes sense [20:35:46] It moves files to using a similar schema to articles [20:36:15] Where everything is a revision and the page table is just a link to the latest and stuff that doesn't change [20:36:29] Than copying entire thing from image to old image when replaced [20:37:32] fair enough [20:57:51] well there we go https://github.com/miraheze/mw-config/pull/6398 [21:00:29] The contributing guide says you should also add it to LocalSettings.php but idk if that actually changes anything [21:05:30] Hm hold on my latest PR won't work I'm fixing this [21:08:02] oh, true. let me do that [21:12:47] Did you guys want to disable user JS and CSS on authwiki like WMF does too [21:13:14] authwiki isn't real [21:13:30] Yeah but you know what I mean [21:13:39] https://tenor.com/view/illuminati-triangle-eye-pyramid-conspiracy-theory-gif-1208222873168159165 [21:13:44] it's a conspiracy [21:14:00] legend has it that authwiki is operated by the miraheze cabal [21:14:18] [1/2] to answer your question a bit more directly: no [21:14:18] [2/2] since there isnt a wiki to disable it for [21:14:37] Oh wait yeah [21:14:41] Because you're not a user [21:14:55] technically on Special:AccountSecurity you can be [21:15:02] its literally just login in the trenchcoat of the wiki you're currently on [21:16:18] I mean that you're already setting $wgUseSiteCss and $wgUseSiteJs [21:16:48] even better the problem is solved [21:17:20] i hate CA and SUL3 [21:18:14] one day I'm gonna end up eating my words and learning mandarin before understanding CA [21:18:35] who doesn't [21:18:42] CA is genuinely the most painful thing [21:18:42] CosmicAlpha and Super UniversaL-om3ga [21:19:12] I think SUL3 "works" for WO but it forces login and attach on first request so you visit it and you auto login immediately [21:19:17] not sure that's intended behaviour but [21:19:20] that's how it works 💀 [21:19:45] I remember that Phab task where Tgr kept trying to disable extensions on auth but everything kept breaking whenever he disabled anything so he just gave up [21:20:16]