[19:22:19] <RhinosF1|NotHere>	 .
[19:22:43] <RhinosF1|NotHere>	 dmehus: done
[19:23:50] <icinga-miraheze>	 PROBLEM - wiki.yumeka.team - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[19:31:27] <dmehus>	 @RhinosF1|NotHere, ty
[19:32:01] <dmehus>	 and lol at your "oh find url yourself" comment :P
[19:34:21] <dmehus>	 @RhinosF1|NotHere What's the process for inviting MH-Discord here?
[19:34:30] <dmehus>	 as that still needs to be done
[19:34:32] <RhinosF1|NotHere>	 dmehus: same as I said earlier
[19:34:36] <dmehus>	 oh right
[19:34:39] <RhinosF1|NotHere>	 Edit a json file
[19:34:48] <miraheze-github>	 miraheze/mw-config - paladox the build passed.
[19:35:00] <dmehus>	 can I do a pull request to a private repo?
[19:35:06] <RhinosF1|NotHere>	 No
[19:35:13] <miraheze-github>	 miraheze/mw-config - paladox the build passed.
[19:35:19] <RhinosF1|NotHere>	 I said it's on server
[19:35:32] <dmehus>	 Are you mobile, or is your laptop available? Can you do the PR and push it to the private repo?
[19:35:42] <dmehus>	 since Reception123 said any MW engineer can do it
[19:35:47] <RhinosF1|NotHere>	 I am mobile
[19:35:52] <dmehus>	 oh ok
[19:35:52] <RhinosF1|NotHere>	 There's no private repo
[19:36:09] <RhinosF1|NotHere>	 And MacFan,Zppix,me,Reception123 and Void have access
[19:36:21] <dmehus>	 okay
[19:36:29] <dmehus>	 so it's not on GitHub?
[19:36:36] <RhinosF1|NotHere>	 No
[19:36:41] <dmehus>	 oh
[19:36:42] <dmehus>	 ok
[19:36:50] <RhinosF1|NotHere>	 If you nag me loud enough on Wednesday I'll do the config file
[19:36:52] <dmehus>	 Let me ask MacFan4000 then
[19:37:02] <RhinosF1|NotHere>	 To puppet
[19:37:10] <RhinosF1|NotHere>	 But it'll still need manual deploy
[19:41:34] <dmehus>	 yeah
[19:41:37] <dmehus>	 that's cool
[19:42:55] <Not-7024>	 [mw-config] paladox closed pull request #3720: Test: Do Not merge - https://git.io/JtVTy
[19:42:57] <Not-7024>	 [mw-config] paladox deleted branch paladox-patch-2 - https://git.io/vbvb3
[19:42:59] <Not-7024>	 [miraheze/mw-config] paladox deleted branch paladox-patch-2
[19:43:17] <miraheze-github>	 miraheze/mw-config - paladox the build passed.
[19:43:28] <Not-7024>	 [mw-config] paladox pushed 1 commit to paladox-patch-2 [+0/-0/±1] https://git.io/JtVTD
[19:43:30] <Not-7024>	 [mw-config] paladox f1d6354 - Test: Do Not merge
[19:43:31] <Not-7024>	 [mw-config] paladox created branch paladox-patch-2 - https://git.io/vbvb3
[19:43:33] <Not-7024>	 [mw-config] paladox reopened pull request #3720: Test: Do Not merge - https://git.io/JtVTy
[19:43:34] <Not-7024>	 [mw-config] paladox closed pull request #3720: Test: Do Not merge - https://git.io/JtVTy
[19:43:36] <Not-7024>	 [mw-config] paladox deleted branch paladox-patch-2 - https://git.io/vbvb3
[19:43:37] <Not-7024>	 [mw-config] paladox deleted branch paladox-patch-2
[19:43:49] <paladox>	 hmm, it sends deleted branch twice
[19:44:04] <icinga-miraheze>	 RECOVERY - ping6 on cp3 is OK: PING OK - Packet loss = 0%, RTA = 236.17 ms
[19:48:15] <icinga-miraheze>	 PROBLEM - ping6 on cp3 is WARNING: PING WARNING - Packet loss = 0%, RTA = 255.33 ms
[19:51:40] <icinga-miraheze>	 RECOVERY - wiki.yumeka.team - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.yumeka.team' will expire on Mon 07 Feb 2022 23:59:59 GMT +0000.
[20:03:53] <RhinosF1|NotHere>	 dmehus: does the relay work?
[20:05:44] <MacFan4000>	 I just added MirahezeBot
[20:05:58] <MacFan4000>	 It should probably get flags
[20:07:58] <RhinosF1|NotHere>	 MacFan4000: rss so it'll disappear on reboot?
[20:08:12] <paladox>	 RhinosF1|NotHere relay doesn't work
[20:08:24] <RhinosF1|NotHere>	 MacFan4000: ^
[20:08:33] <MacFan4000>	 I used .join
[20:08:49] <MacFan4000>	 It auto updates the file
[20:09:08] <RhinosF1|NotHere>	 MacFan4000: doesn't persist iirc because rss messes that up.
[20:11:18] <MacFan4000>	 As for the relay - I added it to the map and to the irc channels so not sure
[20:12:01] <MacFan4000>	 RhinosF1|NotHere: ^ (restarted without an issue)
[20:12:07] <RhinosF1|NotHere>	 Ok
[20:13:25] <wm-bot>	 I trust:  Reception123!.*@miraheze/Reception123 (2admin), .*@miraheze/John (2admin), .*@miraheze/RhinosF1 (2admin), .*@wikimedia/paladox (2admin), .*@wikipedia/Southparkfan (2admin), .*@miraheze/Universal-Omega (2admin),
[20:13:25] <MacFan4000>	 @trusted
[20:16:07] <MacFan4000>	 "#sre":"miraheze-sre"
[20:16:17] <MacFan4000>	 That’s the entry in the map
[20:16:56] <icinga-miraheze>	 PROBLEM - wiki.yumeka.team - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[20:17:45] <dmehus>	 @RhinosF1|NotHere Just saw...doesn't seem to be yet, no
[20:18:10] <Not-7024>	 [02miraheze/mediawiki] 07paladox pushed 031 commit to 03REL1_35 [+0/-0/±1] 13https://git.io/JtVI8
[20:18:11] <Not-7024>	 [02miraheze/mediawiki] 07paladox 03a6c9a53 - Update Medik
[20:18:36] <MacFan4000>	 I’m not sure why it isn’t working
[20:19:02] <dmehus>	 Oh wait
[20:19:04] <Reception123>	 MacFan4000: maybe try restarting MH-Discord? can't hurt, can it?
[20:19:04] <MacFan4000>	 The only thing I can think of would be if it’s not in #sre on discord
[20:19:05] <dmehus>	 I know
[20:19:07] <dmehus>	 I think
[20:19:19] <dmehus>	 I think the Discord bot needs send messages permissions in the channel
[20:19:21] <dmehus>	 Let me look
[20:19:37] <Reception123>	 dmehus: it should have htem
[20:19:38] <Reception123>	 *them
[20:19:51] <Reception123>	 as the "relay bot" role gives permissions out across the server
[20:20:31] <dmehus>	 @Reception123, wasn't sure if the channel permissions were specific but yeah
[20:20:32] <Reception123>	 dmehus: I just saw you tick the "Send messages", but the / means that it keeps the server-wide permission 
[20:20:34] <dmehus>	 that wasn't it
[20:20:45] <Reception123>	 unfortunately yeah, it already has send permission :(
[20:20:48] <Not-7024>	 [02miraheze/ManageWiki] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JtVIu
[20:20:50] <Not-7024>	 [02miraheze/ManageWiki] 07Universal-Omega 03303bd2a - Use #miraheze-sre
[20:20:50] <dmehus>	 yeah :(
[20:20:56] <Reception123>	 I think a restart couldn't hurt, it does fix things sometimes
[20:21:04] <dmehus>	 yeah
[20:21:07] <dmehus>	 we could try that
[20:21:13] <dmehus>	 unless it was Omega's commit
[20:21:18] <dmehus>	 that fixed it
[20:21:32] <Reception123>	 dmehus: that's for github 
[20:21:33] <Not-7024>	 [02miraheze/CreateWiki] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JtVIg
[20:21:34] <Not-7024>	 [02miraheze/CreateWiki] 07Universal-Omega 03824556e - Use #miraheze-sre
[20:21:39] <MacFan4000>	 Restarted
[20:21:46] <miraheze-github>	 miraheze/ManageWiki - Universal-Omega the build passed.
[20:21:50] <Reception123>	 doesn't seem to have fixed it :( oh well it was worth a try
[20:22:09] <dmehus>	 ah
[20:22:11] <Reception123>	 MacFan4000: is there maybe something extra that needs to be ran for it to take effect?
[20:22:21] <Not-7024>	 [02miraheze/landing] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JtVI2
[20:22:22] <Not-7024>	 [02miraheze/landing] 07Universal-Omega 03d4d9976 - Use #miraheze-sre
[20:22:35] <MacFan4000>	 I’ve never tried to add a channel before so no idea
[20:22:41] <miraheze-github>	 miraheze/CreateWiki - Universal-Omega the build passed.
[20:22:55] <Reception123>	 too bad there's not any docs
[20:23:26] <miraheze-github>	 miraheze/landing - Universal-Omega the build passed.
[20:23:38] <Not-7024>	 [02miraheze/DataDump] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JtVIw
[20:23:40] <Not-7024>	 [02miraheze/DataDump] 07Universal-Omega 0312bd021 - Use #miraheze-sre
[20:23:57] <MacFan4000>	 Zppix may know - it’s his bot
[20:24:09] <Reception123>	 yeah, he probably does but he doesn't seem around
[20:24:18] <Not-7024>	 [02miraheze/MatomoAnalytics] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JtVIK
[20:24:19] <Not-7024>	 [02miraheze/MatomoAnalytics] 07Universal-Omega 035605b5f - Use #miraheze-sre
[20:24:41] <miraheze-github>	 miraheze/DataDump - Universal-Omega the build passed.
[20:24:58] <Not-7024>	 [02miraheze/MirahezeMagic] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JtVIP
[20:24:59] <Not-7024>	 [02miraheze/MirahezeMagic] 07Universal-Omega 03089aa63 - Use #miraheze-sre
[20:25:19] <Reception123>	 MacFan4000: do you have any idea where the bot "comes from"?
[20:25:21] <miraheze-github>	 miraheze/MatomoAnalytics - Universal-Omega the build passed.
[20:25:27] <Reception123>	 since Zppix didn't make it himself I don't think
[20:25:32] <Reception123>	 so maybe there would be docs there
[20:25:33] <RhinosF1|NotHere>	 Reception123: they are docs
[20:25:42] <RhinosF1|NotHere>	 And litterally google discord-irc
[20:25:42] <Not-7024>	 [02phabricator-extensions] 07Universal-Omega opened pull request 03#10: Use #miraheze-sre - 13https://git.io/JtVIM
[20:26:00] <RhinosF1|NotHere>	 https://github.com/reactiflux/discord-irc
[20:26:01] <MirahezeBot>	 [ GitHub - reactiflux/discord-irc: Connects Discord and IRC channels by sending messages back and forth. ] - github.com
[20:26:01] <miraheze-github>	 miraheze/MirahezeMagic - Universal-Omega the build passed.
[20:26:21] <Not-7024>	 [02miraheze/WikiDiscover] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JtVI9
[20:26:22] <Not-7024>	 [02miraheze/WikiDiscover] 07Universal-Omega 0351eccd1 - Use #miraheze-sre
[20:26:49] <miraheze-github>	 miraheze/phabricator-extensions - Universal-Omega the build passed.
[20:26:54] <Reception123>	 it doesn't seem to indicate anything else that needs to be done other than the JSON
[20:26:56] <Not-7024>	 [02miraheze/IncidentReporting] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JtVI7
[20:26:57] <Not-7024>	 [02miraheze/IncidentReporting] 07Universal-Omega 03005a636 - Use #miraheze-sre
[20:27:30] <Not-7024>	 [02miraheze/CustomHeader] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JtVIb
[20:27:31] <miraheze-github>	 miraheze/WikiDiscover - Universal-Omega the build passed.
[20:27:32] <Not-7024>	 [02miraheze/CustomHeader] 07Universal-Omega 03a538469 - Use #miraheze-sre
[20:27:56] <miraheze-github>	 miraheze/IncidentReporting - Universal-Omega the build passed.
[20:28:14] <Not-7024>	 [02miraheze/PDFEmbed] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JtVIA
[20:28:16] <Not-7024>	 [02miraheze/PDFEmbed] 07Universal-Omega 035c1efbf - Use #miraheze-sre
[20:28:26] <miraheze-github>	 miraheze/CustomHeader - Universal-Omega the build passed.
[20:29:03] <Not-7024>	 [02miraheze/RottenLinks] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JtVLv
[20:29:04] <Not-7024>	 [02miraheze/RottenLinks] 07Universal-Omega 03d0e3845 - Use #miraheze-sre
[20:29:11] <miraheze-github>	 miraheze/PDFEmbed - Universal-Omega the build passed.
[20:29:36] <Not-7024>	 [02miraheze/SpriteSheet] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JtVLU
[20:29:38] <Not-7024>	 [02miraheze/SpriteSheet] 07Universal-Omega 03594d6da - Use #miraheze-sre
[20:30:01] <miraheze-github>	 miraheze/RottenLinks - Universal-Omega the build passed.
[20:30:06] <Not-7024>	 [02miraheze/YouTube] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JtVLT
[20:30:07] <Not-7024>	 [02miraheze/YouTube] 07Universal-Omega 030941937 - Use #miraheze-sre
[20:30:36] <miraheze-github>	 miraheze/SpriteSheet - Universal-Omega the build passed.
[20:31:11] <miraheze-github>	 miraheze/YouTube - Universal-Omega the build passed.
[20:31:18] <Not-7024>	 [02phabricator-extensions] 07paladox closed pull request 03#10: Use #miraheze-sre - 13https://git.io/JtVIM
[20:31:20] <Not-7024>	 [02miraheze/phabricator-extensions] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JtVLt
[20:31:21] <Not-7024>	 [02miraheze/phabricator-extensions] 07Universal-Omega 030a1b0a8 - Use #miraheze-sre (#10)
[20:32:17] <miraheze-github>	 miraheze/phabricator-extensions - paladox the build passed.
[20:34:31] <RhinosF1|NotHere>	 Reception123: they're isn't
[20:34:37] <RhinosF1|NotHere>	 It's litteraly a json file
[20:34:48] <RhinosF1|NotHere>	 2 extra lines
[20:34:59] <Reception123>	 well something clearly isn't working
[20:35:02] <Reception123>	 see -bots, maybe Void will fix it
[20:35:03] <RhinosF1|NotHere>	 Someone send me the file with passwords redacted
[20:36:29] <Not-7024>	 [02miraheze/mediawiki] 07paladox pushed 031 commit to 03REL1_35 [+0/-0/±1] 13https://git.io/JtVLo
[20:36:31] <Not-7024>	 [02miraheze/mediawiki] 07paladox 03737804c - Update CSS
[20:36:40] <MH-Discord>	 <Void> Ping
[20:36:43] <Voidwalker>	 Ping
[20:37:39] <MH-Discord>	 <Void> Relay
[20:37:43] <MH-Discord>	 <Void> is up
[20:37:44] <Reception123>	 heh of course Voidwalker fixed it :)
[20:37:45] <icinga-miraheze>	 RECOVERY - wiki.yumeka.team - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.yumeka.team' will expire on Mon 07 Feb 2022 23:59:59 GMT +0000.
[20:37:55] * Reception123 gives Voidwalker sysadmin of the year award even though he's not a sysadmin
[20:38:11] <Reception123>	 so how did you fix? :)
[20:38:26] <Voidwalker>	 "#sre":"miraheze-sre" -> "#sre":"#miraheze-sre"
[20:38:33] <Voidwalker>	 just a single missing character :P
[20:38:54] <Reception123>	 ah
[20:39:21] <Reception123>	 it's always that kind of thing that does it
[20:43:53] <MacFan4000>	 I just set the MirahezeBot chanops list
[20:44:28] <MacFan4000>	 If somebody could please give it at least +o you’ll be able to use .op and such
[20:44:50] <MH-Discord>	 <Doug> Nice. 🙂
[20:45:00] <MacFan4000>	 (It’s inheriting from #miraheze)
[20:45:05] <MH-Discord>	 <paladox> I've blacklisted embed rights for this channel and #irc-relay for the relay bot.
[20:45:21] <MH-Discord>	 <Doug> Oh, nice.
[20:45:22] <MH-Discord>	 <paladox> i don't think we need it to embed links (makes things harder to read as it just clutters)
[20:45:29] <MH-Discord>	 <Doug> yeah
[20:45:39] <MH-Discord>	 <Doug> this channel shouldn't need embeds really by anyone
[20:46:02] <MH-Discord>	 <Reception123> yeah, it's for sre business
[20:46:09] <paladox>	 https://git.io/JtVLo
[20:46:10] <MirahezeBot>	 [ Comparing a6c9a53a46dd...737804c46a11 · miraheze/mediawiki · GitHub ] - git.io
[20:46:11] <MH-Discord>	 <Reception123> embeds aren't really useful for that
[20:46:11] <paladox>	 testing
[20:46:28] <MH-Discord>	 <Doug> yeah
[20:46:37] <MH-Discord>	 <Doug> Seems to be working
[20:46:46] <MH-Discord>	 <Doug> I didn't see any embed for that link
[20:46:48] <MacFan4000>	 Reception123: can you please give MirahezeBot +o in here
[20:47:05] <Reception123>	 done
[20:47:05] <MacFan4000>	 ty
[20:47:10] <Reception123>	 no problem :)
[20:47:10] <MH-Discord>	 <Doug> ty @Reception123
[20:47:24] <MacFan4000>	 .op
[20:47:24] <MirahezeBot>	 MacFan4000: Access Denied. If in error, please contact the channel founder.
[20:47:42] <MacFan4000>	 That working as it should :)
[20:47:52] <MH-Discord>	 <Doug> yeah
[20:48:09] <Reception123>	 .op
[20:48:09] <MirahezeBot>	 Attempting to OP...
[20:48:13] <Reception123>	 heh 
[20:50:03] <dmehus>	 heh music troll doesn't know about this channel, so that's a 👍
[20:55:31] <Not-7024>	 [02miraheze/mediawiki] 07paladox pushed 031 commit to 03REL1_35 [+0/-0/±1] 13https://git.io/JtVtP
[20:55:33] <Not-7024>	 [02miraheze/mediawiki] 07paladox 03d1b8f12 - Update CommentStreams
[20:56:35] <Reception123>	 dmehus: heh, don't jinx it, it has been mentioned in #miraheze 
[20:56:45] <Reception123>	 let's just cross our fingers that the troll won't find out about it
[20:57:44] <Reception123>	 just to be sure, we should have this channel "mirror" #miraheze bans like the others
[20:57:51] <Reception123>	 I forgot how to do that though
[20:58:39] <Voidwalker>	 mode +b $j:#miraheze
[20:59:20] <Reception123>	 thanks!
[21:00:01] <wm-bot>	 Seen is now enabled in the channel
[21:00:01] <MacFan4000>	 @seen-on
[21:00:09] <MacFan4000>	 @statistics-on
[21:00:10] <wm-bot>	 Statistics were now enabled
[21:00:20] <wm-bot>	 http://wm-bot.wmflabs.org/dump/%23miraheze-sre.htm
[21:00:20] <MacFan4000>	 @info
[21:00:22] <MirahezeBot>	 [ #miraheze-sre ] - wm-bot.wmflabs.org
[21:00:57] <RhinosF1|NotHere>	 MacFan4000: I'm sure we can do that ourselves
[21:01:11] <Reception123>	 thanks for doing it though :)
[21:01:47] <dmehus>	 MacFa4000, yes, thanks, and apologies for RhinosF1's unnecessary directness :(
[21:01:58] <dmehus>	 s/MacFan4000/MacFan4000
[21:02:04] <dmehus>	 wait
[21:02:09] <dmehus>	 why doesn't that work?
[21:02:18] <dmehus>	 what bot controls that?
[21:03:52] <Voidwalker>	 probably didn't work because it didn't actually match your message
[21:04:15] <Reception123>	 lol yes, you just wrote MacFan twice :P
[21:12:43] <dmehus>	 s/MacFa4000/MacFan4000
[21:12:43] <MirahezeBot>	 dmehus meant to say: MacFan4000, yes, thanks, and apologies for RhinosF1's unnecessary directness :(
[21:12:52] <dmehus>	 @Voidwalker lol I did too
[21:12:57] <dmehus>	 and thanks also @Reception123
[21:18:40] <SPF|Cloud>	 https://grafana.miraheze.org/d/W9MIkA7iz/miraheze-cluster?orgId=1&from=now-1h&to=now-1m&var-job=node&var-node=dbbackup1.miraheze.org&var-port=9100 pity
[21:18:40] <MirahezeBot>	 [ Grafana ] - grafana.miraheze.org
[21:19:15] <SPF|Cloud>	 even with reduced load the disks aren't fast enough to cope well with the replication load
[21:20:27] <dmehus>	 SPF|Cloud, ah, interesting...yeah, we're still seeing a fair bit of slowness right now
[21:20:57] <SPF|Cloud>	 I don't think this is the cause of any of your performance issues
[21:21:38] <dmehus>	 Well, I mean querying user lists and page searching would be a check of the database, so it might be.
[21:21:51] <dmehus>	 Would increasing the number of course from 2 to 4 help?
[21:22:00] <dmehus>	 s/course/cores
[21:22:00] <MirahezeBot>	 dmehus meant to say: Would increasing the number of cores from 2 to 4 help?
[21:22:51] <SPF|Cloud>	 no MediaWiki process accesses a dbbackup server
[21:23:16] <dmehus>	 oh
[21:23:45] <dmehus>	 so what is the downside you're seeing with the dbbackup1 server then?
[21:23:55] <dmehus>	 where does that manifest itself?
[21:24:18] <SPF|Cloud>	 it's a backup server, not a production replica. the first is to help us create proper database backups, the second helps to increase service availability and reduce load on the master
[21:24:50] <SPF|Cloud>	 ability of the backup server to stay in sync with the latest wiki edits, speed of backup creation
[21:25:02] <dmehus>	 yeah, and ah that makes sense
[21:27:00] <SPF|Cloud>	 db12 mariadb has an uptime of almost 17 weeks :o
[21:28:38] <dmehus>	 wow
[21:43:27] <MacFan4000>	 dmehus: heh you’re already top of wm-bots statistics list
[21:46:35] <dmehus>	 MacFan4000, lol yeah. Just in the Miraheze channels list though, I think, or is there statistics for all channels?
[21:46:50] <wm-bot>	 http://wm-bot.wmflabs.org/dump/%23miraheze-sre.htm
[21:46:50] <MacFan4000>	 @info
[21:46:51] <MirahezeBot>	 [ #miraheze-sre ] - wm-bot.wmflabs.org
[21:46:55] <MacFan4000>	 This channel
[21:47:07] <dmehus>	 oh heh
[21:47:43] <dmehus>	 heh we'll actually have a chance of topping the bots in #miraheze now :P
[21:48:05] <MacFan4000>	 In there it’s currently still icinga
[21:48:12] <MacFan4000>	 https://wm-bot.wmflabs.org/dump/%23miraheze.htm
[21:48:13] <MirahezeBot>	 [ #miraheze ] - wm-bot.wmflabs.org
[21:48:30] <dmehus>	 I won't last for long once MH-Discord, icinga-miraheze, and Not-7024 produce more notifications
[21:48:47] <dmehus>	 yeah
[21:48:59] <dmehus>	 even paladox is a distant second at 34,000-ish messages
[21:49:28] <dmehus>	 I almost have more than NDKilla though :P
[21:49:47] <MacFan4000>	 Icinga is at 98935 which is still way more
[21:50:25] <dmehus>	 yeah
[21:59:43] <paladox>	 !log depool/repool mw10 and rebuild lc
[21:59:47] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[21:59:54] <JohnLewis>	 paladox: why?
[22:00:30] <paladox>	 I'm trying to see if this will help with https://phabricator.miraheze.org/T6826. I rebuilt mw9 yesturday which doesn't seem to be experencing the issue anymore.
[22:00:31] <MirahezeBot>	 [ ⚓ T6826 Core Namespace translations having no effect ] - phabricator.miraheze.org
[22:00:39] <JohnLewis>	 it won't
[22:00:42] <JohnLewis>	 We've been over it
[22:00:43] <SPF|Cloud>	 https://phabricator.miraheze.org/T6826
[22:00:44] <MirahezeBot>	 [ ⚓ T6826 Core Namespace translations having no effect ] - phabricator.miraheze.org
[22:01:00] <paladox>	 oh
[22:01:07] <SPF|Cloud>	 when running var_dump( Title::newFromText( 'Plantilla:AD' ) );, you can see differences between mw9 and mw11
[22:01:20] <JohnLewis>	 If you want to see if i18n would, run MediaWikiServices::getInstance()->getLocalisationCache()->getItem( $code, 'namespaceNames' )
[22:01:33] <SPF|Cloud>	 on mw9, Title->mNamespace is 10, but on mw11 Title-mNamespace is 0
[22:01:34] <icinga-miraheze>	 PROBLEM - cp11 Varnish Backends on cp11 is CRITICAL: 1 backends are down. mw10
[22:01:51] <JohnLewis>	 If that returns none-English, LC isn't the problem. And so far, I've ran that on every server and it's worked fine
[22:02:22] <icinga-miraheze>	 PROBLEM - cp12 Varnish Backends on cp12 is CRITICAL: 1 backends are down. mw10
[22:02:40] <icinga-miraheze>	 PROBLEM - cp3 Varnish Backends on cp3 is CRITICAL: 1 backends are down. mw10
[22:02:57] <icinga-miraheze>	 PROBLEM - cp10 Varnish Backends on cp10 is CRITICAL: 1 backends are down. mw10
[22:03:49] <paladox>	 > var_dump(MediaWiki\MediaWikiServices::getInstance()->getLocalisationCache()->getItem( 'es', 'Plantilla' ));
[22:03:49] <paladox>	 NULL
[22:04:49] <SPF|Cloud>	 uh, 'Plantilla' should be 'namespaceNames'
[22:05:00] <paladox>	 https://es.nocyclo.tk/wiki/Plantilla:AD works on mw10 now
[22:05:02] <MirahezeBot>	 [ Plantilla:AD - Risapedia, la enciclopedia seria ] - es.nocyclo.tk
[22:05:04] <paladox>	 after rebuilding lc
[22:05:09] <paladox>	 by wiping it
[22:06:37] <JohnLewis>	 But that’s not the fix
[22:06:53] <JohnLewis>	 If it was, it would have worked the first 3 times you did it :)
[22:06:56] <icinga-miraheze>	 RECOVERY - cp10 Varnish Backends on cp10 is OK: All 7 backends are healthy
[22:07:11] <paladox>	 JohnLewis i mean it did
[22:07:17] <RhinosF1|NotHere>	 I also don't think constantly clearing things and reinstalling them is sustainable
[22:07:17] <JohnLewis>	 The fix it to clear CreateWiki cache *correctly*
[22:07:19] <paladox>	 mw9 is working without issue since i did that yesturday
[22:07:35] <icinga-miraheze>	 RECOVERY - cp11 Varnish Backends on cp11 is OK: All 7 backends are healthy
[22:07:44] <JohnLewis>	 So why isn’t mw10 or mw11 which you did yesterday and the day before?
[22:07:56] <paladox>	 I didn't wipe lc on mw 10/11 though
[22:08:05] <JohnLewis>	 You reinstalled them
[22:08:06] <paladox>	 i just rebuilt it by running the command, i wiped mw9 lc 
[22:08:13] <paladox>	 no, i reinstalled mw9
[22:08:22] <icinga-miraheze>	 RECOVERY - cp12 Varnish Backends on cp12 is OK: All 7 backends are healthy
[22:08:27] <JohnLewis>	 But you ran that command that proves it’s working 
[22:08:31] <JohnLewis>	 aka not LC
[22:08:41] <icinga-miraheze>	 RECOVERY - cp3 Varnish Backends on cp3 is OK: All 7 backends are healthy
[22:11:48] <SPF|Cloud>	 please don't rebuild on mw11 
[22:12:41] <JohnLewis>	 SPF|Cloud; if you’re debugging it, I can tell you problem now because I already debugged earlier and said what the problem was
[22:12:42] <paladox>	 sure i'm not :)
[22:12:57] <SPF|Cloud>	 what was the problem?
[22:13:12] <JohnLewis>	 English namespace names are cached in CreateWiki cache
[22:13:57] <paladox>	 oh
[22:14:06] <paladox>	 what would the fix be to prevent that happening JohnLewis ?
[22:14:29] <JohnLewis>	 Clearing the cache via resetWiki
[22:14:51] <paladox>	 ok, but i mean prevent it happening again
[22:15:49] <JohnLewis>	 a) support English only wikis b) drop support for changing language via CreateWiki c) drop i18n support for namespaces on Miraheze
[22:16:42] <JohnLewis>	 or d) revert CreateWiki/ManageWiki back about a year to using all.dblist and remove pre-setup configuration 
[22:16:53] <paladox>	 oh
[22:18:42] <paladox>	 SPF|Cloud can i clear CW cache?
[22:18:49] <paladox>	 or do you want me to hold off
[22:18:51] <paladox>	 *still
[22:18:55] <SPF|Cloud>	 go for it
[22:19:07] <paladox>	 thanks!
[22:19:07] <Universal_Omega>	 JohnLewis: why is it only happening now and wasn't before the past couple days?
[22:19:54] <paladox>	 !log root@jobrunner3:/home/paladox# /usr/local/bin/foreachwikiindblist /srv/mediawiki/w/cache/databases.json /home/paladox/resetWikiCaches.php
[22:19:57] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[22:21:51] <JohnLewis>	 Universal_Omega; because before we had existing cache that provided the language for namespaces to use. Right now, wikis are assumed English when cache generation happens first time 
[22:22:24] <Universal_Omega>	 Oh
[22:22:31] <Universal_Omega>	 Thanks 
[22:24:52] <RhinosF1|NotHere>	 JohnLewis: couldn't it get that from database when creating the cache
[22:25:49] <MH-Discord>	 <Doug> > <JohnLewis> English namespace names are cached in CreateWiki cache @JohnLewis Oh interesting, is that the cause of #T6826?
[22:29:38] <JohnLewis>	 RhinosF1; it will, the second time. Otherwise you’d need to provision MediaWiki without MediaWiki knowing what wiki to is, what the database server is, nothing 
[22:30:04] <Not-7024>	 [02puppet] 07Universal-Omega edited pull request 03#1638: Remove old - 13https://git.io/JtauB
[22:30:35] <RhinosF1|NotHere>	 JohnLewis: how long does it take to generate the cache a second time?
[22:30:38] <Not-7024>	 [02puppet] 07Universal-Omega edited pull request 03#1638: Remove old infrastructure - 13https://git.io/JtauB
[22:30:48] <JohnLewis>	 An action that causes the cache to regen
[22:31:07] <JohnLewis>	 There’s no real clean fix and it’s a 1 in a 10000 action
[22:31:31] <JohnLewis>	 we don’t do fresh server installs hourly, we don’t harshly wipe cache hourly, we don’t even do it monthly 
[22:31:56] <JohnLewis>	 so investing in a real fix when the solution is so easy and needs to be done so rarely makes it unrealistic
[22:33:40] <RhinosF1|NotHere>	 Ah
[22:34:00] <RhinosF1|NotHere>	 So it just needs cache to warm up on all servers and be cleared once
[22:34:37] <JohnLewis>	 Yeah
[22:34:59] <RhinosF1|NotHere>	 Ah
[22:35:16] <JohnLewis>	 But then once it’s done, it’s done. Unless you remove all cache files entirely, it’s extremely unlikely to reoccur. And if it does, it’s a bug
[22:36:05] <RhinosF1|NotHere>	 If we really wanted to I suppose we could force send a request to every wiki on every app server and then do a proper reset
[22:36:27] <RhinosF1|NotHere>	 But I'm too lazy to write that and it would take a while with the rate limit
[22:36:51] <RhinosF1|NotHere>	 I suppose only non English wikis would matter
[22:37:30] <Not-7024>	 [02puppet] 07dmehus commented on pull request 03#1638: Remove old infrastructure - 13https://git.io/JtV33
[22:37:36] <JohnLewis>	 It’s not even per server
[22:37:50] <JohnLewis>	 cache expiration is centralised in Redis
[22:37:57] <RhinosF1|NotHere>	 Oh that's cool
[22:52:00] <dmehus>	 JohnLewis, ah, that's helpful to know. Do we have a default cache expiry for MediaWiki (i.e., things like "page information using `action=info`)? If so, I'm wondering if we can lower the cache expiry to, say, 24 hours
[22:55:38] <JohnLewis>	 CreateWiki cache is in theory infinite
[23:04:07] <dmehus>	 JohnLewis, oh this is just CreateWiki cache we're talking about, right
[23:05:42] <JohnLewis>	 Yes
[23:12:23] <dmehus>	 ah, ok
[23:22:49] <icinga-miraheze>	 PROBLEM - test3 APT on test3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds.
[23:24:47] <icinga-miraheze>	 RECOVERY - test3 APT on test3 is OK: APT OK: 28 packages available for upgrade (0 critical updates).
[23:40:13] <icinga-miraheze>	 PROBLEM - ping6 on dbbackup2 is CRITICAL: PING CRITICAL - Packet loss = 100%
[23:42:15] <icinga-miraheze>	 PROBLEM - ping6 on dbbackup2 is WARNING: PING WARNING - Packet loss = 0%, RTA = 102.75 ms