[00:02:14] PROBLEM - cloud2 Current Load on cloud2 is CRITICAL: CRITICAL - load average: 25.84, 20.33, 16.41 [00:04:14] RECOVERY - cloud2 Current Load on cloud2 is OK: OK - load average: 12.98, 17.55, 15.88 [05:28:51] PROBLEM - wiki.alirezaivaz.ir - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.alirezaivaz.ir All nameservers failed to answer the query. [05:41:30] PROBLEM - wiki.m.alirezaivaz.ir - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.m.alirezaivaz.ir All nameservers failed to answer the query. [05:42:22] RECOVERY - wiki.alirezaivaz.ir - reverse DNS on sslhost is OK: rDNS OK - wiki.alirezaivaz.ir reverse DNS resolves to cp7.miraheze.org [05:45:39] PROBLEM - wiki.alirezaivaz.ir - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [05:52:32] RECOVERY - wiki.alirezaivaz.ir - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.alirezaivaz.ir' will expire on Mon 28 Dec 2020 20:59:30 GMT +0000. [05:59:56] PROBLEM - wiki.alirezaivaz.ir - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.alirezaivaz.ir All nameservers failed to answer the query. [06:13:24] RECOVERY - wiki.alirezaivaz.ir - reverse DNS on sslhost is OK: rDNS OK - wiki.alirezaivaz.ir reverse DNS resolves to cp7.miraheze.org [06:15:56] RECOVERY - wiki.m.alirezaivaz.ir - reverse DNS on sslhost is OK: rDNS OK - wiki.m.alirezaivaz.ir reverse DNS resolves to cp6.miraheze.org [12:26:13] PROBLEM - ping6 on misc1 is CRITICAL: PING CRITICAL - Packet loss = 37%, RTA = 23.09 ms [12:28:12] RECOVERY - ping6 on misc1 is OK: PING OK - Packet loss = 0%, RTA = 14.99 ms [15:00:20] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JTwTx [15:00:21] [02miraheze/services] 07MirahezeSSLBot 03102f3db - BOT: Updating services config for wikis [15:02:05] PROBLEM - cloud2 Current Load on cloud2 is WARNING: WARNING - load average: 23.41, 21.34, 17.77 [15:04:04] RECOVERY - cloud2 Current Load on cloud2 is OK: OK - load average: 18.29, 19.83, 17.64 [15:10:18] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JTwIU [15:10:20] [02miraheze/services] 07MirahezeSSLBot 030cd4b94 - BOT: Updating services config for wikis [15:19:09] PROBLEM - cloud2 Current Load on cloud2 is CRITICAL: CRITICAL - load average: 27.36, 22.39, 19.35 [15:21:09] PROBLEM - cloud2 Current Load on cloud2 is WARNING: WARNING - load average: 23.48, 23.20, 20.06 [15:27:09] RECOVERY - cloud2 Current Load on cloud2 is OK: OK - load average: 18.60, 20.04, 19.66 [16:07:05] PROBLEM - cloud2 Current Load on cloud2 is WARNING: WARNING - load average: 22.35, 21.12, 18.24 [16:09:05] RECOVERY - cloud2 Current Load on cloud2 is OK: OK - load average: 16.31, 18.70, 17.67 [16:13:33] [02miraheze/mw-config] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/JTw3i [16:13:34] [02miraheze/mw-config] 07paladox 03ce229c0 - Install newsletter namespace when installing newsletter [16:13:36] [02mw-config] 07paladox created branch 03paladox-patch-1 - 13https://git.io/vbvb3 [16:13:37] [02mw-config] 07paladox opened pull request 03#3312: Install newsletter namespace when installing newsletter - 13https://git.io/JTw3P [16:15:08] [02mw-config] 07paladox synchronize pull request 03#3312: Install newsletter namespace when installing newsletter - 13https://git.io/JTw3P [16:15:10] [02miraheze/mw-config] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/JTw3d [16:15:11] [02miraheze/mw-config] 07paladox 03edbfc9e - Update ManageWikiExtensions.php [16:19:23] [02mw-config] 07paladox synchronize pull request 03#3312: Install newsletter namespace when installing newsletter - 13https://git.io/JTw3P [16:19:24] [02miraheze/mw-config] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/JTwsG [16:19:26] [02miraheze/mw-config] 07paladox 03ee897b6 - Update ManageWikiExtensions.php [16:22:14] [02mw-config] 07paladox closed pull request 03#3312: Install newsletter namespace when installing newsletter - 13https://git.io/JTw3P [16:22:16] [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JTws1 [16:22:17] [02miraheze/mw-config] 07paladox 035103d74 - Install newsletter namespace when installing newsletter (#3312) * Install newsletter namespace when installing newsletter * Update ManageWikiExtensions.php * Update ManageWikiExtensions.php [16:22:19] [02mw-config] 07paladox deleted branch 03paladox-patch-1 - 13https://git.io/vbvb3 [16:22:20] [02miraheze/mw-config] 07paladox deleted branch 03paladox-patch-1 [16:24:17] [02miraheze/mw-config] 07paladox pushed 033 commits to 03REL1_35 [+0/-0/±3] 13https://git.io/JTwsx [16:24:19] [02miraheze/mw-config] 07paladox 03f92cd96 - Merge branch 'master' into REL1_35 * master: Install newsletter namespace when installing newsletter (#3312) MWE: add Score note (#3309) [16:25:13] PROBLEM - cloud2 Current Load on cloud2 is WARNING: WARNING - load average: 20.99, 19.81, 17.89 [16:27:12] RECOVERY - cloud2 Current Load on cloud2 is OK: OK - load average: 16.33, 18.58, 17.68 [16:28:06] [02miraheze/mw-config] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/JTwGc [16:28:08] [02miraheze/mw-config] 07paladox 03f486bed - Add NewsletterContent content model to Newsletter namespace [16:28:09] [02mw-config] 07paladox created branch 03paladox-patch-1 - 13https://git.io/vbvb3 [16:28:11] [02mw-config] 07paladox opened pull request 03#3313: Add NewsletterContent content model to Newsletter namespace - 13https://git.io/JTwGC [16:29:24] [02mw-config] 07paladox closed pull request 03#3313: Add NewsletterContent content model to Newsletter namespace - 13https://git.io/JTwGC [16:29:26] [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JTwGg [16:29:27] [02miraheze/mw-config] 07paladox 037b9baa0 - Add NewsletterContent content model to Newsletter namespace (#3313) [16:29:28] [02mw-config] 07paladox deleted branch 03paladox-patch-1 - 13https://git.io/vbvb3 [16:29:30] [02miraheze/mw-config] 07paladox deleted branch 03paladox-patch-1 [16:29:53] [02miraheze/mw-config] 07paladox pushed 032 commits to 03REL1_35 [+0/-0/±2] 13https://git.io/JTwGr [16:29:55] [02miraheze/mw-config] 07paladox 03da79671 - Merge branch 'master' into REL1_35 * master: Add NewsletterContent content model to Newsletter namespace (#3313) [16:31:11] PROBLEM - cloud2 Current Load on cloud2 is CRITICAL: CRITICAL - load average: 24.06, 21.27, 18.93 [16:33:11] PROBLEM - cloud2 Current Load on cloud2 is WARNING: WARNING - load average: 19.85, 20.98, 19.12 [16:35:11] PROBLEM - cloud2 Current Load on cloud2 is CRITICAL: CRITICAL - load average: 27.47, 22.45, 19.82 [16:37:11] PROBLEM - cloud2 Current Load on cloud2 is WARNING: WARNING - load average: 17.47, 20.87, 19.59 [16:38:05] miraheze/mw-config/paladox-patch-1/f486bed - paladox The build has errored. https://travis-ci.org/miraheze/mw-config/builds/738575958 [16:38:35] PROBLEM - ping6 on misc1 is CRITICAL: PING CRITICAL - Packet loss = 0%, RTA = 302.90 ms [16:39:11] RECOVERY - cloud2 Current Load on cloud2 is OK: OK - load average: 15.99, 20.09, 19.51 [16:40:35] RECOVERY - ping6 on misc1 is OK: PING OK - Packet loss = 0%, RTA = 29.30 ms [16:45:11] PROBLEM - cloud2 Current Load on cloud2 is CRITICAL: CRITICAL - load average: 29.14, 23.49, 20.78 [16:47:11] PROBLEM - cloud2 Current Load on cloud2 is WARNING: WARNING - load average: 19.05, 21.81, 20.50 [16:49:28] dmehus: which cache proxy do you use? [16:49:52] SPF|Cloud: he's Canada [16:50:02] so cp9 [16:50:32] It will be [16:50:32] https://meta.miraheze.org/w/index.php?title=MediaWiki:Mainpage&type=revision&diff=138240&oldid=155&diffmode=visual I suppose this will cause erratic behavior in varnish [16:50:33] [ Difference between revisions of "MediaWiki:Mainpage" - Miraheze Meta ] - meta.miraheze.org [16:50:51] SPF|Cloud: why would it? [16:51:03] Unless varnish caches redirects [16:51:11] RECOVERY - cloud2 Current Load on cloud2 is OK: OK - load average: 17.46, 19.95, 20.04 [16:51:24] because the first anonymous user to hit this wiki will define what language is cached in varnish [16:51:35] SPF|Cloud: ah ye [16:51:51] and that's why wgULSAnonCanChangeLanguage should be set to false, but apparently that was enabled for meta in April [16:52:02] SPF|Cloud: wouldn't anon default to English? [16:52:15] no [16:52:31] my browser tells Miraheze to prefer Dutch [16:52:31] It asks for browser lang iric [16:53:17] Ah [16:53:37] https://meta.miraheze.org/wiki/ on cp7 shows the Dutch main page now, because I tricked varnish in caching that one [16:53:38] [ Miraheze - Miraheze Meta ] - meta.miraheze.org [16:53:55] Oof [16:54:07] Yeah we should disable that [16:54:15] SPF|Cloud: should I revert that edit and you kill cache? [16:54:38] technically, setting to wgULSAnonCanChangeLanguage should work [16:54:56] so logged in users would be redirected to the main page in their language [16:55:24] and since we don't cache those requests (for security reasons), that won't cause cache issues [16:55:24] So we do that and undo the edit? [16:55:35] let's try the ULS trick first [16:55:57] Okay [16:58:35] @SPF|Cloud Feel free to revert my change. I did wonder about that, to be honest, as I do see my web browser redirecting me [[Special:MyLanguage/Miraheze]] then to [[Miraheze]]. If there's some setting change you guys can do in the back end, that's definitely preferable, I think [16:58:58] it's an issue @ Wikimedia as well [16:59:18] and the only change we can implement is disabling caching for the meta main page, but I'm not a fan of that [17:00:37] @SPF|Cloud. Ah, yeah, personally, one's Special:Preferences language setting should automatically redirect them to their main page anyway. This really only avoids 1 extra click to the user's language of choice when they click "Main Page" in the sidebar [17:01:39] Want me to undo my edit, or leave for now? [17:01:59] SPF|Cloud: could we not just ban the caching of the redirect between Special:MyLanguage/Miraheze and [17:02:16] disable ULS for anonymous users first [17:02:17] As each page language has its own url [17:03:04] technically possible, but requires to incorporate a list of the translations of 'Special:MyLanguage' itself as well [17:03:24] Ah good point [17:03:29] ah [17:04:09] is there any downside(s) to disabling ULS for anonymous users, @SPF|Cloud? [17:04:10] PROBLEM - cloud2 Current Load on cloud2 is WARNING: WARNING - load average: 23.28, 21.34, 20.04 [17:04:12] Unless we can match meta.miraheze.org/*/*:*/Miraheze [17:04:27] We need to match index.php as well [17:04:53] But I doubt many things would accidentally match *:*/Miraheze [17:05:22] it would be /*:*/Miraheze even [17:05:27] is spending time on a change worth the positive impact of automatically redirecting anonymous users to pages in their language? [17:05:35] Probably not [17:06:06] ?uselang= will still work, by the way [17:06:09] PROBLEM - cloud2 Current Load on cloud2 is CRITICAL: CRITICAL - load average: 24.61, 22.25, 20.51 [17:06:14] Yeah [17:07:33] huh, I issued a ban in varnishadm, but I still see the NL page [17:07:34] heh? [17:08:09] PROBLEM - cloud2 Current Load on cloud2 is WARNING: WARNING - load average: 18.77, 22.25, 20.83 [17:10:56] @SPF|Cloud, hrm, could there still be a cache issue somewhere, kinda like how it takes several days for the "number of subpages" in the "Page Information" page to update after deleting a subpage? [17:11:13] SPF|Cloud: logged out? [17:11:26] of course, I am in incognito mode [17:11:31] Try a different browser? [17:11:38] already did [17:12:14] even all buttons are in Dutch, which is interesting, because I just disabled ULS for anonymous users [17:12:28] @SPF|Cloud, do you have a language preference set in your web browser? *is sure you've thought of that, but sometimes we miss such things* [17:12:42] Not seeing it here [17:12:47] I do, that's very common for browsers to do [17:13:17] that's why you shoud disable ULS if not logged in, so the wiki will only show pages in the wiki's default language [17:13:19] maybe it's the web browser redirecting you to the NL subpage then? Not entirely sure [17:13:31] oh [17:13:41] the web browser needs ULS to be able to do that? [17:14:00] ULS can detect the preferred language sent by your browser [17:14:04] SPF|Cloud: could their be caching on MediaWiki's end? [17:14:10] RECOVERY - cloud2 Current Load on cloud2 is OK: OK - load average: 17.54, 20.11, 20.34 [17:14:12] ?action=purge help? [17:14:31] caching? could be, but I don't know what part would cache such things [17:15:56] the wiki is in Dutch, although the ULS widget (in your personal toolbox on top of the page, if using Vector) tells me to sign in if I want to change the language [17:17:05] @SPF|Cloud Hrm, I just tried changing my preferences to Francais, then loaded up Meta in incognito mode and saw the English version logged out [17:18:33] this is fun [17:18:48] Special:MyLanguage redirects me to /nl, but the MediaWik interface is in English [17:19:11] Redirect cache? [17:19:15] In varnish? [17:20:36] hm [17:20:47] purging everything for meta.miraheze.org on cp6/cp7 now [17:24:46] edited my /etc/hosts to point to cp7, restarted Edge and Firefox, visited meta.mh.o/wiki/ in Edge (Dutch preferred), I get the Dutch page. Then I visit using Firefox (German preferred) and I get the Dutch page as well [17:27:35] it looks like there is some javascript involved rewriting the URL in my address bar from Special:MyLanguage/Miraheze to Miraheze/ as well, there's no redirect doing that [17:27:51] Hmm [17:27:59] That could be interesting [17:28:22] disabling JS on metawiki stops that rewrite [17:28:42] Maybe another layer of cache [17:30:21] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JTwlo [17:30:22] [02miraheze/services] 07MirahezeSSLBot 030300eff - BOT: Updating services config for wikis [17:30:34] brb [17:33:22] PROBLEM - cloud2 Current Load on cloud2 is CRITICAL: CRITICAL - load average: 39.53, 24.21, 20.95 [17:35:21] PROBLEM - cloud2 Current Load on cloud2 is WARNING: WARNING - load average: 15.29, 20.57, 20.04 [17:37:20] RECOVERY - cloud2 Current Load on cloud2 is OK: OK - load average: 13.62, 18.32, 19.28 [17:43:17] PROBLEM - cloud2 Current Load on cloud2 is CRITICAL: CRITICAL - load average: 29.37, 22.29, 20.39 [17:45:17] PROBLEM - cloud2 Current Load on cloud2 is WARNING: WARNING - load average: 21.86, 22.57, 20.77 [17:47:16] RECOVERY - cloud2 Current Load on cloud2 is OK: OK - load average: 11.93, 18.67, 19.57 [17:59:11] I think I've found it [17:59:44] if wgULSLanguageDetection is set to true, the Accept-Language header will be honored [18:07:40] [02miraheze/mw-config] 07Southparkfan pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JTwRm [18:07:41] [02miraheze/mw-config] 07Southparkfan 038f4b94b - Disable language detection (ULS) for all wikis [18:12:32] [02miraheze/mw-config] 07Southparkfan pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JTwRo [18:12:33] [02miraheze/mw-config] 07Southparkfan 032ea5a7a - Enable RottenLinks [18:14:42] dmehus, Spookreeeno: ^ fixed :) [18:15:20] the language detection (which looks at your Accept-Language header) was interfering [18:15:50] Ok [18:21:38] !log ban all URLs containing 'MyLanguage' on cache proxies [18:21:47] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:47:57] PROBLEM - ping6 on misc1 is CRITICAL: PING CRITICAL - Packet loss = 0%, RTA = 297.75 ms [18:49:58] PROBLEM - ping6 on misc1 is WARNING: PING WARNING - Packet loss = 0%, RTA = 181.59 ms [18:51:59] RECOVERY - ping6 on misc1 is OK: PING OK - Packet loss = 0%, RTA = 12.07 ms [19:32:10] PROBLEM - cloud2 Current Load on cloud2 is CRITICAL: CRITICAL - load average: 27.49, 24.54, 20.14 [19:34:09] PROBLEM - cloud2 Current Load on cloud2 is WARNING: WARNING - load average: 22.25, 23.25, 20.18 [19:36:09] RECOVERY - cloud2 Current Load on cloud2 is OK: OK - load average: 15.17, 20.12, 19.40 [20:08:08] PROBLEM - cloud2 Current Load on cloud2 is WARNING: WARNING - load average: 21.02, 20.02, 18.51 [20:10:09] RECOVERY - cloud2 Current Load on cloud2 is OK: OK - load average: 15.67, 18.83, 18.29 [21:13:10] PROBLEM - cloud2 Current Load on cloud2 is WARNING: WARNING - load average: 22.97, 21.27, 18.59 [21:17:11] PROBLEM - cloud2 Current Load on cloud2 is CRITICAL: CRITICAL - load average: 39.78, 29.65, 22.32 [21:17:46] PROBLEM - cp7 Current Load on cp7 is CRITICAL: CRITICAL - load average: 12.36, 10.01, 5.66 [21:19:15] PROBLEM - munwiki.info - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for munwiki.info could not be found [21:19:21] PROBLEM - wiki.teamrelectric.ca - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.teamrelectric.ca could not be found [21:19:46] PROBLEM - cp7 Current Load on cp7 is WARNING: WARNING - load average: 2.83, 7.28, 5.20 [21:22:00] PROBLEM - cp7 Current Load on cp7 is CRITICAL: CRITICAL - load average: 17.12, 11.16, 6.89 [21:25:55] PROBLEM - cp7 Current Load on cp7 is WARNING: WARNING - load average: 3.27, 7.32, 6.36 [21:26:03] RECOVERY - wiki.teamrelectric.ca - reverse DNS on sslhost is OK: rDNS OK - wiki.teamrelectric.ca reverse DNS resolves to cp6.miraheze.org [21:26:12] RECOVERY - munwiki.info - reverse DNS on sslhost is OK: rDNS OK - munwiki.info reverse DNS resolves to cp6.miraheze.org [21:27:53] RECOVERY - cp7 Current Load on cp7 is OK: OK - load average: 2.83, 5.72, 5.88 [21:33:22] PROBLEM - cloud2 Current Load on cloud2 is WARNING: WARNING - load average: 19.77, 21.07, 23.38 [21:39:23] RECOVERY - cloud2 Current Load on cloud2 is OK: OK - load average: 17.92, 16.35, 20.27 [22:28:04] PROBLEM - cp7 Current Load on cp7 is CRITICAL: CRITICAL - load average: 12.12, 8.93, 4.78 [22:28:13] PROBLEM - cloud2 Current Load on cloud2 is CRITICAL: CRITICAL - load average: 50.50, 33.02, 22.64 [22:29:13] PROBLEM - gluster1 Current Load on gluster1 is WARNING: WARNING - load average: 7.17, 6.37, 4.11 [22:29:29] PROBLEM - wiki.cdntennis.ca - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.cdntennis.ca could not be found [22:31:13] RECOVERY - gluster1 Current Load on gluster1 is OK: OK - load average: 2.75, 4.93, 3.86 [22:32:05] RECOVERY - cp7 Current Load on cp7 is OK: OK - load average: 2.00, 5.90, 4.54 [22:34:13] PROBLEM - cloud2 Current Load on cloud2 is WARNING: WARNING - load average: 15.22, 22.42, 21.37 [22:36:13] PROBLEM - cloud2 Current Load on cloud2 is CRITICAL: CRITICAL - load average: 35.85, 27.55, 23.32 [22:36:20] RECOVERY - wiki.cdntennis.ca - reverse DNS on sslhost is OK: rDNS OK - wiki.cdntennis.ca reverse DNS resolves to cp7.miraheze.org [22:37:59] PROBLEM - cp7 Current Load on cp7 is CRITICAL: CRITICAL - load average: 6.75, 9.88, 6.74 [22:38:12] PROBLEM - cloud2 Current Load on cloud2 is WARNING: WARNING - load average: 16.62, 23.54, 22.39 [22:39:57] PROBLEM - cp7 Current Load on cp7 is WARNING: WARNING - load average: 1.69, 6.93, 6.04 [22:41:55] RECOVERY - cp7 Current Load on cp7 is OK: OK - load average: 1.32, 5.15, 5.49 [22:42:11] RECOVERY - cloud2 Current Load on cloud2 is OK: OK - load average: 15.98, 18.06, 20.30 [23:07:01] PROBLEM - cloud2 Current Load on cloud2 is WARNING: WARNING - load average: 21.34, 18.29, 17.27 [23:09:00] RECOVERY - cloud2 Current Load on cloud2 is OK: OK - load average: 15.55, 17.19, 17.00