[00:01:22] PROBLEM - cloud2 Current Load on cloud2 is CRITICAL: CRITICAL - load average: 31.88, 22.86, 22.77 [00:02:52] PROBLEM - cloud1 Current Load on cloud1 is WARNING: WARNING - load average: 22.53, 18.67, 14.67 [00:02:59] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 7 datacenters are down: 128.199.139.216/cpweb, 2400:6180:0:d0::403:f001/cpweb, 51.89.160.142/cpweb, 2001:41d0:800:1056::2/cpweb, 2001:41d0:800:105a::10/cpweb, 51.222.27.129/cpweb, 2607:5300:205:200::2ac4/cpweb [00:03:13] PROBLEM - allthetropes.orain.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for allthetropes.orain.org could not be found [00:03:14] PROBLEM - tl.awiki.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for tl.awiki.org could not be found [00:03:14] PROBLEM - indiancannabis.wiki - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for indiancannabis.wiki could not be found [00:03:14] PROBLEM - adadevelopersacademy.wiki - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for adadevelopersacademy.wiki could not be found [00:03:16] PROBLEM - wiki.climatechange.ai - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.climatechange.ai could not be found [00:04:36] PROBLEM - aman.awiki.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for aman.awiki.org could not be found [00:04:50] RECOVERY - cloud1 Current Load on cloud1 is OK: OK - load average: 17.80, 18.93, 15.28 [00:04:56] PROBLEM - mw6 Current Load on mw6 is WARNING: WARNING - load average: 7.13, 6.14, 5.78 [00:04:58] PROBLEM - mw4 Current Load on mw4 is WARNING: WARNING - load average: 7.05, 6.22, 5.31 [00:04:59] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [00:05:22] PROBLEM - cloud2 Current Load on cloud2 is WARNING: WARNING - load average: 22.26, 23.82, 23.34 [00:05:58] PROBLEM - wiki.parkcity.gay - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.parkcity.gay could not be found [00:06:58] RECOVERY - mw6 Current Load on mw6 is OK: OK - load average: 5.57, 6.17, 5.85 [00:06:59] RECOVERY - mw4 Current Load on mw4 is OK: OK - load average: 6.34, 6.13, 5.39 [00:07:22] PROBLEM - cloud2 Current Load on cloud2 is CRITICAL: CRITICAL - load average: 23.46, 24.34, 23.62 [00:09:58] RECOVERY - allthetropes.orain.org - reverse DNS on sslhost is OK: rDNS OK - allthetropes.orain.org reverse DNS resolves to cp7.miraheze.org [00:09:59] RECOVERY - tl.awiki.org - reverse DNS on sslhost is OK: rDNS OK - tl.awiki.org reverse DNS resolves to cp6.miraheze.org [00:10:00] RECOVERY - indiancannabis.wiki - reverse DNS on sslhost is OK: rDNS OK - indiancannabis.wiki reverse DNS resolves to cp7.miraheze.org [00:10:00] RECOVERY - adadevelopersacademy.wiki - reverse DNS on sslhost is OK: rDNS OK - adadevelopersacademy.wiki reverse DNS resolves to cp7.miraheze.org [00:10:04] RECOVERY - wiki.climatechange.ai - reverse DNS on sslhost is OK: rDNS OK - wiki.climatechange.ai reverse DNS resolves to cp7.miraheze.org [00:11:22] RECOVERY - aman.awiki.org - reverse DNS on sslhost is OK: rDNS OK - aman.awiki.org reverse DNS resolves to cp6.miraheze.org [00:11:23] PROBLEM - cloud2 Current Load on cloud2 is WARNING: WARNING - load average: 20.14, 22.65, 23.18 [00:12:41] RECOVERY - wiki.parkcity.gay - reverse DNS on sslhost is OK: rDNS OK - wiki.parkcity.gay reverse DNS resolves to cp7.miraheze.org [00:16:16] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-5 [+0/-0/±1] 13https://git.io/JIPC6 [00:16:18] [02miraheze/mw-config] 07Universal-Omega 03caec20d - Migrate $wgForeignFileRepos to MWS [00:16:19] [02mw-config] 07Universal-Omega created branch 03Universal-Omega-patch-5 - 13https://git.io/vbvb3 [00:16:21] [02mw-config] 07Universal-Omega opened pull request 03#3563: Migrate $wgForeignFileRepos to MWS - 13https://git.io/JIPCi [00:17:22] PROBLEM - cloud2 Current Load on cloud2 is CRITICAL: CRITICAL - load average: 25.27, 21.86, 22.29 [00:19:22] PROBLEM - cloud2 Current Load on cloud2 is WARNING: WARNING - load average: 22.90, 22.20, 22.37 [00:25:21] PROBLEM - cloud2 Current Load on cloud2 is CRITICAL: CRITICAL - load average: 25.47, 20.82, 21.38 [00:27:20] PROBLEM - cloud2 Current Load on cloud2 is WARNING: WARNING - load average: 15.24, 18.93, 20.64 [00:29:20] RECOVERY - cloud2 Current Load on cloud2 is OK: OK - load average: 19.10, 18.53, 20.24 [00:33:21] PROBLEM - cloud2 Current Load on cloud2 is WARNING: WARNING - load average: 22.37, 20.03, 20.43 [00:35:21] RECOVERY - cloud2 Current Load on cloud2 is OK: OK - load average: 16.13, 18.89, 20.00 [00:35:40] miraheze/mw-config/paladox-patch-1/5ef56f1 - paladox The build has errored. https://travis-ci.org/miraheze/mw-config/builds/749343492 [01:10:10] PROBLEM - cloud2 Current Load on cloud2 is WARNING: WARNING - load average: 20.65, 21.68, 20.03 [01:14:09] PROBLEM - cloud2 Current Load on cloud2 is CRITICAL: CRITICAL - load average: 27.48, 22.01, 20.36 [01:16:09] PROBLEM - cloud2 Current Load on cloud2 is WARNING: WARNING - load average: 23.63, 22.02, 20.54 [01:18:08] RECOVERY - cloud2 Current Load on cloud2 is OK: OK - load average: 18.60, 20.11, 19.98 [01:26:17] @System Administrators @Naleksuh has reported to me a fatal exception. The following is the error. Can someone do the trace? > MediaWiki internal error. > 15:12 Original exception: [a0030fe202c8d0ff3b0713d2] 2020-12-12 23:11:33: Fatal exception of type "ConfigException" > 15:12 Exception caught inside exception handler. > 15:12 Set $wgShowExceptionDetails = true; at the bottom of LocalSettings.php to show detailed debugging information. [01:26:50] What wiki, and what page? [01:27:12] oh MatomoAnalyticsForGetRequest [01:27:22] wait [01:27:26] that was at 11pm? [01:27:30] should have been fixed now [01:27:43] Not sure. Didn't say, I think we normally just trace the a0030fe202c8d0ff3b0713d2 error. [01:28:08] Oh if that is what this error is from, then yeah probably [01:28:13] thanks 🙂 [01:34:57] PROBLEM - mw4 Current Load on mw4 is WARNING: WARNING - load average: 6.82, 5.88, 4.73 [01:36:06] PROBLEM - cloud2 Current Load on cloud2 is WARNING: WARNING - load average: 20.21, 20.74, 20.02 [01:36:57] RECOVERY - mw4 Current Load on mw4 is OK: OK - load average: 3.87, 5.12, 4.59 [01:38:05] RECOVERY - cloud2 Current Load on cloud2 is OK: OK - load average: 17.00, 19.61, 19.70 [01:57:18] I put it in -testwiki for a reason [01:57:27] Granted I did not specify what page (recent changes) [01:57:36] Okay fine its just gonna ping me ever ytime [01:57:46] tbh discord is overrated from the beginning [02:02:03] paladox :fixed now [02:06:44] [02miraheze/WikiDiscover] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-2 [+0/-0/±1] 13https://git.io/JIP01 [02:06:46] [02miraheze/WikiDiscover] 07Universal-Omega 035ce42e4 - add hasSettingValue function [02:06:47] [02WikiDiscover] 07Universal-Omega created branch 03Universal-Omega-patch-2 - 13https://git.io/vhUAp [02:06:52] [02WikiDiscover] 07Universal-Omega opened pull request 03#27: add hasSettingValue function - 13https://git.io/JIP0M [02:18:20] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-5 [+0/-0/±1] 13https://git.io/JIPET [02:18:21] [02miraheze/mw-config] 07Universal-Omega 03ebdf850 - add SharedUpload variables [02:18:23] [02mw-config] 07Universal-Omega synchronize pull request 03#3563: Migrate $wgForeignFileRepos to MWS - 13https://git.io/JIPCi [02:21:22] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-5 [+0/-0/±1] 13https://git.io/JIPEs [02:21:24] [02miraheze/mw-config] 07Universal-Omega 035ee05ff - convert to wmg configs [02:21:25] [02mw-config] 07Universal-Omega synchronize pull request 03#3563: Migrate $wgForeignFileRepos to MWS - 13https://git.io/JIPCi [02:25:00] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JIPE0 [02:25:01] [02miraheze/mw-config] 07Universal-Omega 036cf1dd1 - update comment [02:35:47] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-5 [+0/-0/±1] 13https://git.io/JIPEA [02:35:49] [02miraheze/mw-config] 07Universal-Omega 033a0a7ce - add SharedUpload configs [02:35:50] [02mw-config] 07Universal-Omega synchronize pull request 03#3563: Migrate $wgForeignFileRepos to MWS - 13https://git.io/JIPCi [02:37:48] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-5 [+0/-0/±1] 13https://git.io/JIPuk [02:37:50] [02miraheze/mw-config] 07Universal-Omega 03d302c6f - add $wmgSharedUploadSubdomain [02:37:51] [02mw-config] 07Universal-Omega synchronize pull request 03#3563: Migrate $wgForeignFileRepos to MWS - 13https://git.io/JIPCi [02:48:56] @dmehus you have a sec sometime soon? [02:49:30] If so, DM me on Discord to tell me when (not available until monday) [03:38:13] Universal_Omega: around? I can confirm https://phabricator.miraheze.org/T6362 seems fixed on test2, but it's still broken on spcode.wiki :( [03:38:14] [ ⚓ T6362 Mobile.css / Mobile.js not loading since MW 1.35 upgrade ] - phabricator.miraheze.org [03:38:26] * spcodex.wiki [03:59:23] Okay, so what are you saying? [04:35:15] musikanimal: Make sure your mobile.js is exactly the same as your current minerva.js [04:35:30] and delete minerva.js [06:04:17] !log reception@jobrunner2:/srv/mediawiki/w/maintenance$ sudo -u www-data php deleteBatch.php --wiki awfulmovieswiki --r "Requested (T6586)" /home/reception/awfulmovies.txt [06:04:18] Reception123: 2020-12-12 - 22:56:21CET tell Reception123 Okay, cool, thanks. :) [06:04:20] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [06:12:23] !log reception@jobrunner1:/srv/mediawiki/w/maintenance$ sudo -u www-data php deleteBatch.php --wiki greatestmovieswiki --r "Requested (T6586)" /home/reception/greatestmovies.txt [06:12:25] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [07:39:06] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JI9ID [07:39:08] [02miraheze/mw-config] 07Universal-Omega 03be0bbb1 - fix typo [10:23:45] PROBLEM - wiki.zymonic.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.zymonic.com' expires in 15 day(s) (Tue 29 Dec 2020 10:14:56 GMT +0000). [10:26:36] PROBLEM - cloud2 Current Load on cloud2 is WARNING: WARNING - load average: 21.86, 19.28, 14.75 [10:28:37] RECOVERY - cloud2 Current Load on cloud2 is OK: OK - load average: 19.29, 19.04, 15.21 [10:29:52] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JI7ky [10:29:54] [02miraheze/ssl] 07MirahezeSSLBot 037e814f2 - Bot: Update SSL cert for wiki.zymonic.com [10:34:41] PROBLEM - cloud2 Current Load on cloud2 is CRITICAL: CRITICAL - load average: 26.15, 23.52, 18.27 [10:36:40] PROBLEM - cloud2 Current Load on cloud2 is WARNING: WARNING - load average: 22.53, 23.00, 18.70 [10:38:40] PROBLEM - cloud2 Current Load on cloud2 is CRITICAL: CRITICAL - load average: 28.19, 24.94, 19.92 [10:48:39] PROBLEM - cloud2 Current Load on cloud2 is WARNING: WARNING - load average: 11.60, 21.99, 22.29 [10:54:39] RECOVERY - cloud2 Current Load on cloud2 is OK: OK - load average: 11.92, 17.00, 20.14 [11:04:14] RECOVERY - wiki.zymonic.com - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.zymonic.com' will expire on Sat 13 Mar 2021 09:29:46 GMT +0000. [11:53:58] !log added snapwikiwiki discord webhook [11:54:05] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [11:56:40] !log added batfamilywiki discord webhook [11:56:43] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [15:12:16] [02puppet] 07Universal-Omega opened pull request 03#1554: add wikiplus-app.com to CSP whitelist (T6440) - 13https://git.io/JINIT [15:12:36] [02puppet] 07Universal-Omega closed pull request 03#1554: add wikiplus-app.com to CSP whitelist (T6440) - 13https://git.io/JINIT [15:17:14] [02puppet] 07Universal-Omega opened pull request 03#1555: add wikiplus-app.com to CSP whitelist (T6440) - 13https://git.io/JINOe [15:25:34] [02mw-config] 07MrJaroslavik opened pull request 03#3564: import sources for mrjaroslavikwiki - 13https://git.io/JIN8w [15:40:38] PROBLEM - vedopedia.witches-empire.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:51:38] [02puppet] 07Reception123 closed pull request 03#1555: add wikiplus-app.com to CSP whitelist (T6440) - 13https://git.io/JINOe [15:51:39] [02miraheze/puppet] 07Reception123 pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JIN74 [15:51:41] [02miraheze/puppet] 07Universal-Omega 03af9065b - add wikiplus-app.com (#1555) [15:54:26] RECOVERY - vedopedia.witches-empire.com - LetsEncrypt on sslhost is OK: OK - Certificate 'vedopedia.witches-empire.com' will expire on Sat 06 Feb 2021 23:38:39 GMT +0000. [15:55:11] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JINNn [15:55:12] [02miraheze/services] 07MirahezeSSLBot 037d2322c - BOT: Updating services config for wikis [16:07:00] PROBLEM - nonciclopedia.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for nonciclopedia.org could not be found [16:07:08] PROBLEM - iesd.mobagenie.my.id - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for iesd.mobagenie.my.id could not be found [16:07:10] PROBLEM - wiki.thehall.xyz - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.thehall.xyz could not be found [16:07:10] PROBLEM - pwiki.arkcls.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for pwiki.arkcls.com could not be found [16:11:42] PROBLEM - cloud2 Current Load on cloud2 is CRITICAL: CRITICAL - load average: 27.62, 24.24, 19.19 [16:13:58] RECOVERY - iesd.mobagenie.my.id - reverse DNS on sslhost is OK: rDNS OK - iesd.mobagenie.my.id reverse DNS resolves to cp6.miraheze.org [16:14:00] RECOVERY - nonciclopedia.org - reverse DNS on sslhost is OK: rDNS OK - nonciclopedia.org reverse DNS resolves to cp6.miraheze.org [16:14:02] RECOVERY - wiki.thehall.xyz - reverse DNS on sslhost is OK: rDNS OK - wiki.thehall.xyz reverse DNS resolves to cp6.miraheze.org [16:14:02] RECOVERY - pwiki.arkcls.com - reverse DNS on sslhost is OK: rDNS OK - pwiki.arkcls.com reverse DNS resolves to cp7.miraheze.org [16:19:40] PROBLEM - cloud2 Current Load on cloud2 is WARNING: WARNING - load average: 18.74, 21.89, 20.48 [16:23:41] PROBLEM - cloud2 Current Load on cloud2 is CRITICAL: CRITICAL - load average: 29.37, 24.88, 21.86 [16:25:41] PROBLEM - cloud2 Current Load on cloud2 is WARNING: WARNING - load average: 17.83, 22.81, 21.53 [16:31:41] RECOVERY - cloud2 Current Load on cloud2 is OK: OK - load average: 13.83, 17.90, 19.78 [16:51:00] [02miraheze/mediawiki] 07paladox pushed 031 commit to 03REL1_35 [+0/-0/±1] 13https://git.io/JIAyH [16:51:02] [02miraheze/mediawiki] 07paladox 03d41a41b - Update DataDump [16:51:58] [02miraheze/mediawiki] 07paladox pushed 031 commit to 03REL1_35 [+0/-0/±1] 13https://git.io/JIAS2 [16:52:00] [02miraheze/mediawiki] 07paladox 03fccf733 - Update MirahezeMagic [16:53:41] PROBLEM - cloud2 Current Load on cloud2 is WARNING: WARNING - load average: 20.57, 19.29, 18.73 [16:54:17] !log rebuild lc on test2 [16:54:38] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:55:41] PROBLEM - cloud2 Current Load on cloud2 is CRITICAL: CRITICAL - load average: 28.46, 23.23, 20.25 [16:57:41] PROBLEM - cloud2 Current Load on cloud2 is WARNING: WARNING - load average: 20.32, 21.61, 20.00 [16:58:32] [02miraheze/MirahezeMagic] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/JIA5J [16:58:34] [02miraheze/MirahezeMagic] 07paladox 03ee0c275 - Add managewiki-link-datadump i18n [16:58:35] [02MirahezeMagic] 07paladox created branch 03paladox-patch-1 - 13https://git.io/fQRGX [16:58:37] [02MirahezeMagic] 07paladox opened pull request 03#174: Add managewiki-link-datadump i18n - 13https://git.io/JIA5k [16:59:06] [02miraheze/MirahezeMagic] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/JIA5B [16:59:08] [02miraheze/MirahezeMagic] 07paladox 03f04259f - Update MirahezeMagicHooks.php [16:59:09] [02MirahezeMagic] 07paladox synchronize pull request 03#174: Add managewiki-link-datadump i18n - 13https://git.io/JIA5k [16:59:36] [02miraheze/MirahezeMagic] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/JIA5b [16:59:37] [02miraheze/MirahezeMagic] 07paladox 0328b1a78 - Update en.json [16:59:39] [02MirahezeMagic] 07paladox synchronize pull request 03#174: Add managewiki-link-datadump i18n - 13https://git.io/JIA5k [17:00:03] [02MirahezeMagic] 07paladox closed pull request 03#174: Add managewiki-link-datadump i18n - 13https://git.io/JIA5k [17:00:05] [02miraheze/MirahezeMagic] 07paladox pushed 034 commits to 03master [+0/-0/±5] 13https://git.io/JIAdI [17:00:06] [02miraheze/MirahezeMagic] 07paladox 03502e1dc - Merge pull request #174 from miraheze/paladox-patch-1 Add managewiki-link-datadump i18n [17:00:08] [02MirahezeMagic] 07paladox deleted branch 03paladox-patch-1 - 13https://git.io/fQRGX [17:00:09] [02miraheze/MirahezeMagic] 07paladox deleted branch 03paladox-patch-1 [17:00:34] [02miraheze/mediawiki] 07paladox pushed 031 commit to 03REL1_35 [+0/-0/±1] 13https://git.io/JIAdz [17:00:35] [02miraheze/mediawiki] 07paladox 037026ea9 - Update MirahezeMagic [17:01:22] !log rebuild lc on mw* and jobrunner* [17:01:29] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:03:40] PROBLEM - cloud2 Current Load on cloud2 is CRITICAL: CRITICAL - load average: 42.27, 27.78, 22.52 [17:04:30] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 6 datacenters are down: 128.199.139.216/cpweb, 2400:6180:0:d0::403:f001/cpweb, 2001:41d0:800:1056::2/cpweb, 2001:41d0:800:105a::10/cpweb, 51.222.27.129/cpweb, 2607:5300:205:200::2ac4/cpweb [17:04:45] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 2 datacenters are down: 128.199.139.216/cpweb, 51.222.27.129/cpweb [17:06:46] PROBLEM - mw7 Current Load on mw7 is WARNING: WARNING - load average: 7.97, 6.84, 5.59 [17:06:46] [02miraheze/MirahezeMagic] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/JIApp [17:06:48] [02miraheze/MirahezeMagic] 07paladox 035ffd420 - Fix i18n [17:06:49] [02MirahezeMagic] 07paladox created branch 03paladox-patch-1 - 13https://git.io/fQRGX [17:06:51] [02MirahezeMagic] 07paladox opened pull request 03#175: Fix i18n - 13https://git.io/JIAhJ [17:07:08] PROBLEM - mw4 Current Load on mw4 is CRITICAL: CRITICAL - load average: 8.04, 6.41, 5.19 [17:07:25] [02miraheze/MirahezeMagic] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/JIAhF [17:07:27] [02miraheze/MirahezeMagic] 07paladox 03b30bb1b - Update en.json [17:07:28] [02MirahezeMagic] 07paladox synchronize pull request 03#175: Fix i18n - 13https://git.io/JIAhJ [17:07:46] [02miraheze/MirahezeMagic] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/JIAjm [17:07:48] [02miraheze/MirahezeMagic] 07paladox 03c900cd3 - Update MirahezeMagicHooks.php [17:07:49] [02MirahezeMagic] 07paladox synchronize pull request 03#175: Fix i18n - 13https://git.io/JIAhJ [17:08:02] [02MirahezeMagic] 07paladox closed pull request 03#175: Fix i18n - 13https://git.io/JIAhJ [17:08:04] [02miraheze/MirahezeMagic] 07paladox pushed 034 commits to 03master [+0/-0/±6] 13https://git.io/JIAjl [17:08:05] [02miraheze/MirahezeMagic] 07paladox 0394df54b - Merge pull request #175 from miraheze/paladox-patch-1 Fix i18n [17:08:07] [02MirahezeMagic] 07paladox deleted branch 03paladox-patch-1 - 13https://git.io/fQRGX [17:08:08] [02miraheze/MirahezeMagic] 07paladox deleted branch 03paladox-patch-1 [17:08:10] @Reception123 I need a phab task created [17:08:45] PROBLEM - mw7 Current Load on mw7 is CRITICAL: CRITICAL - load average: 8.65, 7.49, 5.98 [17:08:50] [02miraheze/mediawiki] 07paladox pushed 031 commit to 03REL1_35 [+0/-0/±1] 13https://git.io/JIxeO [17:08:52] [02miraheze/mediawiki] 07paladox 0309da8f9 - Update MirahezeMagic [17:09:17] PROBLEM - mw7 MediaWiki Rendering on mw7 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:09:34] PROBLEM - gluster1 Current Load on gluster1 is CRITICAL: CRITICAL - load average: 8.37, 6.82, 5.19 [17:10:00] PROBLEM - cp3 Stunnel Http for mw6 on cp3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [17:10:12] PROBLEM - cp9 Stunnel Http for mw6 on cp9 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [17:10:17] PROBLEM - cp7 Stunnel Http for mw6 on cp7 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [17:10:22] PROBLEM - mw5 Current Load on mw5 is WARNING: WARNING - load average: 7.24, 6.90, 5.49 [17:10:53] PROBLEM - mw7 Current Load on mw7 is WARNING: WARNING - load average: 7.99, 7.31, 6.09 [17:11:17] PROBLEM - mw4 Current Load on mw4 is WARNING: WARNING - load average: 7.71, 7.02, 5.72 [17:11:35] PROBLEM - gluster1 Current Load on gluster1 is WARNING: WARNING - load average: 6.57, 6.82, 5.40 [17:12:02] PROBLEM - cp6 Stunnel Http for mw7 on cp6 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [17:12:10] RECOVERY - cp3 Stunnel Http for mw6 on cp3 is OK: HTTP OK: HTTP/1.1 200 OK - 15955 bytes in 5.634 second response time [17:12:23] RECOVERY - cp7 Stunnel Http for mw6 on cp7 is OK: HTTP OK: HTTP/1.1 200 OK - 15955 bytes in 7.391 second response time [17:12:24] RECOVERY - cp9 Stunnel Http for mw6 on cp9 is OK: HTTP OK: HTTP/1.1 200 OK - 15955 bytes in 9.304 second response time [17:12:34] PROBLEM - cp7 Stunnel Http for mw7 on cp7 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [17:12:44] PROBLEM - cp6 Stunnel Http for mw4 on cp6 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [17:12:54] PROBLEM - mw7 Current Load on mw7 is CRITICAL: CRITICAL - load average: 9.69, 7.97, 6.47 [17:12:57] PROBLEM - cp3 Stunnel Http for mw4 on cp3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [17:13:19] PROBLEM - mw4 Current Load on mw4 is CRITICAL: CRITICAL - load average: 8.63, 7.49, 6.04 [17:13:31] PROBLEM - cp3 Stunnel Http for mw7 on cp3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [17:13:36] RECOVERY - gluster1 Current Load on gluster1 is OK: OK - load average: 6.67, 6.74, 5.55 [17:14:10] PROBLEM - cp9 Stunnel Http for mw7 on cp9 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [17:14:23] PROBLEM - mw5 Current Load on mw5 is CRITICAL: CRITICAL - load average: 9.21, 7.99, 6.28 [17:14:46] RECOVERY - cp6 Stunnel Http for mw4 on cp6 is OK: HTTP OK: HTTP/1.1 200 OK - 15955 bytes in 0.921 second response time [17:15:04] RECOVERY - cp3 Stunnel Http for mw4 on cp3 is OK: HTTP OK: HTTP/1.1 200 OK - 15955 bytes in 5.547 second response time [17:15:05] PROBLEM - mw6 Current Load on mw6 is CRITICAL: CRITICAL - load average: 9.45, 8.10, 6.47 [17:15:13] PROBLEM - cp3 Varnish Backends on cp3 is CRITICAL: 1 backends are down. mw7 [17:15:33] RECOVERY - cp3 Stunnel Http for mw7 on cp3 is OK: HTTP OK: HTTP/1.1 200 OK - 15955 bytes in 1.046 second response time [17:15:38] RECOVERY - mw7 MediaWiki Rendering on mw7 is OK: HTTP OK: HTTP/1.1 200 OK - 20047 bytes in 0.813 second response time [17:16:13] RECOVERY - cp9 Stunnel Http for mw7 on cp9 is OK: HTTP OK: HTTP/1.1 200 OK - 15955 bytes in 0.349 second response time [17:16:13] RECOVERY - cp6 Stunnel Http for mw7 on cp6 is OK: HTTP OK: HTTP/1.1 200 OK - 15955 bytes in 0.066 second response time [17:16:34] PROBLEM - holonet.pw - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for holonet.pw could not be found [17:16:34] PROBLEM - www.pornwiki.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for www.pornwiki.org could not be found [17:16:36] PROBLEM - wiki.dom64.xyz - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.dom64.xyz could not be found [17:16:37] PROBLEM - cp3 Stunnel Http for mw6 on cp3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [17:16:37] PROBLEM - madgenderscience.wiki - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for madgenderscience.wiki could not be found [17:16:39] RECOVERY - cp7 Stunnel Http for mw7 on cp7 is OK: HTTP OK: HTTP/1.1 200 OK - 15955 bytes in 1.867 second response time [17:16:39] PROBLEM - wikipuk.cl - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wikipuk.cl could not be found [17:16:41] PROBLEM - www.allthetropes.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for www.allthetropes.org could not be found [17:16:42] PROBLEM - sims.miraheze.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for sims.miraheze.org could not be found [17:16:42] PROBLEM - erikapedia.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for erikapedia.com could not be found [17:16:42] PROBLEM - wiki.autocountsoft.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.autocountsoft.com could not be found [17:16:43] PROBLEM - www.wikiescola.com.br - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for www.wikiescola.com.br could not be found [17:16:46] PROBLEM - lg.gyaanipedia.co.in - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for lg.gyaanipedia.co.in could not be found [17:16:47] PROBLEM - wiki.ostans.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.ostans.com could not be found [17:16:47] PROBLEM - vitriol.spoon.army - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for vitriol.spoon.army could not be found [17:16:49] PROBLEM - cp9 Stunnel Http for mw6 on cp9 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [17:16:52] PROBLEM - wiki.worldsofweary.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.worldsofweary.com could not be found [17:16:53] PROBLEM - cp7 Stunnel Http for mw6 on cp7 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [17:16:54] !ops cp9 is extremely latent / sluggish / poorly performing. Can we possibly look at something, maybe depooling and repooling the cache proxies, or changing the way traffic is managed, redirecting users to the least used cache proxy rather than their geographically closest one? In absence of a new NA cache proxy, something needs to be done [17:16:55] * SigmaBotMH yelps for help [17:17:00] PROBLEM - cp6 Stunnel Http for mw6 on cp6 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [17:17:21] PROBLEM - mw6 MediaWiki Rendering on mw6 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:17:27] dmehus: afaik, users should be directed to the closest one already if the current one doesn't work [17:17:31] but I'm not too sure about that process [17:17:46] PROBLEM - spcodex.wiki - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for spcodex.wiki could not be found [17:17:48] PROBLEM - wiki.arru.xyz - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.arru.xyz could not be found [17:17:52] PROBLEM - mh142.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for mh142.com could not be found [17:17:53] PROBLEM - duepedia.uk.to - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for duepedia.uk.to could not be found [17:17:54] PROBLEM - wiki.qadrishattari.xyz - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.qadrishattari.xyz could not be found [17:17:56] PROBLEM - www.iceria.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for www.iceria.org could not be found [17:17:56] PROBLEM - warmage.necrocity.co.uk - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for warmage.necrocity.co.uk could not be found [17:17:56] PROBLEM - wiki.sketch.town - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.sketch.town could not be found [17:18:03] but if you mean redirecting everyone away from cp9, then that would be problematic and wouldn't work [17:18:11] @Reception123 yeah, that would be good if so. I'll ping @SPF|Cloud if he's around to take a look at the recent traffic analysis. [17:18:15] PROBLEM - cloud1 Current Load on cloud1 is WARNING: WARNING - load average: 22.04, 21.81, 18.13 [17:18:25] dmehus: yeah good idea, SPF would be the one to know what to do :) [17:18:28] @Reception123 Not away from cp9, just changing the way our traffic is managed in some way [17:19:19] RECOVERY - cp3 Varnish Backends on cp3 is OK: All 7 backends are healthy [17:19:37] PROBLEM - dc-multiverse.dcwikis.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for dc-multiverse.dcwikis.com could not be found [17:19:38] PROBLEM - pastport.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for pastport.org could not be found [17:19:53] PROBLEM - n64brew.dev - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for n64brew.dev could not be found [17:19:58] PROBLEM - wiki.erisly.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.erisly.com could not be found [17:19:59] PROBLEM - arrowverse.miraheze.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for arrowverse.miraheze.org could not be found [17:20:00] PROBLEM - encyclopediaofastrobiology.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for encyclopediaofastrobiology.org could not be found [17:20:12] PROBLEM - cloud1 Current Load on cloud1 is CRITICAL: CRITICAL - load average: 25.30, 22.34, 18.70 [17:22:09] PROBLEM - guia.cineastas.pt - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for guia.cineastas.pt could not be found [17:22:11] RECOVERY - cloud1 Current Load on cloud1 is OK: OK - load average: 15.52, 19.58, 18.13 [17:22:17] PROBLEM - wiki.danvs.net - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.danvs.net could not be found [17:23:26] RECOVERY - erikapedia.com - reverse DNS on sslhost is OK: rDNS OK - erikapedia.com reverse DNS resolves to cp9.miraheze.org [17:23:29] RECOVERY - holonet.pw - reverse DNS on sslhost is OK: rDNS OK - holonet.pw reverse DNS resolves to cp9.miraheze.org [17:23:30] RECOVERY - www.pornwiki.org - reverse DNS on sslhost is OK: rDNS OK - www.pornwiki.org reverse DNS resolves to cp9.miraheze.org [17:23:31] RECOVERY - vitriol.spoon.army - reverse DNS on sslhost is OK: rDNS OK - vitriol.spoon.army reverse DNS resolves to cp9.miraheze.org [17:23:32] RECOVERY - wiki.dom64.xyz - reverse DNS on sslhost is OK: rDNS OK - wiki.dom64.xyz reverse DNS resolves to cp6.miraheze.org [17:23:35] RECOVERY - madgenderscience.wiki - reverse DNS on sslhost is OK: rDNS OK - madgenderscience.wiki reverse DNS resolves to cp6.miraheze.org [17:23:36] RECOVERY - wikipuk.cl - reverse DNS on sslhost is OK: rDNS OK - wikipuk.cl reverse DNS resolves to cp6.miraheze.org [17:23:36] RECOVERY - wiki.worldsofweary.com - reverse DNS on sslhost is OK: rDNS OK - wiki.worldsofweary.com reverse DNS resolves to cp6.miraheze.org [17:23:41] RECOVERY - sims.miraheze.org - reverse DNS on sslhost is OK: rDNS OK - sims.miraheze.org reverse DNS resolves to cp7.miraheze.org [17:23:41] RECOVERY - www.allthetropes.org - reverse DNS on sslhost is OK: rDNS OK - www.allthetropes.org reverse DNS resolves to cp7.miraheze.org [17:23:43] RECOVERY - www.wikiescola.com.br - reverse DNS on sslhost is OK: rDNS OK - www.wikiescola.com.br reverse DNS resolves to cp6.miraheze.org [17:23:45] RECOVERY - wiki.autocountsoft.com - reverse DNS on sslhost is OK: rDNS OK - wiki.autocountsoft.com reverse DNS resolves to cp7.miraheze.org [17:23:49] RECOVERY - lg.gyaanipedia.co.in - reverse DNS on sslhost is OK: rDNS OK - lg.gyaanipedia.co.in reverse DNS resolves to cp7.miraheze.org [17:23:50] RECOVERY - wiki.ostans.com - reverse DNS on sslhost is OK: rDNS OK - wiki.ostans.com reverse DNS resolves to cp7.miraheze.org [17:23:52] PROBLEM - wiki.arnab.at - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.arnab.at could not be found [17:23:55] PROBLEM - hellointernet.miraheze.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for hellointernet.miraheze.org could not be found [17:23:58] PROBLEM - dc.miraheze.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for dc.miraheze.org could not be found [17:24:02] PROBLEM - wiki.usagihime.ml - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.usagihime.ml could not be found [17:24:02] PROBLEM - monarchists.wiki - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for monarchists.wiki could not be found [17:24:18] PROBLEM - innersphere.tk - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for innersphere.tk could not be found [17:24:22] PROBLEM - wiki.worlduniversityandschool.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.worlduniversityandschool.org could not be found [17:24:24] PROBLEM - poserdazfreebies.orain.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for poserdazfreebies.orain.org could not be found [17:24:30] PROBLEM - pwiki.arkcls.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for pwiki.arkcls.com could not be found [17:24:30] RECOVERY - spcodex.wiki - reverse DNS on sslhost is OK: rDNS OK - spcodex.wiki reverse DNS resolves to cp6.miraheze.org [17:24:32] PROBLEM - gluster1 Current Load on gluster1 is CRITICAL: CRITICAL - load average: 8.10, 7.13, 6.26 [17:24:37] RECOVERY - wiki.arru.xyz - reverse DNS on sslhost is OK: rDNS OK - wiki.arru.xyz reverse DNS resolves to cp6.miraheze.org [17:24:40] RECOVERY - mh142.com - reverse DNS on sslhost is OK: rDNS OK - mh142.com reverse DNS resolves to cp6.miraheze.org [17:24:44] RECOVERY - duepedia.uk.to - reverse DNS on sslhost is OK: rDNS OK - duepedia.uk.to reverse DNS resolves to cp9.miraheze.org [17:24:44] RECOVERY - wiki.qadrishattari.xyz - reverse DNS on sslhost is OK: rDNS OK - wiki.qadrishattari.xyz reverse DNS resolves to cp9.miraheze.org [17:24:51] RECOVERY - www.iceria.org - reverse DNS on sslhost is OK: rDNS OK - www.iceria.org reverse DNS resolves to cp9.miraheze.org [17:24:52] PROBLEM - cp9 Stunnel Http for mw7 on cp9 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [17:24:53] RECOVERY - warmage.necrocity.co.uk - reverse DNS on sslhost is OK: rDNS OK - warmage.necrocity.co.uk reverse DNS resolves to cp9.miraheze.org [17:24:53] RECOVERY - wiki.sketch.town - reverse DNS on sslhost is OK: rDNS OK - wiki.sketch.town reverse DNS resolves to cp9.miraheze.org [17:25:06] PROBLEM - cp7 Stunnel Http for mw7 on cp7 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [17:26:22] RECOVERY - pastport.org - reverse DNS on sslhost is OK: rDNS OK - pastport.org reverse DNS resolves to cp7.miraheze.org [17:26:23] RECOVERY - dc-multiverse.dcwikis.com - reverse DNS on sslhost is OK: rDNS OK - dc-multiverse.dcwikis.com reverse DNS resolves to cp7.miraheze.org [17:26:29] RECOVERY - gluster1 Current Load on gluster1 is OK: OK - load average: 5.61, 6.65, 6.20 [17:26:31] PROBLEM - mw5 Current Load on mw5 is WARNING: WARNING - load average: 5.12, 7.69, 7.42 [17:26:37] RECOVERY - n64brew.dev - reverse DNS on sslhost is OK: rDNS OK - n64brew.dev reverse DNS resolves to cp6.miraheze.org [17:26:42] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [17:26:46] RECOVERY - wiki.erisly.com - reverse DNS on sslhost is OK: rDNS OK - wiki.erisly.com reverse DNS resolves to cp7.miraheze.org [17:26:49] RECOVERY - arrowverse.miraheze.org - reverse DNS on sslhost is OK: rDNS OK - arrowverse.miraheze.org reverse DNS resolves to cp6.miraheze.org [17:26:50] RECOVERY - encyclopediaofastrobiology.org - reverse DNS on sslhost is OK: rDNS OK - encyclopediaofastrobiology.org reverse DNS resolves to cp6.miraheze.org [17:26:52] RECOVERY - cp9 Stunnel Http for mw7 on cp9 is OK: HTTP OK: HTTP/1.1 200 OK - 15955 bytes in 0.386 second response time [17:27:02] RECOVERY - cp7 Stunnel Http for mw7 on cp7 is OK: HTTP OK: HTTP/1.1 200 OK - 15955 bytes in 0.082 second response time [17:27:02] PROBLEM - db7 Check MariaDB Replication on db7 is CRITICAL: MariaDB replication - both - CRITICAL - Slave_IO_Running state : Yes, Slave_SQL_Running state : Yes, Seconds_Behind_Master : 128s [17:27:14] PROBLEM - mw4 Current Load on mw4 is WARNING: WARNING - load average: 5.79, 7.27, 7.08 [17:27:19] RECOVERY - cp9 Stunnel Http for mw6 on cp9 is OK: HTTP OK: HTTP/1.1 200 OK - 15955 bytes in 7.011 second response time [17:27:28] RECOVERY - cp7 Stunnel Http for mw6 on cp7 is OK: HTTP OK: HTTP/1.1 200 OK - 15955 bytes in 8.196 second response time [17:27:47] RECOVERY - cp6 Stunnel Http for mw6 on cp6 is OK: HTTP OK: HTTP/1.1 200 OK - 15969 bytes in 9.339 second response time [17:27:55] PROBLEM - meta.orain.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for meta.orain.org could not be found [17:27:56] PROBLEM - wiki.gesamtschule-nordkirchen.de - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.gesamtschule-nordkirchen.de could not be found [17:27:56] PROBLEM - wiki.ricochetuniverse.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.ricochetuniverse.com could not be found [17:28:10] RECOVERY - mw6 MediaWiki Rendering on mw6 is OK: HTTP OK: HTTP/1.1 200 OK - 20047 bytes in 8.130 second response time [17:28:41] @Doug check your Discord DM [17:28:48] Then I'll explain [17:28:58] RECOVERY - guia.cineastas.pt - reverse DNS on sslhost is OK: rDNS OK - guia.cineastas.pt reverse DNS resolves to cp7.miraheze.org [17:29:02] PROBLEM - db7 Check MariaDB Replication on db7 is WARNING: MariaDB replication - both - WARNING - Slave_IO_Running state : Yes, Slave_SQL_Running state : Yes, Seconds_Behind_Master : 12s [17:29:11] RECOVERY - mw4 Current Load on mw4 is OK: OK - load average: 4.42, 6.35, 6.77 [17:29:13] RECOVERY - wiki.danvs.net - reverse DNS on sslhost is OK: rDNS OK - wiki.danvs.net reverse DNS resolves to cp7.miraheze.org [17:30:38] RECOVERY - wiki.arnab.at - reverse DNS on sslhost is OK: rDNS OK - wiki.arnab.at reverse DNS resolves to cp7.miraheze.org [17:30:39] RECOVERY - hellointernet.miraheze.org - reverse DNS on sslhost is OK: rDNS OK - hellointernet.miraheze.org reverse DNS resolves to cp7.miraheze.org [17:30:44] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 8 datacenters are down: 128.199.139.216/cpweb, 2400:6180:0:d0::403:f001/cpweb, 51.77.107.210/cpweb, 51.89.160.142/cpweb, 2001:41d0:800:1056::2/cpweb, 2001:41d0:800:105a::10/cpweb, 51.222.27.129/cpweb, 2607:5300:205:200::2ac4/cpweb [17:30:45] RECOVERY - dc.miraheze.org - reverse DNS on sslhost is OK: rDNS OK - dc.miraheze.org reverse DNS resolves to cp6.miraheze.org [17:30:52] RECOVERY - monarchists.wiki - reverse DNS on sslhost is OK: rDNS OK - monarchists.wiki reverse DNS resolves to cp6.miraheze.org [17:30:54] RECOVERY - wiki.usagihime.ml - reverse DNS on sslhost is OK: rDNS OK - wiki.usagihime.ml reverse DNS resolves to cp6.miraheze.org [17:31:02] RECOVERY - db7 Check MariaDB Replication on db7 is OK: MariaDB replication - both - OK - Slave_IO_Running state : Yes, Slave_SQL_Running state : Yes, Seconds_Behind_Master : 1s [17:31:05] RECOVERY - innersphere.tk - reverse DNS on sslhost is OK: rDNS OK - innersphere.tk reverse DNS resolves to cp7.miraheze.org [17:31:06] @RhinosF1 Will do as soon as I can, thanks...am trying to post on local bureaucrat's talk page, but it's so bad, I can't even do anything on any Miraheze wikis. [17:31:20] RECOVERY - wiki.worlduniversityandschool.org - reverse DNS on sslhost is OK: rDNS OK - wiki.worlduniversityandschool.org reverse DNS resolves to cp7.miraheze.org [17:31:20] RECOVERY - poserdazfreebies.orain.org - reverse DNS on sslhost is OK: rDNS OK - poserdazfreebies.orain.org reverse DNS resolves to cp7.miraheze.org [17:31:23] PROBLEM - cp9 Stunnel Http for mw7 on cp9 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [17:31:23] RECOVERY - pwiki.arkcls.com - reverse DNS on sslhost is OK: rDNS OK - pwiki.arkcls.com reverse DNS resolves to cp6.miraheze.org [17:31:27] miraheze/MirahezeMagic/paladox-patch-1/c900cd3 - paladox The build has errored. https://travis-ci.org/miraheze/MirahezeMagic/builds/749449784 [17:31:29] @dmehus I can't access half of miraheze [17:31:40] RECOVERY - cp3 Stunnel Http for mw6 on cp3 is OK: HTTP OK: HTTP/1.1 200 OK - 15955 bytes in 8.684 second response time [17:31:51] @blackwidowmovie0 okay, thank you for the added information, that's helpful [17:32:06] is that sarcastic, or truthful? [17:32:12] it's kinda hard to tell over messge [17:32:18] *message [17:32:21] @blackwidowmovie0 not sarcasm [17:32:23] PROBLEM - gluster1 Current Load on gluster1 is CRITICAL: CRITICAL - load average: 8.10, 7.61, 6.70 [17:32:28] RECOVERY - mw5 Current Load on mw5 is OK: OK - load average: 5.27, 6.14, 6.74 [17:32:28] ok 👍🏽 [17:32:35] mcuwiki is taking a while to load the editor [17:32:38] like 30 seconds [17:33:06] [02miraheze/CreateWiki] 07paladox pushed 031 commit to 03paladox-patch-3 [+0/-0/±1] 13https://git.io/JIx0J [17:33:08] [02miraheze/CreateWiki] 07paladox 0332f01e7 - changeDBCluster: Add new --wiki-dbname option In cases where the wiki in question is using the wrong db cluster, using this script will fail. So lets fix this by using for example loginwiki where things won't be broken. [17:33:09] [02CreateWiki] 07paladox created branch 03paladox-patch-3 - 13https://git.io/vpJTL [17:33:11] [02CreateWiki] 07paladox opened pull request 03#177: changeDBCluster: Add new --wiki-dbname option - 13https://git.io/JIx0I [17:33:25] RECOVERY - cp9 Stunnel Http for mw7 on cp9 is OK: HTTP OK: HTTP/1.1 200 OK - 15955 bytes in 5.997 second response time [17:33:39] yeah, I might have to try disabling VE in my [[Special:GlobalPreferences]], but doubt it makes a difference as some wikis don't have it enabled and sluggishness is the same [17:33:48] PROBLEM - cp3 Stunnel Http for mw7 on cp3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [17:34:10] PROBLEM - cp6 Stunnel Http for mw7 on cp6 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [17:34:46] RECOVERY - meta.orain.org - reverse DNS on sslhost is OK: rDNS OK - meta.orain.org reverse DNS resolves to cp6.miraheze.org [17:34:47] RECOVERY - wiki.gesamtschule-nordkirchen.de - reverse DNS on sslhost is OK: rDNS OK - wiki.gesamtschule-nordkirchen.de reverse DNS resolves to cp6.miraheze.org [17:34:50] RECOVERY - wiki.ricochetuniverse.com - reverse DNS on sslhost is OK: rDNS OK - wiki.ricochetuniverse.com reverse DNS resolves to cp7.miraheze.org [17:35:41] @dmehus VE is disabled for me globally [17:35:49] RECOVERY - cp3 Stunnel Http for mw7 on cp3 is OK: HTTP OK: HTTP/1.1 200 OK - 15955 bytes in 0.997 second response time [17:36:09] I have also got an XML for someone to download, if a sysadmin has the chance [17:36:11] PROBLEM - cp3 Stunnel Http for mw6 on cp3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [17:36:29] * blackwidowmovie0 requests sysadmin help, and isn't receiving any...... [17:36:51] PROBLEM - mw6 MediaWiki Rendering on mw6 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:36:55] blackwidowmovie0: as you've been told before, please be patient. you can't report three different errors at once and expect an answer [17:37:00] PROBLEM - cp9 Stunnel Http for mw6 on cp9 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [17:37:12] you can email the XML to tech@ [17:37:16] ok [17:37:20] PROBLEM - marthaspeaks.miraheze.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for marthaspeaks.miraheze.org could not be found [17:37:22] it's for starwarswiki [17:37:24] PROBLEM - towarzystwo.prawa.kredytowego.pl - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for towarzystwo.prawa.kredytowego.pl could not be found [17:37:25] PROBLEM - meta.gyaanipedia.co.in - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for meta.gyaanipedia.co.in could not be found [17:37:25] PROBLEM - guia.esporo.net - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for guia.esporo.net could not be found [17:37:26] PROBLEM - wiki.minecraftathome.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.minecraftathome.com could not be found [17:37:49] PROBLEM - miraheze.gq - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for miraheze.gq could not be found [17:38:15] RECOVERY - cp6 Stunnel Http for mw7 on cp6 is OK: HTTP OK: HTTP/1.1 200 OK - 15955 bytes in 0.028 second response time [17:38:25] PROBLEM - gluster1 Current Load on gluster1 is WARNING: WARNING - load average: 5.60, 7.29, 6.92 [17:38:59] RECOVERY - mw6 MediaWiki Rendering on mw6 is OK: HTTP OK: HTTP/1.1 200 OK - 20047 bytes in 9.405 second response time [17:39:00] RECOVERY - cp9 Stunnel Http for mw6 on cp9 is OK: HTTP OK: HTTP/1.1 200 OK - 15955 bytes in 4.375 second response time [17:39:15] !log root@jobrunner1:/srv/mediawiki/w/maintenance# /usr/local/bin/foreachwikiindblist /srv/mediawiki/w/cache/databases.json /srv/mediawiki/w/maintenance/rebuildmessages.php [17:39:33] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:40:18] RECOVERY - cp3 Stunnel Http for mw6 on cp3 is OK: HTTP OK: HTTP/1.1 200 OK - 15955 bytes in 1.733 second response time [17:40:22] RECOVERY - gluster1 Current Load on gluster1 is OK: OK - load average: 4.51, 6.34, 6.61 [17:40:42] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [17:40:59] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [17:41:57] [02miraheze/mw-config] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/JIxVc [17:41:58]