[00:00:20] [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fjVxU [00:00:21] [02miraheze/mw-config] 07paladox 03cf28b43 - Update RottenLinksCurlTimeout to 10s in MW [00:09:22] !log update cw_wikis set wiki_settings = REPLACE(wiki_settings, '"wgRottenLinksCurlTimeout":"30"', '"wgRottenLinksCurlTimeout":"10"'); [00:09:28] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [00:14:30] !log depool mw3 [00:14:34] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [00:18:03] !log stopped nginx on mw3 && disabled puppet [00:18:07] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [00:19:01] PROBLEM - cp2 Varnish Backends on cp2 is CRITICAL: 1 backends are down. mw3 [00:19:32] PROBLEM - cp3 Varnish Backends on cp3 is CRITICAL: 1 backends are down. mw3 [00:20:21] PROBLEM - mw3 HTTPS on mw3 is CRITICAL: connect to address 81.4.121.113 and port 443: Connection refusedHTTP CRITICAL - Unable to open TCP socket [00:21:03] PROBLEM - cp4 Varnish Backends on cp4 is CRITICAL: 1 backends are down. mw3 [00:21:42] PROBLEM - mw3 Puppet on mw3 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 9 minutes ago with 0 failures [00:31:07] !log reboot mw3 (as there is a proccess in D state, but it's not shown long enough for me to kill it) [00:31:14] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [00:31:52] PROBLEM - mw3 Puppet on mw3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:34:01] PROBLEM - mw3 JobRunner Service on mw3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:34:26] PROBLEM - mw3 Current Load on mw3 is CRITICAL: connect to address 81.4.121.113 port 5666: Connection refusedconnect to host 81.4.121.113 port 5666: Connection refused [00:36:52] !log upgrade firefox-esr on mw3 [00:36:53] RECOVERY - cp2 Varnish Backends on cp2 is OK: All 5 backends are healthy [00:36:58] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [00:37:03] RECOVERY - cp4 Varnish Backends on cp4 is OK: All 5 backends are healthy [00:37:23] !log repool mw3 [00:37:27] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [00:37:32] RECOVERY - cp3 Varnish Backends on cp3 is OK: All 5 backends are healthy [01:52:17] Hello g! If you have any questions feel free to ask and someone should answer soon. [08:20:12] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fjweX [08:20:14] [02miraheze/services] 07MirahezeSSLBot 03fb29220 - BOT: Updating services config for wikis [08:25:11] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fjwvv [08:25:13] [02miraheze/services] 07MirahezeSSLBot 036d06fce - BOT: Updating services config for wikis [08:35:20] PROBLEM - cp3 Disk Space on cp3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [08:35:32] PROBLEM - cp3 Puppet on cp3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [08:35:40] PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 1 datacenter is down: 128.199.139.216/cpweb [08:35:57] PROBLEM - cp3 Varnish Backends on cp3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [08:37:17] RECOVERY - cp3 Disk Space on cp3 is OK: DISK OK - free space: / 5401 MB (22% inode=94%); [08:37:40] RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online [08:37:52] RECOVERY - cp3 Varnish Backends on cp3 is OK: All 5 backends are healthy [08:45:35] RECOVERY - cp3 Puppet on cp3 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [10:38:09] PROBLEM - wiki.galactee.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.galactee.org' expires in 15 day(s) (Sun 07 Jul 2019 10:36:02 AM GMT +0000). [10:38:25] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fjwJj [10:38:26] [02miraheze/ssl] 07MirahezeSSLBot 035bc5917 - Bot: Update SSL cert for wiki.galactee.org [10:44:09] RECOVERY - wiki.galactee.org - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.galactee.org' will expire on Thu 19 Sep 2019 09:38:18 AM GMT +0000. [11:41:49] PROBLEM - thesimswiki.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'www.thesimswiki.com' expires in 15 day(s) (Sun 07 Jul 2019 11:39:42 AM GMT +0000). [11:42:28] PROBLEM - wiki.thesimswiki.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'www.thesimswiki.com' expires in 15 day(s) (Sun 07 Jul 2019 11:39:42 AM GMT +0000). [11:43:03] PROBLEM - www.thesimswiki.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'www.thesimswiki.com' expires in 15 day(s) (Sun 07 Jul 2019 11:39:42 AM GMT +0000). [11:43:20] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fjwT4 [11:43:21] [02miraheze/ssl] 07MirahezeSSLBot 0322d03a2 - Bot: Update SSL cert for www.thesimswiki.com [11:53:03] RECOVERY - www.thesimswiki.com - LetsEncrypt on sslhost is OK: OK - Certificate 'www.thesimswiki.com' will expire on Thu 19 Sep 2019 10:43:13 AM GMT +0000. [11:53:49] RECOVERY - thesimswiki.com - LetsEncrypt on sslhost is OK: OK - Certificate 'www.thesimswiki.com' will expire on Thu 19 Sep 2019 10:43:13 AM GMT +0000. [11:54:28] RECOVERY - wiki.thesimswiki.com - LetsEncrypt on sslhost is OK: OK - Certificate 'www.thesimswiki.com' will expire on Thu 19 Sep 2019 10:43:13 AM GMT +0000. [12:48:18] https://meta.miraheze.org/wiki/Community_noticeboard#How_do_I_add_other_languages_to_my_wiki.3F Can someone relieve me? Apparently, we're not understanding each other. [12:48:18] [ Community noticeboard - Miraheze Meta ] - meta.miraheze.org [13:01:43] !log running updateSearchIndexConfig.php for isvwiki - T4479 [13:01:47] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [13:09:48] !log running forceSearchIndex.php for isvwiki - T4479 [13:09:52] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [13:13:46] !log drop table searchindex for isvwiki [13:13:51] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [13:22:21] [02miraheze/CreateWiki] 07paladox pushed 031 commit to 03paladox-patch-3 [+0/-0/±1] 13https://git.io/fjwIi [13:22:23] [02miraheze/CreateWiki] 07paladox 03cad3a86 - Fix race condition where sometimes you get a 404 just as the list files are updating [13:22:24] [02CreateWiki] 07paladox created branch 03paladox-patch-3 - 13https://git.io/vpJTL [13:22:31] [02CreateWiki] 07paladox opened pull request 03#116: Fix race condition where sometimes you get a 404 just as the list files are updating - 13https://git.io/fjwIP [13:25:11] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fjwIy [13:25:12] [02miraheze/services] 07MirahezeSSLBot 037cf09eb - BOT: Updating services config for wikis [14:15:13] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fjwL5 [14:15:15] [02miraheze/services] 07MirahezeSSLBot 03d71382e - BOT: Updating services config for wikis [14:15:49] Hispano76 hi, i think they want the translator extension. [14:15:57] Like we do for meta's main page (translate) [15:04:34] paladox: can you install https://www.mediawiki.org/wiki/Extension:CategorySortHeaders per T4481 its an ext made by bawolf so i dont think ext review is needed [15:04:35] [ Extension:CategorySortHeaders - MediaWiki ] - www.mediawiki.org [15:04:47] yup [15:12:27] [02miraheze/mediawiki] 07paladox pushed 031 commit to 03REL1_32 [+1/-0/±1] 13https://git.io/fjwqq [15:12:29] [02miraheze/mediawiki] 07paladox 0399f0c28 - Add CategorySortHeaders MW extension Bug: T4481 [15:14:35] [02miraheze/mediawiki] 07paladox pushed 031 commit to 03REL1_33 [+1/-0/±1] 13https://git.io/fjwqs [15:14:36] [02miraheze/mediawiki] 07paladox 03b9b4b5a - Add CategorySortHeaders MW extension Bug: T4481 [15:15:20] [02miraheze/mediawiki] 07paladox pushed 033 commits to 03REL1_33 [+0/-0/±4] 13https://git.io/fjwqG [15:15:21] [02miraheze/mediawiki] 07ayg 030bcd9ad - Third argument to unpack() requires PHP 7.1 On PHP 7.0, it looks like this code would have raised a warning and given random results, because it would interpret the first byte of $tail as the comment length instead of the correct one. Thanks for being permissive about unrecognized function parameters, PHP! (The actual reason I found and fixed this is because the phan [15:15:22] annotation here was giving me an error on PHP 7.3: "UnusedPluginSuppression Plugin BuiltinSuppressionPlugin suppresses issue PhanParamTooManyInternal on this line but this suppression is unused or suppressed elsewhere".) Change-Id: I910f78698d1a53e8a90ddf243a4d65be87f29ca9 (cherry picked from commit 94033330b43f562af7fa80f086e9bc2f591f1af0) [15:15:23] [02miraheze/mediawiki] 07matejsuchanek 0392cdd79 - Style change tags correctly on Special:Newpages Bug: T212613 Change-Id: I6905273bf9bb3d93480df201a2935d8b0e831805 [15:15:24] [02miraheze/mediawiki] 07paladox 03effe19b - Merge branch 'REL1_33' of https://github.com/wikimedia/mediawiki into REL1_33 [15:15:26] [ GitHub - wikimedia/mediawiki: 🌻 The collaborative editing software that runs Wikipedia. This is a mirror from gerrit.wikimedia.org. See https://www.mediawiki.org/wiki/Developer_access for contributing ] - github.com [15:23:21] [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±4] 13https://git.io/fjwqu [15:23:23] [02miraheze/mw-config] 07paladox 030a9663e - Add CategorySortHeaders ext to ManageWiki Change-Id: I759ed98e62cedf8cb2d483c661b63db49823c12f [15:29:13] !log enabling categorytree for all wikis [15:29:13] [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±2] 13https://git.io/fjwqV [15:29:15] [02miraheze/mw-config] 07paladox 03c07bbd0 - Move CategoryTree to ManageWiki Change-Id: Ida36502927baf2a14b0713561a363a2bee68f7c7 [15:29:24] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [15:34:47] [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±2] 13https://git.io/fjwqM [15:34:49] [02miraheze/mw-config] 07paladox 03bea44b0 - Use php entry point for CategorySortHeaders Change-Id: I3044df9d9316cb2e5b330d9ebc6bd07ebf5523fb [15:37:53] !log killing a D state nginx process on mw2 [15:37:58] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [15:48:43] !log depooled mw2 for reboot [15:48:47] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [15:49:18] !log reboot mw2 [15:49:24] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [15:50:37] PROBLEM - mw1 Current Load on mw1 is WARNING: WARNING - load average: 7.56, 6.01, 4.63 [15:51:52] PROBLEM - cp4 Varnish Backends on cp4 is CRITICAL: 1 backends are down. mw2 [15:53:32] PROBLEM - cp3 Varnish Backends on cp3 is CRITICAL: 1 backends are down. mw2 [15:53:47] PROBLEM - cp2 Varnish Backends on cp2 is CRITICAL: 1 backends are down. mw2 [15:54:37] RECOVERY - mw1 Current Load on mw1 is OK: OK - load average: 6.76, 6.65, 5.21 [16:03:23] !log repool mw2 [16:03:28] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [16:03:31] RECOVERY - cp4 Varnish Backends on cp4 is OK: All 5 backends are healthy [16:03:32] RECOVERY - cp3 Varnish Backends on cp3 is OK: All 5 backends are healthy [16:03:47] RECOVERY - cp2 Varnish Backends on cp2 is OK: All 5 backends are healthy [16:08:06] !log depool mw2 [16:08:10] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fjwmB [16:08:12] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [16:08:12] [02miraheze/puppet] 07paladox 0385329e2 - nginx: Switch back to nginx 1.16 (stable) compared to using the very latest [16:13:17] PROBLEM - cp4 Varnish Backends on cp4 is CRITICAL: 1 backends are down. mw2 [16:13:32] PROBLEM - cp3 Varnish Backends on cp3 is CRITICAL: 1 backends are down. mw2 [16:13:47] PROBLEM - cp2 Varnish Backends on cp2 is CRITICAL: 1 backends are down. mw2 [16:13:56] !log downgrading nginx to 1.16 on mw2 [16:14:04] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [16:15:13] RECOVERY - cp4 Varnish Backends on cp4 is OK: All 5 backends are healthy [16:15:24] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fjwma [16:15:26] [02miraheze/puppet] 07paladox 03c9a4be1 - mediawiki::nginx: Fix loading mediawiki-includes before mediawiki.conf [16:15:32] RECOVERY - cp3 Varnish Backends on cp3 is OK: All 5 backends are healthy [16:15:47] RECOVERY - cp2 Varnish Backends on cp2 is OK: All 5 backends are healthy [16:16:44] !log repool mw2 [16:16:48] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [16:17:22] !log downgrading nginx to 1.16 on misc2 [16:17:26] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [16:18:13] !log depool mw3 [16:18:17] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [16:18:19] !log downgrading nginx to 1.16 on mw3 [16:18:23] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [16:22:21] PROBLEM - mw3 HTTPS on mw3 is CRITICAL: connect to address 81.4.121.113 and port 443: Connection refusedHTTP CRITICAL - Unable to open TCP socket [16:23:03] PROBLEM - cp4 Varnish Backends on cp4 is CRITICAL: 1 backends are down. mw3 [16:23:20] !log repool mw3 [16:23:26] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [16:23:43] !log depool mw1 [16:23:47] !log downgrading nginx to 1.16 on mw1 [16:23:48] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [16:23:54] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [16:24:21] RECOVERY - mw3 HTTPS on mw3 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 442 bytes in 0.014 second response time [16:25:03] RECOVERY - cp4 Varnish Backends on cp4 is OK: All 5 backends are healthy [16:27:32] PROBLEM - cp3 Varnish Backends on cp3 is CRITICAL: 1 backends are down. mw1 [16:27:33] !log repool mw1 [16:27:41] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [16:29:32] RECOVERY - cp3 Varnish Backends on cp3 is OK: All 5 backends are healthy [16:31:41] Am I missing something? https://phabricator.miraheze.org/T4477 [16:31:42] [ ⚓ T4477 Can select multiple answer ] - phabricator.miraheze.org [16:31:43] [02miraheze/dns] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fjwmy [16:31:44] [02miraheze/dns] 07paladox 0314a6872 - Depool cp4 [16:31:59] Paladox: ^ [16:32:59] RF1dle Im not sure, that's a john question (as he designed that forum) [16:33:23] !log downgrade nginx on misc4 to 1.16 [16:33:31] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [16:34:19] Paladox: Thx I can't see why it shouldn't be - I've posted at Miraheze Phab. (And answered Andre at WikiMedia Phabricator about Why) (he could have thought of that himself though) [16:34:40] ok [16:35:36] !log upgrade nginx to 1.16 on cp4 [16:35:45] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [16:36:41] [02miraheze/dns] 07paladox pushed 032 commits to 03master [+0/-0/±2] 13https://git.io/fjwm7 [16:36:42] [02miraheze/dns] 07paladox 038c0ea3b - Revert "Depool cp4" This reverts commit 14a687211d7b19feda9f8fedf4fdaad5b59ed3be. [16:36:43] [02miraheze/dns] 07paladox 039f1471b - Depool cp2 [16:38:42] !log upgrade puppet-agent on cp2 and misc1 [16:38:46] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [16:39:09] !log downgrade nginx to 1.16 on misc1 [16:39:14] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [16:39:26] !log upgrade nginx to 1.16 on cp2 [16:39:31] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [16:40:20] [02miraheze/dns] 07paladox pushed 032 commits to 03master [+0/-0/±2] 13https://git.io/fjwmb [16:40:21] [02miraheze/dns] 07paladox 03d0c1d5d - Revert "Depool cp2" This reverts commit 9f1471bec51855c7c8e77b6017bb4d5c54d8d45b. [16:40:23] [02miraheze/dns] 07paladox 03030b34c - Depool cp3 [16:43:55] !log apt-get upgrade on cp3 [16:43:59] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [16:44:03] !log also upgrading kernel on cp3 [16:44:07] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [16:45:24] !log reboot cp3 [16:45:29] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [16:47:16] [02miraheze/dns] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fjwYe [16:47:17] [02miraheze/dns] 07paladox 0385e283b - Revert "Depool cp3" This reverts commit 030b34c2d25c8b297ece779e4a171eee6e784e3f. [16:50:50] !log ran lc on mw* [16:50:55] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [16:51:34] PROBLEM - test1 HTTPS on test1 is CRITICAL: connect to address 185.52.2.243 and port 443: Connection refusedHTTP CRITICAL - Unable to open TCP socket [16:53:34] RECOVERY - test1 HTTPS on test1 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 444 bytes in 0.010 second response time [17:28:30] Hi guys. If there is someone, i have a question :p [17:31:38] A few days ago i translated the Miraheze survey into spanish and I do not see it in my language until today. If I have not seen bad, i remember that the translations on Git were updated yesterday 🤔 [20:09:25] PROBLEM - mw3 JobQueue on mw3 is CRITICAL: JOBQUEUE CRITICAL - job queue greater than 300 jobs. Current queue: 545 [20:19:25] RECOVERY - mw3 JobQueue on mw3 is OK: JOBQUEUE OK - job queue below 300 jobs [21:09:27] PROBLEM - mw3 JobQueue on mw3 is CRITICAL: JOBQUEUE CRITICAL - job queue greater than 300 jobs. Current queue: 2729 [22:04:14] PROBLEM - wiki.bonessdiving.club - LetsEncrypt on sslhost is CRITICAL: Temporary failure in name resolutionHTTP CRITICAL - Unable to open TCP socket [22:06:14] RECOVERY - wiki.bonessdiving.club - LetsEncrypt on sslhost is OK: OK - Certificate 'sni.cloudflaressl.com' will expire on Fri 19 Jun 2020 12:00:00 PM GMT +0000. [22:37:25] RECOVERY - mw3 JobQueue on mw3 is OK: JOBQUEUE OK - job queue below 300 jobs