[00:07:46] [02miraheze/mw-config] 07Southparkfan pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Jvok0 [00:07:48] [02miraheze/mw-config] 07Southparkfan 0310c1492 - Allow editing on nonsensopediawiki [00:08:33] [02miraheze/mw-config] 07Southparkfan pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Jvoku [00:08:34] [02miraheze/mw-config] 07Southparkfan 032260048 - Update Sitenotice.php [00:10:08] !log we're back to business :D [00:10:09] miraheze/mw-config/master/2260048 - Southparkfan The build was broken. https://travis-ci.org/miraheze/mw-config/builds/659943219 [00:10:24] ... not yet [00:10:54] [02miraheze/mw-config] 07Southparkfan pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Jvokz [00:10:56] [02miraheze/mw-config] 07Southparkfan 03166504d - Update Sitenotice.php [00:11:44] PROBLEM - cp4 Varnish Backends on cp4 is CRITICAL: 1 backends are down. lizardfs6 [00:11:49] now it's over xD [00:12:31] miraheze/mw-config/master/166504d - Southparkfan The build was fixed. https://travis-ci.org/miraheze/mw-config/builds/659943492 [00:13:29] sleep well [00:13:47] RECOVERY - cp4 Varnish Backends on cp4 is OK: All 12 backends are healthy [00:19:09] PROBLEM - cp3 Disk Space on cp3 is WARNING: DISK WARNING - free space: / 2650 MB (10% inode=94%); [01:22:16] [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JvoIH [01:22:17] [02miraheze/mw-config] 07paladox 035261273 - Re-enable createwiki [02:26:00] PROBLEM - wiki.antlersg2.info - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.antlersg2.info' expires in 15 day(s) (Wed 25 Mar 2020 02:22:48 AM GMT +0000). [02:33:36] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JvotW [02:33:38] [02miraheze/ssl] 07MirahezeSSLBot 031dc19be - Bot: Update SSL cert for wiki.antlersg2.info [02:44:06] RECOVERY - wiki.antlersg2.info - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.antlersg2.info' will expire on Sun 07 Jun 2020 01:33:30 AM GMT +0000. [03:20:04] PROBLEM - cp8 Stunnel Http for mw7 on cp8 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [03:21:08] PROBLEM - cp3 Stunnel Http for mw7 on cp3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [03:21:11] PROBLEM - cp4 Stunnel Http for mw7 on cp4 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [03:21:39] PROBLEM - cp8 Varnish Backends on cp8 is CRITICAL: 1 backends are down. mw7 [03:21:40] PROBLEM - cp3 Varnish Backends on cp3 is CRITICAL: 1 backends are down. mw7 [03:22:02] RECOVERY - cp8 Stunnel Http for mw7 on cp8 is OK: HTTP OK: HTTP/1.1 200 OK - 15316 bytes in 0.309 second response time [03:23:03] RECOVERY - cp3 Stunnel Http for mw7 on cp3 is OK: HTTP OK: HTTP/1.1 200 OK - 15324 bytes in 0.748 second response time [03:23:08] RECOVERY - cp4 Stunnel Http for mw7 on cp4 is OK: HTTP OK: HTTP/1.1 200 OK - 15316 bytes in 0.078 second response time [03:23:39] RECOVERY - cp3 Varnish Backends on cp3 is OK: All 10 backends are healthy [03:23:39] RECOVERY - cp8 Varnish Backends on cp8 is OK: All 12 backends are healthy [06:27:18] RECOVERY - cp3 Disk Space on cp3 is OK: DISK OK - free space: / 3486 MB (14% inode=94%); [08:20:08] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JvoZs [08:20:09] [02miraheze/services] 07MirahezeSSLBot 03ca7ca43 - BOT: Updating services config for wikis [12:54:04] PROBLEM - cp3 Stunnel Http for mw7 on cp3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [12:54:19] PROBLEM - cp8 Stunnel Http for mw7 on cp8 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [12:56:00] RECOVERY - cp3 Stunnel Http for mw7 on cp3 is OK: HTTP OK: HTTP/1.1 200 OK - 15316 bytes in 0.748 second response time [12:56:19] RECOVERY - cp8 Stunnel Http for mw7 on cp8 is OK: HTTP OK: HTTP/1.1 200 OK - 15330 bytes in 0.312 second response time [14:31:52] Hello BIPU! If you have any questions, feel free to ask and someone should answer soon. [14:33:03] Hi, I'm an user of the Altverse wiki. I need help with an issue. I found this error message when trying to access one of my pages: [14:33:30] The page is: https://wiki.conworlds.org/wiki/Storting [14:33:32] [ Internal error - Constructed Worlds ] - wiki.conworlds.org [14:35:29] paladox, ^ [14:35:40] hi [14:36:24] Yeh, it appears that revision is missing some how. SPF|Cloud ^ (i'm guessing from yesturday's issue). [14:36:37] MariaDB [altversewiki]> select * from revision where rev_id = 36675; [14:36:37] Empty set (0.000 sec) [14:44:56] also, createwiki seems broken [14:49:35] Oh [14:50:14] "Error: 1048 Column 'matomo_id' cannot be null (db4.miraheze.org)" hmm [14:50:20] ohh [14:50:25] right, that's due to piwik [15:40:59] [02miraheze/CreateWiki] 07translatewiki pushed 031 commit to 03master [+1/-0/±1] 13https://git.io/JvoV7 [15:41:01] [02miraheze/CreateWiki] 07translatewiki 030481eba - Localisation updates from https://translatewiki.net. [15:41:01] [ Main page - translatewiki.net ] - translatewiki.net. [15:41:02] [02miraheze/IncidentReporting] 07translatewiki pushed 031 commit to 03master [+1/-0/±0] 13https://git.io/JvoV5 [15:41:04] [02miraheze/IncidentReporting] 07translatewiki 0317a90d1 - Localisation updates from https://translatewiki.net. [15:41:04] [ Main page - translatewiki.net ] - translatewiki.net. [15:41:05] [02miraheze/MirahezeMagic] 07translatewiki pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JvoVd [15:41:07] [02miraheze/MirahezeMagic] 07translatewiki 03417c977 - Localisation updates from https://translatewiki.net. [15:41:07] [ Main page - translatewiki.net ] - translatewiki.net. [15:41:08] [02miraheze/ManageWiki] 07translatewiki pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JvoVF [15:41:10] [02miraheze/ManageWiki] 07translatewiki 03ccd76d1 - Localisation updates from https://translatewiki.net. [15:41:10] [ Main page - translatewiki.net ] - translatewiki.net. [15:43:19] miraheze/ManageWiki/master/ccd76d1 - translatewiki.net The build was broken. https://travis-ci.org/miraheze/ManageWiki/builds/660209665 [15:43:27] miraheze/CreateWiki/master/0481eba - translatewiki.net The build was broken. https://travis-ci.com/miraheze/CreateWiki/builds/152471057 [15:43:37] miraheze/IncidentReporting/master/17a90d1 - translatewiki.net The build was broken. https://travis-ci.com/miraheze/IncidentReporting/builds/152471060 [16:08:17] How did migration go? Sites ready for normal operation? Can log into nenawiki.org but not ndg.nenawiki.org, even after clearing cache. [16:16:53] PROBLEM - cp8 Stunnel Http for mw4 on cp8 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [16:16:58] PROBLEM - cp3 Stunnel Http for mw4 on cp3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [16:17:00] PROBLEM - cp4 Varnish Backends on cp4 is CRITICAL: 1 backends are down. mw6 [16:18:49] RECOVERY - cp8 Stunnel Http for mw4 on cp8 is OK: HTTP OK: HTTP/1.1 200 OK - 15316 bytes in 0.316 second response time [16:18:59] RECOVERY - cp3 Stunnel Http for mw4 on cp3 is OK: HTTP OK: HTTP/1.1 200 OK - 15316 bytes in 0.749 second response time [16:19:01] RECOVERY - cp4 Varnish Backends on cp4 is OK: All 12 backends are healthy [16:19:45] Voidwalker: can you cu&lock the acc I blocked on meta? [16:21:12] sure [16:22:07] Voidwalker: remember locking is a globally banned user / terms of use enforcement now [16:26:20] RhinosF1 stewards doin't enforce the ToU. [16:26:59] paladox: void is also board, I don’t care who locks tbh. Just that it’s done. Go ahead. [16:32:07] technical evidence is a bit shaky on this one, but the behavior seems to check out, so locking as apparent LTA [16:32:39] Thx [16:45:41] PROBLEM - antiguabarbudacalypso.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'antiguabarbudacalypso.com' expires in 15 day(s) (Wed 25 Mar 2020 04:43:38 PM GMT +0000). [16:46:26] PROBLEM - permanentfuturelab.wiki - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'permanentfuturelab.wiki' expires in 15 day(s) (Wed 25 Mar 2020 04:44:19 PM GMT +0000). [16:46:40] PROBLEM - www.permanentfuturelab.wiki - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'permanentfuturelab.wiki' expires in 15 day(s) (Wed 25 Mar 2020 04:44:19 PM GMT +0000). [16:48:15] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JvooL [16:48:16] [02miraheze/ssl] 07MirahezeSSLBot 03ffb0e94 - Bot: Update SSL cert for antiguabarbudacalypso.com [16:50:46] PROBLEM - spiral.wiki - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'spiral.wiki' expires in 15 day(s) (Wed 25 Mar 2020 04:47:52 PM GMT +0000). [16:53:35] RECOVERY - antiguabarbudacalypso.com - LetsEncrypt on sslhost is OK: OK - Certificate 'antiguabarbudacalypso.com' will expire on Sun 07 Jun 2020 03:48:07 PM GMT +0000. [16:55:00] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JvooG [16:55:02] [02miraheze/ssl] 07MirahezeSSLBot 03e54b188 - Bot: Update SSL cert for permanentfuturelab.wiki [17:00:57] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Jvoou [17:00:58] [02miraheze/ssl] 07MirahezeSSLBot 03c5f3313 - Bot: Update SSL cert for spiral.wiki [17:02:23] RECOVERY - permanentfuturelab.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'permanentfuturelab.wiki' will expire on Sun 07 Jun 2020 03:54:54 PM GMT +0000. [17:02:48] RECOVERY - www.permanentfuturelab.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'permanentfuturelab.wiki' will expire on Sun 07 Jun 2020 03:54:54 PM GMT +0000. [17:12:54] RECOVERY - spiral.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'spiral.wiki' will expire on Sun 07 Jun 2020 04:00:49 PM GMT +0000. [17:16:10] PROBLEM - wiki.grottocenter.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.grottocenter.org' expires in 15 day(s) (Wed 25 Mar 2020 05:14:11 PM GMT +0000). [17:16:40] PROBLEM - wiki.dwplive.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.dwplive.com' expires in 15 day(s) (Wed 25 Mar 2020 05:13:22 PM GMT +0000). [17:18:36] PROBLEM - wiki.make717.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.make717.org' expires in 15 day(s) (Wed 25 Mar 2020 05:16:27 PM GMT +0000). [17:21:06] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JvoKJ [17:21:08] [02miraheze/ssl] 07MirahezeSSLBot 031bacb75 - Bot: Update SSL cert for wiki.grottocenter.org [17:29:07] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JvoKm [17:29:08] [02miraheze/ssl] 07MirahezeSSLBot 030131410 - Bot: Update SSL cert for wiki.dwplive.com [17:30:56] PROBLEM - wisdomwiki.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wisdomwiki.org' expires in 15 day(s) (Wed 25 Mar 2020 05:28:27 PM GMT +0000). [17:33:39] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JvoKz [17:33:40] [02miraheze/ssl] 07MirahezeSSLBot 03b0905d4 - Bot: Update SSL cert for wiki.make717.org [17:34:14] RECOVERY - wiki.grottocenter.org - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.grottocenter.org' will expire on Sun 07 Jun 2020 04:20:58 PM GMT +0000. [17:36:13] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JvoKr [17:36:15] [02miraheze/ssl] 07MirahezeSSLBot 03da6413a - Bot: Update SSL cert for wisdomwiki.org [17:39:32] Hi, Reception. Looking for answer... How did migration go? Sites ready for normal operation? Can log into nenawiki.org but not ndg.nenawiki.org, even after clearing cache. [17:40:51] The migration didn't go to well, we had to revert. So only ~70 are on the new infra now. [17:40:53] Your wiki i think is on the old infra? [17:41:39] paladox: yeah, don't think NENAwiki was on the "large wikis" list [17:42:00] paladox: can you please look into the login issues that MikeV is experiencing? [17:42:33] RECOVERY - wiki.dwplive.com - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.dwplive.com' will expire on Sun 07 Jun 2020 04:28:59 PM GMT +0000. [17:42:38] RECOVERY - wiki.make717.org - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.make717.org' will expire on Sun 07 Jun 2020 04:33:31 PM GMT +0000. [17:42:51] it works for me. [17:42:57] RECOVERY - wisdomwiki.org - LetsEncrypt on sslhost is OK: OK - Certificate 'wisdomwiki.org' will expire on Sun 07 Jun 2020 04:36:06 PM GMT +0000. [17:42:57] (logging in) [17:43:07] MikeV: please try again now on NDGwiki? [17:43:41] New information: Was able to log in incognito mode. But not normal mode with cache and cookies cleared. [17:44:33] Yes, login ok. What was wrong? Can I fix this problem myself? [17:46:29] ^ paladox any ideas? [17:47:59] Reception123 nope :(, i think that should fix it's self in a few days? If it's working using incognito then not much we can do. [18:02:38] paladox, Reception: Thanks. [18:23:57] Paladox: There is a database error in wiki creation [18:24:07] [106a70b3b07b5ee18ef68a24] 2020-03-09 18:20:01: Fatal exception of type "Wikimedia\Rdbms\DBQueryError" [18:24:14] Yeh, that's piwik. I have to wait on SPF|Cloud or John to discuss this. [18:24:25] it has occurred multiple times [18:24:49] yeh, it's trying to connect to piwik and failing. It should rather fail silent with it logging. [18:25:59] k [18:53:19] PROBLEM - cp8 Current Load on cp8 is WARNING: WARNING - load average: 1.31, 1.88, 1.26 [18:57:19] RECOVERY - cp8 Current Load on cp8 is OK: OK - load average: 0.74, 1.43, 1.23 [19:01:21] PROBLEM - cp8 Current Load on cp8 is WARNING: WARNING - load average: 1.91, 1.65, 1.35 [19:03:19] RECOVERY - cp8 Current Load on cp8 is OK: OK - load average: 1.20, 1.49, 1.33 [19:04:54] PROBLEM - www.zenbuddhism.info - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'www.zenbuddhism.info' expires in 15 day(s) (Wed 25 Mar 2020 07:01:16 PM GMT +0000). [19:12:54] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JvoXr [19:12:56] [02miraheze/ssl] 07MirahezeSSLBot 038bd32be - Bot: Update SSL cert for www.zenbuddhism.info [19:13:19] PROBLEM - cp8 Current Load on cp8 is CRITICAL: CRITICAL - load average: 2.29, 2.96, 2.00 [19:17:19] PROBLEM - cp8 Current Load on cp8 is WARNING: WARNING - load average: 1.00, 1.91, 1.78 [19:19:52] RECOVERY - cp8 Current Load on cp8 is OK: OK - load average: 1.17, 1.54, 1.65 [19:21:14] PROBLEM - cp4 Varnish Backends on cp4 is CRITICAL: 1 backends are down. mw5 [19:21:15] PROBLEM - cp8 Varnish Backends on cp8 is CRITICAL: 1 backends are down. mw5 [19:21:18] PROBLEM - cp3 Varnish Backends on cp3 is CRITICAL: 1 backends are down. mw5 [19:23:14] RECOVERY - www.zenbuddhism.info - LetsEncrypt on sslhost is OK: OK - Certificate 'www.zenbuddhism.info' will expire on Sun 07 Jun 2020 06:12:47 PM GMT +0000. [19:23:18] RECOVERY - cp4 Varnish Backends on cp4 is OK: All 12 backends are healthy [19:23:18] RECOVERY - cp8 Varnish Backends on cp8 is OK: All 12 backends are healthy [19:23:19] RECOVERY - cp3 Varnish Backends on cp3 is OK: All 10 backends are healthy [19:37:22] PROBLEM - cp8 Current Load on cp8 is CRITICAL: CRITICAL - load average: 2.99, 2.65, 2.03 [19:41:09] Hello, I am aware that everyone is busy with migrations and all but I have not got a response from anyone about the promised investigation of the reception wikis [19:41:17] I think the community deserves an update [19:43:13] I have made a request on the Meta noticeboard on Saturday and no one has answered me [19:46:39] ADeem: Invesitgations can take up to 8 weeks. Teams won’t be able to give information on the investigation until after it has finished. If you have any questions, please email stewards[at]miraheze.org [19:46:45] Voidwalker: reminder [19:47:42] PROBLEM - cp8 Current Load on cp8 is WARNING: WARNING - load average: 0.79, 1.62, 1.90 [19:47:47] RhinosF1: thank you for responding! Is there anywhere where there are these rules (and duration) that I could check? [19:48:15] ADeem: not really as they don’t happen often [19:48:21] The way I see it, the community must feel that nothing is being done since there are no concrete responses from any of the stewards, so even though I know the case is not like this it feels that it is being forgotten [19:50:08] They won’t be able to comment on any open investigations. I suggest reporting via email and they will get in touch with you once they’ve looked into it. [19:50:43] Evening JohnLewis [19:50:47] RhinosF1: I don't have a report myself I have just noticed that no one is getting any responses on Meta so I personally can't report anything [19:51:21] ADeem: they won’t be a public comment until they’ve completed the review. [19:51:57] RhinosF1: Okay. I would at least expect a comment acknowledging that the request was received because on the noticeboard page only non-stewards said things [19:52:29] Evening Rhinos [19:53:12] ADeem: Voidwalker is working on it. I’ll leave a note [19:53:28] RECOVERY - cp8 Current Load on cp8 is OK: OK - load average: 0.39, 1.00, 1.54 [19:53:32] RhinosF1: Okay thank you! [19:54:45] ADeem: {{done}} [19:54:53] JohnLewis: how you doing? [19:56:11] Thank you. I will check the Com. noticeboard for more updates on this matter [19:56:53] I’m good [19:58:51] JohnLewis: nice to hear [20:51:34] !log apt-get install puppet-agent salt-common salt-minion - cp8 [20:51:41] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [21:09:06] RhinosF1: well my browser decided i wanted to change a block for some reason even though i didnt even click seave [21:09:07] save* [21:09:38] Zppix: meh, up to you whether to revert or what [21:10:00] RhinosF1: technically it extended the block by a couple days so no point [21:10:17] As you’ll see in his deleted contribs, we just wanted rid of him. So I hardblocked it. [21:10:45] Zppix: the only change is to anon only [21:11:07] RhinosF1: the global offensive word filter shouldnt of prevented the creation of the page with said word in it [21:11:13] should of* [21:11:23] Can we fix that then? [21:11:49] RhinosF1: im not sure abuse filters arent my strong suit [21:12:02] Voidwalker: ^ [21:12:24] RhinosF1: are you able to see filter 31? [21:12:47] oh wait [21:12:49] no it doesnt [21:12:50] .mh Special:AbuseFilter/31 [21:12:50] it just warns [21:12:50] https://meta.miraheze.org/wiki/Special:AbuseFilter/31 [21:13:13] RhinosF1: i could do a local filter to prevent creation if you want [21:13:19] Zppix: yes and correct [21:13:27] Probably a good idea [21:13:53] RhinosF1: i dont want it to prevent globally as it could detect a false positive due to words in other langs [21:14:22] Yeah true [21:14:49] RhinosF1: im going to refine the words it prevents rather then what filter 31 does [21:14:53] ill pm you my ideas [21:15:07] Kk [21:19:22] paladox: are you any good with abusefilter? [21:27:50] Zppix nope. [21:30:42] [02mw-config] 07Pix1234 closed pull request 03#2927: wgNamespaceRobotPolicies on hispanowiki and ucronias - 13https://git.io/JvVpr [21:30:44] [02miraheze/mw-config] 07Pix1234 pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Jvo9z [21:30:45] [02miraheze/mw-config] 07Hispano76 03a21fae6 - wgNamespaceRobotPolicies on hispanowiki and ucronias (#2927) Avoid indexing pages that are not content. In addition to forcing the indexing of a namespace [21:32:07] [02miraheze/mw-config] 07Pix1234 pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Jvo9K [21:32:09] [02miraheze/mw-config] 07Pix1234 03a84b8d0 - rm php 7.1 as we don't run it in production anymore [22:37:21] paladox: did you manage to talk to JohnLewis or SPF|Cloud about piwik? [22:38:55] SPF|Cloud hasn't been online so nope :) [22:40:33] what about JohnLewis [22:40:54] He's not involved with the new infra. [22:41:20] i've spoken to him, but he said he's not involved, so hence i need SPF|Cloud to be around. [22:41:39] ah [22:51:00] !log reboot misc1 [22:51:10] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master