[00:03:19] RECOVERY - cp8 Disk Space on cp8 is OK: DISK OK - free space: / 3727 MB (19% inode=93%); [00:18:08] paladox: did you see hispano76's error message? [00:26:50] Oh, yes, I got distracted and forgot about it. [00:26:51] I’m now mobile :( [00:30:24] PROBLEM - cp8 Current Load on cp8 is WARNING: WARNING - load average: 0.85, 1.61, 1.71 [00:31:49] I believe it’s on mw[4567] though [00:35:16] PROBLEM - cp8 Current Load on cp8 is CRITICAL: CRITICAL - load average: 0.88, 2.18, 2.01 [00:36:16] paladox: well it probably needs to be fixed [00:40:10] RECOVERY - cp8 Current Load on cp8 is OK: OK - load average: 1.56, 1.39, 1.69 [00:44:54] Zppix: do you have access? [00:45:23] paladox: not atm [00:57:43] Can you guys put flags on users on their own wikis? There is this user who is making 100s of pages automatically and dont have the bot flag even after asking. [00:58:14] Ah nope [00:58:17] They do have it [00:58:30] It’s just the irc feed being dumb [00:59:27] FoxsideMiners: is it being marked as a bot edit? [00:59:37] Zppix: you know if the IRC feed has a list of bots or does it check if the account have a bot flag [00:59:39] https://www.bluepageswiki.org/wiki/Special:Contributions/Reschultzed [00:59:40] [ User contributions for Reschultzed - Bluepages ] - www.bluepageswiki.org [00:59:42] IDK [01:00:01] They said they put it on their account after I asked [01:00:02] FoxsideMiners: its not being marked as a bot edit so the irc feed is doing as expected [01:00:08] Ah [01:00:32] Dam, guess I will need to let them do their thing [01:00:59] That's happened to me and I think I know why and how it can be fixed [01:02:34] In short, there should be enough time between the flag award and the first post-bot edition (it worked for me after 24 hours with the Wiki1776 account). [01:02:52] 12/24h* [01:04:30] they understood me? :P FoxsideMiners Zppix [01:04:55] Ah [01:04:56] (especially with automatic edits) [01:05:05] oh [01:05:06] This is a few days since I told them [01:06:04] It needs to be redone. For example. You remove the flag, wait an hour, give the flag and wait a few hours and make a new edition as BOT [01:06:55] I hope you understood me XD FoxsideMiners Zppix [01:07:25] Ah, could you explain that to them? [01:10:45] FoxsideMiners: me? [01:11:12] Idk, someone needs to explain it to the user [01:26:45] FoxsideMiners: ok [01:37:06] Hello bartleMe! If you have any questions, feel free to ask and someone should answer soon. [01:37:37] the discord link on Meta.miraheze is broken [01:41:44] also, i'm having trouble with Leaflet maps on my wiki [01:46:11] the link wfm [01:53:40] done FoxsideMiners https://www.bluepageswiki.org/wiki/User_talk:Reschultzed#Flooding [01:53:41] [ User talk:Reschultzed - Bluepages ] - www.bluepageswiki.org [01:55:07] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Jv2no [01:55:09] [02miraheze/services] 07MirahezeSSLBot 033f99eb8 - BOT: Updating services config for wikis [02:23:51] hi [02:24:40] hispano76, hi [02:40:07] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Jv2cY [02:40:08] [02miraheze/services] 07MirahezeSSLBot 032a15f6f - BOT: Updating services config for wikis [02:52:11] PROBLEM - cp8 Current Load on cp8 is CRITICAL: CRITICAL - load average: 1.46, 2.59, 1.99 [02:56:38] PROBLEM - db4 Disk Space on db4 is CRITICAL: DISK CRITICAL - free space: / 21978 MB (5% inode=95%); [02:56:41] PROBLEM - cp8 Current Load on cp8 is WARNING: WARNING - load average: 0.71, 1.77, 1.83 [03:01:33] RECOVERY - cp8 Current Load on cp8 is OK: OK - load average: 1.06, 1.39, 1.66 [05:39:55] PROBLEM - cp3 Puppet on cp3 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [05:44:39] RECOVERY - cp3 Puppet on cp3 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [06:29:20] RECOVERY - cp3 Disk Space on cp3 is OK: DISK OK - free space: / 3344 MB (13% inode=94%); [06:29:27] PROBLEM - cp8 Current Load on cp8 is CRITICAL: CRITICAL - load average: 0.34, 2.63, 3.27 [06:43:35] RECOVERY - cp8 Current Load on cp8 is OK: OK - load average: 0.26, 0.68, 1.65 [12:10:08] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Jv2ap [12:10:10] [02miraheze/services] 07MirahezeSSLBot 03fa57f9d - BOT: Updating services config for wikis [12:11:32] PROBLEM - cp4 Varnish Backends on cp4 is CRITICAL: 1 backends are down. mw4 [12:13:12] PROBLEM - cp8 Stunnel Http for mw4 on cp8 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [12:13:15] PROBLEM - cp8 Varnish Backends on cp8 is CRITICAL: 1 backends are down. mw4 [12:13:28] PROBLEM - cp3 Stunnel Http for mw4 on cp3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [12:13:45] PROBLEM - cp4 Stunnel Http for mw4 on cp4 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [12:16:09] PROBLEM - cp3 Varnish Backends on cp3 is CRITICAL: 1 backends are down. mw4 [13:08:26] !log reboot mw4 [13:08:32] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [13:10:38] RECOVERY - cp3 Stunnel Http for mw4 on cp3 is OK: HTTP OK: HTTP/1.1 200 OK - 15343 bytes in 1.735 second response time [13:10:41] RECOVERY - cp8 Stunnel Http for mw4 on cp8 is OK: HTTP OK: HTTP/1.1 200 OK - 15343 bytes in 0.308 second response time [13:11:03] RECOVERY - cp4 Stunnel Http for mw4 on cp4 is OK: HTTP OK: HTTP/1.1 200 OK - 15343 bytes in 0.081 second response time [13:12:14] PROBLEM - cp3 Stunnel Http for mw6 on cp3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [13:13:06] PROBLEM - cp4 Stunnel Http for mw6 on cp4 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [13:15:14] PROBLEM - cp8 Stunnel Http for mw6 on cp8 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [13:25:21] PROBLEM - cp3 Stunnel Http for mw4 on cp3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [13:25:48] PROBLEM - cp8 Stunnel Http for mw4 on cp8 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [13:25:56] PROBLEM - cp4 Stunnel Http for mw4 on cp4 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [13:49:09] [02miraheze/CreateWiki] 07translatewiki pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Jv2rZ [13:49:11] [02miraheze/CreateWiki] 07translatewiki 034ce9cc4 - Localisation updates from https://translatewiki.net. [13:49:12] [ Main page - translatewiki.net ] - translatewiki.net. [13:52:33] RECOVERY - cp4 Stunnel Http for mw4 on cp4 is OK: HTTP OK: HTTP/1.1 200 OK - 15351 bytes in 0.080 second response time [13:53:22] RECOVERY - cp8 Stunnel Http for mw4 on cp8 is OK: HTTP OK: HTTP/1.1 200 OK - 15357 bytes in 0.313 second response time [14:11:57] PROBLEM - cp4 Stunnel Http for mw4 on cp4 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [14:12:31] PROBLEM - cp8 Stunnel Http for mw4 on cp8 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [14:30:19] !log reinstalling mw4 [14:30:28] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [15:11:40] RECOVERY - cp3 Stunnel Http for mw4 on cp3 is OK: HTTP OK: HTTP/1.1 200 OK - 15330 bytes in 0.758 second response time [15:15:48] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Jv263 [15:15:49] [02miraheze/puppet] 07paladox 0301ab6ab - gluster: Update to 7.3 [15:23:44] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Jv26g [15:23:45] [02miraheze/puppet] 07paladox 03371eebf - mediawiki: Install php before installing extensions using composer [15:24:56] PROBLEM - cp3 Stunnel Http for mw4 on cp3 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 344 bytes in 0.756 second response time [15:32:12] PROBLEM - mw2 Puppet on mw2 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 9 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[apt_update] [15:32:18] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Jv26F [15:32:20] [02miraheze/puppet] 07paladox 0389a700f - Update init.pp [15:34:22] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Jv26A [15:34:24] [02miraheze/puppet] 07paladox 037ee8c8c - Update php_fpm.pp [15:35:19] !log apt-get dist-upgrade - lizardfs6 (php & gluster, and a few other packages) [15:37:31] RECOVERY - mw2 Puppet on mw2 is OK: OK: Puppet is currently enabled, last run 5 minutes ago with 0 failures [15:38:50] !log apt-get dist-upgrade - mw[123] (php & gluster, and a few other packages) [15:40:17] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 6 datacenters are down: 128.199.139.216/cpweb, 2400:6180:0:d0::403:f001/cpweb, 51.77.107.210/cpweb, 2001:41d0:800:1056::2/cpweb, 51.161.32.127/cpweb, 2607:5300:205:200::17f6/cpweb [15:42:00] PROBLEM - cp8 HTTP 4xx/5xx ERROR Rate on cp8 is CRITICAL: CRITICAL - NGINX Error Rate is 90% [15:42:04] PROBLEM - cp3 HTTP 4xx/5xx ERROR Rate on cp3 is CRITICAL: CRITICAL - NGINX Error Rate is 75% [15:42:14] PROBLEM - cp3 Stunnel Http for mw7 on cp3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [15:43:17] PROBLEM - mw3 MediaWiki Rendering on mw3 is CRITICAL: HTTP CRITICAL: HTTP/1.1 503 Backend fetch failed - 4224 bytes in 0.466 second response time [15:43:25] PROBLEM - cp4 Stunnel Http for mw7 on cp4 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [15:43:36] PROBLEM - cp4 Stunnel Http for mw5 on cp4 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [15:43:37] PROBLEM - mw1 MediaWiki Rendering on mw1 is CRITICAL: HTTP CRITICAL: HTTP/1.1 503 Backend fetch failed - 4224 bytes in 0.437 second response time [15:44:33] PROBLEM - cp3 Stunnel Http for mw5 on cp3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [15:44:54] PROBLEM - cp8 Stunnel Http for mw5 on cp8 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [15:44:58] PROBLEM - test1 MediaWiki Rendering on test1 is CRITICAL: HTTP CRITICAL: HTTP/1.1 503 Backend fetch failed - 4224 bytes in 0.492 second response time [15:45:10] PROBLEM - mw2 MediaWiki Rendering on mw2 is CRITICAL: HTTP CRITICAL: HTTP/1.1 503 Backend fetch failed - 4224 bytes in 0.411 second response time [15:45:34] PROBLEM - lizardfs6 MediaWiki Rendering on lizardfs6 is CRITICAL: HTTP CRITICAL: HTTP/1.1 503 Backend fetch failed - 4224 bytes in 0.575 second response time [15:45:40] RECOVERY - cp4 Stunnel Http for mw4 on cp4 is OK: HTTP OK: HTTP/1.1 200 OK - 15357 bytes in 0.079 second response time [15:46:07] PROBLEM - cp8 Stunnel Http for mw7 on cp8 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [15:46:09] RECOVERY - cp3 Stunnel Http for mw4 on cp3 is OK: HTTP OK: HTTP/1.1 200 OK - 15343 bytes in 0.756 second response time [15:46:46] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Jv2il [15:46:48] [02miraheze/puppet] 07paladox 0326875a1 - Update php.pp [15:46:54] RECOVERY - cp8 Stunnel Http for mw4 on cp8 is OK: HTTP OK: HTTP/1.1 200 OK - 15343 bytes in 0.324 second response time [15:48:43] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-11 [+0/-0/±1] 13https://git.io/Jv2iE [15:48:44] [02miraheze/puppet] 07paladox 03a62814f - Update init.pp [15:48:46] [02puppet] 07paladox created branch 03paladox-patch-11 - 13https://git.io/vbiAS [15:48:47] [02puppet] 07paladox opened pull request 03#1268: Update init.pp - 13https://git.io/Jv2iu [15:49:17] [02puppet] 07paladox synchronize pull request 03#1268: Update init.pp - 13https://git.io/Jv2iu [15:49:19] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-11 [+0/-0/±1] 13https://git.io/Jv2i2 [15:49:20] [02miraheze/puppet] 07paladox 037fdbd03 - Update init.pp [15:49:35] [02puppet] 07paladox synchronize pull request 03#1268: Update init.pp - 13https://git.io/Jv2iu [15:49:36] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-11 [+0/-0/±1] 13https://git.io/Jv2ia [15:49:38] [02miraheze/puppet] 07paladox 036fc400f - Update init.pp [15:50:30] [02puppet] 07paladox closed pull request 03#1233: matomo: Switch to db6 - 13https://git.io/Jvl6P [15:50:33] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-5 [15:50:35] [02puppet] 07paladox deleted branch 03paladox-patch-5 - 13https://git.io/vbiAS [15:50:45] [02puppet] 07paladox closed pull request 03#1246: Mariadb: Add extra-port - 13https://git.io/Jv8iE [15:50:48] [02puppet] 07paladox deleted branch 03paladox-patch-2 - 13https://git.io/vbiAS [15:50:49] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-2 [15:50:58] [02puppet] 07paladox closed pull request 03#1268: Update init.pp - 13https://git.io/Jv2iu [15:51:01] [02puppet] 07paladox deleted branch 03paladox-patch-11 - 13https://git.io/vbiAS [15:51:03] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-11 [15:51:50] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-2 [+0/-0/±1] 13https://git.io/Jv2iK [15:51:52] [02miraheze/puppet] 07paladox 0362006b4 - Update php.pp [15:51:53] [02puppet] 07paladox created branch 03paladox-patch-2 - 13https://git.io/vbiAS [15:51:55] [02puppet] 07paladox opened pull request 03#1269: Update php.pp - 13https://git.io/Jv2i6 [15:52:33] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-2 [+0/-0/±1] 13https://git.io/Jv2ii [15:52:34] [02miraheze/puppet] 07paladox 03e7f2fb5 - Update init.pp [15:52:36] [02puppet] 07paladox synchronize pull request 03#1269: Update php.pp - 13https://git.io/Jv2i6 [15:52:53] [02puppet] 07paladox closed pull request 03#1269: Update php.pp - 13https://git.io/Jv2i6 [15:52:55] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±2] 13https://git.io/Jv2iP [15:52:56] [02miraheze/puppet] 07paladox 034e4bfa2 - Update php.pp (#1269) * Update php.pp * Update init.pp [15:54:34] PROBLEM - cp4 HTTPS on cp4 is CRITICAL: HTTP CRITICAL: HTTP/1.1 503 Backend fetch failed - 4154 bytes in 0.013 second response time [15:55:43] [02puppet] 07paladox deleted branch 03paladox-patch-2 - 13https://git.io/vbiAS [15:55:44] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-2 [16:02:08] PROBLEM - cp8 Stunnel Http for mw4 on cp8 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [16:04:52] PROBLEM - cp4 Stunnel Http for mw4 on cp4 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [16:06:24] PROBLEM - cp3 Stunnel Http for mw4 on cp3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [16:06:57] paladox: According to the logs, the problem with db4.miraheze.org in privadowiki is fixed or I got confused with another configuration? [16:07:12] Excuse me, curiosity [16:07:25] hispano76 which problem? [16:08:12] RECOVERY - cp8 Stunnel Http for mw6 on cp8 is OK: HTTP OK: HTTP/1.1 200 OK - 15343 bytes in 2.105 second response time [16:08:26] RECOVERY - cp3 Stunnel Http for mw6 on cp3 is OK: HTTP OK: HTTP/1.1 200 OK - 15343 bytes in 0.748 second response time [16:08:30] RECOVERY - cp4 Stunnel Http for mw6 on cp4 is OK: HTTP OK: HTTP/1.1 200 OK - 15343 bytes in 0.078 second response time [16:09:05] oh [16:09:26] paladox: !log MariaDB [altversewiki]> drop database privadowiki; - db4 (on db6 already) Cannot access the database: Unknown error (db4.miraheze.org) [16:10:26] [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Jv2P1 [16:10:27] [02miraheze/mw-config] 07paladox 0325352ab - Switch privadowiki to db6 [16:10:28] hispano76 ^ [16:10:43] paladox: Also I experienced in UcroniasWiki, HispanoWiki and MetaWiki, login errors of my alternate account. This Task summarizes it https://phabricator.miraheze.org/T5296 [16:10:46] [ ⚓ T5296 Error when creating a new user ] - phabricator.miraheze.org [16:11:29] hispano76 i think i've fixed this with https://git.io/Jv2P1 [16:11:31] [ Comparing bf2699769f98...25352ab9fca9 · miraheze/mw-config · GitHub ] - git.io [16:11:32] account Wiki1776* [16:12:48] Ok, thanks y sorry paladox [16:13:02] No need to be sorry :) [16:13:07] hispano76 does it work now? :) [16:16:31] https://phabricator.miraheze.org/T5298#100516 other users have reported error 503 [16:16:34] [ ⚓ T5298 503 error ] - phabricator.miraheze.org [16:16:47] I'll check in a moment. paladox [16:36:59] Same here [503 error, on cwars wiki]; tried to log into phabricator without sucecss :/ [16:37:56] PROBLEM - mw1 php-fpm on mw1 is CRITICAL: PROCS CRITICAL: 0 processes with command name 'php-fpm7.2' [16:37:58] They're working on fixing it Red_1 [16:39:04] Indeed, just noticed its now [Open], didn't know whether it was widespread issue or not yet though. [16:39:21] Yeh, it's widespend, sorry about that :( [16:40:11] RECOVERY - cp3 HTTP 4xx/5xx ERROR Rate on cp3 is OK: OK - NGINX Error Rate is 6% [16:40:27] Hey life is life, I hold no grudge, just wanted to make sure it's being looked at :D [16:41:09] !log reboot mw3 [16:41:15] I still think you guys are doing an amazing job since I woudn't even remotly be in your position to manage all that, so I appreciate everything you guys do! [16:41:18] PROBLEM - cp8 HTTP 4xx/5xx ERROR Rate on cp8 is WARNING: WARNING - NGINX Error Rate is 56% [16:42:19] RECOVERY - mw1 MediaWiki Rendering on mw1 is OK: HTTP OK: HTTP/1.1 200 OK - 18656 bytes in 9.576 second response time [16:42:30] RECOVERY - cp4 HTTPS on cp4 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 1544 bytes in 0.012 second response time [16:43:46] RECOVERY - mw1 php-fpm on mw1 is OK: PROCS OK: 2 processes with command name 'php-fpm7.3' [16:47:07] RECOVERY - cp8 HTTP 4xx/5xx ERROR Rate on cp8 is OK: OK - NGINX Error Rate is 31% [16:48:37] RECOVERY - mw3 MediaWiki Rendering on mw3 is OK: HTTP OK: HTTP/1.1 200 OK - 18656 bytes in 1.678 second response time [16:49:08] paladox: can you keep an eye on instant commons as well to make sure essams doesn't crash us [16:49:49] RECOVERY - test1 MediaWiki Rendering on test1 is OK: HTTP OK: HTTP/1.1 200 OK - 18656 bytes in 1.152 second response time [16:49:58] RECOVERY - mw2 MediaWiki Rendering on mw2 is OK: HTTP OK: HTTP/1.1 200 OK - 18656 bytes in 1.762 second response time [16:50:08] !log downgraded php to 7.3.14 on mw[23] lizardfs6 due to a segfault in 7.3.15, mw1 will be out of the pool till it's upgraded to buster. [16:50:09] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [16:50:27] RhinosF1 ok [16:50:29] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [16:51:22] RECOVERY - lizardfs6 MediaWiki Rendering on lizardfs6 is OK: HTTP OK: HTTP/1.1 200 OK - 18656 bytes in 2.147 second response time [16:51:46] paladox: did that not nearly bring us down during the sept ddos? [16:52:11] It did. [16:52:20] downgraded php to 7.3.14 on mw[23] lizardfs6 due to a [16:52:21] segfault in 7.3.15, mw1 will be out of the pool till it's [16:52:21] upgraded to buster. [16:52:27] crap sorry [16:56:55] PROBLEM - cp4 HTTP 4xx/5xx ERROR Rate on cp4 is WARNING: WARNING - NGINX Error Rate is 41% [16:56:58] paladox: should be good now [16:57:10] Red_1: doon't worry [16:57:28] Red_1 no worries, things should be back. Again sorry about this. [17:03:46] RhinosF1: paladox Curiosity, Commonswiki and MetaWiki are in the new infra?I don't remember, excuse me [17:04:07] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 4 datacenters are down: 128.199.139.216/cpweb, 2400:6180:0:d0::403:f001/cpweb, 51.77.107.210/cpweb, 51.161.32.127/cpweb [17:04:32] hispano76 hi, nope. [17:05:08] Yeah, as soon as that log message I accidently pasted was up our wiki was back online so awesome response time, thanks! [17:06:08] ah, ok paladox [17:09:28] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [17:12:18] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-2 [+0/-0/±1] 13https://git.io/Jv2MI [17:12:19] [02miraheze/puppet] 07paladox 037fd013d - HACK: Due to a buggy 7.3.15 release, we are forcing the use of debian's own repo Which has 7.3.14. [17:12:21] [02puppet] 07paladox created branch 03paladox-patch-2 - 13https://git.io/vbiAS [17:12:22] [02puppet] 07paladox opened pull request 03#1270: HACK: Due to a buggy 7.3.15 release, we are forcing the use of debian… - 13https://git.io/Jv2ML [17:12:31] [02puppet] 07paladox edited pull request 03#1270: HACK: Due to a buggy 7.3.15 release, we are forcing the use of debian's own repo - 13https://git.io/Jv2ML [17:12:36] [02puppet] 07paladox edited pull request 03#1270: HACK: Due to a buggy 7.3.15 release, we are forcing the use of debian's own repo - 13https://git.io/Jv2ML [17:12:59] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-2 [+0/-0/±1] 13https://git.io/Jv2Mq [17:13:01] [02miraheze/puppet] 07paladox 039223d51 - Update php_fpm.pp [17:13:02] [02puppet] 07paladox synchronize pull request 03#1270: HACK: Due to a buggy 7.3.15 release, we are forcing the use of debian's own repo - 13https://git.io/Jv2ML [17:17:22] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-5 [+0/-0/±1] 13https://git.io/Jv2MC [17:17:24] [02miraheze/puppet] 07paladox 030dc0690 - php: Do not use sury php apt repo on 7.3 Due to a bug, we will be forcing usage of debian's repo which has 7.3.14. [17:17:25] [02puppet] 07paladox created branch 03paladox-patch-5 - 13https://git.io/vbiAS [17:17:27] [02puppet] 07paladox opened pull request 03#1271: php: Do not use sury php apt repo on 7.3 - 13https://git.io/Jv2MW [17:18:05] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-5 [+0/-0/±1] 13https://git.io/Jv2MB [17:18:06] [02miraheze/puppet] 07paladox 03304bed9 - Update php_fpm.pp [17:18:08] [02puppet] 07paladox synchronize pull request 03#1271: php: Do not use sury php apt repo on 7.3 - 13https://git.io/Jv2MW [17:18:30] RECOVERY - cp4 HTTP 4xx/5xx ERROR Rate on cp4 is OK: OK - NGINX Error Rate is 22% [17:19:09] [02puppet] 07paladox closed pull request 03#1271: php: Do not use sury php apt repo on 7.3 - 13https://git.io/Jv2MW [17:19:11] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±2] 13https://git.io/Jv2MR [17:19:12] [02miraheze/puppet] 07paladox 03fbe8d11 - php: Do not use sury php apt repo on 7.3 (#1271) * php: Do not use sury php apt repo on 7.3 Due to a bug, we will be forcing usage of debian's repo which has 7.3.14. * Update php_fpm.pp [17:19:14] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-5 [17:19:15] [02puppet] 07paladox deleted branch 03paladox-patch-5 - 13https://git.io/vbiAS [17:22:44] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Jv2MK [17:22:45] [02miraheze/puppet] 07paladox 03b244965 - Update init.pp [17:24:36] !log fixing php on mw[4567] [17:24:46] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [17:25:04] RECOVERY - cp4 Stunnel Http for mw4 on cp4 is OK: HTTP OK: HTTP/1.1 200 OK - 15316 bytes in 0.080 second response time [17:27:01] RECOVERY - cp3 Stunnel Http for mw4 on cp3 is OK: HTTP OK: HTTP/1.1 200 OK - 15316 bytes in 0.770 second response time [17:28:10] RECOVERY - cp3 Stunnel Http for mw5 on cp3 is OK: HTTP OK: HTTP/1.1 200 OK - 15316 bytes in 0.772 second response time [17:28:36] RECOVERY - cp8 Stunnel Http for mw4 on cp8 is OK: HTTP OK: HTTP/1.1 200 OK - 15316 bytes in 0.670 second response time [17:28:59] RECOVERY - cp8 Stunnel Http for mw5 on cp8 is OK: HTTP OK: HTTP/1.1 200 OK - 15316 bytes in 0.312 second response time [17:30:47] RECOVERY - cp4 Stunnel Http for mw5 on cp4 is OK: HTTP OK: HTTP/1.1 200 OK - 15316 bytes in 0.099 second response time [17:33:52] RECOVERY - cp8 Stunnel Http for mw7 on cp8 is OK: HTTP OK: HTTP/1.1 200 OK - 15316 bytes in 0.309 second response time [17:35:56] RECOVERY - cp4 Stunnel Http for mw7 on cp4 is OK: HTTP OK: HTTP/1.1 200 OK - 15316 bytes in 0.079 second response time [17:36:11] RECOVERY - cp3 Stunnel Http for mw7 on cp3 is OK: HTTP OK: HTTP/1.1 200 OK - 15316 bytes in 0.756 second response time [17:36:35] PROBLEM - cp4 Stunnel Http for mw6 on cp4 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [17:36:59] PROBLEM - cp8 Stunnel Http for mw6 on cp8 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [17:37:09] PROBLEM - cp3 Stunnel Http for mw6 on cp3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [17:39:08] Note: possible bug LC (Languages) [17:39:44] hispano76: which is? [17:40:31] !log running lc on mw4 [17:40:56] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [17:44:17] Example https://usercontent.irccloud-cdn.com/file/kP13omcu/Anotaci%C3%B3n%202020-03-02%20114133.jpg [17:46:01] hispano76 does it work now? [17:56:49] RECOVERY - cp4 Stunnel Http for mw6 on cp4 is OK: HTTP OK: HTTP/1.1 200 OK - 15316 bytes in 0.080 second response time [17:57:27] RECOVERY - cp8 Stunnel Http for mw6 on cp8 is OK: HTTP OK: HTTP/1.1 200 OK - 15316 bytes in 0.332 second response time [17:57:29] RECOVERY - cp3 Stunnel Http for mw6 on cp3 is OK: HTTP OK: HTTP/1.1 200 OK - 15316 bytes in 0.747 second response time [17:58:20] paladox: hum, excuse me, is partially. The filtering of recent changes appears in English in my wikis except HispanoWiki. I also miss some edits in PrivadoWiki made in these days in this page. I think the latter is related to the error of the LC. https://privado.miraheze.org/wiki/Usuario:Hispano76/Taller ( There should be a template code between

{{person...

) [17:58:21] [ Error de permisos - Privado ] - privado.miraheze.org [18:00:08] hispano76 hmm, does it work now? [18:00:17] Or it could be two different problems, but it would be strange because yesterday I could see that issue in my subpage [18:00:40] ah, see / viendo [18:05:08] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-5 [+0/-0/±1] 13https://git.io/Jv2Dw [18:05:10] [02miraheze/puppet] 07paladox 039aeb027 - apt: Blacklist php unintended upgrades Today this caused an outage on mw[4567] when php-redis was upgraded. [18:05:11] [02puppet] 07paladox created branch 03paladox-patch-5 - 13https://git.io/vbiAS [18:05:13] [02puppet] 07paladox opened pull request 03#1272: apt: Blacklist php unintended upgrades - 13https://git.io/Jv2Dr [18:06:52] Yeah, it looks fixed. Except for the missing edition on the sub-page that I did between Feb 25 and 26 and that I could see until yesterday. That edition on my sub-page is not important to me and can be redone. But the implication is that other editions could have been lost on other wikis (especially private ones). Let's hope it didn't happen. paladox [18:07:31] hispano76 i think that's related to me not making your wiki use db6. [18:07:37] per the fix i did earlier [18:08:44] ah then again xd Thanks paladox and excuse me. Thank you for your work. :) [18:13:09] !log apt-get upgrade - gluster1 [18:13:37] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [18:15:29] !log downgraded php on jobrunner1 and upgraded gluster + few other pakcages [18:16:04] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [18:31:12] Hello? [18:31:25] hi [18:31:31] i need help [18:31:57] as i said yesterday [18:32:25] this link on meta.miraheze: https://discord.is/miraheze [18:32:27] it's broken [18:32:28] [ Miraheze ] - discord.is [18:32:40] it works for me [18:33:24] i tried it multiple times and it didn't work [18:33:33] What do you get? [18:33:45] i tried it yesterday multiple times and it never worked [18:34:28] What is the issue you get? [18:34:31] What does it say> [18:34:33] *? [18:34:38] did you guys change it yesterday after i mentioned the problem? [18:34:42] no [18:34:58] Zppix even said it worked for him. [18:36:05] it works today. it definitely didn't last night [18:36:57] i notice that "discord.is" is actually not an official discord domain [18:37:35] it's some kind of third party domain that forwards the user to a proper invitation. it's possible that something on that domain was changed / fixed yesterday [18:37:47] No, one of our bot operators for the discord provides that [18:38:00] @Robyul ✨ [18:38:33] changed https://meta.miraheze.org/w/index.php?title=Miraheze&type=revision&diff=99147&oldid=95402 [18:38:34] [ Difference between revisions of "Miraheze" - Miraheze Meta ] - meta.miraheze.org [18:38:38] well i know how to use discord and i'm just letting y'all know that yesterday i would take me to a page that said "the invitation has expired" [18:39:01] obviously something is different since about 12 hours ago when i last tried, as today i was obviously able to join [18:39:16] oh [18:39:31] undid my change [18:40:49] anyway [18:41:12] @brandon: probably a temp issue [18:41:20] i have an issue on my wiki [18:41:35] leaflet maps broke some time ago or recently. i'm not sure [18:41:48] [02puppet] 07paladox synchronize pull request 03#1272: apt: Blacklist php unintended upgrades - 13https://git.io/Jv2Dr [18:41:50] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-5 [+0/-0/±1] 13https://git.io/Jv2yh [18:41:51] [02miraheze/puppet] 07paladox 0317b9aff - Update 50unattended-upgrades [18:42:26] [02puppet] 07paladox closed pull request 03#1272: apt: Blacklist php unintended upgrades - 13https://git.io/Jv2Dr [18:42:28] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Jv2SJ [18:42:29] [02miraheze/puppet] 07paladox 0331b1973 - apt: Blacklist php unintended upgrades (#1272) * apt: Blacklist php unintended upgrades Today this caused an outage on mw[4567] when php-redis was upgraded. * Update 50unattended-upgrades [18:42:31] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-5 [18:42:32] [02puppet] 07paladox deleted branch 03paladox-patch-5 - 13https://git.io/vbiAS [18:42:55] @brandon: Can you give an example? [18:43:21] i was using this: [18:43:29] {{#display_map: 43.892513, -78.971969 |minzoom=6 |zoom=8 |maxzoom=17 |service=leaflet }} [18:44:23] on this page for example: https://jwiki.miraheze.org/wiki/Assembly_Halls_in_Australia [18:44:39] [ Assembly Halls in Australia - JWiki ] - jwiki.miraheze.org [18:44:47] * RhinosF1 looking [18:45:24] paladox: CSP? [18:45:31] Check your console [18:46:20] paladox: I’m mobile [18:46:30] oh, then i'm not sure [18:46:44] @brandon can you check your console log? [18:46:50] oh, you've provided a link [18:47:27] @brandon, what dosen't work? It works for m. [18:47:28] me [18:47:41] paladox: the fact the map is grey [18:48:10] @brandon RhinosF1 https://imgur.com/a/it5NVjk [18:48:10] [ Imgur: The magic of the Internet ] - imgur.com [18:48:14] dosen't look grey to me [18:48:28] Hmm [18:48:34] Does here [18:49:40] https://usercontent.irccloud-cdn.com/file/aZJv7Ob2/IMG_5940.PNG [18:50:38] I've tried in Firefox and Safari and it's grey for me too [18:50:51] paladox: ^ chrome for ios here [18:51:22] how long did you wait on the page before closing? [18:51:33] (I'm using safari too, and also uk) [18:51:53] A while. I used to see partial loading of the leaflet map tiles [18:52:08] I'm in Canada. I've tested on Linux and iOS [18:52:29] paladox: still not loaded [18:52:37] RhinosF1 wait like 5 minutes [18:52:43] Kk [18:52:49] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 2 datacenters are down: 2001:41d0:800:1056::2/cpweb, 51.161.32.127/cpweb [18:53:02] 5 minutes?? [18:54:34] RhinosF1 does it work? [18:54:59] @brandon, could you look at the console to see if it's throwing errors? [18:55:12] the web browser's console? [18:55:29] No [18:55:37] @brandon: yes [18:56:30] https://cdn.discordapp.com/attachments/435711390544560128/684111907426533381/unknown.png [18:56:42] i pasted a screenshot into discord. hope the IRC bridge can handle [18:57:11] paladox: that’s csp then [18:57:20] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [18:57:31] Can we whitelist? [18:58:33] are you asking me? [18:59:02] @brandon: no, I’m talking to paladox now [19:00:10] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Jv2Si [19:00:12] [02miraheze/puppet] 07paladox 03cbea7a8 - Whitelist fastly.net [19:05:47] you're fixing it? [19:06:00] yup [19:10:20] Ty [19:10:36] When should it be able to work? [19:11:31] in a few mins [19:11:32] ~2 mins [19:16:03] @brandon should work now [19:19:33] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 1 datacenter is down: 128.199.139.216/cpweb [19:25:21] Still not working on my iPhone. I'll try again on Firefox on my laptop [19:25:41] !log downgrade php-redis on mw1 [19:26:12] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [19:28:52]