[03:02:35] PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 6 datacenters are down: 107.191.126.23/cpweb, 2604:180:0:33b::2/cpweb, 128.199.139.216/cpweb, 2400:6180:0:d0::403:f001/cpweb, 81.4.109.133/cpweb, 2a00:d880:5:8ea::ebc7/cpweb [03:03:04] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 1 datacenter is down: 2400:6180:0:d0::403:f001/cpweb [03:04:35] RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online [03:05:04] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [06:11:14] PROBLEM - cp4 Current Load on cp4 is CRITICAL: CRITICAL - load average: 2.32, 1.60, 0.89 [06:13:14] RECOVERY - cp4 Current Load on cp4 is OK: OK - load average: 1.52, 1.55, 0.95 [06:25:27] RECOVERY - cp3 Disk Space on cp3 is OK: DISK OK - free space: / 2673 MB (11% inode=94%); [07:03:03] PROBLEM - cp4 Varnish Backends on cp4 is CRITICAL: 3 backends are down. mw1 mw2 mw3 [07:03:13] PROBLEM - cp2 Varnish Backends on cp2 is CRITICAL: 2 backends are down. mw2 mw3 [07:03:21] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 5 datacenters are down: 107.191.126.23/cpweb, 2604:180:0:33b::2/cpweb, 128.199.139.216/cpweb, 2400:6180:0:d0::403:f001/cpweb, 81.4.109.133/cpweb [07:03:38] PROBLEM - cp3 HTTP 4xx/5xx ERROR Rate on cp3 is WARNING: WARNING - NGINX Error Rate is 45% [07:05:05] RECOVERY - cp4 Varnish Backends on cp4 is OK: All 5 backends are healthy [07:05:13] RECOVERY - cp2 Varnish Backends on cp2 is OK: All 5 backends are healthy [07:05:18] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [07:05:37] RECOVERY - cp3 HTTP 4xx/5xx ERROR Rate on cp3 is OK: OK - NGINX Error Rate is 4% [07:31:18] [02miraheze/MirahezeMagic] 07translatewiki pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fjpFN [07:31:20] [02miraheze/MirahezeMagic] 07translatewiki 030ec3c25 - Localisation updates from https://translatewiki.net. [07:31:21] [02miraheze/ManageWiki] 07translatewiki pushed 031 commit to 03master [+0/-0/±2] 13https://git.io/fjpFA [07:31:23] [02miraheze/ManageWiki] 07translatewiki 03e9c3a17 - Localisation updates from https://translatewiki.net. [07:31:24] [02miraheze/CreateWiki] 07translatewiki pushed 031 commit to 03master [+0/-0/±8] 13https://git.io/fjpFx [07:31:26] [02miraheze/CreateWiki] 07translatewiki 03e24fcff - Localisation updates from https://translatewiki.net. [09:20:13] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fjpA3 [09:20:15] [02miraheze/services] 07MirahezeSSLBot 0379f3836 - BOT: Updating services config for wikis [09:22:36] PROBLEM - lizardfs1 Puppet on lizardfs1 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [09:22:43] PROBLEM - puppet1 Puppet on puppet1 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [09:22:44] PROBLEM - misc3 Puppet on misc3 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [09:22:48] PROBLEM - db4 Puppet on db4 is CRITICAL: CRITICAL: Puppet has 18 failures. Last run 2 minutes ago with 18 failures. Failed resources (up to 3 shown) [09:23:12] PROBLEM - misc4 Puppet on misc4 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [09:23:13] PROBLEM - lizardfs3 Puppet on lizardfs3 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [09:23:19] PROBLEM - misc1 Puppet on misc1 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [09:23:21] PROBLEM - misc2 Puppet on misc2 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [09:23:23] PROBLEM - mw2 Puppet on mw2 is CRITICAL: CRITICAL: Puppet has 213 failures. Last run 2 minutes ago with 213 failures. Failed resources (up to 3 shown) [09:23:32] PROBLEM - ns1 Puppet on ns1 is CRITICAL: CRITICAL: Puppet has 15 failures. Last run 2 minutes ago with 15 failures. Failed resources (up to 3 shown) [09:23:33] PROBLEM - lizardfs4 Puppet on lizardfs4 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [09:23:35] PROBLEM - bacula1 Puppet on bacula1 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [09:23:40] PROBLEM - lizardfs2 Puppet on lizardfs2 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [09:23:45] PROBLEM - mw1 Puppet on mw1 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [09:23:46] PROBLEM - db5 Puppet on db5 is CRITICAL: CRITICAL: Puppet has 16 failures. Last run 3 minutes ago with 16 failures. Failed resources (up to 3 shown) [09:23:47] PROBLEM - lizardfs5 Puppet on lizardfs5 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [09:23:53] PROBLEM - cp4 Puppet on cp4 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [09:23:59] PROBLEM - mw3 Puppet on mw3 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [09:24:14] PROBLEM - test1 Puppet on test1 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [09:24:17] PROBLEM - cp3 Puppet on cp3 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [09:24:19] PROBLEM - cp2 Puppet on cp2 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [09:25:08] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fjpAW [09:25:09] [02miraheze/services] 07MirahezeSSLBot 030976e5b - BOT: Updating services config for wikis [09:33:21] RECOVERY - misc2 Puppet on misc2 is OK: OK: Puppet is currently enabled, last run 2 seconds ago with 0 failures [09:33:32] RECOVERY - ns1 Puppet on ns1 is OK: OK: Puppet is currently enabled, last run 19 seconds ago with 0 failures [09:33:33] RECOVERY - lizardfs4 Puppet on lizardfs4 is OK: OK: Puppet is currently enabled, last run 18 seconds ago with 0 failures [09:33:35] RECOVERY - bacula1 Puppet on bacula1 is OK: OK: Puppet is currently enabled, last run 13 seconds ago with 0 failures [09:33:40] RECOVERY - lizardfs2 Puppet on lizardfs2 is OK: OK: Puppet is currently enabled, last run 25 seconds ago with 0 failures [09:33:46] RECOVERY - db5 Puppet on db5 is OK: OK: Puppet is currently enabled, last run 37 seconds ago with 0 failures [09:33:47] RECOVERY - lizardfs5 Puppet on lizardfs5 is OK: OK: Puppet is currently enabled, last run 34 seconds ago with 0 failures [09:33:53] RECOVERY - cp4 Puppet on cp4 is OK: OK: Puppet is currently enabled, last run 1 second ago with 0 failures [09:34:36] RECOVERY - lizardfs1 Puppet on lizardfs1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [09:34:43] RECOVERY - puppet1 Puppet on puppet1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [09:34:45] RECOVERY - misc3 Puppet on misc3 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [09:34:48] RECOVERY - db4 Puppet on db4 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [09:35:12] RECOVERY - misc4 Puppet on misc4 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [09:35:13] RECOVERY - lizardfs3 Puppet on lizardfs3 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [09:35:19] RECOVERY - misc1 Puppet on misc1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [09:35:43] RECOVERY - mw2 Puppet on mw2 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [09:35:45] RECOVERY - mw1 Puppet on mw1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [09:35:58] RECOVERY - mw3 Puppet on mw3 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [09:36:14] RECOVERY - test1 Puppet on test1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [09:36:19] RECOVERY - cp3 Puppet on cp3 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [09:36:20] RECOVERY - cp2 Puppet on cp2 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [09:41:41] PROBLEM - wiki.galactee.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.galactee.org' expires in 15 day(s) (Thu 19 Sep 2019 09:38:18 AM GMT +0000). [09:41:54] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fjpAr [09:41:55] [02miraheze/ssl] 07MirahezeSSLBot 03c365c89 - Bot: Update SSL cert for wiki.galactee.org [09:43:41] RECOVERY - wiki.galactee.org - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.galactee.org' will expire on Mon 02 Dec 2019 08:41:48 AM GMT +0000. [10:46:09] PROBLEM - thesimswiki.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'www.thesimswiki.com' expires in 15 day(s) (Thu 19 Sep 2019 10:43:13 AM GMT +0000). [10:46:52] PROBLEM - www.thesimswiki.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'www.thesimswiki.com' expires in 15 day(s) (Thu 19 Sep 2019 10:43:13 AM GMT +0000). [10:47:07] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fjpxu [10:47:08] [02miraheze/ssl] 07MirahezeSSLBot 035828bd7 - Bot: Update SSL cert for www.thesimswiki.com [10:47:08] PROBLEM - wiki.thesimswiki.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'www.thesimswiki.com' expires in 15 day(s) (Thu 19 Sep 2019 10:43:13 AM GMT +0000). [10:52:52] RECOVERY - www.thesimswiki.com - LetsEncrypt on sslhost is OK: OK - Certificate 'www.thesimswiki.com' will expire on Mon 02 Dec 2019 09:47:00 AM GMT +0000. [10:53:08] RECOVERY - wiki.thesimswiki.com - LetsEncrypt on sslhost is OK: OK - Certificate 'www.thesimswiki.com' will expire on Mon 02 Dec 2019 09:47:00 AM GMT +0000. [10:54:09] RECOVERY - thesimswiki.com - LetsEncrypt on sslhost is OK: OK - Certificate 'www.thesimswiki.com' will expire on Mon 02 Dec 2019 09:47:00 AM GMT +0000. [12:10:11] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fjppF [12:10:12] [02miraheze/services] 07MirahezeSSLBot 03cc483ed - BOT: Updating services config for wikis [12:35:00] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 6.88, 6.50, 5.50 [12:36:59] PROBLEM - mw3 Current Load on mw3 is CRITICAL: CRITICAL - load average: 11.72, 8.49, 6.35 [12:45:21] PROBLEM - mw1 Current Load on mw1 is WARNING: WARNING - load average: 6.65, 6.81, 5.32 [12:47:21] PROBLEM - mw1 Current Load on mw1 is CRITICAL: CRITICAL - load average: 8.05, 7.06, 5.57 [12:49:21] PROBLEM - mw1 Current Load on mw1 is WARNING: WARNING - load average: 7.40, 7.16, 5.79 [12:51:21] PROBLEM - mw1 Current Load on mw1 is CRITICAL: CRITICAL - load average: 9.87, 8.27, 6.36 [12:54:51] PROBLEM - mw2 Current Load on mw2 is CRITICAL: CRITICAL - load average: 8.49, 7.02, 5.56 [12:57:27] PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 2 datacenters are down: 128.199.139.216/cpweb, 2400:6180:0:d0::403:f001/cpweb [12:57:43] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 1 datacenter is down: 2400:6180:0:d0::403:f001/cpweb [12:58:48] PROBLEM - mw2 Current Load on mw2 is WARNING: WARNING - load average: 6.82, 7.14, 5.97 [12:59:26] RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online [12:59:41] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [13:00:47] RECOVERY - mw2 Current Load on mw2 is OK: OK - load average: 3.20, 6.11, 5.76 [13:01:21] PROBLEM - mw1 Current Load on mw1 is WARNING: WARNING - load average: 2.36, 6.93, 6.94 [13:03:15] PROBLEM - cp4 Current Load on cp4 is WARNING: WARNING - load average: 1.62, 1.98, 1.25 [13:03:23] RECOVERY - mw1 Current Load on mw1 is OK: OK - load average: 6.35, 6.58, 6.78 [13:05:14] RECOVERY - cp4 Current Load on cp4 is OK: OK - load average: 1.28, 1.64, 1.20 [13:10:12] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fjphD [13:10:14] [02miraheze/services] 07MirahezeSSLBot 033874345 - BOT: Updating services config for wikis [13:18:27] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 5.63, 6.42, 7.76 [13:24:27] RECOVERY - mw3 Current Load on mw3 is OK: OK - load average: 5.08, 5.27, 6.80 [13:48:00] PROBLEM - cp3 Disk Space on cp3 is WARNING: DISK WARNING - free space: / 2649 MB (10% inode=94%); [14:00:11] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fjpj4 [14:00:13] [02miraheze/services] 07MirahezeSSLBot 03560e687 - BOT: Updating services config for wikis [14:25:59] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 6.25, 7.16, 6.11 [14:27:56] RECOVERY - mw3 Current Load on mw3 is OK: OK - load average: 3.98, 6.14, 5.87 [14:31:39] paladox: we've had a few reports Matamo stats are off [14:31:46] By a large margin [14:31:52] the script runs every 24 hours [14:34:25] paladox: so it should fix itself [14:34:30] yeh [14:34:35] when the script next run [14:34:54] paladox: ah, any reason why they'd suddenly be thousands out [14:35:03] thousands out? [14:35:04] On the visitors count [14:35:15] it'll be because the script ran this morning [14:35:18] the count resets every day [14:35:55] paladox: ok, I'll pm you the report but shouldn't visitors be a rolling count for the last month [14:36:11] RhinosF1 you'll need to ask john [14:36:24] i didn't develop the extension, so i'm not sure [14:36:32] paladox: ok [14:36:35] JohnLewis: ^ [14:36:52] it is a rolling count [14:37:02] it was when I made the extension [14:37:36] JohnLewis: yeh, extension hasn't changed but stats are broke - only thing I can think of is wasn't the dB moved recently [14:37:46] then that would be a paladox issue, not me [14:38:51] JohnLewis: that's what I thought [14:39:28] the db being moved wouldn't cause it, i think? [14:43:20] i've looked in matomo for that wiki, i see no record of it going over 2,000. And also it carn't be the db move as they mention that it showed 2,000 from last week. [14:43:21] PROBLEM - mw1 Current Load on mw1 is CRITICAL: CRITICAL - load average: 9.17, 6.51, 4.81 [14:45:21] RECOVERY - mw1 Current Load on mw1 is OK: OK - load average: 5.18, 6.16, 4.90 [14:46:35] paladox: hmm [15:12:35] PROBLEM - mw3 Current Load on mw3 is CRITICAL: CRITICAL - load average: 8.37, 7.40, 6.25 [15:14:32] RECOVERY - mw3 Current Load on mw3 is OK: OK - load average: 5.07, 6.63, 6.11 [17:02:37] PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 6 datacenters are down: 107.191.126.23/cpweb, 2604:180:0:33b::2/cpweb, 128.199.139.216/cpweb, 2400:6180:0:d0::403:f001/cpweb, 81.4.109.133/cpweb, 2a00:d880:5:8ea::ebc7/cpweb [17:02:52] JohnLewis, paladox, Reception123, SPF|Cloud: ^ [17:02:57] Error 503 Backend fetch failed, forwarded for 2a00:23c6:9201:800:616f:2759:129e:dd9, 127.0.0.1 [17:02:57] (Varnish XID 308969590) via cp4 at Tue, 03 Sep 2019 17:01:52 GMT. [17:02:59] hmm [17:03:03] PROBLEM - cp4 Varnish Backends on cp4 is CRITICAL: 2 backends are down. mw1 mw3 [17:03:04] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 6 datacenters are down: 107.191.126.23/cpweb, 2604:180:0:33b::2/cpweb, 128.199.139.216/cpweb, 2400:6180:0:d0::403:f001/cpweb, 81.4.109.133/cpweb, 2a00:d880:5:8ea::ebc7/cpweb [17:03:13] PROBLEM - cp2 Varnish Backends on cp2 is CRITICAL: 1 backends are down. mw1 [17:03:19] PROBLEM - cp3 Varnish Backends on cp3 is CRITICAL: 1 backends are down. mw1 [17:03:54] paladox: is it sure that it's misc2? because if yes we definitely need an upgrade we can't keep this up [17:04:12] i'm looking [17:04:27] paladox: access is back but Icinga still moaning [17:04:31] i think i've missed it so hard to tell [17:04:35] RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online [17:05:03] RECOVERY - cp4 Varnish Backends on cp4 is OK: All 5 backends are healthy [17:05:05] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [17:05:13] RECOVERY - cp2 Varnish Backends on cp2 is OK: All 5 backends are healthy [17:05:19] RECOVERY - cp3 Varnish Backends on cp3 is OK: All 5 backends are healthy [17:05:51] misc3 had a load spike and cpu spike [17:05:56] (iowait increased) [17:13:05] Ah Thx [17:41:55] PROBLEM - cp4 HTTP 4xx/5xx ERROR Rate on cp4 is CRITICAL: CRITICAL - NGINX Error Rate is 70% [17:42:35] PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 1 datacenter is down: 2400:6180:0:d0::403:f001/cpweb [17:43:55] RECOVERY - cp4 HTTP 4xx/5xx ERROR Rate on cp4 is OK: OK - NGINX Error Rate is 2% [17:44:35] RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online [18:15:47] [02puppet] 07paladox created branch 03paladox-patch-4 - 13https://git.io/vbiAS [18:15:49] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-4 [+0/-0/±1] 13https://git.io/fjhUL [18:15:50] [02miraheze/puppet] 07paladox 035b53ff9 - Add misc3 ip to vpncloud [18:16:07] [02puppet] 07paladox opened pull request 03#1077: Add misc3 ip to vpncloud - 13https://git.io/fjhUt [18:17:02] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-4 [+0/-0/±1] 13https://git.io/fjhUq [18:17:04] [02miraheze/puppet] 07paladox 03029e769 - Update vpncloud.pp [18:17:06] [02puppet] 07paladox synchronize pull request 03#1077: Add misc3 ip to vpncloud - 13https://git.io/fjhUt [18:17:12] [02puppet] 07paladox closed pull request 03#1077: Add misc3 ip to vpncloud - 13https://git.io/fjhUt [18:17:14] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±2] 13https://git.io/fjhUY [18:17:17] [02miraheze/puppet] 07paladox 03628d404 - Add misc3 ip to vpncloud (#1077) * Add misc3 ip to vpncloud * Update vpncloud.pp [18:17:17] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-4 [18:17:18] [02puppet] 07paladox deleted branch 03paladox-patch-4 - 13https://git.io/vbiAS [18:57:16] 6 failed logins since last login. [18:57:18] naisu [18:58:57] hey paladox [18:58:59] PuppyKun: [18:59:07] hey [18:59:15] PuppyKun: how u doing? [19:04:57] RhinosF1: good, you? [19:08:33] hi PuppyKun :) [19:13:19] PuppyKun: pretty good [19:36:41] [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fjhTp [19:36:42] [02miraheze/mw-config] 07paladox 0371e73be - Update LocalSettings.php [19:37:34] paladox: Can You check if phab is up to date? [19:38:46] I only updated phab last week [19:38:58] paladox: date? [19:39:09] you should check https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:39:09] [ Tech:Server admin log - Miraheze Meta ] - meta.miraheze.org [19:39:24] 32st aug [19:39:27] *31st [19:40:06] paladox: Should be then :) wanted to check whether our bug fix was in [19:45:22] PROBLEM - mw1 Current Load on mw1 is WARNING: WARNING - load average: 7.58, 6.24, 5.13 [19:45:59] Hello Pongles! If you have any questions feel free to ask and someone should answer soon. [19:47:22] RECOVERY - mw1 Current Load on mw1 is OK: OK - load average: 5.68, 6.21, 5.27 [19:48:46] Hello, just a few minutes ago, I requested perms on the publictestwiki. Thankfully I got them however I was told to "be mindful of not flooding recent changes". I am a bit unsure how to do that as I won't be importing the pages but rather using a bot script to generate the 2600+ pages. Would having them be marked as bot edits be sufficient? [19:49:22] Pongles: ah, will you be using your current account to run the script [19:49:28] yes [19:49:39] Pongles: how long do you expect it to take? [19:50:12] I'm not sure to be honest, I haven't done something like this before. It really depends on the ratelimit of the API I guess [19:51:25] Pongles: I can grant bot for 24 hours then, ping someone when it's done - would your own wiki be better though? [19:52:08] If you're effectively creating everything on publictestwiki [19:52:16] I didn't want to spam the wiki I am working on, as it's not under my account and my request for a personal test wiki was denied. [19:52:36] Basically I've never done something like this before and I don't want to pollute a wiki that matters [19:52:51] because I am likely to mess it up at least once [19:53:04] Pongles: ah, Iet me look into it [19:53:52] I'll grant bot on Testwiki for the day for you [19:54:10] okay, thank you. [19:54:35] Pongles: done [20:00:29] PROBLEM - test1 Puppet on test1 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Mount[/mnt/mediawiki-static] [20:01:16] hm [20:01:24] would be nice if www.publictestwiki.com didn't show ssl errors :p [20:02:41] PuppyKun: cert looks fine to me. Screenshot? And the above puppet thing usually fixes itself on next run [20:03:21] PROBLEM - mw1 Current Load on mw1 is CRITICAL: CRITICAL - load average: 8.51, 6.29, 5.54 [20:04:46] RhinosF1: www.publictestwiki.com works for you? not https://publictestwiki.com [20:04:47] [ TestWiki ] - publictestwiki.com [20:05:02] I get "Firefox does not trust this site because it uses a certificate that is not valid for www.publictestwiki.com. The certificate is only valid for the following names: *.miraheze.org, miraheze.org" [20:05:22] RECOVERY - mw1 Current Load on mw1 is OK: OK - load average: 5.57, 5.88, 5.47 [20:05:36] PuppyKun: ah I see, it's the www before it then as publictestwiki.com works [20:06:15] :p yeah. would be nice if one redirected to the other [20:06:17] and i could fix it but meh [20:06:28] It would be nice [20:12:44] PROBLEM - misc3 Puppet on misc3 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 10 minutes ago with 0 failures [21:12:33] PuppyKun: i think www.*.miraheze.org should cover the www issue? [21:15:36] no, but www.publictestwiki.com would [21:17:18] i would think that theres a cert wildcard that would fix www.* period [21:18:21] Zppix: no because www.* anything is just a subdomain and certs arent issued for a subdomain with a wildcard domain. that would let me have a valid cert for www.facebook.com :p [21:18:36] the cert just has to be updated to cover www.publictestwiki.com and publictestwiki.com [21:18:49] *shrug* damn CA's being difficult [21:21:10] RhinosF1: Sorry, for some reason the module I am using isn't logging on properly, I work on fixing that. (Blocked my IP on the testwiki) [21:21:39] Pongles: yeah, I guessed it was you. Are you okay with the IP being public? [21:21:47] Yeah, I have no issues with it [21:22:27] Pongles: ah great, Have a try at getting it log in properly - block is anon only so you should be a-ok to go when it uses the right account [21:22:54] sounds good, thanks [21:24:10] If you want you can pm me with the blocked ip in question and I can take a look at potientally unblocking it (if its done by a cvt action) [21:24:15] Pongles: [21:24:25] oh, apparently I have to manually call logIn even though I supplied a username and password, gotta love random libraries [21:24:29] Zppix: nah, it's a standard local block [21:24:33] I just did it [21:24:37] oh [21:32:23] [02mw-config] 07RhinosF1 opened pull request 03#2750: per request T4694 - 13https://git.io/fjhIH [21:32:36] paladox: ^ [21:32:48] .task t4694 [21:32:50] https://phabricator.miraheze.org/T4694 - ApprovedRevs [Open] authored by Rubydesic, assigned to RhinosF1 [21:33:46] [02mw-config] 07paladox reviewed pull request 03#2750 commit - 13https://git.io/fjhI5 [21:36:06] [02mw-config] 07RhinosF1 synchronize pull request 03#2750: per request T4694 - 13https://git.io/fjhIH [21:36:16] paladox: ^ [22:15:11] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fjhLM [22:15:12] [02miraheze/services] 07MirahezeSSLBot 03d4d6906 - BOT: Updating services config for wikis [22:26:30] [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fjhLd [22:26:31] [02miraheze/mw-config] 07paladox 03af2ee72 - Revert "Update LocalSettings.php" This reverts commit 71e73be4acc6918d17a7a3dad1f31762043d74a0. [22:26:44] RECOVERY - misc3 Puppet on misc3 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [22:27:17] paladox: my pr? T4694 [22:27:36] [02mw-config] 07paladox closed pull request 03#2750: per request T4694 - 13https://git.io/fjhIH [22:27:36] done [22:27:37] [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fjhLA [22:27:39] [02miraheze/mw-config] 07RhinosF1 03c13c958 - per request T4694 (#2750) * per request T4694 * per feedback and travis [22:29:24] paladox: thx, watching parliament get worse [22:29:42] i am too [22:30:03] paladox: what a ....? is there words? [22:30:13] huh? [22:30:33] paladox: for our political situation [22:30:47] oh, i'm not sure what words would be used [22:31:07] paladox: no, nothing quite fits [23:27:36] PROBLEM - misc3 Current Load on misc3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [23:27:39] PROBLEM - misc3 Disk Space on misc3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [23:27:40] PROBLEM - misc3 restbase on misc3 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:27:56] PROBLEM - cp3 Stunnel Http for mw3 on cp3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [23:28:35] PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 6 datacenters are down: 107.191.126.23/cpweb, 2604:180:0:33b::2/cpweb, 128.199.139.216/cpweb, 2400:6180:0:d0::403:f001/cpweb, 81.4.109.133/cpweb, 2a00:d880:5:8ea::ebc7/cpweb [23:29:30] RECOVERY - misc3 Current Load on misc3 is OK: OK - load average: 0.26, 0.36, 0.36 [23:29:35] RECOVERY - misc3 Disk Space on misc3 is OK: DISK OK - free space: / 42494 MB (88% inode=94%); [23:29:36] RECOVERY - misc3 restbase on misc3 is OK: TCP OK - 0.001 second response time on 185.52.1.71 port 7231 [23:29:51] RECOVERY - cp3 Stunnel Http for mw3 on cp3 is OK: HTTP OK: HTTP/1.1 200 OK - 24516 bytes in 0.642 second response time [23:30:35] RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online [23:33:07] Voidwalker: I think caused ^ with the 300+ accounts he g/blocked [23:33:35] lol [23:34:07] Voidwalker: My RC is literally Voidwalker, Voidwalker, Voidwalker... [23:34:53] >:D [23:35:10] (should probably hide bots though) :P [23:35:38] Voidwalker: I have that so I can make sure if I do bot edits they went through [23:36:19] that and no one usually just goes and g/blocks 300+ accts [23:37:25] no one? [23:37:35] pretty sure I just did ;) [23:39:57] Voidwalker: I also said usually [23:40:48] well, ideally, I should be doing something similar at least once a week, but 400 accounts in one go is still very unusual [23:41:00] lmao [23:41:01] that, and I blocked an ASN too, so [23:41:20] Voidwalker: Just don't catch the servers on fire with all the blocking [23:41:34] I should be done for now [23:41:41] cause i think thats grounds for immediate removal of steward rights :P [23:41:46] ;)