[00:00:00] RECOVERY - cp2 HTTPS on cp2 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 1516 bytes in 0.623 second response time [00:00:11] RECOVERY - cp3 Stunnel Http for misc2 on cp3 is OK: HTTP OK: HTTP/1.1 200 OK - 43687 bytes in 0.973 second response time [00:00:27] RECOVERY - cp4 Stunnel Http for mw2 on cp4 is OK: HTTP OK: HTTP/1.1 200 OK - 15263 bytes in 3.892 second response time [00:00:29] RECOVERY - cp2 Stunnel Http for misc2 on cp2 is OK: HTTP OK: HTTP/1.1 200 OK - 43687 bytes in 0.568 second response time [00:00:40] RECOVERY - misc2 HTTPS on misc2 is OK: HTTP OK: HTTP/1.1 200 OK - 43695 bytes in 0.146 second response time [00:00:57] RECOVERY - cp2 Varnish Backends on cp2 is OK: All 6 backends are healthy [00:01:14] RECOVERY - lizardfs6 JobChron Service on lizardfs6 is OK: PROCS OK: 1 process with args 'redisJobChronService' [00:01:17] RECOVERY - lizardfs6 JobRunner Service on lizardfs6 is OK: PROCS OK: 1 process with args 'redisJobRunnerService' [00:01:29] RECOVERY - cp4 Stunnel Http for misc2 on cp4 is OK: HTTP OK: HTTP/1.1 200 OK - 43687 bytes in 0.094 second response time [00:01:41] RECOVERY - mw3 MediaWiki Rendering on mw3 is OK: HTTP OK: HTTP/1.1 200 OK - 18668 bytes in 1.076 second response time [00:01:48] RECOVERY - test1 MediaWiki Rendering on test1 is OK: HTTP OK: HTTP/1.1 200 OK - 18674 bytes in 0.272 second response time [00:01:57] RECOVERY - mw1 MediaWiki Rendering on mw1 is OK: HTTP OK: HTTP/1.1 200 OK - 18675 bytes in 0.523 second response time [00:02:00] RECOVERY - lizardfs6 Puppet on lizardfs6 is OK: OK: Puppet is currently enabled, last run 20 seconds ago with 0 failures [00:02:09] RECOVERY - cp4 Varnish Backends on cp4 is OK: All 7 backends are healthy [00:02:10] RECOVERY - cp3 Varnish Backends on cp3 is OK: All 6 backends are healthy [00:02:17] RECOVERY - cp2 Stunnel Http for mw1 on cp2 is OK: HTTP OK: HTTP/1.1 200 OK - 15256 bytes in 0.296 second response time [00:02:24] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [00:02:46] RECOVERY - mw2 MediaWiki Rendering on mw2 is OK: HTTP OK: HTTP/1.1 200 OK - 18675 bytes in 4.555 second response time [00:03:03] RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online [00:03:15] RECOVERY - bacula1 Puppet on bacula1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [00:20:51] 504 [00:20:53] 503* [00:22:04] ok, this seems like it's redis related [00:22:10] PROBLEM - cp4 Stunnel Http for mw3 on cp4 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:22:21] PROBLEM - cp4 Stunnel Http for mw2 on cp4 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:22:35] PROBLEM - cp3 Stunnel Http for misc2 on cp3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:22:45] PROBLEM - cp2 Stunnel Http for misc2 on cp2 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:23:02] PROBLEM - cp2 Varnish Backends on cp2 is CRITICAL: 3 backends are down. mw1 mw2 mw3 [00:23:12] PROBLEM - cp2 Stunnel Http for mw1 on cp2 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:23:14] PROBLEM - misc2 HTTPS on misc2 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 372 bytes in 1.180 second response time [00:23:19] PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 6 datacenters are down: 107.191.126.23/cpweb, 2604:180:0:33b::2/cpweb, 128.199.139.216/cpweb, 2400:6180:0:d0::403:f001/cpweb, 81.4.109.133/cpweb, 2a00:d880:5:8ea::ebc7/cpweb [00:23:20] PROBLEM - mw2 MediaWiki Rendering on mw2 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 1365 bytes in 0.062 second response time [00:23:30] PROBLEM - cp2 Stunnel Http for mw3 on cp2 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:23:31] !log reboot misc2 [00:23:35] PROBLEM - cp3 Stunnel Http for mw3 on cp3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:23:44] PROBLEM - cp3 Stunnel Http for mw2 on cp3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:23:50] PROBLEM - cp3 Stunnel Http for mw1 on cp3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:23:55] PROBLEM - test1 MediaWiki Rendering on test1 is CRITICAL: HTTP CRITICAL: HTTP/1.1 503 Backend fetch failed - 4219 bytes in 0.022 second response time [00:24:05] PROBLEM - cp4 Stunnel Http for mw1 on cp4 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:24:06] PROBLEM - mw3 MediaWiki Rendering on mw3 is CRITICAL: HTTP CRITICAL: HTTP/1.1 503 Backend fetch failed - 4219 bytes in 0.043 second response time [00:24:07] PROBLEM - cp3 Varnish Backends on cp3 is CRITICAL: 4 backends are down. lizardfs6 mw1 mw2 mw3 [00:24:07] PROBLEM - cp4 Varnish Backends on cp4 is CRITICAL: 4 backends are down. lizardfs6 mw1 mw2 mw3 [00:24:09] PROBLEM - mw1 MediaWiki Rendering on mw1 is CRITICAL: HTTP CRITICAL: HTTP/1.1 503 Backend fetch failed - 4219 bytes in 0.021 second response time [00:24:09] PROBLEM - lizardfs6 MediaWiki Rendering on lizardfs6 is CRITICAL: HTTP CRITICAL: HTTP/1.1 503 Backend fetch failed - 4219 bytes in 0.055 second response time [00:24:41] PROBLEM - cp4 Stunnel Http for misc2 on cp4 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:24:42] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 6 datacenters are down: 107.191.126.23/cpweb, 2604:180:0:33b::2/cpweb, 128.199.139.216/cpweb, 2400:6180:0:d0::403:f001/cpweb, 81.4.109.133/cpweb, 2a00:d880:5:8ea::ebc7/cpweb [00:24:43] RECOVERY - cp4 Stunnel Http for mw3 on cp4 is OK: HTTP OK: HTTP/1.1 200 OK - 15257 bytes in 0.004 second response time [00:24:47] PROBLEM - cp2 Stunnel Http for mw2 on cp2 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:25:37] RECOVERY - cp2 Stunnel Http for mw1 on cp2 is OK: HTTP OK: HTTP/1.1 200 OK - 15256 bytes in 0.293 second response time [00:25:54] RECOVERY - cp2 Stunnel Http for mw3 on cp2 is OK: HTTP OK: HTTP/1.1 200 OK - 15263 bytes in 1.599 second response time [00:25:54] RECOVERY - cp3 Stunnel Http for mw3 on cp3 is OK: HTTP OK: HTTP/1.1 200 OK - 15257 bytes in 1.499 second response time [00:26:00] RECOVERY - cp3 Stunnel Http for mw2 on cp3 is OK: HTTP OK: HTTP/1.1 200 OK - 15257 bytes in 0.486 second response time [00:26:01] RECOVERY - cp3 Stunnel Http for mw1 on cp3 is OK: HTTP OK: HTTP/1.1 200 OK - 15262 bytes in 0.475 second response time [00:26:14] RECOVERY - cp4 Stunnel Http for mw1 on cp4 is OK: HTTP OK: HTTP/1.1 200 OK - 15256 bytes in 0.004 second response time [00:30:24] RECOVERY - cp3 Stunnel Http for misc2 on cp3 is OK: HTTP OK: HTTP/1.1 200 OK - 43687 bytes in 0.726 second response time [00:30:25] RECOVERY - cp2 Stunnel Http for misc2 on cp2 is OK: HTTP OK: HTTP/1.1 200 OK - 43687 bytes in 0.555 second response time [00:30:38] RECOVERY - mw2 MediaWiki Rendering on mw2 is OK: HTTP OK: HTTP/1.1 200 OK - 18675 bytes in 0.260 second response time [00:30:41] RECOVERY - misc2 HTTPS on misc2 is OK: HTTP OK: HTTP/1.1 200 OK - 43695 bytes in 0.078 second response time [00:31:03] RECOVERY - test1 MediaWiki Rendering on test1 is OK: HTTP OK: HTTP/1.1 200 OK - 18675 bytes in 0.453 second response time [00:31:11] RECOVERY - mw1 MediaWiki Rendering on mw1 is OK: HTTP OK: HTTP/1.1 200 OK - 18674 bytes in 0.709 second response time [00:31:13] RECOVERY - mw3 MediaWiki Rendering on mw3 is OK: HTTP OK: HTTP/1.1 200 OK - 18674 bytes in 1.358 second response time [00:31:18] RECOVERY - lizardfs6 MediaWiki Rendering on lizardfs6 is OK: HTTP OK: HTTP/1.1 200 OK - 18675 bytes in 0.244 second response time [00:31:39] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [00:31:52] RECOVERY - cp4 Stunnel Http for misc2 on cp4 is OK: HTTP OK: HTTP/1.1 200 OK - 43687 bytes in 0.058 second response time [00:32:01] RECOVERY - cp2 Stunnel Http for mw2 on cp2 is OK: HTTP OK: HTTP/1.1 200 OK - 15257 bytes in 0.293 second response time [00:32:18] RECOVERY - cp4 Stunnel Http for mw2 on cp4 is OK: HTTP OK: HTTP/1.1 200 OK - 15249 bytes in 0.131 second response time [00:32:20] RECOVERY - cp2 Varnish Backends on cp2 is OK: All 6 backends are healthy [00:32:35] RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online [00:32:59] RECOVERY - cp4 Varnish Backends on cp4 is OK: All 7 backends are healthy [00:33:01] RECOVERY - cp3 Varnish Backends on cp3 is OK: All 6 backends are healthy [00:55:03] PROBLEM - cp4 Stunnel Http for mw2 on cp4 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:55:09] PROBLEM - cp2 Stunnel Http for mw2 on cp2 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:55:43] PROBLEM - cp3 Stunnel Http for mw2 on cp3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:57:11] RECOVERY - cp4 Stunnel Http for mw2 on cp4 is OK: HTTP OK: HTTP/1.1 200 OK - 15257 bytes in 3.032 second response time [00:57:16] !log restart php7.3-fpm on mw2 [00:57:16] RECOVERY - cp2 Stunnel Http for mw2 on cp2 is OK: HTTP OK: HTTP/1.1 200 OK - 15263 bytes in 3.300 second response time [00:57:36] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [00:57:50] RECOVERY - cp3 Stunnel Http for mw2 on cp3 is OK: HTTP OK: HTTP/1.1 200 OK - 15249 bytes in 6.338 second response time [00:58:07] PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 1 datacenter is down: 107.191.126.23/cpweb [01:00:07] RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online [02:40:07] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JvepL [02:40:09] [02miraheze/services] 07MirahezeSSLBot 03b5c6f91 - BOT: Updating services config for wikis [04:05:07] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JvejK [04:05:08] [02miraheze/services] 07MirahezeSSLBot 032b3f06f - BOT: Updating services config for wikis [06:25:33] RECOVERY - cp3 Disk Space on cp3 is OK: DISK OK - free space: / 2764 MB (11% inode=94%); [06:38:42] PROBLEM - db4 Disk Space on db4 is CRITICAL: DISK CRITICAL - free space: / 21978 MB (5% inode=95%); [06:54:32] PROBLEM - tep.wiki - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'tep.wiki' expires in 15 day(s) (Sat 25 Jan 2020 06:51:18 AM GMT +0000). [06:54:46] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JvvvV [06:54:47] [02miraheze/ssl] 07MirahezeSSLBot 031551096 - Bot: Update SSL cert for tep.wiki [07:02:35] RECOVERY - tep.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'tep.wiki' will expire on Wed 08 Apr 2020 05:54:39 AM GMT +0000. [07:44:41] PROBLEM - mw2 Puppet on mw2 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[ufw-allow-tcp-from-185.52.3.121-to-any-port-9253] [07:52:38] RECOVERY - mw2 Puppet on mw2 is OK: OK: Puppet is currently enabled, last run 51 seconds ago with 0 failures [10:01:04] Hello Sampo! If you have any questions, feel free to ask and someone should answer soon. [10:30:39] PROBLEM - mw3 MediaWiki Rendering on mw3 is CRITICAL: HTTP CRITICAL: HTTP/1.1 503 Backend fetch failed - 4219 bytes in 0.022 second response time [10:30:52] PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 6 datacenters are down: 107.191.126.23/cpweb, 2604:180:0:33b::2/cpweb, 128.199.139.216/cpweb, 2400:6180:0:d0::403:f001/cpweb, 81.4.109.133/cpweb, 2a00:d880:5:8ea::ebc7/cpweb [10:31:14] PROBLEM - cp3 Stunnel Http for misc2 on cp3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [10:31:23] PROBLEM - cp3 Varnish Backends on cp3 is CRITICAL: 3 backends are down. mw1 mw2 mw3 [10:31:32] PROBLEM - cp2 Stunnel Http for misc2 on cp2 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [10:31:38] PROBLEM - cp3 Stunnel Http for mw3 on cp3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [10:31:42] PROBLEM - cp2 Varnish Backends on cp2 is CRITICAL: 4 backends are down. lizardfs6 mw1 mw2 mw3 [10:31:51] PROBLEM - lizardfs6 MediaWiki Rendering on lizardfs6 is CRITICAL: HTTP CRITICAL: HTTP/1.1 503 Backend fetch failed - 4219 bytes in 0.023 second response time [10:31:54] PROBLEM - mw2 MediaWiki Rendering on mw2 is CRITICAL: HTTP CRITICAL: HTTP/1.1 503 Backend fetch failed - 4219 bytes in 0.045 second response time [10:32:07] PROBLEM - cp3 Stunnel Http for mw2 on cp3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [10:32:20] PROBLEM - cp3 Stunnel Http for mw1 on cp3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [10:32:21] PROBLEM - cp2 Stunnel Http for mw2 on cp2 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [10:32:36] PROBLEM - cp4 Stunnel Http for misc2 on cp4 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [10:32:37] PROBLEM - mw1 MediaWiki Rendering on mw1 is CRITICAL: HTTP CRITICAL: HTTP/1.1 503 Backend fetch failed - 4217 bytes in 0.033 second response time [10:32:47] PROBLEM - cp4 Stunnel Http for mw2 on cp4 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [10:32:50] PROBLEM - cp4 Stunnel Http for mw1 on cp4 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [10:33:07] PROBLEM - misc2 HTTPS on misc2 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:33:07] PROBLEM - cp4 Stunnel Http for mw3 on cp4 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [10:33:08] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 6 datacenters are down: 107.191.126.23/cpweb, 2604:180:0:33b::2/cpweb, 128.199.139.216/cpweb, 2400:6180:0:d0::403:f001/cpweb, 81.4.109.133/cpweb, 2a00:d880:5:8ea::ebc7/cpweb [10:33:21] PROBLEM - test1 MediaWiki Rendering on test1 is CRITICAL: HTTP CRITICAL: HTTP/1.1 503 Backend fetch failed - 4219 bytes in 0.023 second response time [10:33:36] PROBLEM - cp2 Stunnel Http for mw1 on cp2 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [10:34:06] PROBLEM - cp4 Varnish Backends on cp4 is CRITICAL: 4 backends are down. lizardfs6 mw1 mw2 mw3 [10:34:14] RECOVERY - cp3 Stunnel Http for mw3 on cp3 is OK: HTTP OK: HTTP/1.1 200 OK - 15263 bytes in 0.505 second response time [10:34:43] RECOVERY - cp3 Stunnel Http for mw2 on cp3 is OK: HTTP OK: HTTP/1.1 200 OK - 15249 bytes in 2.810 second response time [10:34:50] RECOVERY - cp2 Stunnel Http for mw2 on cp2 is OK: HTTP OK: HTTP/1.1 200 OK - 15257 bytes in 0.293 second response time [10:35:08] RECOVERY - cp4 Stunnel Http for mw2 on cp4 is OK: HTTP OK: HTTP/1.1 200 OK - 15263 bytes in 0.008 second response time [10:35:09] RECOVERY - cp4 Stunnel Http for mw1 on cp4 is OK: HTTP OK: HTTP/1.1 200 OK - 15262 bytes in 0.011 second response time [10:36:01] RECOVERY - cp2 Stunnel Http for mw1 on cp2 is OK: HTTP OK: HTTP/1.1 200 OK - 15256 bytes in 0.294 second response time [10:40:25] PROBLEM - cp2 Stunnel Http for mw3 on cp2 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [10:40:47] PROBLEM - cp3 Stunnel Http for mw3 on cp3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [10:41:19] PROBLEM - cp3 Stunnel Http for mw2 on cp3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [10:42:21] PROBLEM - cp4 Stunnel Http for mw1 on cp4 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [10:42:48] PROBLEM - misc1 Puppet on misc1 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [10:42:49] RECOVERY - cp2 Stunnel Http for mw3 on cp2 is OK: HTTP OK: HTTP/1.1 200 OK - 15263 bytes in 0.294 second response time [10:42:54] PROBLEM - cp2 Stunnel Http for mw1 on cp2 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [10:42:58] RECOVERY - cp3 Stunnel Http for mw3 on cp3 is OK: HTTP OK: HTTP/1.1 200 OK - 15249 bytes in 0.506 second response time [10:43:48] RECOVERY - cp4 Stunnel Http for mw3 on cp4 is OK: HTTP OK: HTTP/1.1 200 OK - 15257 bytes in 0.005 second response time [10:44:37] RECOVERY - cp4 Stunnel Http for mw1 on cp4 is OK: HTTP OK: HTTP/1.1 200 OK - 15256 bytes in 0.006 second response time [10:47:50] PROBLEM - cp2 Stunnel Http for mw2 on cp2 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [10:48:12] PROBLEM - cp4 Stunnel Http for mw2 on cp4 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [10:49:02] PROBLEM - cp2 Stunnel Http for mw3 on cp2 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [10:49:03] PROBLEM - cp3 Stunnel Http for mw3 on cp3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [10:50:12] PROBLEM - cp4 Stunnel Http for mw3 on cp4 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [10:50:54] PROBLEM - cp4 Stunnel Http for mw1 on cp4 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [10:53:23] RECOVERY - cp4 Stunnel Http for mw2 on cp4 is OK: HTTP OK: HTTP/1.1 200 OK - 15257 bytes in 9.451 second response time [10:54:16] RECOVERY - cp3 Stunnel Http for mw2 on cp3 is OK: HTTP OK: HTTP/1.1 200 OK - 15263 bytes in 0.514 second response time [10:55:55] RECOVERY - cp3 Stunnel Http for mw1 on cp3 is OK: HTTP OK: HTTP/1.1 200 OK - 15262 bytes in 0.708 second response time [10:56:01] RECOVERY - cp2 Stunnel Http for mw1 on cp2 is OK: HTTP OK: HTTP/1.1 200 OK - 15256 bytes in 0.294 second response time [10:56:11] RECOVERY - cp3 Stunnel Http for mw3 on cp3 is OK: HTTP OK: HTTP/1.1 200 OK - 15263 bytes in 0.507 second response time [10:56:30] RECOVERY - cp2 Stunnel Http for mw3 on cp2 is OK: HTTP OK: HTTP/1.1 200 OK - 15257 bytes in 0.294 second response time [10:56:46] RECOVERY - mw3 MediaWiki Rendering on mw3 is OK: HTTP OK: HTTP/1.1 200 OK - 18675 bytes in 0.463 second response time [10:58:29] PROBLEM - cp4 Stunnel Http for mw2 on cp4 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [10:59:43] PROBLEM - cp3 Stunnel Http for mw2 on cp3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [11:00:35] RECOVERY - cp3 Stunnel Http for misc2 on cp3 is OK: HTTP OK: HTTP/1.1 200 OK - 43687 bytes in 0.878 second response time [11:00:39] RECOVERY - cp2 Stunnel Http for misc2 on cp2 is OK: HTTP OK: HTTP/1.1 200 OK - 43687 bytes in 0.605 second response time [11:00:46] RECOVERY - cp2 Stunnel Http for mw2 on cp2 is OK: HTTP OK: HTTP/1.1 200 OK - 15263 bytes in 0.295 second response time [11:00:47] RECOVERY - cp4 Stunnel Http for mw1 on cp4 is OK: HTTP OK: HTTP/1.1 200 OK - 15256 bytes in 0.004 second response time [11:00:47] RECOVERY - cp4 Stunnel Http for mw2 on cp4 is OK: HTTP OK: HTTP/1.1 200 OK - 15257 bytes in 0.004 second response time [11:01:48] PROBLEM - cp2 Stunnel Http for mw1 on cp2 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [11:02:04] PROBLEM - mw3 MediaWiki Rendering on mw3 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:06:49] PROBLEM - cp3 Stunnel Http for misc2 on cp3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [11:07:08] PROBLEM - cp2 Stunnel Http for misc2 on cp2 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [11:07:08] PROBLEM - cp3 Stunnel Http for mw3 on cp3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [11:07:34] PROBLEM - cp4 Stunnel Http for mw1 on cp4 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [11:07:42] RECOVERY - cp3 Stunnel Http for mw2 on cp3 is OK: HTTP OK: HTTP/1.1 200 OK - 15257 bytes in 0.481 second response time [11:07:52] PROBLEM - cp2 Stunnel Http for mw3 on cp2 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [11:08:03] PROBLEM - cp4 Puppet on cp4 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 6 minutes ago with 1 failures. Failed resources (up to 3 shown) [11:08:04] RECOVERY - misc1 Puppet on misc1 is OK: OK: Puppet is currently enabled, last run 13 seconds ago with 0 failures [11:08:14] RECOVERY - mw1 MediaWiki Rendering on mw1 is OK: HTTP OK: HTTP/1.1 200 OK - 18673 bytes in 1.834 second response time [11:08:14] RECOVERY - cp4 Stunnel Http for mw3 on cp4 is OK: HTTP OK: HTTP/1.1 200 OK - 15257 bytes in 0.004 second response time [11:08:22] RECOVERY - test1 MediaWiki Rendering on test1 is OK: HTTP OK: HTTP/1.1 200 OK - 18668 bytes in 0.867 second response time [11:08:23] RECOVERY - lizardfs6 MediaWiki Rendering on lizardfs6 is OK: HTTP OK: HTTP/1.1 200 OK - 18668 bytes in 0.928 second response time [11:08:27] RECOVERY - mw2 MediaWiki Rendering on mw2 is OK: HTTP OK: HTTP/1.1 200 OK - 18668 bytes in 0.921 second response time [11:08:48] RECOVERY - cp3 Stunnel Http for misc2 on cp3 is OK: HTTP OK: HTTP/1.1 200 OK - 43687 bytes in 1.118 second response time [11:08:51] RECOVERY - cp3 Varnish Backends on cp3 is OK: All 6 backends are healthy [11:08:55] RECOVERY - cp4 Varnish Backends on cp4 is OK: All 7 backends are healthy [11:09:15] RECOVERY - cp2 Varnish Backends on cp2 is OK: All 6 backends are healthy [11:12:09] RECOVERY - cp3 Stunnel Http for mw3 on cp3 is OK: HTTP OK: HTTP/1.1 200 OK - 15263 bytes in 0.476 second response time [11:12:53] RECOVERY - cp2 Stunnel Http for mw1 on cp2 is OK: HTTP OK: HTTP/1.1 200 OK - 15262 bytes in 0.293 second response time [11:13:14] RECOVERY - cp2 Stunnel Http for mw3 on cp2 is OK: HTTP OK: HTTP/1.1 200 OK - 15263 bytes in 0.295 second response time [11:13:28] PROBLEM - cp4 Stunnel Http for mw2 on cp4 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [11:14:00] PROBLEM - cp2 Stunnel Http for mw2 on cp2 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [11:14:03] PROBLEM - cp3 Stunnel Http for mw2 on cp3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [11:14:47] PROBLEM - mw1 MediaWiki Rendering on mw1 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:14:48] PROBLEM - cp3 Stunnel Http for mw1 on cp3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [11:14:53] PROBLEM - test1 MediaWiki Rendering on test1 is CRITICAL: HTTP CRITICAL: HTTP/1.1 503 Backend fetch failed - 4219 bytes in 0.024 second response time [11:14:53] PROBLEM - lizardfs6 MediaWiki Rendering on lizardfs6 is CRITICAL: HTTP CRITICAL: HTTP/1.1 503 Backend fetch failed - 4219 bytes in 0.034 second response time [11:14:53] PROBLEM - mw2 MediaWiki Rendering on mw2 is CRITICAL: HTTP CRITICAL: HTTP/1.1 503 Backend fetch failed - 4219 bytes in 0.087 second response time [11:15:24] PROBLEM - cp3 Varnish Backends on cp3 is CRITICAL: 4 backends are down. lizardfs6 mw1 mw2 mw3 [11:15:27] PROBLEM - cp4 Varnish Backends on cp4 is CRITICAL: 4 backends are down. lizardfs6 mw1 mw2 mw3 [11:15:49] PROBLEM - cp2 Varnish Backends on cp2 is CRITICAL: 4 backends are down. lizardfs6 mw1 mw2 mw3 [11:15:56] PROBLEM - cp3 Stunnel Http for misc2 on cp3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [11:16:28] RECOVERY - cp4 Stunnel Http for mw2 on cp4 is OK: HTTP OK: HTTP/1.1 200 OK - 15249 bytes in 8.834 second response time [11:16:41] RECOVERY - cp2 Stunnel Http for mw2 on cp2 is OK: HTTP OK: HTTP/1.1 200 OK - 15257 bytes in 0.293 second response time [11:16:42] RECOVERY - cp3 Stunnel Http for mw2 on cp3 is OK: HTTP OK: HTTP/1.1 200 OK - 15263 bytes in 0.493 second response time [11:18:35] PROBLEM - cp3 Stunnel Http for mw3 on cp3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [11:19:01] PROBLEM - cp4 Stunnel Http for mw3 on cp4 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [11:19:34] PROBLEM - cp2 Stunnel Http for mw1 on cp2 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [11:19:55] PROBLEM - cp2 Stunnel Http for mw3 on cp2 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [11:20:29] RECOVERY - lizardfs6 MediaWiki Rendering on lizardfs6 is OK: HTTP OK: HTTP/1.1 200 OK - 18675 bytes in 3.924 second response time [11:22:52] RECOVERY - cp3 Stunnel Http for mw1 on cp3 is OK: HTTP OK: HTTP/1.1 200 OK - 15256 bytes in 0.478 second response time [11:23:00] PROBLEM - cp4 Stunnel Http for mw2 on cp4 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [11:23:10] PROBLEM - cp2 Stunnel Http for mw2 on cp2 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [11:23:38] RECOVERY - cp4 Stunnel Http for mw1 on cp4 is OK: HTTP OK: HTTP/1.1 200 OK - 15256 bytes in 0.004 second response time [11:23:50] RECOVERY - cp3 Stunnel Http for mw3 on cp3 is OK: HTTP OK: HTTP/1.1 200 OK - 15257 bytes in 0.520 second response time [11:24:22] RECOVERY - cp4 Stunnel Http for mw3 on cp4 is OK: HTTP OK: HTTP/1.1 200 OK - 15263 bytes in 0.004 second response time [11:25:37] PROBLEM - cp3 Stunnel Http for mw2 on cp3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [11:26:06] PROBLEM - lizardfs6 MediaWiki Rendering on lizardfs6 is CRITICAL: HTTP CRITICAL: HTTP/1.1 503 Backend fetch failed - 4215 bytes in 0.055 second response time [11:29:07] RECOVERY - cp2 Stunnel Http for misc2 on cp2 is OK: HTTP OK: HTTP/1.1 200 OK - 43687 bytes in 0.636 second response time [11:29:12] RECOVERY - misc2 HTTPS on misc2 is OK: HTTP OK: HTTP/1.1 200 OK - 43695 bytes in 0.207 second response time [11:29:18] RECOVERY - cp4 Stunnel Http for misc2 on cp4 is OK: HTTP OK: HTTP/1.1 200 OK - 43687 bytes in 0.120 second response time [11:29:20] RECOVERY - cp4 Puppet on cp4 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [11:29:25] RECOVERY - cp3 Stunnel Http for misc2 on cp3 is OK: HTTP OK: HTTP/1.1 200 OK - 43687 bytes in 0.973 second response time [11:29:32] Hello jdstroy! If you have any questions, feel free to ask and someone should answer soon. [11:29:49] RECOVERY - mw1 MediaWiki Rendering on mw1 is OK: HTTP OK: HTTP/1.1 200 OK - 18674 bytes in 0.587 second response time [11:29:52] RECOVERY - test1 MediaWiki Rendering on test1 is OK: HTTP OK: HTTP/1.1 200 OK - 18668 bytes in 0.942 second response time [11:29:57] RECOVERY - cp2 Stunnel Http for mw1 on cp2 is OK: HTTP OK: HTTP/1.1 200 OK - 15248 bytes in 0.293 second response time [11:30:18] RECOVERY - cp2 Stunnel Http for mw3 on cp2 is OK: HTTP OK: HTTP/1.1 200 OK - 15263 bytes in 2.439 second response time [11:30:19] Hey y'all, it looks like all the wikis are down (503 Backend fetch failed; Error 503 Backend fetch failed, forwarded for 70.112.29.45, 127.0.0.1 [11:30:21] (Varnish XID 13972124) via cp2.miraheze.org at Thu, 09 Jan 2020 11:29:19 GMT. [11:30:23] ) [11:30:30] RECOVERY - cp4 Stunnel Http for mw2 on cp4 is OK: HTTP OK: HTTP/1.1 200 OK - 15249 bytes in 0.005 second response time [11:30:38] RECOVERY - mw2 MediaWiki Rendering on mw2 is OK: HTTP OK: HTTP/1.1 200 OK - 18674 bytes in 2.149 second response time [11:30:41] RECOVERY - cp3 Varnish Backends on cp3 is OK: All 6 backends are healthy [11:30:44] RECOVERY - cp3 Stunnel Http for mw2 on cp3 is OK: HTTP OK: HTTP/1.1 200 OK - 15257 bytes in 0.489 second response time [11:30:46] RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online [11:30:50] RECOVERY - cp4 Varnish Backends on cp4 is OK: All 7 backends are healthy [11:30:55] RECOVERY - cp2 Varnish Backends on cp2 is OK: All 6 backends are healthy [11:30:59] RECOVERY - cp2 Stunnel Http for mw2 on cp2 is OK: HTTP OK: HTTP/1.1 200 OK - 15263 bytes in 2.004 second response time [11:31:03] RECOVERY - mw3 MediaWiki Rendering on mw3 is OK: HTTP OK: HTTP/1.1 200 OK - 18675 bytes in 2.397 second response time [11:31:07] RECOVERY - lizardfs6 MediaWiki Rendering on lizardfs6 is OK: HTTP OK: HTTP/1.1 200 OK - 18675 bytes in 0.630 second response time [11:31:17] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [11:31:30] aha, looks like someone took care of it. Thank you all! :) [11:40:51] ooor maybe not [11:41:21] still getting 503 errors (503 Backend fetch failed) and I can't login to phabricator because mediawiki isn't serving requests [11:41:36] (Also, happy new year everyone) [11:42:55] PROBLEM - cp4 Stunnel Http for mw3 on cp4 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [11:43:06] PROBLEM - cp2 Stunnel Http for mw1 on cp2 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [11:43:18] PROBLEM - cp2 Stunnel Http for mw2 on cp2 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [11:43:32] PROBLEM - cp4 Stunnel Http for mw2 on cp4 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [11:43:59] PROBLEM - cp2 Stunnel Http for mw3 on cp2 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [11:44:02] PROBLEM - cp2 Stunnel Http for misc2 on cp2 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [11:44:03] PROBLEM - lizardfs6 MediaWiki Rendering on lizardfs6 is CRITICAL: HTTP CRITICAL: HTTP/1.1 503 Backend fetch failed - 4219 bytes in 0.147 second response time [11:44:03] PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 6 datacenters are down: 107.191.126.23/cpweb, 2604:180:0:33b::2/cpweb, 128.199.139.216/cpweb, 2400:6180:0:d0::403:f001/cpweb, 81.4.109.133/cpweb, 2a00:d880:5:8ea::ebc7/cpweb [11:44:17] PROBLEM - cp4 Varnish Backends on cp4 is CRITICAL: 4 backends are down. lizardfs6 mw1 mw2 mw3 [11:44:17] PROBLEM - mw2 MediaWiki Rendering on mw2 is CRITICAL: HTTP CRITICAL: HTTP/1.1 503 Backend fetch failed - 4219 bytes in 0.086 second response time [11:44:25] PROBLEM - cp2 Varnish Backends on cp2 is CRITICAL: 4 backends are down. lizardfs6 mw1 mw2 mw3 [11:44:29] PROBLEM - cp3 Stunnel Http for mw1 on cp3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [11:44:36] PROBLEM - misc2 HTTPS on misc2 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:44:46] PROBLEM - cp3 Varnish Backends on cp3 is CRITICAL: 4 backends are down. lizardfs6 mw1 mw2 mw3 [11:44:47] PROBLEM - cp4 Stunnel Http for misc2 on cp4 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [11:45:03] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 6 datacenters are down: 107.191.126.23/cpweb, 2604:180:0:33b::2/cpweb, 128.199.139.216/cpweb, 2400:6180:0:d0::403:f001/cpweb, 81.4.109.133/cpweb, 2a00:d880:5:8ea::ebc7/cpweb [11:45:23] PROBLEM - mw1 MediaWiki Rendering on mw1 is CRITICAL: HTTP CRITICAL: HTTP/1.1 503 Backend fetch failed - 4219 bytes in 0.025 second response time [11:45:24] PROBLEM - test1 MediaWiki Rendering on test1 is CRITICAL: HTTP CRITICAL: HTTP/1.1 503 Backend fetch failed - 4217 bytes in 0.038 second response time [11:45:24] PROBLEM - cp3 Stunnel Http for misc2 on cp3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [11:45:28] RECOVERY - cp4 Stunnel Http for mw3 on cp4 is OK: HTTP OK: HTTP/1.1 200 OK - 15257 bytes in 0.005 second response time [11:45:36] PROBLEM - mw3 MediaWiki Rendering on mw3 is CRITICAL: HTTP CRITICAL: HTTP/1.1 503 Backend fetch failed - 4219 bytes in 0.033 second response time [11:45:40] RECOVERY - cp2 Stunnel Http for mw1 on cp2 is OK: HTTP OK: HTTP/1.1 200 OK - 15256 bytes in 0.294 second response time [11:45:48] RECOVERY - cp2 Stunnel Http for mw2 on cp2 is OK: HTTP OK: HTTP/1.1 200 OK - 15263 bytes in 0.293 second response time [11:45:49] RECOVERY - cp4 Stunnel Http for mw2 on cp4 is OK: HTTP OK: HTTP/1.1 200 OK - 15257 bytes in 0.004 second response time [11:46:53] PROBLEM - misc1 Puppet on misc1 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [11:50:56] RECOVERY - cp3 Stunnel Http for misc2 on cp3 is OK: HTTP OK: HTTP/1.1 200 OK - 43687 bytes in 0.851 second response time [11:51:29] RECOVERY - cp2 Stunnel Http for mw3 on cp2 is OK: HTTP OK: HTTP/1.1 200 OK - 15249 bytes in 0.467 second response time [11:51:31] RECOVERY - cp2 Stunnel Http for misc2 on cp2 is OK: HTTP OK: HTTP/1.1 200 OK - 43687 bytes in 0.598 second response time [11:51:33] RECOVERY - lizardfs6 MediaWiki Rendering on lizardfs6 is OK: HTTP OK: HTTP/1.1 200 OK - 18675 bytes in 0.285 second response time [11:51:38] RECOVERY - mw2 MediaWiki Rendering on mw2 is OK: HTTP OK: HTTP/1.1 200 OK - 18668 bytes in 0.833 second response time [11:51:39] RECOVERY - cp3 Varnish Backends on cp3 is OK: All 6 backends are healthy [11:51:41] RECOVERY - cp3 Stunnel Http for mw1 on cp3 is OK: HTTP OK: HTTP/1.1 200 OK - 15256 bytes in 0.508 second response time [11:51:42] RECOVERY - misc1 Puppet on misc1 is OK: OK: Puppet is currently enabled, last run 2 seconds ago with 0 failures [11:51:48] RECOVERY - misc2 HTTPS on misc2 is OK: HTTP OK: HTTP/1.1 200 OK - 43695 bytes in 0.065 second response time [11:51:56] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [11:51:59] RECOVERY - cp4 Stunnel Http for misc2 on cp4 is OK: HTTP OK: HTTP/1.1 200 OK - 43687 bytes in 0.060 second response time [11:52:36] RECOVERY - test1 MediaWiki Rendering on test1 is OK: HTTP OK: HTTP/1.1 200 OK - 18667 bytes in 0.827 second response time [11:52:37] RECOVERY - mw1 MediaWiki Rendering on mw1 is OK: HTTP OK: HTTP/1.1 200 OK - 18674 bytes in 0.402 second response time [11:52:50] RECOVERY - mw3 MediaWiki Rendering on mw3 is OK: HTTP OK: HTTP/1.1 200 OK - 18674 bytes in 0.223 second response time [11:53:26] RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online [11:53:29] RECOVERY - cp4 Varnish Backends on cp4 is OK: All 7 backends are healthy [11:53:34] RECOVERY - cp2 Varnish Backends on cp2 is OK: All 6 backends are healthy [14:01:18] jdstroy: hi, apologies for the outage earlier :( [14:01:19] Thank you for notifying us! Seems the issue is fixed [14:36:26] PROBLEM - mw3 MediaWiki Rendering on mw3 is CRITICAL: HTTP CRITICAL: HTTP/1.1 503 Backend fetch failed - 4219 bytes in 0.016 second response time [14:36:56] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 6 datacenters are down: 107.191.126.23/cpweb, 2604:180:0:33b::2/cpweb, 128.199.139.216/cpweb, 2400:6180:0:d0::403:f001/cpweb, 81.4.109.133/cpweb, 2a00:d880:5:8ea::ebc7/cpweb [14:37:22] PROBLEM - cp3 Stunnel Http for mw3 on cp3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [14:37:38] PROBLEM - cp4 Stunnel Http for mw3 on cp4 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [14:37:41] PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 6 datacenters are down: 107.191.126.23/cpweb, 2604:180:0:33b::2/cpweb, 128.199.139.216/cpweb, 2400:6180:0:d0::403:f001/cpweb, 81.4.109.133/cpweb, 2a00:d880:5:8ea::ebc7/cpweb [14:37:59] PROBLEM - cp2 Stunnel Http for mw2 on cp2 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [14:38:15] PROBLEM - cp3 Stunnel Http for mw1 on cp3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [14:38:26] PROBLEM - cp4 Stunnel Http for mw2 on cp4 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [14:38:29] PROBLEM - mw1 MediaWiki Rendering on mw1 is CRITICAL: HTTP CRITICAL: HTTP/1.1 503 Backend fetch failed - 4219 bytes in 0.022 second response time [14:38:33] PROBLEM - cp2 Stunnel Http for misc2 on cp2 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [14:38:40] PROBLEM - cp3 Stunnel Http for misc2 on cp3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [14:38:43] PROBLEM - cp4 Stunnel Http for mw1 on cp4 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [14:38:57] PROBLEM - mw2 MediaWiki Rendering on mw2 is CRITICAL: HTTP CRITICAL: HTTP/1.1 503 Backend fetch failed - 4219 bytes in 0.023 second response time [14:39:08] PROBLEM - cp3 Stunnel Http for mw2 on cp3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [14:39:09] PROBLEM - test1 MediaWiki Rendering on test1 is CRITICAL: HTTP CRITICAL: HTTP/1.1 503 Backend fetch failed - 4219 bytes in 0.024 second response time [14:39:18] PROBLEM - cp4 Stunnel Http for misc2 on cp4 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [14:39:24] PROBLEM - misc2 HTTPS on misc2 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:39:38] PROBLEM - cp2 Stunnel Http for mw1 on cp2 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [14:39:44] PROBLEM - lizardfs6 MediaWiki Rendering on lizardfs6 is CRITICAL: HTTP CRITICAL: HTTP/1.1 503 Backend fetch failed - 4219 bytes in 0.023 second response time [14:39:55] PROBLEM - cp2 Stunnel Http for mw3 on cp2 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [14:39:59] PROBLEM - cp4 Varnish Backends on cp4 is CRITICAL: 3 backends are down. mw1 mw2 mw3 [14:40:10] PROBLEM - cp3 Varnish Backends on cp3 is CRITICAL: 3 backends are down. mw1 mw2 mw3 [14:40:15] PROBLEM - cp2 Varnish Backends on cp2 is CRITICAL: 3 backends are down. mw1 mw2 mw3 [14:40:48] RECOVERY - cp3 Stunnel Http for mw1 on cp3 is OK: HTTP OK: HTTP/1.1 200 OK - 15262 bytes in 8.748 second response time [14:41:04] RECOVERY - cp4 Stunnel Http for mw1 on cp4 is OK: HTTP OK: HTTP/1.1 200 OK - 15256 bytes in 0.008 second response time [14:41:55] RECOVERY - cp2 Stunnel Http for mw1 on cp2 is OK: HTTP OK: HTTP/1.1 200 OK - 15256 bytes in 0.293 second response time [14:42:20] RECOVERY - cp2 Stunnel Http for mw3 on cp2 is OK: HTTP OK: HTTP/1.1 200 OK - 15263 bytes in 3.504 second response time [14:42:28] RECOVERY - cp3 Stunnel Http for mw3 on cp3 is OK: HTTP OK: HTTP/1.1 200 OK - 15257 bytes in 0.505 second response time [14:42:38] RECOVERY - cp4 Stunnel Http for mw3 on cp4 is OK: HTTP OK: HTTP/1.1 200 OK - 15257 bytes in 0.005 second response time [14:46:40] PROBLEM - cp3 Stunnel Http for mw1 on cp3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [14:47:02] PROBLEM - cp4 Stunnel Http for mw1 on cp4 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [14:47:40] RECOVERY - lizardfs6 MediaWiki Rendering on lizardfs6 is OK: HTTP OK: HTTP/1.1 200 OK - 18669 bytes in 2.459 second response time [14:47:48] RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online [14:48:10] RECOVERY - mw1 MediaWiki Rendering on mw1 is OK: HTTP OK: HTTP/1.1 200 OK - 18675 bytes in 1.644 second response time [14:48:18] RECOVERY - cp2 Stunnel Http for misc2 on cp2 is OK: HTTP OK: HTTP/1.1 200 OK - 43687 bytes in 1.982 second response time [14:48:21] RECOVERY - cp2 Stunnel Http for mw2 on cp2 is OK: HTTP OK: HTTP/1.1 200 OK - 15263 bytes in 7.396 second response time [14:48:27] RECOVERY - cp4 Stunnel Http for mw2 on cp4 is OK: HTTP OK: HTTP/1.1 200 OK - 15263 bytes in 5.049 second response time [14:48:33] PROBLEM - cp2 Stunnel Http for mw3 on cp2 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [14:48:35] RECOVERY - cp3 Stunnel Http for misc2 on cp3 is OK: HTTP OK: HTTP/1.1 200 OK - 43687 bytes in 0.980 second response time [14:48:49] RECOVERY - cp3 Stunnel Http for mw1 on cp3 is OK: HTTP OK: HTTP/1.1 200 OK - 15248 bytes in 0.475 second response time [14:48:49] RECOVERY - test1 MediaWiki Rendering on test1 is OK: HTTP OK: HTTP/1.1 200 OK - 18673 bytes in 1.072 second response time [14:48:53] RECOVERY - mw2 MediaWiki Rendering on mw2 is OK: HTTP OK: HTTP/1.1 200 OK - 18675 bytes in 3.995 second response time [14:49:00] RECOVERY - mw3 MediaWiki Rendering on mw3 is OK: HTTP OK: HTTP/1.1 200 OK - 18674 bytes in 0.354 second response time [14:49:00] RECOVERY - cp4 Stunnel Http for mw1 on cp4 is OK: HTTP OK: HTTP/1.1 200 OK - 15256 bytes in 0.005 second response time [14:49:04] RECOVERY - cp3 Stunnel Http for mw2 on cp3 is OK: HTTP OK: HTTP/1.1 200 OK - 15263 bytes in 1.808 second response time [14:49:14] RECOVERY - cp4 Stunnel Http for misc2 on cp4 is OK: HTTP OK: HTTP/1.1 200 OK - 43687 bytes in 0.143 second response time [14:49:16] RECOVERY - misc2 HTTPS on misc2 is OK: HTTP OK: HTTP/1.1 200 OK - 43695 bytes in 0.072 second response time [14:49:32] RECOVERY - cp4 Varnish Backends on cp4 is OK: All 7 backends are healthy [14:49:33] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [14:49:38] RECOVERY - cp3 Varnish Backends on cp3 is OK: All 6 backends are healthy [14:49:43] RECOVERY - cp2 Varnish Backends on cp2 is OK: All 6 backends are healthy [14:50:41] RECOVERY - cp2 Stunnel Http for mw3 on cp2 is OK: HTTP OK: HTTP/1.1 200 OK - 15257 bytes in 4.189 second response time [14:52:18] PROBLEM - cp3 Puppet on cp3 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): File[wiki.exnihilolinux.org_private] [14:54:42] RECOVERY - cp3 Puppet on cp3 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [14:55:59] PROBLEM - mw2 MediaWiki Rendering on mw2 is CRITICAL: HTTP CRITICAL: HTTP/1.1 503 Backend fetch failed - 4217 bytes in 0.024 second response time [14:56:19] PROBLEM - cp4 Stunnel Http for mw1 on cp4 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [14:56:46] PROBLEM - cp3 Stunnel Http for mw2 on cp3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [14:56:57] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 6 datacenters are down: 107.191.126.23/cpweb, 2604:180:0:33b::2/cpweb, 128.199.139.216/cpweb, 2400:6180:0:d0::403:f001/cpweb, 81.4.109.133/cpweb, 2a00:d880:5:8ea::ebc7/cpweb [14:57:17] PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 6 datacenters are down: 107.191.126.23/cpweb, 2604:180:0:33b::2/cpweb, 128.199.139.216/cpweb, 2400:6180:0:d0::403:f001/cpweb, 81.4.109.133/cpweb, 2a00:d880:5:8ea::ebc7/cpweb [14:57:46] PROBLEM - lizardfs6 MediaWiki Rendering on lizardfs6 is CRITICAL: HTTP CRITICAL: HTTP/1.1 503 Backend fetch failed - 4219 bytes in 0.021 second response time [14:58:26] PROBLEM - mw1 MediaWiki Rendering on mw1 is CRITICAL: HTTP CRITICAL: HTTP/1.1 503 Backend fetch failed - 4219 bytes in 0.024 second response time [14:58:27] PROBLEM - cp2 Stunnel Http for mw1 on cp2 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [14:58:47] PROBLEM - cp4 Stunnel Http for mw2 on cp4 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [14:59:03] PROBLEM - cp2 Stunnel Http for misc2 on cp2 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [14:59:03] PROBLEM - test1 MediaWiki Rendering on test1 is CRITICAL: HTTP CRITICAL: HTTP/1.1 503 Backend fetch failed - 4219 bytes in 0.020 second response time [14:59:15] PROBLEM - cp2 Stunnel Http for mw2 on cp2 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [14:59:29] PROBLEM - cp3 Stunnel Http for misc2 on cp3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [14:59:35] PROBLEM - cp4 Stunnel Http for mw3 on cp4 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [14:59:48] PROBLEM - cp2 Stunnel Http for mw3 on cp2 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [14:59:49] PROBLEM - mw3 MediaWiki Rendering on mw3 is CRITICAL: HTTP CRITICAL: HTTP/1.1 503 Backend fetch failed - 4219 bytes in 1.030 second response time [14:59:50] PROBLEM - cp3 Stunnel Http for mw3 on cp3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [14:59:55] PROBLEM - cp4 Varnish Backends on cp4 is CRITICAL: 4 backends are down. lizardfs6 mw1 mw2 mw3 [15:00:06] PROBLEM - cp3 Stunnel Http for mw1 on cp3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [15:00:07] PROBLEM - cp3 Varnish Backends on cp3 is CRITICAL: 4 backends are down. lizardfs6 mw1 mw2 mw3 [15:00:08] PROBLEM - cp2 Varnish Backends on cp2 is CRITICAL: 4 backends are down. lizardfs6 mw1 mw2 mw3 [15:00:17] PROBLEM - misc2 HTTPS on misc2 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:00:29] PROBLEM - cp4 Stunnel Http for misc2 on cp4 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [15:01:44] RECOVERY - cp4 Stunnel Http for mw1 on cp4 is OK: HTTP OK: HTTP/1.1 200 OK - 15256 bytes in 0.003 second response time [15:02:26] RECOVERY - cp3 Stunnel Http for mw1 on cp3 is OK: HTTP OK: HTTP/1.1 200 OK - 15256 bytes in 0.501 second response time [15:06:52] RECOVERY - cp4 Stunnel Http for mw2 on cp4 is OK: HTTP OK: HTTP/1.1 200 OK - 15257 bytes in 0.004 second response time [15:07:05] RECOVERY - cp2 Stunnel Http for misc2 on cp2 is OK: HTTP OK: HTTP/1.1 200 OK - 43687 bytes in 3.551 second response time [15:07:14] RECOVERY - cp2 Stunnel Http for mw2 on cp2 is OK: HTTP OK: HTTP/1.1 200 OK - 15249 bytes in 0.294 second response time [15:07:20] RECOVERY - cp3 Stunnel Http for misc2 on cp3 is OK: HTTP OK: HTTP/1.1 200 OK - 43687 bytes in 3.424 second response time [15:07:53] RECOVERY - misc2 HTTPS on misc2 is OK: HTTP OK: HTTP/1.1 200 OK - 43695 bytes in 0.578 second response time [15:08:09] RECOVERY - lizardfs6 MediaWiki Rendering on lizardfs6 is OK: HTTP OK: HTTP/1.1 200 OK - 18668 bytes in 1.052 second response time [15:08:11] RECOVERY - cp4 Stunnel Http for misc2 on cp4 is OK: HTTP OK: HTTP/1.1 200 OK - 43687 bytes in 0.108 second response time [15:08:20] RECOVERY - mw1 MediaWiki Rendering on mw1 is OK: HTTP OK: HTTP/1.1 200 OK - 18667 bytes in 0.816 second response time [15:08:48] RECOVERY - test1 MediaWiki Rendering on test1 is OK: HTTP OK: HTTP/1.1 200 OK - 18674 bytes in 0.276 second response time [15:08:56] RECOVERY - cp2 Stunnel Http for mw1 on cp2 is OK: HTTP OK: HTTP/1.1 200 OK - 15256 bytes in 0.293 second response time [15:08:56] RECOVERY - mw2 MediaWiki Rendering on mw2 is OK: HTTP OK: HTTP/1.1 200 OK - 18675 bytes in 0.293 second response time [15:09:17] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [15:09:18] RECOVERY - cp3 Varnish Backends on cp3 is OK: All 6 backends are healthy [15:09:19] RECOVERY - mw3 MediaWiki Rendering on mw3 is OK: HTTP OK: HTTP/1.1 200 OK - 18668 bytes in 1.447 second response time [15:09:20] RECOVERY - cp4 Varnish Backends on cp4 is OK: All 7 backends are healthy [15:09:23] RECOVERY - cp4 Stunnel Http for mw3 on cp4 is OK: HTTP OK: HTTP/1.1 200 OK - 15263 bytes in 0.108 second response time [15:09:33] RECOVERY - cp2 Varnish Backends on cp2 is OK: All 6 backends are healthy [15:09:33] RECOVERY - cp3 Stunnel Http for mw3 on cp3 is OK: HTTP OK: HTTP/1.1 200 OK - 15263 bytes in 0.501 second response time [15:09:35] RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online [15:09:37] RECOVERY - cp2 Stunnel Http for mw3 on cp2 is OK: HTTP OK: HTTP/1.1 200 OK - 15257 bytes in 0.303 second response time [15:09:53] RECOVERY - cp3 Stunnel Http for mw2 on cp3 is OK: HTTP OK: HTTP/1.1 200 OK - 15263 bytes in 0.512 second response time [15:53:44] [02miraheze/landing] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/Jvv3r [15:53:45] [02miraheze/landing] 07paladox 03061986d - Update features-extensions-desc text Per @k6ka suggestion [15:53:47] [02landing] 07paladox created branch 03paladox-patch-1 - 13https://git.io/fxEQA [15:53:48] [02landing] 07paladox opened pull request 03#27: Update features-extensions-desc text - 13https://git.io/Jvv3K [15:54:35] miraheze/landing/paladox-patch-1/061986d - paladox The build passed. https://travis-ci.org/miraheze/landing/builds/634809852 [15:55:44] [02landing] 07paladox closed pull request 03#27: Update features-extensions-desc text - 13https://git.io/Jvv3K [15:55:45] [02miraheze/landing] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Jvv3P [15:55:47] [02miraheze/landing] 07paladox 030c70e3e - Update features-extensions-desc text (#27) Per @k6ka suggestion [15:55:48] [02miraheze/landing] 07paladox deleted branch 03paladox-patch-1 [15:55:50] [02landing] 07paladox deleted branch 03paladox-patch-1 - 13https://git.io/fxEQA [16:22:24] paladox: Hi! Happy new year. Sure, and thank you guys for running Miraheze. [16:24:26] jdstroy thank you! We've been in contact with our host who says the network is now stable! [16:24:31] so things should be resolved. [16:25:06] hooray :) [17:32:41] Has anyone done any cost analysis of using baremetal vs cloud services [17:33:45] My belief here from several years of industry experience is, it’s a terrible idea to purchase baremetal at this stage of growth [17:37:42] Well we've done cost analysis of Dedicated Servers, Colocation and Hybrid. Though i doin't think that doc is open. [17:37:55] SPF|Cloud and JohnLewis maybe we should open the doc for public viewing? [17:41:04] How did you do the cost analysis? B/c if you did it purely based on number of cores, amount of ram, etc. that doesn’t translate to cloud. You have to remember you have burst allocations on shared tenancy, which is incredibly valuable. Utilising managed services can be incredibly valuable too. If you’ve absolutely discounted a cloud native approach, [17:41:04] at least consider hybrid cloud. [17:48:15] RECOVERY - wiki.hyperborian.org - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.hyperborian.org' will expire on Wed 29 Jan 2020 12:45:45 PM GMT +0000. [17:52:14] PROBLEM - wiki.hyperborian.org - LetsEncrypt on sslhost is CRITICAL: connect to address wiki.hyperborian.org and port 443: Connection refusedHTTP CRITICAL - Unable to open TCP socket [17:53:31] PROBLEM - cp3 Disk Space on cp3 is WARNING: DISK WARNING - free space: / 2647 MB (10% inode=94%); [17:58:36] [02miraheze/ManageWiki] 07paladox pushed 031 commit to 03paladox-patch-2 [+0/-0/±1] 13https://git.io/JvvZN [17:58:38] [02miraheze/ManageWiki] 07paladox 03e10b211 - Convert mw_namespaces.ns_dbname and mw_permissions.perm_dbname to indexes [17:58:39] [02ManageWiki] 07paladox created branch 03paladox-patch-2 - 13https://git.io/vpSns [17:58:44] [02ManageWiki] 07paladox opened pull request 03#130: Convert mw_namespaces.ns_dbname and mw_permissions.perm_dbname to indexes - 13https://git.io/JvvZA [17:59:34] paladox; SPF’s decision but my understanding is the document is not entirely accurate anymore [17:59:59] ok [18:00:31] (the figures for colocation were underestimated) [18:00:42] [02ManageWiki] 07sonarcloud[bot] commented on pull request 03#130: Convert mw_namespaces.ns_dbname and mw_permissions.perm_dbname to indexes - 13https://git.io/Jvvne [18:00:48] [02ManageWiki] 07sonarcloud[bot] deleted a comment on pull request 03#130: Convert mw_namespaces.ns_dbname and mw_permissions.perm_dbname to indexes - 13https://git.io/Jvvne [18:00:49] [02ManageWiki] 07sonarcloud[bot] commented on pull request 03#130: Convert mw_namespaces.ns_dbname and mw_permissions.perm_dbname to indexes - 13https://git.io/Jvvnv [18:00:58] well [18:00:59] that's new [18:01:17] though, looking at the status of our campaign, that is not the only 'issue' we have [18:02:33] [02ManageWiki] 07sonarcloud[bot] deleted a comment on pull request 03#130: Convert mw_namespaces.ns_dbname and mw_permissions.perm_dbname to indexes - 13https://git.io/Jvvnv [18:02:38] [02ManageWiki] 07sonarcloud[bot] commented on pull request 03#130: Convert mw_namespaces.ns_dbname and mw_permissions.perm_dbname to indexes - 13https://github.com/miraheze/ManageWiki/pull/130#issuecomment-572680832 [18:02:39] [ Convert mw_namespaces.ns_dbname and mw_permissions.perm_dbname to indexes by paladox · Pull Request #130 · miraheze/ManageWiki · GitHub ] - github.com [18:12:12] yeh [18:12:49] [02miraheze/ManageWiki] 07paladox pushed 031 commit to 03paladox-patch-2 [+0/-0/±1] 13https://git.io/JvvnG [18:12:51] [02miraheze/ManageWiki] 07paladox 0358d33d7 - Update mw_permissions.sql [18:12:52] [02ManageWiki] 07paladox synchronize pull request 03#130: Convert mw_namespaces.ns_dbname and mw_permissions.perm_dbname to indexes - 13https://git.io/JvvZA [18:14:47] [02ManageWiki] 07sonarcloud[bot] deleted a comment on pull request 03#130: Convert mw_namespaces.ns_dbname and mw_permissions.perm_dbname to indexes - 13https://git.io/Jvvnk [18:14:48] [02ManageWiki] 07sonarcloud[bot] commented on pull request 03#130: Convert mw_namespaces.ns_dbname and mw_permissions.perm_dbname to indexes - 13https://git.io/JvvnW [18:14:55] [02ManageWiki] 07sonarcloud[bot] deleted a comment on pull request 03#130: Convert mw_namespaces.ns_dbname and mw_permissions.perm_dbname to indexes - 13https://git.io/JvvnW [18:14:57] [02ManageWiki] 07sonarcloud[bot] commented on pull request 03#130: Convert mw_namespaces.ns_dbname and mw_permissions.perm_dbname to indexes - 13https://git.io/Jvvn8 [18:15:31] good lord, the spam [18:16:38] [02ManageWiki] 07sonarcloud[bot] deleted a comment on pull request 03#130: Convert mw_namespaces.ns_dbname and mw_permissions.perm_dbname to indexes - 13https://git.io/Jvvn8 [18:16:40] [02ManageWiki] 07sonarcloud[bot] commented on pull request 03#130: Convert mw_namespaces.ns_dbname and mw_permissions.perm_dbname to indexes - 13https://git.io/JvvnE [18:18:04] [02miraheze/ManageWiki] 07paladox pushed 031 commit to 03paladox-patch-2 [+1/-0/±0] 13https://git.io/Jvvng [18:18:06] [02miraheze/ManageWiki] 07paladox 0344d94a7 - Create patch-namespaces-and-permission-indexes.sql [18:18:07] [02ManageWiki] 07paladox synchronize pull request 03#130: Convert mw_namespaces.ns_dbname and mw_permissions.perm_dbname to indexes - 13https://git.io/JvvZA [18:19:19] [02miraheze/ManageWiki] 07paladox pushed 031 commit to 03paladox-patch-2 [+0/-0/±1] 13https://git.io/Jvvn2 [18:19:20] [02miraheze/ManageWiki] 07paladox 03dedc1f8 - Update ManageWikiHooks.php [18:19:22] [02ManageWiki] 07paladox synchronize pull request 03#130: Convert mw_namespaces.ns_dbname and mw_permissions.perm_dbname to indexes - 13https://git.io/JvvZA [18:20:10] [02miraheze/ManageWiki] 07paladox pushed 031 commit to 03paladox-patch-2 [+1/-0/±0] 13https://git.io/Jvvna [18:20:11] [02miraheze/ManageWiki] 07paladox 03e817fce - Create patch-namespaces-add-indexes.sql [18:20:13] [02ManageWiki] 07paladox synchronize pull request 03#130: Convert mw_namespaces.ns_dbname and mw_permissions.perm_dbname to indexes - 13https://git.io/JvvZA [18:20:14] [02ManageWiki] 07sonarcloud[bot] deleted a comment on pull request 03#130: Convert mw_namespaces.ns_dbname and mw_permissions.perm_dbname to indexes - 13https://git.io/JvvnE [18:20:16] [02ManageWiki] 07sonarcloud[bot] commented on pull request 03#130: Convert mw_namespaces.ns_dbname and mw_permissions.perm_dbname to indexes - 13https://git.io/JvvnV [18:20:27] [02miraheze/ManageWiki] 07paladox pushed 031 commit to 03paladox-patch-2 [+1/-1/±0] 13https://git.io/Jvvnw [18:20:29] [02miraheze/ManageWiki] 07paladox 030f6b84e - Rename patch-namespaces-and-permission-indexes.sql to patch-permissions-add-indexes.sql [18:20:30] [02ManageWiki] 07paladox synchronize pull request 03#130: Convert mw_namespaces.ns_dbname and mw_permissions.perm_dbname to indexes - 13https://git.io/JvvZA [18:20:32] [02ManageWiki] 07sonarcloud[bot] deleted a comment on pull request 03#130: Convert mw_namespaces.ns_dbname and mw_permissions.perm_dbname to indexes - 13https://git.io/JvvnV [18:20:33] [02ManageWiki] 07sonarcloud[bot] commented on pull request 03#130: Convert mw_namespaces.ns_dbname and mw_permissions.perm_dbname to indexes - 13https://git.io/Jvvnr [18:20:42] [02miraheze/ManageWiki] 07paladox pushed 031 commit to 03paladox-patch-2 [+0/-0/±1] 13https://git.io/Jvvno [18:20:44] [02miraheze/ManageWiki] 07paladox 03b0b9026 - Update patch-permissions-add-indexes.sql [18:20:45] [02ManageWiki] 07paladox synchronize pull request 03#130: Convert mw_namespaces.ns_dbname and mw_permissions.perm_dbname to indexes - 13https://git.io/JvvZA [18:21:53] [02ManageWiki] 07sonarcloud[bot] deleted a comment on pull request 03#130: Convert mw_namespaces.ns_dbname and mw_permissions.perm_dbname to indexes - 13https://git.io/Jvvnr [18:21:55] [02ManageWiki] 07sonarcloud[bot] commented on pull request 03#130: Convert mw_namespaces.ns_dbname and mw_permissions.perm_dbname to indexes - 13https://git.io/Jvvn6 [18:22:39] [02ManageWiki] 07sonarcloud[bot] deleted a comment on pull request 03#130: Convert mw_namespaces.ns_dbname and mw_permissions.perm_dbname to indexes - 13https://git.io/Jvvn6 [18:22:41] [02ManageWiki] 07sonarcloud[bot] commented on pull request 03#130: Convert mw_namespaces.ns_dbname and mw_permissions.perm_dbname to indexes - 13https://git.io/JvvnP [18:22:54] miraheze/ManageWiki/paladox-patch-2/b0b9026 - paladox The build passed. https://travis-ci.org/miraheze/ManageWiki/builds/634875277 [18:23:52] [02ManageWiki] 07sonarcloud[bot] deleted a comment on pull request 03#130: Convert mw_namespaces.ns_dbname and mw_permissions.perm_dbname to indexes - 13https://git.io/JvvnP [18:23:53] [02ManageWiki] 07sonarcloud[bot] commented on pull request 03#130: Convert mw_namespaces.ns_dbname and mw_permissions.perm_dbname to indexes - 13https://git.io/Jvvn1 [18:24:30] [02ManageWiki] 07sonarcloud[bot] deleted a comment on pull request 03#130: Convert mw_namespaces.ns_dbname and mw_permissions.perm_dbname to indexes - 13https://git.io/Jvvn1 [18:24:31] [02ManageWiki] 07sonarcloud[bot] commented on pull request 03#130: Convert mw_namespaces.ns_dbname and mw_permissions.perm_dbname to indexes - 13https://git.io/JvvnM [18:45:34] [02ManageWiki] 07paladox deleted branch 03paladox-patch-1 - 13https://git.io/vpSns [18:45:36] [02miraheze/ManageWiki] 07paladox deleted branch 03paladox-patch-1 [18:45:39] [02ManageWiki] 07paladox deleted branch 03DBConnRef - 13https://git.io/vpSns [18:45:40] [02miraheze/ManageWiki] 07paladox deleted branch 03DBConnRef [18:57:37] [02ManageWiki] 07paladox closed pull request 03#130: Convert mw_namespaces.ns_dbname and mw_permissions.perm_dbname to indexes - 13https://git.io/JvvZA [18:57:38] [02miraheze/ManageWiki] 07paladox pushed 031 commit to 03master [+2/-0/±3] 13https://git.io/JvvcW [18:57:40] [02miraheze/ManageWiki] 07paladox 03901daeb - Convert mw_namespaces.ns_dbname and mw_permissions.perm_dbname to indexes (#130) * Convert mw_namespaces.ns_dbname and mw_permissions.perm_dbname to indexes * Update mw_permissions.sql * Create patch-namespaces-and-permission-indexes.sql * Update ManageWikiHooks.php * Create patch-namespaces-add-indexes.sql * Rename patch-namespaces-and-permission-indexes.sql to [18:57:40] patch-permissions-add-indexes.sql * Update patch-permissions-add-indexes.sql [18:57:41] [02miraheze/ManageWiki] 07paladox deleted branch 03paladox-patch-2 [18:57:43] [02ManageWiki] 07paladox deleted branch 03paladox-patch-2 - 13https://git.io/vpSns [18:58:34] [02miraheze/mediawiki] 07paladox pushed 031 commit to 03REL1_34 [+0/-0/±1] 13https://git.io/Jvvc8 [18:58:36] [02miraheze/mediawiki] 07paladox 03de6cd0e - Update ManageWiki [18:59:24] !log MariaDB [mhglobal]> ALTER TABLE /*$wgDBprefix*/mw_permissions ADD INDEX perm_dbname (perm_dbname); - db4 [18:59:35] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [18:59:49] !log MariaDB [mhglobal]> ALTER TABLE /*$wgDBprefix*/mw_namespaces ADD INDEX ns_dbname (ns_dbname); -db4 [18:59:58] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [19:04:29] [02miraheze/CreateWiki] 07paladox pushed 031 commit to 03paladox-patch-2 [+0/-0/±1] 13https://git.io/Jvvca [19:04:31] [02miraheze/CreateWiki] 07paladox 037722f8b - Convert cw_wikis.wiki_dbname to indexes [19:04:33] [02CreateWiki] 07paladox created branch 03paladox-patch-2 - 13https://git.io/vpJTL [19:04:35] [02CreateWiki] 07paladox opened pull request 03#127: Convert cw_wikis.wiki_dbname to indexes - 13https://git.io/JvvcV [19:06:50] [02miraheze/CreateWiki] 07paladox pushed 031 commit to 03paladox-patch-2 [+0/-0/±1] 13https://git.io/Jvvco [19:06:51] [02miraheze/CreateWiki] 07paladox 03c6fc7fe - Update CreateWikiHooks.php [19:06:53] [02CreateWiki] 07paladox synchronize pull request 03#127: Convert cw_wikis.wiki_dbname to indexes - 13https://git.io/JvvcV [19:07:24] miraheze/CreateWiki/paladox-patch-2/7722f8b - paladox The build was broken. https://travis-ci.com/miraheze/CreateWiki/builds/143785429 [19:07:43] [02miraheze/CreateWiki] 07paladox pushed 031 commit to 03paladox-patch-2 [+1/-0/±0] 13https://git.io/Jvvc6 [19:07:44] [02miraheze/CreateWiki] 07paladox 03f2704dc - Create patch-cw_wikis-add-indexes.sql [19:07:46] [02CreateWiki] 07paladox synchronize pull request 03#127: Convert cw_wikis.wiki_dbname to indexes - 13https://git.io/JvvcV [19:08:37] [02CreateWiki] 07sonarcloud[bot] commented on pull request 03#127: Convert cw_wikis.wiki_dbname to indexes - 13https://git.io/JvvcP [19:08:48] [02CreateWiki] 07sonarcloud[bot] deleted a comment on pull request 03#127: Convert cw_wikis.wiki_dbname to indexes - 13https://git.io/JvvcP [19:08:49] [02CreateWiki] 07sonarcloud[bot] commented on pull request 03#127: Convert cw_wikis.wiki_dbname to indexes - 13https://git.io/JvvcX [19:09:03] [02CreateWiki] 07sonarcloud[bot] deleted a comment on pull request 03#127: Convert cw_wikis.wiki_dbname to indexes - 13https://git.io/JvvcX [19:09:04] [02CreateWiki] 07sonarcloud[bot] commented on pull request 03#127: Convert cw_wikis.wiki_dbname to indexes - 13https://git.io/JvvcM [19:11:22] miraheze/CreateWiki/paladox-patch-2/c6fc7fe - paladox The build was broken. https://travis-ci.com/miraheze/CreateWiki/builds/143785725 [19:12:36] [02CreateWiki] 07sonarcloud[bot] deleted a comment on pull request 03#127: Convert cw_wikis.wiki_dbname to indexes - 13https://git.io/JvvcM [19:12:37] [02CreateWiki] 07sonarcloud[bot] commented on pull request 03#127: Convert cw_wikis.wiki_dbname to indexes - 13https://git.io/Jvvc5 [19:13:06] [02CreateWiki] 07sonarcloud[bot] deleted a comment on pull request 03#127: Convert cw_wikis.wiki_dbname to indexes - 13https://git.io/Jvvc5 [19:13:08] [02CreateWiki] 07sonarcloud[bot] commented on pull request 03#127: Convert cw_wikis.wiki_dbname to indexes - 13https://git.io/Jvvcd [19:14:48] [02CreateWiki] 07paladox closed pull request 03#126: Convert wfGetDB() calls to using getConnectionRef() - 13https://git.io/JeFJy [19:14:50] [02miraheze/CreateWiki] 07paladox deleted branch 03paladox-patch-1 [19:14:52] [02CreateWiki] 07paladox deleted branch 03paladox-patch-1 - 13https://git.io/vpJTL [19:15:14] miraheze/CreateWiki/paladox-patch-2/f2704dc - paladox The build is still failing. https://travis-ci.com/miraheze/CreateWiki/builds/143785850 [19:15:50] [02miraheze/CreateWiki] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Jvvcp [19:15:52] [02miraheze/CreateWiki] 07paladox 034208379 - Fix syntax [19:16:00] [02CreateWiki] 07paladox closed pull request 03#127: Convert cw_wikis.wiki_dbname to indexes - 13https://git.io/JvvcV [19:16:01] [02miraheze/CreateWiki] 07paladox pushed 031 commit to 03master [+1/-0/±2] 13https://git.io/Jvvch [19:16:03] [02miraheze/CreateWiki] 07paladox 03f700fba - Convert cw_wikis.wiki_dbname to indexes (#127) * Convert cw_wikis.wiki_dbname to indexes * Update CreateWikiHooks.php * Create patch-cw_wikis-add-indexes.sql [19:16:04] [02CreateWiki] 07sonarcloud[bot] deleted a comment on pull request 03#127: Convert cw_wikis.wiki_dbname to indexes - 13https://git.io/Jvvcd [19:16:06] [02CreateWiki] 07sonarcloud[bot] commented on pull request 03#127: Convert cw_wikis.wiki_dbname to indexes - 13https://git.io/Jvvcj [19:16:27] [02CreateWiki] 07sonarcloud[bot] deleted a comment on pull request 03#127: Convert cw_wikis.wiki_dbname to indexes - 13https://git.io/Jvvcj [19:16:28] [02CreateWiki] 07sonarcloud[bot] commented on pull request 03#127: Convert cw_wikis.wiki_dbname to indexes - 13https://git.io/JvvCe [19:16:31] [02miraheze/mediawiki] 07paladox pushed 031 commit to 03REL1_34 [+0/-0/±1] 13https://git.io/JvvCv [19:16:32] [02miraheze/mediawiki] 07paladox 0305c2bd1 - Update CreateWiki [19:17:43] !log MariaDB [mhglobal]> ALTER TABLE /*$wgDBprefix*/cw_wikis ADD INDEX wiki_dbname (wiki_dbname); -db4 [19:17:49] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [19:18:47] [02CreateWiki] 07paladox deleted branch 03paladox-patch-2 - 13https://git.io/vpJTL [19:18:48] .t 1 [19:18:48] 2020-01-09 - 19:18:48UTC [19:18:48] [02miraheze/CreateWiki] 07paladox deleted branch 03paladox-patch-2 [19:18:52] miraheze/CreateWiki/master/4208379 - paladox The build is still failing. https://travis-ci.com/miraheze/CreateWiki/builds/143786804 [19:18:56] .task [19:19:00] .task 9 [19:19:02] https://phabricator.miraheze.org/T9 - Install anti-spam extensions [Resolved] authored by Southparkfan, assigned to John [19:19:13] sorry ff is acting up it wont let me type in urls [19:19:29] these old tasks are so much fun [19:20:49] miraheze/CreateWiki/master/f700fba - paladox The build is still failing. https://travis-ci.com/miraheze/CreateWiki/builds/143786834 [19:24:05] I wish tasks were still like that :P [19:27:09] [02miraheze/MatomoAnalytics] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/JvvCC [19:27:11] [02miraheze/MatomoAnalytics] 07paladox 03f1ecc42 - Convert matomo.matomo_wiki to indexes [19:27:12] [02MatomoAnalytics] 07paladox created branch 03paladox-patch-1 - 13https://git.io/fN4LT [19:27:17] [02MatomoAnalytics] 07paladox opened pull request 03#15: Convert matomo.matomo_wiki to indexes - 13https://git.io/JvvCW [19:27:19] [02miraheze/ManageWiki] 07translatewiki pushed 031 commit to 03master [+0/-0/±2] 13https://git.io/JvvCl [19:27:20] [02miraheze/ManageWiki] 07translatewiki 0358c66af - Localisation updates from https://translatewiki.net. [19:27:21] [ Main page - translatewiki.net ] - translatewiki.net. [19:28:15] [02miraheze/MatomoAnalytics] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/JvvC4 [19:28:17] [02miraheze/MatomoAnalytics] 07paladox 035acdfa1 - Update MatomoAnalyticsHooks.php [19:28:18] [02MatomoAnalytics] 07paladox synchronize pull request 03#15: Convert matomo.matomo_wiki to indexes - 13https://git.io/JvvCW [19:29:06] [02miraheze/MatomoAnalytics] 07paladox pushed 031 commit to 03paladox-patch-1 [+1/-0/±0] 13https://git.io/JvvCR [19:29:08] [02miraheze/MatomoAnalytics] 07paladox 0370f52ff - Create patch-matomo-add-indexes.sql [19:29:09] [02MatomoAnalytics] 07paladox synchronize pull request 03#15: Convert matomo.matomo_wiki to indexes - 13https://git.io/JvvCW [19:29:23] miraheze/MatomoAnalytics/paladox-patch-1/f1ecc42 - paladox The build was fixed. https://travis-ci.org/miraheze/MatomoAnalytics/builds/634904384 [19:30:18] [02MatomoAnalytics] 07paladox closed pull request 03#10: Add composer.json - 13https://git.io/Je0TY [19:30:20] [02MatomoAnalytics] 07paladox reopened pull request 03#10: Add composer.json - 13https://git.io/Je0TY [19:31:12] [02MatomoAnalytics] 07sonarcloud[bot] commented on pull request 03#15: Convert matomo.matomo_wiki to indexes - 13https://git.io/JvvCa [19:32:56] [02MatomoAnalytics] 07sonarcloud[bot] deleted a comment on pull request 03#15: Convert matomo.matomo_wiki to indexes - 13https://git.io/JvvCa [19:32:58] [02MatomoAnalytics] 07sonarcloud[bot] commented on pull request 03#15: Convert matomo.matomo_wiki to indexes - 13https://git.io/JvvCr [19:33:20] [02MatomoAnalytics] 07sonarcloud[bot] deleted a comment on pull request 03#15: Convert matomo.matomo_wiki to indexes - 13https://git.io/JvvCr [19:33:22] [02MatomoAnalytics] 07sonarcloud[bot] commented on pull request 03#15: Convert matomo.matomo_wiki to indexes - 13https://git.io/JvvCi [19:33:37] [02MatomoAnalytics] 07paladox closed pull request 03#15: Convert matomo.matomo_wiki to indexes - 13https://git.io/JvvCW [19:33:38] [02miraheze/MatomoAnalytics] 07paladox pushed 031 commit to 03master [+1/-0/±2] 13https://git.io/JvvCX [19:33:40] [02miraheze/MatomoAnalytics] 07paladox 039d81fb7 - Convert matomo.matomo_wiki to indexes (#15) * Convert matomo.matomo_wiki to indexes * Update MatomoAnalyticsHooks.php * Create patch-matomo-add-indexes.sql [19:33:41] [02MatomoAnalytics] 07paladox deleted branch 03paladox-patch-1 - 13https://git.io/fN4LT [19:33:43] [02miraheze/MatomoAnalytics] 07paladox deleted branch 03paladox-patch-1 [19:34:09] [02miraheze/mediawiki] 07paladox pushed 031 commit to 03REL1_34 [+0/-0/±1] 13https://git.io/JvvCD [19:34:10] [02miraheze/mediawiki] 07paladox 03d159371 - Update MatomoAnalytics [19:34:53] !log ALTER TABLE /*$wgDBprefix*/matomo ADD INDEX matomo_wiki (matomo_wiki); - db4 [19:35:01] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [19:43:38] [02miraheze/MirahezeMagic] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JvvWT [19:43:39] [02miraheze/MirahezeMagic] 07paladox 036c57d3d - Convert gnf_files.files_dbname to indexes [19:43:42] uh [19:43:52] bugger, didn't click create pull [19:46:06] [02miraheze/MirahezeMagic] 07paladox pushed 031 commit to 03paladox-patch-3 [+0/-0/±1] 13https://git.io/JvvWt [19:46:07] [02miraheze/MirahezeMagic] 07paladox 03f543644 - Add sql patch file for adding indexes [19:46:09] [02MirahezeMagic] 07paladox created branch 03paladox-patch-3 - 13https://git.io/fQRGX [19:46:10] [02MirahezeMagic] 07paladox opened pull request 03#94: Add sql patch file for adding indexes - 13https://git.io/JvvWq [19:46:28] [02miraheze/MirahezeMagic] 07paladox pushed 031 commit to 03paladox-patch-3 [+1/-1/±0] 13https://git.io/JvvWm [19:46:30] [02miraheze/MirahezeMagic] 07paladox 03641595b - Rename sql/patch-gnf_files-binary.sql to sql/patches/patch-gnf_files-binary.sql [19:46:31] [02MirahezeMagic] 07paladox synchronize pull request 03#94: Add sql patch file for adding indexes - 13https://git.io/JvvWq [19:47:08] miraheze/MirahezeMagic/paladox-patch-3/f543644 - paladox The build passed. https://travis-ci.org/miraheze/MirahezeMagic/builds/634913357 [19:47:18] miraheze/MirahezeMagic/paladox-patch-3/f543644 - paladox The build passed. https://travis-ci.com/miraheze/MirahezeMagic/builds/143790617 [19:47:20] [02miraheze/MirahezeMagic] 07paladox pushed 031 commit to 03paladox-patch-3 [+1/-0/±0] 13https://git.io/JvvW3 [19:47:21] [02miraheze/MirahezeMagic] 07paladox 037b5704e - Create patch-gnf_files-add-indexes.sql [19:47:23] [02MirahezeMagic] 07paladox synchronize pull request 03#94: Add sql patch file for adding indexes - 13https://git.io/JvvWq [19:48:02] miraheze/MirahezeMagic/paladox-patch-3/641595b - paladox The build passed. https://travis-ci.org/miraheze/MirahezeMagic/builds/634913538 [19:50:40] [02miraheze/MirahezeMagic] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JvvWG [19:50:41] [02miraheze/MirahezeMagic] 07paladox 031f096c7 - Revert "Convert gnf_files.files_dbname to indexes" This reverts commit 6c57d3d3c9b271a58de4c1ec037185e2f83c1d70. [19:50:51] [02MirahezeMagic] 07paladox closed pull request 03#94: Add sql patch file for adding indexes - 13https://git.io/JvvWq [19:50:54] [02miraheze/MirahezeMagic] 07paladox deleted branch 03paladox-patch-3 [19:50:56] [02MirahezeMagic] 07paladox deleted branch 03paladox-patch-3 - 13https://git.io/fQRGX [20:05:21] PROBLEM - db4 Disk Space on db4 is WARNING: DISK WARNING - free space: / 24019 MB (6% inode=95%); [20:15:08] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JvvWH [20:15:10] [02miraheze/services] 07MirahezeSSLBot 03611c825 - BOT: Updating services config for wikis [21:01:39] [02miraheze/MatomoAnalytics] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/Jvv8l [21:01:41] [02miraheze/MatomoAnalytics] 07paladox 03ad12c28 - Add caching to matomo [21:01:42] [02MatomoAnalytics] 07paladox created branch 03paladox-patch-1 - 13https://git.io/fN4LT [21:01:44] [02MatomoAnalytics] 07paladox opened pull request 03#16: Add caching to matomo - 13https://git.io/Jvv88 [21:03:48] [02MatomoAnalytics] 07sonarcloud[bot] commented on pull request 03#16: Add caching to matomo - 13https://git.io/Jvv8g [21:03:53] [02MatomoAnalytics] 07sonarcloud[bot] deleted a comment on pull request 03#16: Add caching to matomo - 13https://git.io/Jvv8g [21:03:54] [02MatomoAnalytics] 07sonarcloud[bot] commented on pull request 03#16: Add caching to matomo - 13https://git.io/Jvv8a [21:04:00] [02MatomoAnalytics] 07sonarcloud[bot] deleted a comment on pull request 03#16: Add caching to matomo - 13https://git.io/Jvv8a [21:04:02] [02MatomoAnalytics] 07sonarcloud[bot] commented on pull request 03#16: Add caching to matomo - 13https://git.io/Jvv8V [21:05:43] miraheze/MatomoAnalytics/paladox-patch-1/ad12c28 - paladox The build was broken. https://travis-ci.org/miraheze/MatomoAnalytics/builds/634952030 [21:06:21] [02MatomoAnalytics] 07paladox synchronize pull request 03#16: Add caching to matomo - 13https://git.io/Jvv88 [21:06:23] [02miraheze/MatomoAnalytics] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/Jvv8K [21:06:24] [02miraheze/MatomoAnalytics] 07paladox 03f5edeb8 - Update MatomoAnalytics.php [21:08:17] [02MatomoAnalytics] 07sonarcloud[bot] deleted a comment on pull request 03#16: Add caching to matomo - 13https://git.io/Jvv8V [21:08:18] [02MatomoAnalytics] 07sonarcloud[bot] commented on pull request 03#16: Add caching to matomo - 13https://git.io/Jvv8y [21:08:24] [02MatomoAnalytics] 07sonarcloud[bot] deleted a comment on pull request 03#16: Add caching to matomo - 13https://git.io/Jvv8y [21:08:25] [02MatomoAnalytics] 07sonarcloud[bot] commented on pull request 03#16: Add caching to matomo - 13https://git.io/Jvv89 [21:08:32] miraheze/MatomoAnalytics/paladox-patch-1/f5edeb8 - paladox The build was fixed. https://travis-ci.org/miraheze/MatomoAnalytics/builds/634955713 [21:09:35] [02MatomoAnalytics] 07paladox synchronize pull request 03#16: Add caching to matomo - 13https://git.io/Jvv88 [21:09:36] [02miraheze/MatomoAnalytics] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/Jvv8d [21:09:38] [02miraheze/MatomoAnalytics] 07paladox 036cee682 - Update MatomoAnalytics.php [21:10:04] [02MatomoAnalytics] 07sonarcloud[bot] deleted a comment on pull request 03#16: Add caching to matomo - 13https://git.io/Jvv89 [21:10:06] [02MatomoAnalytics] 07sonarcloud[bot] commented on pull request 03#16: Add caching to matomo - 13https://git.io/Jvv8N [21:11:33] [02MatomoAnalytics] 07sonarcloud[bot] deleted a comment on pull request 03#16: Add caching to matomo - 13https://git.io/Jvv8N [21:11:34] [02MatomoAnalytics] 07sonarcloud[bot] commented on pull request 03#16: Add caching to matomo - 13https://git.io/Jvv8h [21:11:38] [02MatomoAnalytics] 07sonarcloud[bot] deleted a comment on pull request 03#16: Add caching to matomo - 13https://git.io/Jvv8h [21:11:40] [02MatomoAnalytics] 07sonarcloud[bot] commented on pull request 03#16: Add caching to matomo - 13https://git.io/Jvv8j [21:11:44] [02MatomoAnalytics] 07sonarcloud[bot] deleted a comment on pull request 03#16: Add caching to matomo - 13https://git.io/Jvv8j [21:11:46] [02MatomoAnalytics] 07sonarcloud[bot] commented on pull request 03#16: Add caching to matomo - 13https://git.io/Jvv4e [21:12:38] [02MatomoAnalytics] 07paladox synchronize pull request 03#16: Add caching to matomo - 13https://git.io/Jvv88 [21:12:39] [02miraheze/MatomoAnalytics] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/Jvv4T [21:12:41] [02miraheze/MatomoAnalytics] 07paladox 03a520fa1 - Update MatomoAnalytics.php [21:14:56] [02MatomoAnalytics] 07sonarcloud[bot] deleted a comment on pull request 03#16: Add caching to matomo - 13https://git.io/Jvv4e [21:14:58] [02MatomoAnalytics] 07sonarcloud[bot] commented on pull request 03#16: Add caching to matomo - 13https://git.io/Jvv4G [21:15:08] [02MatomoAnalytics] 07sonarcloud[bot] deleted a comment on pull request 03#16: Add caching to matomo - 13https://git.io/Jvv4G [21:15:10] [02MatomoAnalytics] 07sonarcloud[bot] commented on pull request 03#16: Add caching to matomo - 13https://git.io/Jvv4Z [21:15:14] [02MatomoAnalytics] 07sonarcloud[bot] deleted a comment on pull request 03#16: Add caching to matomo - 13https://git.io/Jvv4Z [21:15:15] [02MatomoAnalytics] 07sonarcloud[bot] commented on pull request 03#16: Add caching to matomo - 13https://git.io/Jvv4n [21:18:37] [02MatomoAnalytics] 07paladox closed pull request 03#16: Add caching to matomo - 13https://git.io/Jvv88 [21:18:39] [02miraheze/MatomoAnalytics] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Jvv40 [21:18:40] [02miraheze/MatomoAnalytics] 07paladox 03b2b98b0 - Add caching to matomo (#16) * Add caching to matomo * Update MatomoAnalytics.php * Update MatomoAnalytics.php * Update MatomoAnalytics.php [21:18:42] [02MatomoAnalytics] 07paladox deleted branch 03paladox-patch-1 - 13https://git.io/fN4LT [21:18:43] [02miraheze/MatomoAnalytics] 07paladox deleted branch 03paladox-patch-1 [21:19:11] [02miraheze/mediawiki] 07paladox pushed 031 commit to 03REL1_34 [+0/-0/±1] 13https://git.io/Jvv4E [21:19:13] [02miraheze/mediawiki] 07paladox 0336ccf8d - Update MatomoAnalytics [21:20:27] !log rebuild lc on mw* and lizardfs6 [21:20:35] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [21:46:31] [02miraheze/mw-config] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/JvvB8 [21:46:33] [02miraheze/mw-config] 07paladox 031027d06 - Add wgDiscordNotificationNewUser to ManageWiki Per request through email [21:46:34] [02mw-config] 07paladox created branch 03paladox-patch-1 - 13https://git.io/vbvb3 [21:46:36] [02mw-config] 07paladox opened pull request 03#2854: Add wgDiscordNotificationNewUser to ManageWiki - 13https://git.io/JvvB4 [21:53:07] paladox: what about putting thr rest on? [21:53:32] i haven't really looked into that [21:53:40] only got a request for that one through email [21:58:17] [02miraheze/mw-config] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/JvvBX [21:58:18] [02miraheze/mw-config] 07paladox 039cf947f - Update ManageWikiSettings.php [21:58:20] [02mw-config] 07paladox synchronize pull request 03#2854: Add wgDiscordNotificationNewUser to ManageWiki - 13https://git.io/JvvB4 [21:59:26] RhinosF1 feel free to create a pr though :) [22:01:01] paladox: I’m mobile and will probably forget by next time I use PC [22:01:31] ok, maybe create a task? [22:01:54] [02mw-config] 07paladox closed pull request 03#2854: Add wgDiscordNotificationNewUser to ManageWiki - 13https://git.io/JvvB4 [22:01:55] [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±2] 13https://git.io/JvvB5 [22:01:57] [02miraheze/mw-config] 07paladox 034c9d4a9 - Add wgDiscordNotificationNewUser to ManageWiki (#2854) * Add wgDiscordNotificationNewUser to ManageWiki Per request through email * Update ManageWikiSettings.php [22:01:58] * RhinosF1 will do later [22:01:58] [02miraheze/mw-config] 07paladox deleted branch 03paladox-patch-1 [22:02:00] [02mw-config] 07paladox deleted branch 03paladox-patch-1 - 13https://git.io/vbvb3 [22:03:26] paladox: its on my task list [22:04:35] ok [22:07:05] !log sudo -u www-data php /srv/mediawiki/w/maintenance/importDump.php --wiki=inforevivalwiki --report=1 Wiki-20200109132453.xml [22:07:12] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [23:03:57] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 2 datacenters are down: 2604:180:0:33b::2/cpweb, 128.199.139.216/cpweb [23:05:54] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [23:14:04] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 2 datacenters are down: 2400:6180:0:d0::403:f001/cpweb, 81.4.109.133/cpweb [23:15:09] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JvvuZ [23:15:11] [02miraheze/services] 07MirahezeSSLBot 03cf81451 - BOT: Updating services config for wikis [23:16:03] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [23:22:26] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 4 datacenters are down: 2604:180:0:33b::2/cpweb, 128.199.139.216/cpweb, 2400:6180:0:d0::403:f001/cpweb, 81.4.109.133/cpweb [23:22:45] PROBLEM - cp3 Stunnel Http for mw3 on cp3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [23:23:03] PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 2 datacenters are down: 2604:180:0:33b::2/cpweb, 81.4.109.133/cpweb [23:24:47] RECOVERY - cp3 Stunnel Http for mw3 on cp3 is OK: HTTP OK: HTTP/1.1 200 OK - 15196 bytes in 0.829 second response time [23:26:24] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [23:27:02] RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online [23:58:14] [02puppet] 07The-Voidwalker opened pull request 03#1177: Add Access-Control-Allow-Origin to sfnt (font) files - 13https://git.io/JvvzW