[00:03:31] PROBLEM - cp2 Current Load on cp2 is WARNING: WARNING - load average: 0.96, 1.73, 1.09 [00:05:31] RECOVERY - cp2 Current Load on cp2 is OK: OK - load average: 0.16, 0.94, 0.91 [00:46:05] PROBLEM - cp2 Varnish Backends on cp2 is CRITICAL: 3 backends are down. mw1 mw2 mw3 [00:46:08] PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 5 datacenters are down: 107.191.126.23/cpweb, 128.199.139.216/cpweb, 2400:6180:0:d0::403:f001/cpweb, 81.4.109.133/cpweb, 2a00:d880:5:8ea::ebc7/cpweb [00:46:19] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 1 datacenter is down: 81.4.109.133/cpweb [00:46:38] PROBLEM - cp3 Varnish Backends on cp3 is CRITICAL: 1 backends are down. mw3 [00:48:08] RECOVERY - cp2 Varnish Backends on cp2 is OK: All 7 backends are healthy [00:48:10] RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online [00:48:44] RECOVERY - cp3 Varnish Backends on cp3 is OK: All 6 backends are healthy [00:51:20] PROBLEM - cp3 Stunnel Http for mw1 on cp3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:52:07] PROBLEM - cp4 Varnish Backends on cp4 is CRITICAL: 2 backends are down. mw1 mw2 [00:52:36] PROBLEM - cp2 Varnish Backends on cp2 is CRITICAL: 1 backends are down. mw2 [00:53:16] RECOVERY - cp3 Stunnel Http for mw1 on cp3 is OK: HTTP OK: HTTP/1.1 200 OK - 15295 bytes in 0.512 second response time [00:54:07] RECOVERY - cp4 Varnish Backends on cp4 is OK: All 7 backends are healthy [00:54:38] RECOVERY - cp2 Varnish Backends on cp2 is OK: All 7 backends are healthy [00:54:39] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [01:54:56] PROBLEM - cp2 Current Load on cp2 is WARNING: WARNING - load average: 0.57, 1.86, 1.45 [01:56:55] RECOVERY - cp2 Current Load on cp2 is OK: OK - load average: 0.42, 0.96, 1.16 [05:27:56] PROBLEM - wiki.8b8t.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.8b8t.com' expires in 15 day(s) (Fri 21 Feb 2020 05:25:26 AM GMT +0000). [05:28:10] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JvGQV [05:28:11] [02miraheze/ssl] 07MirahezeSSLBot 039a361a7 - Bot: Update SSL cert for wiki.8b8t.com [05:31:54] RECOVERY - wiki.8b8t.com - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.8b8t.com' will expire on Tue 05 May 2020 04:28:03 AM GMT +0000. [06:26:46] RECOVERY - cp3 Disk Space on cp3 is OK: DISK OK - free space: / 3016 MB (12% inode=94%); [11:47:17] Hi! Here is the list of currently open high priority tasks on Phabricator [11:47:24] No updates for 8 days - https://phabricator.miraheze.org/T5174 - enc.for.uz itermittently using both yandex and miraheze dns - authored by Paladox, assigned to Sf7_uz [11:47:30] No updates for 7 days - https://phabricator.miraheze.org/T5130 - Internal error - authored by Kadmin, assigned to None [11:56:46] PROBLEM - cp3 Disk Space on cp3 is WARNING: DISK WARNING - free space: / 2647 MB (10% inode=94%); [13:33:36] PROBLEM - cp4 Current Load on cp4 is WARNING: WARNING - load average: 1.33, 1.79, 1.19 [13:35:34] RECOVERY - cp4 Current Load on cp4 is OK: OK - load average: 0.59, 1.10, 1.04 [15:40:09] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JvZvS [15:40:11] [02miraheze/services] 07MirahezeSSLBot 03114e692 - BOT: Updating services config for wikis [18:19:40] PROBLEM - cp2 Stunnel Http for mw2 on cp2 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [18:19:42] PROBLEM - cp3 Stunnel Http for mw2 on cp3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [18:20:40] PROBLEM - cp2 Varnish Backends on cp2 is CRITICAL: 4 backends are down. lizardfs6 mw1 mw2 mw3 [18:20:40] PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 6 datacenters are down: 107.191.126.23/cpweb, 2604:180:0:33b::2/cpweb, 128.199.139.216/cpweb, 2400:6180:0:d0::403:f001/cpweb, 81.4.109.133/cpweb, 2a00:d880:5:8ea::ebc7/cpweb [18:20:44] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 6 datacenters are down: 107.191.126.23/cpweb, 2604:180:0:33b::2/cpweb, 128.199.139.216/cpweb, 2400:6180:0:d0::403:f001/cpweb, 81.4.109.133/cpweb, 2a00:d880:5:8ea::ebc7/cpweb [18:20:56] PROBLEM - lizardfs6 Current Load on lizardfs6 is CRITICAL: CRITICAL - load average: 16.93, 11.09, 6.96 [18:20:57] PROBLEM - cp3 Varnish Backends on cp3 is CRITICAL: 4 backends are down. lizardfs6 mw1 mw2 mw3 [18:21:46] RECOVERY - cp2 Stunnel Http for mw2 on cp2 is OK: HTTP OK: HTTP/1.1 200 OK - 15288 bytes in 2.584 second response time [18:21:49] RECOVERY - cp3 Stunnel Http for mw2 on cp3 is OK: HTTP OK: HTTP/1.1 200 OK - 15302 bytes in 1.040 second response time [18:22:36] RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online [18:22:40] RECOVERY - cp2 Varnish Backends on cp2 is OK: All 7 backends are healthy [18:22:43] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [18:22:52] RECOVERY - cp3 Varnish Backends on cp3 is OK: All 6 backends are healthy [18:22:55] RECOVERY - lizardfs6 Current Load on lizardfs6 is OK: OK - load average: 8.31, 10.05, 7.11 [18:26:34] PROBLEM - misc2 Puppet on misc2 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 5 minutes ago with 0 failures [18:27:36] paladox|UKInEU: what you blowing up? [18:27:42] nothing. [18:28:29] paladox|UKInEU: good. Saw things go critical and as soon as it recovered you stopped puppet [18:48:09] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JvZIG [18:48:10] [02miraheze/puppet] 07paladox 03cefe678 - php: Do not use php::extension to install php-mail-mime [19:19:12] PROBLEM - prfm.wiki - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Certificate 'prfm.wiki' expired on Wed 29 Jan 2020 01:07:14 PM GMT +0000. [19:23:06] RECOVERY - prfm.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'prfm.wiki' will expire on Tue 10 Mar 2020 11:59:59 PM GMT +0000. [19:31:30] !log initsitestates and rebuildrc for ungamewiki after import [19:31:36] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [19:33:58] PROBLEM - mw3 Puppet on mw3 is CRITICAL: connect to address 81.4.121.113 port 5666: Connection refusedconnect to host 81.4.121.113 port 5666: Connection refused [19:34:11] PROBLEM - mw3 Current Load on mw3 is CRITICAL: connect to address 81.4.121.113 port 5666: Connection refusedconnect to host 81.4.121.113 port 5666: Connection refused [19:34:33] PROBLEM - mw3 Disk Space on mw3 is CRITICAL: connect to address 81.4.121.113 port 5666: Connection refusedconnect to host 81.4.121.113 port 5666: Connection refused [19:35:51] PROBLEM - mw3 php-fpm on mw3 is CRITICAL: connect to address 81.4.121.113 port 5666: Connection refusedconnect to host 81.4.121.113 port 5666: Connection refused [19:37:19] hmm [19:37:57] RECOVERY - mw3 Puppet on mw3 is OK: OK: Puppet is currently enabled, last run 26 minutes ago with 0 failures [19:38:11] RECOVERY - mw3 Current Load on mw3 is OK: OK - load average: 4.41, 3.62, 3.39 [19:38:33] RECOVERY - mw3 Disk Space on mw3 is OK: DISK OK - free space: / 60130 MB (83% inode=93%); [19:39:15] !log restart php7.3-fpm on mw3 [19:39:21] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [19:39:50] RECOVERY - mw3 php-fpm on mw3 is OK: PROCS OK: 7 processes with command name 'php-fpm7.3' [19:43:24] !log upgrade php on mw3 [19:43:39] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [19:46:26] !log upgrade php on mw2 [19:46:37] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [19:47:43] !log upgrade php on mw1 [19:48:00] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [19:48:49] !log upgrade php on lizardfs6 [19:49:15] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [19:49:47] PROBLEM - cp4 Varnish Backends on cp4 is CRITICAL: 1 backends are down. lizardfs6 [19:51:42] RECOVERY - cp4 Varnish Backends on cp4 is OK: All 7 backends are healthy [19:52:55] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JvZLd [19:52:56] [02miraheze/puppet] 07paladox 03b33fec0 - lizardfs6: Lower childs to 36 [19:58:29] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JvZtf [19:58:31] [02miraheze/puppet] 07paladox 03c5bb106 - lizardfs6: cut childs even furthur [20:04:06] Funny question. is there a way in the current or previous version of Mediawiki for interwikis not to work or to be different in a certain namespace? that's because in Wikia that happens at least in the User namespace. [20:24:10] hispano76: wikia uses a very old highly adapted version of mediawiki [20:24:34] I’m not aware of that being possible without them hacking something [20:30:22] Ah, then it must be a "hack". thanks RhinosF1 [21:40:31] PROBLEM - cp2 Varnish Backends on cp2 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [21:42:29] RECOVERY - cp2 Varnish Backends on cp2 is OK: All 7 backends are healthy [22:34:19] PROBLEM - cp2 Current Load on cp2 is CRITICAL: CRITICAL - load average: 8.20, 3.84, 1.83 [22:36:18] PROBLEM - cp2 Current Load on cp2 is WARNING: WARNING - load average: 0.46, 1.95, 1.51 [22:38:17] RECOVERY - cp2 Current Load on cp2 is OK: OK - load average: 0.05, 0.91, 1.19 [22:44:11] PROBLEM - mw3 Current Load on mw3 is CRITICAL: connect to address 81.4.121.113 port 5666: Connection refusedconnect to host 81.4.121.113 port 5666: Connection refused [22:44:35] PROBLEM - mw3 Disk Space on mw3 is CRITICAL: connect to address 81.4.121.113 port 5666: Connection refusedconnect to host 81.4.121.113 port 5666: Connection refused [22:45:49] PROBLEM - mw3 php-fpm on mw3 is CRITICAL: connect to address 81.4.121.113 port 5666: Connection refusedconnect to host 81.4.121.113 port 5666: Connection refused [22:45:55] PROBLEM - mw3 Puppet on mw3 is CRITICAL: connect to address 81.4.121.113 port 5666: Connection refusedconnect to host 81.4.121.113 port 5666: Connection refused [22:52:06] RECOVERY - mw3 php-fpm on mw3 is OK: PROCS OK: 7 processes with command name 'php-fpm7.3' [22:52:07] RECOVERY - mw3 Puppet on mw3 is OK: OK: Puppet is currently enabled, last run 23 seconds ago with 0 failures [22:52:19] RECOVERY - mw3 Current Load on mw3 is OK: OK - load average: 2.01, 3.18, 3.95 [22:52:36] RECOVERY - mw3 Disk Space on mw3 is OK: DISK OK - free space: / 60066 MB (83% inode=93%); [22:53:43] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 4 datacenters are down: 2604:180:0:33b::2/cpweb, 2400:6180:0:d0::403:f001/cpweb, 81.4.109.133/cpweb, 2a00:d880:5:8ea::ebc7/cpweb [22:54:59] PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 2 datacenters are down: 107.191.126.23/cpweb, 2400:6180:0:d0::403:f001/cpweb [22:55:48] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [22:55:53] PROBLEM - cp3 Stunnel Http for mw2 on cp3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [22:56:55] RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online [22:57:52] RECOVERY - cp3 Stunnel Http for mw2 on cp3 is OK: HTTP OK: HTTP/1.1 200 OK - 15288 bytes in 0.515 second response time [23:07:01] PROBLEM - mw2 Puppet on mw2 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [23:23:50] RECOVERY - mw2 Puppet on mw2 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures