[00:16:11] PROBLEM - mw3 Current Load on mw3 is CRITICAL: CRITICAL - load average: 8.88, 7.70, 6.63 [00:18:11] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 7.36, 7.56, 6.71 [00:22:12] PROBLEM - mw3 Current Load on mw3 is CRITICAL: CRITICAL - load average: 9.97, 8.39, 7.19 [00:26:10] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 6.38, 7.83, 7.27 [00:34:07] PROBLEM - mw3 Current Load on mw3 is CRITICAL: CRITICAL - load average: 8.80, 7.44, 7.19 [00:42:06] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 7.53, 7.98, 7.69 [00:56:06] PROBLEM - contraao.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'contraao.com' expires in 15 day(s) (Mon 11 Nov 2019 12:53:57 AM GMT +0000). [00:56:10] PROBLEM - wiki.contraao.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'contraao.com' expires in 15 day(s) (Mon 11 Nov 2019 12:53:57 AM GMT +0000). [00:56:24] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JeEOx [00:56:26] [02miraheze/ssl] 07MirahezeSSLBot 0355264d1 - Bot: Update SSL cert for contraao.com [00:58:12] PROBLEM - mw3 Current Load on mw3 is CRITICAL: CRITICAL - load average: 10.24, 7.98, 7.39 [00:59:26] !log depool mw3 [00:59:32] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [01:00:13] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 5.04, 7.09, 7.15 [01:00:24] !log repool mw3 [01:00:31] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [01:04:04] RECOVERY - contraao.com - LetsEncrypt on sslhost is OK: OK - Certificate 'contraao.com' will expire on Thu 23 Jan 2020 11:56:16 PM GMT +0000. [01:04:09] RECOVERY - wiki.contraao.com - LetsEncrypt on sslhost is OK: OK - Certificate 'contraao.com' will expire on Thu 23 Jan 2020 11:56:16 PM GMT +0000. [01:04:20] !log depool mw3 [01:04:25] PROBLEM - mw1 Current Load on mw1 is CRITICAL: CRITICAL - load average: 10.71, 7.98, 6.16 [01:04:28] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [01:05:25] !log repool mw3 [01:05:33] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [01:06:11] RECOVERY - mw3 Current Load on mw3 is OK: OK - load average: 6.16, 6.23, 6.74 [01:08:17] PROBLEM - mw1 Current Load on mw1 is WARNING: WARNING - load average: 5.02, 7.39, 6.43 [01:08:53] PROBLEM - mw3 Puppet on mw3 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 5 minutes ago with 0 failures [01:12:06] RECOVERY - mw1 Current Load on mw1 is OK: OK - load average: 5.97, 6.73, 6.36 [01:16:51] RECOVERY - mw3 Puppet on mw3 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [01:23:18] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 6.14, 6.83, 6.73 [01:25:18] RECOVERY - mw3 Current Load on mw3 is OK: OK - load average: 5.82, 6.48, 6.61 [01:40:16] PROBLEM - mw3 Current Load on mw3 is CRITICAL: CRITICAL - load average: 8.81, 7.50, 6.78 [01:42:16] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 7.22, 7.22, 6.75 [01:46:15] RECOVERY - mw3 Current Load on mw3 is OK: OK - load average: 5.78, 6.55, 6.58 [01:48:29] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JeE3B [01:48:30] [02miraheze/puppet] 07paladox 03c40b6c5 - varnish: Tweak nginx config [01:52:15] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JeE32 [01:52:16] [02miraheze/puppet] 07paladox 0302b91df - Update nginx.conf.erb [01:53:52] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JeE3w [01:53:53] [02miraheze/puppet] 07paladox 03668f3f2 - Update mediawiki.conf [02:00:54] PROBLEM - mw2 Current Load on mw2 is WARNING: WARNING - load average: 7.07, 6.35, 4.82 [02:02:53] RECOVERY - mw2 Current Load on mw2 is OK: OK - load average: 4.80, 5.87, 4.84 [02:05:02] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JeE3H [02:05:04] [02miraheze/puppet] 07paladox 037df61bd - varnish: Increase http_max_hdr to 128 [02:07:47] PROBLEM - mw1 Puppet on mw1 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [02:13:47] RECOVERY - mw1 Puppet on mw1 is OK: OK: Puppet is currently enabled, last run 53 seconds ago with 0 failures [02:23:12] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 7.97, 7.31, 6.71 [02:24:40] PROBLEM - mw3 Disk Space on mw3 is WARNING: DISK WARNING - free space: / 5438 MB (7% inode=99%); [02:26:39] RECOVERY - mw3 Disk Space on mw3 is OK: DISK OK - free space: / 10577 MB (13% inode=99%); [02:27:11] PROBLEM - mw3 Current Load on mw3 is CRITICAL: CRITICAL - load average: 8.05, 7.63, 6.97 [02:29:10] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 6.91, 7.43, 6.98 [02:35:12] RECOVERY - mw3 Current Load on mw3 is OK: OK - load average: 5.65, 6.46, 6.72 [02:40:49] PROBLEM - mw2 Current Load on mw2 is CRITICAL: CRITICAL - load average: 8.27, 5.98, 4.86 [02:41:12] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 7.76, 7.05, 6.85 [02:42:49] PROBLEM - mw2 Current Load on mw2 is WARNING: WARNING - load average: 6.87, 6.36, 5.14 [02:43:42] PROBLEM - mw1 Current Load on mw1 is WARNING: WARNING - load average: 7.03, 6.56, 5.99 [02:44:49] RECOVERY - mw2 Current Load on mw2 is OK: OK - load average: 4.10, 5.51, 4.97 [02:45:42] RECOVERY - mw1 Current Load on mw1 is OK: OK - load average: 6.38, 6.37, 5.98 [02:49:09] RECOVERY - mw3 Current Load on mw3 is OK: OK - load average: 6.03, 6.76, 6.80 [03:05:16] PROBLEM - bacula1 Bacula Phabricator Static on bacula1 is WARNING: WARNING: Full, 81004 files, 2.632GB, 2019-10-11 03:03:00 (2.1 weeks ago) [03:11:24] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 7.67, 6.96, 6.62 [03:13:24] RECOVERY - mw3 Current Load on mw3 is OK: OK - load average: 6.01, 6.62, 6.54 [03:20:14] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JeEsF [03:20:15] [02miraheze/services] 07MirahezeSSLBot 0342c0805 - BOT: Updating services config for wikis [04:05:43] PROBLEM - mw1 Current Load on mw1 is CRITICAL: CRITICAL - load average: 9.24, 7.45, 5.70 [04:07:42] PROBLEM - mw1 Current Load on mw1 is WARNING: WARNING - load average: 7.57, 7.73, 6.04 [04:09:43] PROBLEM - mw1 Current Load on mw1 is CRITICAL: CRITICAL - load average: 9.48, 8.48, 6.52 [04:16:17] someone should probably check on what that load is from at some point [04:17:42] PROBLEM - mw1 Current Load on mw1 is WARNING: WARNING - load average: 5.96, 7.16, 6.83 [04:19:42] RECOVERY - mw1 Current Load on mw1 is OK: OK - load average: 5.66, 6.61, 6.66 [05:03:44] PROBLEM - mw1 Current Load on mw1 is CRITICAL: CRITICAL - load average: 9.32, 6.92, 5.88 [05:05:43] RECOVERY - mw1 Current Load on mw1 is OK: OK - load average: 5.24, 6.30, 5.78 [05:15:08] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JeEGi [05:15:10] [02miraheze/services] 07MirahezeSSLBot 03830aad7 - BOT: Updating services config for wikis [05:55:45] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 7.80, 7.15, 6.05 [05:57:03] PROBLEM - lizardfs5 Current Load on lizardfs5 is WARNING: WARNING - load average: 3.53, 2.97, 2.04 [05:57:45] RECOVERY - mw3 Current Load on mw3 is OK: OK - load average: 6.11, 6.78, 6.05 [05:59:05] PROBLEM - lizardfs5 Current Load on lizardfs5 is CRITICAL: CRITICAL - load average: 5.50, 3.60, 2.37 [06:01:55] PROBLEM - lizardfs4 Current Load on lizardfs4 is CRITICAL: CRITICAL - load average: 4.31, 3.81, 2.45 [06:02:44] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 6.27, 7.15, 6.46 [06:02:47] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 6 datacenters are down: 107.191.126.23/cpweb, 2604:180:0:33b::2/cpweb, 128.199.139.216/cpweb, 2400:6180:0:d0::403:f001/cpweb, 81.4.109.133/cpweb, 2a00:d880:5:8ea::ebc7/cpweb [06:02:47] PROBLEM - cp2 Varnish Backends on cp2 is CRITICAL: 1 backends are down. mw1 [06:02:55] PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 1 datacenter is down: 2a00:d880:5:8ea::ebc7/cpweb [06:03:17] PROBLEM - cp3 Varnish Backends on cp3 is CRITICAL: 1 backends are down. mw1 [06:04:44] PROBLEM - mw3 Current Load on mw3 is CRITICAL: CRITICAL - load average: 8.15, 7.70, 6.76 [06:05:54] RECOVERY - lizardfs4 Current Load on lizardfs4 is OK: OK - load average: 2.75, 3.34, 2.57 [06:06:36] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [06:06:48] RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online [06:06:49] RECOVERY - cp2 Varnish Backends on cp2 is OK: All 5 backends are healthy [06:07:16] RECOVERY - cp3 Varnish Backends on cp3 is OK: All 5 backends are healthy [06:08:44] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 6.24, 7.13, 6.75 [06:09:24] RECOVERY - lizardfs5 Current Load on lizardfs5 is OK: OK - load average: 1.16, 3.37, 3.15 [06:10:43] RECOVERY - mw3 Current Load on mw3 is OK: OK - load average: 6.04, 6.77, 6.66 [06:27:43] RECOVERY - cp3 Disk Space on cp3 is OK: DISK OK - free space: / 2732 MB (11% inode=94%); [06:53:42] PROBLEM - mw1 Current Load on mw1 is WARNING: WARNING - load average: 7.61, 6.26, 4.78 [06:55:43] RECOVERY - mw1 Current Load on mw1 is OK: OK - load average: 6.38, 6.12, 4.91 [06:56:19] Reception123: what on earth is it moaning about [07:04:47] !log running wikibackups (bash ./wikibackups.sh /home/reception/allpublic.dblist /srv/mediawiki/w/maintenance/dumpBackup.php) [07:04:52] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [07:05:00] RhinosF1: no idea, though mw1 load is probably me [07:06:07] Reception123: ok [07:07:39] RhinosF1: - fix anything broken from meta import [07:19:14] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 7.71, 6.71, 5.98 [07:19:43] PROBLEM - mw1 Current Load on mw1 is CRITICAL: CRITICAL - load average: 8.88, 6.91, 5.94 [07:21:42] RECOVERY - mw1 Current Load on mw1 is OK: OK - load average: 6.63, 6.43, 5.85 [07:23:17] RECOVERY - mw3 Current Load on mw3 is OK: OK - load average: 5.74, 6.42, 6.04 [07:27:24] !log tar zcvf speleowiki26102019.tar.gz /mnt/mediawiki-static/speleowiki on mw2 [07:27:29] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [08:03:42] PROBLEM - mw1 Current Load on mw1 is CRITICAL: CRITICAL - load average: 8.83, 6.80, 5.54 [08:07:42] RECOVERY - mw1 Current Load on mw1 is OK: OK - load average: 5.39, 6.71, 5.84 [08:14:23] PROBLEM - cp3 Disk Space on cp3 is WARNING: DISK WARNING - free space: / 2650 MB (10% inode=94%); [09:24:51] Hello hotlolisex18! If you have any questions, feel free to ask and someone should answer soon. [09:24:55] wassup [09:25:13] anyone here? [09:26:13] alright then [09:26:28] ok [10:52:47] PROBLEM - wiki.kourouklides.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.kourouklides.com' expires in 15 day(s) (Mon 11 Nov 2019 10:50:04 AM GMT +0000). [10:53:00] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JeECi [10:53:01] [02miraheze/ssl] 07MirahezeSSLBot 03b4e9159 - Bot: Update SSL cert for wiki.kourouklides.com [10:53:06] ok [10:54:27] PROBLEM - wiki.ameciclo.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.ameciclo.org' expires in 15 day(s) (Mon 11 Nov 2019 10:52:13 AM GMT +0000). [10:54:40] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JeECP [10:54:42] [02miraheze/ssl] 07MirahezeSSLBot 0369852b4 - Bot: Update SSL cert for wiki.ameciclo.org [10:55:54] PROBLEM - wiki.ldmsys.net - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.ldmsys.net' expires in 15 day(s) (Mon 11 Nov 2019 10:52:25 AM GMT +0000). [10:56:07] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JeECX [10:56:09] [02miraheze/ssl] 07MirahezeSSLBot 031415b1c - Bot: Update SSL cert for wiki.ldmsys.net [10:57:39] PROBLEM - wiki.macc.nyc - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.macc.nyc' expires in 15 day(s) (Mon 11 Nov 2019 10:55:15 AM GMT +0000). [10:57:54] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JeEC1 [10:57:55] [02miraheze/ssl] 07MirahezeSSLBot 030ae04ae - Bot: Update SSL cert for wiki.macc.nyc [10:59:22] PROBLEM - adadevelopersacademy.wiki - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'adadevelopersacademy.wiki' expires in 15 day(s) (Mon 11 Nov 2019 10:55:27 AM GMT +0000). [10:59:28] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JeECM [10:59:29] [02miraheze/ssl] 07MirahezeSSLBot 03a3edd3e - Bot: Update SSL cert for adadevelopersacademy.wiki [10:59:44] PROBLEM - wikibase.revi.wiki - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wikibase.revi.wiki' expires in 15 day(s) (Mon 11 Nov 2019 10:55:47 AM GMT +0000). [10:59:50] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JeECD [10:59:52] [02miraheze/ssl] 07MirahezeSSLBot 03f413ab5 - Bot: Update SSL cert for wikibase.revi.wiki [11:01:21] PROBLEM - wiki.gesamtschule-nordkirchen.de - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.gesamtschule-nordkirchen.de' expires in 15 day(s) (Mon 11 Nov 2019 10:58:14 AM GMT +0000). [11:01:29] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JeECS [11:01:30] [02miraheze/ssl] 07MirahezeSSLBot 03bfb7184 - Bot: Update SSL cert for wiki.gesamtschule-nordkirchen.de [11:02:11] PROBLEM - wiki.ciptamedia.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.ciptamedia.org' expires in 15 day(s) (Mon 11 Nov 2019 10:58:53 AM GMT +0000). [11:02:17] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JeECH [11:02:18] [02miraheze/ssl] 07MirahezeSSLBot 03d881ec8 - Bot: Update SSL cert for wiki.ciptamedia.org [11:03:39] RECOVERY - wiki.macc.nyc - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.macc.nyc' will expire on Fri 24 Jan 2020 09:57:47 AM GMT +0000. [11:03:56] RECOVERY - wiki.ldmsys.net - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.ldmsys.net' will expire on Fri 24 Jan 2020 09:56:01 AM GMT +0000. [11:04:00] PROBLEM - podpedia.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'podpedia.org' expires in 15 day(s) (Mon 11 Nov 2019 11:00:35 AM GMT +0000). [11:04:15] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JeEC7 [11:04:16] [02miraheze/ssl] 07MirahezeSSLBot 038a8ae5a - Bot: Update SSL cert for podpedia.org [11:04:28] PROBLEM - athenapedia.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'athenapedia.org' expires in 15 day(s) (Mon 11 Nov 2019 11:00:48 AM GMT +0000). [11:04:31] RECOVERY - wiki.ameciclo.org - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.ameciclo.org' will expire on Fri 24 Jan 2020 09:54:34 AM GMT +0000. [11:04:42] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JeECF [11:04:44] [02miraheze/ssl] 07MirahezeSSLBot 039a57904 - Bot: Update SSL cert for athenapedia.org [11:04:44] RECOVERY - wiki.kourouklides.com - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.kourouklides.com' will expire on Fri 24 Jan 2020 09:52:54 AM GMT +0000. [11:05:33] PROBLEM - pwiki.arkcls.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'pwiki.arkcls.com' expires in 15 day(s) (Mon 11 Nov 2019 11:01:58 AM GMT +0000). [11:05:46] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JeECb [11:05:47] [02miraheze/ssl] 07MirahezeSSLBot 030a6369e - Bot: Update SSL cert for pwiki.arkcls.com [11:06:27] PROBLEM - kunwok.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'kunwok.org' expires in 15 day(s) (Mon 11 Nov 2019 11:03:13 AM GMT +0000). [11:06:41] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JeECN [11:06:43] [02miraheze/ssl] 07MirahezeSSLBot 03e9073ad - Bot: Update SSL cert for kunwok.org [11:06:51] PROBLEM - nonbinary.wiki - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'nonbinary.wiki' expires in 15 day(s) (Mon 11 Nov 2019 11:04:33 AM GMT +0000). [11:07:06] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JeECx [11:07:07] [02miraheze/ssl] 07MirahezeSSLBot 0355082ba - Bot: Update SSL cert for nonbinary.wiki [11:07:07] PROBLEM - pl.nonbinary.wiki - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'nonbinary.wiki' expires in 15 day(s) (Mon 11 Nov 2019 11:04:33 AM GMT +0000). [11:08:47] PROBLEM - bconnected.aidanmarkham.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'bconnected.aidanmarkham.com' expires in 15 day(s) (Mon 11 Nov 2019 11:06:32 AM GMT +0000). [11:09:01] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JeEWv [11:09:02] [02miraheze/ssl] 07MirahezeSSLBot 030badc65 - Bot: Update SSL cert for bconnected.aidanmarkham.com [11:11:57] PROBLEM - wiki.consentcraft.uk - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.consentcraft.uk' expires in 15 day(s) (Mon 11 Nov 2019 11:08:19 AM GMT +0000). [11:12:10] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JeE8E [11:12:12] [02miraheze/ssl] 07MirahezeSSLBot 031ec69a1 - Bot: Update SSL cert for wiki.consentcraft.uk [11:12:28] RECOVERY - athenapedia.org - LetsEncrypt on sslhost is OK: OK - Certificate 'athenapedia.org' will expire on Fri 24 Jan 2020 10:04:36 AM GMT +0000. [11:12:29] RECOVERY - kunwok.org - LetsEncrypt on sslhost is OK: OK - Certificate 'kunwok.org' will expire on Fri 24 Jan 2020 10:06:35 AM GMT +0000. [11:12:46] RECOVERY - bconnected.aidanmarkham.com - LetsEncrypt on sslhost is OK: OK - Certificate 'bconnected.aidanmarkham.com' will expire on Fri 24 Jan 2020 10:08:55 AM GMT +0000. [11:12:50] RECOVERY - nonbinary.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'nonbinary.wiki' will expire on Fri 24 Jan 2020 10:06:59 AM GMT +0000. [11:13:07] RECOVERY - pl.nonbinary.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'nonbinary.wiki' will expire on Fri 24 Jan 2020 10:06:59 AM GMT +0000. [11:13:21] RECOVERY - wiki.gesamtschule-nordkirchen.de - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.gesamtschule-nordkirchen.de' will expire on Fri 10 Jan 2020 01:02:03 PM GMT +0000. [11:13:22] RECOVERY - adadevelopersacademy.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'adadevelopersacademy.wiki' will expire on Fri 10 Jan 2020 01:06:00 PM GMT +0000. [11:13:29] RECOVERY - pwiki.arkcls.com - LetsEncrypt on sslhost is OK: OK - Certificate 'pwiki.arkcls.com' will expire on Fri 24 Jan 2020 10:05:40 AM GMT +0000. [11:13:38] PROBLEM - wiki.staraves-no.cz - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.staraves-no.cz' expires in 15 day(s) (Mon 11 Nov 2019 11:10:27 AM GMT +0000). [11:13:51] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JeE4u [11:13:52] RECOVERY - wikibase.revi.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'wikibase.revi.wiki' will expire on Fri 10 Jan 2020 01:01:52 PM GMT +0000. [11:13:52] PROBLEM - garrettcountyguide.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'garrettcountyguide.com' expires in 15 day(s) (Mon 11 Nov 2019 11:11:18 AM GMT +0000). [11:13:53] [02miraheze/ssl] 07MirahezeSSLBot 037467fd9 - Bot: Update SSL cert for wiki.staraves-no.cz [11:14:01] RECOVERY - podpedia.org - LetsEncrypt on sslhost is OK: OK - Certificate 'podpedia.org' will expire on Fri 24 Jan 2020 10:04:09 AM GMT +0000. [11:14:06] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JeE4z [11:14:08] [02miraheze/ssl] 07MirahezeSSLBot 032bfd144 - Bot: Update SSL cert for garrettcountyguide.com [11:14:19] RECOVERY - wiki.ciptamedia.org - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.ciptamedia.org' will expire on Fri 10 Jan 2020 01:00:21 PM GMT +0000. [11:15:30] PROBLEM - meregos.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'meregos.com' expires in 15 day(s) (Mon 11 Nov 2019 11:11:30 AM GMT +0000). [11:15:44] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JeE4w [11:15:45] [02miraheze/ssl] 07MirahezeSSLBot 03e3b601a - Bot: Update SSL cert for meregos.com [11:17:19] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 7.25, 6.19, 5.41 [11:19:18] RECOVERY - mw3 Current Load on mw3 is OK: OK - load average: 5.05, 5.69, 5.32 [11:23:27] RECOVERY - meregos.com - LetsEncrypt on sslhost is OK: OK - Certificate 'meregos.com' will expire on Fri 24 Jan 2020 10:15:38 AM GMT +0000. [11:23:38] RECOVERY - wiki.staraves-no.cz - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.staraves-no.cz' will expire on Fri 24 Jan 2020 10:13:45 AM GMT +0000. [11:23:51] RECOVERY - garrettcountyguide.com - LetsEncrypt on sslhost is OK: OK - Certificate 'garrettcountyguide.com' will expire on Fri 24 Jan 2020 10:13:59 AM GMT +0000. [11:24:02] RECOVERY - wiki.consentcraft.uk - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.consentcraft.uk' will expire on Fri 24 Jan 2020 10:12:05 AM GMT +0000. [11:45:12] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JeErE [11:45:13] [02miraheze/services] 07MirahezeSSLBot 03776c74d - BOT: Updating services config for wikis [13:40:38] PROBLEM - mw1 Current Load on mw1 is CRITICAL: CRITICAL - load average: 11.08, 8.07, 6.47 [13:40:56] Zppix: around? [13:41:41] PROBLEM - mw2 Current Load on mw2 is WARNING: WARNING - load average: 7.83, 6.42, 4.77 [13:43:41] RECOVERY - mw2 Current Load on mw2 is OK: OK - load average: 5.03, 6.07, 4.86 [14:03:01] PROBLEM - mw3 Current Load on mw3 is CRITICAL: CRITICAL - load average: 9.41, 7.55, 6.53 [14:05:00] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 7.63, 7.69, 6.71 [14:07:59] [02miraheze/ssl] 07Reception123 pushed 031 commit to 03master [+1/-0/±1] 13https://git.io/JeE6w [14:08:00] [02miraheze/ssl] 07Reception123 03251b603 - add wiki.graalmilitary.com cert [14:09:40] [02puppet] 07paladox created branch 03paladox-patch-7 - 13https://git.io/vbiAS [14:09:41] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-7 [+0/-0/±1] 13https://git.io/JeE6K [14:09:43] [02miraheze/puppet] 07paladox 03d4a54cc - lizardfs: Tweak config [14:09:44] [02puppet] 07paladox opened pull request 03#1121: lizardfs: Tweak config - 13https://git.io/JeE66 [14:12:09] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-7 [+0/-0/±1] 13https://git.io/JeE6X [14:12:11] [02miraheze/puppet] 07paladox 03bf2c488 - Update mfschunkserver.cfg.erb [14:12:13] [02puppet] 07paladox synchronize pull request 03#1121: lizardfs: Tweak config - 13https://git.io/JeE66 [14:12:48] PROBLEM - mw3 Current Load on mw3 is CRITICAL: CRITICAL - load average: 10.51, 8.27, 7.17 [14:16:43] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 6.48, 7.42, 7.09 [14:17:42] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-7 [+0/-0/±1] 13https://git.io/JeE6F [14:17:43] [02miraheze/puppet] 07paladox 0368d1e4f - Update mediawiki.pp [14:17:45] [02puppet] 07paladox synchronize pull request 03#1121: lizardfs: Tweak config - 13https://git.io/JeE66 [14:19:37] [02puppet] 07paladox closed pull request 03#1121: lizardfs: Tweak config - 13https://git.io/JeE66 [14:19:39] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±3] 13https://git.io/JeE6N [14:19:40] [02miraheze/puppet] 07paladox 03c2f5963 - lizardfs: Tweak config (#1121) * lizardfs: Tweak config * Update mfschunkserver.cfg.erb * Update mediawiki.pp [14:19:42] [02puppet] 07paladox deleted branch 03paladox-patch-7 - 13https://git.io/vbiAS [14:19:43] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-7 [14:21:55] PROBLEM - mw3 Puppet on mw3 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 8 minutes ago with 0 failures [14:22:07] PROBLEM - mw2 Puppet on mw2 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 9 minutes ago with 0 failures [14:22:16] PROBLEM - misc3 Puppet on misc3 is WARNING: WARNING: Puppet is currently disabled, message: reason not specified, last run 10 minutes ago with 0 failures [14:22:23] PROBLEM - mw1 Puppet on mw1 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 9 minutes ago with 0 failures [14:22:36] PROBLEM - mw3 Current Load on mw3 is CRITICAL: CRITICAL - load average: 8.71, 7.58, 7.18 [14:23:28] !log restart lizardfs-master on misc3 [14:23:34] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [14:24:20] RECOVERY - misc3 Puppet on misc3 is OK: OK: Puppet is currently enabled, last run 41 seconds ago with 0 failures [14:24:33] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 5.57, 6.85, 6.96 [14:26:29] PROBLEM - mw3 Current Load on mw3 is CRITICAL: CRITICAL - load average: 8.48, 7.25, 7.07 [14:27:58] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 2 datacenters are down: 2400:6180:0:d0::403:f001/cpweb, 2a00:d880:5:8ea::ebc7/cpweb [14:28:29] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 7.18, 7.32, 7.12 [14:29:58] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [14:30:26] PROBLEM - mw3 Current Load on mw3 is CRITICAL: CRITICAL - load average: 9.38, 8.10, 7.43 [14:32:22] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 7.21, 7.64, 7.33 [14:32:23] !log restart lizardfs-chunkserver on lizardfs[45 [14:32:29] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [14:34:22] !log depool mw1 [14:34:35] PROBLEM - mw3 Current Load on mw3 is CRITICAL: CRITICAL - load average: 11.04, 9.00, 7.87 [14:34:40] RECOVERY - mw1 Puppet on mw1 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [14:35:23] PROBLEM - mw3 HTTPS on mw3 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:35:32] paladox: and we're down again [14:35:37] yup [14:35:46] PROBLEM - cp4 Stunnel Http for mw3 on cp4 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [14:35:51] PROBLEM - mw1 Current Load on mw1 is WARNING: WARNING - load average: 3.33, 6.51, 7.64 [14:35:54] PROBLEM - cp3 Varnish Backends on cp3 is CRITICAL: 2 backends are down. mw2 mw3 [14:36:12] PROBLEM - cp2 Varnish Backends on cp2 is CRITICAL: 1 backends are down. mw3 [14:36:13] PROBLEM - cp3 Stunnel Http for mw3 on cp3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [14:36:35] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 6 datacenters are down: 107.191.126.23/cpweb, 2604:180:0:33b::2/cpweb, 128.199.139.216/cpweb, 2400:6180:0:d0::403:f001/cpweb, 81.4.109.133/cpweb, 2a00:d880:5:8ea::ebc7/cpweb [14:36:46] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 4.14, 7.57, 7.53 [14:36:51] PROBLEM - cp4 Varnish Backends on cp4 is CRITICAL: 1 backends are down. mw3 [14:36:59] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JeEik [14:37:01] [02miraheze/puppet] 07paladox 031b98698 - Update mediawiki.pp [14:37:02] PROBLEM - mw2 Current Load on mw2 is CRITICAL: CRITICAL - load average: 10.04, 7.77, 6.22 [14:37:15] PROBLEM - cp2 Stunnel Http for mw3 on cp2 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [14:37:17] PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 6 datacenters are down: 107.191.126.23/cpweb, 2604:180:0:33b::2/cpweb, 128.199.139.216/cpweb, 2400:6180:0:d0::403:f001/cpweb, 81.4.109.133/cpweb, 2a00:d880:5:8ea::ebc7/cpweb [14:37:52] RECOVERY - mw1 Current Load on mw1 is OK: OK - load average: 1.15, 4.55, 6.78 [14:38:17] !log repool mw1 [14:38:23] !log depool mw2 [14:39:09] RECOVERY - cp2 Stunnel Http for mw3 on cp2 is OK: HTTP OK: HTTP/1.1 200 OK - 24639 bytes in 0.390 second response time [14:39:22] RECOVERY - mw3 HTTPS on mw3 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 442 bytes in 0.063 second response time [14:40:09] !log repool mw2 [14:40:37] RECOVERY - mw2 Puppet on mw2 is OK: OK: Puppet is currently enabled, last run 34 seconds ago with 0 failures [14:40:54] !log reboot mw1 - php-fpm froze [14:40:56] PROBLEM - mw3 Current Load on mw3 is CRITICAL: CRITICAL - load average: 10.42, 7.47, 7.33 [14:41:10] PROBLEM - mw2 Current Load on mw2 is WARNING: WARNING - load average: 7.05, 7.44, 6.47 [14:41:31] PROBLEM - cp2 Stunnel Http for mw1 on cp2 is CRITICAL: HTTP CRITICAL - No data received from host [14:41:40] !log reboot mw2 - php-fpm froze [14:41:56] PROBLEM - cp4 Stunnel Http for mw1 on cp4 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [14:42:44] PROBLEM - mw1 Current Load on mw1 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [14:43:12] PROBLEM - cp3 Stunnel Http for mw1 on cp3 is CRITICAL: HTTP CRITICAL - No data received from host [14:43:40] PROBLEM - cp3 Stunnel Http for mw2 on cp3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [14:43:42] PROBLEM - cp2 Stunnel Http for mw2 on cp2 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [14:43:59] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 3.58, 6.59, 7.09 [14:44:28] PROBLEM - mw2 Current Load on mw2 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [14:45:03] PROBLEM - mw1 Puppet on mw1 is CRITICAL: CRITICAL: Puppet has 7 failures. Last run 3 minutes ago with 7 failures. Failed resources (up to 3 shown) [14:45:07] PROBLEM - cp2 Stunnel Http for mw3 on cp2 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 309 bytes in 0.294 second response time [14:45:12] RECOVERY - mw1 Current Load on mw1 is OK: OK - load average: 1.77, 0.62, 0.22 [14:45:42] RECOVERY - cp2 Stunnel Http for mw2 on cp2 is OK: HTTP OK: HTTP/1.1 200 OK - 24639 bytes in 4.737 second response time [14:45:42] RECOVERY - cp3 Stunnel Http for mw2 on cp3 is OK: HTTP OK: HTTP/1.1 200 OK - 24639 bytes in 4.915 second response time [14:45:55] RECOVERY - mw3 Current Load on mw3 is OK: OK - load average: 4.44, 5.58, 6.63 [14:46:26] RECOVERY - mw2 Current Load on mw2 is OK: OK - load average: 4.26, 1.39, 0.48 [14:47:36] RECOVERY - cp3 Stunnel Http for mw3 on cp3 is OK: HTTP OK: HTTP/1.1 200 OK - 24639 bytes in 0.641 second response time [14:49:05] PROBLEM - mw1 php-fpm on mw1 is CRITICAL: PROCS CRITICAL: 0 processes with command name 'php-fpm7.2' [14:50:35] PROBLEM - mw2 Current Load on mw2 is CRITICAL: CRITICAL - load average: 10.37, 5.85, 2.47 [14:51:10] RECOVERY - cp2 Stunnel Http for mw3 on cp2 is OK: HTTP OK: HTTP/1.1 200 OK - 24639 bytes in 0.544 second response time [14:51:19] RECOVERY - cp4 Stunnel Http for mw3 on cp4 is OK: HTTP OK: HTTP/1.1 200 OK - 24655 bytes in 0.003 second response time [14:51:27] !log rebooting mw1 [14:51:34] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [14:52:00] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [14:52:14] PROBLEM - mw3 Current Load on mw3 is CRITICAL: CRITICAL - load average: 9.83, 8.23, 7.41 [14:52:23] RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online [14:55:25] PROBLEM - mw1 Current Load on mw1 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [14:55:33] PROBLEM - mw1 HTTPS on mw1 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:56:06] RECOVERY - cp2 Stunnel Http for mw1 on cp2 is OK: HTTP OK: HTTP/1.1 200 OK - 24655 bytes in 0.504 second response time [14:56:11] RECOVERY - mw1 php-fpm on mw1 is OK: PROCS OK: 15 processes with command name 'php-fpm7.2' [14:56:31] RECOVERY - cp3 Stunnel Http for mw1 on cp3 is OK: HTTP OK: HTTP/1.1 200 OK - 24655 bytes in 0.643 second response time [14:57:21] RECOVERY - cp4 Varnish Backends on cp4 is OK: All 5 backends are healthy [14:57:23] RECOVERY - mw1 Current Load on mw1 is OK: OK - load average: 3.78, 0.97, 0.32 [14:57:28] RECOVERY - mw1 HTTPS on mw1 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 442 bytes in 0.016 second response time [14:57:47] RECOVERY - cp3 Varnish Backends on cp3 is OK: All 5 backends are healthy [14:57:49] RECOVERY - cp4 Stunnel Http for mw1 on cp4 is OK: HTTP OK: HTTP/1.1 200 OK - 24639 bytes in 0.005 second response time [14:57:50] PROBLEM - mw2 Current Load on mw2 is WARNING: WARNING - load average: 6.67, 7.56, 4.60 [14:58:32] RECOVERY - cp2 Varnish Backends on cp2 is OK: All 5 backends are healthy [14:59:48] RECOVERY - mw2 Current Load on mw2 is OK: OK - load average: 4.34, 6.27, 4.47 [15:01:07] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 5.29, 7.06, 7.48 [15:02:03] RECOVERY - test1 Puppet on test1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [15:04:55] RECOVERY - mw1 Puppet on mw1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [15:06:15] PROBLEM - mw1 Current Load on mw1 is CRITICAL: CRITICAL - load average: 10.57, 7.41, 3.74 [15:08:13] PROBLEM - mw1 Current Load on mw1 is WARNING: WARNING - load average: 7.09, 7.26, 4.12 [15:12:08] RECOVERY - mw1 Current Load on mw1 is OK: OK - load average: 5.22, 6.70, 4.63 [15:15:00] PROBLEM - mw3 Current Load on mw3 is CRITICAL: CRITICAL - load average: 9.12, 7.26, 7.18 [15:16:05] PROBLEM - mw1 Current Load on mw1 is WARNING: WARNING - load average: 6.53, 7.39, 5.45 [15:16:57] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 6.66, 7.24, 7.20 [15:18:05] PROBLEM - mw1 Current Load on mw1 is CRITICAL: CRITICAL - load average: 8.83, 7.85, 5.84 [15:20:06] PROBLEM - mw1 Current Load on mw1 is WARNING: WARNING - load average: 6.16, 6.96, 5.74 [15:26:40] PROBLEM - mw3 Current Load on mw3 is CRITICAL: CRITICAL - load average: 8.73, 7.45, 7.19 [15:32:08] PROBLEM - mw1 Current Load on mw1 is CRITICAL: CRITICAL - load average: 8.09, 7.83, 6.85 [15:32:10] PROBLEM - mw2 Current Load on mw2 is WARNING: WARNING - load average: 7.41, 6.72, 6.09 [15:32:34] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 6.09, 7.60, 7.48 [15:36:03] PROBLEM - mw1 Current Load on mw1 is WARNING: WARNING - load average: 6.97, 7.75, 7.07 [15:38:08] RECOVERY - mw2 Current Load on mw2 is OK: OK - load average: 4.27, 6.08, 6.11 [15:39:59] PROBLEM - mw1 Current Load on mw1 is CRITICAL: CRITICAL - load average: 8.51, 8.03, 7.32 [15:40:21] PROBLEM - mw3 Current Load on mw3 is CRITICAL: CRITICAL - load average: 8.80, 7.67, 7.48 [15:41:57] PROBLEM - mw1 Current Load on mw1 is WARNING: WARNING - load average: 7.02, 7.94, 7.40 [15:42:17] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 6.46, 7.15, 7.30 [15:53:52] RECOVERY - mw1 Current Load on mw1 is OK: OK - load average: 6.10, 6.22, 6.72 [15:53:58] RECOVERY - mw3 Current Load on mw3 is OK: OK - load average: 6.28, 6.43, 6.80 [15:57:53] PROBLEM - mw1 Current Load on mw1 is WARNING: WARNING - load average: 7.62, 7.02, 6.94 [15:57:53] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 7.07, 6.92, 6.92 [16:03:52] PROBLEM - mw1 Current Load on mw1 is CRITICAL: CRITICAL - load average: 8.13, 7.56, 7.21 [16:05:53] RECOVERY - mw3 Current Load on mw3 is OK: OK - load average: 6.33, 6.37, 6.70 [16:05:53] PROBLEM - mw1 Current Load on mw1 is WARNING: WARNING - load average: 6.37, 7.12, 7.08 [16:09:22] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JeEPz [16:09:23] [02miraheze/puppet] 07paladox 03b93669d - Update mediawiki.pp [16:10:10] PROBLEM - mw3 Current Load on mw3 is CRITICAL: CRITICAL - load average: 8.30, 7.15, 6.91 [16:11:25] PROBLEM - mw2 Current Load on mw2 is CRITICAL: CRITICAL - load average: 8.95, 6.79, 5.91 [16:12:06] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 7.46, 7.32, 7.00 [16:14:05] PROBLEM - mw3 Current Load on mw3 is CRITICAL: CRITICAL - load average: 9.58, 7.90, 7.23 [16:16:32] PROBLEM - mw2 Puppet on mw2 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Mount[/mnt/mediawiki-static] [16:17:41] PROBLEM - mw2 Current Load on mw2 is WARNING: WARNING - load average: 5.60, 7.09, 6.47 [16:18:10] PROBLEM - mw1 Current Load on mw1 is CRITICAL: CRITICAL - load average: 8.11, 8.09, 7.50 [16:19:40] RECOVERY - mw2 Current Load on mw2 is OK: OK - load average: 5.54, 6.50, 6.32 [16:20:13] PROBLEM - mw1 Current Load on mw1 is WARNING: WARNING - load average: 6.25, 7.39, 7.32 [16:24:46] RECOVERY - mw2 Puppet on mw2 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [16:26:12] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 6.68, 7.96, 7.92 [16:28:14] RECOVERY - mw1 Current Load on mw1 is OK: OK - load average: 4.96, 5.89, 6.66 [16:38:08] RECOVERY - mw3 Current Load on mw3 is OK: OK - load average: 5.70, 6.00, 6.78 [16:42:15] PROBLEM - mw1 Current Load on mw1 is WARNING: WARNING - load average: 7.45, 6.72, 6.53 [16:42:21] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JeEXU [16:42:22] [02miraheze/puppet] 07paladox 0398c9a12 - mediawiki: Use thumb_handler [16:44:14] RECOVERY - mw1 Current Load on mw1 is OK: OK - load average: 5.45, 6.27, 6.39 [16:46:53] PROBLEM - mw3 Puppet on mw3 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 39 seconds ago with 1 failures. Failed resources (up to 3 shown): Mount[/mnt/mediawiki-static] [16:48:49] paladox: ^ [16:48:49] RECOVERY - mw3 Puppet on mw3 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [16:48:54] Meh back [16:49:12] yes, no need to ping me for that :) [16:49:17] i was already aware :) [16:50:22] Ok [17:02:44] PROBLEM - mw2 Current Load on mw2 is WARNING: WARNING - load average: 7.05, 6.26, 5.70 [17:03:54] PROBLEM - mw1 Current Load on mw1 is WARNING: WARNING - load average: 7.74, 6.88, 6.44 [17:04:44] RECOVERY - mw2 Current Load on mw2 is OK: OK - load average: 5.34, 6.09, 5.71 [17:05:53] PROBLEM - mw1 Current Load on mw1 is CRITICAL: CRITICAL - load average: 8.19, 7.10, 6.55 [17:07:52] PROBLEM - mw1 Current Load on mw1 is WARNING: WARNING - load average: 7.31, 6.89, 6.52 [17:09:52] RECOVERY - mw1 Current Load on mw1 is OK: OK - load average: 6.42, 6.57, 6.44 [17:33:51] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 6.60, 6.87, 6.36 [17:33:53] PROBLEM - mw1 Current Load on mw1 is WARNING: WARNING - load average: 7.06, 6.80, 6.52 [17:35:52] RECOVERY - mw3 Current Load on mw3 is OK: OK - load average: 5.67, 6.44, 6.26 [17:35:54] RECOVERY - mw1 Current Load on mw1 is OK: OK - load average: 5.98, 6.34, 6.37 [17:45:52] PROBLEM - mw3 Current Load on mw3 is CRITICAL: CRITICAL - load average: 8.74, 7.31, 6.62 [17:47:57] RECOVERY - mw3 Current Load on mw3 is OK: OK - load average: 5.20, 6.60, 6.45 [17:54:00] PROBLEM - lizardfs4 Current Load on lizardfs4 is WARNING: WARNING - load average: 3.61, 3.50, 2.73 [17:54:51] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 7.01, 7.06, 6.70 [17:56:15] PROBLEM - mw2 Current Load on mw2 is WARNING: WARNING - load average: 7.22, 7.48, 6.63 [17:56:58] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 5 datacenters are down: 107.191.126.23/cpweb, 128.199.139.216/cpweb, 2400:6180:0:d0::403:f001/cpweb, 81.4.109.133/cpweb, 2a00:d880:5:8ea::ebc7/cpweb [17:57:41] PROBLEM - lizardfs5 Current Load on lizardfs5 is WARNING: WARNING - load average: 3.11, 3.45, 2.76 [17:57:49] RECOVERY - lizardfs4 Current Load on lizardfs4 is OK: OK - load average: 1.47, 2.96, 2.72 [17:58:08] PROBLEM - mw1 Current Load on mw1 is CRITICAL: CRITICAL - load average: 8.95, 7.61, 7.01 [17:58:56] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [17:59:38] RECOVERY - lizardfs5 Current Load on lizardfs5 is OK: OK - load average: 1.85, 2.87, 2.63 [18:00:08] PROBLEM - mw1 Current Load on mw1 is WARNING: WARNING - load average: 7.92, 7.50, 7.03 [18:00:44] PROBLEM - mw3 Current Load on mw3 is CRITICAL: CRITICAL - load average: 8.17, 7.42, 6.96 [18:02:17] RECOVERY - mw2 Current Load on mw2 is OK: OK - load average: 4.69, 5.96, 6.31 [18:02:41] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 6.10, 6.86, 6.80 [18:04:39] RECOVERY - mw3 Current Load on mw3 is OK: OK - load average: 5.23, 6.40, 6.64 [18:06:01] RECOVERY - mw1 Current Load on mw1 is OK: OK - load average: 5.32, 6.30, 6.68 [18:12:38] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 7.97, 7.47, 6.96 [18:16:30] PROBLEM - mw3 Current Load on mw3 is CRITICAL: CRITICAL - load average: 8.02, 7.55, 7.08 [18:19:44] PROBLEM - cp2 Stunnel Http for mw1 on cp2 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [18:19:58] PROBLEM - cp2 Varnish Backends on cp2 is CRITICAL: 2 backends are down. mw1 mw2 [18:20:07] PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 6 datacenters are down: 107.191.126.23/cpweb, 2604:180:0:33b::2/cpweb, 128.199.139.216/cpweb, 2400:6180:0:d0::403:f001/cpweb, 81.4.109.133/cpweb, 2a00:d880:5:8ea::ebc7/cpweb [18:20:34] PROBLEM - cp3 Stunnel Http for mw2 on cp3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [18:20:47] PROBLEM - cp3 Varnish Backends on cp3 is CRITICAL: 1 backends are down. mw3 [18:20:48] PROBLEM - cp4 Varnish Backends on cp4 is CRITICAL: 1 backends are down. mw3 [18:21:09] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 5 datacenters are down: 107.191.126.23/cpweb, 2604:180:0:33b::2/cpweb, 128.199.139.216/cpweb, 2400:6180:0:d0::403:f001/cpweb, 2a00:d880:5:8ea::ebc7/cpweb [18:21:31] PROBLEM - lizardfs5 Current Load on lizardfs5 is CRITICAL: CRITICAL - load average: 3.87, 4.68, 3.29 [18:21:46] RECOVERY - cp2 Stunnel Http for mw1 on cp2 is OK: HTTP OK: HTTP/1.1 200 OK - 24639 bytes in 0.392 second response time [18:21:59] RECOVERY - cp2 Varnish Backends on cp2 is OK: All 5 backends are healthy [18:22:04] RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online [18:22:27] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 7.19, 7.78, 7.40 [18:22:32] RECOVERY - cp3 Stunnel Http for mw2 on cp3 is OK: HTTP OK: HTTP/1.1 200 OK - 24661 bytes in 0.639 second response time [18:22:42] RECOVERY - cp3 Varnish Backends on cp3 is OK: All 5 backends are healthy [18:22:49] RECOVERY - cp4 Varnish Backends on cp4 is OK: All 5 backends are healthy [18:23:05] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [18:23:28] PROBLEM - lizardfs5 Current Load on lizardfs5 is WARNING: WARNING - load average: 1.26, 3.45, 3.01 [18:24:27] PROBLEM - mw3 Current Load on mw3 is CRITICAL: CRITICAL - load average: 8.28, 7.86, 7.46 [18:26:23] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 7.21, 7.60, 7.42 [18:27:20] RECOVERY - lizardfs5 Current Load on lizardfs5 is OK: OK - load average: 1.81, 2.86, 2.89 [18:28:21] PROBLEM - mw3 Current Load on mw3 is CRITICAL: CRITICAL - load average: 8.67, 7.89, 7.53 [18:31:07] PROBLEM - mw1 Current Load on mw1 is CRITICAL: CRITICAL - load average: 8.76, 7.62, 6.92 [18:35:28] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JeE17 [18:35:30] [02miraheze/puppet] 07paladox 03886d12a - Update mediawiki.pp [18:37:16] PROBLEM - lizardfs5 Current Load on lizardfs5 is WARNING: WARNING - load average: 3.01, 3.52, 3.29 [18:39:04] PROBLEM - mw1 Current Load on mw1 is WARNING: WARNING - load average: 7.04, 7.90, 7.39 [18:39:13] PROBLEM - lizardfs5 Current Load on lizardfs5 is CRITICAL: CRITICAL - load average: 5.60, 4.69, 3.76 [18:41:10] PROBLEM - lizardfs5 Current Load on lizardfs5 is WARNING: WARNING - load average: 2.17, 3.66, 3.49 [18:42:10] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 7.04, 7.89, 7.94 [18:43:05] PROBLEM - mw1 Current Load on mw1 is CRITICAL: CRITICAL - load average: 8.42, 7.85, 7.45 [18:43:14] PROBLEM - lizardfs5 Current Load on lizardfs5 is CRITICAL: CRITICAL - load average: 5.40, 4.08, 3.64 [18:44:07] PROBLEM - mw3 Current Load on mw3 is CRITICAL: CRITICAL - load average: 8.48, 8.24, 8.07 [18:45:11] PROBLEM - mw1 Current Load on mw1 is WARNING: WARNING - load average: 5.95, 7.28, 7.31 [18:45:54] PROBLEM - cp3 Varnish Backends on cp3 is CRITICAL: 1 backends are down. mw2 [18:46:05] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 4 datacenters are down: 107.191.126.23/cpweb, 2604:180:0:33b::2/cpweb, 81.4.109.133/cpweb, 2a00:d880:5:8ea::ebc7/cpweb [18:46:14] PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 6 datacenters are down: 107.191.126.23/cpweb, 2604:180:0:33b::2/cpweb, 128.199.139.216/cpweb, 2400:6180:0:d0::403:f001/cpweb, 81.4.109.133/cpweb, 2a00:d880:5:8ea::ebc7/cpweb [18:47:50] RECOVERY - cp3 Varnish Backends on cp3 is OK: All 5 backends are healthy [18:48:00] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 7.10, 7.55, 7.80 [18:48:01] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [18:48:12] RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online [18:51:10] PROBLEM - mw1 Current Load on mw1 is CRITICAL: CRITICAL - load average: 8.13, 7.27, 7.24 [18:51:12] PROBLEM - lizardfs5 Current Load on lizardfs5 is WARNING: WARNING - load average: 3.86, 3.89, 3.91 [18:52:58] PROBLEM - lizardfs4 Current Load on lizardfs4 is CRITICAL: CRITICAL - load average: 5.48, 3.72, 3.10 [18:53:13] PROBLEM - mw1 Current Load on mw1 is WARNING: WARNING - load average: 7.43, 7.30, 7.25 [18:53:15] PROBLEM - lizardfs5 Current Load on lizardfs5 is CRITICAL: CRITICAL - load average: 6.15, 4.72, 4.21 [18:53:53] PROBLEM - mw3 Current Load on mw3 is CRITICAL: CRITICAL - load average: 8.35, 7.49, 7.61 [18:54:21] PROBLEM - lizardfs4 Puppet on lizardfs4 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 19 seconds ago with 0 failures [18:57:06] we're seeing really slow site performance [18:57:21] paladox: ^ [18:57:31] I bet it has something to do with all this high load [18:57:41] yup [18:57:42] k6ka: did you see my Discord ping [18:57:51] looking [18:58:00] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 5.56, 7.21, 7.53 [18:58:07] RhinosF1: yes [18:58:43] Voidwalker it's lizard me thinks [18:58:45] PROBLEM - lizardfs4 Current Load on lizardfs4 is WARNING: WARNING - load average: 2.79, 3.44, 3.19 [18:59:08] Someone writes to lizard, requests gets hold and everything goes down i think. At least judging by the graphs [18:59:15] very high i/o on mw [18:59:23] RECOVERY - mw1 Current Load on mw1 is OK: OK - load average: 5.03, 6.00, 6.70 [18:59:38] lizard is extreamly slow for me [19:00:00] i've been waiting minutes for puppet to disable on lizardfs5... still hasen't disabled. [19:00:27] k6ka: good [19:00:40] RECOVERY - lizardfs4 Current Load on lizardfs4 is OK: OK - load average: 2.53, 2.98, 3.04 [19:00:51] https://grafana.miraheze.org/d/W9MIkA7iz/miraheze-cluster?orgId=1&var-job=node&var-node=lizardfs5.miraheze.org&var-port=9100 [19:00:51] [ Grafana ] - grafana.miraheze.org [19:00:52] good lord [19:02:08] PROBLEM - mw3 Current Load on mw3 is CRITICAL: CRITICAL - load average: 9.60, 8.13, 7.80 [19:02:47] !log restart lizardfs-chunkserver on lizardfs[45] [19:04:55] PROBLEM - lizardfs5 Puppet on lizardfs5 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 58 seconds ago with 0 failures [19:04:58] PROBLEM - cp3 Varnish Backends on cp3 is CRITICAL: 3 backends are down. mw1 mw2 mw3 [19:04:58] PROBLEM - cp4 Varnish Backends on cp4 is CRITICAL: 3 backends are down. mw1 mw2 mw3 [19:05:00] PROBLEM - cp2 Varnish Backends on cp2 is CRITICAL: 3 backends are down. mw1 mw2 mw3 [19:05:33] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 6 datacenters are down: 107.191.126.23/cpweb, 2604:180:0:33b::2/cpweb, 128.199.139.216/cpweb, 2400:6180:0:d0::403:f001/cpweb, 81.4.109.133/cpweb, 2a00:d880:5:8ea::ebc7/cpweb [19:05:37] PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 6 datacenters are down: 107.191.126.23/cpweb, 2604:180:0:33b::2/cpweb, 128.199.139.216/cpweb, 2400:6180:0:d0::403:f001/cpweb, 81.4.109.133/cpweb, 2a00:d880:5:8ea::ebc7/cpweb [19:06:06] https://www.youtube.com/watch?v=sonLd-32ns4 [19:06:07] [ YouTube ] - www.youtube.com [19:06:07] PROBLEM - cp3 Stunnel Http for mw3 on cp3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [19:06:38] paladox: (see video) :P [19:06:55] lol [19:06:57] old video :P [19:07:28] PROBLEM - cp2 Stunnel Http for mw3 on cp2 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [19:07:45] PROBLEM - cp3 Stunnel Http for mw2 on cp3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [19:08:13] Zppix https://www.youtube.com/watch?v=9jK-NcRmVcw :P [19:08:14] [ YouTube ] - www.youtube.com [19:08:17] PROBLEM - cp4 Stunnel Http for mw3 on cp4 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [19:08:20] PROBLEM - mw3 HTTPS on mw3 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:08:26] And just for JohnLewis ^^ :D "Europe - It's the final countdown" :P [19:09:02] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 2.67, 6.11, 7.33 [19:09:23] lol [19:09:30] RECOVERY - cp2 Stunnel Http for mw3 on cp2 is OK: HTTP OK: HTTP/1.1 200 OK - 24655 bytes in 0.389 second response time [19:09:43] PROBLEM - cp4 Stunnel Http for mw2 on cp4 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [19:09:57] PROBLEM - cp2 Stunnel Http for mw2 on cp2 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [19:10:23] What did we blow up this time paladox? [19:10:27] RECOVERY - mw3 HTTPS on mw3 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 442 bytes in 0.013 second response time [19:10:47] lizard has decided for the past few months it's going to play hard ball and reuin everyones life :( [19:11:09] knew we should of paid the mob for its Lizardfs protection money [19:12:24] PROBLEM - lizardfs5 Current Load on lizardfs5 is WARNING: WARNING - load average: 1.23, 2.00, 3.67 [19:12:26] 503s everywhere https://upload.wikimedia.org/wikipedia/commons/f/fb/Bomba_atomica.gif [19:12:32] k6ka: all the 503s [19:12:43] all your 503s belong to us [19:13:10] k6ka lol [19:13:27] you really op'd up just for that space lol [19:13:33] Zppix: yes [19:13:38] thats some dedication [19:13:47] I'm in that mood [19:13:57] PROBLEM - cp2 Stunnel Http for mw3 on cp2 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [19:14:04] PROBLEM - cp4 Stunnel Http for mw1 on cp4 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 309 bytes in 0.003 second response time [19:14:21] RECOVERY - lizardfs5 Current Load on lizardfs5 is OK: OK - load average: 1.14, 1.73, 3.37 [19:14:25] PROBLEM - cp3 Stunnel Http for mw1 on cp3 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 309 bytes in 0.506 second response time [19:14:30] Campfire songs while the servers burn? [19:14:53] PROBLEM - cp2 Stunnel Http for mw1 on cp2 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [19:15:37] Zppix: too cold and wet for a fire [19:15:55] RhinosF1: tell that to RN datacenter staff rn :P [19:16:09] Zppix: heh [19:16:25] poor guys probably had to order more extingshers just for our servers [19:17:05] RECOVERY - mw3 Current Load on mw3 is OK: OK - load average: 5.14, 5.72, 6.67 [19:17:06] RECOVERY - cp2 Stunnel Http for mw1 on cp2 is OK: HTTP OK: HTTP/1.1 200 OK - 24661 bytes in 9.115 second response time [19:17:20] * Zppix plays a violin [19:17:40] lol [19:17:57] RECOVERY - cp4 Stunnel Http for mw1 on cp4 is OK: HTTP OK: HTTP/1.1 200 OK - 24655 bytes in 5.358 second response time [19:18:02] O_o are we back up? [19:18:37] uh nope [19:19:00] RECOVERY - cp4 Stunnel Http for mw3 on cp4 is OK: HTTP OK: HTTP/1.1 200 OK - 24661 bytes in 8.467 second response time [19:19:12] RECOVERY - cp3 Stunnel Http for mw3 on cp3 is OK: HTTP OK: HTTP/1.1 200 OK - 24655 bytes in 0.667 second response time [19:19:31] paladox: i think swap being at 97% may be an issue xD [19:19:41] where? [19:19:44] paladox: cp2 [19:20:18] well https://grafana.miraheze.org/d/W9MIkA7iz/miraheze-cluster?orgId=1&var-job=node&var-node=cp2.miraheze.org&var-port=9100&fullscreen&panelId=78 [19:20:19] [ Grafana ] - grafana.miraheze.org [19:20:24] paladox: cp4 is 100% swap [19:20:28] swap can be used to store non-frequent stuff [19:21:25] PROBLEM - cp2 Stunnel Http for mw1 on cp2 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [19:22:08] PROBLEM - cp4 Stunnel Http for mw1 on cp4 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [19:22:11] RECOVERY - cp4 Stunnel Http for mw2 on cp4 is OK: HTTP OK: HTTP/1.1 200 OK - 24661 bytes in 0.013 second response time [19:22:21] RECOVERY - cp2 Stunnel Http for mw2 on cp2 is OK: HTTP OK: HTTP/1.1 200 OK - 24661 bytes in 0.390 second response time [19:22:27] RECOVERY - cp3 Stunnel Http for mw1 on cp3 is OK: HTTP OK: HTTP/1.1 200 OK - 24661 bytes in 0.633 second response time [19:22:33] We probably should tweet this [19:22:36] RECOVERY - cp3 Stunnel Http for mw2 on cp3 is OK: HTTP OK: HTTP/1.1 200 OK - 24655 bytes in 1.778 second response time [19:23:06] PROBLEM - cp4 Stunnel Http for mw3 on cp4 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [19:23:18] PROBLEM - cp3 Stunnel Http for mw3 on cp3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [19:24:01] paladox: we are getting blips of non-503 fyi [19:25:24] lizard still scanning it's dir [19:25:34] so will prevent uploads till it says "completed" [19:25:39] RECOVERY - cp2 Stunnel Http for mw1 on cp2 is OK: HTTP OK: HTTP/1.1 200 OK - 24639 bytes in 3.018 second response time [19:26:08] it's complete! [19:26:12] at least on lizardfs5 [19:26:14] paladox: bad lizard timing [19:26:17] RECOVERY - cp2 Stunnel Http for mw3 on cp2 is OK: HTTP OK: HTTP/1.1 200 OK - 24655 bytes in 0.390 second response time [19:26:31] things are back [19:27:04] RECOVERY - cp4 Stunnel Http for mw3 on cp4 is OK: HTTP OK: HTTP/1.1 200 OK - 24639 bytes in 0.007 second response time [19:27:07] RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online [19:27:20] RECOVERY - cp3 Stunnel Http for mw3 on cp3 is OK: HTTP OK: HTTP/1.1 200 OK - 24639 bytes in 0.632 second response time [19:27:33] RECOVERY - cp4 Varnish Backends on cp4 is OK: All 5 backends are healthy [19:27:43] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [19:27:50] RECOVERY - cp2 Varnish Backends on cp2 is OK: All 5 backends are healthy [19:28:10] RECOVERY - cp4 Stunnel Http for mw1 on cp4 is OK: HTTP OK: HTTP/1.1 200 OK - 24639 bytes in 0.014 second response time [19:28:57] RECOVERY - cp3 Varnish Backends on cp3 is OK: All 5 backends are healthy [19:36:26] [02mw-config] 07Pix1234 deleted branch 03revert-2759-RhinosF1-patch-2 - 13https://git.io/vbvb3 [19:36:27] [02miraheze/mw-config] 07Pix1234 deleted branch 03revert-2759-RhinosF1-patch-2 [19:36:36] just deleting stale branches from our gh repos ^ [19:36:56] Zppix: I always forget thx [19:37:06] RhinosF1: no worries i enjoy trival stuff like this [19:37:53] [02mediawiki] 07Pix1234 deleted branch 03revert-125-REL1_33 - 13https://git.io/vbL5b [19:37:54] [02miraheze/mediawiki] 07Pix1234 deleted branch 03revert-125-REL1_33 [19:38:30] [02miraheze/CreateWiki] 07Pix1234 deleted branch 03echo [19:38:31] [02CreateWiki] 07Pix1234 deleted branch 03echo - 13https://git.io/vpJTL [19:38:33] [02CreateWiki] 07Pix1234 deleted branch 03paladox-patch-2 - 13https://git.io/vpJTL [19:38:34] [02miraheze/CreateWiki] 07Pix1234 deleted branch 03paladox-patch-2 [19:38:36] [02CreateWiki] 07Pix1234 deleted branch 03paladox-patch-1 - 13https://git.io/vpJTL [19:38:37] [02miraheze/CreateWiki] 07Pix1234 deleted branch 03paladox-patch-1 [19:38:38] [02CreateWiki] 07Pix1234 deleted branch 03RhinosF1-sonar-patch - 13https://git.io/vpJTL [19:38:40] [02miraheze/CreateWiki] 07Pix1234 deleted branch 03RhinosF1-sonar-patch [19:39:10] [02ManageWiki] 07Pix1234 deleted branch 03single-page - 13https://git.io/vpSns [19:39:12] [02miraheze/ManageWiki] 07Pix1234 deleted branch 03single-page [19:39:13] [02ManageWiki] 07Pix1234 deleted branch 03paladox-patch-1 - 13https://git.io/vpSns [19:39:15] [02miraheze/ManageWiki] 07Pix1234 deleted branch 03paladox-patch-1 [19:39:16] [02miraheze/ManageWiki] 07Pix1234 deleted branch 03paladox-patch-2 [19:39:17] Zppix: I like people to watch what I'm forgetting so ++ [19:39:18] [02ManageWiki] 07Pix1234 deleted branch 03paladox-patch-2 - 13https://git.io/vpSns [19:41:58] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 7.75, 6.81, 6.16 [19:42:05] [02landing] 07Pix1234 synchronize pull request 03#12: japanese translation - 13https://git.io/fjpeh [19:43:31] RhinosF1: if i resolve the conflict on ^ would it be good to merge? [19:43:34] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 1 datacenter is down: 128.199.139.216/cpweb [19:43:54] RECOVERY - mw3 Current Load on mw3 is OK: OK - load average: 5.56, 6.65, 6.20 [19:44:33] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JeEMo [19:44:34] [02miraheze/puppet] 07paladox 034e7135c - Update sysctl.pp [19:45:13] [02landing] 07RhinosF1 synchronize pull request 03#12: japanese translation - 13https://git.io/fjpeh [19:45:34] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [19:45:42] RhinosF1: i take that as a yes then xD [19:47:24] PROBLEM - mw1 Current Load on mw1 is WARNING: WARNING - load average: 6.74, 7.14, 6.37 [19:47:42] Zppix: you can merge at your risk [19:48:11] RhinosF1: i mean do you think it will break anytihng? [19:48:37] if not i could have pioneer take a look at the translatinos [19:48:56] Zppix: it won't break anything, just haven't checked the translations for accuracy [19:49:04] ill ask pioneer if he would [19:49:06] You can ask the pioneer to check [19:51:22] PROBLEM - mw1 Current Load on mw1 is CRITICAL: CRITICAL - load average: 9.59, 8.15, 6.89 [19:51:31] RhinosF1: {{done}} [19:51:39] anyway im off to work [19:52:42] Zppix: enjoy [19:52:58] PROBLEM - mw3 Current Load on mw3 is CRITICAL: CRITICAL - load average: 8.45, 7.34, 6.66 [19:54:05] paladox: 503 [19:54:36] ok [19:54:54] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 6.28, 7.10, 6.67 [19:55:17] Back [19:55:43] PROBLEM - mw1 Current Load on mw1 is WARNING: WARNING - load average: 5.31, 7.22, 6.87 [19:56:51] RECOVERY - mw3 Current Load on mw3 is OK: OK - load average: 4.52, 6.32, 6.44 [19:59:38] RECOVERY - mw1 Current Load on mw1 is OK: OK - load average: 5.52, 6.47, 6.64 [20:02:45] [02puppet] 07paladox created branch 03paladox-patch-7 - 13https://git.io/vbiAS [20:02:47] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-7 [+0/-0/±1] 13https://git.io/JeEDT [20:02:49] [02miraheze/puppet] 07paladox 032d53c78 - Varnish: Tweak timeout config [20:02:50] [02puppet] 07paladox opened pull request 03#1122: Varnish: Tweak timeout config - 13https://git.io/JeEDk [20:04:14] PROBLEM - mw1 Disk Space on mw1 is WARNING: DISK WARNING - free space: / 8348 MB (10% inode=98%); [20:13:32] PROBLEM - mw2 Current Load on mw2 is CRITICAL: CRITICAL - load average: 8.55, 6.28, 5.23 [20:14:23] PROBLEM - mw1 Current Load on mw1 is WARNING: WARNING - load average: 6.55, 6.92, 6.73 [20:15:32] PROBLEM - mw2 Current Load on mw2 is WARNING: WARNING - load average: 7.95, 6.38, 5.37 [20:17:30] RECOVERY - mw2 Current Load on mw2 is OK: OK - load average: 4.89, 5.80, 5.28 [20:18:27] RECOVERY - mw1 Current Load on mw1 is OK: OK - load average: 5.18, 6.42, 6.61 [20:29:14] [02puppet] 07paladox synchronize pull request 03#1122: Varnish: Tweak timeout config - 13https://git.io/JeEDk [20:29:15] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-7 [+0/-0/±1] 13https://git.io/JeEDg [20:29:17] [02miraheze/puppet] 07paladox 03b9a54bd - Update default.vcl [20:30:26] PROBLEM - mw1 Current Load on mw1 is CRITICAL: CRITICAL - load average: 8.40, 6.64, 6.44 [20:32:27] PROBLEM - mw1 Current Load on mw1 is WARNING: WARNING - load average: 6.92, 6.55, 6.42 [20:38:28] RECOVERY - mw1 Current Load on mw1 is OK: OK - load average: 6.02, 6.29, 6.38 [20:39:43] paladox: it's hard to use staffwiki in a 503 [20:40:05] ok [20:40:24] paladox: usual? [20:40:36] yes? [20:41:01] ok [20:41:02] back [20:41:34] PROBLEM - mw3 Current Load on mw3 is CRITICAL: CRITICAL - load average: 9.09, 7.96, 6.99 [20:42:29] PROBLEM - mw1 Current Load on mw1 is WARNING: WARNING - load average: 7.49, 6.92, 6.57 [20:43:31] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 7.32, 7.65, 6.99 [20:44:27] RECOVERY - mw1 Current Load on mw1 is OK: OK - load average: 6.25, 6.71, 6.54 [20:51:27] PROBLEM - mw3 Current Load on mw3 is CRITICAL: CRITICAL - load average: 9.38, 7.93, 7.26 [20:52:57] PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 4 datacenters are down: 107.191.126.23/cpweb, 2604:180:0:33b::2/cpweb, 128.199.139.216/cpweb, 81.4.109.133/cpweb [20:53:09] PROBLEM - cp2 Varnish Backends on cp2 is CRITICAL: 2 backends are down. mw1 mw2 [20:54:20] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 6 datacenters are down: 107.191.126.23/cpweb, 2604:180:0:33b::2/cpweb, 128.199.139.216/cpweb, 2400:6180:0:d0::403:f001/cpweb, 81.4.109.133/cpweb, 2a00:d880:5:8ea::ebc7/cpweb [20:54:50] PROBLEM - cp4 Varnish Backends on cp4 is CRITICAL: 2 backends are down. mw1 mw2 [20:54:58] PROBLEM - mw2 Current Load on mw2 is CRITICAL: CRITICAL - load average: 9.12, 8.05, 6.51 [20:56:32] PROBLEM - cp3 Varnish Backends on cp3 is CRITICAL: 1 backends are down. mw1 [20:57:03] * RhinosF1 sighs [20:58:44] [02puppet] 07paladox closed pull request 03#1122: Varnish: Tweak timeout config - 13https://git.io/JeEDk [20:58:45] RECOVERY - cp4 Varnish Backends on cp4 is OK: All 5 backends are healthy [20:58:45] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JeEDp [20:58:47] [02miraheze/puppet] 07paladox 0309b1459 - Varnish: Tweak timeout config (#1122) * Varnish: Tweak timeout config * Update default.vcl [20:58:48] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-7 [20:58:50] [02puppet] 07paladox deleted branch 03paladox-patch-7 - 13https://git.io/vbiAS [20:59:05] PROBLEM - mw2 Current Load on mw2 is WARNING: WARNING - load average: 7.94, 7.97, 6.85 [21:00:28] RECOVERY - cp3 Varnish Backends on cp3 is OK: All 5 backends are healthy [21:01:22] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 5.33, 7.43, 7.48 [21:03:08] RECOVERY - mw2 Current Load on mw2 is OK: OK - load average: 6.29, 6.35, 6.37 [21:04:55] PROBLEM - cp4 Varnish Backends on cp4 is CRITICAL: 1 backends are down. mw2 [21:05:25] PROBLEM - mw3 Current Load on mw3 is CRITICAL: CRITICAL - load average: 8.81, 7.85, 7.61 [21:05:35] PROBLEM - cp4 HTTPS on cp4 is CRITICAL: HTTP CRITICAL: HTTP/1.1 503 Backend fetch failed - 4040 bytes in 0.019 second response time [21:06:13] * RhinosF1 looks at paladox [21:06:31] PROBLEM - cp3 Varnish Backends on cp3 is CRITICAL: 1 backends are down. mw3 [21:07:21] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 7.24, 7.40, 7.46 [21:07:27] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JeEyt [21:07:29] [02miraheze/puppet] 07paladox 03c0c3b73 - Revert "Varnish: Tweak timeout config (#1122)" This reverts commit 09b145994656e3da5b3247b2e4d95197c1e5a3c3. [21:08:11] PROBLEM - cp4 Stunnel Http for mw2 on cp4 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [21:08:31] PROBLEM - cp3 Stunnel Http for mw2 on cp3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [21:10:09] RECOVERY - cp4 Stunnel Http for mw2 on cp4 is OK: HTTP OK: HTTP/1.1 200 OK - 24639 bytes in 0.013 second response time [21:10:22] PROBLEM - test1 MediaWiki Rendering on test1 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:10:30] PROBLEM - cp3 Stunnel Http for mw1 on cp3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [21:11:43] RECOVERY - cp2 Varnish Backends on cp2 is OK: All 5 backends are healthy [21:12:19] RECOVERY - test1 MediaWiki Rendering on test1 is OK: HTTP OK: HTTP/1.1 200 OK - 18975 bytes in 0.035 second response time [21:13:32] PROBLEM - mw3 Current Load on mw3 is CRITICAL: CRITICAL - load average: 8.66, 7.29, 7.29 [21:14:08] RECOVERY - cp4 HTTPS on cp4 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 1503 bytes in 0.824 second response time [21:14:34] RECOVERY - cp3 Stunnel Http for mw1 on cp3 is OK: HTTP OK: HTTP/1.1 200 OK - 24655 bytes in 0.633 second response time [21:15:57] PROBLEM - cp2 Varnish Backends on cp2 is CRITICAL: 2 backends are down. mw1 mw3 [21:17:35] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 7.17, 7.13, 7.19 [21:17:50] !log upgrade php 7.2 on mw1 [21:18:37] PROBLEM - cp2 Stunnel Http for mw2 on cp2 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [21:19:03] PROBLEM - cp3 Stunnel Http for mw1 on cp3 is CRITICAL: HTTP CRITICAL - No data received from host [21:19:06] PROBLEM - cp4 Stunnel Http for mw1 on cp4 is CRITICAL: HTTP CRITICAL - No data received from host [21:19:47] PROBLEM - cp4 Stunnel Http for mw2 on cp4 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [21:20:34] RECOVERY - cp2 Stunnel Http for mw2 on cp2 is OK: HTTP OK: HTTP/1.1 200 OK - 24655 bytes in 0.391 second response time [21:21:08] RECOVERY - cp3 Stunnel Http for mw2 on cp3 is OK: HTTP OK: HTTP/1.1 200 OK - 24661 bytes in 9.584 second response time [21:21:17] PROBLEM - cp2 Stunnel Http for mw1 on cp2 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 309 bytes in 0.357 second response time [21:21:39] PROBLEM - mw1 Puppet on mw1 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 8 minutes ago with 0 failures [21:21:45] RECOVERY - cp4 Stunnel Http for mw2 on cp4 is OK: HTTP OK: HTTP/1.1 200 OK - 24661 bytes in 0.009 second response time [21:22:34] RECOVERY - cp3 Varnish Backends on cp3 is OK: All 5 backends are healthy [21:25:32] PROBLEM - cp3 Stunnel Http for mw2 on cp3 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 309 bytes in 0.526 second response time [21:25:50] RECOVERY - cp2 Stunnel Http for mw1 on cp2 is OK: HTTP OK: HTTP/1.1 200 OK - 24661 bytes in 0.396 second response time [21:26:03] PROBLEM - cp4 Stunnel Http for mw2 on cp4 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 309 bytes in 0.003 second response time [21:26:30] PROBLEM - cp3 Varnish Backends on cp3 is CRITICAL: 1 backends are down. mw2 [21:26:39] PROBLEM - cp2 Stunnel Http for mw2 on cp2 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [21:28:04] RECOVERY - cp4 Stunnel Http for mw2 on cp4 is OK: HTTP OK: HTTP/1.1 200 OK - 24639 bytes in 0.005 second response time [21:28:35] RECOVERY - cp2 Stunnel Http for mw2 on cp2 is OK: HTTP OK: HTTP/1.1 200 OK - 24655 bytes in 0.390 second response time [21:33:26] PROBLEM - cp2 Stunnel Http for mw1 on cp2 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [21:33:36] PROBLEM - cp4 Stunnel Http for mw2 on cp4 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [21:33:49] PROBLEM - cp4 Stunnel Http for mw3 on cp4 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [21:34:01] PROBLEM - cp3 Stunnel Http for mw3 on cp3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [21:34:21] PROBLEM - cp2 Stunnel Http for mw3 on cp2 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [21:34:33] PROBLEM - mw1 HTTPS on mw1 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:34:34] RECOVERY - mw3 Current Load on mw3 is OK: OK - load average: 2.71, 5.31, 6.30 [21:34:35] RECOVERY - cp3 Stunnel Http for mw1 on cp3 is OK: HTTP OK: HTTP/1.1 200 OK - 24655 bytes in 0.658 second response time [21:37:45] RECOVERY - cp4 Stunnel Http for mw2 on cp4 is OK: HTTP OK: HTTP/1.1 200 OK - 24655 bytes in 0.003 second response time [21:39:11] PROBLEM - cp3 Stunnel Http for mw1 on cp3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [21:40:04] RECOVERY - cp4 Stunnel Http for mw3 on cp4 is OK: HTTP OK: HTTP/1.1 200 OK - 24639 bytes in 0.013 second response time [21:40:27] RECOVERY - cp3 Stunnel Http for mw3 on cp3 is OK: HTTP OK: HTTP/1.1 200 OK - 24655 bytes in 0.683 second response time [21:40:32] RECOVERY - cp2 Stunnel Http for mw3 on cp2 is OK: HTTP OK: HTTP/1.1 200 OK - 24639 bytes in 0.400 second response time [21:41:20] RECOVERY - cp3 Stunnel Http for mw2 on cp3 is OK: HTTP OK: HTTP/1.1 200 OK - 24661 bytes in 0.634 second response time [21:43:01] RECOVERY - mw1 HTTPS on mw1 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 442 bytes in 0.029 second response time [21:43:23] RECOVERY - cp3 Stunnel Http for mw1 on cp3 is OK: HTTP OK: HTTP/1.1 200 OK - 24661 bytes in 0.695 second response time [21:45:19] PROBLEM - cp3 Stunnel Http for mw3 on cp3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [21:45:25] PROBLEM - cp2 Stunnel Http for mw3 on cp2 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [21:47:07] RECOVERY - cp4 Stunnel Http for mw1 on cp4 is OK: HTTP OK: HTTP/1.1 200 OK - 24661 bytes in 0.004 second response time [21:48:55] PROBLEM - mw3 HTTPS on mw3 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:49:07] PROBLEM - cp2 Stunnel Http for mw2 on cp2 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [21:49:09] RECOVERY - cp2 Stunnel Http for mw1 on cp2 is OK: HTTP OK: HTTP/1.1 200 OK - 24639 bytes in 0.390 second response time [21:49:17] PROBLEM - cp4 Stunnel Http for mw2 on cp4 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [21:49:30] PROBLEM - mw1 Current Load on mw1 is WARNING: WARNING - load average: 7.45, 5.39, 4.21 [21:49:35] PROBLEM - cp4 Stunnel Http for mw3 on cp4 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [21:49:52] PROBLEM - cp3 Stunnel Http for mw2 on cp3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [21:50:30] 503 [21:50:43] oh, topic xd [21:50:50] RECOVERY - mw3 HTTPS on mw3 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 442 bytes in 0.015 second response time [21:51:38] PROBLEM - mw1 Current Load on mw1 is CRITICAL: CRITICAL - load average: 8.56, 6.56, 4.80 [21:51:39] RECOVERY - cp4 Stunnel Http for mw3 on cp4 is OK: HTTP OK: HTTP/1.1 200 OK - 24655 bytes in 0.011 second response time [21:51:42] RECOVERY - cp3 Stunnel Http for mw3 on cp3 is OK: HTTP OK: HTTP/1.1 200 OK - 24647 bytes in 0.683 second response time [21:51:49] RECOVERY - cp2 Stunnel Http for mw3 on cp2 is OK: HTTP OK: HTTP/1.1 200 OK - 24655 bytes in 0.390 second response time [21:53:13] hispano76_: yeah unfortunately [21:53:19] * RhinosF1 needs to sleep [21:56:28] PROBLEM - cp4 Stunnel Http for mw3 on cp4 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [21:56:43] PROBLEM - cp3 Stunnel Http for mw3 on cp3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [21:56:57] PROBLEM - cp2 Stunnel Http for mw3 on cp2 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [21:57:55] PROBLEM - mw1 Current Load on mw1 is WARNING: WARNING - load average: 6.17, 7.34, 5.79 [22:01:50] RECOVERY - cp2 Stunnel Http for mw2 on cp2 is OK: HTTP OK: HTTP/1.1 200 OK - 24639 bytes in 0.402 second response time [22:02:50] RECOVERY - cp3 Stunnel Http for mw2 on cp3 is OK: HTTP OK: HTTP/1.1 200 OK - 24639 bytes in 9.300 second response time [22:04:01] PROBLEM - mw1 Current Load on mw1 is CRITICAL: CRITICAL - load average: 10.78, 8.87, 6.93 [22:05:32] PROBLEM - mw3 Current Load on mw3 is CRITICAL: CRITICAL - load average: 8.37, 6.63, 5.75 [22:05:55] PROBLEM - cp2 Stunnel Http for mw2 on cp2 is CRITICAL: HTTP CRITICAL - No data received from host [22:07:38] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 7.03, 6.69, 5.87 [22:07:55] RECOVERY - cp2 Stunnel Http for mw2 on cp2 is OK: HTTP OK: HTTP/1.1 200 OK - 24655 bytes in 0.393 second response time [22:09:16] RECOVERY - cp2 Stunnel Http for mw3 on cp2 is OK: HTTP OK: HTTP/1.1 200 OK - 24639 bytes in 1.438 second response time [22:09:16] RECOVERY - cp3 Stunnel Http for mw3 on cp3 is OK: HTTP OK: HTTP/1.1 200 OK - 24639 bytes in 1.916 second response time [22:09:37] PROBLEM - mw3 Current Load on mw3 is CRITICAL: CRITICAL - load average: 8.53, 7.59, 6.30 [22:12:21] PROBLEM - mw1 Current Load on mw1 is WARNING: WARNING - load average: 5.54, 7.43, 7.12 [22:13:13] PROBLEM - cp3 Stunnel Http for mw2 on cp3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [22:13:50] PROBLEM - mw2 HTTPS on mw2 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:14:01] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 4.34, 6.98, 6.45 [22:14:09] PROBLEM - cp2 Stunnel Http for mw3 on cp2 is CRITICAL: HTTP CRITICAL - No data received from host [22:14:27] PROBLEM - cp3 Stunnel Http for mw3 on cp3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [22:14:31] PROBLEM - mw1 Current Load on mw1 is CRITICAL: CRITICAL - load average: 8.48, 7.93, 7.35 [22:14:37] PROBLEM - mw3 HTTPS on mw3 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:15:16] PROBLEM - cp2 Stunnel Http for mw2 on cp2 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [22:16:11] RECOVERY - mw3 Current Load on mw3 is OK: OK - load average: 3.80, 5.56, 5.97 [22:16:53] RECOVERY - mw3 HTTPS on mw3 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 442 bytes in 0.016 second response time [22:19:00] RECOVERY - cp3 Stunnel Http for mw3 on cp3 is OK: HTTP OK: HTTP/1.1 200 OK - 24639 bytes in 8.188 second response time [22:22:41] RECOVERY - mw2 HTTPS on mw2 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 442 bytes in 0.016 second response time [22:22:50] RECOVERY - cp2 Stunnel Http for mw3 on cp2 is OK: HTTP OK: HTTP/1.1 200 OK - 24639 bytes in 0.411 second response time [22:23:11] !log reboot mw1 [22:23:13] RECOVERY - mw1 Current Load on mw1 is OK: OK - load average: 0.00, 0.00, 0.00 [22:23:40] PROBLEM - cp3 Stunnel Http for mw3 on cp3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [22:25:38] PROBLEM - cp3 Stunnel Http for mw1 on cp3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [22:27:21] PROBLEM - cp2 Stunnel Http for mw1 on cp2 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [22:27:42] PROBLEM - cp2 Stunnel Http for mw3 on cp2 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [22:28:13] PROBLEM - cp4 Stunnel Http for mw1 on cp4 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [22:29:27] RECOVERY - cp2 Stunnel Http for mw1 on cp2 is OK: HTTP OK: HTTP/1.1 200 OK - 24639 bytes in 6.039 second response time [22:29:40] RECOVERY - cp2 Stunnel Http for mw3 on cp2 is OK: HTTP OK: HTTP/1.1 200 OK - 24655 bytes in 0.389 second response time [22:29:59] RECOVERY - cp3 Stunnel Http for mw3 on cp3 is OK: HTTP OK: HTTP/1.1 200 OK - 24639 bytes in 1.582 second response time [22:30:19] RECOVERY - cp4 Stunnel Http for mw1 on cp4 is OK: HTTP OK: HTTP/1.1 200 OK - 24639 bytes in 9.540 second response time [22:30:24] RECOVERY - cp2 Stunnel Http for mw2 on cp2 is OK: HTTP OK: HTTP/1.1 200 OK - 24655 bytes in 0.397 second response time [22:30:43] RECOVERY - cp3 Stunnel Http for mw2 on cp3 is OK: HTTP OK: HTTP/1.1 200 OK - 24639 bytes in 0.635 second response time [22:30:49] RECOVERY - cp4 Stunnel Http for mw2 on cp4 is OK: HTTP OK: HTTP/1.1 200 OK - 24639 bytes in 1.201 second response time [22:32:49] RECOVERY - cp4 Stunnel Http for mw3 on cp4 is OK: HTTP OK: HTTP/1.1 200 OK - 24639 bytes in 3.792 second response time [22:33:25] RECOVERY - cp2 Varnish Backends on cp2 is OK: All 5 backends are healthy [22:33:40] RECOVERY - cp4 Varnish Backends on cp4 is OK: All 5 backends are healthy [22:33:41] PROBLEM - lizardfs4 Puppet on lizardfs4 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [22:34:19] RECOVERY - cp3 Stunnel Http for mw1 on cp3 is OK: HTTP OK: HTTP/1.1 200 OK - 24639 bytes in 6.885 second response time [22:34:34] PROBLEM - lizardfs5 Current Load on lizardfs5 is CRITICAL: CRITICAL - load average: 5.26, 3.76, 2.19 [22:35:42] PROBLEM - lizardfs4 Puppet on lizardfs4 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 3 hours ago with 0 failures [22:36:14] PROBLEM - mw3 Current Load on mw3 is CRITICAL: CRITICAL - load average: 9.71, 7.97, 6.58 [22:36:14] PROBLEM - cp2 Stunnel Http for mw3 on cp2 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [22:36:47] RECOVERY - lizardfs5 Current Load on lizardfs5 is OK: OK - load average: 1.51, 2.81, 2.04 [22:37:01] PROBLEM - cp2 Stunnel Http for mw2 on cp2 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [22:38:13] PROBLEM - mw3 HTTPS on mw3 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:38:28] PROBLEM - cp4 Varnish Backends on cp4 is CRITICAL: 3 backends are down. mw1 mw2 mw3 [22:38:31] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 5.92, 7.60, 6.66 [22:39:07] RECOVERY - cp2 Stunnel Http for mw2 on cp2 is OK: HTTP OK: HTTP/1.1 200 OK - 24655 bytes in 0.390 second response time [22:39:12] PROBLEM - cp3 Stunnel Http for mw3 on cp3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [22:40:03] PROBLEM - cp4 Stunnel Http for mw3 on cp4 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [22:40:26] PROBLEM - cp2 Varnish Backends on cp2 is CRITICAL: 2 backends are down. mw2 mw3 [22:40:44] RECOVERY - mw3 Current Load on mw3 is OK: OK - load average: 2.34, 5.51, 6.00 [22:42:19] PROBLEM - mw2 Puppet on mw2 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 10 minutes ago with 0 failures [22:42:19] RECOVERY - mw3 HTTPS on mw3 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 442 bytes in 0.031 second response time [22:42:25] PROBLEM - mw1 Current Load on mw1 is CRITICAL: CRITICAL - load average: 8.81, 6.60, 3.80 [22:42:35] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JeE9v [22:42:36] [02miraheze/puppet] 07paladox 03f7ebd2c - Update mediawiki.conf [22:45:21] PROBLEM - mw2 Current Load on mw2 is CRITICAL: CRITICAL - load average: 9.75, 7.96, 6.41 [22:46:36] PROBLEM - mw1 Current Load on mw1 is WARNING: WARNING - load average: 7.36, 7.12, 4.65 [22:47:19] RECOVERY - cp2 Stunnel Http for mw3 on cp2 is OK: HTTP OK: HTTP/1.1 200 OK - 24639 bytes in 0.391 second response time [22:47:38] PROBLEM - mw2 Current Load on mw2 is WARNING: WARNING - load average: 5.97, 7.42, 6.42 [22:48:50] PROBLEM - mw1 Current Load on mw1 is CRITICAL: CRITICAL - load average: 10.74, 8.49, 5.50 [22:48:57] RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online [22:49:04] :/ [22:49:50] PROBLEM - mw2 Current Load on mw2 is CRITICAL: CRITICAL - load average: 11.84, 8.89, 7.06 [22:51:03] PROBLEM - mw3 HTTPS on mw3 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:51:55] PROBLEM - cp2 Stunnel Http for mw3 on cp2 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [22:52:57] PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 3 datacenters are down: 107.191.126.23/cpweb, 2604:180:0:33b::2/cpweb, 128.199.139.216/cpweb [22:52:58] RECOVERY - mw3 HTTPS on mw3 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 442 bytes in 0.013 second response time [22:53:50] I guess I'd better wait until tomorrow to edit? [22:53:51] RECOVERY - cp2 Stunnel Http for mw3 on cp2 is OK: HTTP OK: HTTP/1.1 200 OK - 24639 bytes in 0.398 second response time [22:54:08] RECOVERY - cp3 Stunnel Http for mw3 on cp3 is OK: HTTP OK: HTTP/1.1 200 OK - 24661 bytes in 0.674 second response time [22:54:29] PROBLEM - lizardfs5 Puppet on lizardfs5 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [22:54:41] paladox Voidwalker JohnLewis Reception123 RhinosF1 Zppix sparr [22:55:03] hispano76_ yup [22:56:28] PROBLEM - lizardfs5 Puppet on lizardfs5 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 3 hours ago with 0 failures [22:56:47] RECOVERY - cp4 Stunnel Http for mw3 on cp4 is OK: HTTP OK: HTTP/1.1 200 OK - 24639 bytes in 0.014 second response time [22:56:49] PROBLEM - lizardfs4 Current Load on lizardfs4 is CRITICAL: CRITICAL - load average: 4.51, 2.45, 1.84 [22:57:39] PROBLEM - lizardfs5 Current Load on lizardfs5 is CRITICAL: CRITICAL - load average: 5.91, 4.68, 2.92 [22:57:59] PROBLEM - mw2 Current Load on mw2 is WARNING: WARNING - load average: 4.88, 7.02, 7.08 [22:58:47] RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online [22:59:00] PROBLEM - mw1 Current Load on mw1 is WARNING: WARNING - load average: 4.52, 7.13, 6.75 [22:59:04] RECOVERY - cp3 Varnish Backends on cp3 is OK: All 5 backends are healthy [22:59:12] RECOVERY - cp4 Varnish Backends on cp4 is OK: All 5 backends are healthy [22:59:13] RECOVERY - cp2 Varnish Backends on cp2 is OK: All 5 backends are healthy [23:00:25] PROBLEM - mw3 Current Load on mw3 is CRITICAL: CRITICAL - load average: 10.43, 7.94, 6.56 [23:00:46] RECOVERY - lizardfs4 Current Load on lizardfs4 is OK: OK - load average: 3.13, 3.16, 2.31 [23:00:58] RECOVERY - mw1 Current Load on mw1 is OK: OK - load average: 4.50, 6.14, 6.42 [23:01:05] so, what's going on exactly? [23:02:28] RECOVERY - mw2 Current Load on mw2 is OK: OK - load average: 4.41, 6.07, 6.69 [23:03:03] PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 6 datacenters are down: 107.191.126.23/cpweb, 2604:180:0:33b::2/cpweb, 128.199.139.216/cpweb, 2400:6180:0:d0::403:f001/cpweb, 81.4.109.133/cpweb, 2a00:d880:5:8ea::ebc7/cpweb [23:03:24] PROBLEM - cp3 Varnish Backends on cp3 is CRITICAL: 2 backends are down. mw2 mw3 [23:03:32] PROBLEM - cp4 Varnish Backends on cp4 is CRITICAL: 1 backends are down. mw2 [23:03:39] PROBLEM - cp2 Varnish Backends on cp2 is CRITICAL: 1 backends are down. mw2 [23:04:41] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 7.49, 7.79, 6.84 [23:04:56] PROBLEM - cp3 Stunnel Http for mw3 on cp3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [23:05:05] PROBLEM - lizardfs4 Current Load on lizardfs4 is CRITICAL: CRITICAL - load average: 4.91, 4.01, 2.85 [23:07:01] RECOVERY - cp3 Stunnel Http for mw3 on cp3 is OK: HTTP OK: HTTP/1.1 200 OK - 24639 bytes in 9.349 second response time [23:07:05] PROBLEM - lizardfs4 Current Load on lizardfs4 is WARNING: WARNING - load average: 3.75, 3.64, 2.83 [23:07:59] RECOVERY - lizardfs5 Current Load on lizardfs5 is OK: OK - load average: 1.90, 3.33, 3.39 [23:08:50] RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online [23:09:01] RECOVERY - lizardfs4 Current Load on lizardfs4 is OK: OK - load average: 3.22, 3.36, 2.82 [23:09:22] PROBLEM - mw1 Current Load on mw1 is CRITICAL: CRITICAL - load average: 8.90, 8.28, 7.17 [23:09:25] RECOVERY - cp3 Varnish Backends on cp3 is OK: All 5 backends are healthy [23:09:40] RECOVERY - cp4 Varnish Backends on cp4 is OK: All 5 backends are healthy [23:09:47] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [23:10:36] PROBLEM - mw3 Current Load on mw3 is CRITICAL: CRITICAL - load average: 8.90, 8.10, 7.25 [23:11:18] PROBLEM - lizardfs5 Puppet on lizardfs5 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [23:11:21] PROBLEM - mw1 Current Load on mw1 is WARNING: WARNING - load average: 6.41, 7.78, 7.14 [23:11:58] PROBLEM - lizardfs5 Current Load on lizardfs5 is CRITICAL: CRITICAL - load average: 6.50, 4.64, 3.85 [23:13:16] PROBLEM - lizardfs5 Puppet on lizardfs5 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 4 hours ago with 0 failures [23:13:37] PROBLEM - mw2 Current Load on mw2 is WARNING: WARNING - load average: 6.65, 7.46, 7.05 [23:14:50] PROBLEM - lizardfs4 Current Load on lizardfs4 is WARNING: WARNING - load average: 3.52, 3.24, 2.88 [23:15:18] PROBLEM - mw1 Current Load on mw1 is CRITICAL: CRITICAL - load average: 8.09, 8.20, 7.46 [23:15:54] RECOVERY - cp2 Varnish Backends on cp2 is OK: All 5 backends are healthy [23:15:57] PROBLEM - lizardfs5 Current Load on lizardfs5 is WARNING: WARNING - load average: 2.34, 3.75, 3.70 [23:16:12] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JeE9l [23:16:14] [02miraheze/puppet] 07paladox 03537d792 - php: Increase opcache [23:16:36] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JeE98 [23:16:38] [02miraheze/puppet] 07paladox 036370b53 - Update mediawiki.pp [23:16:46] RECOVERY - lizardfs4 Current Load on lizardfs4 is OK: OK - load average: 1.88, 2.73, 2.74 [23:17:34] PROBLEM - mw2 Current Load on mw2 is CRITICAL: CRITICAL - load average: 9.30, 7.62, 7.12 [23:17:56] RECOVERY - mw1 Puppet on mw1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [23:18:50] PROBLEM - mw2 Puppet on mw2 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 18 seconds ago with 1 failures. Failed resources (up to 3 shown): Mount[/mnt/mediawiki-static] [23:19:34] PROBLEM - mw2 Current Load on mw2 is WARNING: WARNING - load average: 6.91, 7.68, 7.23 [23:20:23] PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 5 datacenters are down: 107.191.126.23/cpweb, 2604:180:0:33b::2/cpweb, 128.199.139.216/cpweb, 2400:6180:0:d0::403:f001/cpweb, 2a00:d880:5:8ea::ebc7/cpweb [23:20:54] PROBLEM - lizardfs4 Current Load on lizardfs4 is CRITICAL: CRITICAL - load average: 4.82, 5.03, 3.77 [23:21:24] PROBLEM - lizardfs5 Puppet on lizardfs5 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [23:21:53] PROBLEM - mw3 Puppet on mw3 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Mount[/mnt/mediawiki-static] [23:22:05] PROBLEM - lizardfs5 Current Load on lizardfs5 is CRITICAL: CRITICAL - load average: 5.50, 4.15, 3.78 [23:22:48] RECOVERY - mw2 Puppet on mw2 is OK: OK: Puppet is currently enabled, last run 20 seconds ago with 0 failures [23:22:51] boom, another 503! [23:23:14] PROBLEM - mw1 Current Load on mw1 is WARNING: WARNING - load average: 7.24, 7.92, 7.67 [23:23:22] PROBLEM - lizardfs5 Puppet on lizardfs5 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 4 hours ago with 0 failures [23:24:03] RECOVERY - mw3 Puppet on mw3 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [23:24:22] PROBLEM - cp2 Varnish Backends on cp2 is CRITICAL: 1 backends are down. mw1 [23:24:52] PROBLEM - cp4 Varnish Backends on cp4 is CRITICAL: 1 backends are down. mw2 [23:25:21] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 5 datacenters are down: 107.191.126.23/cpweb, 128.199.139.216/cpweb, 2400:6180:0:d0::403:f001/cpweb, 81.4.109.133/cpweb, 2a00:d880:5:8ea::ebc7/cpweb [23:25:55] RECOVERY - mw2 Current Load on mw2 is OK: OK - load average: 4.68, 6.09, 6.75 [23:26:22] RECOVERY - cp2 Varnish Backends on cp2 is OK: All 5 backends are healthy [23:26:29] RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online [23:26:49] RECOVERY - cp4 Varnish Backends on cp4 is OK: All 5 backends are healthy [23:27:20] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [23:28:20] k6ka yeh :( [23:29:04] RECOVERY - lizardfs4 Puppet on lizardfs4 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [23:29:17] PROBLEM - mw1 Current Load on mw1 is CRITICAL: CRITICAL - load average: 10.25, 8.35, 7.83 [23:29:39] PROBLEM - lizardfs5 Puppet on lizardfs5 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [23:31:23] PROBLEM - mw1 Current Load on mw1 is WARNING: WARNING - load average: 6.19, 7.72, 7.68 [23:31:40] RECOVERY - lizardfs5 Puppet on lizardfs5 is OK: OK: Puppet is currently enabled, last run 4 minutes ago with 0 failures [23:33:09] PROBLEM - cp4 Varnish Backends on cp4 is CRITICAL: 3 backends are down. mw1 mw2 mw3 [23:33:29] PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 6 datacenters are down: 107.191.126.23/cpweb, 2604:180:0:33b::2/cpweb, 128.199.139.216/cpweb, 2400:6180:0:d0::403:f001/cpweb, 81.4.109.133/cpweb, 2a00:d880:5:8ea::ebc7/cpweb [23:33:29] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 6 datacenters are down: 107.191.126.23/cpweb, 2604:180:0:33b::2/cpweb, 128.199.139.216/cpweb, 2400:6180:0:d0::403:f001/cpweb, 81.4.109.133/cpweb, 2a00:d880:5:8ea::ebc7/cpweb [23:33:53] PROBLEM - cp3 Varnish Backends on cp3 is CRITICAL: 3 backends are down. mw1 mw2 mw3 [23:35:06] RECOVERY - cp4 Varnish Backends on cp4 is OK: All 5 backends are healthy [23:35:25] RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online [23:35:27] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [23:37:30] PROBLEM - mw1 Current Load on mw1 is CRITICAL: CRITICAL - load average: 8.42, 7.02, 7.33 [23:37:57] RECOVERY - cp3 Varnish Backends on cp3 is OK: All 5 backends are healthy [23:43:26] PROBLEM - mw1 Current Load on mw1 is WARNING: WARNING - load average: 8.00, 7.97, 7.69 [23:46:25] !log restart lizardfs-chunkserver [23:47:04] PROBLEM - mw2 Puppet on mw2 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 4 minutes ago with 1 failures. Failed resources (up to 3 shown): Mount[/mnt/mediawiki-static] [23:47:11] PROBLEM - mw1 Puppet on mw1 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Mount[/mnt/mediawiki-static] [23:49:37] PROBLEM - lizardfs4 Current Load on lizardfs4 is WARNING: WARNING - load average: 1.15, 2.37, 3.99 [23:49:38] PROBLEM - cp4 Stunnel Http for mw2 on cp4 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [23:50:42] PROBLEM - mw3 HTTPS on mw3 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:50:53] PROBLEM - cp4 Stunnel Http for mw1 on cp4 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [23:50:54] PROBLEM - cp3 Stunnel Http for mw2 on cp3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [23:51:06] PROBLEM - cp3 Stunnel Http for mw3 on cp3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [23:51:06] RECOVERY - mw1 Current Load on mw1 is OK: OK - load average: 1.84, 5.08, 6.60 [23:51:07] PROBLEM - cp2 Stunnel Http for mw1 on cp2 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [23:51:21] PROBLEM - cp4 Varnish Backends on cp4 is CRITICAL: 3 backends are down. mw1 mw2 mw3 [23:51:30] PROBLEM - cp2 Stunnel Http for mw3 on cp2 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [23:51:49] PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 6 datacenters are down: 107.191.126.23/cpweb, 2604:180:0:33b::2/cpweb, 128.199.139.216/cpweb, 2400:6180:0:d0::403:f001/cpweb, 81.4.109.133/cpweb, 2a00:d880:5:8ea::ebc7/cpweb [23:51:51] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 6 datacenters are down: 107.191.126.23/cpweb, 2604:180:0:33b::2/cpweb, 128.199.139.216/cpweb, 2400:6180:0:d0::403:f001/cpweb, 81.4.109.133/cpweb, 2a00:d880:5:8ea::ebc7/cpweb [23:51:54] PROBLEM - cp3 Varnish Backends on cp3 is CRITICAL: 3 backends are down. mw1 mw2 mw3 [23:52:01] PROBLEM - cp3 Stunnel Http for mw1 on cp3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [23:52:01] PROBLEM - cp2 Varnish Backends on cp2 is CRITICAL: 2 backends are down. mw1 mw3 [23:52:01] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 3.03, 5.49, 7.17 [23:52:02] RECOVERY - cp4 Stunnel Http for mw2 on cp4 is OK: HTTP OK: HTTP/1.1 200 OK - 24639 bytes in 0.018 second response time [23:52:52] RECOVERY - cp3 Stunnel Http for mw2 on cp3 is OK: HTTP OK: HTTP/1.1 200 OK - 24655 bytes in 0.642 second response time [23:53:09] RECOVERY - cp2 Stunnel Http for mw1 on cp2 is OK: HTTP OK: HTTP/1.1 200 OK - 24639 bytes in 0.395 second response time [23:54:08] RECOVERY - lizardfs4 Current Load on lizardfs4 is OK: OK - load average: 1.10, 1.67, 3.28 [23:54:12] PROBLEM - lizardfs5 Current Load on lizardfs5 is WARNING: WARNING - load average: 1.09, 2.09, 3.65 [23:54:12] PROBLEM - mw3 Current Load on mw3 is CRITICAL: CRITICAL - load average: 8.79, 6.95, 7.49 [23:54:26] RECOVERY - mw2 Puppet on mw2 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [23:54:32] RECOVERY - mw1 Puppet on mw1 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [23:54:45] RECOVERY - mw3 HTTPS on mw3 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 442 bytes in 0.021 second response time [23:56:21] PROBLEM - cp4 Stunnel Http for mw3 on cp4 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [23:56:30] RECOVERY - lizardfs5 Current Load on lizardfs5 is OK: OK - load average: 1.01, 1.69, 3.29 [23:56:31] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 3.80, 5.81, 7.00 [23:57:04] PROBLEM - cp2 Stunnel Http for mw2 on cp2 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [23:57:21] PROBLEM - cp4 Stunnel Http for mw2 on cp4 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [23:57:53] PROBLEM - cp3 Stunnel Http for mw2 on cp3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [23:58:08] PROBLEM - cp2 Stunnel Http for mw1 on cp2 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [23:58:42] RECOVERY - mw3 Current Load on mw3 is OK: OK - load average: 3.15, 4.71, 6.42 [23:58:45] PROBLEM - mw1 HTTPS on mw1 is CRITICAL: CRITICAL - Socket timeout after 10 seconds