[03:20:10] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JeCTl [03:20:11] [02miraheze/services] 07MirahezeSSLBot 038b54226 - BOT: Updating services config for wikis [04:47:00] PROBLEM - cp3 Varnish Backends on cp3 is CRITICAL: 1 backends are down. mw1 [04:47:24] PROBLEM - cp4 Varnish Backends on cp4 is CRITICAL: 1 backends are down. mw1 [04:47:24] PROBLEM - cp2 Varnish Backends on cp2 is CRITICAL: 1 backends are down. mw1 [04:49:57] RECOVERY - cp3 Varnish Backends on cp3 is OK: All 5 backends are healthy [04:50:28] RECOVERY - cp4 Varnish Backends on cp4 is OK: All 5 backends are healthy [04:50:29] RECOVERY - cp2 Varnish Backends on cp2 is OK: All 5 backends are healthy [06:26:20] RECOVERY - cp3 Disk Space on cp3 is OK: DISK OK - free space: / 3005 MB (12% inode=94%); [08:18:38] PROBLEM - mw3 Puppet on mw3 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 5 minutes ago with 1 failures. Failed resources (up to 3 shown): Package[php7.2-redis] [08:24:00] RECOVERY - mw3 Puppet on mw3 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [09:02:51] PROBLEM - cp4 Varnish Backends on cp4 is CRITICAL: 1 backends are down. mw1 [09:02:56] PROBLEM - cp2 Varnish Backends on cp2 is CRITICAL: 1 backends are down. mw1 [09:05:48] RECOVERY - cp4 Varnish Backends on cp4 is OK: All 5 backends are healthy [09:08:48] RECOVERY - cp2 Varnish Backends on cp2 is OK: All 5 backends are healthy [12:25:08] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JeCtM [12:25:10] [02miraheze/services] 07MirahezeSSLBot 03a0b1d30 - BOT: Updating services config for wikis [12:40:25] ok [12:42:52] PROBLEM - cp4 Stunnel Http for mw3 on cp4 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [12:43:25] PROBLEM - cp3 Varnish Backends on cp3 is CRITICAL: 1 backends are down. mw3 [12:43:26] PROBLEM - cp4 Varnish Backends on cp4 is CRITICAL: 1 backends are down. mw3 [12:45:35] RECOVERY - cp4 Stunnel Http for mw3 on cp4 is OK: HTTP OK: HTTP/1.1 200 OK - 24586 bytes in 0.004 second response time [12:46:00] RECOVERY - cp3 Varnish Backends on cp3 is OK: All 5 backends are healthy [12:46:03] RECOVERY - cp4 Varnish Backends on cp4 is OK: All 5 backends are healthy [13:02:00] PROBLEM - cp3 Disk Space on cp3 is WARNING: DISK WARNING - free space: / 2625 MB (10% inode=94%); [13:22:35] RECOVERY - cp3 Disk Space on cp3 is OK: DISK OK - free space: / 2725 MB (11% inode=94%); [15:01:45] PROBLEM - cp3 Disk Space on cp3 is WARNING: DISK WARNING - free space: / 2647 MB (10% inode=94%); [17:30:10] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JeC3G [17:30:11] [02miraheze/services] 07MirahezeSSLBot 034cf4190 - BOT: Updating services config for wikis [18:09:38] PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 2 datacenters are down: 2604:180:0:33b::2/cpweb, 128.199.139.216/cpweb [18:11:39] PROBLEM - cp3 Varnish Backends on cp3 is CRITICAL: 2 backends are down. mw1 mw3 [18:14:35] RECOVERY - cp3 Varnish Backends on cp3 is OK: All 5 backends are healthy [18:15:11] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JeC3p [18:15:13] [02miraheze/services] 07MirahezeSSLBot 03ba7c63c - BOT: Updating services config for wikis [18:15:28] RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online [19:19:47] Hey hispano76_ [19:34:07] PROBLEM - mw3 Current Load on mw3 is CRITICAL: CRITICAL - load average: 14.88, 9.78, 6.75 [19:41:55] RECOVERY - mw3 Current Load on mw3 is OK: OK - load average: 3.74, 6.79, 6.77 [19:45:20] hi RhinosF1 :) [19:45:37] hispano76_: how are you? [19:46:04] RhinosF1 good ;) [19:53:07] Nice to hear! [20:28:46] !log sudo -u www-data php /srv/mediawiki/w/maintenance/resetUserEmail.php ZppixBot tools.zppixbot@tools.wmflabs.org --wiki=loginwiki (requested, operator known on irc) [20:28:50] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [20:46:42] PROBLEM - wiesepedia.ga - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiesepedia.ga' expires in 15 day(s) (Mon 21 Oct 2019 08:41:44 PM GMT +0000). [20:46:48] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JeCZo [20:46:50] [02miraheze/ssl] 07MirahezeSSLBot 03cf28a6e - Bot: Update SSL cert for wiesepedia.ga [20:54:29] RECOVERY - wiesepedia.ga - LetsEncrypt on sslhost is OK: OK - Certificate 'wiesepedia.ga' will expire on Fri 20 Dec 2019 11:44:56 PM GMT +0000. [21:03:49] PROBLEM - tradwiki.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'www.tradwiki.org' expires in 15 day(s) (Mon 21 Oct 2019 09:00:23 PM GMT +0000). [21:05:46] PROBLEM - www.tradwiki.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'www.tradwiki.org' expires in 15 day(s) (Mon 21 Oct 2019 09:00:23 PM GMT +0000). [21:05:59] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JeCZx [21:06:01] [02miraheze/ssl] 07MirahezeSSLBot 03eaaa2f3 - Bot: Update SSL cert for www.tradwiki.org [21:13:36] RECOVERY - www.tradwiki.org - LetsEncrypt on sslhost is OK: OK - Certificate 'www.tradwiki.org' will expire on Fri 03 Jan 2020 08:05:53 PM GMT +0000. [21:14:12] RECOVERY - tradwiki.org - LetsEncrypt on sslhost is OK: OK - Certificate 'www.tradwiki.org' will expire on Fri 03 Jan 2020 08:05:53 PM GMT +0000. [22:51:59] !log depool mw1 and repool [22:52:08] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [22:54:12] PROBLEM - test1 Puppet on test1 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 1 minute ago with 1 failures [22:55:24] [02miraheze/mediawiki] 07paladox pushed 031 commit to 03REL1_33 [+0/-0/±1] 13https://git.io/JeCcc [22:55:26] [02miraheze/mediawiki] 07paladox 030bed956 - Update DN [22:58:16] PROBLEM - mw1 Puppet on mw1 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 5 minutes ago with 1 failures. Failed resources (up to 3 shown): Mount[/mnt/mediawiki-static] [23:00:54] PROBLEM - mw1 Puppet on mw1 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 8 minutes ago with 1 failures [23:02:11] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JeCcB [23:02:13] [02miraheze/puppet] 07paladox 0329c0b36 - mediawiki: Set mfsioretries to 1 for lizardfs client [23:04:07] !log depool and repool mw[23] [23:04:12] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [23:10:24] !log reboot mw3 [23:12:18] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 6 datacenters are down: 107.191.126.23/cpweb, 2604:180:0:33b::2/cpweb, 128.199.139.216/cpweb, 2400:6180:0:d0::403:f001/cpweb, 81.4.109.133/cpweb, 2a00:d880:5:8ea::ebc7/cpweb [23:12:33] known ^ [23:14:58] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [23:19:06] PROBLEM - test1 Puppet on test1 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Mount[/mnt/mediawiki-static] [23:27:04] RECOVERY - test1 Puppet on test1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [23:45:46] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JeCcj [23:45:48] [02miraheze/puppet] 07paladox 03e860223 - Update mfschunkserver.cfg.erb [23:50:02] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JeCCJ [23:50:03] [02miraheze/puppet] 07paladox 039d23218 - Update mfsmaster.cfg.erb [23:58:47] PROBLEM - mw1 Puppet on mw1 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Mount[/mnt/mediawiki-static]