[00:03:44] Hispano76: I will tomorrow (Sunday) [00:06:24] PROBLEM - cp3 Disk Space on cp3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:08:30] PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 2 datacenters are down: 128.199.139.216/cpweb, 2400:6180:0:d0::403:f001/cpweb [00:09:05] PROBLEM - cp3 Puppet on cp3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:09:33] PROBLEM - cp3 Varnish Backends on cp3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:09:44] PROBLEM - cp3 SSH on cp3 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:09:52] PROBLEM - cp3 HTTP 4xx/5xx ERROR Rate on cp3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:10:27] PROBLEM - cp3 Disk Space on cp3 is WARNING: DISK WARNING - free space: / 2258 MB (9% inode=94%); [00:10:30] RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online [00:11:01] RECOVERY - cp3 Puppet on cp3 is OK: OK: Puppet is currently enabled, last run 7 minutes ago with 0 failures [00:11:31] RECOVERY - cp3 Varnish Backends on cp3 is OK: All 5 backends are healthy [00:11:39] RECOVERY - cp3 SSH on cp3 is OK: SSH OK - OpenSSH_7.4p1 Debian-10+deb9u6 (protocol 2.0) [00:11:50] RECOVERY - cp3 HTTP 4xx/5xx ERROR Rate on cp3 is OK: OK - NGINX Error Rate is 13% [01:28:24] PROBLEM - cp3 Disk Space on cp3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [01:29:05] PROBLEM - cp3 Puppet on cp3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [01:29:33] PROBLEM - cp3 Varnish Backends on cp3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [01:29:44] PROBLEM - cp3 SSH on cp3 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:29:52] PROBLEM - cp3 HTTP 4xx/5xx ERROR Rate on cp3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [01:30:06] PROBLEM - cp3 Current Load on cp3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [01:30:14] PROBLEM - cp3 HTTPS on cp3 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:30:24] PROBLEM - cp3 Disk Space on cp3 is WARNING: DISK WARNING - free space: / 2216 MB (9% inode=94%); [01:30:30] PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 1 datacenter is down: 128.199.139.216/cpweb [01:31:01] RECOVERY - cp3 Puppet on cp3 is OK: OK: Puppet is currently enabled, last run 7 minutes ago with 0 failures [01:31:31] RECOVERY - cp3 Varnish Backends on cp3 is OK: All 5 backends are healthy [01:31:39] RECOVERY - cp3 SSH on cp3 is OK: SSH OK - OpenSSH_7.4p1 Debian-10+deb9u6 (protocol 2.0) [01:31:50] RECOVERY - cp3 HTTP 4xx/5xx ERROR Rate on cp3 is OK: OK - NGINX Error Rate is 3% [01:32:01] RECOVERY - cp3 Current Load on cp3 is OK: OK - load average: 0.03, 0.07, 0.11 [01:32:13] RECOVERY - cp3 HTTPS on cp3 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 1514 bytes in 0.697 second response time [01:32:30] RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online [02:48:15] RECOVERY - bacula1 Bacula Databases db4 on bacula1 is OK: OK: Full, 764611 files, 70.58GB, 2019-08-04 02:46:00 (2.2 minutes ago) [06:26:15] RECOVERY - cp3 Disk Space on cp3 is OK: DISK OK - free space: / 2970 MB (12% inode=94%); [13:12:15] PROBLEM - cp3 Disk Space on cp3 is WARNING: DISK WARNING - free space: / 2649 MB (10% inode=94%); [13:24:50] !log renaming ucroniaswiki to ucroniasantiguowiki - T4606 [13:24:55] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [13:30:13] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fjHWV [13:30:15] [02miraheze/services] 07MirahezeSSLBot 034d7d560 - BOT: Updating services config for wikis [13:32:46] thanks paladox [13:36:52] your welcome :) [13:50:10] https://ucronias.miraheze.org/wiki/Especial:CambiosRecientes bug? paladox [13:50:11] [ Cambios recientes - ucroniaswiki ] - ucronias.miraheze.org [13:50:37] that has got to be a bug [13:50:48] it was 404 only a minute after i renamed the wiki [13:50:52] cc JohnLewis ^ [13:51:41] !log drop ucroniaswiki db [13:51:46] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [13:52:11] 20190804133439 [13:52:15] that is a def bug [14:00:06] [02puppet] 07paladox created branch 03paladox-patch-2 - 13https://git.io/vbiAS [14:00:08] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-2 [+0/-0/±1] 13https://git.io/fjHlJ [14:00:09] [02miraheze/puppet] 07paladox 030e71d98 - Add "Bacula Databases db5" monitoring check [14:00:16] [02puppet] 07paladox opened pull request 03#1059: Add "Bacula Databases db5" monitoring check - 13https://git.io/fjHlU [14:01:00] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-2 [+0/-0/±1] 13https://git.io/fjHlk [14:01:02] [02miraheze/puppet] 07paladox 03d0189be - Update nrpe.cfg.erb [14:01:03] [02puppet] 07paladox synchronize pull request 03#1059: Add "Bacula Databases db5" monitoring check - 13https://git.io/fjHlU [14:05:13] !log change /etc/hosts on db5 to have "185.52.1.89 db5.miraheze.org db5" instead of "127.0.1.1 db5.miraheze.org db5" [14:05:17] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [14:13:03] [02puppet] 07paladox closed pull request 03#1059: Add "Bacula Databases db5" monitoring check - 13https://git.io/fjHlU [14:13:04] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±2] 13https://git.io/fjHll [14:13:06] [02miraheze/puppet] 07paladox 036d68fe5 - Add "Bacula Databases db5" monitoring check (#1059) * Add "Bacula Databases db5" monitoring check * Update nrpe.cfg.erb [14:13:07] [02puppet] 07paladox deleted branch 03paladox-patch-2 - 13https://git.io/vbiAS [14:13:09] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-2 [14:14:25] paladox, Could you see if everything's in order? that's to recreate UcroniasWiki. without any bug or other inconvenience. https://meta.miraheze.org/wiki/Special:RequestWikiQueue/8851 [14:14:26] [ Wiki requests queue - Miraheze Meta ] - meta.miraheze.org [14:14:26] !log drop from cw_wikis where wiki_dbname = 'ucroniaswiki'; -db4 [14:14:30] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [14:14:37] oh :) [14:15:06] !log delete from cw_wikis where wiki_dbname = 'ucroniaswiki'; -db4 [14:15:15] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [14:16:42] !log MariaDB [mhglobal]> drop table mw_permissions2; -db4 [14:16:46] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [14:18:17] Hispano76 created [14:19:07] ok, thanks, thanks, thanks paladox :) [14:19:13] your welcome :) [14:21:49] paladox, no flag admin in ucroniaswiki :p [14:22:23] Not sure what you mean? [14:22:30] * paladox sees your an admin on https://ucronias.miraheze.org/wiki/Especial:ListaUsuarios [14:22:31] [ Lista de usuarios - ucroniaswiki ] - ucronias.miraheze.org [14:22:56] Ah, it was the cache. xD [14:28:45] hum, paladox [8f8824cf4e432c3dd26d8447] 2019-08-04 14:28:15: Excepción grave de tipo "MediaWiki\Storage\NameTableAccessException" [14:28:45] paladox [14:29:07] hmm [14:29:53] and https://prnt.sc/oo6bff paladox [14:29:53] [ Screenshot by Lightshot ] - prnt.sc [14:30:24] [8f8824cf4e432c3dd26d8447] /w/index.php?title=P%C3%A1gina_principal&action=edit MediaWiki\Storage\NameTableAccessException from line 42 of /srv/mediawiki/w/includes/Storage/NameTableAccessException.php: Failed to access name from content_models using id = 3 [14:31:32] !log recreating ucroniaswiki [14:31:36] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [14:33:56] PROBLEM - bacula1 Bacula Databases db5 on bacula1 is CRITICAL: CRITICAL: no terminated jobs [14:38:20] Hispano76 try now? [14:58:21] PROBLEM - misc3 Lizardfs Master Port 1 on misc3 is CRITICAL: connect to address 185.52.1.144 and port 9419: Connection refused [14:58:28] PROBLEM - cp2 Varnish Backends on cp2 is CRITICAL: 3 backends are down. mw1 mw2 mw3 [14:58:30] PROBLEM - misc3 Lizardfs Master Port 3 on misc3 is CRITICAL: connect to address 185.52.1.144 and port 9421: Connection refused [14:58:53] PROBLEM - cp4 Varnish Backends on cp4 is CRITICAL: 3 backends are down. mw1 mw2 mw3 [14:58:59] PROBLEM - cp3 Varnish Backends on cp3 is CRITICAL: 3 backends are down. mw1 mw2 mw3 [14:59:12] PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 6 datacenters are down: 107.191.126.23/cpweb, 2604:180:0:33b::2/cpweb, 128.199.139.216/cpweb, 2400:6180:0:d0::403:f001/cpweb, 81.4.109.133/cpweb, 2a00:d880:5:8ea::ebc7/cpweb [14:59:20] PROBLEM - cp2 HTTP 4xx/5xx ERROR Rate on cp2 is WARNING: WARNING - NGINX Error Rate is 48% [14:59:37] PROBLEM - misc3 Lizardfs Master Port 2 on misc3 is CRITICAL: connect to address 185.52.1.144 and port 9420: Connection refused [14:59:49] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 6 datacenters are down: 107.191.126.23/cpweb, 2604:180:0:33b::2/cpweb, 128.199.139.216/cpweb, 2400:6180:0:d0::403:f001/cpweb, 81.4.109.133/cpweb, 2a00:d880:5:8ea::ebc7/cpweb [15:01:20] PROBLEM - cp2 HTTP 4xx/5xx ERROR Rate on cp2 is CRITICAL: CRITICAL - NGINX Error Rate is 62% [15:02:20] RECOVERY - misc3 Lizardfs Master Port 1 on misc3 is OK: TCP OK - 0.001 second response time on 185.52.1.144 port 9419 [15:02:30] RECOVERY - misc3 Lizardfs Master Port 3 on misc3 is OK: TCP OK - 0.001 second response time on 185.52.1.144 port 9421 [15:03:00] RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online [15:03:19] RECOVERY - cp2 HTTP 4xx/5xx ERROR Rate on cp2 is OK: OK - NGINX Error Rate is 8% [15:03:33] RECOVERY - misc3 Lizardfs Master Port 2 on misc3 is OK: TCP OK - 0.001 second response time on 185.52.1.144 port 9420 [15:03:49] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [15:04:22] RECOVERY - cp2 Varnish Backends on cp2 is OK: All 5 backends are healthy [15:04:44] RECOVERY - cp4 Varnish Backends on cp4 is OK: All 5 backends are healthy [15:04:49] RECOVERY - cp3 Varnish Backends on cp3 is OK: All 5 backends are healthy [15:05:56] hmm [15:06:28] the master restarted (thus cp* depooled mw) [15:27:46] sorry paladox; hum, won't let me edit, could you manually give me the flags? [15:28:58] Oh, wait. paladox [15:29:48] paladox, https://ucronias.miraheze.org/wiki/Especial:ListaDerechosGrupos [15:29:49] [ Permisos de los grupos de usuarios - ucroniaswiki ] - ucronias.miraheze.org [15:30:18] I wanted to see if it was the cache, but still, the problem persists. [15:32:55] judging from Special:ManageWiki/permissions, the rights are assigned in the database, but I'm guessing the cdb hasn't been updated yet [15:42:34] Hello vesper11! If you have any questions feel free to ask and someone should answer soon. [15:45:27] Hispano76, paladox, fixed, I just used Special:ManageWiki/permissions, didn't change anything, and it worked [16:21:54] thanks Voidwalker. [18:05:11] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fjHKX [18:05:12] [02miraheze/services] 07MirahezeSSLBot 03196a855 - BOT: Updating services config for wikis [18:38:26] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 7.32, 6.36, 5.25 [18:40:22] RECOVERY - mw3 Current Load on mw3 is OK: OK - load average: 5.45, 5.97, 5.24 [18:55:26] [02puppet] 07paladox created branch 03paladox-patch-2 - 13https://git.io/vbiAS [18:55:27] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-2 [+0/-0/±1] 13https://git.io/fjH6W [18:55:29] [02miraheze/puppet] 07paladox 037c0e15f - stunnel: Set TCP_NODELAY=1 for both the server and client [18:55:30] [02puppet] 07paladox opened pull request 03#1060: stunnel: Set TCP_NODELAY=1 for both the server and client - 13https://git.io/fjH6l [19:23:07] !log depooling cp2 (testing some configuation changes && package update) [19:23:13] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [19:23:34] [02miraheze/dns] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fjH6H [19:23:35] [02miraheze/dns] 07paladox 03861831c - Depool cp2 [19:31:49] PROBLEM - cp2 Puppet on cp2 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 7 minutes ago with 0 failures [19:58:18] sorry to bother you again but you can attend task https://phabricator.miraheze.org/T4607 [19:58:19] [ ⚓ T4607 Import for ucronias.miraheze.org ] - phabricator.miraheze.org [20:08:40] Hispano76: I think Reception does imports now and they're not around for a few days [20:10:36] PROBLEM - cp4 Puppet on cp4 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 7 minutes ago with 0 failures [20:11:03] PROBLEM - cp3 Puppet on cp3 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 6 minutes ago with 0 failures [20:50:41] PROBLEM - cp2 HTTP 4xx/5xx ERROR Rate on cp2 is WARNING: WARNING - NGINX Error Rate is 51% [20:54:41] RECOVERY - cp2 HTTP 4xx/5xx ERROR Rate on cp2 is OK: OK - NGINX Error Rate is 29% [20:56:52] [02puppet] 07paladox created branch 03paladox-patch-4 - 13https://git.io/vbiAS [20:56:54] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-4 [+0/-0/±1] 13https://git.io/fjHPU [20:56:55] [02miraheze/puppet] 07paladox 03991f474 - Set wildcard cert permissions to 0600 [20:56:57] [02puppet] 07paladox opened pull request 03#1061: Set wildcard cert permissions to 0600 - 13https://git.io/fjHPT [21:03:43] [02puppet] 07paladox closed pull request 03#1061: Set wildcard cert permissions to 0600 - 13https://git.io/fjHPT [21:03:44] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fjHP3 [21:03:46] [02miraheze/puppet] 07paladox 032e28204 - Set wildcard cert permissions to 0600 (#1061) [21:03:53] paladox: uh [21:04:03] hmm? [21:04:14] 600 is way too restrictive for most uses [21:04:17] [02miraheze/puppet] 07paladox pushed 031 commit to 03revert-1061-paladox-patch-4 [+0/-0/±1] 13https://git.io/fjHPG [21:04:19] [02miraheze/puppet] 07paladox 03ea70b23 - Revert "Set wildcard cert permissions to 0600 (#1061)" This reverts commit 2e2820493af9a0dd0393296a3ae5b5549c6b3c63. [21:04:20] ok [21:04:20] [02puppet] 07paladox created branch 03revert-1061-paladox-patch-4 - 13https://git.io/vbiAS [21:04:26] JohnLewis what should it be? [21:04:29] 660 [21:04:31] [02puppet] 07paladox opened pull request 03#1062: Revert "Set wildcard cert permissions to 0600" - 13https://git.io/fjHPn [21:04:35] [02puppet] 07paladox closed pull request 03#1062: Revert "Set wildcard cert permissions to 0600" - 13https://git.io/fjHPn [21:04:36] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fjHPc [21:04:38] [02miraheze/puppet] 07paladox 03a5070f5 - Revert "Set wildcard cert permissions to 0600 (#1061)" (#1062) This reverts commit 2e2820493af9a0dd0393296a3ae5b5549c6b3c63. [21:04:39] [02puppet] 07paladox deleted branch 03revert-1061-paladox-patch-4 - 13https://git.io/vbiAS [21:04:41] [02miraheze/puppet] 07paladox deleted branch 03revert-1061-paladox-patch-4 [21:04:45] JohnLewis ok, thanks! [21:04:59] though you could have committed the fix without the need to revert [21:05:05] JohnLewis for both the cert and key? [21:05:12] for the key definitely [21:05:22] cert can be 777 for all anyone cares [21:06:08] ok [21:06:35] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fjHP8 [21:06:37] [02miraheze/puppet] 07paladox 03e7094fd - ssl: Set private key [21:06:39] bah [21:07:02] anyways that was ment to be "Set private key to 660" [21:07:39] [02puppet] 07paladox created branch 03paladox-patch-5 - 13https://git.io/vbiAS [21:07:41] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-5 [+0/-0/±1] 13https://git.io/fjHPE [21:07:42] [02miraheze/puppet] 07paladox 031e7009d - ssl: Set private ssl key to mode 660 [21:07:44] [02puppet] 07paladox opened pull request 03#1063: ssl: Set private ssl key to mode 660 - 13https://git.io/fjHPu [21:08:02] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-5 [+0/-0/±1] 13https://git.io/fjHPz [21:08:03] [02miraheze/puppet] 07paladox 037b61bdf - Update cert.pp [21:08:05] [02puppet] 07paladox synchronize pull request 03#1063: ssl: Set private ssl key to mode 660 - 13https://git.io/fjHPu [21:14:33] PROBLEM - cp3 Varnish Backends on cp3 is CRITICAL: 1 backends are down. mw1 [21:14:34] PROBLEM - cp2 Varnish Backends on cp2 is CRITICAL: 1 backends are down. mw1 [21:14:44] PROBLEM - cp4 Varnish Backends on cp4 is CRITICAL: 1 backends are down. mw1 [21:15:24] that wasen't me [21:18:17] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 2 datacenters are down: 107.191.126.23/cpweb, 2604:180:0:33b::2/cpweb [21:18:36] PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 2 datacenters are down: 107.191.126.23/cpweb, 2604:180:0:33b::2/cpweb [21:20:23] that's me (cp2, already depooled) [21:22:41] PROBLEM - cp2 HTTP 4xx/5xx ERROR Rate on cp2 is CRITICAL: CRITICAL - NGINX Error Rate is 62% [21:31:49] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [21:32:36] RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online [21:38:41] PROBLEM - cp2 HTTP 4xx/5xx ERROR Rate on cp2 is WARNING: WARNING - NGINX Error Rate is 58% [21:42:41] RECOVERY - cp2 HTTP 4xx/5xx ERROR Rate on cp2 is OK: OK - NGINX Error Rate is 28% [21:51:38] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 7.40, 6.42, 5.19 [21:53:38] RECOVERY - mw3 Current Load on mw3 is OK: OK - load average: 5.17, 6.04, 5.20 [22:01:22] [02miraheze/CreateWiki] 07paladox pushed 031 commit to 03paladox-patch-4 [+0/-0/±1] 13https://git.io/fjHXn [22:01:24] [02miraheze/CreateWiki] 07paladox 03744a746 - Remove "NULL" from cw_id Bug: T4608 [22:01:25] [02CreateWiki] 07paladox created branch 03paladox-patch-4 - 13https://git.io/vpJTL [22:01:27] [02CreateWiki] 07paladox opened pull request 03#118: Remove "NULL" from cw_id - 13https://git.io/fjHXc [22:07:01] [02CreateWiki] 07paladox closed pull request 03#118: Remove "NULL" from cw_id - 13https://git.io/fjHXc [22:07:03] [02CreateWiki] 07paladox reopened pull request 03#118: Remove "NULL" from cw_id - 13https://git.io/fjHXc [22:07:30] [02CreateWiki] 07paladox closed pull request 03#118: Remove "NULL" from cw_id - 13https://git.io/fjHXc [22:07:32] [02miraheze/CreateWiki] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fjHXR [22:07:33] [02miraheze/CreateWiki] 07paladox 03cfe2cf9 - Remove "NULL" from cw_id (#118) Bug: T4608 [22:07:35] [02CreateWiki] 07paladox deleted branch 03paladox-patch-4 - 13https://git.io/vpJTL [22:07:36] [02miraheze/CreateWiki] 07paladox deleted branch 03paladox-patch-4 [22:09:38] PROBLEM - mw3 Current Load on mw3 is CRITICAL: CRITICAL - load average: 8.40, 6.74, 5.61 [22:13:38] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 7.88, 7.26, 6.09 [22:17:38] RECOVERY - mw3 Current Load on mw3 is OK: OK - load average: 5.37, 6.79, 6.21 [22:20:49] [02CreateWiki] 07paladox created branch 03paladox-patch-4 - 13https://git.io/vpJTL [22:20:50] [02miraheze/CreateWiki] 07paladox pushed 031 commit to 03paladox-patch-4 [+0/-0/±1] 13https://git.io/fjHXa [22:20:51] [02miraheze/CreateWiki] 07paladox 03740a104 - wikimanager: Stop setting storage_engine to innodb it's the default under mysql 5.5+ and MariaDB 10.2+. Users can set innodb to be the default in the configuration. (which should be preferred). Bug: T4611 [22:20:53] [02CreateWiki] 07paladox opened pull request 03#119: wikimanager: Stop setting storage_engine to innodb - 13https://git.io/fjHXV [22:23:52] [02CreateWiki] 07paladox closed pull request 03#119: wikimanager: Stop setting storage_engine to innodb - 13https://git.io/fjHXV [22:23:54] [02miraheze/CreateWiki] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fjHXo [22:23:55] [02miraheze/CreateWiki] 07paladox 03b2f683d - wikimanager: Stop setting storage_engine to innodb (#119) it's the default under mysql 5.5+ and MariaDB 10.2+. Users can set innodb to be the default in the configuration. (which should be preferred). Bug: T4611 [22:25:05] [02puppet] 07paladox closed pull request 03#1063: ssl: Set private ssl key to mode 660 - 13https://git.io/fjHPu [22:25:06] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±2] 13https://git.io/fjHX6 [22:25:08] [02miraheze/puppet] 07paladox 03bfdc7cc - ssl: Set private ssl key to mode 660 (#1063) * ssl: Set private ssl key to mode 660 * Update cert.pp [22:25:09] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-5 [22:25:11] [02puppet] 07paladox deleted branch 03paladox-patch-5 - 13https://git.io/vbiAS [22:25:12] [02CreateWiki] 07paladox deleted branch 03paladox-patch-4 - 13https://git.io/vpJTL [22:25:14] [02miraheze/CreateWiki] 07paladox deleted branch 03paladox-patch-4 [22:30:57] PROBLEM - test1 HTTPS on test1 is CRITICAL: connect to address 185.52.2.243 and port 443: Connection refusedHTTP CRITICAL - Unable to open TCP socket [22:32:57] RECOVERY - test1 HTTPS on test1 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 444 bytes in 0.010 second response time [22:39:09] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 7.58, 7.09, 6.57 [22:39:49] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 2 datacenters are down: 107.191.126.23/cpweb, 2604:180:0:33b::2/cpweb [22:40:41] PROBLEM - cp2 HTTP 4xx/5xx ERROR Rate on cp2 is WARNING: WARNING - NGINX Error Rate is 58% [22:41:05] RECOVERY - mw3 Current Load on mw3 is OK: OK - load average: 6.27, 6.62, 6.45 [22:41:49] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [22:42:41] RECOVERY - cp2 HTTP 4xx/5xx ERROR Rate on cp2 is OK: OK - NGINX Error Rate is 7% [22:55:53] !log reboot mw1 (having mysql conn issues) [22:55:57] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [22:58:24] PROBLEM - mw1 MirahezeRenewSsl on mw1 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:58:26] PROBLEM - mw1 Disk Space on mw1 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [22:58:55] PROBLEM - mw1 Current Load on mw1 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [22:59:00] PROBLEM - mw1 SSH on mw1 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:59:23] PROBLEM - Host mw1 is DOWN: PING CRITICAL - Packet loss = 100% [23:01:24] RECOVERY - Host mw1 is UP: PING OK - Packet loss = 0%, RTA = 0.49 ms [23:01:29] PROBLEM - mw1 php-fpm on mw1 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [23:01:29] PROBLEM - mw1 HTTPS on mw1 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:01:29] PROBLEM - mw1 Puppet on mw1 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [23:02:06] RECOVERY - mw1 Puppet on mw1 is OK: OK: Puppet is currently enabled, last run 9 minutes ago with 0 failures [23:02:22] RECOVERY - mw1 MirahezeRenewSsl on mw1 is OK: TCP OK - 0.001 second response time on 185.52.1.75 port 5000 [23:02:25] RECOVERY - mw1 Disk Space on mw1 is OK: DISK OK - free space: / 26800 MB (34% inode=99%); [23:02:51] RECOVERY - mw1 Current Load on mw1 is OK: OK - load average: 0.25, 0.08, 0.02 [23:03:31] RECOVERY - mw1 HTTPS on mw1 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 442 bytes in 0.008 second response time [23:03:38] RECOVERY - mw1 php-fpm on mw1 is OK: PROCS OK: 4 processes with command name 'php-fpm7.2' [23:04:47] !log install mysql-client on mw1 [23:04:52] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [23:10:04] RECOVERY - cp2 Varnish Backends on cp2 is OK: All 5 backends are healthy [23:10:12] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fjH1k [23:10:14] [02miraheze/services] 07MirahezeSSLBot 03f7cdee3 - BOT: Updating services config for wikis [23:10:33] RECOVERY - cp3 Varnish Backends on cp3 is OK: All 5 backends are healthy [23:10:44] RECOVERY - cp4 Varnish Backends on cp4 is OK: All 5 backends are healthy [23:14:04] PROBLEM - cp2 Varnish Backends on cp2 is CRITICAL: 1 backends are down. mw1 [23:15:48] PROBLEM - mw1 Puppet on mw1 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 3 minutes ago with 0 failures [23:16:33] PROBLEM - cp3 Varnish Backends on cp3 is CRITICAL: 1 backends are down. mw1 [23:16:44] PROBLEM - cp4 Varnish Backends on cp4 is CRITICAL: 1 backends are down. mw1 [23:17:40] [02puppet] 07paladox created branch 03paladox-patch-5 - 13https://git.io/vbiAS [23:17:41] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-5 [+0/-0/±1] 13https://git.io/fjH1Y [23:17:43] [02miraheze/puppet] 07paladox 031ebb3cd - ssl: Fix ownership of the private key [23:17:44] [02puppet] 07paladox opened pull request 03#1064: ssl: Fix ownership of the private key - 13https://git.io/fjH1O [23:17:57] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-5 [+0/-0/±1] 13https://git.io/fjH13 [23:17:58] [02miraheze/puppet] 07paladox 03599801e - Update cert.pp [23:18:00] [02puppet] 07paladox synchronize pull request 03#1064: ssl: Fix ownership of the private key - 13https://git.io/fjH1O [23:18:10] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-5 [+0/-0/±1] 13https://git.io/fjH1s [23:18:12] [02miraheze/puppet] 07paladox 03d442aa5 - Update certs.pp [23:18:13] [02puppet] 07paladox synchronize pull request 03#1064: ssl: Fix ownership of the private key - 13https://git.io/fjH1O [23:18:18] [02puppet] 07paladox closed pull request 03#1064: ssl: Fix ownership of the private key - 13https://git.io/fjH1O [23:18:19] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±3] 13https://git.io/fjH1G [23:18:21] [02miraheze/puppet] 07paladox 034ba6ba5 - ssl: Fix ownership of the private key (#1064) * ssl: Fix ownership of the private key * Update cert.pp * Update certs.pp [23:18:22] [02puppet] 07paladox deleted branch 03paladox-patch-5 - 13https://git.io/vbiAS [23:18:24] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-5 [23:19:48] RECOVERY - mw1 Puppet on mw1 is OK: OK: Puppet is currently enabled, last run 20 seconds ago with 0 failures [23:20:04] RECOVERY - cp2 Varnish Backends on cp2 is OK: All 5 backends are healthy [23:20:33] RECOVERY - cp3 Varnish Backends on cp3 is OK: All 5 backends are healthy [23:20:44] RECOVERY - cp4 Varnish Backends on cp4 is OK: All 5 backends are healthy [23:24:36] RECOVERY - cp4 Puppet on cp4 is OK: OK: Puppet is currently enabled, last run 40 seconds ago with 0 failures [23:25:17] RECOVERY - cp3 Puppet on cp3 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [23:28:21] PROBLEM - db4 Disk Space on db4 is CRITICAL: DISK CRITICAL - free space: / 21974 MB (5% inode=96%); [23:35:11] PROBLEM - test1 Puppet on test1 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 2 minutes ago with 0 failures [23:54:04] !log hacked stunnel conf on puppet1 (and rolling it out to cp*)