[02:13:45] PROBLEM - dbbackup2 Check MariaDB Replication c3 on dbbackup2 is CRITICAL: MariaDB replication - both - CRITICAL - Slave_IO_Running state : Yes, Slave_SQL_Running state : Yes, Seconds_Behind_Master : 235s [02:19:54] RECOVERY - dbbackup2 Check MariaDB Replication c3 on dbbackup2 is OK: MariaDB replication - both - OK - Slave_IO_Running state : Yes, Slave_SQL_Running state : Yes, Seconds_Behind_Master : 0s [02:30:16] PROBLEM - wiki.mlpwiki.net - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.mlpwiki.net reverse DNS resolves to 192-185-16-85.unifiedlayer.com [03:18:53] PROBLEM - dbbackup2 Check MariaDB Replication c3 on dbbackup2 is CRITICAL: MariaDB replication - both - CRITICAL - Slave_IO_Running state : Yes, Slave_SQL_Running state : Yes, Seconds_Behind_Master : 212s [03:20:53] RECOVERY - dbbackup2 Check MariaDB Replication c3 on dbbackup2 is OK: MariaDB replication - both - OK - Slave_IO_Running state : Yes, Slave_SQL_Running state : Yes, Seconds_Behind_Master : 0s [03:26:58] PROBLEM - dbbackup2 Check MariaDB Replication c3 on dbbackup2 is WARNING: MariaDB replication - both - WARNING - Slave_IO_Running state : Yes, Slave_SQL_Running state : Yes, Seconds_Behind_Master : 142s [03:29:04] PROBLEM - dbbackup2 Check MariaDB Replication c3 on dbbackup2 is CRITICAL: MariaDB replication - both - CRITICAL - Slave_IO_Running state : Yes, Slave_SQL_Running state : Yes, Seconds_Behind_Master : 237s [03:30:45] [02miraheze/ManageWiki] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JYqOH [03:30:46] [02miraheze/ManageWiki] 07Universal-Omega 03c280e2f - Fix disabled attribute for MWN content model field [03:31:46] miraheze/ManageWiki - Universal-Omega the build passed. [03:33:11] PROBLEM - dbbackup2 Check MariaDB Replication c3 on dbbackup2 is WARNING: MariaDB replication - both - WARNING - Slave_IO_Running state : Yes, Slave_SQL_Running state : Yes, Seconds_Behind_Master : 145s [03:33:48] [02miraheze/ManageWiki] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JYq30 [03:33:50] [02miraheze/ManageWiki] 07Universal-Omega 034923969 - Dont need to pass disabled [03:34:44] miraheze/ManageWiki - Universal-Omega the build passed. [03:35:15] RECOVERY - dbbackup2 Check MariaDB Replication c3 on dbbackup2 is OK: MariaDB replication - both - OK - Slave_IO_Running state : Yes, Slave_SQL_Running state : Yes, Seconds_Behind_Master : 0s [03:46:22] PROBLEM - bacula2 Bacula Phabricator Static on bacula2 is WARNING: WARNING: Full, 87218 files, 3.984GB, 2021-03-10 23:10:00 (2.2 weeks ago) [03:46:55] PROBLEM - bacula2 Bacula Databases db13 on bacula2 is WARNING: WARNING: Full, 238729 files, 84.61GB, 2021-03-10 22:42:00 (2.2 weeks ago) [03:46:55] PROBLEM - bacula2 Bacula Private Git on bacula2 is WARNING: WARNING: Full, 5531 files, 22.29MB, 2021-03-14 00:05:00 (1.7 weeks ago) [03:46:55] PROBLEM - bacula2 Bacula Static on bacula2 is CRITICAL: CRITICAL: no terminated jobs [03:46:55] PROBLEM - bacula2 Disk Space on bacula2 is CRITICAL: DISK CRITICAL - free space: / 11 MB (0% inode=99%); [03:46:55] PROBLEM - bacula2 Bacula Phabricator Static on bacula2 is WARNING: WARNING: Full, 87218 files, 3.984GB, 2021-03-10 23:10:00 (2.2 weeks ago) [06:11:23] !log GDPR script [06:11:29] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [06:17:11] [02miraheze/ssl] 07Reception123 pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JYq1U [06:17:12] [02miraheze/ssl] 07Reception123 03d3d909c - renew miraheze.wiki [11:34:05] RECOVERY - wiki.mlpwiki.net - reverse DNS on sslhost is OK: rDNS OK - wiki.mlpwiki.net reverse DNS resolves to cp11.miraheze.org [12:08:23] Reception123: there are two issues here [12:08:39] first, the nginx config points to /etc/ssl/certs/miraheze.wiki.crt, but we have switched to /etc/ssl/localcerts [12:09:15] second, the certificate is valid for 'miraheze.wiki', but not phab.miraheze.wiki [12:10:35] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JYmXY [12:10:36] [02miraheze/puppet] 07paladox 0356fbd38 - Fix path [12:12:33] (the certificate = /etc/ssl/localcerts/miraheze.wiki.crt) [12:13:06] SPF|Cloud: heh I just figured that out as you said it :P [12:13:08] Anyways fixed [12:13:10] It’s a wildcard cert SPF|Cloud [12:13:23] it's not [12:13:36] perhaps the old one was, but the new one isn't [12:14:00] I can't find SANs in the new cert [12:14:55] SPF|Cloud https://phab.miraheze.wiki/ works now [12:15:03] huh [12:15:18] it doesn't work [12:15:57] 13:09:17 <+SPF|Cloud> second, the certificate is valid for 'miraheze.wiki', but not phab.miraheze.wiki [12:16:32] it is valid for miraheze.wiki ONLY ;) [12:20:28] ok, Reception123 renewed the cert this morning it appears but converted it from wildcard to just a singular domain. [12:21:05] I followed the steps on the docs page to renew wildcard certs [12:22:16] [02miraheze/ssl] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JYmMU [12:22:17] [02miraheze/ssl] 07paladox 03a7617e6 - Update miraheze.wiki.crt [12:23:01] paladox: what command did you use then? [12:23:13] X509v3 Subject Alternative Name: [12:23:14] DNS:*.miraheze.wiki, DNS:miraheze.wiki [12:23:19] that'll work [12:24:33] closed T7041 [12:25:21] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JYmMy [12:25:22] [02miraheze/puppet] 07paladox 03554c0a9 - Fix [12:26:08] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JYmMA [12:26:09] [02miraheze/puppet] 07paladox 03f51bd8b - phabricator: Bring down ssl stanzas so it restarts nginx after update [12:34:13] PROBLEM - dbbackup2 Check MariaDB Replication c3 on dbbackup2 is CRITICAL: MariaDB replication - both - CRITICAL - Slave_IO_Running state : Yes, Slave_SQL_Running state : Yes, Seconds_Behind_Master : 231s [13:03:41] Hello SREs - Is this template used somewhere? https://meta.miraheze.org/wiki/Template:Lock_notice [13:03:42] [ Template:Lock notice - Miraheze Meta ] - meta.miraheze.org [13:11:05] Not anymore, that used to be the old template when it was done differently [13:12:49] RECOVERY - dbbackup2 Check MariaDB Replication c3 on dbbackup2 is OK: MariaDB replication - both - OK - Slave_IO_Running state : Yes, Slave_SQL_Running state : Yes, Seconds_Behind_Master : 0s [13:13:53] i saw something like this in MirahezeMagic (?) [13:13:56] thank you [13:15:20] @MrJaroslavik yeah, the ones currently used are in MirahezeMagic, this one was used when the system was different and on-wiki templates were used [13:31:01] PROBLEM - dbbackup2 Check MariaDB Replication c3 on dbbackup2 is WARNING: MariaDB replication - both - WARNING - Slave_IO_Running state : Yes, Slave_SQL_Running state : Yes, Seconds_Behind_Master : 118s [13:33:03] RECOVERY - dbbackup2 Check MariaDB Replication c3 on dbbackup2 is OK: MariaDB replication - both - OK - Slave_IO_Running state : Yes, Slave_SQL_Running state : Yes, Seconds_Behind_Master : 0s [13:55:01] PROBLEM - puppet3 Puppet on puppet3 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Package[postgresql-contrib-11] [13:57:01] RECOVERY - puppet3 Puppet on puppet3 is OK: OK: Puppet is currently enabled, last run 38 seconds ago with 0 failures [14:21:53] PROBLEM - dbbackup2 Check MariaDB Replication c3 on dbbackup2 is WARNING: MariaDB replication - both - WARNING - Slave_IO_Running state : Yes, Slave_SQL_Running state : Yes, Seconds_Behind_Master : 149s [14:23:55] RECOVERY - dbbackup2 Check MariaDB Replication c3 on dbbackup2 is OK: MariaDB replication - both - OK - Slave_IO_Running state : Yes, Slave_SQL_Running state : Yes, Seconds_Behind_Master : 0s [14:38:11] PROBLEM - wiki.mlpwiki.net - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.mlpwiki.net reverse DNS resolves to 192-185-16-85.unifiedlayer.com [14:41:12] !log rebuilding cp3 [14:41:18] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [14:42:10] RECOVERY - Host cp3 is UP: PING OK - Packet loss = 0%, RTA = 255.84 ms [14:42:12] PROBLEM - cp3 Varnish Backends on cp3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [14:42:12] PROBLEM - cp3 SSH on cp3 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:43:12] PROBLEM - cp3 Disk Space on cp3 is CRITICAL: connect to address 128.199.139.216 port 5666: Connection refusedconnect to host 128.199.139.216 port 5666: Connection refused [14:43:25] RECOVERY - cp3 SSH on cp3 is OK: SSH OK - OpenSSH_7.9p1 Debian-10+deb10u2 (protocol 2.0) [14:43:30] RECOVERY - ping6 on cp3 is OK: PING OK - Packet loss = 0%, RTA = 257.20 ms [14:47:16] > PROBLEM - wiki.mlpwiki.net - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.mlpwiki.net reverse DNS resolves to 192-185-16-85.unifiedlayer.com [14:47:16] Reception123, looking at that, that would seem to suggest that wiki, or at least its domain, has moved away from us, eh? [14:47:46] dmehus: I've already contacted them, they seem to have set their namespaces to 4 servers (2 are Miraheze's, two are not) [14:48:08] Reception123, ah, they're using backup nameservers. Yeah, we wouldn't like that [14:48:10] thanks :) [14:49:07] yeah [14:49:10] that explains why I got a wiki not found error, too, since I obviously hit a Miraheze nameserver [14:49:20] what's the wiki db name? [14:49:27] if you look on icinga, I left a comment on the warning [14:49:40] Reception123, oh nice, heh let me look [14:49:42] dmehus: https://phabricator.miraheze.org/T6954 [14:49:42] [ ⚓ T6954 Moving wiki to subdomain on other website ] - phabricator.miraheze.org [14:49:50] a very recent one too [14:50:03] ah, right yeah [14:51:05] oh, cool, if you comment on one icinga warning, if that warning is repeated, it's the same record entry so the comment is attached to it [14:58:11] now that's a coincidence [14:58:11] https://phabricator.miraheze.org/T7043#139403 [14:58:12] [ ⚓ T7043 took down my site? ] - phabricator.miraheze.org [15:01:14] Reception123 they use cp3, it went down when cp3 went down. [15:01:24] host dreamsit.com.br [15:01:24] dreamsit.com.br has address 128.199.139.216 [15:02:25] paladox: oh, I thought that was fixed because the task was resolved [15:03:03] thanks for seeing that, I'll tell them [15:04:55] PROBLEM - cp3 HTTP 4xx/5xx ERROR Rate on cp3 is UNKNOWN: NRPE: Unable to read output [15:05:01] PROBLEM - cp3 Puppet on cp3 is UNKNOWN: NRPE: Unable to read output [15:05:03] RECOVERY - cp3 PowerDNS Recursor on cp3 is OK: DNS OK: 0.688 seconds response time. miraheze.org returns 2001:41d0:800:178a::5,2001:41d0:800:1bbd::4,51.195.236.219,51.195.236.250 [15:05:05] RECOVERY - cp3 NTP time on cp3 is OK: NTP OK: Offset 0.02784839272 secs [15:05:14] PROBLEM - cp12 Current Load on cp12 is WARNING: WARNING - load average: 1.79, 1.60, 1.00 [15:05:16] RECOVERY - cp3 Disk Space on cp3 is OK: DISK OK - free space: / 22189 MB (92% inode=95%); [15:05:49] PROBLEM - cp3 Varnish Backends on cp3 is UNKNOWN: NRPE: Unable to read output [15:06:30] RECOVERY - cp3 Current Load on cp3 is OK: OK - load average: 1.21, 0.88, 0.37 [15:06:46] RECOVERY - cp3 APT on cp3 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [15:07:12] RECOVERY - cp12 Current Load on cp12 is OK: OK - load average: 0.71, 1.30, 0.96 [15:07:54] RECOVERY - dreamsit.com.br - LetsEncrypt on sslhost is OK: OK - Certificate 'dreamsit.com.br' will expire on Fri 11 Jun 2021 15:23:19 GMT +0000. [15:11:48] PROBLEM - cp3 Varnish Backends on cp3 is WARNING: No backends detected. If this is an error, see readme.txt [15:12:55] RECOVERY - cp3 HTTP 4xx/5xx ERROR Rate on cp3 is OK: OK - NGINX Error Rate is 0% [15:20:30] PROBLEM - cp3 Puppet on cp3 is CRITICAL: CRITICAL: Puppet has 3 failures. Last run 36 seconds ago with 3 failures. Failed resources (up to 3 shown): Package[prometheus-varnish-exporter],Package[varnish],Package[varnish-modules] [15:21:12] RECOVERY - dreamsit.com.br - reverse DNS on sslhost is OK: rDNS OK - dreamsit.com.br reverse DNS resolves to cp11.miraheze.org [15:27:05] PROBLEM - cp3 HTTP 4xx/5xx ERROR Rate on cp3 is CRITICAL: CRITICAL - NGINX Error Rate is 64% [15:31:01] RECOVERY - cp3 Stunnel Http for mw9 on cp3 is OK: HTTP OK: HTTP/1.1 200 OK - 15200 bytes in 1.035 second response time [15:31:22] RECOVERY - cp3 Stunnel Http for mw11 on cp3 is OK: HTTP OK: HTTP/1.1 200 OK - 15201 bytes in 1.026 second response time [15:31:45] RECOVERY - cp3 Stunnel Http for test3 on cp3 is OK: HTTP OK: HTTP/1.1 200 OK - 15202 bytes in 1.008 second response time [15:32:19] RECOVERY - cp3 Stunnel Http for mw8 on cp3 is OK: HTTP OK: HTTP/1.1 200 OK - 15200 bytes in 1.062 second response time [15:32:29] RECOVERY - cp3 Stunnel Http for mon2 on cp3 is OK: HTTP OK: HTTP/1.1 200 OK - 35435 bytes in 1.036 second response time [15:32:32] RECOVERY - cp3 Stunnel Http for mw10 on cp3 is OK: HTTP OK: HTTP/1.1 200 OK - 15201 bytes in 1.016 second response time [15:35:16] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [15:35:27] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [15:35:40] RECOVERY - cp3 Varnish Backends on cp3 is OK: All 7 backends are healthy [15:35:54] [02miraheze/dns] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JYYcw [15:35:56] [02miraheze/dns] 07paladox 0358e84de - Revert "Depool sg" [15:36:09] RECOVERY - cp3 HTTPS on cp3 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 2210 bytes in 1.993 second response time [15:36:19] RECOVERY - cp3 Puppet on cp3 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [15:37:06] RECOVERY - cp3 HTTP 4xx/5xx ERROR Rate on cp3 is OK: OK - NGINX Error Rate is 4% [16:06:04] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JYYBD [16:06:05] [02miraheze/puppet] 07paladox 0305d683e - Update mediawiki.pp [16:08:27] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 6.97, 6.37, 5.49 [16:10:27] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 6.60, 6.40, 5.60 [18:20:16] PROBLEM - dbbackup2 Check MariaDB Replication c3 on dbbackup2 is CRITICAL: MariaDB replication - both - CRITICAL - Slave_IO_Running state : Yes, Slave_SQL_Running state : Yes, Seconds_Behind_Master : 257s [18:36:17] RECOVERY - dbbackup2 Check MariaDB Replication c3 on dbbackup2 is OK: MariaDB replication - both - OK - Slave_IO_Running state : Yes, Slave_SQL_Running state : Yes, Seconds_Behind_Master : 0s [18:55:54] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 7.35, 6.68, 5.87 [18:59:50] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 6.33, 6.69, 6.11 [21:35:01] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 9.10, 7.09, 5.75 [21:37:01] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 5.39, 6.32, 5.63 [23:42:00] RECOVERY - wiki.mlpwiki.net - reverse DNS on sslhost is OK: rDNS OK - wiki.mlpwiki.net reverse DNS resolves to cp11.miraheze.org [23:45:12] PROBLEM - cp12 Current Load on cp12 is WARNING: WARNING - load average: 1.70, 1.72, 1.24 [23:47:12] RECOVERY - cp12 Current Load on cp12 is OK: OK - load average: 0.44, 1.23, 1.12 [23:51:00] PROBLEM - wiki.mlpwiki.net - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.mlpwiki.net reverse DNS resolves to 192-185-16-85.unifiedlayer.com