[00:09:01] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JeRhZ [00:09:03] [02miraheze/puppet] 07paladox 033f4cb73 - lizardfs: Do not try and create /var/lib/lizardfs/metadata.mfs This is dangerous and only needs to be done if the master is new [00:10:51] [02puppet] 07paladox created branch 03paladox-patch-1 - 13https://git.io/vbiAS [00:10:53] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/JeRhn [00:10:54] [02miraheze/puppet] 07paladox 03a83795e - lizardfs: Add globaliolimits and configure it Limit bandwith to 1MiB/s [00:10:56] [02puppet] 07paladox opened pull request 03#1114: lizardfs: Add globaliolimits and configure it - 13https://git.io/JeRhc [00:11:44] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-1 [+1/-0/±0] 13https://git.io/JeRhC [00:11:46] [02miraheze/puppet] 07paladox 03e3f5290 - Create globaliolimits.cfg.erb [00:11:47] [02puppet] 07paladox synchronize pull request 03#1114: lizardfs: Add globaliolimits and configure it - 13https://git.io/JeRhc [00:12:22] PROBLEM - cp2 Stunnel Http for misc2 on cp2 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 372 bytes in 0.295 second response time [00:12:32] PROBLEM - db5 Puppet on db5 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:12:38] [02puppet] 07paladox synchronize pull request 03#1114: lizardfs: Add globaliolimits and configure it - 13https://git.io/JeRhc [00:12:40] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/JeRhl [00:12:41] [02miraheze/puppet] 07paladox 03eb1e18f - Update globaliolimits.cfg.erb [00:12:45] [02puppet] 07paladox closed pull request 03#1114: lizardfs: Add globaliolimits and configure it - 13https://git.io/JeRhc [00:12:47] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+1/-0/±1] 13https://git.io/JeRh8 [00:12:48] [02miraheze/puppet] 07paladox 03afe6216 - lizardfs: Add globaliolimits and configure it (#1114) * lizardfs: Add globaliolimits and configure it Limit bandwith to 1MiB/s * Create globaliolimits.cfg.erb * Update globaliolimits.cfg.erb [00:12:50] [02puppet] 07paladox deleted branch 03paladox-patch-1 - 13https://git.io/vbiAS [00:12:51] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-1 [00:13:08] PROBLEM - misc2 HTTPS on misc2 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 372 bytes in 0.009 second response time [00:13:23] uhh [00:13:40] PROBLEM - cp2 Varnish Backends on cp2 is CRITICAL: 2 backends are down. mw1 mw3 [00:14:08] PROBLEM - cp4 Varnish Backends on cp4 is CRITICAL: 1 backends are down. mw1 [00:14:09] PROBLEM - cp3 Varnish Backends on cp3 is CRITICAL: 1 backends are down. mw1 [00:14:19] RECOVERY - cp2 Stunnel Http for misc2 on cp2 is OK: HTTP OK: HTTP/1.1 200 OK - 41802 bytes in 0.570 second response time [00:14:29] RECOVERY - db5 Puppet on db5 is OK: OK: Puppet is currently enabled, last run 12 minutes ago with 0 failures [00:14:49] !log restart lizardfs-master on misc3 - config change [00:15:18] RECOVERY - misc2 HTTPS on misc2 is OK: HTTP OK: HTTP/1.1 200 OK - 41810 bytes in 0.112 second response time [00:16:05] RECOVERY - cp4 Varnish Backends on cp4 is OK: All 5 backends are healthy [00:17:55] RECOVERY - cp2 Varnish Backends on cp2 is OK: All 5 backends are healthy [00:18:03] RECOVERY - cp3 Varnish Backends on cp3 is OK: All 5 backends are healthy [00:21:43] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 7.81, 6.69, 6.23 [00:23:42] PROBLEM - mw3 Current Load on mw3 is CRITICAL: CRITICAL - load average: 8.45, 7.20, 6.46 [00:25:17] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JeRhz [00:25:18] [02miraheze/puppet] 07paladox 03a257e9c - lizardfs: Set GLOBALIOLIMITS_FILENAME [00:25:43] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 7.78, 7.15, 6.52 [00:26:22] !log [01:14:49] <+paladox> !log restart lizardfs-master on misc3 - config change [00:26:28] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [00:26:30] !log restart lizardfs-master [00:26:35] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [00:27:43] RECOVERY - mw3 Current Load on mw3 is OK: OK - load average: 5.46, 6.49, 6.35 [00:29:05] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 2 datacenters are down: 2400:6180:0:d0::403:f001/cpweb, 2a00:d880:5:8ea::ebc7/cpweb [00:31:06] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [00:31:43] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 6.82, 7.05, 6.64 [00:45:43] RECOVERY - mw3 Current Load on mw3 is OK: OK - load average: 5.91, 6.50, 6.73 [00:46:38] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JeRhN [00:46:39] [02miraheze/puppet] 07paladox 03e02693c - Update globaliolimits.cfg.erb [00:47:40] !log restart lizardfs-master [00:52:01] !log [01:47:40] <+paladox> !log restart lizardfs-master [00:52:06] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [01:02:28] PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 5 datacenters are down: 107.191.126.23/cpweb, 128.199.139.216/cpweb, 2400:6180:0:d0::403:f001/cpweb, 81.4.109.133/cpweb, 2a00:d880:5:8ea::ebc7/cpweb [01:03:15] PROBLEM - mw3 Current Load on mw3 is CRITICAL: CRITICAL - load average: 10.21, 7.20, 6.50 [01:04:28] RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online [01:07:14] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 7.15, 7.28, 6.69 [01:11:24] RECOVERY - mw3 Current Load on mw3 is OK: OK - load average: 5.46, 6.17, 6.35 [01:24:42] Paladox: Are you ever going to renew the *.miraheze.org ssl cert? [01:27:48] PROBLEM - mw3 Current Load on mw3 is CRITICAL: CRITICAL - load average: 9.06, 7.90, 6.97 [01:29:47] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 5.82, 6.96, 6.73 [01:31:46] RECOVERY - mw3 Current Load on mw3 is OK: OK - load average: 6.21, 6.62, 6.62 [01:34:54] Yes, we are working on that. [01:46:17] Paladox: Good [01:55:11] Paladox: What are the requirements for helping as a MediaWiki sysadmin or mw-admin? [02:00:30] Examknow: I think https://meta.miraheze.org/wiki/Contributing#Tweaking_the_servers may be what you want to look at [02:00:31] [ Contributing - Miraheze Meta ] - meta.miraheze.org [02:03:10] * paladox bed [02:17:04] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 7.56, 7.06, 6.58 [02:19:05] RECOVERY - mw3 Current Load on mw3 is OK: OK - load average: 4.88, 6.28, 6.36 [02:25:09] PROBLEM - mw3 Current Load on mw3 is CRITICAL: CRITICAL - load average: 8.54, 7.11, 6.63 [02:29:11] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 5.91, 7.04, 6.75 [02:35:21] RECOVERY - mw3 Current Load on mw3 is OK: OK - load average: 4.77, 6.10, 6.47 [03:06:09] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 7.12, 6.99, 6.38 [03:14:12] RECOVERY - mw3 Current Load on mw3 is OK: OK - load average: 5.29, 6.34, 6.35 [03:41:04] PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 6 datacenters are down: 107.191.126.23/cpweb, 2604:180:0:33b::2/cpweb, 128.199.139.216/cpweb, 2400:6180:0:d0::403:f001/cpweb, 81.4.109.133/cpweb, 2a00:d880:5:8ea::ebc7/cpweb [03:43:05] RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online [03:46:52] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 7.55, 6.89, 6.36 [03:48:16] PROBLEM - cp3 Varnish Backends on cp3 is CRITICAL: 3 backends are down. mw1 mw2 mw3 [03:48:17] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 6 datacenters are down: 107.191.126.23/cpweb, 2604:180:0:33b::2/cpweb, 128.199.139.216/cpweb, 2400:6180:0:d0::403:f001/cpweb, 81.4.109.133/cpweb, 2a00:d880:5:8ea::ebc7/cpweb [03:48:19] PROBLEM - cp2 Varnish Backends on cp2 is CRITICAL: 3 backends are down. mw1 mw2 mw3 [03:48:51] RECOVERY - mw3 Current Load on mw3 is OK: OK - load average: 5.93, 6.62, 6.32 [03:50:11] RECOVERY - cp3 Varnish Backends on cp3 is OK: All 5 backends are healthy [03:50:14] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [03:50:16] RECOVERY - cp2 Varnish Backends on cp2 is OK: All 5 backends are healthy [03:53:41] PROBLEM - cp4 Varnish Backends on cp4 is CRITICAL: 1 backends are down. mw2 [03:53:55] PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 4 datacenters are down: 107.191.126.23/cpweb, 2604:180:0:33b::2/cpweb, 81.4.109.133/cpweb, 2a00:d880:5:8ea::ebc7/cpweb [03:54:34] PROBLEM - cp3 Varnish Backends on cp3 is CRITICAL: 2 backends are down. mw1 mw2 [03:54:38] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 5 datacenters are down: 107.191.126.23/cpweb, 128.199.139.216/cpweb, 2400:6180:0:d0::403:f001/cpweb, 81.4.109.133/cpweb, 2a00:d880:5:8ea::ebc7/cpweb [03:56:08] RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online [03:56:29] RECOVERY - cp3 Varnish Backends on cp3 is OK: All 5 backends are healthy [03:56:35] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [03:57:54] RECOVERY - cp4 Varnish Backends on cp4 is OK: All 5 backends are healthy [04:05:05] PROBLEM - mw1 Current Load on mw1 is CRITICAL: CRITICAL - load average: 8.49, 6.71, 5.77 [04:07:00] PROBLEM - mw1 Current Load on mw1 is WARNING: WARNING - load average: 7.74, 7.24, 6.08 [04:08:55] PROBLEM - mw1 Current Load on mw1 is CRITICAL: CRITICAL - load average: 8.24, 7.64, 6.36 [04:12:47] PROBLEM - mw1 Current Load on mw1 is WARNING: WARNING - load average: 6.66, 7.71, 6.71 [04:22:28] PROBLEM - mw1 Current Load on mw1 is CRITICAL: CRITICAL - load average: 8.47, 7.78, 7.02 [04:24:26] PROBLEM - mw1 Current Load on mw1 is WARNING: WARNING - load average: 5.37, 7.19, 6.91 [04:25:30] PROBLEM - mw3 Current Load on mw3 is CRITICAL: CRITICAL - load average: 8.23, 7.34, 6.23 [04:28:24] RECOVERY - mw1 Current Load on mw1 is OK: OK - load average: 4.37, 6.02, 6.52 [04:33:38] PROBLEM - lizardfs4 Puppet on lizardfs4 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [04:34:43] PROBLEM - lizardfs4 Current Load on lizardfs4 is CRITICAL: CRITICAL - load average: 4.12, 3.79, 2.19 [04:35:39] RECOVERY - lizardfs4 Puppet on lizardfs4 is OK: OK: Puppet is currently enabled, last run 25 seconds ago with 0 failures [04:36:45] RECOVERY - lizardfs4 Current Load on lizardfs4 is OK: OK - load average: 2.92, 3.35, 2.22 [04:47:22] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 6.16, 7.70, 7.96 [04:49:22] PROBLEM - mw3 Current Load on mw3 is CRITICAL: CRITICAL - load average: 7.64, 7.99, 8.05 [04:51:21] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 7.28, 7.79, 7.96 [04:57:22] PROBLEM - mw3 Current Load on mw3 is CRITICAL: CRITICAL - load average: 9.55, 8.16, 7.99 [05:59:38] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 6.12, 6.59, 7.74 [06:07:42] RECOVERY - mw3 Current Load on mw3 is OK: OK - load average: 4.54, 5.38, 6.72 [06:20:50] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 7.02, 7.08, 6.80 [06:26:43] RECOVERY - cp3 Disk Space on cp3 is OK: DISK OK - free space: / 2931 MB (12% inode=94%); [06:28:51] RECOVERY - mw3 Current Load on mw3 is OK: OK - load average: 5.77, 6.58, 6.74 [08:23:22] [02MatomoAnalytics] 07dgrammlich opened pull request 03#10: Add composer.json - 13https://git.io/Je0TY [09:13:24] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 7.38, 6.82, 6.01 [09:15:27] RECOVERY - mw3 Current Load on mw3 is OK: OK - load average: 6.24, 6.66, 6.05 [09:30:36] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 7.35, 7.19, 6.62 [09:32:34] RECOVERY - mw3 Current Load on mw3 is OK: OK - load average: 5.80, 6.64, 6.48 [09:36:31] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 6.50, 7.20, 6.78 [09:38:32] RECOVERY - mw3 Current Load on mw3 is OK: OK - load average: 4.96, 6.37, 6.52 [11:55:29] PROBLEM - mw3 Current Load on mw3 is CRITICAL: CRITICAL - load average: 8.32, 7.27, 6.43 [11:57:28] RECOVERY - mw3 Current Load on mw3 is OK: OK - load average: 5.72, 6.61, 6.29 [12:07:26] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 6.82, 6.68, 6.35 [12:09:25] RECOVERY - mw3 Current Load on mw3 is OK: OK - load average: 6.78, 6.62, 6.36 [12:20:23] PROBLEM - mw3 Current Load on mw3 is CRITICAL: CRITICAL - load average: 12.08, 8.17, 6.92 [12:24:20] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 5.34, 7.46, 7.00 [12:24:43] PROBLEM - cp3 Disk Space on cp3 is WARNING: DISK WARNING - free space: / 2648 MB (10% inode=94%); [12:26:18] RECOVERY - mw3 Current Load on mw3 is OK: OK - load average: 4.66, 6.47, 6.69 [12:50:16] PROBLEM - mw3 Current Load on mw3 is CRITICAL: CRITICAL - load average: 8.98, 7.71, 6.92 [12:52:18] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 7.32, 7.43, 6.91 [12:54:18] RECOVERY - mw3 Current Load on mw3 is OK: OK - load average: 4.80, 6.50, 6.63 [13:00:15] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 6.68, 6.87, 6.74 [13:02:14] RECOVERY - mw3 Current Load on mw3 is OK: OK - load average: 6.32, 6.62, 6.65 [13:08:18] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 7.72, 7.10, 6.82 [13:10:17] RECOVERY - mw3 Current Load on mw3 is OK: OK - load average: 5.58, 6.63, 6.69 [13:14:17] PROBLEM - mw3 Current Load on mw3 is CRITICAL: CRITICAL - load average: 8.95, 7.50, 7.00 [13:16:16] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 5.82, 6.95, 6.87 [13:18:16] RECOVERY - mw3 Current Load on mw3 is OK: OK - load average: 5.14, 6.30, 6.64 [13:35:17] !log shutdown -r on lizardfs6 [13:35:21] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [13:39:38] PROBLEM - lizardfs6 SSH on lizardfs6 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:39:42] PROBLEM - lizardfs6 Current Load on lizardfs6 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [13:39:48] PROBLEM - lizardfs6 Puppet on lizardfs6 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [13:40:34] PROBLEM - lizardfs6 Lizardfs Master Port 3 on lizardfs6 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:40:49] PROBLEM - lizardfs6 Lizardfs Master Port 2 on lizardfs6 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:40:52] PROBLEM - lizardfs6 lizard.miraheze.org HTTPS on lizardfs6 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:41:16] Scheduled downtime [13:53:13] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Je0Y5 [13:53:15] [02miraheze/puppet] 07paladox 03a35240a - Update globaliolimits.cfg.erb [13:54:19] !log restart lizardfs-master on misc3 for configuation change [13:54:25] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [13:55:56] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Je0OT [13:55:58] [02miraheze/puppet] 07paladox 0300fe064 - Update mediawiki.pp [14:34:08] !log revoke lizardfs6 puppet cert [14:34:14] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [14:58:27] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Je0sv [14:58:29] [02miraheze/puppet] 07paladox 03d39dff6 - lizardfs: Make lizardfs6 a master only [15:15:54] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 6 datacenters are down: 107.191.126.23/cpweb, 2604:180:0:33b::2/cpweb, 128.199.139.216/cpweb, 2400:6180:0:d0::403:f001/cpweb, 81.4.109.133/cpweb, 2a00:d880:5:8ea::ebc7/cpweb [15:15:56] PROBLEM - cp3 Varnish Backends on cp3 is CRITICAL: 3 backends are down. mw1 mw2 mw3 [15:16:02] PROBLEM - cp2 Varnish Backends on cp2 is CRITICAL: 3 backends are down. mw1 mw2 mw3 [15:16:12] PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 6 datacenters are down: 107.191.126.23/cpweb, 2604:180:0:33b::2/cpweb, 128.199.139.216/cpweb, 2400:6180:0:d0::403:f001/cpweb, 81.4.109.133/cpweb, 2a00:d880:5:8ea::ebc7/cpweb [15:16:57] hmm [15:17:52] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [15:17:59] RECOVERY - cp3 Varnish Backends on cp3 is OK: All 5 backends are healthy [15:18:04] RECOVERY - cp2 Varnish Backends on cp2 is OK: All 5 backends are healthy [15:18:11] RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online [15:19:07] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Je0sV [15:19:09] [02miraheze/puppet] 07paladox 034b37252 - Update config.yaml [15:25:10] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Je0sP [15:25:11] [02miraheze/services] 07MirahezeSSLBot 03c3736c1 - BOT: Updating services config for wikis [15:31:39] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Je0s5 [15:31:41] [02miraheze/puppet] 07paladox 034cb54ef - Update config.yaml [15:35:10] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Je0sx [15:35:11] [02miraheze/services] 07MirahezeSSLBot 0310c5ae5 - BOT: Updating services config for wikis [15:36:11] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Je0sp [15:36:13] [02miraheze/puppet] 07paladox 03995bad1 - parsoid: Add timeout to nginx [15:40:57] RECOVERY - lizardfs6 lizard.miraheze.org HTTPS on lizardfs6 is OK: HTTP OK: Status line output matched "HTTP/1.1 401 Unauthorized" - 381 bytes in 0.058 second response time [15:40:57] RECOVERY - lizardfs6 Lizardfs Master Port 3 on lizardfs6 is OK: TCP OK - 0.023 second response time on 51.158.176.139 port 9421 [15:40:57] RECOVERY - lizardfs6 SSH on lizardfs6 is OK: SSH OK - OpenSSH_7.9p1 Debian-10+deb10u1 (protocol 2.0) [15:41:42] RECOVERY - lizardfs6 Current Load on lizardfs6 is OK: OK - load average: 0.36, 0.13, 0.08 [15:41:57] RECOVERY - lizardfs6 Lizardfs Master Port 2 on lizardfs6 is OK: TCP OK - 0.013 second response time on 51.158.176.139 port 9420 [15:46:30] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Je0Gm [15:46:31] [02miraheze/puppet] 07paladox 03e968564 - Update client.pp [15:52:23] !log remove global file locking from all mounts on mw[123] and test1 [15:52:28] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [16:43:29] can somebody close this discussion for me? i withdraw my request. https://meta.miraheze.org/wiki/Meta:Requests_for_permissions#Apap04_.28Wiki_Creator.29 [16:43:30] [ Meta:Requests for permissions - Miraheze Meta ] - meta.miraheze.org [16:44:03] apap: can you state that you're withdrawing on the request please? [16:44:25] yes, i will [16:45:54] done [16:50:34] and closed [16:50:38] thanks [16:50:46] yw [18:28:03] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 7.47, 6.96, 6.55 [18:30:00] RECOVERY - mw3 Current Load on mw3 is OK: OK - load average: 6.04, 6.59, 6.46 [18:37:42] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 7.95, 7.19, 6.76 [18:39:41] PROBLEM - mw3 Current Load on mw3 is CRITICAL: CRITICAL - load average: 11.63, 9.00, 7.47 [18:40:41] Apap: Hi [18:41:36] hello [18:44:24] [02puppet] 07paladox created branch 03paladox-patch-1 - 13https://git.io/vbiAS [18:44:25] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/Je0C8 [18:44:27] [02miraheze/puppet] 07paladox 03e091adf - Convert wildcard (miraheze) certificate to LetsEncrypt [18:44:34] Apap: I have not really seen you much lately [18:44:42] [02puppet] 07paladox opened pull request 03#1115: Convert wildcard (miraheze) certificate to LetsEncrypt - 13https://git.io/Je0C4 [18:46:14] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/Je0C0 [18:46:16] [02miraheze/puppet] 07paladox 032571f26 - Update restbase [18:46:17] [02puppet] 07paladox synchronize pull request 03#1115: Convert wildcard (miraheze) certificate to LetsEncrypt - 13https://git.io/Je0C4 [18:47:02] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/Je0CE [18:47:04] [02miraheze/puppet] 07paladox 03179e0ff - Update grafana.conf [18:47:05] [02puppet] 07paladox synchronize pull request 03#1115: Convert wildcard (miraheze) certificate to LetsEncrypt - 13https://git.io/Je0C4 [18:47:28] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/Je0Cu [18:47:30] [02miraheze/puppet] 07paladox 03c1748f9 - Update roundcubemail.conf [18:47:31] [02puppet] 07paladox synchronize pull request 03#1115: Convert wildcard (miraheze) certificate to LetsEncrypt - 13https://git.io/Je0C4 [18:48:04] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/Je0Cg [18:48:06] [02miraheze/puppet] 07paladox 03258cba1 - Update nginx.conf [18:48:07] [02puppet] 07paladox synchronize pull request 03#1115: Convert wildcard (miraheze) certificate to LetsEncrypt - 13https://git.io/Je0C4 [18:48:28] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/Je0C2 [18:48:30] [02miraheze/puppet] 07paladox 0326cbd4a - Update mail.miraheze.org.conf [18:48:31] [02puppet] 07paladox synchronize pull request 03#1115: Convert wildcard (miraheze) certificate to LetsEncrypt - 13https://git.io/Je0C4 [18:49:06] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/Je0Cw [18:49:07] [02miraheze/puppet] 07paladox 03adeffdd - Update icinga2.conf [18:49:09] [02puppet] 07paladox synchronize pull request 03#1115: Convert wildcard (miraheze) certificate to LetsEncrypt - 13https://git.io/Je0C4 [18:49:36] yeah im kinda new here, been here for almost a month [18:49:43] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/Je0Co [18:49:44] [02miraheze/puppet] 07paladox 03a12553f - Update phabricator.miraheze.org.conf [18:49:46] [02puppet] 07paladox synchronize pull request 03#1115: Convert wildcard (miraheze) certificate to LetsEncrypt - 13https://git.io/Je0C4 [18:50:11] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/Je0C6 [18:50:13] [02miraheze/puppet] 07paladox 03026b319 - Update mw.cnf.erb [18:50:14] [02puppet] 07paladox synchronize pull request 03#1115: Convert wildcard (miraheze) certificate to LetsEncrypt - 13https://git.io/Je0C4 [18:51:51] Apap: Would you like to join my personal channel ##Examknow? [18:51:55] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/Je0Ci [18:51:56] [02miraheze/puppet] 07paladox 03fc8d747 - Update init.pp [18:51:57] [02puppet] 07paladox synchronize pull request 03#1115: Convert wildcard (miraheze) certificate to LetsEncrypt - 13https://git.io/Je0C4 [18:51:59] sure [18:53:03] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/Je0C1 [18:53:03] Apap: I have invited you to the channel [18:53:04] [02miraheze/puppet] 07paladox 03cda418e - Update mediawiki.conf [18:53:05] [02puppet] 07paladox synchronize pull request 03#1115: Convert wildcard (miraheze) certificate to LetsEncrypt - 13https://git.io/Je0C4 [18:54:52] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Je0CQ [18:54:54] [02miraheze/puppet] 07paladox 03e6ebe60 - Update wildcard.pp [19:01:34] PROBLEM - bacula1 Bacula Private Git on bacula1 is WARNING: WARNING: Full, 4217 files, 8.761MB, 2019-10-15 18:59:00 (1.1 weeks ago) [19:01:40] [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Je0WU [19:01:41] [02miraheze/mw-config] 07paladox 03078e96e - Update Sitenotice.php [19:03:41] [02ssl] 07paladox created branch 03paladox-patch-2 - 13https://git.io/vxP9L [19:03:43] [02miraheze/ssl] 07paladox pushed 031 commit to 03paladox-patch-2 [+0/-0/±1] 13https://git.io/Je0WI [19:03:44] [02miraheze/ssl] 07paladox 03480a2b7 - Convert wildcard (miraheze) cert to letsencrypt [19:03:46] [02ssl] 07paladox opened pull request 03#229: Convert wildcard (miraheze) cert to letsencrypt - 13https://git.io/Je0WL [19:09:06] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 6.65, 6.84, 7.97 [19:09:54] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/Je0Wc [19:09:56] [02miraheze/puppet] 07paladox 030689e35 - Update mathoid [19:09:57] [02puppet] 07paladox synchronize pull request 03#1115: Convert wildcard (miraheze) certificate to LetsEncrypt - 13https://git.io/Je0C4 [19:10:21] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/Je0WC [19:10:23] [02miraheze/puppet] 07paladox 03a2ab4fc - Update restbase [19:10:24] [02puppet] 07paladox synchronize pull request 03#1115: Convert wildcard (miraheze) certificate to LetsEncrypt - 13https://git.io/Je0C4 [19:13:55] !log disable puppet everywhere [19:14:00] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [19:15:47] [02miraheze/dns] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Je0WR [19:15:49] [02miraheze/dns] 07paladox 033f3acb2 - Add acme challenges to miraheze.org [19:16:18] PROBLEM - db5 Puppet on db5 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 4 minutes ago with 0 failures [19:16:21] PROBLEM - mw2 Puppet on mw2 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 3 minutes ago with 0 failures [19:16:25] PROBLEM - mw1 Puppet on mw1 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 3 minutes ago with 0 failures [19:16:28] PROBLEM - test1 Puppet on test1 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 3 minutes ago with 0 failures [19:16:45] PROBLEM - misc2 Puppet on misc2 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 4 minutes ago with 0 failures [19:16:55] PROBLEM - bacula1 Puppet on bacula1 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 4 minutes ago with 0 failures [19:16:56] PROBLEM - cp4 Puppet on cp4 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 4 minutes ago with 0 failures [19:17:06] PROBLEM - misc3 Puppet on misc3 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 4 minutes ago with 0 failures [19:17:07] PROBLEM - db4 Puppet on db4 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 4 minutes ago with 0 failures [19:17:37] PROBLEM - lizardfs5 Puppet on lizardfs5 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 5 minutes ago with 0 failures [19:17:40] PROBLEM - mw3 Puppet on mw3 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 4 minutes ago with 0 failures [19:17:50] PROBLEM - cp2 Puppet on cp2 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 4 minutes ago with 0 failures [19:17:56] PROBLEM - lizardfs4 Puppet on lizardfs4 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 5 minutes ago with 0 failures [19:18:00] PROBLEM - puppet1 Puppet on puppet1 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 5 minutes ago with 0 failures [19:18:15] PROBLEM - misc4 Puppet on misc4 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 5 minutes ago with 0 failures [19:18:16] PROBLEM - cp3 Puppet on cp3 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 4 minutes ago with 0 failures [19:18:20] [02miraheze/ssl] 07paladox pushed 031 commit to 03paladox-patch-2 [+0/-0/±1] 13https://git.io/Je0Wu [19:18:22] [02miraheze/ssl] 07paladox 03f000053 - Update wildcard.miraheze.org.crt [19:18:23] [02ssl] 07paladox synchronize pull request 03#229: Convert wildcard (miraheze) cert to letsencrypt - 13https://git.io/Je0WL [19:23:00] [02ssl] 07paladox closed pull request 03#229: Convert wildcard (miraheze) cert to letsencrypt - 13https://git.io/Je0WL [19:23:00] PROBLEM - misc1 Puppet on misc1 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 2 minutes ago with 0 failures [19:23:01] [02miraheze/ssl] 07paladox pushed 031 commit to 03master [+0/-0/±2] 13https://git.io/Je0Ww [19:23:03] [02miraheze/ssl] 07paladox 030e0748b - Convert wildcard (miraheze) cert to letsencrypt (#229) * Convert wildcard (miraheze) cert to letsencrypt * Update wildcard.miraheze.org.crt [19:23:04] [02ssl] 07paladox deleted branch 03paladox-patch-2 - 13https://git.io/vxP9L [19:23:05] PROBLEM - mw3 Current Load on mw3 is CRITICAL: CRITICAL - load average: 8.33, 6.49, 6.96 [19:23:06] [02miraheze/ssl] 07paladox deleted branch 03paladox-patch-2 [19:23:08] [02puppet] 07paladox closed pull request 03#1115: Convert wildcard (miraheze) certificate to LetsEncrypt - 13https://git.io/Je0C4 [19:23:09] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±12] 13https://git.io/Je0Wr [19:23:10] [02miraheze/puppet] 07paladox 037cfca80 - Convert wildcard (miraheze) certificate to LetsEncrypt (#1115) * Convert wildcard (miraheze) certificate to LetsEncrypt * Update restbase * Update grafana.conf * Update roundcubemail.conf * Update nginx.conf * Update mail.miraheze.org.conf * Update icinga2.conf * Update phabricator.miraheze.org.conf * Update mw.cnf.erb * Update init.pp * Update mediawiki.conf * Update [19:23:11] mathoid * Update restbase [19:23:26] !log update wildcard.miraheze.org in the private puppet repo [19:23:31] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [19:24:47] !log updating wildcard.miraheze.org on db[45] [19:24:52] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [19:25:06] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 7.61, 6.75, 7.00 [19:25:09] RECOVERY - db4 Puppet on db4 is OK: OK: Puppet is currently enabled, last run 4 seconds ago with 0 failures [19:25:59] !log upgrade mariadb on db[45] [19:26:05] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [19:26:18] RECOVERY - db5 Puppet on db5 is OK: OK: Puppet is currently enabled, last run 55 seconds ago with 0 failures [19:27:03] RECOVERY - mw3 Current Load on mw3 is OK: OK - load average: 4.31, 5.88, 6.65 [19:28:17] PROBLEM - db4 MySQL on db4 is CRITICAL: Can't connect to MySQL server on '81.4.109.166' (111 "Connection refused") [19:28:18] PROBLEM - misc4 phab.miraheze.wiki HTTPS on misc4 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/1.1 500 Internal Server Error [19:28:32] PROBLEM - cp3 Varnish Backends on cp3 is CRITICAL: 3 backends are down. mw1 mw2 mw3 [19:28:38] PROBLEM - misc1 webmail.miraheze.org HTTPS on misc1 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/1.1 200 OK [19:28:39] PROBLEM - cp4 Varnish Backends on cp4 is CRITICAL: 3 backends are down. mw1 mw2 mw3 [19:29:10] PROBLEM - cp4 Stunnel Http for misc2 on cp4 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 2370 bytes in 0.079 second response time [19:29:11] known ^ [19:29:19] PROBLEM - cp2 Stunnel Http for misc2 on cp2 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 2370 bytes in 0.359 second response time [19:29:19] PROBLEM - misc4 phabricator.miraheze.org HTTPS on misc4 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 4221 bytes in 0.032 second response time [19:29:33] PROBLEM - misc2 HTTPS on misc2 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 2370 bytes in 0.068 second response time [19:29:33] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 6 datacenters are down: 107.191.126.23/cpweb, 2604:180:0:33b::2/cpweb, 128.199.139.216/cpweb, 2400:6180:0:d0::403:f001/cpweb, 81.4.109.133/cpweb, 2a00:d880:5:8ea::ebc7/cpweb [19:30:13] PROBLEM - cp2 Varnish Backends on cp2 is CRITICAL: 3 backends are down. mw1 mw2 mw3 [19:30:16] PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 6 datacenters are down: 107.191.126.23/cpweb, 2604:180:0:33b::2/cpweb, 128.199.139.216/cpweb, 2400:6180:0:d0::403:f001/cpweb, 81.4.109.133/cpweb, 2a00:d880:5:8ea::ebc7/cpweb [19:30:37] PROBLEM - cp3 Stunnel Http for misc2 on cp3 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 2370 bytes in 0.600 second response time [19:33:09] PROBLEM - db4 Puppet on db4 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Package[mariadb-server] [19:36:17] RECOVERY - db4 MySQL on db4 is OK: Uptime: 195 Threads: 26 Questions: 165 Slow queries: 12 Opens: 28 Flush tables: 1 Open tables: 22 Queries per second avg: 0.846 [19:39:06] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Je0Wp [19:39:06] PROBLEM - db4 Puppet on db4 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 1 minute ago with 0 failures [19:39:07] [02miraheze/puppet] 07paladox 03c2aa065 - Update mw.cnf.erb [19:42:16] PROBLEM - db4 MySQL on db4 is CRITICAL: Can't connect to MySQL server on '81.4.109.166' (111 "Connection refused") [20:04:04] 503 [20:06:24] Error 503 Backend fetch failed, forwarded for XXX.246.XXX.146, XXX.0.0.1 [20:06:42] (Varnish XID 181122057) via cp2 at [20:07:10] [02miraheze/ssl] 07paladox pushed 031 commit to 03master [+1/-0/±0] 13https://git.io/Je0l2 [20:07:11] [02miraheze/ssl] 07paladox 03697254b - Add lets-encrypt-x3-cross-signed.crt [20:08:07] [02puppet] 07paladox created branch 03paladox-patch-2 - 13https://git.io/vbiAS [20:08:08] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-2 [+0/-0/±1] 13https://git.io/Je0la [20:08:10] [02miraheze/puppet] 07paladox 03fc8c445 - Use lets-encrypt-x3-cross-signed.crt ca for mysql [20:08:11] [02puppet] 07paladox opened pull request 03#1116: Use lets-encrypt-x3-cross-signed.crt ca for mysql - 13https://git.io/Je0lV [20:08:57] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-2 [+0/-0/±1] 13https://git.io/Je0lw [20:08:59] [02miraheze/puppet] 07paladox 03898d630 - Update wildcard.pp [20:09:00] [02puppet] 07paladox synchronize pull request 03#1116: Use lets-encrypt-x3-cross-signed.crt ca for mysql - 13https://git.io/Je0lV [20:09:12] [02puppet] 07paladox closed pull request 03#1116: Use lets-encrypt-x3-cross-signed.crt ca for mysql - 13https://git.io/Je0lV [20:09:13] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±2] 13https://git.io/Je0lo [20:09:15] [02miraheze/puppet] 07paladox 03479efb5 - Use lets-encrypt-x3-cross-signed.crt ca for mysql (#1116) * Use lets-encrypt-x3-cross-signed.crt ca for mysql * Update wildcard.pp [20:10:19] PROBLEM - db5 Puppet on db5 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 9 minutes ago with 0 failures [20:13:07] RECOVERY - db4 Puppet on db4 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [20:13:28] :/ [20:13:58] it's to be expected right now, basically emergency maintenance [20:13:59] [02puppet] 07paladox deleted branch 03paladox-patch-2 - 13https://git.io/vbiAS [20:14:01] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-2 [20:14:30] !log deploying new wildcard cert to misc[1234] [20:14:57] RECOVERY - misc1 Puppet on misc1 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [20:16:11] PROBLEM - hellointernet.miraheze.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Certificate '*.miraheze.org' expired on Wed 23 Oct 2019 08:12:13 PM GMT +0000. [20:16:12] PROBLEM - unmade.miraheze.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Certificate '*.miraheze.org' expired on Wed 23 Oct 2019 08:12:13 PM GMT +0000. [20:16:16] [02puppet] 07paladox created branch 03paladox-patch-2 - 13https://git.io/vbiAS [20:16:16] PROBLEM - tallguysfree.miraheze.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Certificate '*.miraheze.org' expired on Wed 23 Oct 2019 08:12:13 PM GMT +0000. [20:16:17] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-2 [+0/-0/±1] 13https://git.io/Je0lD [20:16:19] [02miraheze/puppet] 07paladox 03af61efc - Use /etc/ssl/certs/lets-encrypt-x3-cross-signed everywhere [20:16:20] PROBLEM - commonswiki.miraheze.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Certificate '*.miraheze.org' expired on Wed 23 Oct 2019 08:12:13 PM GMT +0000. [20:16:31] PROBLEM - cnt.miraheze.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Certificate '*.miraheze.org' expired on Wed 23 Oct 2019 08:12:13 PM GMT +0000. [20:16:35] right.. [20:16:37] PROBLEM - studynotekr.miraheze.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Certificate '*.miraheze.org' expired on Wed 23 Oct 2019 08:12:13 PM GMT +0000. [20:16:47] PROBLEM - marthaspeaks.miraheze.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Certificate '*.miraheze.org' expired on Wed 23 Oct 2019 08:12:13 PM GMT +0000. [20:16:48] PROBLEM - intp.miraheze.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Certificate '*.miraheze.org' expired on Wed 23 Oct 2019 08:12:13 PM GMT +0000. [20:16:52] PROBLEM - sims.miraheze.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Certificate '*.miraheze.org' expired on Wed 23 Oct 2019 08:12:13 PM GMT +0000. [20:16:56] [02puppet] 07paladox created branch 03paladox-patch-4 - 13https://git.io/vbiAS [20:16:57] PROBLEM - piwik.miraheze.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Certificate '*.miraheze.org' expired on Wed 23 Oct 2019 08:12:13 PM GMT +0000. [20:16:57] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-4 [+0/-0/±1] 13https://git.io/Je0ly [20:16:59] [02miraheze/puppet] 07paladox 0314ebd88 - Use /etc/ssl/certs/lets-encrypt-x3-cross-signed.crt everywhere [20:17:00] [02puppet] 07paladox opened pull request 03#1117: Use /etc/ssl/certs/lets-encrypt-x3-cross-signed.crt everywhere - 13https://git.io/Je0lS [20:17:46] RECOVERY - misc4 phabricator.miraheze.org HTTPS on misc4 is OK: HTTP OK: HTTP/1.1 200 OK - 19058 bytes in 0.126 second response time [20:17:59] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-4 [+0/-0/±1] 13https://git.io/Je0l9 [20:18:00] [02miraheze/puppet] 07paladox 03108dedd - Update icinga2.conf [20:18:01] [02puppet] 07paladox synchronize pull request 03#1117: Use /etc/ssl/certs/lets-encrypt-x3-cross-signed.crt everywhere - 13https://git.io/Je0lS [20:18:15] RECOVERY - db4 MySQL on db4 is OK: Uptime: 201 Threads: 72 Questions: 3395 Slow queries: 79 Opens: 139 Flush tables: 1 Open tables: 133 Queries per second avg: 16.890 [20:18:19] RECOVERY - misc4 phab.miraheze.wiki HTTPS on misc4 is OK: HTTP OK: Status line output matched "HTTP/1.1 200" - 17724 bytes in 0.077 second response time [20:18:29] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-4 [+0/-0/±1] 13https://git.io/Je0lQ [20:18:31] [02miraheze/puppet] 07paladox 0395b503b - Update nginx.conf [20:18:32] [02puppet] 07paladox synchronize pull request 03#1117: Use /etc/ssl/certs/lets-encrypt-x3-cross-signed.crt everywhere - 13https://git.io/Je0lS [20:19:10] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-4 [+0/-0/±1] 13https://git.io/Je0l7 [20:19:11] [02miraheze/puppet] 07paladox 03483d529 - Update mediawiki.conf [20:19:13] [02puppet] 07paladox synchronize pull request 03#1117: Use /etc/ssl/certs/lets-encrypt-x3-cross-signed.crt everywhere - 13https://git.io/Je0lS [20:19:33] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-4 [+0/-0/±1] 13https://git.io/Je0l5 [20:19:34] [02miraheze/puppet] 07paladox 0306cdcdf - Update init.pp [20:19:36] [02puppet] 07paladox synchronize pull request 03#1117: Use /etc/ssl/certs/lets-encrypt-x3-cross-signed.crt everywhere - 13https://git.io/Je0lS [20:19:51] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-4 [+0/-0/±1] 13https://git.io/Je0ld [20:19:53] [02miraheze/puppet] 07paladox 037eacc98 - Update phabricator.miraheze.org.conf [20:19:54] [02puppet] 07paladox synchronize pull request 03#1117: Use /etc/ssl/certs/lets-encrypt-x3-cross-signed.crt everywhere - 13https://git.io/Je0lS [20:20:06] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-4 [+0/-0/±1] 13https://git.io/Je0lb [20:20:07] [02miraheze/puppet] 07paladox 03f1633ad - Update phab.miraheze.wiki.conf [20:20:09] [02puppet] 07paladox synchronize pull request 03#1117: Use /etc/ssl/certs/lets-encrypt-x3-cross-signed.crt everywhere - 13https://git.io/Je0lS [20:20:20] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-4 [+0/-0/±1] 13https://git.io/Je0lN [20:20:21] [02miraheze/puppet] 07paladox 03772e071 - Update mail.miraheze.org.conf [20:20:23] [02puppet] 07paladox synchronize pull request 03#1117: Use /etc/ssl/certs/lets-encrypt-x3-cross-signed.crt everywhere - 13https://git.io/Je0lS [20:20:48] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-4 [+0/-0/±1] 13https://git.io/Je0lA [20:20:49] [02miraheze/puppet] 07paladox 038d7d867 - Update roundcubemail.conf [20:20:51] [02puppet] 07paladox synchronize pull request 03#1117: Use /etc/ssl/certs/lets-encrypt-x3-cross-signed.crt everywhere - 13https://git.io/Je0lS [20:21:08] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-4 [+0/-0/±1] 13https://git.io/Je0lx [20:21:10] [02miraheze/puppet] 07paladox 03183d704 - Update mathoid [20:21:11] [02puppet] 07paladox synchronize pull request 03#1117: Use /etc/ssl/certs/lets-encrypt-x3-cross-signed.crt everywhere - 13https://git.io/Je0lS [20:21:29] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-4 [+0/-0/±1] 13https://git.io/Je0lp [20:21:31] [02miraheze/puppet] 07paladox 03a2f270a - Update restbase [20:21:32] [02puppet] 07paladox synchronize pull request 03#1117: Use /etc/ssl/certs/lets-encrypt-x3-cross-signed.crt everywhere - 13https://git.io/Je0lS [20:21:56] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-4 [+0/-0/±1] 13https://git.io/Je0lj [20:21:57] [02miraheze/puppet] 07paladox 035d6beb4 - Update mediawiki.conf [20:21:59] [02puppet] 07paladox synchronize pull request 03#1117: Use /etc/ssl/certs/lets-encrypt-x3-cross-signed.crt everywhere - 13https://git.io/Je0lS [20:22:11] [02puppet] 07paladox closed pull request 03#1117: Use /etc/ssl/certs/lets-encrypt-x3-cross-signed.crt everywhere - 13https://git.io/Je0lS [20:22:13] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±12] 13https://git.io/Je08v [20:22:14] [02miraheze/puppet] 07paladox 03f33111e - Use /etc/ssl/certs/lets-encrypt-x3-cross-signed.crt everywhere (#1117) * Use /etc/ssl/certs/lets-encrypt-x3-cross-signed.crt everywhere * Update icinga2.conf * Update nginx.conf * Update mediawiki.conf * Update init.pp * Update phabricator.miraheze.org.conf * Update phab.miraheze.wiki.conf * Update mail.miraheze.org.conf * Update roundcubemail.conf * Update mathoid * [20:22:14] Update restbase * Update mediawiki.conf [20:24:21] RECOVERY - db5 Puppet on db5 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [20:26:44] RECOVERY - misc2 Puppet on misc2 is OK: OK: Puppet is currently enabled, last run 25 seconds ago with 0 failures [20:27:14] RECOVERY - misc3 Puppet on misc3 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [20:27:41] PROBLEM - misc4 phabricator.miraheze.org HTTPS on misc4 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 4221 bytes in 0.043 second response time [20:27:50] RECOVERY - misc4 Puppet on misc4 is OK: OK: Puppet is currently enabled, last run 2 seconds ago with 0 failures [20:28:29] !log deploying new wildcard cert to mw[123] and test1 [20:28:29] RECOVERY - test1 Puppet on test1 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [20:30:02] PROBLEM - db4 MySQL on db4 is CRITICAL: Can't connect to MySQL server on '81.4.109.166' (111 "Connection refused") [20:30:10] RECOVERY - mw1 Puppet on mw1 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [20:30:16] !log restarting postgresql to pickup new cert [20:30:17] PROBLEM - misc4 phab.miraheze.wiki HTTPS on misc4 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/1.1 500 Internal Server Error [20:32:18] !log restart puppetdb [20:32:24] PROBLEM - test1 Puppet on test1 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [20:32:32] Hello ErtasVideos! If you have any questions, feel free to ask and someone should answer soon. [20:32:36] hi [20:33:02] Hi [20:33:16] irc is just old school discord [20:33:28] PROBLEM - db4 Puppet on db4 is WARNING: WARNING: Puppet is currently disabled, message: reason not specified, last run 12 minutes ago with 0 failures [20:33:50] im just here for no reason [20:34:14] ok [20:34:19] PROBLEM - misc1 Puppet on misc1 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [20:34:32] this is pretty dope [20:34:35] at least we get all the alerts here immediately [20:34:44] PROBLEM - misc2 Puppet on misc2 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [20:34:56] PROBLEM - misc3 Puppet on misc3 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [20:35:13] PROBLEM - ns1 Puppet on ns1 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [20:35:28] PROBLEM - lizardfs6 Puppet on lizardfs6 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [20:35:30] PROBLEM - misc4 Puppet on misc4 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [20:36:04] PROBLEM - mw1 Puppet on mw1 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 3 minutes ago with 0 failures [20:36:21] PROBLEM - db5 Puppet on db5 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 4 minutes ago with 0 failures [20:37:38] RECOVERY - misc4 phabricator.miraheze.org HTTPS on misc4 is OK: HTTP OK: HTTP/1.1 200 OK - 19058 bytes in 7.617 second response time [20:37:51] RECOVERY - db4 MySQL on db4 is OK: Uptime: 319 Threads: 77 Questions: 5803 Slow queries: 204 Opens: 138 Flush tables: 1 Open tables: 132 Queries per second avg: 18.191 [20:38:17] RECOVERY - misc4 phab.miraheze.wiki HTTPS on misc4 is OK: HTTP OK: Status line output matched "HTTP/1.1 200" - 17724 bytes in 0.074 second response time [20:51:06] RECOVERY - ns1 Puppet on ns1 is OK: OK: Puppet is currently enabled, last run 37 seconds ago with 0 failures [20:51:19] RECOVERY - lizardfs6 Puppet on lizardfs6 is OK: OK: Puppet is currently enabled, last run 46 seconds ago with 0 failures [20:56:33] PROBLEM - misc4 phab.miraheze.wiki HTTPS on misc4 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:56:37] PROBLEM - misc4 phabricator.miraheze.org HTTPS on misc4 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:03:55] RECOVERY - intp.miraheze.org - LetsEncrypt on sslhost is OK: OK - Certificate 'miraheze.org' will expire on Tue 21 Jan 2020 06:57:00 PM GMT +0000. [21:03:56] RECOVERY - cnt.miraheze.org - LetsEncrypt on sslhost is OK: OK - Certificate 'miraheze.org' will expire on Tue 21 Jan 2020 06:57:00 PM GMT +0000. [21:04:13] RECOVERY - studynotekr.miraheze.org - LetsEncrypt on sslhost is OK: OK - Certificate 'miraheze.org' will expire on Tue 21 Jan 2020 06:57:00 PM GMT +0000. [21:04:18] RECOVERY - marthaspeaks.miraheze.org - LetsEncrypt on sslhost is OK: OK - Certificate 'miraheze.org' will expire on Tue 21 Jan 2020 06:57:00 PM GMT +0000. [21:04:40] RECOVERY - tallguysfree.miraheze.org - LetsEncrypt on sslhost is OK: OK - Certificate 'miraheze.org' will expire on Tue 21 Jan 2020 06:57:00 PM GMT +0000. [21:04:45] RECOVERY - commonswiki.miraheze.org - LetsEncrypt on sslhost is OK: OK - Certificate 'miraheze.org' will expire on Tue 21 Jan 2020 06:57:00 PM GMT +0000. [21:05:07] RECOVERY - hellointernet.miraheze.org - LetsEncrypt on sslhost is OK: OK - Certificate 'miraheze.org' will expire on Tue 21 Jan 2020 06:57:00 PM GMT +0000. [21:05:22] RECOVERY - unmade.miraheze.org - LetsEncrypt on sslhost is OK: OK - Certificate 'miraheze.org' will expire on Tue 21 Jan 2020 06:57:00 PM GMT +0000. [21:05:23] RECOVERY - sims.miraheze.org - LetsEncrypt on sslhost is OK: OK - Certificate 'miraheze.org' will expire on Tue 21 Jan 2020 06:57:00 PM GMT +0000. [21:05:29] RECOVERY - piwik.miraheze.org - LetsEncrypt on sslhost is OK: OK - Certificate 'miraheze.org' will expire on Tue 21 Jan 2020 06:57:00 PM GMT +0000. [21:05:35] RECOVERY - wc.miraheze.org on sslhost is OK: OK - Certificate 'miraheze.org' will expire on Tue 21 Jan 2020 06:57:00 PM GMT +0000. [21:05:45] that's a good sign [21:12:06] Are they not set to automatically renew themselves? [21:12:10] The SSL certificates, I mean [21:12:35] k6ka miraheze.org nope [21:12:45] we used GlobalSign, but there support was awful [21:12:58] And so they didn't renew it in time [21:13:14] We're having issues with using LE with mysql [21:13:14] PROBLEM - ns1 Puppet on ns1 is WARNING: WARNING: Puppet is currently disabled, message: reason not specified, last run 2 minutes ago with 0 failures [21:13:33] I thought you always used Let's Encrypt, since LE is free [21:13:48] PROBLEM - db4 MySQL on db4 is CRITICAL: Can't connect to MySQL server on '81.4.109.166' (111 "Connection refused") [21:13:52] GlobalSign provide a free wildcard cert for opensource projects [21:14:01] is that you paladox? [21:14:04] Nope [21:14:50] it segfaulted [21:17:49] SPF|Cloud puppet? [21:40:17] paladox: MariaDB [(none)]> SHOW VARIABLES LIKE '%ssl%'; [21:40:33] ssl_ca | /etc/mysql/ssl/ca-cert.pem [21:40:46] get the CA cert files from letsencrypt [21:40:46] we carn't get onto the commandline :( [21:40:50] make sure it's a .pem [21:40:50] root@db5:/home/paladox# mysql [21:40:51] ERROR 2026 (HY000): SSL connection error: tlsv1 alert unknown ca [21:40:56] change the ssl_ca value [21:41:13] paladox: oh, you are completely locked out from mysql ?? [21:41:20] how many servers are there? [21:41:22] yeh due to the ssl issue [21:41:26] we have 2 dbs [21:41:31] you still have shell access and root, right [21:41:31] both having the same problems [21:41:35] yup [21:41:57] so you could change the config for one or both [21:42:05] to allow connection without TLS from localhost [21:42:09] then connect from localhost [21:42:16] and then change the path to the ssl_ca [21:42:25] and then it should hopefully all just work with LE [21:42:47] or you could use a self-signed cert [21:42:52] or you could re-use puppet certs like WMF does [21:43:16] the biggest question is probably how to change config to allow connection without TLS from localhost (temp) [21:43:20] without breaking more [21:43:33] mutante but that wouldn't fix the ssl issue with mysql connecting to mysql [21:43:39] and wouldn't force a ssl connection [21:43:55] paladox: why not? the problem is "unknown CA" right? [21:44:00] so if you add the path to the right CA [21:44:01] yup [21:44:07] then it's not unknown anymore.. and success [21:44:07] we did [21:44:08] :) [21:44:20] how, if you say you are locked out? [21:44:32] you changed the path to ssl_ca in mysql console? [21:44:59] see comment on https://askubuntu.com/questions/772588/enable-mariadb-ssl [21:44:59] [ server - Enable MariaDB SSL - Ask Ubuntu ] - askubuntu.com [21:45:08] under SHOW VARIABLES LIKE '%ssl%'; [21:45:17] there is the cert, the CA and the CA path [21:45:55] the ssl_ca path should point to a .pem file that contains the Letsencrypt root certs [21:45:58] from https://letsencrypt.org/certificates/ [21:45:59] [ Chain of Trust - Let's Encrypt - Free SSL/TLS Certificates ] - letsencrypt.org [21:46:31] also.. if you buy a cert.. wouldnt you have to do the exact same thing? [21:46:42] just a different CA file [21:47:09] Yeh, the CA would be trusted [21:47:15] at least GlobalSign was [21:48:54] it trusts whatever you tell it to trust [21:49:15] GlobalSign just happens to be in some default file [21:49:29] that was shipped with the package or whatever installed it [21:50:05] since you want to use LE you just have to tell it to trust LE [21:51:21] mutante: easier said than done [21:51:32] I have installed it in a thousand different ways [21:51:39] tried the other X3 signed cert [21:52:09] so here is what i dont get: [21:52:15] whether you buy a cert or use LE [21:52:31] there are two people trying to buy a certificate now [21:52:42] because apparently LE doesn't work [21:53:34] you still have u [21:53:41] you still have to do the same thing now [21:53:53] which is - get on mysql again somehow [21:53:57] we can [21:53:58] --skip-ssl [21:53:59] by disabling TLS [21:54:09] and once you did that [21:54:19] you can change the ssl_ca value [21:54:33] mutante: explain to me why we couldn't by editing my.cnf and restarting mysql [21:55:00] even passing various parameters to the client doesn't work [21:55:10] SPF|Cloud: did you download the root CA files from Letsencrypt? [21:55:15] of course [21:55:17] both of em [21:55:23] what did you put in my.cnf [21:55:23] tried both of em as well [21:55:43] ssl-ca = /etc/ssl/certs/lets-encrypt-x3-cross-signed.crt [21:56:59] is that the same as https://letsencrypt.org/certs/isrgrootx1.pem.txt [21:57:22] sounds like it might be about ethe format of the cert [21:57:23] since the examples use a .pem [21:57:54] and you use a .crt [21:58:04] and there are different formats [21:58:21] the downloadable file is a pem [21:58:31] that is the ISRG signed one [21:58:37] this is the cross signed one [21:58:54] https://letsencrypt.org/certs/lets-encrypt-x3-cross-signed.pem.txt [21:58:59] this also ends in .pem [21:59:03] how did you get to .crt ? [21:59:23] you might have to combine root cert and intermediate cert [21:59:44] so that it gets the whole chain [22:00:00] uh wait a minute [22:00:03] there are also openssl commands to convert between the formats [22:00:04] let me try that? [22:04:14] PROBLEM - mw2 MediaWiki Rendering on mw2 is CRITICAL: HTTP CRITICAL: HTTP/1.1 503 Backend fetch failed - 4112 bytes in 0.045 second response time [22:04:50] PROBLEM - mw1 MediaWiki Rendering on mw1 is CRITICAL: HTTP CRITICAL: HTTP/1.1 503 Backend fetch failed - 4112 bytes in 0.075 second response time [22:05:00] PROBLEM - test1 MediaWiki Rendering on test1 is CRITICAL: HTTP CRITICAL: HTTP/1.1 503 Backend fetch failed - 4112 bytes in 0.033 second response time [22:05:06] PROBLEM - mw3 MediaWiki Rendering on mw3 is CRITICAL: HTTP CRITICAL: HTTP/1.1 503 Backend fetch failed - 4112 bytes in 0.030 second response time [22:07:15] [02miraheze/ssl] 07Southparkfan pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Je04F [22:07:17] [02miraheze/ssl] 07Southparkfan 034b8ae5e - Update lets-encrypt-x3-cross-signed.crt [22:07:43] paladox ^ [22:07:48] SPF|Cloud thanks! [22:07:54] I've brought it now :( [22:07:55] this did the trick on db5 [22:07:59] so might as well use it [22:08:13] gives us a year (and so we doin't have to restart mysql in 3 months) [22:08:52] let's bring the site up first [22:09:41] ok [22:09:53] great, allthetropes.org is now only complaining about db4 [22:10:02] that is a sign of recovery [22:10:52] [02miraheze/dns] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Je04b [22:10:54] [02miraheze/dns] 07paladox 039c49588 - Add namecheap challenge [22:11:03] SPF|Cloud awesome! [22:11:08] RECOVERY - db4 MySQL on db4 is OK: Uptime: 220 Threads: 42 Questions: 128 Slow queries: 7 Opens: 22 Flush tables: 1 Open tables: 16 Queries per second avg: 0.581 [22:11:14] RECOVERY - misc1 Puppet on misc1 is OK: OK: Puppet is currently enabled, last run 37 seconds ago with 0 failures [22:12:37] miraheze/dns/master/9c49588 - paladox The build was broken. https://travis-ci.org/miraheze/dns/builds/602046642 [22:13:19] [02miraheze/dns] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Je04A [22:13:20] [02miraheze/dns] 07paladox 038ade2bb - Update miraheze.org [22:14:38] PROBLEM - cp2 Stunnel Http for mw1 on cp2 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [22:15:08] PROBLEM - cp4 Stunnel Http for mw1 on cp4 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [22:15:24] PROBLEM - cp3 Stunnel Http for mw1 on cp3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [22:16:34] RECOVERY - cp2 Stunnel Http for mw1 on cp2 is OK: HTTP OK: HTTP/1.1 200 OK - 24655 bytes in 0.391 second response time [22:18:15] RECOVERY - cp2 Puppet on cp2 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [22:18:18] RECOVERY - cp3 Puppet on cp3 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [22:18:20] !log running puppet everywhere [22:18:23] RECOVERY - mw2 Puppet on mw2 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [22:18:34] RECOVERY - cp4 Puppet on cp4 is OK: OK: Puppet is currently enabled, last run 25 seconds ago with 0 failures [22:18:58] RECOVERY - bacula1 Puppet on bacula1 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [22:18:58] RECOVERY - lizardfs5 Puppet on lizardfs5 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [22:19:00] RECOVERY - lizardfs4 Puppet on lizardfs4 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [22:19:04] RECOVERY - db5 Puppet on db5 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [22:19:14] miraheze/dns/master/8ade2bb - paladox The build has errored. https://travis-ci.org/miraheze/dns/builds/602047493 [22:19:48] RECOVERY - mw1 Puppet on mw1 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [22:19:58] RECOVERY - db4 Puppet on db4 is OK: OK: Puppet is currently enabled, last run 3 seconds ago with 0 failures [22:19:59] RECOVERY - mw3 Puppet on mw3 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [22:20:04] RECOVERY - puppet1 Puppet on puppet1 is OK: OK: Puppet is currently enabled, last run 8 seconds ago with 0 failures [22:20:10] paladox: sudo -u www-data php sql.php --wiki allthetropeswiki [22:20:10] > show databases; [22:20:11] RECOVERY - misc2 Puppet on misc2 is OK: OK: Puppet is currently enabled, last run 11 seconds ago with 0 failures [22:20:12] works [22:20:18] SPF|Cloud awesome! [22:20:41] for some reason db4 is not as forgiving [22:20:53] RECOVERY - misc4 Puppet on misc4 is OK: OK: Puppet is currently enabled, last run 20 seconds ago with 0 failures [22:21:09] <7JTAAV2IP> miraheze/dns/master/8ade2bb - paladox The build was fixed. https://travis-ci.org/miraheze/dns/builds/602047493 [22:21:09] PROBLEM - cp2 Stunnel Http for mw1 on cp2 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [22:21:09] Hello 7JTAAV2IP! If you have any questions, feel free to ask and someone should answer soon. [22:21:10] [ Travis CI - Test and Deploy Your Code with Confidence ] - travis-ci.org [22:21:36] RECOVERY - cp3 Stunnel Http for mw1 on cp3 is OK: HTTP OK: HTTP/1.1 200 OK - 24639 bytes in 0.634 second response time [22:22:10] RECOVERY - misc3 Puppet on misc3 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [22:23:04] RECOVERY - cp2 Stunnel Http for mw1 on cp2 is OK: HTTP OK: HTTP/1.1 200 OK - 24661 bytes in 0.389 second response time [22:23:16] RECOVERY - cp4 Stunnel Http for mw1 on cp4 is OK: HTTP OK: HTTP/1.1 200 OK - 24661 bytes in 0.004 second response time [22:23:17] RECOVERY - test1 Puppet on test1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [22:24:50] PROBLEM - db4 MySQL on db4 is CRITICAL: Can't connect to MySQL server on '81.4.109.166' (111 "Connection refused") [22:26:57] new attempt to start mysql [22:27:48] ah! back.. i was offline [22:27:57] so updating Letsecnrypt root CA cert fixed it? [22:28:10] i see https://github.com/miraheze/ssl/compare/697254ba38bd...4b8ae5ee22cf [22:28:11] [ Comparing 697254ba38bd...4b8ae5ee22cf · miraheze/ssl · GitHub ] - github.com [22:28:15] it looks like we were only using the intermediate [22:28:18] not root+intermediate [22:28:24] makes sense [22:28:30] and that explains the error [22:28:43] so you concatenated them both and saved as -chained.pem or something? [22:29:02] that's the longer-term solution [22:29:05] that's like i've seen it before, ack [22:29:08] +1 [22:29:18] priority no 1 is getting the database servers up [22:29:21] everything else comes later [22:29:21] paladox: maybe you can just cancel the order [22:29:30] SPF|Cloud: full ack [22:29:35] I doin't think i can cancle [22:29:42] RECOVERY - misc1 webmail.miraheze.org HTTPS on misc1 is OK: HTTP OK: Status line output matched "HTTP/1.1 401 Unauthorized" - 5799 bytes in 0.068 second response time [22:29:45] but we need it so we doin't have to restart mysql in 3 months [22:29:49] if it worked on db5, just go on to the next and i stop distracting you [22:30:20] paladox: let him do quick fix and then do a planned maintenace for switching [22:30:28] if still desired then [22:30:31] ok [22:31:03] RECOVERY - db4 MySQL on db4 is OK: Uptime: 251 Threads: 98 Questions: 2854 Slow queries: 185 Opens: 158 Flush tables: 1 Open tables: 152 Queries per second avg: 11.370 [22:31:45] [02miraheze/dns] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Je0Bs [22:31:47] [02miraheze/dns] 07paladox 03ec3ba72 - Update miraheze.org [22:33:02] [02miraheze/puppet] 07Southparkfan pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Je0BG [22:33:03] [02miraheze/puppet] 07Southparkfan 03c71ff74 - Update config.ini.php.erb [22:33:43] PROBLEM - cp2 Stunnel Http for mw3 on cp2 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [22:33:46] ^ matomo is a service on db5, if this change makes it work then I can be sure we found the issue [22:34:07] PROBLEM - cp4 Stunnel Http for mw3 on cp4 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [22:34:12] yep [22:34:14] that fixed it [22:34:16] :) [22:34:17] PROBLEM - cp2 Stunnel Http for mw2 on cp2 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [22:34:25] RECOVERY - cp4 Stunnel Http for misc2 on cp4 is OK: HTTP OK: HTTP/1.1 200 OK - 41802 bytes in 0.106 second response time [22:34:39] PROBLEM - cp2 Stunnel Http for mw1 on cp2 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [22:34:49] PROBLEM - cp4 Stunnel Http for mw1 on cp4 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [22:34:50] RECOVERY - cp2 Stunnel Http for misc2 on cp2 is OK: HTTP OK: HTTP/1.1 200 OK - 41802 bytes in 0.580 second response time [22:35:01] PROBLEM - cp3 Stunnel Http for mw1 on cp3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [22:35:03] paladox: for the record, you have missed a lot of references to the CA in puppet [22:35:05] PROBLEM - cp4 Stunnel Http for mw2 on cp4 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [22:35:13] RECOVERY - cp3 Stunnel Http for misc2 on cp3 is OK: HTTP OK: HTTP/1.1 200 OK - 41802 bytes in 0.948 second response time [22:35:25] PROBLEM - cp3 Stunnel Http for mw3 on cp3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [22:35:29] SPF|Cloud oh? [22:35:37] https://github.com/miraheze/puppet/search?q=GlobalSign&unscoped_q=GlobalSign [22:35:38] [ Search · GlobalSign · GitHub ] - github.com [22:35:49] and yet db4 is still stuck for some reason [22:35:49] RECOVERY - misc2 HTTPS on misc2 is OK: HTTP OK: HTTP/1.1 200 OK - 41810 bytes in 0.089 second response time [22:35:52] PROBLEM - cp3 Stunnel Http for mw2 on cp3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [22:36:19] damn [22:36:28] SPF|Cloud ssl still fails? [22:36:42] mediawiki refuses to connect [22:38:32] sudo -u www-data mysql -hdb5.miraheze.org -umediawiki -pREDACTED --ssl-cert=/etc/ssl/certs/wildcard.miraheze.org.crt --ssl-key=/etc/ssl/private/wildcard.miraheze.org.key --ssl-ca=/etc/ssl/certs/lets-encrypt-x3-cross-signed.crt [22:38:50] SSL error: Unable to get private key from '/etc/ssl/private/wildcard.miraheze.org.key' [22:38:50] ERROR 2026 (HY000): SSL connection error: Unable to get private key [22:40:10] RECOVERY - cp3 Stunnel Http for mw2 on cp3 is OK: HTTP OK: HTTP/1.1 200 OK - 24661 bytes in 0.674 second response time [22:40:24] on mw1 it can't get the private key? [22:40:35] sudo -u www-data cat /etc/ssl/private/wildcard.miraheze.org.key [22:40:36] works [22:41:39] on mw1 [22:42:39] RECOVERY - cp2 Stunnel Http for mw3 on cp2 is OK: HTTP OK: HTTP/1.1 200 OK - 24655 bytes in 9.290 second response time [22:42:56] RECOVERY - cp4 Stunnel Http for mw3 on cp4 is OK: HTTP OK: HTTP/1.1 200 OK - 24639 bytes in 0.018 second response time [22:45:06] PROBLEM - cp3 Stunnel Http for mw2 on cp3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [22:45:44] RECOVERY - cp4 Stunnel Http for mw1 on cp4 is OK: HTTP OK: HTTP/1.1 200 OK - 24661 bytes in 2.925 second response time [22:45:44] RECOVERY - cp2 Stunnel Http for mw1 on cp2 is OK: HTTP OK: HTTP/1.1 200 OK - 24661 bytes in 4.398 second response time [22:45:48] RECOVERY - cp3 Stunnel Http for mw1 on cp3 is OK: HTTP OK: HTTP/1.1 200 OK - 24661 bytes in 0.693 second response time [22:45:54] RECOVERY - cp4 Stunnel Http for mw2 on cp4 is OK: HTTP OK: HTTP/1.1 200 OK - 24655 bytes in 0.003 second response time [22:46:25] RECOVERY - cp3 Stunnel Http for mw3 on cp3 is OK: HTTP OK: HTTP/1.1 200 OK - 24655 bytes in 0.676 second response time [22:47:04] RECOVERY - cp3 Stunnel Http for mw2 on cp3 is OK: HTTP OK: HTTP/1.1 200 OK - 24661 bytes in 0.642 second response time [22:47:36] RECOVERY - cp2 Stunnel Http for mw2 on cp2 is OK: HTTP OK: HTTP/1.1 200 OK - 24655 bytes in 0.389 second response time [22:48:06] PROBLEM - db4 MySQL on db4 is CRITICAL: Can't connect to MySQL server on '81.4.109.166' (111 "Connection refused") [22:49:22] PROBLEM - misc1 webmail.miraheze.org HTTPS on misc1 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/1.1 200 OK [22:49:40] !log rebooting db4 [22:50:22] paladox: other than the mess called db4, you can fix the other stuff [22:50:40] the stuff you mentioned above? [22:50:41] yup [22:50:43] paladox: he said "sudo -u www-data mysql .." so maybe the issue is with permission for mysql client ? [22:50:47] grafana, ... [22:50:54] he was running mysql as www-data [22:51:04] doing [22:51:04] mutante: but as www-data I could cat the contents of the private key file [22:51:24] and it was also affecting connections to db5 but db5 is working [22:51:38] [02puppet] 07paladox created branch 03paladox-patch-5 - 13https://git.io/vbiAS [22:51:40] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-5 [+0/-0/±1] 13https://git.io/Je0Bo [22:51:41] [02miraheze/puppet] 07paladox 036311553 - Use lets-encrypt-x3-cross-signed.crt everywhere [22:51:43] [02puppet] 07paladox opened pull request 03#1118: Use lets-encrypt-x3-cross-signed.crt everywhere - 13https://git.io/Je0BK [22:52:09] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-5 [+0/-0/±1] 13https://git.io/Je0BP [22:52:10] [02miraheze/puppet] 07paladox 03e0ce748 - Update grafana.ini.erb [22:52:12] [02puppet] 07paladox synchronize pull request 03#1118: Use lets-encrypt-x3-cross-signed.crt everywhere - 13https://git.io/Je0BK [22:52:18] SPF|Cloud: so the mysql client says "can't get" it.. but now it works? nice! [22:52:27] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-5 [+0/-0/±1] 13https://git.io/Je0BX [22:52:28] something like that [22:52:28] [02miraheze/puppet] 07paladox 03eb568a3 - Update parsoid [22:52:30] [02puppet] 07paladox synchronize pull request 03#1118: Use lets-encrypt-x3-cross-signed.crt everywhere - 13https://git.io/Je0BK [22:52:32] alrighty [22:52:50] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-5 [+0/-0/±1] 13https://git.io/Je0B1 [22:52:52] [02miraheze/puppet] 07paladox 03d4f7068 - Update nginx.conf [22:52:53] [02puppet] 07paladox synchronize pull request 03#1118: Use lets-encrypt-x3-cross-signed.crt everywhere - 13https://git.io/Je0BK [22:53:10] [02puppet] 07paladox closed pull request 03#1118: Use lets-encrypt-x3-cross-signed.crt everywhere - 13https://git.io/Je0BK [22:53:12] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±4] 13https://git.io/Je0By [22:53:13] [02miraheze/puppet] 07paladox 036f7b7ea - Use lets-encrypt-x3-cross-signed.crt everywhere (#1118) * Use lets-encrypt-x3-cross-signed.crt everywhere * Update grafana.ini.erb * Update parsoid * Update nginx.conf [22:53:15] [02puppet] 07paladox deleted branch 03paladox-patch-5 - 13https://git.io/vbiAS [22:53:16] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-5 [22:54:15] RECOVERY - db4 MySQL on db4 is OK: Uptime: 251 Threads: 95 Questions: 3194 Slow queries: 151 Opens: 146 Flush tables: 1 Open tables: 140 Queries per second avg: 12.725 [22:54:23] RECOVERY - cp4 Varnish Backends on cp4 is OK: All 5 backends are healthy [22:55:29] RECOVERY - mw3 MediaWiki Rendering on mw3 is OK: HTTP OK: HTTP/1.1 200 OK - 19917 bytes in 0.405 second response time [22:55:47] uh? [22:56:25] RECOVERY - mw1 MediaWiki Rendering on mw1 is OK: HTTP OK: HTTP/1.1 200 OK - 19909 bytes in 0.120 second response time [22:56:25] RECOVERY - mw2 MediaWiki Rendering on mw2 is OK: HTTP OK: HTTP/1.1 200 OK - 19909 bytes in 0.033 second response time [22:56:45] that looks good [22:57:12] still 503's in cp4 [22:58:02] PROBLEM - cp4 Stunnel Http for mw3 on cp4 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [22:58:02] RECOVERY - test1 MediaWiki Rendering on test1 is OK: HTTP OK: HTTP/1.1 200 OK - 19910 bytes in 0.166 second response time [22:58:37] PROBLEM - cp3 Stunnel Http for mw3 on cp3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [22:58:52] PROBLEM - cp3 Stunnel Http for mw2 on cp3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [22:59:06] PROBLEM - cp2 Stunnel Http for mw3 on cp2 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [22:59:07] PROBLEM - cp2 Stunnel Http for mw1 on cp2 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [22:59:22] SPF|Cloud: MariaDB [(none)]> use metawiki; [22:59:24] is just stuck [22:59:25] PROBLEM - cp2 Stunnel Http for mw2 on cp2 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [22:59:29] on db4 [22:59:43] PROBLEM - cp3 Stunnel Http for mw1 on cp3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [22:59:54] PROBLEM - cp4 Stunnel Http for mw1 on cp4 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [22:59:59] paladox: mysql metawiki -A works somehow [23:00:00] PROBLEM - cp4 Varnish Backends on cp4 is CRITICAL: 3 backends are down. mw1 mw2 mw3 [23:00:11] PROBLEM - cp4 Stunnel Http for mw2 on cp4 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [23:00:30] MariaDB [metawiki]> select * from ajaxpoll_vote; [23:00:31] is stuck [23:00:38] in fact connections are coming in fine, it's somehow not reading files [23:01:53] Did someone turn on verbose mode? [23:04:29] PROBLEM - misc1 grafana.miraheze.org HTTPS on misc1 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 309 bytes in 0.025 second response time [23:05:57] PROBLEM - misc1 HTTPS on misc1 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 309 bytes in 0.014 second response time [23:09:23] PROBLEM - mw3 HTTPS on mw3 is CRITICAL: connect to address 81.4.121.113 and port 443: Connection refusedHTTP CRITICAL - Unable to open TCP socket [23:09:34] RECOVERY - cp3 Stunnel Http for mw2 on cp3 is OK: HTTP OK: HTTP/1.1 200 OK - 24661 bytes in 2.783 second response time [23:09:52] RECOVERY - cp2 Stunnel Http for mw1 on cp2 is OK: HTTP OK: HTTP/1.1 200 OK - 24655 bytes in 0.389 second response time [23:10:23] RECOVERY - cp2 Stunnel Http for mw2 on cp2 is OK: HTTP OK: HTTP/1.1 200 OK - 24661 bytes in 0.390 second response time [23:10:29] RECOVERY - cp4 Stunnel Http for mw1 on cp4 is OK: HTTP OK: HTTP/1.1 200 OK - 24661 bytes in 0.005 second response time [23:10:31] RECOVERY - cp3 Stunnel Http for mw1 on cp3 is OK: HTTP OK: HTTP/1.1 200 OK - 24655 bytes in 0.693 second response time [23:11:04] RECOVERY - cp4 Stunnel Http for mw2 on cp4 is OK: HTTP OK: HTTP/1.1 200 OK - 24661 bytes in 0.005 second response time [23:11:31] PROBLEM - db4 MySQL on db4 is CRITICAL: Can't connect to MySQL server on '81.4.109.166' (111 "Connection refused") [23:12:57] RECOVERY - cp4 Stunnel Http for mw3 on cp4 is OK: HTTP OK: HTTP/1.1 200 OK - 24655 bytes in 0.004 second response time [23:13:12] RECOVERY - mw3 HTTPS on mw3 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 442 bytes in 0.022 second response time [23:13:20] RECOVERY - cp2 Stunnel Http for mw3 on cp2 is OK: HTTP OK: HTTP/1.1 200 OK - 24655 bytes in 0.393 second response time [23:13:33] RECOVERY - cp3 Stunnel Http for mw3 on cp3 is OK: HTTP OK: HTTP/1.1 200 OK - 24655 bytes in 0.853 second response time [23:27:26] PROBLEM - db4 Puppet on db4 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Package[mariadb-server] [23:36:48] RECOVERY - db4 MySQL on db4 is OK: Uptime: 195 Threads: 54 Questions: 758 Slow queries: 25 Opens: 114 Flush tables: 1 Open tables: 108 Queries per second avg: 3.887 [23:37:14] RECOVERY - misc4 phab.miraheze.wiki HTTPS on misc4 is OK: HTTP OK: Status line output matched "HTTP/1.1 200" - 17724 bytes in 0.074 second response time [23:37:22] RECOVERY - misc1 webmail.miraheze.org HTTPS on misc1 is OK: HTTP OK: Status line output matched "HTTP/1.1 401 Unauthorized" - 5799 bytes in 0.061 second response time [23:37:38] RECOVERY - cp3 Varnish Backends on cp3 is OK: All 5 backends are healthy [23:37:39] thank god [23:38:32] RECOVERY - misc4 phabricator.miraheze.org HTTPS on misc4 is OK: HTTP OK: HTTP/1.1 200 OK - 19073 bytes in 0.134 second response time [23:38:41] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [23:39:06] RECOVERY - cp4 Varnish Backends on cp4 is OK: All 5 backends are healthy [23:39:07] RECOVERY - cp2 Varnish Backends on cp2 is OK: All 5 backends are healthy [23:39:12] RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online [23:41:54] RECOVERY - misc1 HTTPS on misc1 is OK: HTTP OK: HTTP/1.1 302 Found - 408 bytes in 0.021 second response time [23:41:58] RECOVERY - misc1 grafana.miraheze.org HTTPS on misc1 is OK: HTTP OK: HTTP/1.1 302 Found - 408 bytes in 0.025 second response time [23:44:32] PROBLEM - misc4 phabricator.miraheze.org HTTPS on misc4 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 4221 bytes in 0.036 second response time [23:44:35] PROBLEM - db4 MySQL on db4 is CRITICAL: Can't connect to MySQL server on '81.4.109.166' (111 "Connection refused") [23:44:39] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 6 datacenters are down: 107.191.126.23/cpweb, 2604:180:0:33b::2/cpweb, 128.199.139.216/cpweb, 2400:6180:0:d0::403:f001/cpweb, 81.4.109.133/cpweb, 2a00:d880:5:8ea::ebc7/cpweb [23:45:03] PROBLEM - cp4 Varnish Backends on cp4 is CRITICAL: 3 backends are down. mw1 mw2 mw3 [23:45:05] PROBLEM - cp2 Varnish Backends on cp2 is CRITICAL: 3 backends are down. mw1 mw2 mw3 [23:45:16] PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 6 datacenters are down: 107.191.126.23/cpweb, 2604:180:0:33b::2/cpweb, 128.199.139.216/cpweb, 2400:6180:0:d0::403:f001/cpweb, 81.4.109.133/cpweb, 2a00:d880:5:8ea::ebc7/cpweb [23:45:16] PROBLEM - misc4 phab.miraheze.wiki HTTPS on misc4 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/1.1 500 Internal Server Error [23:45:24] PROBLEM - misc1 webmail.miraheze.org HTTPS on misc1 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/1.1 200 OK [23:45:32] PROBLEM - cp3 Varnish Backends on cp3 is CRITICAL: 3 backends are down. mw1 mw2 mw3 [23:51:32] RECOVERY - db4 Puppet on db4 is OK: OK: Puppet is currently enabled, last run 4 seconds ago with 0 failures [23:52:30] RECOVERY - db4 MySQL on db4 is OK: Uptime: 253 Threads: 104 Questions: 1370 Slow queries: 142 Opens: 64 Flush tables: 1 Open tables: 57 Queries per second avg: 5.415 [23:55:46] PROBLEM - db4 Puppet on db4 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 4 minutes ago with 0 failures [23:56:22] PROBLEM - db4 MySQL on db4 is CRITICAL: Can't connect to MySQL server on '81.4.109.166' (111 "Connection refused")