[00:01:49] RECOVERY - cp8 Disk Space on cp8 is OK: DISK OK - free space: / 3485 MB (18% inode=93%); [00:02:38] RECOVERY - cp6 Disk Space on cp6 is OK: DISK OK - free space: / 3051 MB (16% inode=93%); [00:02:52] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Jv73Q [00:02:54] [02miraheze/puppet] 07paladox 03d1c3b53 - Update mon1.yaml [00:03:48] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Jv737 [00:03:50] [02miraheze/puppet] 07paladox 0360515cd - Fix setting php::php_fpm::fpm_workers_multiplier [00:21:38] [02puppet] 07paladox closed pull request 03#1263: mariadb: Tweak config - 13https://git.io/JvEWO [00:21:40] [02puppet] 07paladox deleted branch 03paladox-patch-9 - 13https://git.io/vbiAS [00:21:42] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-9 [00:30:34] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Jv7sg [00:30:36] [02miraheze/puppet] 07paladox 03494def9 - nutcracker: Pool in rdb2 [00:44:18] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Jv7GY [00:44:20] [02miraheze/puppet] 07paladox 03e9bc15c - Revert "nutcracker: Pool in rdb2" This reverts commit 494def9c5afd2e9f416d0b5856be8d7677c24c3a. [00:54:44] [02miraheze/dns] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Jv7GD [00:54:46] [02miraheze/dns] 07paladox 03194237e - Remove bacula1 [00:55:55] [02miraheze/dns] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Jv7Gy [00:55:57] [02miraheze/dns] 07paladox 038a139a0 - Update miraheze.org [00:59:38] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Jv7G5 [00:59:39] [02miraheze/puppet] 07paladox 03951c38a - services: Change restbase systemd script [01:00:12] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Jv7Gb [01:00:14] [02miraheze/puppet] 07paladox 0312d91da - Services: Change citoid systemd script [01:00:31] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Jv7Gx [01:00:32] [02miraheze/puppet] 07paladox 03e553f57 - Zotero: Change systemd script [01:04:09] PROBLEM - ns1 Puppet on ns1 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[ufw-allow-udp-from-any-to-any-port-53] [01:04:23] !log apt-get dist-upgrade - services [01:04:30] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [01:06:49] !log apt-get dist-upgrade - services2 [01:06:55] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [01:12:09] RECOVERY - ns1 Puppet on ns1 is OK: OK: Puppet is currently enabled, last run 31 seconds ago with 0 failures [02:25:08] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Jv7Cv [02:25:09] [02miraheze/services] 07MirahezeSSLBot 037ce3081 - BOT: Updating services config for wikis [03:33:22] PROBLEM - cp3 Current Load on cp3 is WARNING: WARNING - load average: 1.75, 1.52, 1.38 [03:37:22] RECOVERY - cp3 Current Load on cp3 is OK: OK - load average: 1.36, 1.52, 1.43 [03:43:23] PROBLEM - cp3 Current Load on cp3 is CRITICAL: CRITICAL - load average: 2.02, 1.82, 1.57 [03:47:22] PROBLEM - cp3 Current Load on cp3 is WARNING: WARNING - load average: 1.64, 1.87, 1.65 [03:49:22] RECOVERY - cp3 Current Load on cp3 is OK: OK - load average: 1.37, 1.69, 1.61 [06:26:23] RECOVERY - cp3 Disk Space on cp3 is OK: DISK OK - free space: / 2669 MB (11% inode=93%); [06:57:14] PROBLEM - cp3 Current Load on cp3 is CRITICAL: CRITICAL - load average: 2.04, 1.70, 1.50 [07:01:09] PROBLEM - cp3 Current Load on cp3 is WARNING: WARNING - load average: 1.81, 1.74, 1.56 [07:03:06] RECOVERY - cp3 Current Load on cp3 is OK: OK - load average: 1.49, 1.69, 1.56 [08:02:23] PROBLEM - cp3 Current Load on cp3 is WARNING: WARNING - load average: 1.90, 1.63, 1.46 [08:04:21] RECOVERY - cp3 Current Load on cp3 is OK: OK - load average: 1.61, 1.58, 1.46 [08:52:19] PROBLEM - cp3 Current Load on cp3 is CRITICAL: CRITICAL - load average: 2.02, 1.86, 1.65 [08:54:17] PROBLEM - cp3 Current Load on cp3 is WARNING: WARNING - load average: 1.65, 1.75, 1.63 [08:58:12] RECOVERY - cp3 Current Load on cp3 is OK: OK - load average: 1.68, 1.69, 1.64 [09:04:11] PROBLEM - cp3 Current Load on cp3 is WARNING: WARNING - load average: 1.87, 1.76, 1.68 [09:04:49] Hello Not-1447! If you have any questions, feel free to ask and someone should answer soon. [09:04:50] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-5 [+0/-0/±1] 13https://git.io/Jv7z6 [09:04:51] [02miraheze/puppet] 07paladox 032ff8744 - Introduce php::fpm::memory_limit [09:04:52] [02puppet] 07paladox created branch 03paladox-patch-5 - 13https://git.io/vbiAS [09:04:54] [02puppet] 07paladox opened pull request 03#1308: Introduce php::fpm::memory_limit - 13https://git.io/Jv7zi [09:05:56] we should probably block Not-* from welcome [09:06:00] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-5 [+0/-0/±1] 13https://git.io/Jv7z1 [09:06:02] [02miraheze/puppet] 07paladox 035f880d5 - Update init.pp [09:06:03] [02puppet] 07paladox synchronize pull request 03#1308: Introduce php::fpm::memory_limit - 13https://git.io/Jv7zi [09:06:11] RECOVERY - cp3 Current Load on cp3 is OK: OK - load average: 1.53, 1.68, 1.66 [09:06:51] Hello RH9! If you have any questions, feel free to ask and someone should answer soon. [09:06:56] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-5 [+0/-0/±1] 13https://git.io/Jv7zS [09:06:57] [02miraheze/puppet] 07paladox 03b90715c - Update mon1.yaml [09:06:59] [02puppet] 07paladox synchronize pull request 03#1308: Introduce php::fpm::memory_limit - 13https://git.io/Jv7zi [09:07:05] [02puppet] 07paladox closed pull request 03#1308: Introduce php::fpm::memory_limit - 13https://git.io/Jv7zi [09:07:07] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±3] 13https://git.io/Jv7zQ [09:07:08] [02miraheze/puppet] 07paladox 03562b8ec - Introduce php::fpm::memory_limit (#1308) * Introduce php::fpm::memory_limit * Update init.pp * Update mon1.yaml [09:07:10] [02puppet] 07paladox deleted branch 03paladox-patch-5 - 13https://git.io/vbiAS [09:07:11] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-5 [10:33:30] PROBLEM - cp3 Current Load on cp3 is WARNING: WARNING - load average: 1.73, 1.58, 1.46 [10:37:25] RECOVERY - cp3 Current Load on cp3 is OK: OK - load average: 1.36, 1.54, 1.48 [12:01:22] PROBLEM - cp8 Disk Space on cp8 is WARNING: DISK WARNING - free space: / 2114 MB (10% inode=93%); [12:13:23] PROBLEM - cp3 Current Load on cp3 is WARNING: WARNING - load average: 1.77, 1.64, 1.47 [12:17:22] RECOVERY - cp3 Current Load on cp3 is OK: OK - load average: 1.43, 1.59, 1.48 [12:21:23] PROBLEM - cp3 Current Load on cp3 is WARNING: WARNING - load average: 2.00, 1.93, 1.66 [12:25:22] PROBLEM - cp3 Current Load on cp3 is CRITICAL: CRITICAL - load average: 2.26, 1.94, 1.71 [12:26:21] afternoon JohnLewis [12:27:22] PROBLEM - cp3 Current Load on cp3 is WARNING: WARNING - load average: 1.64, 1.84, 1.71 [12:29:06] Afternoon [12:29:22] RECOVERY - cp3 Current Load on cp3 is OK: OK - load average: 1.38, 1.65, 1.65 [12:33:24] PROBLEM - cp3 Current Load on cp3 is WARNING: WARNING - load average: 1.81, 1.80, 1.72 [12:37:22] RECOVERY - cp3 Current Load on cp3 is OK: OK - load average: 1.12, 1.52, 1.63 [12:42:19] JohnLewis: how's week 2 of lockdown looking? [12:48:10] Still working [13:13:22] PROBLEM - cp3 Current Load on cp3 is CRITICAL: CRITICAL - load average: 2.04, 1.65, 1.49 [13:15:22] PROBLEM - cp3 Current Load on cp3 is WARNING: WARNING - load average: 2.00, 1.83, 1.58 [13:17:22] RECOVERY - cp3 Current Load on cp3 is OK: OK - load average: 1.22, 1.58, 1.51 [14:13:48] PROBLEM - cp3 Disk Space on cp3 is WARNING: DISK WARNING - free space: / 2644 MB (10% inode=93%); [14:26:21] PROBLEM - cp6 Disk Space on cp6 is WARNING: DISK WARNING - free space: / 2079 MB (10% inode=93%); [15:10:27] PROBLEM - misc1 Disk Space on misc1 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [15:11:27] PROBLEM - cp7 HTTP 4xx/5xx ERROR Rate on cp7 is WARNING: WARNING - NGINX Error Rate is 42% [15:12:50] RECOVERY - misc1 Disk Space on misc1 is OK: DISK OK - free space: / 34448 MB (84% inode=98%); [15:12:55] PROBLEM - ping4 on ns1 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 103.35 ms [15:13:25] RECOVERY - cp7 HTTP 4xx/5xx ERROR Rate on cp7 is OK: OK - NGINX Error Rate is 9% [15:14:53] PROBLEM - ping4 on ns1 is WARNING: PING WARNING - Packet loss = 0%, RTA = 102.42 ms [15:38:57] PROBLEM - mw6 MediaWiki Rendering on mw6 is CRITICAL: HTTP CRITICAL: HTTP/1.1 503 Backend fetch failed - 4225 bytes in 0.024 second response time [15:39:18] PROBLEM - cp3 Stunnel Http for mw7 on cp3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [15:39:33] PROBLEM - cp7 Stunnel Http for mw7 on cp7 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [15:39:48] PROBLEM - cp3 Stunnel Http for mw4 on cp3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [15:39:52] PROBLEM - cp6 Stunnel Http for mw4 on cp6 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [15:39:55] PROBLEM - cp3 Stunnel Http for mw6 on cp3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [15:40:03] PROBLEM - cp3 Stunnel Http for mw5 on cp3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [15:40:14] PROBLEM - cp8 HTTP 4xx/5xx ERROR Rate on cp8 is CRITICAL: CRITICAL - NGINX Error Rate is 83% [15:40:16] PROBLEM - cp8 Stunnel Http for mw6 on cp8 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [15:40:22] PROBLEM - cp6 Stunnel Http for mw5 on cp6 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [15:40:31] PROBLEM - cp7 Stunnel Http for mw5 on cp7 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [15:40:31] PROBLEM - jobrunner1 MediaWiki Rendering on jobrunner1 is CRITICAL: HTTP CRITICAL: HTTP/1.1 503 Backend fetch failed - 4227 bytes in 7.216 second response time [15:40:31] PROBLEM - cp7 Varnish Backends on cp7 is CRITICAL: 4 backends are down. mw4 mw5 mw6 mw7 [15:40:32] PROBLEM - ns1 Auth DNS on ns1 is CRITICAL: CRITICAL - Plugin timed out while executing system call [15:40:36] PROBLEM - mw4 MediaWiki Rendering on mw4 is CRITICAL: HTTP CRITICAL: HTTP/1.1 503 Backend fetch failed - 4227 bytes in 1.047 second response time [15:40:36] PROBLEM - mw5 MediaWiki Rendering on mw5 is CRITICAL: HTTP CRITICAL: HTTP/1.1 503 Backend fetch failed - 4227 bytes in 0.025 second response time [15:40:46] Uh oh [15:40:51] PROBLEM - misc1 SMTP on misc1 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:40:52] PROBLEM - phab1 phabricator.miraheze.org HTTPS on phab1 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:40:53] PROBLEM - mw7 MediaWiki Rendering on mw7 is CRITICAL: HTTP CRITICAL: HTTP/1.1 503 Backend fetch failed - 4227 bytes in 5.064 second response time [15:41:02] PROBLEM - mon1 icinga.miraheze.org HTTPS on mon1 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 295 bytes in 0.004 second response time [15:41:03] PROBLEM - cp7 HTTP 4xx/5xx ERROR Rate on cp7 is CRITICAL: CRITICAL - NGINX Error Rate is 65% [15:41:04] PROBLEM - cp6 Stunnel Http for mw6 on cp6 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [15:41:04] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 6 datacenters are down: 128.199.139.216/cpweb, 2400:6180:0:d0::403:f001/cpweb, 51.77.107.210/cpweb, 2001:41d0:800:1056::2/cpweb, 51.161.32.127/cpweb, 2607:5300:205:200::17f6/cpweb [15:41:13] PROBLEM - cp8 Stunnel Http for mw7 on cp8 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [15:41:13] paladox, JohnLewis: ^ [15:41:22] PROBLEM - cp6 Stunnel Http for mw7 on cp6 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [15:41:23] PROBLEM - cp8 Stunnel Http for mw5 on cp8 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [15:41:25] hmm [15:41:28] Well, then [15:41:46] PROBLEM - test2 MediaWiki Rendering on test2 is CRITICAL: HTTP CRITICAL: HTTP/1.1 503 Backend fetch failed - 4225 bytes in 6.066 second response time [15:41:52] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [15:41:55] PROBLEM - cp7 Stunnel Http for mw6 on cp7 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [15:41:59] paladox: everything is down afaics. [15:42:05] PROBLEM - cp7 Stunnel Http for mw4 on cp7 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [15:42:06] PROBLEM - cp3 Varnish Backends on cp3 is CRITICAL: 4 backends are down. mw4 mw5 mw6 mw7 [15:42:07] PROBLEM - cp8 Stunnel Http for mw4 on cp8 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [15:42:17] PROBLEM - phab1 phab.miraheze.wiki HTTPS on phab1 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:42:17] PROBLEM - cp6 Varnish Backends on cp6 is CRITICAL: 4 backends are down. mw4 mw5 mw6 mw7 [15:42:21] oh [15:42:28] PROBLEM - cp3 HTTP 4xx/5xx ERROR Rate on cp3 is CRITICAL: CRITICAL - NGINX Error Rate is 88% [15:42:30] PROBLEM - cp6 HTTP 4xx/5xx ERROR Rate on cp6 is CRITICAL: CRITICAL - NGINX Error Rate is 77% [15:42:35] PROBLEM - cp8 Varnish Backends on cp8 is CRITICAL: 4 backends are down. mw4 mw5 mw6 mw7 [15:43:10] QuIRC wiki is 503: Backend fetch failed. [15:43:47] quirc: everything is down [15:44:07] well i see why [15:44:15] RECOVERY - cp7 Stunnel Http for mw4 on cp7 is OK: HTTP OK: HTTP/1.1 200 OK - 15316 bytes in 0.004 second response time [15:44:16] RECOVERY - cp8 Stunnel Http for mw4 on cp8 is OK: HTTP OK: HTTP/1.1 200 OK - 15316 bytes in 0.307 second response time [15:44:19] RECOVERY - cp3 Stunnel Http for mw7 on cp3 is OK: HTTP OK: HTTP/1.1 200 OK - 15330 bytes in 0.697 second response time [15:44:31] RECOVERY - cp7 Stunnel Http for mw7 on cp7 is OK: HTTP OK: HTTP/1.1 200 OK - 15324 bytes in 0.003 second response time [15:44:32] PROBLEM - cp6 HTTP 4xx/5xx ERROR Rate on cp6 is WARNING: WARNING - NGINX Error Rate is 50% [15:44:35] RECOVERY - cp6 Stunnel Http for mw4 on cp6 is OK: HTTP OK: HTTP/1.1 200 OK - 15330 bytes in 0.003 second response time [15:44:37] RECOVERY - cp3 Stunnel Http for mw4 on cp3 is OK: HTTP OK: HTTP/1.1 200 OK - 15316 bytes in 0.701 second response time [15:44:38] RECOVERY - cp3 Stunnel Http for mw5 on cp3 is OK: HTTP OK: HTTP/1.1 200 OK - 15316 bytes in 0.736 second response time [15:44:42] PROBLEM - cp8 HTTP 4xx/5xx ERROR Rate on cp8 is WARNING: WARNING - NGINX Error Rate is 57% [15:44:50] RECOVERY - cp6 Stunnel Http for mw5 on cp6 is OK: HTTP OK: HTTP/1.1 200 OK - 15316 bytes in 0.004 second response time [15:44:56] and its back [15:45:02] RECOVERY - cp7 Stunnel Http for mw5 on cp7 is OK: HTTP OK: HTTP/1.1 200 OK - 15330 bytes in 0.003 second response time [15:45:06] PROBLEM - phab1 Puppet on phab1 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_clone_phabricator-extensions] [15:45:17] Yay! [15:45:18] paladox: what happened? [15:45:21] oh [15:45:23] no [15:45:38] PROBLEM - ns2 Puppet on ns2 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_dns] [15:45:38] RECOVERY - cp6 Stunnel Http for mw6 on cp6 is OK: HTTP OK: HTTP/1.1 200 OK - 15330 bytes in 0.003 second response time [15:45:42] RECOVERY - cp8 Stunnel Http for mw7 on cp8 is OK: HTTP OK: HTTP/1.1 200 OK - 15324 bytes in 0.320 second response time [15:45:44] RECOVERY - cp6 Stunnel Http for mw7 on cp6 is OK: HTTP OK: HTTP/1.1 200 OK - 15324 bytes in 0.004 second response time [15:45:47] RECOVERY - cp8 Stunnel Http for mw5 on cp8 is OK: HTTP OK: HTTP/1.1 200 OK - 15316 bytes in 0.320 second response time [15:45:51] PROBLEM - misc1 IMAP on misc1 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:46:06] network is down [15:46:24] RECOVERY - cp7 Stunnel Http for mw6 on cp7 is OK: HTTP OK: HTTP/1.1 200 OK - 15324 bytes in 0.005 second response time [15:46:32] PROBLEM - mw5 Puppet on mw5 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki config] [15:46:35] paladox: update topic pls [15:46:36] PROBLEM - cp6 HTTP 4xx/5xx ERROR Rate on cp6 is CRITICAL: CRITICAL - NGINX Error Rate is 74% [15:46:48] PROBLEM - cp8 HTTP 4xx/5xx ERROR Rate on cp8 is CRITICAL: CRITICAL - NGINX Error Rate is 69% [15:46:51] RECOVERY - cp3 Stunnel Http for mw6 on cp3 is OK: HTTP OK: HTTP/1.1 200 OK - 15330 bytes in 0.696 second response time [15:46:55] PROBLEM - jobrunner1 Puppet on jobrunner1 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_landing] [15:47:14] PROBLEM - misc1 Current Load on misc1 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [15:47:15] RECOVERY - cp8 Stunnel Http for mw6 on cp8 is OK: HTTP OK: HTTP/1.1 200 OK - 15324 bytes in 0.313 second response time [15:47:55] PROBLEM - ns1 SSH on ns1 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:48:11] RECOVERY - misc1 IMAP on misc1 is OK: IMAP OK - 0.045 second response time on 185.52.1.76 port 143 [* OK [CAPABILITY IMAP4rev1 LITERAL+ SASL-IR LOGIN-REFERRALS ID ENABLE IDLE STARTTLS LOGINDISABLED] Dovecot ready.] [15:48:53] PROBLEM - ns1 Disk Space on ns1 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [15:49:01] PROBLEM - mw6 Puppet on mw6 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki config] [15:49:06] PROBLEM - ns1 Puppet on ns1 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [15:49:53] PROBLEM - cp3 Stunnel Http for mw5 on cp3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [15:50:13] PROBLEM - cp6 Stunnel Http for mw5 on cp6 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [15:50:20] PROBLEM - cp7 Stunnel Http for mw5 on cp7 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [15:50:31] RECOVERY - ns1 SSH on ns1 is OK: SSH OK - OpenSSH_7.9p1 Debian-10+deb10u2 (protocol 2.0) [15:51:05] PROBLEM - misc1 HTTPS on misc1 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:51:16] Hello dreamcast99! If you have any questions, feel free to ask and someone should answer soon. [15:51:29] PROBLEM - cp8 Stunnel Http for mw7 on cp8 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [15:51:37] PROBLEM - cp6 HTTP 4xx/5xx ERROR Rate on cp6 is WARNING: WARNING - NGINX Error Rate is 42% [15:51:39] PROBLEM - cp8 Stunnel Http for mw5 on cp8 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [15:51:44] Hey dreamcast99 [15:51:47] hey [15:51:48] RECOVERY - ns1 Puppet on ns1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [15:52:18] PROBLEM - cp3 Stunnel Http for mw7 on cp3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [15:52:18] RECOVERY - misc1 Current Load on misc1 is OK: OK - load average: 0.04, 0.06, 0.05 [15:52:20] RECOVERY - cp3 Stunnel Http for mw5 on cp3 is OK: HTTP OK: HTTP/1.1 200 OK - 15316 bytes in 0.703 second response time [15:52:30] PROBLEM - cp7 Stunnel Http for mw7 on cp7 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [15:52:39] RECOVERY - cp6 Stunnel Http for mw5 on cp6 is OK: HTTP OK: HTTP/1.1 200 OK - 15316 bytes in 0.003 second response time [15:52:42] RECOVERY - cp7 Stunnel Http for mw5 on cp7 is OK: HTTP OK: HTTP/1.1 200 OK - 15316 bytes in 0.004 second response time [15:52:49] RECOVERY - ns2 Puppet on ns2 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [15:52:52] PROBLEM - cp3 Stunnel Http for mw6 on cp3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [15:53:24] PROBLEM - cp8 Stunnel Http for mw6 on cp8 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [15:53:24] RECOVERY - misc1 HTTPS on misc1 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 427 bytes in 0.087 second response time [15:53:43] RECOVERY - cp8 Stunnel Http for mw7 on cp8 is OK: HTTP OK: HTTP/1.1 200 OK - 15316 bytes in 1.465 second response time [15:53:46] PROBLEM - cp6 HTTP 4xx/5xx ERROR Rate on cp6 is CRITICAL: CRITICAL - NGINX Error Rate is 83% [15:53:46] RECOVERY - cp8 Stunnel Http for mw5 on cp8 is OK: HTTP OK: HTTP/1.1 200 OK - 15316 bytes in 0.333 second response time [15:53:46] RECOVERY - ns1 Disk Space on ns1 is OK: DISK OK - free space: / 11309 MB (88% inode=92%); [15:53:50] PROBLEM - cp6 Stunnel Http for mw6 on cp6 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [15:54:16] PROBLEM - cp8 Stunnel Http for mw4 on cp8 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [15:54:16] PROBLEM - cp7 Stunnel Http for mw6 on cp7 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [15:54:26] thanks RhinosF1 for the pm [15:54:32] RECOVERY - cp3 Stunnel Http for mw7 on cp3 is OK: HTTP OK: HTTP/1.1 200 OK - 15316 bytes in 0.703 second response time [15:54:34] PROBLEM - cp6 Stunnel Http for mw4 on cp6 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [15:54:35] PROBLEM - cp3 Puppet on cp3 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [15:54:47] paladox: that’ll be why [15:54:48] RECOVERY - cp7 Stunnel Http for mw7 on cp7 is OK: HTTP OK: HTTP/1.1 200 OK - 15330 bytes in 0.002 second response time [15:55:04] PROBLEM - cp3 Stunnel Http for mw4 on cp3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [15:55:58] who's the provider? [15:56:13] OVH [15:56:16] PROBLEM - cp3 HTTP 4xx/5xx ERROR Rate on cp3 is WARNING: WARNING - NGINX Error Rate is 48% [15:56:23] RECOVERY - cp8 Stunnel Http for mw4 on cp8 is OK: HTTP OK: HTTP/1.1 200 OK - 15316 bytes in 0.320 second response time [15:56:41] RECOVERY - cp6 Stunnel Http for mw4 on cp6 is OK: HTTP OK: HTTP/1.1 200 OK - 15316 bytes in 0.004 second response time [15:56:46] RECOVERY - ns1 Auth DNS on ns1 is OK: DNS OK: 0.214 seconds response time. miraheze.org returns 2001:41d0:800:1056::2,51.77.107.210 [15:56:53] paladox: update topic pls [15:57:20] RECOVERY - cp3 Stunnel Http for mw4 on cp3 is OK: HTTP OK: HTTP/1.1 200 OK - 15316 bytes in 0.703 second response time [15:57:20] PROBLEM - misc1 Current Load on misc1 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [15:57:31] RECOVERY - cp3 Stunnel Http for mw6 on cp3 is OK: HTTP OK: HTTP/1.1 200 OK - 15330 bytes in 6.317 second response time [15:57:33] PROBLEM - misc1 Disk Space on misc1 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [15:57:58] RECOVERY - cp8 Stunnel Http for mw6 on cp8 is OK: HTTP OK: HTTP/1.1 200 OK - 15324 bytes in 0.327 second response time [15:58:37] RECOVERY - cp6 Stunnel Http for mw6 on cp6 is OK: HTTP OK: HTTP/1.1 200 OK - 15324 bytes in 0.003 second response time [15:58:43] PROBLEM - cp3 HTTP 4xx/5xx ERROR Rate on cp3 is CRITICAL: CRITICAL - NGINX Error Rate is 88% [15:58:59] RECOVERY - cp7 Stunnel Http for mw6 on cp7 is OK: HTTP OK: HTTP/1.1 200 OK - 15316 bytes in 0.003 second response time [15:59:20] PROBLEM - db6 Puppet on db6 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [16:00:02] RECOVERY - misc1 Disk Space on misc1 is OK: DISK OK - free space: / 34446 MB (84% inode=98%); [16:01:28] Thank you [16:09:51] any word from OVH [16:10:49] They are still investigating [16:14:36] dreamcast99: status page is down [16:14:56] What status page? OVH's? [16:22:13] RECOVERY - mw7 Puppet on mw7 is OK: OK: Puppet is currently enabled, last run 9 seconds ago with 0 failures [16:22:25] RECOVERY - mw5 Puppet on mw5 is OK: OK: Puppet is currently enabled, last run 21 seconds ago with 0 failures [16:22:28] Things are restoring, OVH have confirmed it is a physical network issue [16:23:58] RECOVERY - mw4 Puppet on mw4 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [16:30:43] As of 5 mins ago: Services are gradually recovering from the isolation of a faulty network infrastructure. [16:30:43] We continue to ensure that services are restored [16:32:28] [02miraheze/IncidentReporting] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Jv7Ho [16:32:29] [02miraheze/IncidentReporting] 07paladox 0372697ff - Fix typo [16:33:13] [02miraheze/IncidentReporting] 07paladox pushed 031 commit to 03paladox-patch-2 [+0/-0/±1] 13https://git.io/Jv7H6 [16:33:14] [02miraheze/IncidentReporting] 07paladox 0399b59ad - Bump version to 1.1.3 Bug fix [16:33:16] [02IncidentReporting] 07paladox created branch 03paladox-patch-2 - 13https://git.io/fh5YJ [16:33:17] [02IncidentReporting] 07paladox opened pull request 03#11: Bump version to 1.1.3 - 13https://git.io/Jv7HP [16:33:35] paladox: Captial U [16:33:42] [02miraheze/IncidentReporting] 07paladox pushed 031 commit to 03paladox-patch-2 [+0/-0/±1] 13https://git.io/Jv7HX [16:33:44] [02miraheze/IncidentReporting] 07paladox 03b719aea - Update CHANGELOG [16:33:45] [02IncidentReporting] 07paladox synchronize pull request 03#11: Bump version to 1.1.3 - 13https://git.io/Jv7HP [16:33:58] Where? [16:34:17] [02IncidentReporting] 07paladox closed pull request 03#11: Bump version to 1.1.3 - 13https://git.io/Jv7HP [16:34:18] [02miraheze/IncidentReporting] 07paladox pushed 033 commits to 03master [+0/-0/±4] 13https://git.io/Jv7Hy [16:34:20] [02miraheze/IncidentReporting] 07paladox 03d66dfc8 - Merge pull request #11 from miraheze/paladox-patch-2 Bump version to 1.1.3 [16:34:21] [02miraheze/IncidentReporting] 07paladox deleted branch 03paladox-patch-2 [16:34:23] [02IncidentReporting] 07paladox deleted branch 03paladox-patch-2 - 13https://git.io/fh5YJ [16:34:26] paladox: In the topic [16:34:47] [02miraheze/mediawiki] 07paladox pushed 031 commit to 03REL1_34 [+0/-0/±1] 13https://git.io/Jv7HH [16:34:48] [02miraheze/mediawiki] 07paladox 0393f5877 - Update IncidentReporting [16:35:18] miraheze/IncidentReporting/paladox-patch-2/99b59ad - paladox The build has errored. https://travis-ci.com/miraheze/IncidentReporting/builds/156699695 [16:36:28] miraheze/IncidentReporting/paladox-patch-2/b719aea - paladox The build has errored. https://travis-ci.com/miraheze/IncidentReporting/builds/156699789 [16:36:31] paladox: ^ [16:36:42] Looks like it’s checking your deleted branch? [16:36:45] yes [16:37:10] [02miraheze/IncidentReporting] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Jv7HN [16:37:12] [02miraheze/IncidentReporting] 07paladox 03723a84f - Fix typo part 2 [16:37:17] Stupid travis [16:37:25] JohnLewis do i need to bump the version again? [16:37:36] no [16:38:07] [02miraheze/mediawiki] 07paladox pushed 031 commit to 03REL1_34 [+0/-0/±1] 13https://git.io/Jv7Hp [16:38:09] [02miraheze/mediawiki] 07paladox 039ec58dd - Update IncidentReporting [16:38:47] thanks! [17:11:15] PROBLEM - cp8 Current Load on cp8 is CRITICAL: CRITICAL - load average: 3.48, 2.31, 1.48 [17:19:03] PROBLEM - cp8 Current Load on cp8 is WARNING: WARNING - load average: 1.25, 1.88, 1.67 [17:22:58] PROBLEM - cp8 Current Load on cp8 is CRITICAL: CRITICAL - load average: 2.67, 2.05, 1.77 [17:31:34] PROBLEM - cp3 Current Load on cp3 is WARNING: WARNING - load average: 1.72, 1.58, 1.44 [17:33:31] RECOVERY - cp3 Current Load on cp3 is OK: OK - load average: 1.57, 1.60, 1.46 [17:42:51] PROBLEM - cp8 Current Load on cp8 is WARNING: WARNING - load average: 1.13, 1.85, 2.00 [17:48:51] RECOVERY - cp8 Current Load on cp8 is OK: OK - load average: 0.87, 1.26, 1.69 [17:52:51] PROBLEM - cp8 Current Load on cp8 is WARNING: WARNING - load average: 1.51, 1.58, 1.72 [17:54:51] RECOVERY - cp8 Current Load on cp8 is OK: OK - load average: 0.78, 1.26, 1.59 [18:15:09] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Jv7dR [18:15:11] [02miraheze/services] 07MirahezeSSLBot 030184b78 - BOT: Updating services config for wikis [20:06:25] [02miraheze/puppet] 07Reception123 pushed 031 commit to 03Reception123-patch-1 [+0/-23/±8] 13https://git.io/Jv7N9 [20:06:27] [02miraheze/puppet] 07Reception123 03dbf773e - remove some old infra config [20:06:28] [02puppet] 07Reception123 created branch 03Reception123-patch-1 - 13https://git.io/vbiAS [20:06:56] [02puppet] 07Reception123 opened pull request 03#1309: remove some old infra config - 13https://git.io/Jv7N7 [20:07:02] ^ paladox [20:08:18] yes [20:08:40] please review and merge if you can [20:09:23] [02puppet] 07paladox reviewed pull request 03#1309 commit - 13https://git.io/Jv7NN [20:10:07] [02miraheze/puppet] 07Reception123 pushed 031 commit to 03Reception123-patch-2 [+0/-14/±1] 13https://git.io/Jv7Np [20:10:09] [02miraheze/puppet] 07Reception123 03dfec265 - remove lizardfs [20:10:10] [02puppet] 07Reception123 created branch 03Reception123-patch-2 - 13https://git.io/vbiAS [20:10:33] [02puppet] 07Reception123 opened pull request 03#1310: remove lizardfs - 13https://git.io/Jv7Nh [20:11:37] [02miraheze/CreateWiki] 07JohnFLewis pushed 031 commit to 03master [+1/-0/±1] 13https://git.io/Jv7Ae [20:11:38] [02miraheze/CreateWiki] 07JohnFLewis 037de85e4 - add JsonBuilder code for new caching backend [20:11:43] [02miraheze/ManageWiki] 07JohnFLewis pushed 031 commit to 03master [+0/-0/±2] 13https://git.io/Jv7Av [20:11:45] [02miraheze/ManageWiki] 07JohnFLewis 03067fb86 - add JsonBuilder code for Hooks for CreateWiki [20:12:05] paladox / Reception123 ^ fancy giving those a quick review please? [20:12:39] [02miraheze/puppet] 07Reception123 pushed 031 commit to 03Reception123-patch-1 [+0/-0/±1] 13https://git.io/Jv7AU [20:12:41] [02miraheze/puppet] 07Reception123 0331607b5 - change per paladox [20:12:42] [02puppet] 07Reception123 synchronize pull request 03#1309: remove some old infra config - 13https://git.io/Jv7N7 [20:12:45] paladox: fixed :) [20:12:49] JohnLewis: sure :D [20:13:07] i'll check in a sec [20:18:02] not the best person to ask, but LGTM (mostly had a look to check for typos and didn't find anyway, those always get us) [20:18:07] also when you're done with all this, maybe you could take a look at getting dbcluster to work with createwiki rather than having the config file :) though it's not that important considering the other tasks [20:21:39] Reception123: this work is required for that [20:21:55] That's why it's taken so long to be able to implement it and why I stopped half way months ago [20:22:31] This would move configuration to being done before MediaWiki loads, while currently it's done post MediaWiki so things like database server changes don't work [20:22:32] PROBLEM - cp8 Disk Space on cp8 is CRITICAL: DISK CRITICAL - free space: / 1147 MB (5% inode=93%); [20:22:53] JohnLewis why not use MediaWikiServices to get the config? :) [20:22:56] and if paladox says that looks okay, I can potentially get https://phabricator.miraheze.org/T4133 and https://phabricator.miraheze.org/T4132 closed today [20:22:58] [ ⚓ T4133 Remove wiki_extensions and wiki_settings (and maybe more?) from CreateWiki ] - phabricator.miraheze.org [20:22:58] [ ⚓ T4132 Convert wiki_extensions to use JSON ] - phabricator.miraheze.org [20:23:06] JohnLewis: oh when I saw it in the code I imagined it was somewhat related and that reminded me of it :) [20:23:08] paladox: because I don't want to have the extension in a half-half state [20:23:15] ok [20:24:37] JohnLewis: so then if you get https://phabricator.miraheze.org/T4133 done, if I needed to manually edit something in MW I'd need to edit the file right? (And for now the only instance where that's needed is custom domains, because if they're invalidated the settings obviously can't be edited via the interface) [20:24:38] [ ⚓ T4133 Remove wiki_extensions and wiki_settings (and maybe more?) from CreateWiki ] - phabricator.miraheze.org [20:24:54] no more db insertions I hope :) [20:25:15] JohnLewis https://github.com/miraheze/CreateWiki/compare/984823406259...7de85e4837dd#diff-489411c805a2d89ff5ae823eb6b490a8R90 should probably create a temporary file then copy it over? Like we already do? [20:25:16] [ Comparing 984823406259...7de85e4837dd · miraheze/CreateWiki · GitHub ] - github.com [20:26:01] paladox: the event would be such a rare occurrence, I'm still not convinced it's a real race condition [20:26:30] ok, if you think we won't experence the same race condition you can ignore my comment :) [20:26:39] "same" is the issue [20:27:01] As my comment directly said "I don't believe it existed originally" [20:27:17] Reception123: editing the cache files would really not be recommended [20:27:28] JohnLewis should we add a if (!.....) check here https://github.com/miraheze/ManageWiki/compare/83ccb2c51fa1...067fb86b5105#diff-6379dbe476c71358e01325ed674fbc21R46 ? [20:27:28] [ Comparing 83ccb2c51fa1...067fb86b5105 · miraheze/ManageWiki · GitHub ] - github.com [20:27:37] JohnLewis: well then how would I be able to deal with that case? [20:27:47] paladox: for? [20:27:57] JohnLewis for setObject [20:28:01] * $setObject [20:28:02] there's no other way to do it since if a domain is set to another server the wiki can't be accessed anymore without first removing the domain from wgServer [20:28:02] oh, I see now [20:28:06] Not for that reason [20:28:09] Should we not move the security issue reporting from readme.md to security.md so it follows normal expectation and populated in the GitHub security tab? [20:28:38] As the non-existence of those values while the module is enable is more severe than that [20:29:02] PROBLEM - bacula2 Bacula Phabricator Static on bacula2 is WARNING: WARNING: Diff, 5266 files, 19.16MB, 2020-03-15 20:26:00 (2.1 weeks ago) [20:29:08] Reception123: Same as now [20:29:41] Though with changes to how the whole system works, RemoteWiki and WikiManager will be get changes to be more complete tools [20:29:49] ah ok [20:29:54] Guess I'll wait and see what changes then [20:31:46] JohnLewis looks fine [20:34:59] [02miraheze/ManageWiki] 07JohnFLewis pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Jv7xv [20:35:01] [02miraheze/ManageWiki] 07JohnFLewis 03af5864e - add module checks for Hooks [21:18:46] !log created dummy wiki dbs to prevent conflict with services/servers (mon1, icinga, grafana, dbt1) [21:18:52] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [21:34:34] !log drop dummy wiki dbs for old infra (DONE: mw[123], db[12345] lizardfs[12345] cp[1245]) [21:34:44] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [22:41:07] can the autoconfirmed be "removed/delete" or will it cause an error? [22:41:37] * hispano76 you prefer to ask before you make mistakes [22:41:47] PROBLEM - cp8 Current Load on cp8 is CRITICAL: CRITICAL - load average: 1.62, 2.20, 1.69 [22:42:34] on ManageWiki* [22:43:43] PROBLEM - cp8 Current Load on cp8 is WARNING: WARNING - load average: 1.00, 1.75, 1.58 [22:45:40] RECOVERY - cp8 Current Load on cp8 is OK: OK - load average: 0.36, 1.28, 1.43 [22:48:55] [02mw-config] 07Hispano76 opened pull request 03#2959: reconfigurations of wgNamespaceRobotPolicies on HispanoWiki, Privado… - 13https://git.io/Jv5vE [22:50:44] [02mw-config] 07Pix1234 closed pull request 03#2959: reconfigurations of wgNamespaceRobotPolicies on HispanoWiki, Privado… - 13https://git.io/Jv5vE [22:50:45] [02miraheze/mw-config] 07Pix1234 pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Jv5v6 [22:50:47] [02miraheze/mw-config] 07Hispano76 03ac7e1a5 - reconfigurations of wgNamespaceRobotPolicies on HispanoWiki, PrivadoWiki and UcroniasWiki (#2959) [22:52:25] thanks Zppix [23:14:26] [02mw-config] 07Hispano76 opened pull request 03#2960: Reconfigurations on Hispanowiki, Privadowiki and Ucroniaswiki - 13https://git.io/Jv5f8