[00:23:30] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 7.89, 7.10, 6.44 [00:25:01] PROBLEM - ns1 Puppet on ns1 is CRITICAL: CRITICAL: Puppet has 9 failures. Last run 3 minutes ago with 9 failures. Failed resources (up to 3 shown): Service[ssh],Service[postfix],Exec[ufw-logging-low],Exec[ufw-allow-tcp-from-any-to-any-port-22] [00:25:31] RECOVERY - mw3 Current Load on mw3 is OK: OK - load average: 6.07, 6.61, 6.40 [00:29:23] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 7.25, 7.21, 6.78 [00:33:01] RECOVERY - ns1 Puppet on ns1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [00:35:06] RECOVERY - mw3 Current Load on mw3 is OK: OK - load average: 5.21, 6.45, 6.71 [00:50:33] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 7.83, 7.01, 6.50 [00:52:34] PROBLEM - mw3 Current Load on mw3 is CRITICAL: CRITICAL - load average: 8.90, 7.70, 6.88 [00:54:29] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 5.80, 7.15, 6.88 [00:58:22] RECOVERY - mw3 Current Load on mw3 is OK: OK - load average: 6.66, 6.58, 6.74 [01:13:49] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 7.79, 7.36, 6.77 [01:15:46] RECOVERY - mw3 Current Load on mw3 is OK: OK - load average: 4.94, 6.17, 6.43 [01:35:10] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 7.22, 7.12, 6.56 [01:37:15] RECOVERY - mw3 Current Load on mw3 is OK: OK - load average: 6.17, 6.69, 6.52 [01:42:22] Hello ilby! If you have any questions, feel free to ask and someone should answer soon. [01:42:42] ok [01:42:45] bye [01:55:01] PROBLEM - ns1 Puppet on ns1 is CRITICAL: CRITICAL: Puppet has 12 failures. Last run 3 minutes ago with 12 failures. Failed resources (up to 3 shown): Package[openssh-server],Service[ssh],Package[exim4],Service[postfix] [02:02:20] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 6.93, 6.93, 6.67 [02:03:01] RECOVERY - ns1 Puppet on ns1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [02:06:19] RECOVERY - mw3 Current Load on mw3 is OK: OK - load average: 5.79, 6.36, 6.57 [02:15:04] PROBLEM - ns1 Puppet on ns1 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[ops_ensure_members] [02:16:19] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 6.81, 6.89, 6.66 [02:22:18] PROBLEM - mw3 Current Load on mw3 is CRITICAL: CRITICAL - load average: 8.18, 7.66, 7.15 [02:23:27] PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 1 datacenter is down: 2a00:d880:5:8ea::ebc7/cpweb [02:23:59] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 1 datacenter is down: 128.199.139.216/cpweb [02:24:28] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 6.81, 7.61, 7.32 [02:25:21] PROBLEM - cp3 Stunnel Http for mw2 on cp3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [02:27:20] RECOVERY - cp3 Stunnel Http for mw2 on cp3 is OK: HTTP OK: HTTP/1.1 200 OK - 24662 bytes in 3.504 second response time [02:27:31] RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online [02:28:21] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [02:30:25] PROBLEM - mw3 Current Load on mw3 is CRITICAL: CRITICAL - load average: 8.11, 7.75, 7.53 [02:32:22] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 7.27, 7.66, 7.55 [02:33:01] RECOVERY - ns1 Puppet on ns1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [02:45:05] PROBLEM - ns1 Puppet on ns1 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[ops_ensure_members] [02:47:47] PROBLEM - cp3 Puppet on cp3 is CRITICAL: CRITICAL: Puppet has 119 failures. Last run 4 minutes ago with 119 failures. Failed resources (up to 3 shown): File[sahitya.shaunak.in_private],File[wiki.galactee.org],File[wiki.galactee.org_private],File[www.thesimswiki.com] [02:50:19] PROBLEM - mw3 Current Load on mw3 is CRITICAL: CRITICAL - load average: 8.28, 7.48, 7.32 [02:52:18] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 6.94, 7.43, 7.36 [02:53:01] RECOVERY - ns1 Puppet on ns1 is OK: OK: Puppet is currently enabled, last run 13 seconds ago with 0 failures [02:55:48] RECOVERY - cp3 Puppet on cp3 is OK: OK: Puppet is currently enabled, last run 58 seconds ago with 0 failures [03:06:19] RECOVERY - mw3 Current Load on mw3 is OK: OK - load average: 6.09, 6.12, 6.63 [03:25:03] PROBLEM - ns1 Puppet on ns1 is CRITICAL: CRITICAL: Puppet has 12 failures. Last run 3 minutes ago with 12 failures. Failed resources (up to 3 shown): Package[openssh-server],Service[ssh],Package[exim4],Service[postfix] [03:33:01] RECOVERY - ns1 Puppet on ns1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [03:49:10] PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 2 datacenters are down: 107.191.126.23/cpweb, 128.199.139.216/cpweb [03:49:44] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 7.45, 7.46, 6.73 [03:51:07] RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online [03:57:22] RECOVERY - mw3 Current Load on mw3 is OK: OK - load average: 4.31, 5.44, 6.18 [04:31:06] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 7.50, 7.26, 6.72 [04:34:56] RECOVERY - mw3 Current Load on mw3 is OK: OK - load average: 4.22, 6.13, 6.54 [04:51:19] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 7.16, 6.62, 6.21 [04:55:07] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Je1hO [04:55:08] [02miraheze/services] 07MirahezeSSLBot 037e9e281 - BOT: Updating services config for wikis [04:55:09] RECOVERY - mw3 Current Load on mw3 is OK: OK - load average: 5.94, 6.27, 6.27 [05:17:39] PROBLEM - cp3 Disk Space on cp3 is CRITICAL: DISK CRITICAL - free space: / 1444 MB (5% inode=94%); [05:25:03] PROBLEM - ns1 Puppet on ns1 is CRITICAL: CRITICAL: Puppet has 12 failures. Last run 3 minutes ago with 12 failures. Failed resources (up to 3 shown): Package[openssh-server],Service[ssh],Package[exim4],Service[postfix] [05:53:01] RECOVERY - ns1 Puppet on ns1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [06:12:36] PROBLEM - mw3 Current Load on mw3 is CRITICAL: CRITICAL - load average: 8.08, 6.86, 6.27 [06:14:31] RECOVERY - mw3 Current Load on mw3 is OK: OK - load average: 5.71, 6.35, 6.21 [06:15:01] PROBLEM - ns1 Puppet on ns1 is CRITICAL: CRITICAL: Puppet has 12 failures. Last run 3 minutes ago with 12 failures. Failed resources (up to 3 shown): Package[openssh-server],Service[ssh],Package[exim4],Service[postfix] [06:23:01] RECOVERY - ns1 Puppet on ns1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [06:23:08] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 7.74, 7.35, 6.69 [06:25:39] PROBLEM - cp3 Disk Space on cp3 is WARNING: DISK WARNING - free space: / 2047 MB (8% inode=94%); [06:28:52] RECOVERY - mw3 Current Load on mw3 is OK: OK - load average: 6.02, 6.38, 6.63 [06:35:03] PROBLEM - ns1 Puppet on ns1 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[rsyslog] [06:43:02] RECOVERY - ns1 Puppet on ns1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [06:43:19] PROBLEM - mw3 Current Load on mw3 is CRITICAL: CRITICAL - load average: 8.27, 7.33, 6.84 [06:45:14] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 7.86, 7.25, 6.89 [06:47:09] RECOVERY - mw3 Current Load on mw3 is OK: OK - load average: 5.04, 6.44, 6.66 [07:08:19] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 6.28, 6.82, 6.54 [07:16:18] RECOVERY - mw3 Current Load on mw3 is OK: OK - load average: 5.37, 6.60, 6.77 [07:35:01] PROBLEM - ns1 Puppet on ns1 is CRITICAL: CRITICAL: Puppet has 12 failures. Last run 3 minutes ago with 12 failures. Failed resources (up to 3 shown): Package[openssh-server],Service[ssh],Package[exim4],Service[postfix] [07:43:02] RECOVERY - ns1 Puppet on ns1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [08:05:01] PROBLEM - ns1 Puppet on ns1 is CRITICAL: CRITICAL: Puppet has 8 failures. Last run 3 minutes ago with 8 failures. Failed resources (up to 3 shown): Exec[ufw-logging-low],Exec[ufw-allow-tcp-from-any-to-any-port-22],Exec[ufw-allow-tcp-from-any-to-any-port-5666],Exec[ufw-allow-tcp-from-185.52.3.121-to-any-port-9100] [08:13:01] RECOVERY - ns1 Puppet on ns1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [08:41:59] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 6.73, 7.02, 6.37 [08:43:53] RECOVERY - mw3 Current Load on mw3 is OK: OK - load average: 6.18, 6.75, 6.41 [09:15:01] PROBLEM - ns1 Puppet on ns1 is CRITICAL: CRITICAL: Puppet has 8 failures. Last run 3 minutes ago with 8 failures. Failed resources (up to 3 shown): Exec[ufw-logging-low],Exec[ufw-allow-tcp-from-any-to-any-port-22],Exec[ufw-allow-tcp-from-any-to-any-port-5666],Exec[ufw-allow-tcp-from-185.52.3.121-to-any-port-9100] [09:33:01] RECOVERY - ns1 Puppet on ns1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [09:55:01] PROBLEM - ns1 Puppet on ns1 is CRITICAL: CRITICAL: Puppet has 8 failures. Last run 3 minutes ago with 8 failures. Failed resources (up to 3 shown): Exec[ufw-logging-low],Exec[ufw-allow-tcp-from-any-to-any-port-22],Exec[ufw-allow-tcp-from-any-to-any-port-5666],Exec[ufw-allow-tcp-from-185.52.3.121-to-any-port-9100] [09:56:19] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 7.21, 6.93, 6.24 [10:03:03] RECOVERY - ns1 Puppet on ns1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [10:04:19] RECOVERY - mw3 Current Load on mw3 is OK: OK - load average: 4.62, 5.64, 5.99 [10:12:24] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 7.43, 6.80, 6.19 [10:15:09] PROBLEM - ns1 Puppet on ns1 is CRITICAL: CRITICAL: Puppet has 12 failures. Last run 3 minutes ago with 12 failures. Failed resources (up to 3 shown): Package[openssh-server],Service[ssh],Package[exim4],Service[postfix] [10:18:18] RECOVERY - mw3 Current Load on mw3 is OK: OK - load average: 5.77, 6.27, 6.35 [10:33:01] RECOVERY - ns1 Puppet on ns1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [10:54:59] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 6.87, 7.22, 6.60 [10:58:47] RECOVERY - mw3 Current Load on mw3 is OK: OK - load average: 6.01, 6.46, 6.49 [11:24:51] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 7.21, 7.03, 6.54 [11:28:39] RECOVERY - mw3 Current Load on mw3 is OK: OK - load average: 6.24, 6.60, 6.56 [11:45:03] PROBLEM - ns1 Puppet on ns1 is CRITICAL: CRITICAL: Puppet has 8 failures. Last run 3 minutes ago with 8 failures. Failed resources (up to 3 shown): Exec[ufw-logging-low],Exec[ufw-allow-tcp-from-any-to-any-port-22],Exec[ufw-allow-tcp-from-any-to-any-port-5666],Exec[ufw-allow-tcp-from-185.52.3.121-to-any-port-9100] [11:53:01] RECOVERY - ns1 Puppet on ns1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [12:13:06] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 7.42, 6.96, 6.57 [12:15:04] RECOVERY - mw3 Current Load on mw3 is OK: OK - load average: 4.65, 5.78, 6.19 [13:15:05] PROBLEM - ns1 Puppet on ns1 is CRITICAL: CRITICAL: Puppet has 2 failures. Last run 3 minutes ago with 2 failures. Failed resources (up to 3 shown): Exec[ufw-allow-tcp-from-185.52.1.76-to-any-port-3506],Exec[ops_ensure_members] [13:23:02] RECOVERY - ns1 Puppet on ns1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [13:35:01] PROBLEM - ns1 Puppet on ns1 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[ops_ensure_members] [13:42:04] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 3 datacenters are down: 107.191.126.23/cpweb, 2604:180:0:33b::2/cpweb, 2a00:d880:5:8ea::ebc7/cpweb [13:42:24] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 6.83, 6.82, 6.53 [13:44:03] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [13:44:20] RECOVERY - mw3 Current Load on mw3 is OK: OK - load average: 6.32, 6.48, 6.46 [13:53:01] RECOVERY - ns1 Puppet on ns1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [13:57:58] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 7.73, 7.25, 6.82 [13:59:52] RECOVERY - mw3 Current Load on mw3 is OK: OK - load average: 5.30, 6.55, 6.66 [14:03:44] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 7.42, 7.40, 7.02 [14:03:55] Hi JohnLewis [14:07:34] RECOVERY - mw3 Current Load on mw3 is OK: OK - load average: 6.00, 6.59, 6.79 [14:15:01] PROBLEM - ns1 Puppet on ns1 is CRITICAL: CRITICAL: Puppet has 12 failures. Last run 3 minutes ago with 12 failures. Failed resources (up to 3 shown): Package[openssh-server],Service[ssh],Package[exim4],Service[postfix] [14:23:02] RECOVERY - ns1 Puppet on ns1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [14:25:01] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 6.41, 7.13, 6.82 [14:32:38] PROBLEM - mw3 Current Load on mw3 is CRITICAL: CRITICAL - load average: 8.19, 7.72, 7.26 [14:34:33] RECOVERY - mw3 Current Load on mw3 is OK: OK - load average: 5.24, 6.36, 6.80 [14:44:18] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 7.43, 7.31, 6.98 [14:46:19] RECOVERY - mw3 Current Load on mw3 is OK: OK - load average: 5.97, 6.56, 6.74 [15:01:52] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 7.86, 7.43, 7.01 [15:03:49] PROBLEM - mw3 Current Load on mw3 is CRITICAL: CRITICAL - load average: 8.41, 8.19, 7.43 [15:05:44] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 7.22, 7.76, 7.42 [15:23:00] PROBLEM - mw3 Current Load on mw3 is CRITICAL: CRITICAL - load average: 8.38, 7.68, 7.27 [15:24:56] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 6.71, 7.34, 7.24 [15:30:45] RECOVERY - mw3 Current Load on mw3 is OK: OK - load average: 6.27, 6.23, 6.71 [15:44:25] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 6.84, 7.20, 6.75 [15:48:20] PROBLEM - mw3 Current Load on mw3 is CRITICAL: CRITICAL - load average: 8.04, 7.57, 7.09 [15:50:21] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 7.44, 6.98, 6.92 [15:56:21] RECOVERY - mw3 Current Load on mw3 is OK: OK - load average: 6.06, 6.58, 6.79 [16:05:10] PROBLEM - mw3 Current Load on mw3 is CRITICAL: CRITICAL - load average: 9.40, 8.31, 7.42 [16:18:31] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 6.55, 7.53, 7.94 [16:20:28] PROBLEM - mw3 Current Load on mw3 is CRITICAL: CRITICAL - load average: 9.07, 8.06, 8.04 [16:25:03] PROBLEM - ns1 Puppet on ns1 is CRITICAL: CRITICAL: Puppet has 11 failures. Last run 3 minutes ago with 11 failures. Failed resources (up to 3 shown): Service[ssh],Package[exim4],Service[postfix],Exec[ufw-logging-low] [16:29:15] PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 2 datacenters are down: 2400:6180:0:d0::403:f001/cpweb, 2a00:d880:5:8ea::ebc7/cpweb [16:29:20] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 2 datacenters are down: 2604:180:0:33b::2/cpweb, 81.4.109.133/cpweb [16:32:47] PROBLEM - cp4 Stunnel Http for mw3 on cp4 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [16:33:09] PROBLEM - cp3 Stunnel Http for mw3 on cp3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [16:33:29] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [16:34:46] RECOVERY - cp4 Stunnel Http for mw3 on cp4 is OK: HTTP OK: HTTP/1.1 200 OK - 24639 bytes in 0.469 second response time [16:35:12] RECOVERY - cp3 Stunnel Http for mw3 on cp3 is OK: HTTP OK: HTTP/1.1 200 OK - 24661 bytes in 5.684 second response time [16:35:15] RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online [16:38:31] !log update php on misc4 including libidn2-0 libsodium23 [16:38:40] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [16:43:18] RECOVERY - ns1 Puppet on ns1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [16:45:04] PROBLEM - cp2 Stunnel Http for mw3 on cp2 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [16:45:41] PROBLEM - cp4 Stunnel Http for mw3 on cp4 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [16:46:05] PROBLEM - cp3 Stunnel Http for mw3 on cp3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [16:46:29] !log restart php-fpm on mw2 [16:46:42] PROBLEM - mw3 Puppet on mw3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [16:46:57] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [16:47:00] RECOVERY - cp2 Stunnel Http for mw3 on cp2 is OK: HTTP OK: HTTP/1.1 200 OK - 24639 bytes in 0.390 second response time [16:47:40] RECOVERY - cp4 Stunnel Http for mw3 on cp4 is OK: HTTP OK: HTTP/1.1 200 OK - 24661 bytes in 0.457 second response time [16:48:07] RECOVERY - cp3 Stunnel Http for mw3 on cp3 is OK: HTTP OK: HTTP/1.1 200 OK - 24655 bytes in 2.986 second response time [16:49:01] paladox: is phpfpm restartable for me admins yet [16:49:22] oh [16:49:23] nope [16:49:44] Can you fix it [16:49:52] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JeMIL [16:49:54] [02miraheze/puppet] 07paladox 03ca9ec95 - user: Allow mediawiki-admins to restart php7.3-fpm [16:49:54] done ^ [16:51:50] paladox: thx [16:52:42] PROBLEM - cp3 Stunnel Http for mw3 on cp3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [16:53:46] PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 2 datacenters are down: 107.191.126.23/cpweb, 2604:180:0:33b::2/cpweb [16:54:20] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 7.56, 7.54, 7.89 [16:54:43] RECOVERY - cp3 Stunnel Http for mw3 on cp3 is OK: HTTP OK: HTTP/1.1 200 OK - 24639 bytes in 1.920 second response time [16:55:53] PROBLEM - cp3 Stunnel Http for mw2 on cp3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [16:56:40] PROBLEM - mw3 Current Load on mw3 is CRITICAL: CRITICAL - load average: 7.59, 8.00, 8.03 [16:57:57] RECOVERY - cp3 Stunnel Http for mw2 on cp3 is OK: HTTP OK: HTTP/1.1 200 OK - 24640 bytes in 0.683 second response time [16:58:38] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 6.24, 7.01, 7.62 [17:00:10] RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online [17:00:32] PROBLEM - mw3 Current Load on mw3 is CRITICAL: CRITICAL - load average: 8.22, 7.25, 7.52 [17:02:26] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 7.59, 7.32, 7.49 [17:03:46] RECOVERY - mw3 Puppet on mw3 is OK: OK: Puppet is currently enabled, last run 39 seconds ago with 0 failures [17:04:24] !log sudo -u www-data php createLocalAccount.php --wiki=loginwiki QuIRC (on mw2) and per RhinosF1 request [17:04:55] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [17:05:00] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 1 datacenter is down: 128.199.139.216/cpweb [17:06:20] PROBLEM - cp3 Stunnel Http for mw2 on cp3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [17:07:00] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [17:08:16] RECOVERY - cp3 Stunnel Http for mw2 on cp3 is OK: HTTP OK: HTTP/1.1 200 OK - 24640 bytes in 0.770 second response time [17:10:09] PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 4 datacenters are down: 2604:180:0:33b::2/cpweb, 128.199.139.216/cpweb, 2400:6180:0:d0::403:f001/cpweb, 2a00:d880:5:8ea::ebc7/cpweb [17:10:21] PROBLEM - mw3 Current Load on mw3 is CRITICAL: CRITICAL - load average: 8.24, 7.24, 7.15 [17:14:23] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 7.62, 7.58, 7.37 [17:15:22] Hello Giuseppe11! If you have any questions, feel free to ask and someone should answer soon. [17:15:34] Hi Giuseppe11 [17:15:34] Is there anyone here ? [17:15:55] Giuseppe11: mostly [17:16:01] Thanks ! [17:16:01] What do you need? [17:17:05] I have a wiki here called programmingred whose domain is programming.red I have written it is unreachable and you replied but things are so messy I could not even read your reply: [17:17:06] https://meta.miraheze.org/wiki/Special:Special:RequestWikiQueue/10019 [17:17:20] [ No such special page - Miraheze Meta ] - meta.miraheze.org [17:17:21] Miraheze will report page does not exists [17:17:34] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 3 datacenters are down: 107.191.126.23/cpweb, 128.199.139.216/cpweb, 81.4.109.133/cpweb [17:19:18] RhinosF1 are you human or a bot ? [17:19:35] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [17:20:06] RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online [17:21:21] Giuseppe11: humab [17:21:30] s/b/n [17:21:30] RhinosF1 meant to say: Giuseppe11: human [17:23:57] Giuseppe11: I think your SSL cert expired [17:24:13] So the custom domain would have been removed [17:24:23] Let me check [17:25:00] Thnkas [17:25:05] PROBLEM - ns1 Puppet on ns1 is CRITICAL: CRITICAL: Puppet has 12 failures. Last run 3 minutes ago with 12 failures. Failed resources (up to 3 shown): Package[openssh-server],Service[ssh],Package[exim4],Service[postfix] [17:25:36] Giuseppe11: yep, it was removed 24 days ago - did your domain expire or something [17:26:28] Giuseppe11: what’s your on wiki username? [17:26:55] Yes, my domain expired and then I have restored it 3 days ago and my SIP unpaused it. [17:27:10] should be programmingred or redprogramming [17:27:17] Giuseppe11: I’ll have a look [17:27:32] *ISP [17:28:10] !log depool mw2 [17:28:30] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [17:29:24] paladox: ^ can you look at that it should be programmingredwiki but it’s redirecting to the custom domain [17:29:46] Giuseppe11: if you open another phabricator task we can set the custom domain back up [17:30:00] Please make sure you renew it in time to avoid issues [17:30:21] Actually it is full and active... [17:30:38] Had money shortage 1 month ago.. [17:31:44] Hello Eduardo2019-2! If you have any questions, feel free to ask and someone should answer soon. [17:31:51] [02miraheze/ssl] 07paladox pushed 031 commit to 03paladox-patch-2 [+0/-0/±1] 13https://git.io/JeMIM [17:31:52] [02miraheze/ssl] 07paladox 039d4158d - Add programming.red ssl certificate [17:31:54] [02ssl] 07paladox created branch 03paladox-patch-2 - 13https://git.io/vxP9L [17:31:55] [02ssl] 07paladox opened pull request 03#240: Add programming.red ssl certificate - 13https://git.io/JeMID [17:32:16] Giuseppe11: looks like paladox is going to set it back up [17:32:23] RhinosF1 Hi!! [17:32:29] paladox: www. Was a redirect as well [17:32:32] Thanks a lot ! [17:32:40] oh [17:32:50] RhinosF1 Hi!! [17:33:11] RECOVERY - ns1 Puppet on ns1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [17:33:14] Eduardo2019-2: one moment, what do you need [17:33:37] RhinosF1 I need to access to shell. [17:34:19] Eduardo2019-2: you need to follow the access policy then [17:34:24] [02miraheze/ssl] 07paladox pushed 031 commit to 03paladox-patch-2 [+0/-0/±1] 13https://git.io/JeMI9 [17:34:26] [02miraheze/ssl] 07paladox 035425637 - Update redirects.yaml [17:34:27] [02ssl] 07paladox synchronize pull request 03#240: Add programming.red ssl certificate - 13https://git.io/JeMID [17:34:51] PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 2 datacenters are down: 107.191.126.23/cpweb, 2400:6180:0:d0::403:f001/cpweb [17:35:08] [02miraheze/ssl] 07paladox pushed 031 commit to 03paladox-patch-2 [+1/-0/±0] 13https://git.io/JeMIH [17:35:09] [02miraheze/ssl] 07paladox 0322fdc68 - Create programming.red.crt [17:35:11] [02ssl] 07paladox synchronize pull request 03#240: Add programming.red ssl certificate - 13https://git.io/JeMID [17:35:21] [02ssl] 07paladox closed pull request 03#240: Add programming.red ssl certificate - 13https://git.io/JeMID [17:35:22] [02miraheze/ssl] 07paladox pushed 031 commit to 03master [+1/-0/±2] 13https://git.io/JeMIQ [17:35:24] [02miraheze/ssl] 07paladox 0362f19c5 - Add programming.red ssl certificate (#240) * Add programming.red ssl certificate * Update redirects.yaml * Create programming.red.crt [17:36:00] Eduardo2019-2: https://meta.miraheze.org/wiki/Tech:Access_policy [17:36:01] [ Tech:Access policy - Miraheze Meta ] - meta.miraheze.org [17:36:09] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 3 datacenters are down: 128.199.139.216/cpweb, 81.4.109.133/cpweb, 2a00:d880:5:8ea::ebc7/cpweb [17:36:14] What’s your on wiki account Eduardo2019-2 ? [17:36:22] Yes [17:36:49] RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online [17:36:58] Eduardo2019-2: what is your Miraheze username? [17:37:08] RhinosF1 Eduardo2019-2 [17:37:18] This message: PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 2 datacenters are down; me me feel really anxious. [17:37:37] .mhca Eduardo2019-2 [17:37:37] https://meta.miraheze.org/wiki/Special:CentralAuth/Eduardo2019-2 [17:37:52] Eduardo2019-2: and why do you want shell access? [17:38:14] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [17:38:19] RECOVERY - mw3 Current Load on mw3 is OK: OK - load average: 6.62, 6.56, 6.77 [17:38:28] paladox: now I receive a "404 Not Found [17:38:44] Giuseppe11: give it 10 mins [17:38:48] ok [17:39:15] Eduardo2019-2: you’ll probably want a few months non shell experience first [17:39:31] Doing technical stuff [17:39:47] You’re account is very new and has no advanced rights [17:41:23] [02puppet] 07joaquinito01 opened pull request 03#1156: Adding Alt-Svc and changing X-Cache headers - 13https://git.io/JeMLe [17:41:36] https://github.com/miraheze/puppet/pull/1156 [17:41:36] [ Adding Alt-Svc and changing X-Cache headers by joaquinito01 · Pull Request #1156 · miraheze/puppet · GitHub ] - github.com [17:42:08] Giuseppe11 should work now. [17:42:10] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 6.93, 6.91, 6.90 [17:42:28] paladox https://github.com/miraheze/puppet/pull/1156 [17:42:29] [ Adding Alt-Svc and changing X-Cache headers by joaquinito01 · Pull Request #1156 · miraheze/puppet · GitHub ] - github.com [17:42:45] I dunno what that is solving? [17:42:57] I see it ! paladox, thank you ! [17:46:06] RECOVERY - mw3 Current Load on mw3 is OK: OK - load average: 5.61, 6.38, 6.66 [17:46:36] RhinosF1Could you explain why Eduardo has been banned ? Was he a bot ? Just curious [17:46:52] @Stewards https://meta.miraheze.org/wiki/Special:CentralAuth/Eduardo2019-2 - sock puppet [17:46:57] [ Global account information for Eduardo2019-2 - Miraheze Meta ] - meta.miraheze.org [17:47:08] Giuseppe11: ban evasion [17:47:24] [02ssl] 07paladox deleted branch 03paladox-patch-2 - 13https://git.io/vxP9L [17:47:25] [02miraheze/ssl] 07paladox deleted branch 03paladox-patch-2 [17:47:58] Wow, what a battle managing here ! [17:54:24] PROBLEM - cp4 Stunnel Http for mw3 on cp4 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [17:54:55] PROBLEM - cp2 Stunnel Http for mw3 on cp2 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [17:55:23] PROBLEM - cp3 Stunnel Http for mw3 on cp3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [17:55:44] Giuseppe11: they are a known troll [17:57:17] PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 4 datacenters are down: 128.199.139.216/cpweb, 2400:6180:0:d0::403:f001/cpweb, 81.4.109.133/cpweb, 2a00:d880:5:8ea::ebc7/cpweb [17:58:33] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 4 datacenters are down: 107.191.126.23/cpweb, 2604:180:0:33b::2/cpweb, 128.199.139.216/cpweb, 2400:6180:0:d0::403:f001/cpweb [17:58:38] RECOVERY - cp4 Stunnel Http for mw3 on cp4 is OK: HTTP OK: HTTP/1.1 200 OK - 24661 bytes in 9.456 second response time [17:59:14] RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online [18:03:17] RECOVERY - cp2 Stunnel Http for mw3 on cp2 is OK: HTTP OK: HTTP/1.1 200 OK - 24661 bytes in 6.432 second response time [18:03:42] RECOVERY - cp3 Stunnel Http for mw3 on cp3 is OK: HTTP OK: HTTP/1.1 200 OK - 24639 bytes in 1.466 second response time [18:04:26] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 6.51, 6.81, 6.61 [18:04:30] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [18:09:27] [02puppet] 07RhinosF1 commented on pull request 03#1156: Adding Alt-Svc and changing X-Cache headers - 13https://git.io/JeMLy [18:09:47] paladox: ^ can you close and lock [18:09:57] [02puppet] 07paladox closed pull request 03#1156: Adding Alt-Svc and changing X-Cache headers - 13https://git.io/JeMLe [18:10:08] done [18:10:23] paladox: thanks, now to see who they hit next [18:10:27] Voidwalker: welcome [18:10:33] You have a tell with ZppixBot [18:10:45] hi [18:10:46] Voidwalker: 2019-11-30 - 19:00:53UTC tell Voidwalker ping pong - cleanup required in -logs due to a parser issue - let me know when you're around and I'll explain - there is good news though [18:10:55] there it is :P [18:12:18] PROBLEM - mw3 Current Load on mw3 is CRITICAL: CRITICAL - load average: 8.21, 7.55, 7.09 [18:14:17] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 7.67, 7.44, 7.15 [18:18:15] PROBLEM - mw3 Current Load on mw3 is CRITICAL: CRITICAL - load average: 8.16, 7.82, 7.38 [18:22:19] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 7.67, 7.57, 7.42 [18:35:31] PROBLEM - mw3 Puppet on mw3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [18:36:10] I saw Eduardo2019-2 making a resuest to create a new MikiWiki (originally founded by Prueba) - so after all they were the same person 😛 [18:38:52] @The Pioneer they’re well known. Been around years. [18:39:23] On Miraheze? [18:39:42] @The Pioneer both MH and WMF projects [18:40:08] For us we’ve known them at least 2 years (or the GitHub username matches an account locked 2 years ago) [18:40:41] Really? Strange that I haven't heard them before. [18:41:09] I knew them based on behaviour [18:41:33] I've been on Miraheze for about one and a half years, so I guess I've missed something regarding this guy. [18:41:44] They always demand advanced rights (normally shell access) and technical changes very quickly and get rude if you say no [18:42:04] Could you share some information perhaps on cvt wiki? [18:42:07] It’s either Apache Traffic Server or HTTP headers [18:42:17] Am I supposed to poke round on it! [18:43:17] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 1 datacenter is down: 2604:180:0:33b::2/cpweb [18:43:23] PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 2 datacenters are down: 107.191.126.23/cpweb, 2400:6180:0:d0::403:f001/cpweb [18:43:45] RECOVERY - mw3 Puppet on mw3 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [18:44:38] PROBLEM - ns1 Puppet on ns1 is CRITICAL: CRITICAL: Puppet has 11 failures. Last run 3 minutes ago with 11 failures. Failed resources (up to 3 shown): Service[ssh],Package[exim4],Service[postfix],Exec[ufw-logging-low] [18:45:16] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [18:45:20] RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online [18:45:28] Oh, wait, it's for CVT and Stewards. Maybe on staff wiki instead. (though personally speaking, CVT sounds more fit because it's a counter vandalism related topic) [18:46:00] I have access but I’m not sure whether I should be using sysadmin tools to do it [18:46:16] I can post on staff wiki and someone can copy across [18:46:25] PROBLEM - mw3 Current Load on mw3 is CRITICAL: CRITICAL - load average: 8.08, 8.14, 7.83 [18:46:28] paladox: ^ thoughts? [18:46:42] ? [18:47:04] oh [18:47:20] paladox: posting on CVT wiki with information on that troll or using sysadmin or staff wiki and someone can copy across to it [18:47:21] RhinosF1 Nope, speak to John or SPF|Cloud [18:47:36] SPF|Cloud: ^ [18:48:23] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 5.17, 6.99, 7.46 [18:50:18] paladox, i assigned you on [[phab:T4813]] based on what you said on discord.. [18:50:21] PROBLEM - mw3 Current Load on mw3 is CRITICAL: CRITICAL - load average: 8.42, 7.20, 7.41 [18:50:25] ok [18:50:28] thanks [18:50:30] Sometimes this division blocks of sysadmin, stews, etc. is making things harder... My biggest interest is if there's an on-wiki way (rather than on-IRC way) to detect this user. [18:52:18] @The Pioneer they likely won’t target much on wiki - it’s going to be phab, github, irc, discord - anywhere they can pester technical staff [18:52:19] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 7.05, 6.85, 7.20 [18:52:39] RECOVERY - ns1 Puppet on ns1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [18:53:03] Detecting them is: are they demanding some sort of technical thing? Yes does it involve headers or Apache? Yes it's them [18:53:23] And are they stupid if to use the same GitHub account to make the PR [18:53:55] SPF|Cloud: they might want GitHub blocking as well [18:54:18] they also behave inappropriately on testwiki, blocking other users for invalid reasons, and creating multiple accounts that they can block themselves, without making it look like they operate those accounts [18:55:22] Voidwalker: I've never seen them on wiki [18:55:33] hmm... so I guess testwiki is one place I can check their moves relatively easily. [18:55:50] Seen as you have legitimate CVT wiki access could you post information on them their Voidwalker [18:56:12] I can give you some example phab tasks and GitHub PRs later [18:56:52] RhinosF1, actually, you blocked them https://publictestwiki.com/w/index.php?title=Special:Log/block&page=User%3APrueba [18:56:55] [ Block log - TestWiki ] - publictestwiki.com [18:57:38] Voidwalker: after I saw them on wiki [18:57:43] On IRC [18:58:00] I just wanted to make sure they couldn’t do anything until they could be locked [18:58:23] Oh that's them as well [18:58:39] I remember that person [18:58:57] They must be as bad on wiki as in technical areas [19:00:15] PROBLEM - mw3 Current Load on mw3 is CRITICAL: CRITICAL - load average: 8.82, 7.76, 7.42 [19:00:27] remembers them now [19:00:35] also doesn't block easily [19:03:10] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 4 datacenters are down: 2604:180:0:33b::2/cpweb, 2400:6180:0:d0::403:f001/cpweb, 81.4.109.133/cpweb, 2a00:d880:5:8ea::ebc7/cpweb [19:05:07] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [19:05:09] So I guess vandals and trolls get more and more complex as the wiki farm develops. [19:05:20] Yep [19:06:35] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 7.58, 7.90, 7.73 [19:06:41] Which is why sharing information within a team will be important. Covering everything alone will be a deadly work. [19:07:31] Voidwalker: you okay to set up a page on CVT wiki for them? [19:07:43] I will, but not just now, got a few things to handle [19:07:53]