[00:12:10] PROBLEM - cp4 Current Load on cp4 is WARNING: WARNING - load average: 1.67, 1.85, 1.22 [00:14:09] PROBLEM - cp4 Current Load on cp4 is CRITICAL: CRITICAL - load average: 2.57, 1.95, 1.32 [00:16:07] PROBLEM - cp4 Current Load on cp4 is WARNING: WARNING - load average: 1.90, 1.84, 1.34 [00:18:05] RECOVERY - cp4 Current Load on cp4 is OK: OK - load average: 0.96, 1.45, 1.26 [00:23:48] PROBLEM - test1 Current Load on test1 is WARNING: WARNING - load average: 1.07, 1.62, 1.98 [00:25:48] PROBLEM - test1 Current Load on test1 is CRITICAL: CRITICAL - load average: 1.86, 1.82, 2.01 [00:26:55] PROBLEM - cp4 Current Load on cp4 is WARNING: WARNING - load average: 1.77, 1.73, 1.43 [00:28:53] RECOVERY - cp4 Current Load on cp4 is OK: OK - load average: 0.91, 1.44, 1.36 [00:31:48] PROBLEM - test1 Current Load on test1 is WARNING: WARNING - load average: 1.58, 1.83, 1.96 [00:32:47] PROBLEM - cp4 Current Load on cp4 is WARNING: WARNING - load average: 1.62, 1.84, 1.55 [00:33:48] PROBLEM - test1 Current Load on test1 is CRITICAL: CRITICAL - load average: 2.10, 1.99, 2.01 [00:35:48] PROBLEM - test1 Current Load on test1 is WARNING: WARNING - load average: 1.72, 1.95, 1.98 [00:36:43] RECOVERY - cp4 Current Load on cp4 is OK: OK - load average: 1.61, 1.70, 1.56 [00:39:48] PROBLEM - test1 Current Load on test1 is CRITICAL: CRITICAL - load average: 3.93, 2.61, 2.20 [00:42:36] PROBLEM - cp4 Current Load on cp4 is WARNING: WARNING - load average: 1.58, 1.80, 1.64 [00:44:35] RECOVERY - cp4 Current Load on cp4 is OK: OK - load average: 1.02, 1.53, 1.56 [00:45:48] PROBLEM - test1 Current Load on test1 is WARNING: WARNING - load average: 1.28, 1.71, 1.93 [00:49:48] RECOVERY - test1 Current Load on test1 is OK: OK - load average: 0.77, 1.28, 1.70 [00:57:48] PROBLEM - test1 Current Load on test1 is CRITICAL: CRITICAL - load average: 2.18, 1.71, 1.67 [00:59:48] RECOVERY - test1 Current Load on test1 is OK: OK - load average: 1.38, 1.62, 1.64 [01:03:13] PROBLEM - cp4 Current Load on cp4 is CRITICAL: CRITICAL - load average: 1.86, 2.02, 1.65 [01:03:48] PROBLEM - test1 Current Load on test1 is CRITICAL: CRITICAL - load average: 2.70, 2.01, 1.79 [01:07:10] PROBLEM - cp4 Current Load on cp4 is WARNING: WARNING - load average: 1.14, 1.81, 1.65 [01:09:12] PROBLEM - cp4 Current Load on cp4 is CRITICAL: CRITICAL - load average: 2.12, 1.73, 1.62 [01:13:10] PROBLEM - cp4 Current Load on cp4 is WARNING: WARNING - load average: 1.33, 1.94, 1.79 [01:15:10] PROBLEM - cp4 Current Load on cp4 is CRITICAL: CRITICAL - load average: 3.22, 2.05, 1.82 [01:17:10] RECOVERY - cp4 Current Load on cp4 is OK: OK - load average: 0.90, 1.55, 1.66 [01:19:48] PROBLEM - test1 Current Load on test1 is WARNING: WARNING - load average: 0.81, 1.54, 1.98 [01:25:48] PROBLEM - test1 Current Load on test1 is CRITICAL: CRITICAL - load average: 3.57, 2.19, 2.06 [01:48:22] PROBLEM - bacula1 Disk Space on bacula1 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [01:50:22] PROBLEM - bacula1 Disk Space on bacula1 is WARNING: DISK WARNING - free space: / 44856 MB (9% inode=99%); [02:11:39] PROBLEM - cp4 Current Load on cp4 is CRITICAL: CRITICAL - load average: 7.62, 3.91, 2.23 [02:13:40] PROBLEM - misc4 Current Load on misc4 is CRITICAL: CRITICAL - load average: 3.96, 4.08, 2.48 [02:15:40] RECOVERY - misc4 Current Load on misc4 is OK: OK - load average: 1.63, 3.20, 2.36 [02:16:37] PROBLEM - cp4 Puppet on cp4 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 2 minutes ago with 0 failures [02:18:03] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fjMGt [02:18:05] [02miraheze/puppet] 07paladox 035b3e5b1 - redis: Lower heap by 100mb [02:18:12] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fjMGq [02:18:14] [02miraheze/puppet] 07paladox 030c853ae - Update redis.pp [02:19:31] PROBLEM - cp4 Current Load on cp4 is WARNING: WARNING - load average: 1.18, 1.90, 1.98 [02:23:27] RECOVERY - cp4 Current Load on cp4 is OK: OK - load average: 0.62, 1.24, 1.69 [02:23:52] !log apt-get upgrade - misc2 [02:23:57] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [02:34:03] [02miraheze/dns] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fjMGZ [02:34:05] [02miraheze/dns] 07paladox 03498cf49 - Update admin_state [02:34:10] !log depool cp4 [02:34:14] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [02:36:33] RECOVERY - cp4 Puppet on cp4 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [02:37:48] PROBLEM - test1 Current Load on test1 is WARNING: WARNING - load average: 1.66, 1.62, 1.95 [02:37:54] !log apt upgrade - cp4 [02:38:03] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [02:38:09] !log reboot cp4 [02:38:21] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [02:41:59] [02miraheze/dns] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fjMGW [02:42:00] [02miraheze/dns] 07paladox 0305d5162 - Revert "Update admin_state" This reverts commit 498cf49b0e116ce78e48b49f4fa1733ebea69712. [02:45:48] PROBLEM - test1 Current Load on test1 is CRITICAL: CRITICAL - load average: 2.22, 1.69, 1.78 [02:47:48] PROBLEM - test1 Current Load on test1 is WARNING: WARNING - load average: 1.81, 1.67, 1.76 [02:49:48] RECOVERY - test1 Current Load on test1 is OK: OK - load average: 1.13, 1.49, 1.68 [02:58:35] PROBLEM - test1 Current Load on test1 is WARNING: WARNING - load average: 1.90, 1.73, 1.66 [03:00:29] RECOVERY - test1 Current Load on test1 is OK: OK - load average: 1.00, 1.43, 1.55 [03:06:10] PROBLEM - test1 Current Load on test1 is CRITICAL: CRITICAL - load average: 2.48, 1.79, 1.63 [03:08:04] RECOVERY - test1 Current Load on test1 is OK: OK - load average: 1.68, 1.70, 1.61 [03:36:23] PROBLEM - test1 Current Load on test1 is CRITICAL: CRITICAL - load average: 2.86, 2.21, 1.76 [03:43:59] PROBLEM - test1 Current Load on test1 is WARNING: WARNING - load average: 1.58, 1.85, 1.77 [03:45:53] PROBLEM - test1 Current Load on test1 is CRITICAL: CRITICAL - load average: 2.30, 2.02, 1.84 [03:49:48] PROBLEM - test1 Current Load on test1 is WARNING: WARNING - load average: 1.35, 1.77, 1.78 [03:55:48] PROBLEM - test1 Current Load on test1 is CRITICAL: CRITICAL - load average: 2.51, 2.00, 1.83 [03:57:48] PROBLEM - test1 Current Load on test1 is WARNING: WARNING - load average: 1.70, 1.88, 1.81 [03:59:48] PROBLEM - test1 Current Load on test1 is CRITICAL: CRITICAL - load average: 1.88, 2.13, 1.92 [04:03:48] PROBLEM - test1 Current Load on test1 is WARNING: WARNING - load average: 1.11, 1.78, 1.84 [04:05:48] PROBLEM - test1 Current Load on test1 is CRITICAL: CRITICAL - load average: 2.49, 2.16, 1.97 [04:11:48] PROBLEM - test1 Current Load on test1 is WARNING: WARNING - load average: 1.04, 1.78, 1.89 [04:21:48] RECOVERY - test1 Current Load on test1 is OK: OK - load average: 0.76, 1.25, 1.60 [04:27:48] PROBLEM - test1 Current Load on test1 is CRITICAL: CRITICAL - load average: 2.20, 1.79, 1.70 [04:29:48] RECOVERY - test1 Current Load on test1 is OK: OK - load average: 0.96, 1.48, 1.59 [05:08:00] PROBLEM - test1 Current Load on test1 is CRITICAL: CRITICAL - load average: 2.94, 2.45, 1.86 [05:13:48] PROBLEM - test1 Current Load on test1 is WARNING: WARNING - load average: 0.63, 1.72, 1.77 [05:17:48] RECOVERY - test1 Current Load on test1 is OK: OK - load average: 0.81, 1.35, 1.61 [05:21:58] PROBLEM - elasticsearch1 SSH on elasticsearch1 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [05:22:05] PROBLEM - elasticsearch1 Current Load on elasticsearch1 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [05:22:23] PROBLEM - Host elasticsearch1 is DOWN: PING CRITICAL - Packet loss = 100% [05:27:48] PROBLEM - test1 Current Load on test1 is CRITICAL: CRITICAL - load average: 2.69, 2.05, 1.73 [05:29:48] PROBLEM - test1 Current Load on test1 is WARNING: WARNING - load average: 1.67, 1.92, 1.72 [05:34:15] PROBLEM - es1 Current Load on es1 is CRITICAL: CRITICAL - load average: 4.17, 3.42, 1.75 [05:35:48] PROBLEM - test1 Current Load on test1 is CRITICAL: CRITICAL - load average: 2.71, 2.09, 1.82 [05:36:10] RECOVERY - es1 Current Load on es1 is OK: OK - load average: 0.84, 2.39, 1.57 [05:51:48] PROBLEM - test1 Current Load on test1 is WARNING: WARNING - load average: 1.43, 1.92, 1.94 [05:57:48] PROBLEM - test1 Current Load on test1 is CRITICAL: CRITICAL - load average: 2.08, 1.95, 1.93 [05:59:48] PROBLEM - test1 Current Load on test1 is WARNING: WARNING - load average: 1.18, 1.77, 1.86 [06:01:48] PROBLEM - test1 Current Load on test1 is CRITICAL: CRITICAL - load average: 2.28, 2.10, 1.97 [06:03:48] PROBLEM - test1 Current Load on test1 is WARNING: WARNING - load average: 1.77, 1.97, 1.94 [06:05:48] PROBLEM - test1 Current Load on test1 is CRITICAL: CRITICAL - load average: 2.51, 2.18, 2.02 [06:33:48] PROBLEM - test1 Current Load on test1 is WARNING: WARNING - load average: 1.68, 1.85, 1.99 [06:35:48] PROBLEM - test1 Current Load on test1 is CRITICAL: CRITICAL - load average: 2.02, 1.91, 1.99 [06:37:48] PROBLEM - test1 Current Load on test1 is WARNING: WARNING - load average: 1.53, 1.89, 1.98 [06:39:48] PROBLEM - test1 Current Load on test1 is CRITICAL: CRITICAL - load average: 2.63, 2.36, 2.15 [06:43:48] PROBLEM - test1 Current Load on test1 is WARNING: WARNING - load average: 1.29, 1.83, 1.98 [06:45:48] PROBLEM - test1 Current Load on test1 is CRITICAL: CRITICAL - load average: 2.24, 1.93, 1.99 [06:51:48] PROBLEM - test1 Current Load on test1 is WARNING: WARNING - load average: 0.94, 1.72, 1.91 [07:01:48] PROBLEM - test1 Current Load on test1 is CRITICAL: CRITICAL - load average: 2.30, 1.92, 1.85 [07:03:48] PROBLEM - test1 Current Load on test1 is WARNING: WARNING - load average: 1.39, 1.67, 1.76 [07:11:48] RECOVERY - test1 Current Load on test1 is OK: OK - load average: 1.17, 1.43, 1.62 [07:18:35] PROBLEM - test1 Current Load on test1 is CRITICAL: CRITICAL - load average: 2.98, 2.20, 1.88 [07:22:23] PROBLEM - test1 Current Load on test1 is WARNING: WARNING - load average: 1.37, 1.81, 1.79 [07:29:58] RECOVERY - test1 Current Load on test1 is OK: OK - load average: 1.03, 1.47, 1.64 [07:35:48] PROBLEM - test1 Current Load on test1 is CRITICAL: CRITICAL - load average: 3.06, 2.12, 1.82 [07:43:48] PROBLEM - test1 Current Load on test1 is WARNING: WARNING - load average: 1.52, 2.00, 1.94 [07:47:48] PROBLEM - test1 Current Load on test1 is CRITICAL: CRITICAL - load average: 2.94, 2.25, 2.02 [07:53:48] PROBLEM - test1 Current Load on test1 is WARNING: WARNING - load average: 1.72, 1.92, 1.94 [07:55:48] PROBLEM - test1 Current Load on test1 is CRITICAL: CRITICAL - load average: 2.29, 2.03, 1.97 [07:59:48] PROBLEM - test1 Current Load on test1 is WARNING: WARNING - load average: 1.54, 1.80, 1.90 [08:03:28] PROBLEM - cp4 Current Load on cp4 is CRITICAL: CRITICAL - load average: 2.14, 2.10, 1.49 [08:03:48] PROBLEM - test1 Current Load on test1 is CRITICAL: CRITICAL - load average: 2.45, 2.13, 2.01 [08:05:11] PROBLEM - misc2 Puppet on misc2 is CRITICAL: CRITICAL: Puppet has 2 failures. Last run 2 minutes ago with 2 failures. Failed resources (up to 3 shown) [08:05:17] PROBLEM - cp4 Puppet on cp4 is CRITICAL: CRITICAL: Puppet has 321 failures. Last run 2 minutes ago with 321 failures. Failed resources (up to 3 shown) [08:05:27] PROBLEM - mw1 Puppet on mw1 is CRITICAL: CRITICAL: Puppet has 279 failures. Last run 2 minutes ago with 279 failures. Failed resources (up to 3 shown) [08:05:34] PROBLEM - es4 Puppet on es4 is CRITICAL: CRITICAL: Puppet has 9 failures. Last run 2 minutes ago with 9 failures. Failed resources (up to 3 shown) [08:05:36] PROBLEM - misc4 Puppet on misc4 is CRITICAL: CRITICAL: Puppet has 11 failures. Last run 2 minutes ago with 11 failures. Failed resources (up to 3 shown) [08:05:47] PROBLEM - mw2 Puppet on mw2 is CRITICAL: CRITICAL: Puppet has 255 failures. Last run 2 minutes ago with 255 failures. Failed resources (up to 3 shown) [08:05:55] PROBLEM - cp2 Puppet on cp2 is CRITICAL: CRITICAL: Puppet has 319 failures. Last run 2 minutes ago with 319 failures. Failed resources (up to 3 shown) [08:06:00] PROBLEM - misc1 Puppet on misc1 is CRITICAL: CRITICAL: Puppet has 25 failures. Last run 3 minutes ago with 25 failures. Failed resources (up to 3 shown) [08:06:09] PROBLEM - es1 Puppet on es1 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [08:06:10] PROBLEM - misc3 Puppet on misc3 is CRITICAL: CRITICAL: Puppet has 9 failures. Last run 3 minutes ago with 9 failures. Failed resources (up to 3 shown) [08:06:25] PROBLEM - es3 Puppet on es3 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [08:06:29] PROBLEM - mw3 Puppet on mw3 is CRITICAL: CRITICAL: Puppet has 275 failures. Last run 3 minutes ago with 275 failures. Failed resources (up to 3 shown) [08:07:00] PROBLEM - cp3 Puppet on cp3 is CRITICAL: CRITICAL: Puppet has 235 failures. Last run 3 minutes ago with 235 failures. Failed resources (up to 3 shown): File[www.openonderwijs.org],File[www.openonderwijs.org_private],File[wiki.meeusen.net],File[wiki.meeusen.net_private] [08:09:22] RECOVERY - cp4 Current Load on cp4 is OK: OK - load average: 1.20, 1.63, 1.48 [08:13:10] RECOVERY - misc2 Puppet on misc2 is OK: OK: Puppet is currently enabled, last run 8 seconds ago with 0 failures [08:13:37] RECOVERY - misc4 Puppet on misc4 is OK: OK: Puppet is currently enabled, last run 17 seconds ago with 0 failures [08:13:43] RECOVERY - mw2 Puppet on mw2 is OK: OK: Puppet is currently enabled, last run 7 seconds ago with 0 failures [08:14:00] RECOVERY - misc1 Puppet on misc1 is OK: OK: Puppet is currently enabled, last run 43 seconds ago with 0 failures [08:14:08] PROBLEM - es2 Puppet on es2 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [08:14:10] RECOVERY - misc3 Puppet on misc3 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [08:14:31] RECOVERY - mw3 Puppet on mw3 is OK: OK: Puppet is currently enabled, last run 45 seconds ago with 0 failures [08:14:59] RECOVERY - cp3 Puppet on cp3 is OK: OK: Puppet is currently enabled, last run 17 seconds ago with 0 failures [08:15:19] RECOVERY - cp4 Puppet on cp4 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [08:15:19] RECOVERY - mw1 Puppet on mw1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [08:15:55] RECOVERY - cp2 Puppet on cp2 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [08:16:29] RECOVERY - es3 Puppet on es3 is OK: OK: Puppet is currently enabled, last run 51 seconds ago with 0 failures [08:17:31] RECOVERY - es4 Puppet on es4 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [08:24:08] RECOVERY - es2 Puppet on es2 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [08:24:13] RECOVERY - es1 Puppet on es1 is OK: OK: Puppet is currently enabled, last run 30 seconds ago with 0 failures [08:32:26] PROBLEM - es3 Current Load on es3 is CRITICAL: CRITICAL - load average: 4.48, 3.70, 2.44 [08:34:25] RECOVERY - es3 Current Load on es3 is OK: OK - load average: 1.34, 2.80, 2.25 [09:03:10] PROBLEM - cp4 Current Load on cp4 is CRITICAL: CRITICAL - load average: 2.14, 1.87, 1.30 [09:05:10] RECOVERY - cp4 Current Load on cp4 is OK: OK - load average: 1.46, 1.52, 1.23 [09:05:48] PROBLEM - test1 Current Load on test1 is WARNING: WARNING - load average: 1.14, 1.37, 1.99 [09:09:48] RECOVERY - test1 Current Load on test1 is OK: OK - load average: 0.22, 0.70, 1.57 [09:38:53] !log purge binary logs before '2019-07-20 07:00:00'; [09:38:57] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [10:03:16] PROBLEM - cp4 Current Load on cp4 is WARNING: WARNING - load average: 1.85, 1.75, 1.23 [10:07:14] RECOVERY - cp4 Current Load on cp4 is OK: OK - load average: 0.72, 1.28, 1.15 [10:22:03] PROBLEM - cp4 Current Load on cp4 is CRITICAL: CRITICAL - load average: 2.49, 1.78, 1.44 [10:27:58] PROBLEM - cp4 Current Load on cp4 is WARNING: WARNING - load average: 0.72, 1.82, 1.68 [10:29:56] RECOVERY - cp4 Current Load on cp4 is OK: OK - load average: 0.80, 1.50, 1.57 [10:35:01] PROBLEM - es3 Current Load on es3 is WARNING: WARNING - load average: 3.83, 3.32, 2.08 [10:36:57] RECOVERY - es3 Current Load on es3 is OK: OK - load average: 0.70, 2.35, 1.88 [10:37:48] PROBLEM - test1 Current Load on test1 is WARNING: WARNING - load average: 2.00, 1.54, 1.27 [10:39:57] PROBLEM - test1 Current Load on test1 is CRITICAL: CRITICAL - load average: 3.14, 1.87, 1.40 [10:41:57] PROBLEM - cp4 Current Load on cp4 is CRITICAL: CRITICAL - load average: 9.85, 6.39, 3.47 [10:41:59] PROBLEM - misc4 Puppet on misc4 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [10:42:07] PROBLEM - cp4 Varnish Backends on cp4 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [10:42:39] PROBLEM - misc4 Current Load on misc4 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [10:42:51] http://wm-bot.wmflabs.org/dump/%23miraheze.htm [10:42:51] @info [10:42:53] [ #miraheze ] - wm-bot.wmflabs.org [10:44:00] RECOVERY - misc4 Puppet on misc4 is OK: OK: Puppet is currently enabled, last run 11 minutes ago with 0 failures [10:44:02] RECOVERY - cp4 Varnish Backends on cp4 is OK: All 5 backends are healthy [10:50:29] PROBLEM - misc4 Current Load on misc4 is WARNING: WARNING - load average: 1.14, 3.49, 3.71 [10:51:48] PROBLEM - test1 Current Load on test1 is WARNING: WARNING - load average: 1.04, 1.75, 1.81 [10:53:48] RECOVERY - test1 Current Load on test1 is OK: OK - load average: 0.88, 1.44, 1.69 [10:54:24] RECOVERY - misc4 Current Load on misc4 is OK: OK - load average: 0.90, 2.15, 3.11 [10:59:40] PROBLEM - cp4 Current Load on cp4 is WARNING: WARNING - load average: 0.60, 0.99, 1.91 [11:01:39] PROBLEM - cp4 Current Load on cp4 is CRITICAL: CRITICAL - load average: 10.78, 4.55, 3.08 [11:02:39] PROBLEM - misc4 Current Load on misc4 is CRITICAL: CRITICAL - load average: 16.40, 6.98, 4.33 [11:03:48] PROBLEM - test1 Current Load on test1 is CRITICAL: CRITICAL - load average: 2.49, 2.54, 2.04 [11:07:48] PROBLEM - test1 Current Load on test1 is WARNING: WARNING - load average: 1.18, 1.75, 1.82 [11:08:32] PROBLEM - misc4 Current Load on misc4 is WARNING: WARNING - load average: 0.58, 3.50, 3.79 [11:11:48] RECOVERY - test1 Current Load on test1 is OK: OK - load average: 0.87, 1.48, 1.70 [11:14:24] RECOVERY - misc4 Current Load on misc4 is OK: OK - load average: 0.83, 2.07, 3.08 [11:23:18] PROBLEM - cp4 Current Load on cp4 is WARNING: WARNING - load average: 0.65, 1.23, 1.85 [11:25:16] PROBLEM - cp4 Current Load on cp4 is CRITICAL: CRITICAL - load average: 3.18, 1.55, 1.86 [11:27:14] RECOVERY - cp4 Current Load on cp4 is OK: OK - load average: 0.81, 1.20, 1.70 [11:43:08] PROBLEM - es3 Current Load on es3 is WARNING: WARNING - load average: 3.82, 3.37, 2.35 [11:45:10] PROBLEM - es3 Current Load on es3 is CRITICAL: CRITICAL - load average: 15.79, 7.71, 4.05 [11:51:47] PROBLEM - es3 Puppet on es3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [11:56:45] PROBLEM - es3 SSH on es3 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:58:07] RECOVERY - es3 Puppet on es3 is OK: OK: Puppet is currently enabled, last run 28 seconds ago with 0 failures [11:58:44] RECOVERY - es3 SSH on es3 is OK: SSH OK - OpenSSH_7.4p1 Debian-10+deb9u6 (protocol 2.0) [12:01:42] PROBLEM - es4 Current Load on es4 is CRITICAL: CRITICAL - load average: 10.94, 5.10, 3.16 [12:03:39] PROBLEM - es4 Current Load on es4 is WARNING: WARNING - load average: 2.29, 3.78, 2.90 [12:05:36] RECOVERY - es4 Current Load on es4 is OK: OK - load average: 0.63, 2.63, 2.58 [12:11:48] PROBLEM - test1 Current Load on test1 is WARNING: WARNING - load average: 1.66, 1.71, 1.25 [12:12:42] PROBLEM - cp4 Current Load on cp4 is CRITICAL: CRITICAL - load average: 1.67, 2.23, 1.57 [12:13:48] RECOVERY - test1 Current Load on test1 is OK: OK - load average: 0.99, 1.47, 1.22 [12:14:40] RECOVERY - cp4 Current Load on cp4 is OK: OK - load average: 0.54, 1.65, 1.44 [12:15:31] PROBLEM - es4 Current Load on es4 is CRITICAL: CRITICAL - load average: 2.96, 4.37, 3.23 [12:17:28] RECOVERY - es4 Current Load on es4 is OK: OK - load average: 0.68, 3.09, 2.90 [12:26:26] PROBLEM - es3 Current Load on es3 is WARNING: WARNING - load average: 0.57, 2.00, 3.62 [12:28:26] RECOVERY - es3 Current Load on es3 is OK: OK - load average: 0.42, 1.46, 3.22 [12:32:21] PROBLEM - cp4 Current Load on cp4 is CRITICAL: CRITICAL - load average: 2.01, 1.30, 1.11 [12:34:19] RECOVERY - cp4 Current Load on cp4 is OK: OK - load average: 1.66, 1.54, 1.22 [12:34:29] PROBLEM - test1 Current Load on test1 is CRITICAL: CRITICAL - load average: 3.50, 2.34, 1.64 [12:38:13] PROBLEM - cp4 Current Load on cp4 is WARNING: WARNING - load average: 1.60, 1.85, 1.44 [12:42:09] RECOVERY - cp4 Current Load on cp4 is OK: OK - load average: 0.96, 1.40, 1.35 [12:43:58] PROBLEM - test1 Current Load on test1 is WARNING: WARNING - load average: 0.96, 1.88, 1.88 [12:46:13] PROBLEM - es1 Current Load on es1 is CRITICAL: CRITICAL - load average: 4.80, 3.96, 2.49 [12:46:20] PROBLEM - es4 Current Load on es4 is CRITICAL: CRITICAL - load average: 8.96, 5.42, 3.01 [12:47:48] RECOVERY - test1 Current Load on test1 is OK: OK - load average: 0.62, 1.32, 1.65 [12:48:08] RECOVERY - es1 Current Load on es1 is OK: OK - load average: 1.99, 3.31, 2.44 [12:50:14] PROBLEM - es4 Current Load on es4 is WARNING: WARNING - load average: 1.16, 3.63, 2.88 [12:52:11] RECOVERY - es4 Current Load on es4 is OK: OK - load average: 1.45, 2.89, 2.69 [13:03:46] PROBLEM - cp4 Current Load on cp4 is WARNING: WARNING - load average: 1.31, 1.96, 1.39 [13:05:44] RECOVERY - cp4 Current Load on cp4 is OK: OK - load average: 1.26, 1.70, 1.36 [13:16:13] PROBLEM - es2 Current Load on es2 is CRITICAL: CRITICAL - load average: 4.74, 3.79, 2.06 [13:18:11] RECOVERY - es2 Current Load on es2 is OK: OK - load average: 1.04, 2.75, 1.89 [13:36:27] PROBLEM - es3 Current Load on es3 is WARNING: WARNING - load average: 3.93, 2.68, 1.66 [13:38:26] RECOVERY - es3 Current Load on es3 is OK: OK - load average: 0.87, 1.94, 1.51 [13:43:43] !log removing the archive script from misc2 (temporarily, to stop space filling up on the db, this will stop stats from updating) [13:43:47] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [13:43:57] PROBLEM - es4 Puppet on es4 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [13:44:36] !log root@misc2:/var/spool/cron/crontabs# rm www-data [13:44:41] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [13:45:17] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fjMl4 [13:45:19] [02miraheze/puppet] 07paladox 0377b4c76 - matomo: Disable archive_matomo cron temporarily [13:45:28] PROBLEM - es4 Current Load on es4 is CRITICAL: CRITICAL - load average: 7.24, 4.79, 2.64 [13:47:58] RECOVERY - es4 Puppet on es4 is OK: OK: Puppet is currently enabled, last run 13 minutes ago with 0 failures [13:56:26] PROBLEM - es3 Current Load on es3 is CRITICAL: CRITICAL - load average: 4.15, 3.46, 2.49 [13:57:11] PROBLEM - es4 Current Load on es4 is WARNING: WARNING - load average: 0.39, 3.38, 3.86 [13:57:48] PROBLEM - test1 Current Load on test1 is WARNING: WARNING - load average: 1.89, 1.71, 1.33 [13:58:26] RECOVERY - es3 Current Load on es3 is OK: OK - load average: 0.87, 2.44, 2.23 [13:59:48] RECOVERY - test1 Current Load on test1 is OK: OK - load average: 0.85, 1.44, 1.27 [14:01:05] RECOVERY - es4 Current Load on es4 is OK: OK - load average: 1.26, 2.03, 3.18 [14:03:48] PROBLEM - test1 Current Load on test1 is WARNING: WARNING - load average: 1.49, 1.89, 1.53 [14:09:48] PROBLEM - test1 Current Load on test1 is CRITICAL: CRITICAL - load average: 2.80, 2.28, 1.79 [14:11:31] PROBLEM - cp4 Current Load on cp4 is CRITICAL: CRITICAL - load average: 2.14, 1.85, 1.41 [14:11:48] PROBLEM - test1 Current Load on test1 is WARNING: WARNING - load average: 1.37, 1.99, 1.74 [14:13:29] RECOVERY - cp4 Current Load on cp4 is OK: OK - load average: 0.85, 1.52, 1.34 [14:13:48] RECOVERY - test1 Current Load on test1 is OK: OK - load average: 0.93, 1.59, 1.62 [14:28:16] PROBLEM - test1 Current Load on test1 is CRITICAL: CRITICAL - load average: 2.38, 2.19, 1.81 [14:30:10] PROBLEM - test1 Current Load on test1 is WARNING: WARNING - load average: 1.51, 1.90, 1.74 [14:35:52] PROBLEM - test1 Current Load on test1 is CRITICAL: CRITICAL - load average: 2.65, 1.98, 1.77 [14:37:48] PROBLEM - test1 Current Load on test1 is WARNING: WARNING - load average: 1.73, 1.85, 1.75 [14:39:48] PROBLEM - test1 Current Load on test1 is CRITICAL: CRITICAL - load average: 2.34, 2.22, 1.90 [14:41:48] PROBLEM - test1 Current Load on test1 is WARNING: WARNING - load average: 1.08, 1.82, 1.80 [14:43:48] RECOVERY - test1 Current Load on test1 is OK: OK - load average: 0.94, 1.51, 1.68 [14:43:53] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fjM8k [14:43:54] [02miraheze/puppet] 07paladox 03aeb4a7e - php: Raise cli memory_limit to 256M [14:44:59] PROBLEM - es4 Current Load on es4 is CRITICAL: CRITICAL - load average: 5.18, 3.14, 2.13 [14:45:15] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fjM8t [14:45:17] [02miraheze/puppet] 07paladox 03001be1e - matomo: Raise max_execution_time to 256M [14:46:57] !log root@misc2:/etc/php/7.3# apt-get upgrade [14:47:01] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [14:52:34] PROBLEM - es3 Current Load on es3 is CRITICAL: CRITICAL - load average: 13.95, 6.93, 3.71 [14:54:51] RECOVERY - es4 Current Load on es4 is OK: OK - load average: 0.98, 2.98, 2.82 [14:56:23] PROBLEM - test1 Current Load on test1 is CRITICAL: CRITICAL - load average: 2.17, 1.70, 1.62 [14:56:26] PROBLEM - es3 Current Load on es3 is WARNING: WARNING - load average: 0.57, 3.49, 3.03 [14:58:25] RECOVERY - es3 Current Load on es3 is OK: OK - load average: 0.46, 2.49, 2.71 [15:00:10] RECOVERY - test1 Current Load on test1 is OK: OK - load average: 1.25, 1.64, 1.62 [15:02:50] PROBLEM - cp4 Current Load on cp4 is WARNING: WARNING - load average: 1.79, 1.73, 1.39 [15:03:02] PROBLEM - es4 Puppet on es4 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [15:03:10] PROBLEM - es4 Current Load on es4 is CRITICAL: CRITICAL - load average: 11.52, 5.82, 3.71 [15:04:48] RECOVERY - cp4 Current Load on cp4 is OK: OK - load average: 0.60, 1.29, 1.27 [15:05:00] RECOVERY - es4 Puppet on es4 is OK: OK: Puppet is currently enabled, last run 12 minutes ago with 0 failures [15:08:59] PROBLEM - es4 Current Load on es4 is WARNING: WARNING - load average: 1.96, 3.69, 3.49 [15:10:52] PROBLEM - es1 Puppet on es1 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [15:10:56] PROBLEM - es1 SSH on es1 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:11:16] PROBLEM - es1 Current Load on es1 is CRITICAL: CRITICAL - load average: 10.76, 6.71, 3.71 [15:12:51] RECOVERY - es1 Puppet on es1 is OK: OK: Puppet is currently enabled, last run 9 minutes ago with 0 failures [15:12:52] PROBLEM - es4 Current Load on es4 is CRITICAL: CRITICAL - load average: 4.80, 4.11, 3.68 [15:12:55] RECOVERY - es1 SSH on es1 is OK: SSH OK - OpenSSH_7.4p1 Debian-10+deb9u6 (protocol 2.0) [15:16:55] PROBLEM - es4 Current Load on es4 is WARNING: WARNING - load average: 3.69, 3.62, 3.60 [15:17:01] PROBLEM - es1 Current Load on es1 is WARNING: WARNING - load average: 1.72, 3.62, 3.36 [15:17:48] PROBLEM - test1 Current Load on test1 is CRITICAL: CRITICAL - load average: 2.33, 1.69, 1.52 [15:18:52] RECOVERY - es4 Current Load on es4 is OK: OK - load average: 1.18, 2.76, 3.29 [15:18:56] RECOVERY - es1 Current Load on es1 is OK: OK - load average: 0.87, 2.68, 3.04 [15:19:48] RECOVERY - test1 Current Load on test1 is OK: OK - load average: 1.53, 1.70, 1.54 [15:26:59] PROBLEM - es1 Current Load on es1 is CRITICAL: CRITICAL - load average: 12.06, 6.69, 4.35 [15:31:18] PROBLEM - es3 Current Load on es3 is CRITICAL: CRITICAL - load average: 4.47, 4.37, 3.04 [15:34:44] PROBLEM - es1 Current Load on es1 is WARNING: WARNING - load average: 1.77, 3.65, 3.81 [15:38:34] RECOVERY - es1 Current Load on es1 is OK: OK - load average: 1.70, 2.52, 3.30 [15:39:03] RECOVERY - es3 Current Load on es3 is OK: OK - load average: 0.91, 3.03, 3.15 [15:47:48] PROBLEM - test1 Current Load on test1 is WARNING: WARNING - load average: 1.88, 1.72, 1.51 [15:49:48] RECOVERY - test1 Current Load on test1 is OK: OK - load average: 0.73, 1.39, 1.41 [15:52:06] PROBLEM - es3 Puppet on es3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [15:52:52] PROBLEM - es3 Current Load on es3 is CRITICAL: CRITICAL - load average: 6.63, 4.05, 3.10 [15:56:14] RECOVERY - es3 Puppet on es3 is OK: OK: Puppet is currently enabled, last run 13 minutes ago with 0 failures [15:58:38] RECOVERY - es3 Current Load on es3 is OK: OK - load average: 0.77, 3.14, 3.18 [16:05:41] PROBLEM - es4 Current Load on es4 is WARNING: WARNING - load average: 3.22, 3.44, 2.42 [16:07:38] RECOVERY - es4 Current Load on es4 is OK: OK - load average: 2.41, 3.18, 2.45 [16:14:54] PROBLEM - es1 Current Load on es1 is CRITICAL: CRITICAL - load average: 8.69, 5.36, 3.38 [16:16:04] [02miraheze/mediawiki] 07paladox pushed 032 commits to 03REL1_33 [+0/-0/±3] 13https://git.io/fjM4O [16:16:05] [02miraheze/mediawiki] 07paladox 03d8f89f1 - Update PortableInfobox [16:16:07] [02miraheze/mediawiki] 07paladox 0393361b4 - Merge branch 'REL1_33' of github.com:miraheze/mediawiki into REL1_33 [16:17:46] PROBLEM - es4 Current Load on es4 is CRITICAL: CRITICAL - load average: 5.46, 5.00, 3.49 [16:18:45] RECOVERY - es1 Current Load on es1 is OK: OK - load average: 0.91, 3.40, 3.06 [16:21:41] PROBLEM - es4 Current Load on es4 is WARNING: WARNING - load average: 1.98, 3.51, 3.25 [16:23:38] RECOVERY - es4 Current Load on es4 is OK: OK - load average: 2.31, 3.10, 3.13 [16:33:42] PROBLEM - es3 Puppet on es3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [16:34:33] PROBLEM - es3 Current Load on es3 is CRITICAL: CRITICAL - load average: 10.51, 5.18, 3.07 [16:35:41] RECOVERY - es3 Puppet on es3 is OK: OK: Puppet is currently enabled, last run 13 minutes ago with 0 failures [16:38:26] PROBLEM - es3 Current Load on es3 is WARNING: WARNING - load average: 1.43, 3.50, 2.89 [16:40:26] RECOVERY - es3 Current Load on es3 is OK: OK - load average: 0.38, 2.41, 2.56 [16:44:12] PROBLEM - es3 Puppet on es3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [16:44:43] PROBLEM - es3 Current Load on es3 is CRITICAL: CRITICAL - load average: 9.69, 7.36, 4.61 [16:46:12] RECOVERY - es3 Puppet on es3 is OK: OK: Puppet is currently enabled, last run 8 minutes ago with 0 failures [16:50:29] PROBLEM - es3 Current Load on es3 is WARNING: WARNING - load average: 0.56, 3.14, 3.61 [16:54:26] RECOVERY - es3 Current Load on es3 is OK: OK - load average: 1.79, 2.53, 3.25 [16:58:07] !log root@mw1:/srv/mediawiki/w/maintenance# sudo -u www-data php deleteBatch.php --wiki=cwarswiki page_deletion [16:58:11] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [17:00:00] PROBLEM - es4 Current Load on es4 is WARNING: WARNING - load average: 3.60, 3.31, 2.71 [17:01:58] PROBLEM - es4 Current Load on es4 is CRITICAL: CRITICAL - load average: 7.90, 5.07, 3.44 [17:02:05] [02miraheze/ssl] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/fjM4i [17:02:07] [02miraheze/ssl] 07paladox 03fbcd280 - Remove theliteratureproject.org ssl certificate domain is not pointing at our namespaces/mw-lb [17:02:08] [02ssl] 07paladox created branch 03paladox-patch-1 - 13https://git.io/vxP9L [17:02:10] [02ssl] 07paladox opened pull request 03#199: Remove theliteratureproject.org ssl certificate - 13https://git.io/fjM4P [17:02:43] [02miraheze/ssl] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-1/±0] 13https://git.io/fjM41 [17:02:45] [02miraheze/ssl] 07paladox 032be793c - Delete theliteratureproject.org.crt [17:02:46] [02ssl] 07paladox synchronize pull request 03#199: Remove theliteratureproject.org ssl certificate - 13https://git.io/fjM4P [17:02:51] [02ssl] 07paladox closed pull request 03#199: Remove theliteratureproject.org ssl certificate - 13https://git.io/fjM4P [17:02:52] [02miraheze/ssl] 07paladox pushed 031 commit to 03master [+0/-1/±1] 13https://git.io/fjM4M [17:02:54] [02miraheze/ssl] 07paladox 03eb2dc24 - Remove theliteratureproject.org ssl certificate (#199) * Remove theliteratureproject.org ssl certificate domain is not pointing at our namespaces/mw-lb * Delete theliteratureproject.org.crt [17:03:10] [02ssl] 07paladox deleted branch 03paladox-patch-1 - 13https://git.io/vxP9L [17:03:12] [02miraheze/ssl] 07paladox deleted branch 03paladox-patch-1 [17:03:55] PROBLEM - es4 Current Load on es4 is WARNING: WARNING - load average: 1.72, 3.71, 3.13 [17:04:15] [02ssl] 07paladox created branch 03paladox-patch-1 - 13https://git.io/vxP9L [17:04:17] [02miraheze/ssl] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/fjM4S [17:04:18] [02miraheze/ssl] 07paladox 034b52308 - Remove toonpedia.cf ssl certificate [17:04:20] [02ssl] 07paladox opened pull request 03#200: Remove toonpedia.cf ssl certificate - 13https://git.io/fjM49 [17:04:35] [02ssl] 07paladox edited pull request 03#200: Remove toonpedia.cf ssl certificate - 13https://git.io/fjM49 [17:04:48] [02miraheze/ssl] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-1/±0] 13https://git.io/fjM4H [17:04:50] [02miraheze/ssl] 07paladox 034db6339 - Delete toonpedia.cf.crt [17:04:51] [02ssl] 07paladox synchronize pull request 03#200: Remove toonpedia.cf ssl certificate - 13https://git.io/fjM49 [17:05:01] [02ssl] 07paladox closed pull request 03#200: Remove toonpedia.cf ssl certificate - 13https://git.io/fjM49 [17:05:02] [02miraheze/ssl] 07paladox pushed 031 commit to 03master [+0/-1/±1] 13https://git.io/fjM4Q [17:05:04] [02miraheze/ssl] 07paladox 03dd9baa3 - Remove toonpedia.cf ssl certificate (#200) * Remove toonpedia.cf ssl certificate * Delete toonpedia.cf.crt [17:05:40] [02ssl] 07paladox deleted branch 03paladox-patch-1 - 13https://git.io/vxP9L [17:05:42] [02miraheze/ssl] 07paladox deleted branch 03paladox-patch-1 [17:05:52] RECOVERY - es4 Current Load on es4 is OK: OK - load average: 0.62, 2.64, 2.81 [17:06:04] [02ssl] 07paladox created branch 03paladox-patch-1 - 13https://git.io/vxP9L [17:06:05] [02miraheze/ssl] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/fjM47 [17:06:07] [02miraheze/ssl] 07paladox 0321d99fb - Remove wiki.kybermc.tk ssl certificate Domain is not pointing at our nameserver/mw-lb [17:06:08] [02ssl] 07paladox opened pull request 03#201: Remove wiki.kybermc.tk ssl certificate - 13https://git.io/fjM45 [17:06:21] [02miraheze/ssl] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-1/±0] 13https://git.io/fjM4b [17:06:23] [02miraheze/ssl] 07paladox 03c5f3288 - Delete wiki.kybermc.tk.crt [17:06:24] [02ssl] 07paladox synchronize pull request 03#201: Remove wiki.kybermc.tk ssl certificate - 13https://git.io/fjM45 [17:06:30] [02ssl] 07paladox closed pull request 03#201: Remove wiki.kybermc.tk ssl certificate - 13https://git.io/fjM45 [17:06:31] [02miraheze/ssl] 07paladox pushed 031 commit to 03master [+0/-1/±1] 13https://git.io/fjM4N [17:06:33] [02miraheze/ssl] 07paladox 033604a5f - Remove wiki.kybermc.tk ssl certificate (#201) * Remove wiki.kybermc.tk ssl certificate Domain is not pointing at our nameserver/mw-lb * Delete wiki.kybermc.tk.crt [17:06:34] [02ssl] 07paladox deleted branch 03paladox-patch-1 - 13https://git.io/vxP9L [17:06:36] [02miraheze/ssl] 07paladox deleted branch 03paladox-patch-1 [17:07:00] PROBLEM - cp3 Puppet on cp3 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): File[theliteratureproject.org_private] [17:07:48] PROBLEM - test1 Current Load on test1 is CRITICAL: CRITICAL - load average: 2.24, 1.79, 1.47 [17:09:18] [02ssl] 07paladox created branch 03paladox-patch-1 - 13https://git.io/vxP9L [17:09:20] [02miraheze/ssl] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/fjM4p [17:09:21] [02miraheze/ssl] 07paladox 03c24530a - Remove wiki.pyxxl.tk ssl certificate Domain is no longer pointing to our nameserver/mw-lb [17:09:23] [02ssl] 07paladox opened pull request 03#202: Remove wiki.pyxxl.tk ssl certificate - 13https://git.io/fjM4h [17:09:41] [02miraheze/ssl] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-1/±0] 13https://git.io/fjM4j [17:09:42] [02miraheze/ssl] 07paladox 03daef7d4 - Delete wiki.pyxxl.tk.crt [17:09:44] [02ssl] 07paladox synchronize pull request 03#202: Remove wiki.pyxxl.tk ssl certificate - 13https://git.io/fjM4h [17:09:48] RECOVERY - test1 Current Load on test1 is OK: OK - load average: 1.67, 1.69, 1.46 [17:10:10] [02ssl] 07paladox closed pull request 03#202: Remove wiki.pyxxl.tk ssl certificate - 13https://git.io/fjM4h [17:10:11] [02miraheze/ssl] 07paladox pushed 031 commit to 03master [+0/-1/±1] 13https://git.io/fjMBe [17:10:13] [02miraheze/ssl] 07paladox 03d455332 - Remove wiki.pyxxl.tk ssl certificate (#202) * Remove wiki.pyxxl.tk ssl certificate Domain is no longer pointing to our nameserver/mw-lb * Delete wiki.pyxxl.tk.crt [17:10:14] [02ssl] 07paladox deleted branch 03paladox-patch-1 - 13https://git.io/vxP9L [17:10:16] [02miraheze/ssl] 07paladox deleted branch 03paladox-patch-1 [17:14:24] !log apt upgrade - puppet1 [17:14:28] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [17:15:50] PROBLEM - mw2 Puppet on mw2 is CRITICAL: CRITICAL: Puppet has 9 failures. Last run 3 minutes ago with 9 failures. Failed resources (up to 3 shown) [17:15:52] PROBLEM - cp2 Puppet on cp2 is CRITICAL: CRITICAL: Puppet has 232 failures. Last run 2 minutes ago with 232 failures. Failed resources (up to 3 shown) [17:16:04] PROBLEM - cp4 Puppet on cp4 is CRITICAL: CRITICAL: Puppet has 41 failures. Last run 3 minutes ago with 41 failures. Failed resources (up to 3 shown) [17:16:08] PROBLEM - es2 Puppet on es2 is CRITICAL: CRITICAL: Puppet has 17 failures. Last run 3 minutes ago with 17 failures. Failed resources (up to 3 shown) [17:16:14] PROBLEM - es1 Current Load on es1 is CRITICAL: CRITICAL - load average: 7.49, 5.41, 3.26 [17:16:35] PROBLEM - mw1 Puppet on mw1 is CRITICAL: CRITICAL: Puppet has 9 failures. Last run 3 minutes ago with 9 failures. Failed resources (up to 3 shown) [17:16:39] PROBLEM - mw3 Puppet on mw3 is CRITICAL: CRITICAL: Puppet has 33 failures. Last run 3 minutes ago with 33 failures. Failed resources (up to 3 shown) [17:17:03] PROBLEM - es3 Current Load on es3 is WARNING: WARNING - load average: 2.61, 3.53, 2.89 [17:19:00] RECOVERY - es3 Current Load on es3 is OK: OK - load average: 2.55, 3.19, 2.84 [17:19:11] PROBLEM - es4 Puppet on es4 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [17:19:33] PROBLEM - es1 Puppet on es1 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [17:20:12] PROBLEM - es1 Current Load on es1 is WARNING: WARNING - load average: 1.67, 3.55, 2.99 [17:22:10] RECOVERY - es1 Current Load on es1 is OK: OK - load average: 1.23, 2.83, 2.80 [17:23:00] PROBLEM - es3 Current Load on es3 is CRITICAL: CRITICAL - load average: 4.44, 4.50, 3.47 [17:23:44] RECOVERY - cp4 Puppet on cp4 is OK: OK: Puppet is currently enabled, last run 5 seconds ago with 0 failures [17:24:02] RECOVERY - es1 Puppet on es1 is OK: OK: Puppet is currently enabled, last run 35 seconds ago with 0 failures RECOVERY - es2 Puppet on es2 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [17:24:31] RECOVERY - mw3 Puppet on mw3 is OK: OK: Puppet is currently enabled, last run 41 seconds ago with 0 failures [17:24:43] RECOVERY - mw1 Puppet on mw1 is OK: OK: Puppet is currently enabled, last run 56 seconds ago with 0 failures [17:24:55] RECOVERY - cp3 Puppet on cp3 is OK: OK: Puppet is currently enabled, last run 7 seconds ago with 0 failures [17:24:56] RECOVERY - es3 Current Load on es3 is OK: OK - load average: 0.81, 3.13, 3.09 [17:24:56] RECOVERY - es4 Puppet on es4 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [17:25:23] RECOVERY - mw2 Puppet on mw2 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [17:26:08] RECOVERY - cp2 Puppet on cp2 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [17:33:37] PROBLEM - es1 Current Load on es1 is CRITICAL: CRITICAL - load average: 4.29, 2.89, 2.59 [17:33:58] PROBLEM - es4 Current Load on es4 is CRITICAL: CRITICAL - load average: 6.71, 3.66, 2.41 [17:34:28] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fjMBa [17:34:29] [02miraheze/puppet] 07paladox 0323b47a3 - Remove "elasticsearch1.miraheze.org" [17:35:33] !log apt upgrade - misc4 [17:35:36] RECOVERY - es1 Current Load on es1 is OK: OK - load average: 1.20, 2.24, 2.39 [17:35:37] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [17:35:59] PROBLEM - es4 Current Load on es4 is WARNING: WARNING - load average: 3.39, 3.70, 2.60 [17:36:54] !log apt upgrade - misc1 [17:36:58]