[00:03:02] PROBLEM - misc4 Current Load on misc4 is CRITICAL: CRITICAL - load average: 4.79, 3.38, 1.52 [00:03:53] PROBLEM - cp4 Current Load on cp4 is CRITICAL: CRITICAL - load average: 2.22, 2.62, 1.28 [00:04:59] RECOVERY - misc4 Current Load on misc4 is OK: OK - load average: 1.02, 2.44, 1.39 [00:05:53] PROBLEM - cp4 Current Load on cp4 is WARNING: WARNING - load average: 0.38, 1.79, 1.14 [00:07:53] RECOVERY - cp4 Current Load on cp4 is OK: OK - load average: 0.12, 1.22, 1.00 [00:44:27] Hi! Here is the list of currently open high priority tasks on Phabricator [00:44:34] No updates for 3 days - https://phabricator.miraheze.org/T4564 - Mediawiki internal error (DB query error) - authored by Kees_Langeveld, assigned to Reception123 [00:44:41] No updates for 9 days - https://phabricator.miraheze.org/T4547 - Fix WikiBase Client - authored by AmandaCath, assigned to None [00:44:47] No updates for 3 days - https://phabricator.miraheze.org/T4540 - Purchase db5 - authored by Reception123, assigned to None [00:44:54] No updates for 3 days - https://phabricator.miraheze.org/T4260 - Migrate all wikis to elasticsearch - authored by Southparkfan, assigned to None [00:50:13] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fjDDF [00:50:14] [02miraheze/services] 07MirahezeSSLBot 03cca3ae6 - BOT: Updating services config for wikis [01:05:04] RECOVERY - test1 Disk Space on test1 is OK: DISK OK - free space: / 5447 MB (13% inode=98%); [05:01:04] PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 6 datacenters are down: 107.191.126.23/cpweb, 2604:180:0:33b::2/cpweb, 128.199.139.216/cpweb, 2400:6180:0:d0::403:f001/cpweb, 81.4.109.133/cpweb, 2a00:d880:5:8ea::ebc7/cpweb [05:01:23] PROBLEM - misc3 Lizardfs Master Port 2 on misc3 is CRITICAL: connect to address 185.52.1.144 and port 9420: Connection refused [05:01:49] PROBLEM - misc3 Lizardfs Master Port 1 on misc3 is CRITICAL: connect to address 185.52.1.144 and port 9419: Connection refused [05:01:58] PROBLEM - misc3 Lizardfs Master Port 3 on misc3 is CRITICAL: connect to address 185.52.1.144 and port 9421: Connection refused [05:02:14] PROBLEM - cp4 HTTP 4xx/5xx ERROR Rate on cp4 is CRITICAL: CRITICAL - NGINX Error Rate is 76% [05:02:17] PROBLEM - cp3 Varnish Backends on cp3 is CRITICAL: 3 backends are down. mw1 mw2 mw3 [05:02:20] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 6 datacenters are down: 107.191.126.23/cpweb, 2604:180:0:33b::2/cpweb, 128.199.139.216/cpweb, 2400:6180:0:d0::403:f001/cpweb, 81.4.109.133/cpweb, 2a00:d880:5:8ea::ebc7/cpweb [05:02:39] PROBLEM - cp4 Varnish Backends on cp4 is CRITICAL: 3 backends are down. mw1 mw2 mw3 [05:02:47] PROBLEM - cp2 Varnish Backends on cp2 is CRITICAL: 3 backends are down. mw1 mw2 mw3 [05:03:23] RECOVERY - misc3 Lizardfs Master Port 2 on misc3 is OK: TCP OK - 0.001 second response time on 185.52.1.144 port 9420 [05:03:49] RECOVERY - misc3 Lizardfs Master Port 1 on misc3 is OK: TCP OK - 0.002 second response time on 185.52.1.144 port 9419 [05:03:58] RECOVERY - misc3 Lizardfs Master Port 3 on misc3 is OK: TCP OK - 0.001 second response time on 185.52.1.144 port 9421 [05:04:14] RECOVERY - cp4 HTTP 4xx/5xx ERROR Rate on cp4 is OK: OK - NGINX Error Rate is 14% [05:04:17] RECOVERY - cp3 Varnish Backends on cp3 is OK: All 5 backends are healthy [05:04:20] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [05:04:39] RECOVERY - cp4 Varnish Backends on cp4 is OK: All 5 backends are healthy [05:04:47] RECOVERY - cp2 Varnish Backends on cp2 is OK: All 5 backends are healthy [05:05:03] RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online [05:35:10] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fjD98 [05:35:12] [02miraheze/services] 07MirahezeSSLBot 039235408 - BOT: Updating services config for wikis [06:27:14] RECOVERY - cp3 Disk Space on cp3 is OK: DISK OK - free space: / 3170 MB (13% inode=94%); [08:38:33] Reception123: Still around? Eyes or a block needed on https://meta.miraheze.org/wiki/User_talk:2804:431:B724:964C:8859:46F1:241D:4179 [08:38:34] [ User talk:2804:431:B724:964C:8859:46F1:241D:4179 - Miraheze Meta ] - meta.miraheze.org [08:43:59] Meta admin needed: Pinging Reception123, PuppyKun, SPF|Cloud: ^^ [08:44:08] We should get an !admin ping [09:56:35] RhinosF1: thanks for notifying [09:57:08] Your final warning seems sufficient for now, if they keep adding nonsense a block of >= 3 days is warranted [10:01:01] SPF|Cloud: np [11:42:16] PROBLEM - mw1 Current Load on mw1 is CRITICAL: CRITICAL - load average: 8.48, 5.77, 4.66 [11:44:16] RECOVERY - mw1 Current Load on mw1 is OK: OK - load average: 3.71, 4.94, 4.50 [13:16:18] !log deleting es* instances [13:16:22] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [13:16:36] PROBLEM - es1 Puppet on es1 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [13:16:53] PROBLEM - es3 Puppet on es3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [13:17:10] PROBLEM - es2 Puppet on es2 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [13:18:30] PROBLEM - es4 Puppet on es4 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [13:18:53] PROBLEM - es3 Current Load on es3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [13:19:11] PROBLEM - es1 Disk Space on es1 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [13:19:29] PROBLEM - es2 Current Load on es2 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [13:19:44] PROBLEM - es1 Current Load on es1 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [13:19:57] PROBLEM - es2 Disk Space on es2 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [13:19:59] PROBLEM - es4 SSH on es4 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:20:17] PROBLEM - es2 SSH on es2 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:20:24] PROBLEM - Host es3 is DOWN: PING CRITICAL - Packet loss = 100% [13:20:38] PROBLEM - es4 Disk Space on es4 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [13:20:41] PROBLEM - es1 SSH on es1 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:20:49] PROBLEM - es4 Current Load on es4 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [13:21:30] PROBLEM - Host es4 is DOWN: PING CRITICAL - Packet loss = 100% [13:21:48] PROBLEM - Host es2 is DOWN: PING CRITICAL - Packet loss = 100% [13:22:16] PROBLEM - Host es1 is DOWN: PING CRITICAL - Packet loss = 100% [13:23:25] RECOVERY - misc4 Prometheus on misc4 is OK: TCP OK - 0.001 second response time on 185.52.3.121 port 9090 [13:49:00] Hello Skoppy! If you have any questions feel free to ask and someone should answer soon. [14:21:27] RECOVERY - Host es1 is UP: PING WARNING - Packet loss = 80%, RTA = 78.22 ms [14:39:19] PROBLEM - es1 SSH on es1 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:41:08] PROBLEM - Host es1 is DOWN: PING CRITICAL - Packet loss = 100% [14:50:35] RECOVERY - Host es1 is UP: PING OK - Packet loss = 0%, RTA = 78.08 ms [14:59:49] PROBLEM - es1 SSH on es1 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:00:05] PROBLEM - Host es1 is DOWN: PING CRITICAL - Packet loss = 100% [15:09:16] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+2/-0/±14] 13https://git.io/fjDpO [15:09:17] [02miraheze/puppet] 07paladox 03a32d87a - Update mailalias to 1.5.0 [16:36:46] PROBLEM - mw3 Current Load on mw3 is CRITICAL: CRITICAL - load average: 10.53, 9.22, 5.71 [16:40:39] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 4.58, 7.10, 5.66 [16:42:38] RECOVERY - mw3 Current Load on mw3 is OK: OK - load average: 3.59, 5.89, 5.39 [17:17:02] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fjDjH [17:17:04] [02miraheze/puppet] 07paladox 03ccc6daa - salt: Update to 2019.2 release [17:18:30] !log apt-upgrade - misc4 [17:18:35] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [17:20:37] PROBLEM - misc4 Puppet on misc4 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 8 minutes ago with 0 failures [17:22:19] [02puppet] 07paladox created branch 03paladox-patch-5 - 13https://git.io/vbiAS [17:22:21] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-5 [+0/-0/±1] 13https://git.io/fjDjp [17:22:22] [02miraheze/puppet] 07paladox 03076cca7 - Update salt master configuration file [17:22:24] [02puppet] 07paladox opened pull request 03#1051: Update salt master configuration file - 13https://git.io/fjyee [17:30:33] [02puppet] 07paladox synchronize pull request 03#1051: Update salt master configuration file - 13https://git.io/fjyee [17:30:34] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-5 [+0/-0/±1] 13https://git.io/fjyeq [17:30:36] [02miraheze/puppet] 07paladox 038a99824 - Update master.erb [17:33:22] [02puppet] 07paladox synchronize pull request 03#1051: Update salt master configuration file - 13https://git.io/fjyee [17:33:24] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-5 [+0/-0/±1] 13https://git.io/fjyeO [17:33:25] [02miraheze/puppet] 07paladox 03927ae97 - Update master.erb [17:36:41] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-5 [+0/-0/±1] 13https://git.io/fjye3 [17:36:42] [02miraheze/puppet] 07paladox 03a525146 - Update minion.pp [17:36:44] [02puppet] 07paladox synchronize pull request 03#1051: Update salt master configuration file - 13https://git.io/fjyee [17:38:38] RECOVERY - misc4 Puppet on misc4 is OK: OK: Puppet is currently enabled, last run 52 seconds ago with 0 failures [17:41:22] PROBLEM - misc4 Current Load on misc4 is CRITICAL: CRITICAL - load average: 10.35, 5.01, 2.30 [17:43:22] PROBLEM - misc4 Current Load on misc4 is WARNING: WARNING - load average: 2.69, 4.00, 2.27 [17:45:22] RECOVERY - misc4 Current Load on misc4 is OK: OK - load average: 1.42, 3.03, 2.12 [17:52:36] PROBLEM - misc4 Puppet on misc4 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 10 minutes ago with 0 failures [17:53:11] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fjyeE [17:53:13] [02miraheze/puppet] 07paladox 03a7b0570 - Update minion_master.pub [17:56:36] RECOVERY - misc4 Puppet on misc4 is OK: OK: Puppet is currently enabled, last run 43 seconds ago with 0 failures [17:57:32] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fjyew [17:57:33] [02miraheze/puppet] 07paladox 038db6e99 - salt: Fix support for python3 in keys.py [17:59:22] PROBLEM - misc4 Current Load on misc4 is CRITICAL: CRITICAL - load average: 6.35, 3.47, 2.34 [18:01:22] RECOVERY - misc4 Current Load on misc4 is OK: OK - load average: 2.24, 3.30, 2.45 [18:04:26] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fjyei [18:04:27] [02miraheze/puppet] 07paladox 0399f1b9d - salt: Update pub key [18:12:11] [02puppet] 07paladox created branch 03paladox-patch-6 - 13https://git.io/vbiAS [18:12:13] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-6 [+0/-0/±1] 13https://git.io/fjyeQ [18:12:14] [02miraheze/puppet] 07paladox 03f9106b9 - misc4: Set role::salt::minions::salt_master_key to true [18:12:16] [02puppet] 07paladox opened pull request 03#1052: misc4: Set role::salt::minions::salt_master_key to true - 13https://git.io/fjye7 [18:12:32] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-6 [+0/-0/±1] 13https://git.io/fjyeF [18:12:34] [02miraheze/puppet] 07paladox 030d82160 - Update misc4.yaml [18:12:35] [02puppet] 07paladox synchronize pull request 03#1052: misc4: Set role::salt::minions::salt_master_key to true - 13https://git.io/fjye7 [18:12:39] [02puppet] 07paladox closed pull request 03#1052: misc4: Set role::salt::minions::salt_master_key to true - 13https://git.io/fjye7 [18:12:41] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±2] 13https://git.io/fjyeb [18:12:42] [02miraheze/puppet] 07paladox 033237340 - misc4: Set role::salt::minions::salt_master_key to true (#1052) * misc4: Set role::salt::minions::salt_master_key to true * Update misc4.yaml [18:12:44] [02puppet] 07paladox deleted branch 03paladox-patch-6 - 13https://git.io/vbiAS [18:12:45] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-6 [18:12:48] [02puppet] 07paladox closed pull request 03#1033: Rename es_heap to es_jvm_options - 13https://git.io/fj1as [18:12:50] [02puppet] 07paladox deleted branch 03paladox-patch-2 - 13https://git.io/vbiAS [18:12:52] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-2 [18:13:00] [02puppet] 07paladox closed pull request 03#1050: Add grant to add REPLICATION CLIENT to wikiadmin - 13https://git.io/fjMMr [18:13:02] [02puppet] 07paladox deleted branch 03paladox-patch-4 - 13https://git.io/vbiAS [18:13:03] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-4 [18:13:08] [02puppet] 07paladox closed pull request 03#1049: Add REPLICATION CLIENT to mediawiki grants - 13https://git.io/fjMMB [18:13:10] [02puppet] 07paladox deleted branch 03paladox-patch-1 - 13https://git.io/vbiAS [18:13:12] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-1 [18:13:55] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fjyeN [18:13:57] [02miraheze/puppet] 07paladox 03dddf4ca - misc4: Fix syntax [18:15:13] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fjyeA [18:15:14] [02miraheze/services] 07MirahezeSSLBot 035cf9c6d - BOT: Updating services config for wikis [18:15:55] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fjyej [18:15:57] [02miraheze/puppet] 07paladox 03a8546dd - salt: update master finger print [18:19:46] [02puppet] 07paladox created branch 03paladox-patch-1 - 13https://git.io/vbiAS [18:19:48] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/fjyvI [18:19:49] [02miraheze/puppet] 07paladox 03952cc85 - Include ssl::wildcard in minion.pp and master.pp [18:19:51] [02puppet] 07paladox opened pull request 03#1053: Include ssl::wildcard in minion.pp and master.pp - 13https://git.io/fjyvL [18:20:12] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/fjyvq [18:20:14] [02miraheze/puppet] 07paladox 03b83d43a - Update master.pp [18:20:15] [02puppet] 07paladox synchronize pull request 03#1053: Include ssl::wildcard in minion.pp and master.pp - 13https://git.io/fjyvL [18:20:22] [02puppet] 07paladox closed pull request 03#1053: Include ssl::wildcard in minion.pp and master.pp - 13https://git.io/fjyvL [18:20:23] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±2] 13https://git.io/fjyvm [18:20:25] [02miraheze/puppet] 07paladox 033247d8f - Include ssl::wildcard in minion.pp and master.pp (#1053) * Include ssl::wildcard in minion.pp and master.pp * Update master.pp [18:23:16] [02puppet] 07paladox closed pull request 03#1051: Update salt master configuration file - 13https://git.io/fjyee [18:23:18] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±2] 13https://git.io/fjyvZ [18:23:19] [02miraheze/puppet] 07paladox 0380fda1d - Update salt master configuration file (#1051) * Update salt master configuration file * Update master.erb * Update master.erb * Update minion.pp [18:23:30] [02puppet] 07paladox deleted branch 03paladox-patch-5 - 13https://git.io/vbiAS [18:23:32] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-5 [18:32:41] PROBLEM - misc3 Puppet on misc3 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 10 minutes ago with 0 failures [18:38:43] !log accidentally removed /var/log on misc3 (was ment to remove /var/log/salt but missed 'salt') [18:38:48] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [18:40:41] RECOVERY - misc3 Puppet on misc3 is OK: OK: Puppet is currently enabled, last run 42 seconds ago with 0 failures [18:43:29] !log upgrading salt accross all hosts [18:43:37] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [20:26:02] !log reboot misc4 [20:26:07] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [20:28:22] PROBLEM - misc4 Prometheus on misc4 is CRITICAL: connect to address 185.52.3.121 and port 9090: Connection refused [20:30:22] RECOVERY - misc4 Prometheus on misc4 is OK: TCP OK - 0.002 second response time on 185.52.3.121 port 9090 [20:39:58] PROBLEM - cp3 Disk Space on cp3 is WARNING: DISK WARNING - free space: / 2650 MB (10% inode=94%); [20:51:35] RhinosF1: that would ping me as I'm an admin on enwiki [22:26:52] PROBLEM - cp3 Puppet on cp3 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[ufw-allow-tcp-from-185.52.3.121-to-any-port-9131] [22:34:51] RECOVERY - cp3 Puppet on cp3 is OK: OK: Puppet is currently enabled, last run 36 seconds ago with 0 failures