[00:18:06] PROBLEM - mw2 Puppet on mw2 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:26:09] RECOVERY - mw2 Puppet on mw2 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [00:30:53] Zppix: What are the requirements of helping out with CVT [00:42:05] Examknow: you can anytime by combating vandalism to become CVT you have to be trustworthy, have a good reputation in miraheze, and be able to properly determine vandalism [00:42:20] ok [00:42:33] What is the best way to combat vandalism [00:42:42] I dont see much of it on meta [00:43:23] Examknow: You have to look for it on different wikis, that you contribute too [00:43:31] ok [04:12:06] PROBLEM - cp3 Varnish Backends on cp3 is CRITICAL: 1 backends are down. mw1 [04:14:44] RECOVERY - cp3 Varnish Backends on cp3 is OK: All 5 backends are healthy [06:26:11] RECOVERY - cp3 Disk Space on cp3 is OK: DISK OK - free space: / 3099 MB (12% inode=94%); [06:50:10] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JeWaK [06:50:11] [02miraheze/services] 07MirahezeSSLBot 03f65e69a - BOT: Updating services config for wikis [08:07:49] PROBLEM - lizardfs5 Puppet on lizardfs5 is CRITICAL: CRITICAL: Puppet has 12 failures. Last run 5 minutes ago with 12 failures. Failed resources (up to 3 shown) [08:08:14] PROBLEM - mw3 Puppet on mw3 is CRITICAL: CRITICAL: Puppet has 217 failures. Last run 5 minutes ago with 217 failures. Failed resources (up to 3 shown) [08:08:18] PROBLEM - test1 Puppet on test1 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [08:08:37] PROBLEM - mw2 Puppet on mw2 is CRITICAL: CRITICAL: Puppet has 214 failures. Last run 6 minutes ago with 214 failures. Failed resources (up to 3 shown) [08:08:48] PROBLEM - bacula1 Puppet on bacula1 is CRITICAL: CRITICAL: Puppet has 13 failures. Last run 6 minutes ago with 13 failures. Failed resources (up to 3 shown) [08:09:09] PROBLEM - cp2 Puppet on cp2 is CRITICAL: CRITICAL: Puppet has 203 failures. Last run 6 minutes ago with 203 failures. Failed resources (up to 3 shown) [08:09:10] PROBLEM - misc4 Puppet on misc4 is CRITICAL: CRITICAL: Puppet has 32 failures. Last run 7 minutes ago with 32 failures. Failed resources (up to 3 shown) [08:09:16] PROBLEM - cp3 Puppet on cp3 is CRITICAL: CRITICAL: Puppet has 202 failures. Last run 6 minutes ago with 202 failures. Failed resources (up to 3 shown) [08:09:28] PROBLEM - mw1 Puppet on mw1 is CRITICAL: CRITICAL: Puppet has 220 failures. Last run 7 minutes ago with 220 failures. Failed resources (up to 3 shown) [08:09:29] PROBLEM - misc2 Puppet on misc2 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [08:09:32] PROBLEM - ns1 Puppet on ns1 is CRITICAL: CRITICAL: Puppet has 14 failures. Last run 7 minutes ago with 14 failures. Failed resources (up to 3 shown) [08:09:32] PROBLEM - lizardfs4 Puppet on lizardfs4 is CRITICAL: CRITICAL: Puppet has 12 failures. Last run 7 minutes ago with 12 failures. Failed resources (up to 3 shown) [08:09:52] PROBLEM - misc3 Puppet on misc3 is CRITICAL: CRITICAL: Puppet has 22 failures. Last run 7 minutes ago with 22 failures. Failed resources (up to 3 shown) [08:09:53] PROBLEM - db4 Puppet on db4 is CRITICAL: CRITICAL: Puppet has 16 failures. Last run 7 minutes ago with 16 failures. Failed resources (up to 3 shown) [08:10:01] PROBLEM - misc1 Puppet on misc1 is CRITICAL: CRITICAL: Puppet has 51 failures. Last run 8 minutes ago with 51 failures. Failed resources (up to 3 shown) [08:10:05] PROBLEM - cp4 Puppet on cp4 is CRITICAL: CRITICAL: Puppet has 204 failures. Last run 8 minutes ago with 204 failures. Failed resources (up to 3 shown) [08:11:08] PROBLEM - db5 Puppet on db5 is CRITICAL: CRITICAL: Puppet has 15 failures. Last run 9 minutes ago with 15 failures. Failed resources (up to 3 shown) [08:11:10] PROBLEM - puppet1 Puppet on puppet1 is CRITICAL: CRITICAL: Puppet has 20 failures. Last run 9 minutes ago with 20 failures. Failed resources (up to 3 shown) [08:13:52] RECOVERY - lizardfs5 Puppet on lizardfs5 is OK: OK: Puppet is currently enabled, last run 24 seconds ago with 0 failures [08:13:55] RECOVERY - db5 Puppet on db5 is OK: OK: Puppet is currently enabled, last run 23 seconds ago with 0 failures [08:13:56] RECOVERY - puppet1 Puppet on puppet1 is OK: OK: Puppet is currently enabled, last run 18 seconds ago with 0 failures [08:14:36] RECOVERY - bacula1 Puppet on bacula1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [08:14:48] RECOVERY - misc4 Puppet on misc4 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [08:15:11] RECOVERY - mw1 Puppet on mw1 is OK: OK: Puppet is currently enabled, last run 33 seconds ago with 0 failures [08:15:11] RECOVERY - lizardfs4 Puppet on lizardfs4 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [08:15:12] RECOVERY - misc2 Puppet on misc2 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [08:15:14] RECOVERY - ns1 Puppet on ns1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [08:15:34] RECOVERY - db4 Puppet on db4 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [08:15:35] RECOVERY - misc3 Puppet on misc3 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [08:15:43] RECOVERY - misc1 Puppet on misc1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [08:15:43] RECOVERY - cp4 Puppet on cp4 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [08:16:47] RECOVERY - mw3 Puppet on mw3 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [08:16:52] RECOVERY - test1 Puppet on test1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [08:16:57] RECOVERY - mw2 Puppet on mw2 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [08:17:22] RECOVERY - cp3 Puppet on cp3 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [08:17:27] RECOVERY - cp2 Puppet on cp2 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [09:33:15] PROBLEM - cp4 Stunnel Http for mw2 on cp4 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [09:33:15] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 6 datacenters are down: 107.191.126.23/cpweb, 2604:180:0:33b::2/cpweb, 128.199.139.216/cpweb, 2400:6180:0:d0::403:f001/cpweb, 81.4.109.133/cpweb, 2a00:d880:5:8ea::ebc7/cpweb [09:34:07] PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 5 datacenters are down: 107.191.126.23/cpweb, 2604:180:0:33b::2/cpweb, 128.199.139.216/cpweb, 2400:6180:0:d0::403:f001/cpweb, 81.4.109.133/cpweb [09:34:41] PROBLEM - cp2 Stunnel Http for mw1 on cp2 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [09:34:41] PROBLEM - cp4 Varnish Backends on cp4 is CRITICAL: 1 backends are down. mw1 [09:36:05] PROBLEM - cp2 Varnish Backends on cp2 is CRITICAL: 3 backends are down. mw1 mw2 mw3 [09:36:43] PROBLEM - cp3 Varnish Backends on cp3 is CRITICAL: 1 backends are down. mw3 [09:37:03] RECOVERY - cp4 Stunnel Http for mw2 on cp4 is OK: HTTP OK: HTTP/1.1 200 OK - 24592 bytes in 3.765 second response time [09:38:08] RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online [09:38:24] RECOVERY - cp2 Stunnel Http for mw1 on cp2 is OK: HTTP OK: HTTP/1.1 200 OK - 24570 bytes in 0.389 second response time [09:38:27] RECOVERY - cp4 Varnish Backends on cp4 is OK: All 5 backends are healthy [09:38:59] RECOVERY - cp2 Varnish Backends on cp2 is OK: All 5 backends are healthy [09:39:28] RECOVERY - cp3 Varnish Backends on cp3 is OK: All 5 backends are healthy [09:39:47] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [12:45:09] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JeW6N [12:45:11] [02miraheze/services] 07MirahezeSSLBot 03f24e64b - BOT: Updating services config for wikis [13:30:11] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JeWi5 [13:30:13] [02miraheze/services] 07MirahezeSSLBot 032eea031 - BOT: Updating services config for wikis [14:42:29] 503 Backend fetch failed [14:43:49] PROBLEM - cp4 Varnish Backends on cp4 is CRITICAL: 2 backends are down. mw1 mw2 [14:44:02] PROBLEM - cp3 Varnish Backends on cp3 is CRITICAL: 3 backends are down. mw1 mw2 mw3 [14:44:15] PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 6 datacenters are down: 107.191.126.23/cpweb, 2604:180:0:33b::2/cpweb, 128.199.139.216/cpweb, 2400:6180:0:d0::403:f001/cpweb, 81.4.109.133/cpweb, 2a00:d880:5:8ea::ebc7/cpweb [14:46:49] RECOVERY - cp4 Varnish Backends on cp4 is OK: All 5 backends are healthy [14:47:01] RECOVERY - cp3 Varnish Backends on cp3 is OK: All 5 backends are healthy [14:47:04] BurningPrincess: try now [14:47:07] RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online [16:03:33] 503s... [16:04:01] it should be back up now, sorry :( [16:04:06] (works for me) [16:15:39] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 3 datacenters are down: 2604:180:0:33b::2/cpweb, 128.199.139.216/cpweb, 2400:6180:0:d0::403:f001/cpweb [16:18:28] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [16:51:03] PROBLEM - cp3 Varnish Backends on cp3 is CRITICAL: 1 backends are down. mw2 [16:51:37] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 4 datacenters are down: 2604:180:0:33b::2/cpweb, 128.199.139.216/cpweb, 2400:6180:0:d0::403:f001/cpweb, 81.4.109.133/cpweb [16:52:44] PROBLEM - cp4 Varnish Backends on cp4 is CRITICAL: 2 backends are down. mw1 mw2 [16:52:45] PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 5 datacenters are down: 107.191.126.23/cpweb, 128.199.139.216/cpweb, 2400:6180:0:d0::403:f001/cpweb, 81.4.109.133/cpweb, 2a00:d880:5:8ea::ebc7/cpweb [16:53:01] PROBLEM - cp2 Varnish Backends on cp2 is CRITICAL: 2 backends are down. mw1 mw2 [16:56:03] RECOVERY - cp4 Varnish Backends on cp4 is OK: All 5 backends are healthy [16:56:03] RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online [16:56:24] RECOVERY - cp2 Varnish Backends on cp2 is OK: All 5 backends are healthy [16:57:23] RECOVERY - cp3 Varnish Backends on cp3 is OK: All 5 backends are healthy [16:57:58] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [17:03:37] Miraheze seems a bit slow loading to me [17:05:09] PROBLEM - wikiverte.pl - LetsEncrypt on sslhost is CRITICAL: Name or service not knownHTTP CRITICAL - Unable to open TCP socket [17:06:59] looking [17:07:37] RECOVERY - wikiverte.pl - LetsEncrypt on sslhost is OK: OK - Certificate 'wikiverte.pl' will expire on Fri 01 Nov 2019 07:53:18 PM GMT +0000. [17:16:25] PROBLEM - wikiverte.pl - LetsEncrypt on sslhost is CRITICAL: Name or service not knownHTTP CRITICAL - Unable to open TCP socket [17:28:20] Well, it was slow, seem sokay now [18:07:58] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 3 datacenters are down: 128.199.139.216/cpweb, 2400:6180:0:d0::403:f001/cpweb, 2a00:d880:5:8ea::ebc7/cpweb [18:07:59] PROBLEM - cp2 Varnish Backends on cp2 is CRITICAL: 1 backends are down. mw2 [18:11:15] RECOVERY - cp2 Varnish Backends on cp2 is OK: All 5 backends are healthy [18:11:15] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [18:41:40] PROBLEM - cp3 Varnish Backends on cp3 is CRITICAL: 1 backends are down. mw2 [18:43:07] PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 6 datacenters are down: 107.191.126.23/cpweb, 2604:180:0:33b::2/cpweb, 128.199.139.216/cpweb, 2400:6180:0:d0::403:f001/cpweb, 81.4.109.133/cpweb, 2a00:d880:5:8ea::ebc7/cpweb [18:43:34] PROBLEM - cp4 Varnish Backends on cp4 is CRITICAL: 3 backends are down. mw1 mw2 mw3 [18:43:52] PROBLEM - cp2 Varnish Backends on cp2 is CRITICAL: 2 backends are down. mw2 mw3 [18:51:01] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 6 datacenters are down: 107.191.126.23/cpweb, 2604:180:0:33b::2/cpweb, 128.199.139.216/cpweb, 2400:6180:0:d0::403:f001/cpweb, 81.4.109.133/cpweb, 2a00:d880:5:8ea::ebc7/cpweb [18:53:45] PROBLEM - cp4 Stunnel Http for mw1 on cp4 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [18:54:34] 504 Gateway Time-out [18:55:18] paladox ? [18:55:31] that'll be because you hit the timeout [18:55:49] PROBLEM - cp2 Stunnel Http for mw3 on cp2 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [18:58:11] RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online [18:58:27] RECOVERY - cp2 Varnish Backends on cp2 is OK: All 5 backends are healthy [18:58:33] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [19:00:25] !log depool and repool mw[23] [19:00:33] !log hack fstab on mw[23] [19:01:01] RECOVERY - cp4 Stunnel Http for mw1 on cp4 is OK: HTTP OK: HTTP/1.1 200 OK - 24586 bytes in 0.004 second response time [19:02:36] RECOVERY - cp2 Stunnel Http for mw3 on cp2 is OK: HTTP OK: HTTP/1.1 200 OK - 24570 bytes in 0.389 second response time [19:03:29] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [19:04:27] RECOVERY - cp4 Varnish Backends on cp4 is OK: All 5 backends are healthy [19:06:06] RECOVERY - cp3 Varnish Backends on cp3 is OK: All 5 backends are healthy [19:08:04] PROBLEM - mw3 Puppet on mw3 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 5 minutes ago with 0 failures [19:15:00] PROBLEM - cp3 Varnish Backends on cp3 is CRITICAL: 2 backends are down. mw1 mw2 [19:15:08] PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 6 datacenters are down: 107.191.126.23/cpweb, 2604:180:0:33b::2/cpweb, 128.199.139.216/cpweb, 2400:6180:0:d0::403:f001/cpweb, 81.4.109.133/cpweb, 2a00:d880:5:8ea::ebc7/cpweb [19:15:18] PROBLEM - cp2 HTTP 4xx/5xx ERROR Rate on cp2 is WARNING: WARNING - NGINX Error Rate is 58% [19:15:18] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 6 datacenters are down: 107.191.126.23/cpweb, 2604:180:0:33b::2/cpweb, 128.199.139.216/cpweb, 2400:6180:0:d0::403:f001/cpweb, 81.4.109.133/cpweb, 2a00:d880:5:8ea::ebc7/cpweb [19:16:16] PROBLEM - cp2 Varnish Backends on cp2 is CRITICAL: 1 backends are down. mw2 [19:16:59] PROBLEM - misc3 Current Load on misc3 is WARNING: WARNING - load average: 2.14, 3.74, 2.78 [19:17:25] RECOVERY - cp3 Varnish Backends on cp3 is OK: All 5 backends are healthy [19:17:36] RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online [19:17:47] RECOVERY - cp2 HTTP 4xx/5xx ERROR Rate on cp2 is OK: OK - NGINX Error Rate is 5% [19:17:49] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [19:18:45] RECOVERY - cp2 Varnish Backends on cp2 is OK: All 5 backends are healthy [19:19:33] RECOVERY - misc3 Current Load on misc3 is OK: OK - load average: 0.79, 2.57, 2.48 [19:21:10] PROBLEM - mw2 Puppet on mw2 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 8 minutes ago with 0 failures [19:43:14] PROBLEM - cp3 Disk Space on cp3 is WARNING: DISK WARNING - free space: / 2649 MB (10% inode=94%); [19:57:55] PROBLEM - cp4 Varnish Backends on cp4 is CRITICAL: 2 backends are down. mw1 mw3 [19:59:23] PROBLEM - cp3 Varnish Backends on cp3 is CRITICAL: 3 backends are down. mw1 mw2 mw3 [19:59:52] PROBLEM - cp2 Varnish Backends on cp2 is CRITICAL: 1 backends are down. mw1 [20:01:48] PROBLEM - cp4 Stunnel Http for mw2 on cp4 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [20:02:24] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 6 datacenters are down: 107.191.126.23/cpweb, 2604:180:0:33b::2/cpweb, 128.199.139.216/cpweb, 2400:6180:0:d0::403:f001/cpweb, 81.4.109.133/cpweb, 2a00:d880:5:8ea::ebc7/cpweb [20:02:25] PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 6 datacenters are down: 107.191.126.23/cpweb, 2604:180:0:33b::2/cpweb, 128.199.139.216/cpweb, 2400:6180:0:d0::403:f001/cpweb, 81.4.109.133/cpweb, 2a00:d880:5:8ea::ebc7/cpweb [20:03:19] looking [20:04:01] PROBLEM - cp3 Stunnel Http for mw2 on cp3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [20:07:16] RECOVERY - cp3 Stunnel Http for mw2 on cp3 is OK: HTTP OK: HTTP/1.1 200 OK - 24586 bytes in 0.641 second response time [20:08:41] RECOVERY - cp4 Stunnel Http for mw2 on cp4 is OK: HTTP OK: HTTP/1.1 200 OK - 24586 bytes in 0.004 second response time [20:11:33] RECOVERY - cp4 Varnish Backends on cp4 is OK: All 5 backends are healthy [20:12:05] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [20:12:07] RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online [20:12:27] RECOVERY - cp3 Varnish Backends on cp3 is OK: All 5 backends are healthy [20:12:58] RECOVERY - cp2 Varnish Backends on cp2 is OK: All 5 backends are healthy [20:36:56] PROBLEM - cp3 Varnish Backends on cp3 is CRITICAL: 1 backends are down. mw2 [20:37:00] PROBLEM - cp2 Varnish Backends on cp2 is CRITICAL: 1 backends are down. mw2 [20:38:58] PROBLEM - cp4 Varnish Backends on cp4 is CRITICAL: 2 backends are down. mw1 mw2 [20:39:09] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 4 datacenters are down: 107.191.126.23/cpweb, 2604:180:0:33b::2/cpweb, 81.4.109.133/cpweb, 2a00:d880:5:8ea::ebc7/cpweb [20:39:16] PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 6 datacenters are down: 107.191.126.23/cpweb, 2604:180:0:33b::2/cpweb, 128.199.139.216/cpweb, 2400:6180:0:d0::403:f001/cpweb, 81.4.109.133/cpweb, 2a00:d880:5:8ea::ebc7/cpweb [20:41:57] RECOVERY - cp4 Varnish Backends on cp4 is OK: All 5 backends are healthy [20:41:58] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [20:42:05] RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online [20:46:34] What does all this mean? [20:46:46] RECOVERY - cp3 Varnish Backends on cp3 is OK: All 5 backends are healthy [20:46:58] It's telling us when mw[123] stops working. [20:47:00] RECOVERY - cp2 Varnish Backends on cp2 is OK: All 5 backends are healthy [20:47:14] one of the checks tells us that our caching proxy get depooled too. [20:48:30] It seems to be working/not working a lot [20:49:20] yeh [20:49:25] * paladox looking for a solution [20:50:44] !log hack puppet on puppet1 [20:50:51] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [20:52:14] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JeWH8 [20:52:14] Hello Not-def6! If you have any questions, feel free to ask and someone should answer soon. [20:52:15] [02miraheze/puppet] 07paladox 03822fc50 - lizardfs: Allow misc3 to mount to the master [20:52:57] !log reload lizardfs-master [20:53:15] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [20:56:05] Zppix: can you do a regex like Not-* for seen on welcome messages? is that possible? [21:09:19] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 6 datacenters are down: 107.191.126.23/cpweb, 2604:180:0:33b::2/cpweb, 128.199.139.216/cpweb, 2400:6180:0:d0::403:f001/cpweb, 81.4.109.133/cpweb, 2a00:d880:5:8ea::ebc7/cpweb [21:09:19] PROBLEM - cp4 Varnish Backends on cp4 is CRITICAL: 3 backends are down. mw1 mw2 mw3 [21:09:20] PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 6 datacenters are down: 107.191.126.23/cpweb, 2604:180:0:33b::2/cpweb, 128.199.139.216/cpweb, 2400:6180:0:d0::403:f001/cpweb, 81.4.109.133/cpweb, 2a00:d880:5:8ea::ebc7/cpweb [21:09:51] PROBLEM - cp4 Stunnel Http for mw2 on cp4 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [21:11:20] PROBLEM - cp3 Varnish Backends on cp3 is CRITICAL: 3 backends are down. mw1 mw2 mw3 [21:11:35] PROBLEM - cp3 Stunnel Http for mw1 on cp3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [21:11:44] PROBLEM - cp2 Varnish Backends on cp2 is CRITICAL: 1 backends are down. mw1 [21:14:02] RECOVERY - cp4 Stunnel Http for mw2 on cp4 is OK: HTTP OK: HTTP/1.1 200 OK - 24592 bytes in 7.691 second response time [21:15:08] RECOVERY - cp3 Stunnel Http for mw1 on cp3 is OK: HTTP OK: HTTP/1.1 200 OK - 24570 bytes in 1.899 second response time [21:17:43] RECOVERY - cp3 Varnish Backends on cp3 is OK: All 5 backends are healthy [21:18:06] RECOVERY - cp2 Varnish Backends on cp2 is OK: All 5 backends are healthy [21:19:27] RECOVERY - cp4 Varnish Backends on cp4 is OK: All 5 backends are healthy [21:19:28] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [21:19:30] RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online [21:21:50] .status mhtest offline [21:21:54] RhinosF1 updating User:RhinosF1/Status! [21:22:00] RhinosF1: Done! [21:22:25] .mh test User:RhinosF1 [21:22:25] https://test.miraheze.org/wiki/User:RhinosF1 [21:23:37] ^ will rewrite .status to hopefully be a tad faster [21:37:30] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 6 datacenters are down: 107.191.126.23/cpweb, 2604:180:0:33b::2/cpweb, 128.199.139.216/cpweb, 2400:6180:0:d0::403:f001/cpweb, 81.4.109.133/cpweb, 2a00:d880:5:8ea::ebc7/cpweb [21:37:35] PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 6 datacenters are down: 107.191.126.23/cpweb, 2604:180:0:33b::2/cpweb, 128.199.139.216/cpweb, 2400:6180:0:d0::403:f001/cpweb, 81.4.109.133/cpweb, 2a00:d880:5:8ea::ebc7/cpweb [21:37:55] PROBLEM - cp4 Varnish Backends on cp4 is CRITICAL: 1 backends are down. mw2 [21:39:41] PROBLEM - cp3 Varnish Backends on cp3 is CRITICAL: 2 backends are down. mw2 mw3 [21:39:58] PROBLEM - cp3 Stunnel Http for mw2 on cp3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [21:40:05] PROBLEM - misc3 Puppet on misc3 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 6 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[nfs-kernel-server] [21:40:11] PROBLEM - cp2 Varnish Backends on cp2 is CRITICAL: 1 backends are down. mw2 [21:40:44] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [21:40:48] RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online [21:41:09] RECOVERY - cp4 Varnish Backends on cp4 is OK: All 5 backends are healthy [21:42:46] RECOVERY - cp3 Varnish Backends on cp3 is OK: All 5 backends are healthy [21:42:51] RECOVERY - cp3 Stunnel Http for mw2 on cp3 is OK: HTTP OK: HTTP/1.1 200 OK - 24570 bytes in 0.637 second response time [21:42:57] RECOVERY - misc3 Puppet on misc3 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [21:42:59] RECOVERY - cp2 Varnish Backends on cp2 is OK: All 5 backends are healthy [21:48:18] PROBLEM - test1 Puppet on test1 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 5 minutes ago with 1 failures. Failed resources (up to 3 shown): Mount[/mnt/static] [22:14:00] hi guys [22:23:31] RECOVERY - test1 Puppet on test1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [22:33:25] [02puppet] 07paladox created branch 03paladox-patch-2 - 13https://git.io/vbiAS [22:33:26] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-2 [+1/-0/±0] 13https://git.io/JeW5m [22:33:28] [02miraheze/puppet] 07paladox 0329fa628 - Reintroduce nfs We will connect nfs to the lizard master which will have a mount which connects to the lizard master. [22:33:29] [02puppet] 07paladox opened pull request 03#1095: Reintroduce nfs - 13https://git.io/JeW5Y [22:34:08] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-2 [+1/-0/±0] 13https://git.io/JeW53 [22:34:10] [02miraheze/puppet] 07paladox 031254c2a - Create server.pp [22:34:11] [02puppet] 07paladox synchronize pull request 03#1095: Reintroduce nfs - 13https://git.io/JeW5Y [22:36:33] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-2 [+1/-0/±0] 13https://git.io/JeW5l [22:36:34] [02miraheze/puppet] 07paladox 030a421b8 - Create staticserver.pp [22:36:36] [02puppet] 07paladox synchronize pull request 03#1095: Reintroduce nfs - 13https://git.io/JeW5Y [22:37:03] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-2 [+0/-0/±1] 13https://git.io/JeW58 [22:37:05] [02miraheze/puppet] 07paladox 03528eedb - Update staticserver.pp [22:37:06] [02puppet] 07paladox synchronize pull request 03#1095: Reintroduce nfs - 13https://git.io/JeW5Y [22:39:05] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-2 [+1/-0/±0] 13https://git.io/JeW5B [22:39:06] [02miraheze/puppet] 07paladox 030462337 - Create exports [22:39:08] [02puppet] 07paladox synchronize pull request 03#1095: Reintroduce nfs - 13https://git.io/JeW5Y [22:40:22] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-2 [+0/-0/±1] 13https://git.io/JeW5R [22:40:23] [02miraheze/puppet] 07paladox 031678483 - Update site.pp [22:40:25] [02puppet] 07paladox synchronize pull request 03#1095: Reintroduce nfs - 13https://git.io/JeW5Y [22:45:01] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-2 [+0/-0/±1] 13https://git.io/JeW5z [22:45:02] [02miraheze/puppet] 07paladox 03e756162 - Update mediawiki.pp [22:45:04] [02puppet] 07paladox synchronize pull request 03#1095: Reintroduce nfs - 13https://git.io/JeW5Y [22:47:48] [02puppet] 07paladox synchronize pull request 03#1095: Reintroduce nfs - 13https://git.io/JeW5Y [22:47:49] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-2 [+0/-0/±1] 13https://git.io/JeW52 [22:47:51] [02miraheze/puppet] 07paladox 037bd3513 - Update staticserver.pp [22:48:52] [02puppet] 07paladox closed pull request 03#1095: Reintroduce nfs - 13https://git.io/JeW5Y [22:48:53] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+4/-0/±2] 13https://git.io/JeW5a [22:48:55] [02miraheze/puppet] 07paladox 0309ffc1e - Reintroduce nfs (#1095) * Reintroduce nfs We will connect nfs to the lizard master which will have a mount which connects to the lizard master. * Create server.pp * Create staticserver.pp * Update staticserver.pp * Create exports * Update site.pp * Update mediawiki.pp * Update staticserver.pp [22:49:11] !log depool mw1 [22:49:17] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [22:49:34] [02puppet] 07paladox deleted branch 03paladox-patch-2 - 13https://git.io/vbiAS [22:49:36] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-2 [22:50:09] PROBLEM - mw1 Puppet on mw1 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 7 minutes ago with 0 failures [22:52:59] RECOVERY - mw1 Puppet on mw1 is OK: OK: Puppet is currently enabled, last run 53 seconds ago with 0 failures [22:54:04] How often are the wikis backed up? [22:54:32] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 7.95, 6.86, 5.25 [22:54:57] PROBLEM - puppet1 Puppet on puppet1 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_puppet] [22:55:53] !log repool mw1 [22:55:58] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [22:56:28] BurningPrincess we backup wikis on a best effort using bacula. We allow users to backup there own wikis using DataDump that generates a xml backup. [22:57:02] RECOVERY - mw3 Current Load on mw3 is OK: OK - load average: 4.63, 6.27, 5.29 [22:57:58] !log depool mw2 [22:58:02] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [22:58:44] RECOVERY - mw2 Puppet on mw2 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [22:59:34] !log repool mw2 [22:59:38] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [23:00:31] !log depool mw3 [23:00:36] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [23:02:21] RECOVERY - puppet1 Puppet on puppet1 is OK: OK: Puppet is currently enabled, last run 31 seconds ago with 0 failures [23:03:07] !log repool mw3 [23:03:11] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [23:03:13] RECOVERY - mw3 Puppet on mw3 is OK: OK: Puppet is currently enabled, last run 13 seconds ago with 0 failures [23:44:11] !log MatomoAnalyticsHooks::wikiCreation( 'bestmusicandsongswiki' ); in eval.php [23:44:15] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [23:55:59] I think i hit the nail in the bud, still monitoring!