[00:18:07] PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 2 datacenters are down: 2604:180:0:33b::2/cpweb, 128.199.139.216/cpweb [00:20:07] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 2 datacenters are down: 107.191.126.23/cpweb, 128.199.139.216/cpweb [00:22:08] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [00:22:10] RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online [01:00:09] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JeK20 [01:00:10] [02miraheze/services] 07MirahezeSSLBot 0309329ae - BOT: Updating services config for wikis [01:30:08] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JeK27 [01:30:10] [02miraheze/services] 07MirahezeSSLBot 03a4ffec7 - BOT: Updating services config for wikis [05:26:37] PROBLEM - mw1 Puppet on mw1 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_checkout_landing] [05:34:46] RECOVERY - mw1 Puppet on mw1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [05:56:38] PROBLEM - mw1 Puppet on mw1 is CRITICAL: CRITICAL: Puppet has 3 failures. Last run 3 minutes ago with 3 failures. Failed resources (up to 3 shown): Exec[git_checkout_landing],Exec[ufw-allow-tcp-from-any-to-any-port-80],Exec[ufw-allow-tcp-from-any-to-any-port-443] [06:23:19] RECOVERY - mw1 Puppet on mw1 is OK: OK: Puppet is currently enabled, last run 1 second ago with 0 failures [06:26:41] RECOVERY - cp3 Disk Space on cp3 is OK: DISK OK - free space: / 2730 MB (11% inode=94%); [07:06:38] PROBLEM - mw1 Puppet on mw1 is CRITICAL: CRITICAL: Puppet has 9 failures. Last run 3 minutes ago with 9 failures. Failed resources (up to 3 shown): Exec[ufw-allow-tcp-from-any-to-any-port-443],Exec[ufw-allow-tcp-from-185.52.3.121-to-any-port-9253],Exec[ufw-allow-tcp-from-185.52.3.121-to-any-port-9113],Package[php7.3-apcu] [07:24:56] RECOVERY - mw1 Puppet on mw1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:36:58] PROBLEM - mw1 Puppet on mw1 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [07:45:06] RECOVERY - mw1 Puppet on mw1 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [07:54:42] PROBLEM - cp3 Disk Space on cp3 is WARNING: DISK WARNING - free space: / 2635 MB (10% inode=94%); [07:58:46] RECOVERY - cp3 Disk Space on cp3 is OK: DISK OK - free space: / 2656 MB (11% inode=94%); [08:02:44] PROBLEM - cp3 Disk Space on cp3 is WARNING: DISK WARNING - free space: / 2590 MB (10% inode=94%); [11:25:05] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 3 datacenters are down: 107.191.126.23/cpweb, 2604:180:0:33b::2/cpweb, 81.4.109.133/cpweb [11:27:05] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [13:20:12] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 1 datacenter is down: 81.4.109.133/cpweb [13:20:29] PROBLEM - cp3 Stunnel Http for mw2 on cp3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [13:22:33] RECOVERY - cp3 Stunnel Http for mw2 on cp3 is OK: HTTP OK: HTTP/1.1 200 OK - 24661 bytes in 6.214 second response time [13:22:55] PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 2 datacenters are down: 2604:180:0:33b::2/cpweb, 81.4.109.133/cpweb [13:26:57] RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online [13:28:14] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [13:31:02] PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 3 datacenters are down: 128.199.139.216/cpweb, 2400:6180:0:d0::403:f001/cpweb, 81.4.109.133/cpweb [13:35:20] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 2 datacenters are down: 2604:180:0:33b::2/cpweb, 2400:6180:0:d0::403:f001/cpweb [13:37:02] RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online [13:37:20] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [14:13:31] PROBLEM - cp2 Stunnel Http for mw2 on cp2 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [14:14:32] PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 2 datacenters are down: 128.199.139.216/cpweb, 2a00:d880:5:8ea::ebc7/cpweb [14:15:28] RECOVERY - cp2 Stunnel Http for mw2 on cp2 is OK: HTTP OK: HTTP/1.1 200 OK - 24655 bytes in 0.390 second response time [14:16:31] RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online [14:21:06] PROBLEM - cp3 Stunnel Http for mw2 on cp3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [14:21:09] PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 3 datacenters are down: 107.191.126.23/cpweb, 128.199.139.216/cpweb, 81.4.109.133/cpweb [14:22:05] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 1 datacenter is down: 2a00:d880:5:8ea::ebc7/cpweb [14:23:04] RECOVERY - cp3 Stunnel Http for mw2 on cp3 is OK: HTTP OK: HTTP/1.1 200 OK - 24661 bytes in 3.746 second response time [14:23:10] RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online [14:24:03] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [14:34:24] PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 1 datacenter is down: 81.4.109.133/cpweb [14:36:22] RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online [14:42:41] PROBLEM - cp4 Stunnel Http for mw2 on cp4 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [14:44:47] RECOVERY - cp4 Stunnel Http for mw2 on cp4 is OK: HTTP OK: HTTP/1.1 200 OK - 24661 bytes in 6.571 second response time [14:52:22] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 1 datacenter is down: 128.199.139.216/cpweb [14:54:21] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [15:55:07] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JeKHf [15:55:09] [02miraheze/services] 07MirahezeSSLBot 0366475f2 - BOT: Updating services config for wikis [16:02:54] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 2 datacenters are down: 2400:6180:0:d0::403:f001/cpweb, 81.4.109.133/cpweb [16:03:18] PROBLEM - cp4 Stunnel Http for mw2 on cp4 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [16:03:23] PROBLEM - cp2 Stunnel Http for mw2 on cp2 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [16:03:59] PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 3 datacenters are down: 2604:180:0:33b::2/cpweb, 81.4.109.133/cpweb, 2a00:d880:5:8ea::ebc7/cpweb [16:07:35] RECOVERY - cp2 Stunnel Http for mw2 on cp2 is OK: HTTP OK: HTTP/1.1 200 OK - 24661 bytes in 9.429 second response time [16:07:38] RECOVERY - cp4 Stunnel Http for mw2 on cp4 is OK: HTTP OK: HTTP/1.1 200 OK - 24661 bytes in 7.611 second response time [16:09:05] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [16:10:13] RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online [16:14:15] PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 2 datacenters are down: 107.191.126.23/cpweb, 2604:180:0:33b::2/cpweb [16:16:09] PROBLEM - cp2 Stunnel Http for mw2 on cp2 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [16:16:40] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 3 datacenters are down: 107.191.126.23/cpweb, 2604:180:0:33b::2/cpweb, 128.199.139.216/cpweb [16:18:07] RECOVERY - cp2 Stunnel Http for mw2 on cp2 is OK: HTTP OK: HTTP/1.1 200 OK - 24661 bytes in 1.686 second response time [16:18:38] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [16:22:43] PROBLEM - cp2 Stunnel Http for mw2 on cp2 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [16:22:46] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 1 datacenter is down: 2a00:d880:5:8ea::ebc7/cpweb [16:28:01] PROBLEM - cp4 Stunnel Http for mw2 on cp4 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [16:28:23] PROBLEM - cp3 Stunnel Http for mw2 on cp3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [16:28:26] erm, that doesn't look good [16:29:00] Can someone restart php-fpm please? [16:29:03] Only seems to affect mw2 [16:29:09] Maybe even nginx [16:29:16] paladox: what's the commands? [16:29:18] Cc johnLewis if your on your laptop ^ [16:29:30] sudo service php7.3-fpm restart [16:29:38] sudo service nginx restart [16:29:55] * RhinosF1 looks [16:30:35] paladox: asking for sudo password for rhinosf1 :( [16:32:16] Ok [16:32:27] RECOVERY - cp4 Stunnel Http for mw2 on cp4 is OK: HTTP OK: HTTP/1.1 200 OK - 24661 bytes in 8.553 second response time [16:32:37] RECOVERY - cp3 Stunnel Http for mw2 on cp3 is OK: HTTP OK: HTTP/1.1 200 OK - 24661 bytes in 9.074 second response time [16:34:02] !log restart ngninx and php7.3-fpm on mw2 [16:34:17] thanks john [16:34:23] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [16:34:46] JohnLewis: am I right that mw-admin don't have access to that based on being asked for sudo pw [16:35:06] Thanks johnLewis ! [16:35:29] RhinosF1: they don't have access to 7.3 [16:35:37] only 7.2. Apparently. [16:35:51] JohnLewis: would nginx have worked if i'd have done it? [16:35:58] Yes [16:36:36] I'll remember for next time and JohnLewis that's maybe something we should look at, giving us 7.3 access [16:36:48] RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online [16:37:05] Oh [16:37:06] Damn, need to tweek that to use the hiera [16:37:11] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [16:37:40] RECOVERY - cp2 Stunnel Http for mw2 on cp2 is OK: HTTP OK: HTTP/1.1 200 OK - 24661 bytes in 3.242 second response time [16:48:48] PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 4 datacenters are down: 107.191.126.23/cpweb, 2604:180:0:33b::2/cpweb, 2400:6180:0:d0::403:f001/cpweb, 2a00:d880:5:8ea::ebc7/cpweb [16:49:25] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 2 datacenters are down: 2604:180:0:33b::2/cpweb, 81.4.109.133/cpweb [16:52:00] PROBLEM - mw1 MediaWiki Rendering on mw1 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:52:40] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JeK7T [16:52:42] [02miraheze/puppet] 07paladox 035144bb5 - Update mount.pp [16:52:59] PROBLEM - cp2 Stunnel Http for mw2 on cp2 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [16:53:29] for pete sake [16:53:45] !log reboot mw2 [16:54:06] RECOVERY - mw1 MediaWiki Rendering on mw1 is OK: HTTP OK: HTTP/1.1 200 OK - 19020 bytes in 3.381 second response time [16:54:36] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [16:54:58] RECOVERY - cp2 Stunnel Http for mw2 on cp2 is OK: HTTP OK: HTTP/1.1 200 OK - 24639 bytes in 0.392 second response time [16:55:13] RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online [16:57:42] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [17:01:42] !log upgrade some php packages on mw* [17:02:03] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [17:05:50] PROBLEM - cp4 Stunnel Http for mw2 on cp4 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [17:07:53] RECOVERY - cp4 Stunnel Http for mw2 on cp4 is OK: HTTP OK: HTTP/1.1 200 OK - 24661 bytes in 3.793 second response time [17:12:00] PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 1 datacenter is down: 2400:6180:0:d0::403:f001/cpweb [17:12:39] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 5 datacenters are down: 2604:180:0:33b::2/cpweb, 128.199.139.216/cpweb, 2400:6180:0:d0::403:f001/cpweb, 81.4.109.133/cpweb, 2a00:d880:5:8ea::ebc7/cpweb [17:12:41] PROBLEM - cp2 Stunnel Http for mw2 on cp2 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [17:13:58] RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online [17:14:37] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [17:14:39] RECOVERY - cp2 Stunnel Http for mw2 on cp2 is OK: HTTP OK: HTTP/1.1 200 OK - 24639 bytes in 1.657 second response time [17:16:58] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+1/-2/±1] 13https://git.io/JeK77 [17:16:59] [02miraheze/puppet] 07paladox 038e14a80 - mediawiki: Fix installing tideways under php7.3 [17:17:56] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+1/-2/±1] 13https://git.io/JeK7d [17:17:58] [02miraheze/puppet] 07paladox 03ba09d84 - mediawiki: Fix installing tideways under php7.3 [17:18:03] PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 2 datacenters are down: 128.199.139.216/cpweb, 2400:6180:0:d0::403:f001/cpweb [17:19:09] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+1/-2/±1] 13https://git.io/JeK7A [17:19:10] [02miraheze/puppet] 07paladox 03a15192e - mediawiki: Fix installing tideways under php7.3 [17:21:14] PROBLEM - cp3 Stunnel Http for mw2 on cp3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [17:22:54] PROBLEM - mw2 Puppet on mw2 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [17:22:59] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 1 datacenter is down: 2a00:d880:5:8ea::ebc7/cpweb [17:24:26] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+1/-2/±1] 13https://git.io/JeK5J [17:24:28] [02miraheze/puppet] 07paladox 03d9a83e6 - mediawiki: Fix installing tideways under php7.3 [17:24:39] RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online [17:24:49] RECOVERY - mw2 Puppet on mw2 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [17:24:58] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [17:29:11] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+1/-2/±1] 13https://git.io/JeK5G [17:29:12] [02miraheze/puppet] 07paladox 03c997dc7 - mediawiki: Fix installing tideways under php7.3 [17:29:15] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 3 datacenters are down: 107.191.126.23/cpweb, 2604:180:0:33b::2/cpweb, 128.199.139.216/cpweb [17:29:31] RECOVERY - cp3 Stunnel Http for mw2 on cp3 is OK: HTTP OK: HTTP/1.1 200 OK - 24661 bytes in 7.712 second response time [17:30:57] PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 1 datacenter is down: 2a00:d880:5:8ea::ebc7/cpweb [17:31:28] PROBLEM - mw2 Puppet on mw2 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Package[php-tideways-xhprof] [17:33:25] RECOVERY - mw2 Puppet on mw2 is OK: OK: Puppet is currently enabled, last run 12 seconds ago with 0 failures [17:34:54] RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online [17:36:33] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JeKdt [17:36:34] [02miraheze/puppet] 07paladox 03e8877b0 - Revert "varnish: Remove lizardfs6" This reverts commit 87f980cdeddf6e2b53b0acc8c1de672fbcafd5bd. [17:39:21] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [17:56:35] !log install haveged on mw2 [17:56:45] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [17:57:31] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JeKd6 [17:57:32] [02miraheze/puppet] 07paladox 033aaf61e - varnish: Remove lizardfs6 This reverts commit e8877b046d9f144f73c8085c490b1151bce2329f. [18:00:37] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JeKdX [18:00:38] [02miraheze/puppet] 07paladox 03d8777bc - Revert "varnish: Remove lizardfs6" This reverts commit 3aaf61e003b463dd619d9193035eb6807918ad77. [18:02:35] PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 5 datacenters are down: 107.191.126.23/cpweb, 2604:180:0:33b::2/cpweb, 128.199.139.216/cpweb, 81.4.109.133/cpweb, 2a00:d880:5:8ea::ebc7/cpweb [18:02:49] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 1 datacenter is down: 2604:180:0:33b::2/cpweb [18:03:34] !log depool mw2 temporarily [18:03:39] !log remove haveged from mw2 [18:03:41] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [18:03:57] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [18:04:36] !log repool mw2 [18:04:55] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [18:07:32] PROBLEM - cp3 Stunnel Http for mw2 on cp3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [18:08:35] RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online [18:08:51] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [18:09:31] RECOVERY - cp3 Stunnel Http for mw2 on cp3 is OK: HTTP OK: HTTP/1.1 200 OK - 24661 bytes in 3.312 second response time [18:17:51] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 2 datacenters are down: 107.191.126.23/cpweb, 81.4.109.133/cpweb [18:17:56] PROBLEM - cp3 Stunnel Http for mw2 on cp3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [18:18:17] PROBLEM - cp2 Stunnel Http for mw2 on cp2 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [18:18:52] PROBLEM - cp4 Stunnel Http for mw2 on cp4 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [18:19:49] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [18:19:52] RECOVERY - cp3 Stunnel Http for mw2 on cp3 is OK: HTTP OK: HTTP/1.1 200 OK - 24639 bytes in 0.642 second response time [18:20:15] RECOVERY - cp2 Stunnel Http for mw2 on cp2 is OK: HTTP OK: HTTP/1.1 200 OK - 24639 bytes in 0.400 second response time [18:20:34] PROBLEM - wiki.mxlinuxusers.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.mxlinuxusers.org' expires in 15 day(s) (Thu 05 Dec 2019 06:17:21 PM GMT +0000). [18:20:48] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JeKFL [18:20:50] [02miraheze/ssl] 07MirahezeSSLBot 0355896ed - Bot: Update SSL cert for wiki.mxlinuxusers.org [18:20:52] RECOVERY - cp4 Stunnel Http for mw2 on cp4 is OK: HTTP OK: HTTP/1.1 200 OK - 24655 bytes in 0.006 second response time [18:24:19] PROBLEM - storytime.jdstroy.cf - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'storytime.jdstroy.cf' expires in 15 day(s) (Thu 05 Dec 2019 06:20:25 PM GMT +0000). [18:24:33] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JeKFs [18:24:34] [02miraheze/ssl] 07MirahezeSSLBot 032b0669c - Bot: Update SSL cert for storytime.jdstroy.cf [18:30:40] paladox: Do you see yellow zones on the map? because I only see red and blue, and I uploaded a new file with yellow zones. https://commons.miraheze.org/wiki/File:Mapa_electoral_de_las_elecciones_presidenciales_de_M%C3%A9xico_de_2018_(Segundo_Imperio_Mexicano).svg [18:30:41] [ File:Mapa electoral de las elecciones presidenciales de México de 2018 (Segundo Imperio Mexicano).svg - Miraheze Commons ] - commons.miraheze.org [18:32:28] This is what I've modified https://usercontent.irccloud-cdn.com/file/v7Q0NuHk/Anotaci%C3%B3n%202019-11-19%20123138.png [18:34:12] RECOVERY - storytime.jdstroy.cf - LetsEncrypt on sslhost is OK: OK - Certificate 'storytime.jdstroy.cf' will expire on Mon 17 Feb 2020 05:24:26 PM GMT +0000. [18:34:28] RECOVERY - wiki.mxlinuxusers.org - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.mxlinuxusers.org' will expire on Mon 17 Feb 2020 05:20:42 PM GMT +0000. [18:40:05] hispano76 yes i see yellow zones [18:48:06] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JeKFS [18:48:07] [02miraheze/puppet] 07paladox 03366be5d - varnish: Remove lizardfs6 This reverts commit d8777bc920281bd9c0f7207d63265c947c54cec4. [18:56:15] !log reboot mw1 [18:56:24] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [19:01:44] PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 4 datacenters are down: 2604:180:0:33b::2/cpweb, 128.199.139.216/cpweb, 2400:6180:0:d0::403:f001/cpweb, 2a00:d880:5:8ea::ebc7/cpweb [19:03:06] !log restart php7.3-fpm on mw* [19:03:12] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [19:03:44] RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online [19:12:39] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 2 datacenters are down: 107.191.126.23/cpweb, 2604:180:0:33b::2/cpweb [19:14:37] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [19:16:46] Oh, all right. So it's my browser. Thank you paladox :) [19:22:56] PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 3 datacenters are down: 2604:180:0:33b::2/cpweb, 128.199.139.216/cpweb, 2400:6180:0:d0::403:f001/cpweb [19:24:44] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 2 datacenters are down: 107.191.126.23/cpweb, 81.4.109.133/cpweb [19:28:40] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [19:28:56] RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online [19:44:58] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 3 datacenters are down: 2604:180:0:33b::2/cpweb, 2400:6180:0:d0::403:f001/cpweb, 2a00:d880:5:8ea::ebc7/cpweb [19:45:15] PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 3 datacenters are down: 2604:180:0:33b::2/cpweb, 128.199.139.216/cpweb, 2a00:d880:5:8ea::ebc7/cpweb [19:45:23] PROBLEM - cp2 Stunnel Http for mw1 on cp2 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [19:46:25] PROBLEM - mw2 MediaWiki Rendering on mw2 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:47:27] RECOVERY - cp2 Stunnel Http for mw1 on cp2 is OK: HTTP OK: HTTP/1.1 200 OK - 24655 bytes in 0.865 second response time [19:48:23] RECOVERY - mw2 MediaWiki Rendering on mw2 is OK: HTTP OK: HTTP/1.1 200 OK - 19025 bytes in 0.305 second response time [19:49:03] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [19:49:20] RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online [19:55:10] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 1 datacenter is down: 107.191.126.23/cpweb [19:57:18] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [19:58:33] PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 5 datacenters are down: 107.191.126.23/cpweb, 2604:180:0:33b::2/cpweb, 128.199.139.216/cpweb, 81.4.109.133/cpweb, 2a00:d880:5:8ea::ebc7/cpweb [20:02:54] RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online [20:20:15] [02mw-config] 07Pix1234 deleted branch 03paladox-patch-1 - 13https://git.io/vbvb3 [20:20:17] [02miraheze/mw-config] 07Pix1234 deleted branch 03paladox-patch-1 [20:45:16] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 1 datacenter is down: 107.191.126.23/cpweb [20:46:03] PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 1 datacenter is down: 2604:180:0:33b::2/cpweb [20:47:17] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [20:50:07] RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online [20:54:01] PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 1 datacenter is down: 2604:180:0:33b::2/cpweb [20:58:20] RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online