[01:15:31] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JeoZO [01:15:32] [02miraheze/puppet] 07paladox 03e0c7d9b - Update config.yaml [01:21:11] !log dpkg -i libjs-moment-timezone_0.5.23+dfsg1-1_all.deb (https://packages.debian.org/buster/all/libjs-moment-timezone/download) required by the prometheus upgrade 2.3 -> 2.7 in stretch-backports, apt package not avilable. [01:21:11] [ Debian -- Error ] - packages.debian.org [01:21:22] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [01:29:03] PROBLEM - lizardfs6 Current Load on lizardfs6 is WARNING: WARNING - load average: 5.28, 10.58, 15.18 [01:30:15] [02mw-config] 07paladox closed pull request 03#2810: Move all wikis that start with the letter m to the new mount - 13https://git.io/JeoON [01:30:16] [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JeoZ8 [01:30:18] [02miraheze/mw-config] 07paladox 0396d1064 - Move all wikis that start with the letter m to the new mount (#2810) [01:31:05] [02mw-config] 07paladox deleted branch 03paladox-patch-2 - 13https://git.io/vbvb3 [01:31:06] [02miraheze/mw-config] 07paladox deleted branch 03paladox-patch-2 [01:33:03] RECOVERY - lizardfs6 Current Load on lizardfs6 is OK: OK - load average: 8.11, 8.83, 13.35 [01:39:20] [02miraheze/mw-config] 07paladox pushed 031 commit to 03paladox-patch-2 [+0/-0/±1] 13https://git.io/JeoZE [01:39:22] [02miraheze/mw-config] 07paladox 03e7b9a13 - Move all wikis that start with the letter o to the new mount [01:39:23] [02mw-config] 07paladox created branch 03paladox-patch-2 - 13https://git.io/vbvb3 [01:39:25] [02mw-config] 07paladox opened pull request 03#2811: Move all wikis that start with the letter o to the new mount - 13https://git.io/JeoZu [01:40:06] !log root@lizardfs6:/mnt/mediawiki-static# rm -rf m* [01:40:42] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [01:45:10] PROBLEM - lizardfs6 Current Load on lizardfs6 is CRITICAL: CRITICAL - load average: 27.60, 18.66, 14.99 [02:25:05] PROBLEM - lizardfs6 Current Load on lizardfs6 is WARNING: WARNING - load average: 5.50, 9.88, 15.06 [02:29:03] RECOVERY - lizardfs6 Current Load on lizardfs6 is OK: OK - load average: 6.07, 7.68, 12.98 [06:26:13] RECOVERY - cp3 Disk Space on cp3 is OK: DISK OK - free space: / 2725 MB (11% inode=94%); [08:13:43] PROBLEM - cp3 Disk Space on cp3 is WARNING: DISK WARNING - free space: / 2649 MB (10% inode=94%); [08:44:57] PROBLEM - wiki.galactee.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.galactee.org' expires in 15 day(s) (Mon 02 Dec 2019 08:41:48 AM GMT +0000). [08:45:04] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JeoWN [08:45:05] [02miraheze/ssl] 07MirahezeSSLBot 030034200 - Bot: Update SSL cert for wiki.galactee.org [08:52:57] RECOVERY - wiki.galactee.org - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.galactee.org' will expire on Fri 31 Jan 2020 03:45:05 PM GMT +0000. [09:50:03] PROBLEM - www.thesimswiki.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'www.thesimswiki.com' expires in 15 day(s) (Mon 02 Dec 2019 09:47:00 AM GMT +0000). [09:50:13] PROBLEM - wiki.thesimswiki.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'www.thesimswiki.com' expires in 15 day(s) (Mon 02 Dec 2019 09:47:00 AM GMT +0000). [09:50:20] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Jeol6 [09:50:21] [02miraheze/ssl] 07MirahezeSSLBot 03a72b4f2 - Bot: Update SSL cert for www.thesimswiki.com [09:50:47] PROBLEM - thesimswiki.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'www.thesimswiki.com' expires in 15 day(s) (Mon 02 Dec 2019 09:47:00 AM GMT +0000). [10:04:07] RECOVERY - wiki.thesimswiki.com - LetsEncrypt on sslhost is OK: OK - Certificate 'www.thesimswiki.com' will expire on Fri 14 Feb 2020 08:50:14 AM GMT +0000. [10:04:11] RECOVERY - www.thesimswiki.com - LetsEncrypt on sslhost is OK: OK - Certificate 'www.thesimswiki.com' will expire on Fri 14 Feb 2020 08:50:14 AM GMT +0000. [10:04:47] RECOVERY - thesimswiki.com - LetsEncrypt on sslhost is OK: OK - Certificate 'www.thesimswiki.com' will expire on Fri 14 Feb 2020 08:50:14 AM GMT +0000. [12:10:06] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Jeo4i [12:10:08] [02miraheze/services] 07MirahezeSSLBot 039928f32 - BOT: Updating services config for wikis [14:34:24] [02mw-config] 07paladox closed pull request 03#2811: Move all wikis that start with the letter o to the new mount - 13https://git.io/JeoZu [14:34:25] [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Jeo0O [14:34:26] [02miraheze/mw-config] 07paladox 03aa31233 - Move all wikis that start with the letter o to the new mount (#2811) [14:34:51] [02mw-config] 07paladox deleted branch 03paladox-patch-2 - 13https://git.io/vbvb3 [14:34:53] [02miraheze/mw-config] 07paladox deleted branch 03paladox-patch-2 [14:36:32] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Jeo03 [14:36:34] [02miraheze/puppet] 07paladox 034a9523c - Temporarily removing lizardfs6 from varnish [14:38:46] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Jeo0Z [14:38:47] [02miraheze/puppet] 07paladox 03c46af51 - Revert "Temporarily removing lizardfs6 from varnish" This reverts commit 4a9523c77f4c74154126429edc05d3e9c36920be. [14:53:29] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-6 [+0/-0/±1] 13https://git.io/Jeo0g [14:53:31] [02miraheze/puppet] 07paladox 03340466e - varnish: Check syntax before restarting varnish [14:53:32] [02puppet] 07paladox created branch 03paladox-patch-6 - 13https://git.io/vbiAS [14:53:34] [02puppet] 07paladox opened pull request 03#1147: varnish: Check syntax before restarting varnish - 13https://git.io/Jeo02 [14:56:19] [02puppet] 07paladox closed pull request 03#1147: varnish: Check syntax before restarting varnish - 13https://git.io/Jeo02 [14:56:21] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Jeo0w [14:56:22] [02miraheze/puppet] 07paladox 031d01e86 - varnish: Check syntax before restarting varnish (#1147) [14:59:15] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Jeo0i [14:59:16] [02miraheze/puppet] 07paladox 03f61a72d - varnish: Temporarily remove lizardfs6 [15:02:27] !log root@lizardfs6:/mnt/mediawiki-static# rm -rf o* [15:03:14] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [15:04:19] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Jeo09 [15:04:20] [02miraheze/puppet] 07paladox 032909f3e - Revert "varnish: Temporarily remove lizardfs6" This reverts commit f61a72d267c350c52b70d4482df764ddab6c817e. [15:05:24] PROBLEM - cp3 Stunnel Http for mw2 on cp3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [15:06:11] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 6 datacenters are down: 107.191.126.23/cpweb, 2604:180:0:33b::2/cpweb, 128.199.139.216/cpweb, 2400:6180:0:d0::403:f001/cpweb, 81.4.109.133/cpweb, 2a00:d880:5:8ea::ebc7/cpweb [15:06:13] [02miraheze/mw-config] 07paladox pushed 031 commit to 03paladox-patch-2 [+0/-0/±1] 13https://git.io/Jeo07 [15:06:15] [02miraheze/mw-config] 07paladox 038cdb768 - Move all wikis that start with the letter p to the new mount [15:06:16] [02mw-config] 07paladox created branch 03paladox-patch-2 - 13https://git.io/vbvb3 [15:06:18] [02mw-config] 07paladox opened pull request 03#2812: Move all wikis that start with the letter p to the new mount - 13https://git.io/Jeo05 [15:06:32] PROBLEM - lizardfs6 MediaWiki Rendering on lizardfs6 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:07:12] [02miraheze/mw-config] 07paladox pushed 031 commit to 03paladox-patch-2 [+0/-0/±1] 13https://git.io/Jeo0F [15:07:13] [02miraheze/mw-config] 07paladox 0338b0117 - Update LocalWiki.php [15:07:15] [02mw-config] 07paladox synchronize pull request 03#2812: Move all wikis that start with the letter p to the new mount - 13https://git.io/Jeo05 [15:07:18] PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 6 datacenters are down: 107.191.126.23/cpweb, 2604:180:0:33b::2/cpweb, 128.199.139.216/cpweb, 2400:6180:0:d0::403:f001/cpweb, 81.4.109.133/cpweb, 2a00:d880:5:8ea::ebc7/cpweb [15:07:41] RECOVERY - cp3 Stunnel Http for mw2 on cp3 is OK: HTTP OK: HTTP/1.1 200 OK - 24661 bytes in 4.454 second response time [15:08:11] PROBLEM - cp2 Varnish Backends on cp2 is CRITICAL: 2 backends are down. mw1 mw3 [15:08:59] PROBLEM - cp4 Varnish Backends on cp4 is CRITICAL: 2 backends are down. mw2 mw3 [15:09:00] PROBLEM - mw3 MediaWiki Rendering on mw3 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:09:15] PROBLEM - mw1 MediaWiki Rendering on mw1 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 342 bytes in 0.017 second response time [15:09:51] PROBLEM - cp3 Varnish Backends on cp3 is CRITICAL: 1 backends are down. mw1 [15:10:53] RECOVERY - cp4 Varnish Backends on cp4 is OK: All 6 backends are healthy [15:11:09] RECOVERY - mw3 MediaWiki Rendering on mw3 is OK: HTTP OK: HTTP/1.1 200 OK - 19036 bytes in 3.962 second response time [15:12:17] RECOVERY - cp3 Varnish Backends on cp3 is OK: All 6 backends are healthy [15:12:30] RECOVERY - cp2 Varnish Backends on cp2 is OK: All 6 backends are healthy [15:12:35] PROBLEM - lizardfs6 Current Load on lizardfs6 is CRITICAL: CRITICAL - load average: 26.78, 16.90, 9.20 [15:12:37] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [15:13:05] RECOVERY - lizardfs6 MediaWiki Rendering on lizardfs6 is OK: HTTP OK: HTTP/1.1 200 OK - 19026 bytes in 5.858 second response time [15:13:32] RECOVERY - mw1 MediaWiki Rendering on mw1 is OK: HTTP OK: HTTP/1.1 200 OK - 19032 bytes in 2.119 second response time [15:17:33] RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online [15:56:30] PROBLEM - lizardfs6 Current Load on lizardfs6 is WARNING: WARNING - load average: 12.01, 14.57, 15.99 [15:58:32] PROBLEM - lizardfs6 Current Load on lizardfs6 is CRITICAL: CRITICAL - load average: 19.70, 15.91, 16.27 [16:08:56] PROBLEM - lizardfs6 Current Load on lizardfs6 is WARNING: WARNING - load average: 11.31, 14.37, 15.81 [16:14:53] PROBLEM - lizardfs6 Current Load on lizardfs6 is CRITICAL: CRITICAL - load average: 16.23, 13.56, 14.87 [16:18:48] PROBLEM - lizardfs6 Current Load on lizardfs6 is WARNING: WARNING - load average: 10.57, 13.14, 14.50 [16:24:38] RECOVERY - lizardfs6 Current Load on lizardfs6 is OK: OK - load average: 8.21, 11.29, 13.35 [16:34:09] [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JeouR [16:34:10] [02miraheze/mw-config] 07paladox 032dc7000 - Change $wmgUploadHostname to static-temp temporarily [16:38:51] [02miraheze/landing] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/Jeouu [16:38:52] [02miraheze/landing] 07paladox 03c06505f - Temporarily changing to static-temp [16:38:54] [02landing] 07paladox created branch 03paladox-patch-1 - 13https://git.io/fxEQA [16:38:55] [02landing] 07paladox opened pull request 03#18: Temporarily changing to static-temp - 13https://git.io/Jeouz [16:38:58] [02landing] 07paladox closed pull request 03#18: Temporarily changing to static-temp - 13https://git.io/Jeouz [16:39:00] [02miraheze/landing] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Jeoug [16:39:02] [02miraheze/landing] 07paladox 037f6bbc3 - Temporarily changing to static-temp (#18) [16:39:03] [02landing] 07paladox deleted branch 03paladox-patch-1 - 13https://git.io/fxEQA [16:39:05] [02miraheze/landing] 07paladox deleted branch 03paladox-patch-1 [16:39:46] miraheze/landing/paladox-patch-1/c06505f - paladox The build has errored. https://travis-ci.org/miraheze/landing/builds/612840952 [17:12:25] [02miraheze/mw-config] 07paladox pushed 031 commit to 03paladox-patch-2 [+0/-0/±1] 13https://git.io/JeozL [17:12:27] [02miraheze/mw-config] 07paladox 03adc6229 - Update LocalExtensions.php [17:12:28] [02mw-config] 07paladox synchronize pull request 03#2812: Move all wikis that start with the letter p to the new mount - 13https://git.io/Jeo05 [17:36:21] [02mw-config] 07paladox closed pull request 03#2812: Move all wikis that start with the letter p to the new mount - 13https://git.io/Jeo05 [17:36:23] [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±3] 13https://git.io/Jeogk [17:36:24] [02miraheze/mw-config] 07paladox 03b1f3339 - Move all wikis that start with the letter p to the new mount (#2812) * Move all wikis that start with the letter p to the new mount * Update LocalWiki.php * Update LocalExtensions.php [17:36:54] [02mw-config] 07paladox deleted branch 03paladox-patch-2 - 13https://git.io/vbvb3 [17:36:55] [02miraheze/mw-config] 07paladox deleted branch 03paladox-patch-2 [17:53:52] !log root@lizardfs6:/mnt/mediawiki-static# rm -rf p* [17:54:10] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [17:58:22] PROBLEM - lizardfs6 Current Load on lizardfs6 is CRITICAL: CRITICAL - load average: 21.16, 14.43, 10.18 [17:59:31] [02miraheze/mw-config] 07paladox pushed 031 commit to 03paladox-patch-2 [+0/-0/±1] 13https://git.io/Jeogi [17:59:32] [02miraheze/mw-config] 07paladox 0380823a7 - Move all wikis that start with the letter q to the new mount [17:59:34] [02mw-config] 07paladox created branch 03paladox-patch-2 - 13https://git.io/vbvb3 [17:59:35] [02mw-config] 07paladox opened pull request 03#2813: Move all wikis that start with the letter q to the new mount - 13https://git.io/JeogP [18:00:20] PROBLEM - lizardfs6 Current Load on lizardfs6 is WARNING: WARNING - load average: 15.13, 15.08, 10.98 [18:02:37] PROBLEM - lizardfs6 Current Load on lizardfs6 is CRITICAL: CRITICAL - load average: 34.68, 21.96, 14.03 [18:03:48] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 2 datacenters are down: 107.191.126.23/cpweb, 2400:6180:0:d0::403:f001/cpweb [18:04:17] !log reception@mw3:/srv/mediawiki/w/maintenance$ sudo -u www-data php deleteBatch.php --wiki metawiki --r="Mass deletion of Twinkle pages, to be reimported" --u="Reception123" /home/reception/delete.txt [18:04:37] PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 1 datacenter is down: 2a00:d880:5:8ea::ebc7/cpweb [18:04:38] PROBLEM - cp2 Stunnel Http for mw2 on cp2 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [18:04:54] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [18:06:34] RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online [18:06:37] RECOVERY - cp2 Stunnel Http for mw2 on cp2 is OK: HTTP OK: HTTP/1.1 200 OK - 24675 bytes in 0.399 second response time [18:07:23] [02mw-config] 07paladox closed pull request 03#2813: Move all wikis that start with the letter q to the new mount - 13https://git.io/JeogP [18:07:24] [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JeogN [18:07:26] [02miraheze/mw-config] 07paladox 032a1e6b2 - Move all wikis that start with the letter q to the new mount (#2813) [18:07:27] [02miraheze/mw-config] 07paladox deleted branch 03paladox-patch-2 [18:07:29] [02mw-config] 07paladox deleted branch 03paladox-patch-2 - 13https://git.io/vbvb3 [18:07:53] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [18:09:24] !log root@lizardfs6:/mnt/mediawiki-static# rm -rf q* [18:09:48] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [18:11:57] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 3 datacenters are down: 107.191.126.23/cpweb, 2604:180:0:33b::2/cpweb, 128.199.139.216/cpweb [18:12:41] PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 1 datacenter is down: 2400:6180:0:d0::403:f001/cpweb [18:13:49] !log restart php7.2-fpm on mw2 [18:13:53] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [18:14:04] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [18:14:33] !log restart nginx on mw2 [18:15:03] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [18:16:50] RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online [18:23:49] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 4 datacenters are down: 128.199.139.216/cpweb, 2400:6180:0:d0::403:f001/cpweb, 81.4.109.133/cpweb, 2a00:d880:5:8ea::ebc7/cpweb [18:24:18] PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 1 datacenter is down: 128.199.139.216/cpweb [18:25:48] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [18:28:14] RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online [18:52:57] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 1 datacenter is down: 128.199.139.216/cpweb [18:53:03] PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 2 datacenters are down: 107.191.126.23/cpweb, 81.4.109.133/cpweb [18:55:05] RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online [18:57:44] PROBLEM - mw2 Puppet on mw2 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [18:59:07] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [19:35:38] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 1 datacenter is down: 107.191.126.23/cpweb [19:37:32] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [19:38:19] PROBLEM - lizardfs6 Current Load on lizardfs6 is WARNING: WARNING - load average: 7.45, 11.78, 15.67 [19:39:09] [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JeoVa [19:39:11] [02miraheze/mw-config] 07paladox 03b6909a5 - Move all wikis that start with the letter r to the new mount [19:41:49] !log root@lizardfs6:/mnt/mediawiki-static# rm r* [19:42:03] !log root@lizardfs6:/mnt/mediawiki-static# rm -rf r* [19:42:07] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [19:43:38] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [19:44:38] PROBLEM - lizardfs6 Current Load on lizardfs6 is CRITICAL: CRITICAL - load average: 25.50, 16.30, 15.78 [19:52:15] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 4 datacenters are down: 128.199.139.216/cpweb, 2400:6180:0:d0::403:f001/cpweb, 81.4.109.133/cpweb, 2a00:d880:5:8ea::ebc7/cpweb [19:54:13] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [20:02:59] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 2 datacenters are down: 107.191.126.23/cpweb, 2400:6180:0:d0::403:f001/cpweb [20:04:54] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [20:17:28] PROBLEM - cp3 Stunnel Http for mw1 on cp3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [20:18:47] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 2 datacenters are down: 107.191.126.23/cpweb, 2a00:d880:5:8ea::ebc7/cpweb [20:19:29] RECOVERY - cp3 Stunnel Http for mw1 on cp3 is OK: HTTP OK: HTTP/1.1 200 OK - 24681 bytes in 0.712 second response time [20:22:17] [02miraheze/mw-config] 07paladox pushed 031 commit to 03paladox-patch-2 [+0/-0/±1] 13https://git.io/JeowB [20:22:18] [02miraheze/mw-config] 07paladox 0355c96d4 - Move all wikis that start with the letter s to the new mount [20:22:20] [02mw-config] 07paladox created branch 03paladox-patch-2 - 13https://git.io/vbvb3 [20:22:21] [02mw-config] 07paladox opened pull request 03#2814: Move all wikis that start with the letter s to the new mount - 13https://git.io/JeowR [20:22:45] PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 3 datacenters are down: 2604:180:0:33b::2/cpweb, 128.199.139.216/cpweb, 81.4.109.133/cpweb [20:24:44] RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online [20:25:03] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [20:29:18] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 3 datacenters are down: 128.199.139.216/cpweb, 2400:6180:0:d0::403:f001/cpweb, 2a00:d880:5:8ea::ebc7/cpweb [20:31:25] PROBLEM - cp2 Stunnel Http for mw2 on cp2 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [20:31:53] PROBLEM - cp3 Stunnel Http for mw2 on cp3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [20:32:04] PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 6 datacenters are down: 107.191.126.23/cpweb, 2604:180:0:33b::2/cpweb, 128.199.139.216/cpweb, 2400:6180:0:d0::403:f001/cpweb, 81.4.109.133/cpweb, 2a00:d880:5:8ea::ebc7/cpweb [20:33:35] RECOVERY - cp2 Stunnel Http for mw2 on cp2 is OK: HTTP OK: HTTP/1.1 200 OK - 24675 bytes in 0.397 second response time [20:33:54] RECOVERY - cp3 Stunnel Http for mw2 on cp3 is OK: HTTP OK: HTTP/1.1 200 OK - 24659 bytes in 0.673 second response time [20:37:58] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [20:38:10] RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online [20:53:32] PROBLEM - cp2 Stunnel Http for mw2 on cp2 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [20:54:11] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 1 datacenter is down: 81.4.109.133/cpweb [20:57:40] RECOVERY - cp2 Stunnel Http for mw2 on cp2 is OK: HTTP OK: HTTP/1.1 200 OK - 24659 bytes in 0.391 second response time [20:58:02] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [21:02:58] PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 1 datacenter is down: 2a00:d880:5:8ea::ebc7/cpweb [21:05:00] RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online [21:09:27] PROBLEM - cp3 Stunnel Http for mw2 on cp3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [21:11:38] RECOVERY - cp3 Stunnel Http for mw2 on cp3 is OK: HTTP OK: HTTP/1.1 200 OK - 24675 bytes in 0.677 second response time [21:13:35] PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 1 datacenter is down: 128.199.139.216/cpweb [21:15:21] PROBLEM - lizardfs6 Current Load on lizardfs6 is WARNING: WARNING - load average: 15.13, 14.08, 15.60 [21:17:35] RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online [21:21:28] PROBLEM - lizardfs6 Current Load on lizardfs6 is CRITICAL: CRITICAL - load average: 20.16, 15.85, 15.67 [21:23:24] PROBLEM - lizardfs6 Current Load on lizardfs6 is WARNING: WARNING - load average: 15.10, 15.57, 15.59 [21:27:28] PROBLEM - lizardfs6 Current Load on lizardfs6 is CRITICAL: CRITICAL - load average: 21.25, 17.00, 16.04 [21:31:37] PROBLEM - lizardfs6 Current Load on lizardfs6 is WARNING: WARNING - load average: 13.95, 15.67, 15.73 [21:33:46] PROBLEM - lizardfs6 Current Load on lizardfs6 is CRITICAL: CRITICAL - load average: 19.72, 17.04, 16.21 [21:43:15] PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 1 datacenter is down: 2400:6180:0:d0::403:f001/cpweb [21:45:13] RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online [22:12:08] PROBLEM - lizardfs6 Current Load on lizardfs6 is WARNING: WARNING - load average: 14.23, 14.79, 15.88 [22:16:06] PROBLEM - lizardfs6 Current Load on lizardfs6 is CRITICAL: CRITICAL - load average: 16.66, 14.52, 15.45 [22:20:01] PROBLEM - lizardfs6 Current Load on lizardfs6 is WARNING: WARNING - load average: 9.99, 13.82, 15.09 [22:29:44] PROBLEM - lizardfs6 Current Load on lizardfs6 is CRITICAL: CRITICAL - load average: 24.16, 15.68, 14.92 [22:31:45] PROBLEM - lizardfs6 Current Load on lizardfs6 is WARNING: WARNING - load average: 15.19, 15.26, 14.86 [22:35:52] PROBLEM - lizardfs6 Current Load on lizardfs6 is CRITICAL: CRITICAL - load average: 16.36, 15.60, 15.05 [22:37:50] PROBLEM - lizardfs6 Current Load on lizardfs6 is WARNING: WARNING - load average: 10.26, 13.66, 14.41 [22:41:52] RECOVERY - lizardfs6 Current Load on lizardfs6 is OK: OK - load average: 9.01, 11.70, 13.44 [22:48:08] PROBLEM - lizardfs6 Current Load on lizardfs6 is CRITICAL: CRITICAL - load average: 18.19, 16.97, 15.22 [22:50:05] PROBLEM - lizardfs6 Current Load on lizardfs6 is WARNING: WARNING - load average: 11.47, 15.06, 14.72 [22:57:52] PROBLEM - lizardfs6 Current Load on lizardfs6 is CRITICAL: CRITICAL - load average: 19.34, 14.50, 14.20 [23:03:56] PROBLEM - lizardfs6 Current Load on lizardfs6 is WARNING: WARNING - load average: 14.25, 15.29, 14.66 [23:14:06] PROBLEM - lizardfs6 Current Load on lizardfs6 is CRITICAL: CRITICAL - load average: 18.95, 14.82, 14.35 [23:16:11] RECOVERY - lizardfs6 Current Load on lizardfs6 is OK: OK - load average: 9.52, 12.48, 13.53 [23:17:46] PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 1 datacenter is down: 107.191.126.23/cpweb [23:19:43] RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online [23:22:17] PROBLEM - lizardfs6 Current Load on lizardfs6 is WARNING: WARNING - load average: 15.19, 14.55, 14.05 [23:28:20] RECOVERY - lizardfs6 Current Load on lizardfs6 is OK: OK - load average: 11.42, 12.83, 13.50 [23:32:28] PROBLEM - lizardfs6 Current Load on lizardfs6 is CRITICAL: CRITICAL - load average: 16.99, 14.39, 13.88 [23:34:26] PROBLEM - lizardfs6 Current Load on lizardfs6 is WARNING: WARNING - load average: 11.78, 13.72, 13.73 [23:36:26] RECOVERY - lizardfs6 Current Load on lizardfs6 is OK: OK - load average: 9.00, 11.94, 13.07 [23:40:39] PROBLEM - lizardfs6 Current Load on lizardfs6 is CRITICAL: CRITICAL - load average: 19.18, 14.74, 13.90 [23:42:48] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 1 datacenter is down: 2400:6180:0:d0::403:f001/cpweb [23:42:58] PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 1 datacenter is down: 81.4.109.133/cpweb [23:44:40] PROBLEM - lizardfs6 Current Load on lizardfs6 is WARNING: WARNING - load average: 12.26, 14.35, 14.00 [23:44:43] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [23:44:56] RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online [23:46:42] PROBLEM - lizardfs6 Current Load on lizardfs6 is CRITICAL: CRITICAL - load average: 16.32, 14.56, 14.07 [23:48:41] PROBLEM - lizardfs6 Current Load on lizardfs6 is WARNING: WARNING - load average: 13.98, 15.50, 14.58 [23:52:34] PROBLEM - lizardfs6 Current Load on lizardfs6 is CRITICAL: CRITICAL - load average: 23.45, 17.71, 15.53 [23:59:54] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 4 datacenters are down: 2604:180:0:33b::2/cpweb, 128.199.139.216/cpweb, 2400:6180:0:d0::403:f001/cpweb, 2a00:d880:5:8ea::ebc7/cpweb