[00:01:03] [ANNOUNCEMENT] Happy New year to all in UTC [00:14:57] PROBLEM - cp4 Current Load on cp4 is CRITICAL: CRITICAL - load average: 3.06, 3.23, 2.16 [00:16:29] icinga-miraheze: congrats on your first issue of the decade [00:18:55] PROBLEM - cp4 Current Load on cp4 is WARNING: WARNING - load average: 0.42, 1.34, 1.73 [00:21:50] Yey welcome to 2020 MH-Discord [00:24:55] PROBLEM - cp4 Current Load on cp4 is CRITICAL: CRITICAL - load average: 5.00, 3.27, 2.39 [00:32:55] PROBLEM - cp4 Current Load on cp4 is WARNING: WARNING - load average: 1.65, 1.33, 1.93 [00:34:55] RECOVERY - cp4 Current Load on cp4 is OK: OK - load average: 0.17, 0.80, 1.58 [00:37:11] !log restart varnish cp2,cp4 [00:37:26] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [02:22:55] PROBLEM - cp4 Current Load on cp4 is CRITICAL: CRITICAL - load average: 0.93, 2.21, 1.59 [02:24:55] RECOVERY - cp4 Current Load on cp4 is OK: OK - load average: 0.20, 1.27, 1.36 [02:33:46] PROBLEM - cp4 Current Load on cp4 is WARNING: WARNING - load average: 1.39, 1.82, 1.50 [02:35:45] PROBLEM - cp4 Current Load on cp4 is CRITICAL: CRITICAL - load average: 0.98, 3.18, 2.33 [02:37:42] PROBLEM - cp4 Current Load on cp4 is WARNING: WARNING - load average: 0.16, 1.54, 1.85 [02:39:37] RECOVERY - cp4 Current Load on cp4 is OK: OK - load average: 0.21, 0.91, 1.54 [02:47:17] PROBLEM - cp4 Current Load on cp4 is CRITICAL: CRITICAL - load average: 0.29, 4.60, 4.28 [02:54:58] PROBLEM - cp4 Current Load on cp4 is WARNING: WARNING - load average: 0.08, 0.40, 1.76 [02:56:56] PROBLEM - cp4 Current Load on cp4 is CRITICAL: CRITICAL - load average: 12.96, 4.30, 2.80 [03:02:55] PROBLEM - cp4 Current Load on cp4 is WARNING: WARNING - load average: 0.56, 1.37, 1.84 [03:04:55] RECOVERY - cp4 Current Load on cp4 is OK: OK - load average: 0.12, 0.69, 1.46 [04:32:43] [02miraheze/mediawiki] 07paladox pushed 031 commit to 03REL1_34 [+0/-0/±2] 13https://git.io/JeAh1 [04:32:45] [02miraheze/mediawiki] 07paladox 03f108632 - Update QuizGame [04:35:58] !log root@mw1:/srv/mediawiki/w# /usr/local/bin/foreachwikiindblist /srv/mediawiki/dblist/all.dblist /srv/mediawiki/w/maintenance/sql.php /srv/mediawiki/w/extensions/QuizGame/sql/patches/patch-add-default-choice_answer_count.sql [04:36:06] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [04:41:42] [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JeAhy [04:41:44] [02miraheze/mw-config] 07paladox 0384706cb - Change https://mediawiki.org to https://www.mediawiki.org [04:41:45] [ MediaWiki ] - mediawiki.org [04:41:46] [ MediaWiki ] - www.mediawiki.org [05:07:28] PROBLEM - cp4 Current Load on cp4 is WARNING: WARNING - load average: 0.89, 2.00, 1.78 [05:09:22] RECOVERY - cp4 Current Load on cp4 is OK: OK - load average: 0.20, 1.19, 1.51 [06:01:34] PROBLEM - cp4 Current Load on cp4 is CRITICAL: CRITICAL - load average: 1.66, 2.62, 1.89 [06:05:29] RECOVERY - cp4 Current Load on cp4 is OK: OK - load average: 0.63, 1.21, 1.55 [06:16:05] PROBLEM - cp4 Current Load on cp4 is CRITICAL: CRITICAL - load average: 0.75, 1.99, 2.02 [06:19:54] PROBLEM - cp4 Current Load on cp4 is WARNING: WARNING - load average: 0.36, 1.26, 1.82 [06:21:51] RECOVERY - cp4 Current Load on cp4 is OK: OK - load average: 0.62, 0.95, 1.57 [06:26:38] RECOVERY - cp3 Disk Space on cp3 is OK: DISK OK - free space: / 3327 MB (13% inode=94%); [07:14:16] hi [07:14:29] Anyone? [08:18:55] PROBLEM - cp4 Current Load on cp4 is CRITICAL: CRITICAL - load average: 0.60, 2.40, 1.96 [08:20:56] RECOVERY - cp4 Current Load on cp4 is OK: OK - load average: 0.12, 1.23, 1.59 [10:17:33] PROBLEM - cp4 Current Load on cp4 is CRITICAL: CRITICAL - load average: 1.97, 2.17, 1.36 [10:19:29] RECOVERY - cp4 Current Load on cp4 is OK: OK - load average: 0.63, 1.23, 1.14 [11:09:30] PROBLEM - cp4 Current Load on cp4 is CRITICAL: CRITICAL - load average: 1.65, 3.29, 2.29 [11:17:08] PROBLEM - cp4 Current Load on cp4 is WARNING: WARNING - load average: 0.09, 1.20, 1.92 [11:19:03] RECOVERY - cp4 Current Load on cp4 is OK: OK - load average: 0.16, 0.60, 1.51 [11:52:42] PROBLEM - eerstelijnszones.be - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'eerstelijnszones.be' expires in 15 day(s) (Fri 17 Jan 2020 11:50:22 AM GMT +0000). [11:52:57] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JexJF [11:52:59] [02miraheze/ssl] 07MirahezeSSLBot 03fc571d1 - Bot: Update SSL cert for eerstelijnszones.be [11:53:11] PROBLEM - www.eerstelijnszones.be - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'eerstelijnszones.be' expires in 15 day(s) (Fri 17 Jan 2020 11:50:22 AM GMT +0000). [11:56:49] PROBLEM - poserdazfreebies.orain.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'orain.org' expires in 15 day(s) (Fri 17 Jan 2020 11:54:38 AM GMT +0000). [11:56:56] PROBLEM - meta.orain.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'orain.org' expires in 15 day(s) (Fri 17 Jan 2020 11:54:38 AM GMT +0000). [11:57:11] PROBLEM - orain.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'orain.org' expires in 15 day(s) (Fri 17 Jan 2020 11:54:38 AM GMT +0000). [11:57:52] PROBLEM - allthetropes.orain.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'orain.org' expires in 15 day(s) (Fri 17 Jan 2020 11:54:38 AM GMT +0000). [11:59:42] PROBLEM - miraheze.wiki - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'miraheze.wiki' expires in 15 day(s) (Fri 17 Jan 2020 11:57:22 AM GMT +0000). [12:01:51] PROBLEM - cp4 Current Load on cp4 is CRITICAL: CRITICAL - load average: 2.51, 1.59, 1.12 [12:02:00] PROBLEM - miraheze.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'miraheze.com' expires in 15 day(s) (Fri 17 Jan 2020 11:59:53 AM GMT +0000). [12:02:44] RECOVERY - eerstelijnszones.be - LetsEncrypt on sslhost is OK: OK - Certificate 'eerstelijnszones.be' will expire on Tue 31 Mar 2020 10:52:50 AM GMT +0000. [12:03:48] RECOVERY - www.eerstelijnszones.be - LetsEncrypt on sslhost is OK: OK - Certificate 'eerstelijnszones.be' will expire on Tue 31 Mar 2020 10:52:50 AM GMT +0000. [12:07:34] PROBLEM - m.miraheze.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'm.miraheze.org' expires in 15 day(s) (Fri 17 Jan 2020 12:04:11 PM GMT +0000). [12:19:28] PROBLEM - cp4 Current Load on cp4 is WARNING: WARNING - load average: 0.91, 1.23, 1.83 [12:21:23] PROBLEM - cp4 Current Load on cp4 is CRITICAL: CRITICAL - load average: 3.16, 2.32, 2.17 [12:25:12] PROBLEM - cp4 Current Load on cp4 is WARNING: WARNING - load average: 0.33, 1.25, 1.80 [12:27:06] RECOVERY - cp4 Current Load on cp4 is OK: OK - load average: 0.04, 0.63, 1.43 [12:43:00] PROBLEM - cp4 Current Load on cp4 is CRITICAL: CRITICAL - load average: 1.99, 2.18, 1.58 [12:48:56] RECOVERY - cp4 Current Load on cp4 is OK: OK - load average: 0.07, 1.00, 1.57 [12:54:55] PROBLEM - cp4 Current Load on cp4 is CRITICAL: CRITICAL - load average: 0.44, 2.21, 2.34 [13:00:55] PROBLEM - cp4 Current Load on cp4 is WARNING: WARNING - load average: 1.23, 1.62, 1.89 [13:06:55] PROBLEM - cp4 Current Load on cp4 is CRITICAL: CRITICAL - load average: 2.74, 2.18, 2.04 [13:08:55] PROBLEM - cp4 Current Load on cp4 is WARNING: WARNING - load average: 0.83, 1.30, 1.71 [13:10:55] PROBLEM - cp4 Current Load on cp4 is CRITICAL: CRITICAL - load average: 5.33, 4.40, 2.89 [13:20:55] PROBLEM - cp4 Current Load on cp4 is WARNING: WARNING - load average: 1.25, 1.36, 1.83 [13:22:55] PROBLEM - cp4 Current Load on cp4 is CRITICAL: CRITICAL - load average: 5.31, 3.41, 2.52 [13:40:59] PROBLEM - cp4 Current Load on cp4 is WARNING: WARNING - load average: 1.67, 1.15, 1.87 [13:42:55] RECOVERY - cp4 Current Load on cp4 is OK: OK - load average: 0.54, 0.90, 1.62 [13:46:58] PROBLEM - cp4 Current Load on cp4 is CRITICAL: CRITICAL - load average: 3.51, 3.86, 2.68 [13:52:58] PROBLEM - cp4 Current Load on cp4 is WARNING: WARNING - load average: 0.14, 0.98, 1.74 [13:54:57] RECOVERY - cp4 Current Load on cp4 is OK: OK - load average: 0.43, 0.68, 1.45 [13:58:07] Huh [13:58:08] Why is it doing that? [14:06:18] don't ask me :P [14:11:52] * hispano76 greeting [14:12:55] PROBLEM - cp4 Current Load on cp4 is CRITICAL: CRITICAL - load average: 0.72, 2.17, 1.49 [14:14:55] RECOVERY - cp4 Current Load on cp4 is OK: OK - load average: 0.15, 1.05, 1.18 [14:24:54] !log MariaDB [phabricator_herald]> UPDATE herald_rule SET isDisabled = '1' WHERE id = '21'; (empty) [14:24:56] hispano76: hi [14:25:04] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [14:26:03] hi Reception123 [14:28:45] [02miraheze/ssl] 07Reception123 pushed 031 commit to 03Reception123-patch-1 [+1/-0/±0] 13https://git.io/Jexke [14:28:47] [02miraheze/ssl] 07Reception123 03696fd67 - add wiki.twilightsignal.com cert [14:28:48] [02ssl] 07Reception123 created branch 03Reception123-patch-1 - 13https://git.io/vxP9L [14:28:50] [02ssl] 07Reception123 opened pull request 03#252: add wiki.twilightsignal.com cert - 13https://git.io/Jexkv [14:29:30] [02miraheze/ssl] 07Reception123 pushed 031 commit to 03Reception123-patch-1 [+0/-0/±1] 13https://git.io/JexkJ [14:29:32] [02miraheze/ssl] 07Reception123 03743c9a6 - add wiki.twilightsignal.com cert [14:29:33] [02ssl] 07Reception123 synchronize pull request 03#252: add wiki.twilightsignal.com cert - 13https://git.io/Jexkv [14:29:46] [02ssl] 07Reception123 closed pull request 03#252: add wiki.twilightsignal.com cert - 13https://git.io/Jexkv [14:29:47] [02miraheze/ssl] 07Reception123 pushed 031 commit to 03master [+1/-0/±1] 13https://git.io/JexkU [14:29:49] [02miraheze/ssl] 07Reception123 032fd6c38 - add wiki.twilightsignal.com cert (#252) * add wiki.twilightsignal.com cert * add wiki.twilightsignal.com cert [14:29:50] [02ssl] 07Reception123 deleted branch 03Reception123-patch-1 - 13https://git.io/vxP9L [14:29:52] [02miraheze/ssl] 07Reception123 deleted branch 03Reception123-patch-1 [14:35:07] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Jexkm [14:35:08] [02miraheze/services] 07MirahezeSSLBot 036438fd9 - BOT: Updating services config for wikis [14:44:41] Reception123: https://phabricator.miraheze.org/T4554 [14:44:42] [ ⚓ T4554 Extension:StopForumSpam ] - phabricator.miraheze.org [14:45:46] And https://phabricator.miraheze.org/T4962#96148 [14:45:47] [ ⚓ T4962 Please add filetype odg to sdiywiki ] - phabricator.miraheze.org [15:31:34] Happy new year JohnLewis [15:53:17] !log sudo -u www-data php fixStuckGlobalRename.php --wiki=200xwiki PaperClip "FireBarrier101" --logwiki=metawiki [15:53:28] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [15:53:49] RhinosF1: looking at the second [15:53:56] but haven't got time to figure out the first [15:56:13] Reception123: ok [16:54:59] RhinosF1: happy new year to you too [16:58:31] PROBLEM - cp4 Current Load on cp4 is CRITICAL: CRITICAL - load average: 2.05, 2.67, 1.96 [17:00:26] RECOVERY - cp4 Current Load on cp4 is OK: OK - load average: 0.62, 1.48, 1.62 [17:12:02] PROBLEM - cp4 Current Load on cp4 is CRITICAL: CRITICAL - load average: 2.17, 2.57, 1.83 [17:13:57] PROBLEM - cp4 Current Load on cp4 is WARNING: WARNING - load average: 0.77, 1.84, 1.71 [17:14:35] PROBLEM - cp3 Disk Space on cp3 is WARNING: DISK WARNING - free space: / 2649 MB (10% inode=94%); [17:19:49] RECOVERY - cp4 Current Load on cp4 is OK: OK - load average: 1.24, 1.24, 1.58 [17:42:52] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-11 [+0/-0/±1] 13https://git.io/Jexti [17:42:53] [02miraheze/puppet] 07paladox 03bab0508 - Remove labster from tech@ Per board [17:42:55] [02puppet] 07paladox created branch 03paladox-patch-11 - 13https://git.io/vbiAS [17:42:56] [02puppet] 07paladox opened pull request 03#1173: Remove labster from tech@ - 13https://git.io/JextP [17:43:06] [02puppet] 07paladox closed pull request 03#1173: Remove labster from tech@ - 13https://git.io/JextP [17:43:08] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JextX [17:43:09] [02miraheze/puppet] 07paladox 03c012916 - Remove labster from tech@ (#1173) Per board [17:43:11] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-11 [17:43:12] [02puppet] 07paladox deleted branch 03paladox-patch-11 - 13https://git.io/vbiAS [17:43:40] paladox: has that come from ferran? [17:46:00] RhinosF1 it came from owen [17:46:15] since NDA is now in-effect [17:49:33] !log root@misc1:/home/paladox# deluser labster [17:49:39] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [17:53:12] paladox: okay, he wasnt sure yesterday [17:55:44] PROBLEM - cp4 Current Load on cp4 is CRITICAL: CRITICAL - load average: 3.45, 2.88, 2.10 [17:57:44] PROBLEM - cp4 Current Load on cp4 is WARNING: WARNING - load average: 1.50, 1.92, 1.86 [18:01:44] RECOVERY - cp4 Current Load on cp4 is OK: OK - load average: 0.96, 1.46, 1.69 [18:05:13] [02miraheze/mediawiki] 07paladox pushed 031 commit to 03REL1_34 [+0/-0/±1] 13https://git.io/JexqJ [18:05:15] [02miraheze/mediawiki] 07paladox 03c721140 - Update SocialProfile [18:09:44] PROBLEM - cp4 Current Load on cp4 is CRITICAL: CRITICAL - load average: 4.53, 2.82, 2.13 [18:11:44] PROBLEM - cp4 Current Load on cp4 is WARNING: WARNING - load average: 1.41, 1.84, 1.86 [18:13:44] RECOVERY - cp4 Current Load on cp4 is OK: OK - load average: 1.48, 1.44, 1.68 [18:19:44] PROBLEM - cp4 Current Load on cp4 is CRITICAL: CRITICAL - load average: 2.16, 2.05, 1.87 [18:21:44] RECOVERY - cp4 Current Load on cp4 is OK: OK - load average: 1.25, 1.47, 1.66 [18:43:21] PROBLEM - cp4 Current Load on cp4 is CRITICAL: CRITICAL - load average: 0.98, 2.07, 1.78 [18:45:18] RECOVERY - cp4 Current Load on cp4 is OK: OK - load average: 0.40, 1.15, 1.47 [19:31:38] PROBLEM - cp3 HTTP 4xx/5xx ERROR Rate on cp3 is WARNING: WARNING - NGINX Error Rate is 49% [19:33:39] RECOVERY - cp3 HTTP 4xx/5xx ERROR Rate on cp3 is OK: OK - NGINX Error Rate is 38% [20:12:54] PROBLEM - cp4 Current Load on cp4 is CRITICAL: CRITICAL - load average: 5.12, 3.01, 1.98 [20:14:53] RECOVERY - cp4 Current Load on cp4 is OK: OK - load average: 0.57, 1.68, 1.66 [20:21:31] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Jex3e [20:21:32] [02miraheze/puppet] 07paladox 0318e5824 - varnish: Set thread_pool_min to 250 Default is 5. [20:21:54] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Jex3f [20:21:56] [02miraheze/puppet] 07paladox 03c18a193 - varnish: Set thread_pool_timeout to 120 Default is 300 [20:27:03] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Jex3G [20:27:05] [02miraheze/puppet] 07paladox 0301ab08d - varnish: Set workspace_backend to 128k [20:55:14] paladox: check tech@ [20:57:38] PROBLEM - cp3 HTTP 4xx/5xx ERROR Rate on cp3 is WARNING: WARNING - NGINX Error Rate is 43% [20:59:43] RECOVERY - cp3 HTTP 4xx/5xx ERROR Rate on cp3 is OK: OK - NGINX Error Rate is 30% [21:48:45] RhinosF1 yes you can close it for now. I doin't think we will be using it yet? [21:59:09] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JexGr [21:59:11] [02miraheze/puppet] 07paladox 0381ea842 - varnish: Set timeout_idle to 120 [22:12:50] [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JexGj [22:12:51] [02miraheze/mw-config] 07paladox 036d4aa7c - Move $wgConf->settings['wgMobileUrlTemplate'] outside foreach [22:18:14] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JexZ3 [22:18:15] [02miraheze/puppet] 07paladox 03256a0ac - Fix [22:21:44] PROBLEM - cp4 Current Load on cp4 is CRITICAL: CRITICAL - load average: 0.27, 4.54, 3.54 [22:25:51] paladox: I shouldn’t have had that email. I don’t have access anymore. [22:26:05] oh, i dunno if i have access. [22:26:13] as i didn't set that up. [22:26:23] paladox: you do [22:26:34] I gave you access [22:26:41] It’ll be an email somewhere [22:26:49] Or you’ll have an account [22:27:08] oh [22:27:47] PROBLEM - cp4 Current Load on cp4 is WARNING: WARNING - load average: 0.53, 1.00, 2.00 [22:27:48] ah, i do [22:28:14] paladox: all I want to know is whether stats on miraheze should be being sent to me and whether I should query it [22:28:31] Because I don’t have access to our account [22:29:47] PROBLEM - cp4 Current Load on cp4 is CRITICAL: CRITICAL - load average: 3.39, 2.56, 2.38 [22:30:33] I dunno how your getting it as i doin't see you under users [22:30:47] paladox: that’s my issue [22:31:03] Would you like me to email mailerlite and query it? [22:31:22] yup [22:31:42] .at 08:15 query that ml email [22:31:42] RhinosF1: Okay, I will set the reminder for: 2020-01-02 - 08:14:59UTC [22:31:52] Night [22:33:46] RECOVERY - cp4 Current Load on cp4 is OK: OK - load average: 0.42, 1.02, 1.69 [22:40:57] Hello Operator873_! If you have any questions, feel free to ask and someone should answer soon. [22:56:22] PROBLEM - cp3 Varnish Backends on cp3 is CRITICAL: 1 backends are down. mw2 [22:56:33] PROBLEM - test1 MediaWiki Rendering on test1 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:56:58] PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 3 datacenters are down: 107.191.126.23/cpweb, 2604:180:0:33b::2/cpweb, 128.199.139.216/cpweb [22:57:11] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 2 datacenters are down: 2400:6180:0:d0::403:f001/cpweb, 2a00:d880:5:8ea::ebc7/cpweb [22:58:28] RECOVERY - cp3 Varnish Backends on cp3 is OK: All 6 backends are healthy [22:58:31] RECOVERY - test1 MediaWiki Rendering on test1 is OK: HTTP OK: HTTP/1.1 200 OK - 18604 bytes in 0.404 second response time [22:58:55] RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online [23:03:30] PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 2 datacenters are down: 2604:180:0:33b::2/cpweb, 128.199.139.216/cpweb [23:09:28] PROBLEM - cp3 Varnish Backends on cp3 is CRITICAL: 1 backends are down. mw2 [23:13:34] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [23:13:34] RECOVERY - cp3 Varnish Backends on cp3 is OK: All 6 backends are healthy [23:16:33] PROBLEM - cp3 Stunnel Http for mw3 on cp3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [23:16:45] PROBLEM - cp4 Stunnel Http for mw3 on cp4 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [23:17:18] PROBLEM - test1 MediaWiki Rendering on test1 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:18:03] RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online [23:18:30] RECOVERY - cp3 Stunnel Http for mw3 on cp3 is OK: HTTP OK: HTTP/1.1 200 OK - 15190 bytes in 0.666 second response time [23:18:43] RECOVERY - cp4 Stunnel Http for mw3 on cp4 is OK: HTTP OK: HTTP/1.1 200 OK - 15190 bytes in 0.011 second response time [23:19:17] RECOVERY - test1 MediaWiki Rendering on test1 is OK: HTTP OK: HTTP/1.1 200 OK - 18602 bytes in 0.255 second response time [23:21:45] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 4 datacenters are down: 107.191.126.23/cpweb, 2604:180:0:33b::2/cpweb, 2400:6180:0:d0::403:f001/cpweb, 2a00:d880:5:8ea::ebc7/cpweb [23:22:31] PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 2 datacenters are down: 2604:180:0:33b::2/cpweb, 81.4.109.133/cpweb [23:26:14] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [23:26:34] RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online