[00:03:54] PROBLEM - lizardfs2 Puppet on lizardfs2 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [00:11:53] RECOVERY - lizardfs2 Puppet on lizardfs2 is OK: OK: Puppet is currently enabled, last run 59 seconds ago with 0 failures [05:08:39] RECOVERY - mw3 JobQueue on mw3 is OK: JOBQUEUE OK - job queue below 300 jobs [09:04:45] nihao ma [09:18:15] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/fAxN0 [09:18:16] [02miraheze/puppet] 07paladox 030bb5b2c - Update www-7.2.conf [09:18:18] [02puppet] 07paladox synchronize pull request 03#857: Update php-fpm config - 13https://git.io/fAxva [11:41:36] r 10 seconds [11:41:44] PROBLEM - wiki.zymonic.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:42:40] PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 1 datacenter is down: 2a00:d880:5:8ea::ebc7/cpweb [11:43:30] RECOVERY - antiguabarbudacalypso.com - LetsEncrypt on sslhost is OK: OK - Certificate 'antiguabarbudacalypso.com' will expire on Fri 19 Oct 2018 12:02:36 AM GMT +0000. [11:43:32] RECOVERY - wikipuk.cl - LetsEncrypt on sslhost is OK: OK - Certificate 'wikipuk.cl' will expire on Tue 04 Dec 2018 02:53:58 PM GMT +0000. [11:43:42] RECOVERY - wiki.zymonic.com - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.zymonic.com' will expire on Thu 25 Oct 2018 01:37:27 PM GMT +0000. [11:46:17] PROBLEM - adadevelopersacademy.wiki - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:46:39] PROBLEM - astrapedia.ru - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:48:17] RECOVERY - adadevelopersacademy.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'adadevelopersacademy.wiki' will expire on Thu 18 Oct 2018 01:36:00 AM GMT +0000. [11:48:37] RECOVERY - astrapedia.ru - LetsEncrypt on sslhost is OK: OK - Certificate 'astrapedia.ru' will expire on Fri 14 Dec 2018 01:26:46 PM GMT +0000. [11:50:37] PROBLEM - nonbinary.wiki - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:51:51] PROBLEM - meregos.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:52:41] PROBLEM - spiral.wiki - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:52:59] PROBLEM - thelonsdalebattalion.co.uk - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:53:25] PROBLEM - russopedia.info - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:53:29] PROBLEM - miraheze.wiki - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:53:49] RECOVERY - meregos.com - LetsEncrypt on sslhost is OK: OK - Certificate 'meregos.com' will expire on Wed 31 Oct 2018 05:26:54 PM GMT +0000. [11:54:39] RECOVERY - nonbinary.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'nonbinary.wiki' will expire on Wed 19 Dec 2018 03:48:18 PM GMT +0000. [11:54:49] PROBLEM - astrapedia.ru - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:55:19] RECOVERY - russopedia.info - LetsEncrypt on sslhost is OK: OK - Certificate 'russopedia.info' will expire on Fri 12 Oct 2018 12:32:34 PM GMT +0000. [11:55:21] PROBLEM - kkutu.wiki - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:55:25] RECOVERY - miraheze.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'miraheze.wiki' will expire on Wed 19 Dec 2018 03:51:18 PM GMT +0000. [11:56:45] RECOVERY - astrapedia.ru - LetsEncrypt on sslhost is OK: OK - Certificate 'astrapedia.ru' will expire on Fri 14 Dec 2018 01:26:46 PM GMT +0000. [11:56:51] RECOVERY - spiral.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'spiral.wiki' will expire on Thu 01 Nov 2018 08:51:11 PM GMT +0000. [11:57:03] RECOVERY - thelonsdalebattalion.co.uk - LetsEncrypt on sslhost is OK: OK - Certificate 'thelonsdalebattalion.co.uk' will expire on Sun 21 Oct 2018 09:52:39 AM GMT +0000. [11:57:17] RECOVERY - kkutu.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'kkutu.wiki' will expire on Fri 07 Dec 2018 10:20:18 AM GMT +0000. [11:57:35] PROBLEM - antiguabarbudacalypso.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:59:11] PROBLEM - savage-wiki.com - RapidSSL on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:01:32] PROBLEM - hgwiki.ga - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:03:10] RECOVERY - savage-wiki.com - RapidSSL on sslhost is OK: OK - Certificate 'savage-wiki.com' will expire on Fri 04 Dec 2020 12:00:00 PM GMT +0000. [12:03:28] RECOVERY - hgwiki.ga - LetsEncrypt on sslhost is OK: OK - Certificate 'hgwiki.ga' will expire on Tue 18 Dec 2018 04:13:07 PM GMT +0000. [12:05:44] RECOVERY - antiguabarbudacalypso.com - LetsEncrypt on sslhost is OK: OK - Certificate 'antiguabarbudacalypso.com' will expire on Fri 19 Oct 2018 12:02:36 AM GMT +0000. [12:12:36] PROBLEM - www.splat-teams.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:14:34] RECOVERY - www.splat-teams.com - LetsEncrypt on sslhost is OK: OK - Certificate 'www.splat-teams.com' will expire on Sat 08 Dec 2018 11:00:40 AM GMT +0000. [12:17:11] PROBLEM - www.programming.red - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:21:15] RECOVERY - www.programming.red - LetsEncrypt on sslhost is OK: OK - Certificate 'programming.red' will expire on Fri 19 Oct 2018 07:07:40 PM GMT +0000. [12:34:01] PROBLEM - russopedia.info - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'russopedia.info' expires in 15 day(s) (Fri 12 Oct 2018 12:32:34 PM GMT +0000). [12:34:16] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fAps5 [12:34:18] [02miraheze/ssl] 07MirahezeSSLBot 035978a87 - Bot: Update SSL cert for russopedia.info [12:34:51] PROBLEM - kkutu.wiki - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:38:53] RECOVERY - kkutu.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'kkutu.wiki' will expire on Fri 07 Dec 2018 10:20:18 AM GMT +0000. [12:41:49] RECOVERY - russopedia.info - LetsEncrypt on sslhost is OK: OK - Certificate 'russopedia.info' will expire on Tue 25 Dec 2018 11:34:11 AM GMT +0000. [12:42:04] paladox: bot? [12:42:30] Yep [12:42:36] Speaking gibberish [12:42:48] Lol [12:44:17] PROBLEM - astrapedia.ru - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:44:48] PROBLEM - pc-architect.tk - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:46:13] RECOVERY - astrapedia.ru - LetsEncrypt on sslhost is OK: OK - Certificate 'astrapedia.ru' will expire on Fri 14 Dec 2018 01:26:46 PM GMT +0000. [12:46:35] RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online [12:46:41] RECOVERY - pc-architect.tk - LetsEncrypt on sslhost is OK: OK - Certificate 'pc-architect.tk' will expire on Mon 17 Dec 2018 02:43:05 PM GMT +0000. [14:44:41] PROBLEM - mw3 JobQueue on mw3 is CRITICAL: JOBQUEUE CRITICAL - job queue greater than 300 jobs. Current queue: 6457 [15:01:16] https://phabricator.miraheze.org/T3658 [15:01:17] Title: [ ⚓ T3658 Extension:Duplicator does not work and MediaWiki error appears\Storage\IncompleteRevisionException ] - phabricator.miraheze.org [15:34:22] [02mw-config] 07ucronistaw opened pull request 03#2487: [Needs Review Before Approval] add tallercentralwiki on wgExtraInterlanguageLinkPrefixes - 13https://git.io/fApKS [17:18:40] RECOVERY - mw3 JobQueue on mw3 is OK: JOBQUEUE OK - job queue below 300 jobs [17:41:09] [02mw-config] 07Amanda-Catherine reviewed pull request 03#2487 commit - 13https://git.io/fApFG [17:51:50] [02mw-config] 07ucronistaw synchronize pull request 03#2487: [Needs Review Before Approval] add tallercentralwiki on wgExtraInterlanguageLinkPrefixes - 13https://git.io/fApKS [17:51:52] [02mw-config] 07ucronistaw reviewed pull request 03#2487 commit - 13https://git.io/fApbm [17:55:34] Hi [17:55:43] Hi [17:57:23] Can anyone see request #2487 and verify that there are no errors in Github? [18:10:16] [02mw-config] 07MacFan4000 synchronize pull request 03#2487: [Needs Review Before Approval] add tallercentralwiki on wgExtraInterlanguageLinkPrefixes - 13https://git.io/fApKS [18:10:49] [02mw-config] 07MacFan4000 closed pull request 03#2487: [Needs Review Before Approval] add tallercentralwiki on wgExtraInterlanguageLinkPrefixes - 13https://git.io/fApKS [18:10:50] [02miraheze/mw-config] 07MacFan4000 pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fApA8 [18:10:52] [02miraheze/mw-config] 07ucronistaw 03a381748 - [Needs Review Before Approval] add tallercentralwiki on wgExtraInterlanguageLinkPrefixes (#2487) * add tallercentralwiki on wgExtraInterlanguageLinkPrefixes * correction * array symtax [18:11:29] Wiki-1776: ^^ [18:12:03] ok, thanks [18:14:40] [02mw-config] 07JohnFLewis commented on pull request 03#2487: [Needs Review Before Approval] add tallercentralwiki on wgExtraInterlanguageLinkPrefixes - 13https://git.io/fApA7 [18:16:19] [02mw-config] 07paladox reviewed pull request 03#2487 commit - 13https://git.io/fApxt [18:31:46] [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fAppz [18:31:48] [02miraheze/mw-config] 07paladox 037e718c3 - fix spelling [18:45:37] PROBLEM - papelor.io - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'papelor.io' expires in 15 day(s) (Fri 12 Oct 2018 06:42:54 PM GMT +0000). [18:45:50] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fApjq [18:45:51] [02miraheze/ssl] 07MirahezeSSLBot 03576f4d8 - Bot: Update SSL cert for papelor.io [18:51:38] RECOVERY - papelor.io - LetsEncrypt on sslhost is OK: OK - Certificate 'papelor.io' will expire on Tue 25 Dec 2018 05:45:44 PM GMT +0000. [20:20:45] PROBLEM - takethatwiki.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:35:13] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fAhqo [20:35:15] [02miraheze/services] 07MirahezeSSLBot 0354ab3bf - BOT: Updating services config for wikis [20:40:15] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fAhmZ [20:40:17] [02miraheze/services] 07MirahezeSSLBot 039a32dea - BOT: Updating services config for wikis [21:32:43] PROBLEM - mw3 JobQueue on mw3 is CRITICAL: JOBQUEUE CRITICAL - job queue greater than 300 jobs. Current queue: 3192 [22:51:06] PROBLEM - takethatwiki.com - LetsEncrypt on sslhost is UNKNOWN: [22:53:14] PROBLEM - takethatwiki.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:58:24] !log upgraded icinga to icinga 2.9.2 on misc1 [22:58:29] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [23:02:52] Yeh, takethatwiki.com is expired [23:03:27] let's hope soon it will be "Back For Good" /me shows himself out [23:04:05] lol [23:06:52] why does expired cert mean = socket timeout ? [23:07:01] wouldnt it just mean a big fat browser warning [23:07:08] but not totally stopping the service [23:07:16] mutante because icinga cannot connect to port 443 [23:07:24] because the expired domain page is http:// [23:07:25] only [23:07:30] why cant it connect to the port? [23:07:32] No, the domain is expired, not the cert [23:07:46] ooh. that's different . ok! [23:15:13] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fAh8S [23:15:15] [02miraheze/services] 07MirahezeSSLBot 03fd95eab - BOT: Updating services config for wikis [23:16:24] [02ssl] 07MacFan4000 opened pull request 03#62: Rm takethatwiki.com cert - 13https://git.io/fAh8F [23:16:40] paladox: ^^ [23:16:48] ok [23:16:49] thanks [23:16:57] [02ssl] 07paladox closed pull request 03#62: Rm takethatwiki.com cert - 13https://git.io/fAh8F [23:16:59] [02miraheze/ssl] 07paladox pushed 031 commit to 03master [+0/-1/±1] 13https://git.io/fAh8x [23:17:00] [02miraheze/ssl] 07MacFan4000 031cbab9b - Rm takethatwiki.com cert (#62) * Update certs.yaml * Delete takethatwiki.com.crt [23:28:18] [02dns] 07MacFan4000 opened pull request 03#99: Rm takethatwiki.com zone - 13https://git.io/fAh4w [23:28:35] [02dns] 07paladox closed pull request 03#99: Rm takethatwiki.com zone - 13https://git.io/fAh4w [23:28:36] [02miraheze/dns] 07paladox pushed 031 commit to 03master [+0/-1/±0] 13https://git.io/fAh46 [23:28:38] [02miraheze/dns] 07MacFan4000 0351b734f - Rm takethatwiki.com zone (#99) [23:30:11] miraheze/dns/master/51b734f - MacFan4000 The build has errored. https://travis-ci.org/miraheze/dns/builds/433812385 [23:34:01] miraheze/dns/master/51b734f - MacFan4000 The build has errored. https://travis-ci.org/miraheze/dns/builds/433812385 [23:34:27] gzip: /home/travis/GeoLite2-Country.tar.gz: not in gzip format [23:34:36] Uh [23:34:42] paladox: ^^ [23:35:02] Restarted and same result [23:36:19] [02miraheze/dns] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fAhBJ [23:36:21] [02miraheze/dns] 07paladox 0338856b3 - Update travis [23:36:22] what format does it say it actuall is .. with "file /home/travis/GeoLite2-Country.tar.gz" [23:37:44] miraheze/dns/master/38856b3 - paladox The build has errored. https://travis-ci.org/miraheze/dns/builds/433815066 [23:38:25] Same error [23:39:52] [02miraheze/dns] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fAhBt [23:39:54] [02miraheze/dns] 07paladox 033ef434a - Use wget [23:41:05] " sudo curl https://geolite.maxmind.com/download/geoip/database/GeoLite2-Country.tar.gz --output " [23:41:09] curl: option --output: requires parameter [23:41:19] miraheze/dns/master/3ef434a - paladox The build has errored. https://travis-ci.org/miraheze/dns/builds/433816336 [23:41:28] running those commands doesnt work for me [23:41:32] different curl version? [23:41:50] https://www.irccloud.com/pastebin/rOZw0riG [23:41:50] Title: [ Snippet | IRCCloud ] - www.irccloud.com [23:42:13] It looks to be dns [23:42:26] saved [1756471/1756471] [23:42:31] GeoLite2-Country.tar.gz: gzip compressed data, last modified: Wed Sep 26 16:41:03 2018, from Unix [23:43:00] it resolves and then connects.. [23:43:10] if it was DNS how would it resolve to an IP [23:43:55] though.. you are right about this: [23:44:01] host 104.16.38.47 [23:44:01] Host 47.38.16.104.in-addr.arpa not found: 2(SERVFAIL) [23:44:05] no reverse at least [23:44:12] and SERVFAIL is bad [23:44:23] yep [23:44:46] Connecting to geolite.maxmind.com (geolite.maxmind.com)|104.16.37.47|:443... connected. [23:44:52] that is a different IP [23:45:12] 38.47 vs 37.47 [23:45:32] try using 104.16.37.47 with curl or wget [23:45:39] or hack /etc/hosts to it [23:46:11] hmm [23:46:25] miraheze/dns/master/3ef434a - paladox The build has errored. https://travis-ci.org/miraheze/dns/builds/433816336 [23:46:41] maybe they are rate limiting travis ip [23:47:36] they would/should use a different error code for that [23:47:37] 429 or so [23:47:44] not 503 or any 5xx [23:47:46] hmm [23:48:12] let's just try using that IP that works for me? [23:48:30] miraheze/dns/master/3ef434a - paladox The build has errored. https://travis-ci.org/miraheze/dns/builds/433816336 [23:48:39] ok [23:48:52] mutante how do i get it into /etc/hosts [23:49:33] curl --resolve 'geolite.maxmind.com:443:104.16.37.47' [23:49:40] try this .. might be even easier [23:49:40] oh i see [23:50:13] the /etc/hosts version would be a line: [23:50:24] 104.16.37.47 geolinte.maxmind.com [23:50:30] 104.16.37.47 geolite.maxmind.com [23:50:59] [02miraheze/dns] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fAhBV [23:51:01] [02miraheze/dns] 07paladox 0344b1db7 - GO back to curl [23:52:22] still fails with 'gzip: /home/travis/GeoLite2-Country.tar.gz: not in gzip format' [23:52:23] Back to not in gzip format [23:52:25] miraheze/dns/master/44b1db7 - paladox The build has errored. https://travis-ci.org/miraheze/dns/builds/433819980 [23:52:41] paladox: what is actually in that file? can you run "file" on it ? [23:53:04] can you open it with a text editor? [23:53:08] mutante it stores the mmm db i think [23:53:09] for ips [23:53:15] mutante and nope it's in travis [23:53:33] i dont mean what it _would_ be if it as gzip [23:53:36] it should contain "GeoLite2-Country.mmdb" [23:53:38] i mean what is it right now [23:53:41] since it's NOT gzip [23:53:42] ah ok [23:53:58] is it a text file with a curl error in it? [23:54:03] [02miraheze/dns] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fAhBP [23:54:05] [02miraheze/dns] 07paladox 030e18846 - Update .travis.yml [23:55:28] miraheze/dns/master/0e18846 - paladox The build has errored. https://travis-ci.org/miraheze/dns/builds/433820734 [23:55:34] it's a html document mutante [23:55:39] so i guess it's the same 503 [23:55:39] HTML document, ASCII text, with very long lines [23:55:53] [02miraheze/dns] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fAhBQ [23:55:54] [02miraheze/dns] 07paladox 036a0f949 - Update .travis.yml [23:57:19] miraheze/dns/master/6a0f949 - paladox The build has errored. https://travis-ci.org/miraheze/dns/builds/433821166 [23:57:22] hmm. ok.. the whole command works for me lilke that [23:58:07] Looking at that cat output it is showing that there is CloudFlare DDoS filtering [23:58:09] are you downloading this file often? [23:58:34] mutante everytime we edit the dns repo [23:58:46] why do you need to download it each and every time? [23:59:02] because we need it for our config [23:59:07] which we have geoip enabled [23:59:11] why dont you keep it after downloading it once [23:59:20] because travis deletes it [23:59:26] and also because it's not a free license [23:59:39] eh.. how about replacing curl with "cp" (from somewhere else in the file system) [23:59:59] travis deletes it after