[00:04:12] PROBLEM - lizardfs2 Puppet on lizardfs2 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [00:12:11] RECOVERY - lizardfs2 Puppet on lizardfs2 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [00:28:42] PROBLEM - mw3 JobQueue on mw3 is CRITICAL: JOBQUEUE CRITICAL - job queue greater than 300 jobs. Current queue: 2807 [01:25:11] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fxZFm [01:25:12] [02miraheze/services] 07MirahezeSSLBot 03295a734 - BOT: Updating services config for wikis [01:33:52] PROBLEM - bacula1 Bacula Static Lizardfs2 on bacula1 is WARNING: WARNING: Full, 866559 files, 85.22GB, 2018-09-24 01:30:00 (2.1 weeks ago) [03:32:39] RECOVERY - mw3 JobQueue on mw3 is OK: JOBQUEUE OK - job queue below 300 jobs [05:47:04] PROBLEM - cp5 Varnish Backends on cp5 is CRITICAL: 1 backends are down. mw1 [05:47:20] PROBLEM - cp4 Varnish Backends on cp4 is CRITICAL: 1 backends are down. mw2 [05:49:20] RECOVERY - cp4 Varnish Backends on cp4 is OK: All 5 backends are healthy [05:50:14] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fxneh [05:50:16] [02miraheze/services] 07MirahezeSSLBot 03009c4ff - BOT: Updating services config for wikis [05:51:04] RECOVERY - cp5 Varnish Backends on cp5 is OK: All 5 backends are healthy [05:52:28] PROBLEM - misc4 phabricator.miraheze.org HTTPS on misc4 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [05:57:14] PROBLEM - misc4 phab.miraheze.wiki HTTPS on misc4 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [05:57:18] PROBLEM - cp5 Varnish Backends on cp5 is CRITICAL: 3 backends are down. mw1 mw2 mw3 [05:57:20] PROBLEM - cp4 Varnish Backends on cp4 is CRITICAL: 3 backends are down. mw1 mw2 mw3 [05:57:46] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 6 datacenters are down: 107.191.126.23/cpweb, 2604:180:0:33b::2/cpweb, 81.4.109.133/cpweb, 2a00:d880:5:8ea::ebc7/cpweb, 172.104.111.8/cpweb, 2400:8902::f03c:91ff:fe07:444e/cpweb [05:57:50] PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 6 datacenters are down: 107.191.126.23/cpweb, 2604:180:0:33b::2/cpweb, 81.4.109.133/cpweb, 2a00:d880:5:8ea::ebc7/cpweb, 172.104.111.8/cpweb, 2400:8902::f03c:91ff:fe07:444e/cpweb [05:57:54] PROBLEM - guiasdobrasil.com.br - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [05:59:37] PROBLEM - cp4 HTTP 4xx/5xx ERROR Rate on cp4 is WARNING: WARNING - NGINX Error Rate is 56% [06:00:11] PROBLEM - www.eerstelijnszones.be - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:00:29] PROBLEM - mw1 Puppet on mw1 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki config] [06:01:07] PROBLEM - docs.websmart.media - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:01:23] PROBLEM - cp5 HTTP 4xx/5xx ERROR Rate on cp5 is WARNING: WARNING - NGINX Error Rate is 42% [06:01:37] RECOVERY - cp4 HTTP 4xx/5xx ERROR Rate on cp4 is OK: OK - NGINX Error Rate is 37% [06:01:39] PROBLEM - hellointernet.miraheze.org - GlobalSign on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:01:45] PROBLEM - www.zenbuddhism.info - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:01:49] PROBLEM - wiki.ombre.io - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:01:55] PROBLEM - athenapedia.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:02:01] RECOVERY - www.guiasdobrasil.com.br - LetsEncrypt on sslhost is OK: OK - Certificate 'sni61771.cloudflaressl.com' will expire on Sat 30 Mar 2019 11:59:59 PM GMT +0000. [06:02:05] RECOVERY - guiasdobrasil.com.br - LetsEncrypt on sslhost is OK: OK - Certificate 'sni61771.cloudflaressl.com' will expire on Sat 30 Mar 2019 11:59:59 PM GMT +0000. [06:02:09] RECOVERY - www.eerstelijnszones.be - LetsEncrypt on sslhost is OK: OK - Certificate 'www.eerstelijnszones.be' will expire on Wed 31 Oct 2018 05:27:38 PM GMT +0000. [06:02:47] RECOVERY - misc4 phabricator.miraheze.org HTTPS on misc4 is OK: HTTP OK: HTTP/1.1 200 OK - 18290 bytes in 0.278 second response time [06:03:03] RECOVERY - docs.websmart.media - LetsEncrypt on sslhost is OK: OK - Certificate 'docs.websmart.media' will expire on Sat 15 Dec 2018 08:04:36 AM GMT +0000. [06:03:23] RECOVERY - misc4 phab.miraheze.wiki HTTPS on misc4 is OK: HTTP OK: Status line output matched "HTTP/1.1 500 Internal Server Error" - 9709 bytes in 0.165 second response time [06:03:35] RECOVERY - hellointernet.miraheze.org - GlobalSign on sslhost is OK: OK - Certificate '*.miraheze.org' will expire on Wed 23 Oct 2019 08:12:13 PM GMT +0000. [06:03:45] RECOVERY - wiki.ombre.io - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.ombre.io' will expire on Fri 21 Dec 2018 05:54:51 PM GMT +0000. [06:03:49] RECOVERY - wiki.valentinaproject.org - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.valentinaproject.org' will expire on Tue 04 Dec 2018 03:27:45 PM GMT +0000. [06:03:51] RECOVERY - enc.for.uz - LetsEncrypt on sslhost is OK: OK - Certificate 'enc.for.uz' will expire on Thu 01 Nov 2018 09:40:02 AM GMT +0000. [06:04:29] RECOVERY - mw1 Puppet on mw1 is OK: OK: Puppet is currently enabled, last run 55 seconds ago with 0 failures [06:05:21] RECOVERY - cp4 Varnish Backends on cp4 is OK: All 5 backends are healthy [06:05:25] PROBLEM - cp5 HTTP 4xx/5xx ERROR Rate on cp5 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [06:05:37] PROBLEM - private.revi.wiki - Comodo on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:05:45] PROBLEM - vsfan.net - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:06:03] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 2 datacenters are down: 172.104.111.8/cpweb, 2400:8902::f03c:91ff:fe07:444e/cpweb [06:06:23] PROBLEM - www.guiasdobrasil.com.br - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:06:32] PROBLEM - guiasdobrasil.com.br - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:07:13] RECOVERY - cp5 Varnish Backends on cp5 is OK: All 5 backends are healthy [06:07:19] RECOVERY - cp5 HTTP 4xx/5xx ERROR Rate on cp5 is OK: OK - NGINX Error Rate is 35% [06:07:35] PROBLEM - private.revi.wiki - Comodo on sslhost is WARNING: WARNING - Certificate 'private.revi.wiki' expires in 29 day(s) (Wed 07 Nov 2018 11:59:59 PM GMT +0000). [06:07:41] PROBLEM - vsfan.net - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'vsfan.net' expires in 14 day(s) (Wed 24 Oct 2018 01:46:20 AM GMT +0000). [06:08:07] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [06:10:29] RECOVERY - www.guiasdobrasil.com.br - LetsEncrypt on sslhost is OK: OK - Certificate 'sni61771.cloudflaressl.com' will expire on Sat 30 Mar 2019 11:59:59 PM GMT +0000. [06:10:39] RECOVERY - guiasdobrasil.com.br - LetsEncrypt on sslhost is OK: OK - Certificate 'sni61771.cloudflaressl.com' will expire on Sat 30 Mar 2019 11:59:59 PM GMT +0000. [06:11:53] RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online [06:18:38] what [06:18:46] sni61771.cloudflaressl.com? [06:18:50] that's not miraheze fleet [07:08:43] PROBLEM - mw3 JobQueue on mw3 is CRITICAL: JOBQUEUE CRITICAL - job queue greater than 300 jobs. Current queue: 2778 [08:06:40] RECOVERY - mw3 JobQueue on mw3 is OK: JOBQUEUE OK - job queue below 300 jobs [08:27:23] HTTP CRITICAL - Unable to open TCP socket [08:27:46] [02miraheze/CreateWiki] 07translatewiki pushed 031 commit to 03master [+0/-0/±2] 13https://git.io/fxnt7 [08:27:47] [02miraheze/CreateWiki] 07translatewiki 03ccb216a - Localisation updates from https://translatewiki.net. [08:27:49] [02miraheze/WikiDiscover] 07translatewiki pushed 031 commit to 03master [+1/-0/±1] 13https://git.io/fxnt5 [08:27:50] [02miraheze/WikiDiscover] 07translatewiki 03da49c0b - Localisation updates from https://translatewiki.net. [08:27:52] [02miraheze/MirahezeMagic] 07translatewiki pushed 031 commit to 03master [+13/-0/±17] 13https://git.io/fxntd [08:27:53] [02miraheze/MirahezeMagic] 07translatewiki 03b6abe98 - Localisation updates from https://translatewiki.net. [08:27:55] [02miraheze/ManageWiki] 07translatewiki pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fxntF [08:27:56] [02miraheze/ManageWiki] 07translatewiki 03acab11c - Localisation updates from https://translatewiki.net. [08:29:24] PROBLEM - private.revi.wiki - Comodo on sslhost is WARNING: WARNING - Certificate 'private.revi.wiki' expires in 29 day(s) (Wed 07 Nov 2018 11:59:59 PM GMT +0000). [08:30:06] HTTP CRITICAL - Unable to open TCP socket [08:30:15] ok [08:30:28] RECOVERY - public.revi.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'public.revi.wiki' will expire on Tue 13 Nov 2018 08:30:27 PM GMT +0000. [08:32:06] RECOVERY - www.reviwiki.info - PositiveSSLDV on sslhost is OK: OK - Certificate 'reviwiki.info' will expire on Wed 03 Feb 2021 11:59:59 PM GMT +0000. [08:47:37] RECOVERY - reviwiki.info - PositiveSSLDV on sslhost is OK: OK - Certificate 'reviwiki.info' will expire on Wed 03 Feb 2021 11:59:59 PM GMT +0000. [09:50:16] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fxnnO [09:50:17] [02miraheze/services] 07MirahezeSSLBot 037234feb - BOT: Updating services config for wikis [09:56:43] PROBLEM - mw3 JobQueue on mw3 is CRITICAL: JOBQUEUE CRITICAL - job queue greater than 300 jobs. Current queue: 5676 [10:32:53] PROBLEM - phab.miraheze.wiki on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:34:49] RECOVERY - phab.miraheze.wiki on sslhost is OK: OK - Certificate 'phab.miraheze.wiki' will expire on Thu 25 Oct 2018 01:35:01 PM GMT +0000. [10:53:28] mlange: are you sure you correctly entered your email [10:58:48] If you have, then someone on the Ops team (not me) can check the server logs [11:02:10] paladox: ^^ [11:13:26] Reception123: ^^ [11:14:02] At this point I have pinged 2 ops in hopes that they will respond [11:27:59] mlange: hi [11:29:04] I presume https://phabricator.miraheze.org/p/M.lange/ that’s you? [11:29:05] Title: [ ♟ M.lange ] - phabricator.miraheze.org [11:29:11] Looks like you resolved it? :) [11:30:15] :) [11:39:11] :) [11:43:57] [02mw-config] 07Amanda-Catherine opened pull request 03#2513: VisualEditor config for several wikis T3683 - 13https://git.io/fxngR [11:45:13] Amanda-Catherine/mw-config/patch-2/d114b26 - Amanda The build failed. https://travis-ci.com/Amanda-Catherine/mw-config/builds/87325896 [11:50:26] [02mw-config] 07Amanda-Catherine synchronize pull request 03#2513: VisualEditor config for several wikis T3683 - 13https://git.io/fxngR [11:57:07] [02mw-config] 07MacFan4000 reviewed pull request 03#2513 commit - 13https://git.io/fxnae [12:01:22] [02mw-config] 07Amanda-Catherine reviewed pull request 03#2513 commit - 13https://git.io/fxnaW [12:10:29] [02mw-config] 07JohnFLewis reviewed pull request 03#2513 commit - 13https://git.io/fxnVC [12:11:01] [02mw-config] 07JohnFLewis reviewed pull request 03#2513 commit - 13https://git.io/fxnV4 [12:16:01] [02mw-config] 07Amanda-Catherine reviewed pull request 03#2513 commit - 13https://git.io/fxnVS [12:48:21] [02mw-config] 07JohnFLewis synchronize pull request 03#2513: VisualEditor config for several wikis T3683 - 13https://git.io/fxngR [12:50:00] Amanda-Catherine/mw-config/patch-2/9402790 - John F. Lewis The build is still failing. https://travis-ci.com/Amanda-Catherine/mw-config/builds/87333621 [12:52:54] [02mw-config] 07MacFan4000 synchronize pull request 03#2513: VisualEditor config for several wikis T3683 - 13https://git.io/fxngR [12:54:17] Amanda-Catherine/mw-config/patch-2/e821457 - MacFan4000 The build was fixed. https://travis-ci.com/Amanda-Catherine/mw-config/builds/87334261 [12:54:51] [02mw-config] 07MacFan4000 closed pull request 03#2513: VisualEditor config for several wikis T3683 - 13https://git.io/fxngR [12:54:52] [02miraheze/mw-config] 07MacFan4000 pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fxnKl [12:54:53] [02miraheze/mw-config] 07Amanda-Catherine 03a6e4eaa - VisualEditor config for several wikis T3683 (#2513) * VisualEditor config for several wikis T3683 * Don’t know where that came from * em stray def * fix [13:10:43] Works for me [13:20:40] RECOVERY - mw3 JobQueue on mw3 is OK: JOBQUEUE OK - job queue below 300 jobs [13:37:13] PROBLEM - phab.miraheze.wiki on sslhost is WARNING: WARNING - Certificate 'phab.miraheze.wiki' expires in 15 day(s) (Thu 25 Oct 2018 01:35:01 PM GMT +0000). [13:39:01] PROBLEM - wiki.consentcraft.uk - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.consentcraft.uk' expires in 15 day(s) (Thu 25 Oct 2018 01:36:36 PM GMT +0000). [13:39:31] PROBLEM - wiki.zymonic.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.zymonic.com' expires in 15 day(s) (Thu 25 Oct 2018 01:37:27 PM GMT +0000). [14:02:49] PROBLEM - www.iceposeidonwiki.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'www.iceposeidonwiki.com' expires in 15 day(s) (Thu 25 Oct 2018 02:00:38 PM GMT +0000). [15:58:39] paladox: ^^ looks like the SSL bot isn’t working [15:58:44] yup [15:58:50] will look into that [16:08:47] PROBLEM - misc1 Puppet on misc1 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 6 minutes ago with 0 failures [16:11:19] Hola [16:17:29] Hola [16:36:29] PROBLEM - mw1 Puppet on mw1 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 5 minutes ago with 0 failures [16:44:23] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fxckP [16:44:25] [02miraheze/ssl] 07MirahezeSSLBot 03fdaa76d - Bot: Update SSL cert for www.iceposeidonwiki.com [16:52:48] RECOVERY - www.iceposeidonwiki.com - LetsEncrypt on sslhost is OK: OK - Certificate 'www.iceposeidonwiki.com' will expire on Mon 07 Jan 2019 03:44:18 PM GMT +0000. [17:11:07] [02miraheze/ssl] 07Reception123 pushed 031 commit to 03master [+1/-0/±1] 13https://git.io/fxctg [17:11:09] [02miraheze/ssl] 07Reception123 0396ea1a0 - add wiki.kybermc.tk cert [17:20:17] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fxcqE [17:20:18] [02miraheze/services] 07MirahezeSSLBot 0301fe144 - BOT: Updating services config for wikis [17:38:15] [02mw-config] 07Amanda-Catherine opened pull request 03#2514: Add parameter namespace for ecoepiwiki T3691 - 13https://git.io/fxcYL [17:39:30] Amanda-Catherine/mw-config/patch-1/82951fa - Amanda The build passed. https://travis-ci.com/Amanda-Catherine/mw-config/builds/87376415 [17:49:42] [02mw-config] 07Reception123 reviewed pull request 03#2514 commit - 13https://git.io/fxcOg [17:50:50] [02mw-config] 07Amanda-Catherine reviewed pull request 03#2514 commit - 13https://git.io/fxcOX [17:58:59] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fxc3Z [17:59:01] [02miraheze/puppet] 07paladox 035f9fa0a - Update mirahezerenewssl.py [17:59:53] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fxc3W [17:59:54] [02miraheze/puppet] 07paladox 038c683b3 - Fix script [18:05:53] PROBLEM - www.iceposeidonwiki.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'www.iceposeidonwiki.com' expires in 15 day(s) (Thu 25 Oct 2018 02:00:38 PM GMT +0000). [18:06:12] meh it's not renewing :? [18:13:31] RECOVERY - www.iceposeidonwiki.com - LetsEncrypt on sslhost is OK: OK - Certificate 'www.iceposeidonwiki.com' will expire on Mon 07 Jan 2019 03:44:18 PM GMT +0000. [18:14:25] PROBLEM - www.iceposeidonwiki.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'www.iceposeidonwiki.com' expires in 15 day(s) (Thu 25 Oct 2018 02:00:38 PM GMT +0000). [18:16:57] RECOVERY - www.iceposeidonwiki.com - LetsEncrypt on sslhost is OK: OK - Certificate 'www.iceposeidonwiki.com' will expire on Mon 07 Jan 2019 03:44:18 PM GMT +0000. [18:19:54] PROBLEM - www.iceposeidonwiki.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'www.iceposeidonwiki.com' expires in 15 day(s) (Thu 25 Oct 2018 02:00:38 PM GMT +0000). [18:22:28] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fxcGR [18:22:29] [02miraheze/puppet] 07paladox 038b11a0f - Add debug output to ssl-renew.sh [18:22:52] [02mw-config] 07mlange-42 reviewed pull request 03#2514 commit - 13https://git.io/fxcGE [18:24:00] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±0] 13https://git.io/fxcG2 [18:24:02] [02miraheze/puppet] 07paladox 033b00bf4 - Update ssl-renew.sh [18:26:37] [02puppet] 07paladox created branch 03paladox-patch-2 - 13https://git.io/vbiAS [18:26:38] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-2 [+0/-0/±1] 13https://git.io/fxcGQ [18:26:40] [02miraheze/puppet] 07paladox 030dcf715 - Remove -a param from ssl-renew It was unused. [18:26:41] [02puppet] 07paladox opened pull request 03#875: Remove -a param from ssl-renew - 13https://git.io/fxcG7 [18:27:24] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-2 [+0/-0/±1] 13https://git.io/fxcGd [18:27:25] [02miraheze/puppet] 07paladox 034c2d5d1 - Update ssl-acme [18:27:27] [02puppet] 07paladox synchronize pull request 03#875: Remove -a param from ssl-renew - 13https://git.io/fxcG7 [18:28:03] [02puppet] 07paladox closed pull request 03#875: Remove -a param from ssl-renew - 13https://git.io/fxcG7 [18:28:04] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±2] 13https://git.io/fxcGA [18:28:06] [02miraheze/puppet] 07paladox 0373ebe41 - Remove -a param from ssl-renew (#875) * Remove -a param from ssl-renew It was unused. * Update ssl-acme [18:28:07] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-2 [18:28:09] [02puppet] 07paladox deleted branch 03paladox-patch-2 - 13https://git.io/vbiAS [18:29:45] [02puppet] 07paladox created branch 03paladox-patch-2 - 13https://git.io/vbiAS [18:29:46] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-2 [+0/-0/±1] 13https://git.io/fxcZU [18:29:48] [02miraheze/puppet] 07paladox 034103b8c - Create acme log folder [18:29:49] [02puppet] 07paladox opened pull request 03#876: Create acme log folder - 13https://git.io/fxcZT [18:30:35] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-2 [+1/-0/±0] 13https://git.io/fxcZm [18:30:36] [02miraheze/puppet] 07paladox 038d9bfc2 - Create acme.logrotate.conf [18:30:38] [02puppet] 07paladox synchronize pull request 03#876: Create acme log folder - 13https://git.io/fxcZT [18:32:05] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-2 [+0/-0/±1] 13https://git.io/fxcZW [18:32:06] [02miraheze/puppet] 07paladox 03e671545 - Update mirahezerenewssl.py [18:32:08] [02puppet] 07paladox synchronize pull request 03#876: Create acme log folder - 13https://git.io/fxcZT [18:32:27] [02puppet] 07paladox closed pull request 03#876: Create acme log folder - 13https://git.io/fxcZT [18:32:29] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+1/-0/±2] 13https://git.io/fxcZl [18:32:30] [02miraheze/puppet] 07paladox 03380f3e0 - Create acme log folder (#876) * Create acme log folder * Create acme.logrotate.conf * Update mirahezerenewssl.py [18:32:32] [02puppet] 07paladox deleted branch 03paladox-patch-2 - 13https://git.io/vbiAS [18:32:33] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-2 [18:34:29] RECOVERY - mw1 Puppet on mw1 is OK: OK: Puppet is currently enabled, last run 57 seconds ago with 0 failures [18:34:47] RECOVERY - misc1 Puppet on misc1 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [18:35:13] RECOVERY - mw1 Puppet on mw1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [18:35:51] RECOVERY - misc1 Puppet on misc1 is OK: OK: Puppet is currently enabled, last run 13 seconds ago with 0 failures [18:37:09] RECOVERY - www.iceposeidonwiki.com - LetsEncrypt on sslhost is OK: OK - Certificate 'www.iceposeidonwiki.com' will expire on Mon 07 Jan 2019 03:44:18 PM GMT +0000. [18:38:13] PROBLEM - www.iceposeidonwiki.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'www.iceposeidonwiki.com' expires in 15 day(s) (Thu 25 Oct 2018 02:00:38 PM GMT +0000). [18:44:40] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fxcnR [18:44:42] [02miraheze/puppet] 07paladox 039494ecd - Update ssl-renew.sh [18:46:01] RECOVERY - www.iceposeidonwiki.com - LetsEncrypt on sslhost is OK: OK - Certificate 'www.iceposeidonwiki.com' will expire on Mon 07 Jan 2019 03:44:18 PM GMT +0000. [18:47:17] PROBLEM - www.iceposeidonwiki.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'www.iceposeidonwiki.com' expires in 15 day(s) (Thu 25 Oct 2018 02:00:38 PM GMT +0000). [18:49:12] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fxcnS [18:49:13] [02miraheze/ssl] 07MirahezeSSLBot 03ad7b450 - Bot: Update SSL cert for www.iceposeidonwiki.com [18:49:15] oh found the error [18:53:41] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fxcc3 [18:53:42] PROBLEM - puppet1 Puppet on puppet1 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_ssl] [18:53:43] [02miraheze/ssl] 07MirahezeSSLBot 038e02cb7 - Bot: Update SSL cert for www.iceposeidonwiki.com [18:53:50] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fxccs [18:53:51] [02miraheze/puppet] 07paladox 03fe9f665 - Fix script [18:54:56] RECOVERY - www.iceposeidonwiki.com - LetsEncrypt on sslhost is OK: OK - Certificate 'www.iceposeidonwiki.com' will expire on Mon 07 Jan 2019 05:53:35 PM GMT +0000. [18:56:00] PROBLEM - www.iceposeidonwiki.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'www.iceposeidonwiki.com' expires in 15 day(s) (Thu 25 Oct 2018 02:00:38 PM GMT +0000). [18:56:12] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fxccu [18:56:13] [02miraheze/ssl] 07MirahezeSSLBot 03e72f171 - Bot: Update SSL cert for www.iceposeidonwiki.com [18:56:35] woo works now :D [18:57:47] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fxccX [18:57:49] [02miraheze/ssl] 07MirahezeSSLBot 035c2d734 - Bot: Update SSL cert for wiki.consentcraft.uk [18:57:54] RECOVERY - www.iceposeidonwiki.com - LetsEncrypt on sslhost is OK: OK - Certificate 'www.iceposeidonwiki.com' will expire on Mon 07 Jan 2019 05:56:04 PM GMT +0000. [18:58:15] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fxccy [18:58:17] [02miraheze/ssl] 07MirahezeSSLBot 03cd9536a - Bot: Update SSL cert for vsfan.net [18:58:41] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fxccH [18:58:42] [02miraheze/ssl] 07MirahezeSSLBot 03033ea37 - Bot: Update SSL cert for wiki.zymonic.com [18:59:36] RECOVERY - puppet1 Puppet on puppet1 is OK: OK: Puppet is currently enabled, last run 1 second ago with 0 failures [18:59:54] RECOVERY - wiki.zymonic.com - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.zymonic.com' will expire on Mon 07 Jan 2019 05:58:35 PM GMT +0000. [18:59:59] RECOVERY - wiki.consentcraft.uk - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.consentcraft.uk' will expire on Mon 07 Jan 2019 05:57:42 PM GMT +0000. [19:00:04] RECOVERY - vsfan.net - LetsEncrypt on sslhost is OK: OK - Certificate 'vsfan.net' will expire on Mon 07 Jan 2019 05:58:10 PM GMT +0000. [19:02:26] [02puppet] 07paladox created branch 03paladox-patch-2 - 13https://git.io/vbiAS [19:02:28] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-2 [+0/-0/±1] 13https://git.io/fxcCI [19:02:29] [02miraheze/puppet] 07paladox 03a994963 - Move prometheus-varnish-exporter.systemd from varnish puppet repo [19:02:31] [02puppet] 07paladox opened pull request 03#877: Move prometheus-varnish-exporter.systemd from varnish puppet repo - 13https://git.io/fxcCL [19:02:52] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-2 [+1/-1/±0] 13https://git.io/fxcCY [19:02:54] [02miraheze/puppet] 07paladox 033c002e0 - Rename modules/varnish/templates/prometheus-varnish-exporter.systemd to modules/prometheus/templates/prometheus-varnish-exporter.systemd [19:02:55] [02puppet] 07paladox synchronize pull request 03#877: Move prometheus-varnish-exporter.systemd from varnish puppet repo - 13https://git.io/fxcCL [19:03:01] [02puppet] 07paladox closed pull request 03#877: Move prometheus-varnish-exporter.systemd from varnish puppet repo - 13https://git.io/fxcCL [19:03:03] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+1/-1/±1] 13https://git.io/fxcC3 [19:03:04] [02miraheze/puppet] 07paladox 03e154372 - Move prometheus-varnish-exporter.systemd from varnish puppet repo (#877) * Move prometheus-varnish-exporter.systemd from varnish puppet repo * Rename modules/varnish/templates/prometheus-varnish-exporter.systemd to modules/prometheus/templates/prometheus-varnish-exporter.systemd [19:03:06] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-2 [19:03:07] [02puppet] 07paladox deleted branch 03paladox-patch-2 - 13https://git.io/vbiAS [19:06:22] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fxcCV [19:06:23] [02miraheze/puppet] 07paladox 0306f1a24 - Add threaded=true to flask [19:21:16] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fxcWQ [19:21:18] [02miraheze/ssl] 07MirahezeSSLBot 03735fb07 - Bot: Update SSL cert for phab.miraheze.wiki [19:21:30] JohnLewis ^^ [19:22:55] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fxcWN [19:22:56] [02miraheze/puppet] 07paladox 03519a5eb - Redirect acme renewals to mw1 [19:23:08] PROBLEM - misc4 Puppet on misc4 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 12 minutes ago with 0 failures [19:24:42] RECOVERY - phab.miraheze.wiki on sslhost is OK: OK - Certificate 'phab.miraheze.wiki' will expire on Mon 07 Jan 2019 06:21:11 PM GMT +0000. [19:25:06] RECOVERY - misc4 Puppet on misc4 is OK: OK: Puppet is currently enabled, last run 29 seconds ago with 0 failures [19:29:47] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fxcl2 [19:29:49] [02miraheze/ssl] 07MirahezeSSLBot 03d27967c - Bot: Update SSL cert for phab.miraheze.wiki [19:29:58] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fxcla [19:30:00] [02miraheze/ssl] 07MirahezeSSLBot 03d360945 - Bot: Update SSL cert for phab.miraheze.wiki [19:30:04] just confirming. [22:34:50] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fxcPY [22:34:52] [02miraheze/puppet] 07paladox 035d896f9 - Reduce timeout from 400s to 240s in varnish [22:37:52] !log PURGE BINARY LOGS BEFORE '2018-10-09 23:37:00'; on db4 [22:37:58] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [22:38:46] PROBLEM - db4 Disk Space on db4 is WARNING: DISK WARNING - free space: / 55310 MB (15% inode=95%); [23:37:00] PROBLEM - mw3 JobQueue on mw3 is CRITICAL: JOBQUEUE CRITICAL - job queue greater than 300 jobs. Current queue: 2766