[00:04:25] [02mw-config] 07Universal-Omega closed pull request 03#3755: add HasSomeColours skin (T6863) - 13https://git.io/JqTdj [00:04:27] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±4] 13https://git.io/JqTFa [00:04:28] [02miraheze/mw-config] 07Universal-Omega 03d32ab7a - add HasSomeColours skin (T6863) (#3755) [00:04:30] [02mw-config] 07Universal-Omega deleted branch 03Universal-Omega-patch-5 - 13https://git.io/vbvb3 [00:04:31] [02miraheze/mw-config] 07Universal-Omega deleted branch 03Universal-Omega-patch-5 [00:05:37] miraheze/mw-config - Universal-Omega the build passed. [00:12:56] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JqTF1 [00:12:58] [02miraheze/mw-config] 07Universal-Omega 0399ac949 - fix section for HasSomeColours [00:13:58] miraheze/mw-config - Universal-Omega the build passed. [01:12:43] [02puppet] 07Universal-Omega opened pull request 03#1680: Add X-Robots-Tag http header (To fix T6680) - 13https://git.io/JqTN1 [01:13:08] [02puppet] 07Universal-Omega synchronize pull request 03#1680: Add X-Robots-Tag http header (To fix T6680) - 13https://git.io/JqTN1 [01:26:04] [02puppet] 07Southparkfan commented on pull request 03#1680: Add X-Robots-Tag http header (To fix T6680) - 13https://git.io/JqTA4 [01:29:12] [02puppet] 07Universal-Omega commented on pull request 03#1680: Add X-Robots-Tag http header (To fix T6680) - 13https://git.io/JqTAg [01:47:37] [02miraheze/mediawiki] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://git.io/JqTx4 [01:47:39] [02miraheze/mediawiki] 07Universal-Omega 03f4c0f6e - Set Cargo to use REL1_35 branch [01:47:40] [02mediawiki] 07Universal-Omega created branch 03Universal-Omega-patch-1 - 13https://git.io/vbL5b [01:53:30] [02mediawiki] 07Universal-Omega opened pull request 03#1215: Set Cargo to use REL1_35 branch - 13https://git.io/JqTxK [05:27:13] PROBLEM - runzeppelin.ru - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for runzeppelin.ru could not be found [05:27:13] PROBLEM - stablestate.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for stablestate.org could not be found [05:33:53] RECOVERY - stablestate.org - reverse DNS on sslhost is OK: rDNS OK - stablestate.org reverse DNS resolves to cp11.miraheze.org [05:33:53] RECOVERY - runzeppelin.ru - reverse DNS on sslhost is OK: rDNS OK - runzeppelin.ru reverse DNS resolves to cp10.miraheze.org [06:39:02] !log renamed civclassicwiki to civwikiwiki. [06:39:06] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [06:40:48] [02miraheze/ssl] 07Reception123 pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JqkU3 [06:40:50] [02miraheze/ssl] 07Reception123 0336dfb27 - civclassic->civwikiwiki redirect [12:25:45] !log sudo -u www-data php /srv/mediawiki/w/maintenance/importDump.php --wiki callipediawiki /home/reception/Wikidump[01-07] --username-prefix="w" [01-03 on jbr3, 04-07 on jbr4) [12:25:48] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [12:48:34] [02miraheze/puppet] 07JohnFLewis pushed 031 commit to 03ssl-overhaul [+0/-0/±15] 13https://git.io/JqkCc [12:48:36] [02miraheze/puppet] 07JohnFLewis 0327d8941 - ssl: update ca regularly and move to capath config over cafile [12:48:37] [02puppet] 07JohnFLewis created branch 03ssl-overhaul - 13https://git.io/vbiAS [12:48:55] [02puppet] 07JohnFLewis opened pull request 03#1681: ssl: update ca regularly and move to capath config over cafile - 13https://git.io/JqkC8 [12:49:24] paladox: ^ can you review and merge if that looks good? Tested on test3 but want second review [13:03:43] Sure, I’ll look just after I have lunch. [13:53:43] i'm back [13:55:19] JohnLewis around? [13:55:33] yeah [13:56:49] JohnLewis is changing from /certs to /localcerts expect? [13:57:06] and if so, shouldn't the path be changed in nginx for certs/+ other places [13:57:14] yes, and I now realise that I didn't do that :P [14:01:35] :) [14:02:49] [02miraheze/puppet] 07JohnFLewis pushed 031 commit to 03ssl-overhaul [+0/-0/±19] 13https://git.io/Jqk43 [14:02:50] [02miraheze/puppet] 07JohnFLewis 0361728e3 - ssl: update ca regularly and move to capath config over cafile [14:02:52] [02puppet] 07JohnFLewis synchronize pull request 03#1681: ssl: update ca regularly and move to capath config over cafile - 13https://git.io/JqkC8 [14:02:59] paladox: there you go [14:03:07] looking [14:05:10] Looks good to me JohnLewis, merging! [14:05:33] great :) [14:05:45] [02puppet] 07paladox closed pull request 03#1681: ssl: update ca regularly and move to capath config over cafile - 13https://git.io/JqkC8 [14:05:47] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±19] 13https://git.io/Jqk48 [14:05:48] [02miraheze/puppet] 07JohnFLewis 0344ad3d5 - ssl: update ca regularly and move to capath config over cafile (#1681) [14:05:50] [02miraheze/puppet] 07paladox deleted branch 03ssl-overhaul [14:05:51] [02puppet] 07paladox deleted branch 03ssl-overhaul - 13https://git.io/vbiAS [14:06:57] ?[1;31mError: Could not retrieve catalog from remote server: Error 500 on SERVER: Server Error: Evaluation Error: Error while evaluating a Resource Statement, Ssl::Hiera::Certs[antiguabarbudacalypso]: has no parameter named 'ca' (file: /etc/puppetlabs/puppet/environments/production/modules/ssl/manifests/hiera.pp, line: 9) on node cp11.miraheze.org?[0m [14:06:59] JohnLewis [14:08:36] oh, i see what the $ca param is used for. [14:08:40] [02miraheze/puppet] 07JohnFLewis pushed 032 commits to 03master [+0/-0/±32] 13https://git.io/Jqk4K [14:08:41] [02miraheze/puppet] 07JohnFLewis 0327d8941 - ssl: update ca regularly and move to capath config over cafile [14:08:43] [02miraheze/puppet] 07JohnFLewis 03bcf2ea9 - add ca back to hiera.pp [14:08:48] PROBLEM - cp11 Puppet on cp11 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [14:08:53] move conflict JohnLewis [14:08:54] :P [14:08:57] *merge [14:09:32] [02miraheze/puppet] 07JohnFLewis pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Jqk4i [14:09:34] [02miraheze/puppet] 07JohnFLewis 038cadf85 - local conflict [14:10:34] PROBLEM - cp12 Puppet on cp12 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [14:10:35] PROBLEM - cp3 Puppet on cp3 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [14:10:48] PROBLEM - cp10 Puppet on cp10 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 14 seconds ago with 1 failures. Failed resources (up to 3 shown): Exec[nginx-syntax] [14:10:51] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Jqk4M [14:10:52] [02miraheze/puppet] 07paladox 032859761 - Add a note that $ca is expected to not be used [14:11:18] is it used in nginx? [14:11:26] I thought it was monitoring only [14:11:34] it's used in nginx [14:11:42] Where? [14:11:46] mediawiki [14:11:48] PROBLEM - services4 Puppet on services4 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[nginx] [14:11:52] ?[mNotice: /Stage[main]/Varnish::Nginx/Exec[nginx-syntax]/returns: nginx: [emerg] BIO_new_file("/etc/ssl/localcerts/wildcard.miraheze.org-2020-2.crt") failed (SSL: error:02001002:system library:fopen:No such file or directory:fopen('/etc/ssl/localcerts/wildcard.miraheze.org-2020-2.crt','r') error:2006D080:BIO routines:BIO_new_file:no such file)?[0m [14:12:07] where? [14:12:33] and that error is either the directory isn't created and thus the certs aren't, it's an order issue that one but a first run issue [14:12:46] oh [14:12:47] i'm wrong [14:13:11] it's cuss you have to update wildcard.pp JohnLewis [14:13:25] i could have sworn ca was used [14:13:47] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Jqk4F [14:13:48] [02miraheze/puppet] 07paladox 03387fe31 - Fix not for $ca [14:14:04] PROBLEM - services3 Puppet on services3 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[nginx] [14:14:15] [02miraheze/puppet] 07JohnFLewis pushed 032 commits to 03master [+0/-0/±2] 13https://git.io/Jqk4b [14:14:16] [02miraheze/puppet] 07JohnFLewis 034348538 - fix path, why is this a param? [14:14:18] [02miraheze/puppet] 07JohnFLewis 03ae869fd - Merge branch 'master' of github.com:/miraheze/puppet [14:14:21] ca was used, until you merged the change [14:15:04] ah [14:17:16] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JqkBO [14:17:17] [02miraheze/puppet] 07paladox 039b96ddb - ssl::cert: Fix path for ssl file [14:17:57] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JqkBs [14:17:59] [02miraheze/puppet] 07paladox 034f77255 - Update cert.pp [14:21:47] PROBLEM - dcmultiverse.miraheze.org - Sectigo on sslhost is CRITICAL: connect to address dcmultiverse.miraheze.org and port 443: Connection refusedHTTP CRITICAL - Unable to open TCP socket [14:21:47] PROBLEM - wiki.erisly.com - LetsEncrypt on sslhost is CRITICAL: connect to address wiki.erisly.com and port 443: Connection refusedHTTP CRITICAL - Unable to open TCP socket [14:21:47] PROBLEM - miraheze.wiki - LetsEncrypt on sslhost is CRITICAL: connect to address miraheze.wiki and port 443: Connection refusedHTTP CRITICAL - Unable to open TCP socket [14:21:47] PROBLEM - miraheze.ga - LetsEncrypt on sslhost is CRITICAL: connect to address miraheze.ga and port 443: Connection refusedHTTP CRITICAL - Unable to open TCP socket [14:21:47] PROBLEM - archive.stellurgists.wiki - LetsEncrypt on sslhost is CRITICAL: connect to address archive.stellurgists.wiki and port 443: Connection refusedHTTP CRITICAL - Unable to open TCP socket [14:21:55] PROBLEM - wiki.zymonic.com - LetsEncrypt on sslhost is CRITICAL: connect to address wiki.zymonic.com and port 443: Connection refusedHTTP CRITICAL - Unable to open TCP socket [14:22:04] PROBLEM - rangpurpedia.xyz - LetsEncrypt on sslhost is CRITICAL: connect to address rangpurpedia.xyz and port 443: Connection refusedHTTP CRITICAL - Unable to open TCP socket [14:22:05] PROBLEM - icclopedia.org - LetsEncrypt on sslhost is CRITICAL: connect to address icclopedia.org and port 443: Connection refusedHTTP CRITICAL - Unable to open TCP socket [14:22:05] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 6 datacenters are down: 51.195.236.219/cpweb, 51.195.236.250/cpweb, 2001:41d0:800:178a::5/cpweb, 2001:41d0:800:1bbd::4/cpweb, 51.222.25.132/cpweb, 2607:5300:205:200::1c30/cpweb [14:22:07] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 6 datacenters are down: 51.195.236.219/cpweb, 51.195.236.250/cpweb, 2001:41d0:800:178a::5/cpweb, 2001:41d0:800:1bbd::4/cpweb, 51.222.25.132/cpweb, 2607:5300:205:200::1c30/cpweb [14:22:08] PROBLEM - wiki.tulpa.info - LetsEncrypt on sslhost is CRITICAL: connect to address wiki.tulpa.info and port 443: Connection refusedHTTP CRITICAL - Unable to open TCP socket [14:22:18] PROBLEM - wiki.helioss.co - LetsEncrypt on sslhost is CRITICAL: connect to address wiki.helioss.co and port 443: Connection refusedHTTP CRITICAL - Unable to open TCP socket [14:22:22] PROBLEM - wiki.xysspon.com - LetsEncrypt on sslhost is CRITICAL: connect to address wiki.xysspon.com and port 443: Connection refusedHTTP CRITICAL - Unable to open TCP socket [14:22:35] PROBLEM - wiki.macc.nyc - LetsEncrypt on sslhost is CRITICAL: connect to address wiki.macc.nyc and port 443: Connection refusedHTTP CRITICAL - Unable to open TCP socket [14:22:43] PROBLEM - wiki.ricochetuniverse.com - LetsEncrypt on sslhost is CRITICAL: connect to address wiki.ricochetuniverse.com and port 443: Connection refusedHTTP CRITICAL - Unable to open TCP socket [14:22:46] PROBLEM - enc.for.uz - LetsEncrypt on sslhost is CRITICAL: connect to address enc.for.uz and port 443: Connection refusedHTTP CRITICAL - Unable to open TCP socket [14:22:47] PROBLEM - www.thesimswiki.com - LetsEncrypt on sslhost is CRITICAL: connect to address www.thesimswiki.com and port 443: Connection refusedHTTP CRITICAL - Unable to open TCP socket [14:22:47] PROBLEM - www.fraterniterebellion.com - LetsEncrypt on sslhost is CRITICAL: connect to address www.fraterniterebellion.com and port 443: Connection refusedHTTP CRITICAL - Unable to open TCP socket [14:22:47] PROBLEM - vitriol.spoon.army - LetsEncrypt on sslhost is CRITICAL: connect to address vitriol.spoon.army and port 443: Connection refusedHTTP CRITICAL - Unable to open TCP socket [14:22:48] PROBLEM - wiki.qadrishattari.xyz - LetsEncrypt on sslhost is CRITICAL: connect to address wiki.qadrishattari.xyz and port 443: Connection refusedHTTP CRITICAL - Unable to open TCP socket [14:22:48] PROBLEM - lukaibrisimovic.org - LetsEncrypt on sslhost is CRITICAL: connect to address lukaibrisimovic.org and port 443: Connection refusedHTTP CRITICAL - Unable to open TCP socket [14:22:49] PROBLEM - wiki.ddr.red - LetsEncrypt on sslhost is CRITICAL: connect to address wiki.ddr.red and port 443: Connection refusedHTTP CRITICAL - Unable to open TCP socket [14:22:52] PROBLEM - bharatwiki.online - LetsEncrypt on sslhost is CRITICAL: connect to address bharatwiki.online and port 443: Connection refusedHTTP CRITICAL - Unable to open TCP socket [14:22:52] PROBLEM - meta.orain.org - LetsEncrypt on sslhost is CRITICAL: connect to address meta.orain.org and port 443: Connection refusedHTTP CRITICAL - Unable to open TCP socket [14:22:52] PROBLEM - meta.gyaanipedia.com - LetsEncrypt on sslhost is CRITICAL: connect to address meta.gyaanipedia.com and port 443: Connection refusedHTTP CRITICAL - Unable to open TCP socket [14:22:53] PROBLEM - allthetropes.miraheze.org - Sectigo on sslhost is CRITICAL: connect to address allthetropes.miraheze.org and port 443: Connection refusedHTTP CRITICAL - Unable to open TCP socket [14:22:53] PROBLEM - private.revi.wiki - Sectigo on sslhost is CRITICAL: connect to address private.revi.wiki and port 443: Connection refusedHTTP CRITICAL - Unable to open TCP socket [14:22:56] PROBLEM - bluevies.org - LetsEncrypt on sslhost is CRITICAL: connect to address bluevies.org and port 443: Connection refusedHTTP CRITICAL - Unable to open TCP socket [14:22:59] PROBLEM - hololive.wiki - LetsEncrypt on sslhost is CRITICAL: connect to address hololive.wiki and port 443: Connection refusedHTTP CRITICAL - Unable to open TCP socket [14:22:59] PROBLEM - piwik.miraheze.org - Sectigo on sslhost is CRITICAL: connect to address piwik.miraheze.org and port 443: Connection refusedHTTP CRITICAL - Unable to open TCP socket [14:22:59] PROBLEM - marthaspeaks.miraheze.org - Sectigo on sslhost is CRITICAL: connect to address marthaspeaks.miraheze.org and port 443: Connection refusedHTTP CRITICAL - Unable to open TCP socket [14:22:59] PROBLEM - wc.miraheze.org on sslhost is CRITICAL: connect to address miraheze.org and port 443: Connection refusedHTTP CRITICAL - Unable to open TCP socket [14:23:01] paladox [14:23:04] uh [14:23:05] Mar 07 14:20:52 cp12.miraheze.org nginx[11092]: nginx: [emerg] BIO_new_file("/etc/ssl/localcerts/civclassic.civwiki.org.crt") failed (SSL: error:02001002:system library:fopen:No such file or directory:fopen('/etc/ssl/localcerts/civclassic.civwiki.org.crt','r') error:2006D080:BIO routines:BIO_new_file:no such file) [14:23:06] PROBLEM - pt.graalmilitary.com - LetsEncrypt on sslhost is CRITICAL: connect to address pt.graalmilitary.com and port 443: Connection refusedHTTP CRITICAL - Unable to open TCP socket [14:23:07] oh [14:23:08] PROBLEM - wiki.mineland.eu - LetsEncrypt on sslhost is CRITICAL: connect to address wiki.mineland.eu and port 443: Connection refusedHTTP CRITICAL - Unable to open TCP socket [14:23:09] PROBLEM - wiki.worldsofweary.com - LetsEncrypt on sslhost is CRITICAL: connect to address wiki.worldsofweary.com and port 443: Connection refusedHTTP CRITICAL - Unable to open TCP socket [14:23:10] PROBLEM - www.mh142.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for www.mh142.com could not be found [14:23:10] PROBLEM - marinebiodiversitymatrix.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for marinebiodiversitymatrix.org could not be found [14:23:13] PROBLEM - wikien.wildterra2.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wikien.wildterra2.com could not be found [14:23:14] PROBLEM - en.phgalaxy.xyz - LetsEncrypt on sslhost is CRITICAL: connect to address en.phgalaxy.xyz and port 443: Connection refusedHTTP CRITICAL - Unable to open TCP socket [14:23:15] going to revert [14:23:16] PROBLEM - wiki.cdntennis.ca - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.cdntennis.ca could not be found [14:23:17] PROBLEM - enc.for.uz - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for enc.for.uz could not be found [14:23:18] PROBLEM - nonbinary.wiki - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for nonbinary.wiki could not be found [14:23:18] PROBLEM - wiki.warframestat.us - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.warframestat.us could not be found [14:23:18] PROBLEM - wiki.make717.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.make717.org could not be found [14:23:24] PROBLEM - wiki.yunachannel.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.yunachannel.com could not be found [14:23:24] PROBLEM - wiki.etn.link - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.etn.link could not be found [14:23:25] no such file? [14:23:26] PROBLEM - wiki.counterculturelabs.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.counterculturelabs.org could not be found [14:23:26] PROBLEM - n64brew.dev - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for n64brew.dev could not be found [14:23:26] PROBLEM - ecole.science - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for ecole.science could not be found [14:23:27] PROBLEM - guia.esporo.net - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for guia.esporo.net could not be found [14:23:30] PROBLEM - wiki.warframestat.us - LetsEncrypt on sslhost is CRITICAL: connect to address wiki.warframestat.us and port 443: Connection refusedHTTP CRITICAL - Unable to open TCP socket [14:23:30] PROBLEM - pubwiki.lab.co.pl - LetsEncrypt on sslhost is CRITICAL: connect to address pubwiki.lab.co.pl and port 443: Connection refusedHTTP CRITICAL - Unable to open TCP socket [14:23:32] PROBLEM - ao90.pinho.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for ao90.pinho.org could not be found [14:23:33] PROBLEM - cp10 HTTPS on cp10 is CRITICAL: connect to address 51.195.236.219 and port 443: Connection refusedHTTP CRITICAL - Unable to open TCP socket [14:23:33] PROBLEM - wiki.elgeis.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.elgeis.com could not be found [14:23:34] PROBLEM - wiki.dom64.xyz - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.dom64.xyz could not be found [14:23:34] So the new files haven't actually be created? [14:23:34] PROBLEM - cp11 HTTPS on cp11 is CRITICAL: connect to address 51.195.236.250 and port 443: Connection refusedHTTP CRITICAL - Unable to open TCP socket [14:23:34] PROBLEM - en.nocyclo.tk - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for en.nocyclo.tk could not be found [14:23:34] PROBLEM - wiki.jacksonheights.nyc - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.jacksonheights.nyc could not be found [14:23:35] PROBLEM - www.fraterniterebellion.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for www.fraterniterebellion.com could not be found [14:23:35] PROBLEM - www.highdeologywiki.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for www.highdeologywiki.com could not be found [14:23:39] PROBLEM - orain.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for orain.org could not be found [14:23:40] PROBLEM - pastport.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for pastport.org could not be found [14:23:40] PROBLEM - wiki.caker18.com - LetsEncrypt on sslhost is CRITICAL: connect to address wiki.caker18.com and port 443: Connection refusedHTTP CRITICAL - Unable to open TCP socket [14:23:40] PROBLEM - miraheze.gq - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for miraheze.gq could not be found [14:23:43] PROBLEM - www.burnout.wiki - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for www.burnout.wiki could not be found [14:23:47] PROBLEM - dc.miraheze.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for dc.miraheze.org could not be found [14:23:47] PROBLEM - tradwiki.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for tradwiki.org could not be found [14:23:47] PROBLEM - wiki.raghuveer.net - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.raghuveer.net could not be found [14:23:49] PROBLEM - fr.gyaanipedia.com - LetsEncrypt on sslhost is CRITICAL: connect to address fr.gyaanipedia.com and port 443: Connection refusedHTTP CRITICAL - Unable to open TCP socket [14:23:49] PROBLEM - infectowiki.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for infectowiki.com could not be found [14:23:49] PROBLEM - datacrondatabase.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for datacrondatabase.com could not be found [14:23:49] PROBLEM - archive.a2b2.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for archive.a2b2.org could not be found [14:23:50] PROBLEM - wiki.fraterniterebellion.com - LetsEncrypt on sslhost is CRITICAL: connect to address wiki.fraterniterebellion.com and port 443: Connection refusedHTTP CRITICAL - Unable to open TCP socket [14:23:51] PROBLEM - wiki.nevillepedia.eu - LetsEncrypt on sslhost is CRITICAL: connect to address wiki.nevillepedia.eu and port 443: Connection refusedHTTP CRITICAL - Unable to open TCP socket [14:23:51] PROBLEM - wiki.teamrelectric.ca - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.teamrelectric.ca could not be found [14:23:51] PROBLEM - ipv6bolivia.tk - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for ipv6bolivia.tk could not be found [14:23:52] PROBLEM - fr.gyaanipedia.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for fr.gyaanipedia.com could not be found [14:23:53] JohnLewis yeh [14:23:53] PROBLEM - permanentfuturelab.wiki - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for permanentfuturelab.wiki could not be found [14:23:54] PROBLEM - wiki.park-city.club - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.park-city.club could not be found [14:23:55] PROBLEM - www.iceria.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for www.iceria.org could not be found [14:23:55] PROBLEM - iceria.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for iceria.org could not be found [14:23:55] PROBLEM - electowiki.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for electowiki.org could not be found [14:23:55] PROBLEM - wikiescola.com.br - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wikiescola.com.br could not be found [14:24:04] though [14:24:07] PROBLEM - allthetropes.miraheze.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for allthetropes.miraheze.org could not be found [14:24:07] PROBLEM - www.lab612.at - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for www.lab612.at could not be found [14:24:07] i see files in /etc/ssl/localcerts/ [14:24:09] PROBLEM - data.nonbinary.wiki - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for data.nonbinary.wiki could not be found [14:24:09] PROBLEM - savagepedia.wiki - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for savagepedia.wiki could not be found [14:24:10] PROBLEM - iesd.mobagenie.my.id - LetsEncrypt on sslhost is CRITICAL: connect to address iesd.mobagenie.my.id and port 443: Connection refusedHTTP CRITICAL - Unable to open TCP socket [14:24:12] PROBLEM - wiki.shaazzz.ir - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.shaazzz.ir could not be found [14:24:12] PROBLEM - wiki.jafaron.net - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.jafaron.net could not be found [14:24:13] PROBLEM - wiki.wysteria.ga - LetsEncrypt on sslhost is CRITICAL: connect to address wiki.wysteria.ga and port 443: Connection refusedHTTP CRITICAL - Unable to open TCP socket [14:24:14] PROBLEM - wiki.conworlds.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.conworlds.org could not be found [14:24:14] PROBLEM - es.publictestwiki.com - LetsEncrypt on sslhost is CRITICAL: connect to address es.publictestwiki.com and port 443: Connection refusedHTTP CRITICAL - Unable to open TCP socket [14:24:15] PROBLEM - dc-multiverse.dcwikis.com - LetsEncrypt on sslhost is CRITICAL: connect to address dc-multiverse.dcwikis.com and port 443: Connection refusedHTTP CRITICAL - Unable to open TCP socket [14:24:21] PROBLEM - mediawiki.ecfin.eu - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for mediawiki.ecfin.eu could not be found [14:24:22] PROBLEM - test2.miraheze.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for test2.miraheze.org could not be found [14:24:23] PROBLEM - pubwiki.lab.co.pl - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for pubwiki.lab.co.pl could not be found [14:24:23] PROBLEM - pa.gyaanipedia.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for pa.gyaanipedia.com could not be found [14:24:27] PROBLEM - runzeppelin.ru - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for runzeppelin.ru could not be found [14:24:34] PROBLEM - wiki.invinciblewellbeing.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.invinciblewellbeing.com could not be found [14:24:34] PROBLEM - wiki.wysteria.ga - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.wysteria.ga could not be found [14:24:35] PROBLEM - viileapedia.ga - LetsEncrypt on sslhost is CRITICAL: connect to address viileapedia.ga and port 443: Connection refusedHTTP CRITICAL - Unable to open TCP socket [14:24:36] PROBLEM - wiki.ciptamedia.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.ciptamedia.org could not be found [14:24:37] PROBLEM - www.wikimicrofinanza.it - LetsEncrypt on sslhost is CRITICAL: connect to address www.wikimicrofinanza.it and port 443: Connection refusedHTTP CRITICAL - Unable to open TCP socket [14:24:37] PROBLEM - encyclopediaofastrobiology.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for encyclopediaofastrobiology.org could not be found [14:24:37] PROBLEM - wiki.twilightsignal.com - LetsEncrypt on sslhost is CRITICAL: connect to address wiki.twilightsignal.com and port 443: Connection refusedHTTP CRITICAL - Unable to open TCP socket [14:24:38] PROBLEM - oecumene.org - LetsEncrypt on sslhost is CRITICAL: connect to address oecumene.org and port 443: Connection refusedHTTP CRITICAL - Unable to open TCP socket [14:24:38] PROBLEM - speleo.wiki - LetsEncrypt on sslhost is CRITICAL: connect to address speleo.wiki and port 443: Connection refusedHTTP CRITICAL - Unable to open TCP socket [14:24:41] PROBLEM - cp12 HTTPS on cp12 is CRITICAL: connect to address 51.222.25.132 and port 443: Connection refusedHTTP CRITICAL - Unable to open TCP socket [14:24:41] PROBLEM - wiki.climatechange.ai - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.climatechange.ai could not be found [14:24:41] PROBLEM - en.phgalaxy.xyz - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for en.phgalaxy.xyz could not be found [14:24:41] PROBLEM - wiki.fraterniterebellion.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.fraterniterebellion.com could not be found [14:24:42] PROBLEM - www.erikapedia.com - LetsEncrypt on sslhost is CRITICAL: connect to address www.erikapedia.com and port 443: Connection refusedHTTP CRITICAL - Unable to open TCP socket [14:24:42] PROBLEM - allthetropes.orain.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for allthetropes.orain.org could not be found [14:24:42] PROBLEM - wiki.candela.digital - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.candela.digital could not be found [14:24:42] PROBLEM - meta.gyaanipedia.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for meta.gyaanipedia.com could not be found [14:24:53] PROBLEM - chouverse.cludisciples.cyou - LetsEncrypt on sslhost is CRITICAL: connect to address chouverse.cludisciples.cyou and port 443: Connection refusedHTTP CRITICAL - Unable to open TCP socket [14:24:54] PROBLEM - vise.dayid.org - LetsEncrypt on sslhost is CRITICAL: connect to address vise.dayid.org and port 443: Connection refusedHTTP CRITICAL - Unable to open TCP socket [14:24:58] RECOVERY - wc.miraheze.org on sslhost is OK: OK - Certificate '*.miraheze.org' will expire on Sat 23 Oct 2021 23:59:59 GMT +0000. [14:25:33] RECOVERY - cp10 HTTPS on cp10 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 2208 bytes in 0.005 second response time [14:25:34] RECOVERY - cp11 HTTPS on cp11 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 2211 bytes in 0.012 second response time [14:26:20] oh [14:26:27] nvm the oh [14:26:40] RECOVERY - cp12 HTTPS on cp12 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 2211 bytes in 0.322 second response time [14:26:43] * dmehus scrolls through the icinga errors to find the conversations between paladox and JohnLewis [14:27:00] so the problem is the files never got created as mw10 has no files [14:28:04] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [14:28:05] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [14:28:34] RECOVERY - wiki.erisly.com - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.erisly.com' will expire on Fri 26 Mar 2021 00:23:45 GMT +0000. [14:28:34] RECOVERY - dcmultiverse.miraheze.org - Sectigo on sslhost is OK: OK - Certificate '*.miraheze.org' will expire on Sat 23 Oct 2021 23:59:59 GMT +0000. [14:28:34] RECOVERY - miraheze.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'miraheze.wiki' will expire on Fri 26 Mar 2021 02:32:28 GMT +0000. [14:28:35] RECOVERY - miraheze.ga - LetsEncrypt on sslhost is OK: OK - Certificate 'miraheze.ga' will expire on Fri 26 Mar 2021 01:16:07 GMT +0000. [14:28:35] RECOVERY - archive.stellurgists.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'archive.stellurgists.wiki' will expire on Fri 26 Mar 2021 00:59:54 GMT +0000. [14:28:46] RECOVERY - rangpurpedia.xyz - LetsEncrypt on sslhost is OK: OK - Certificate 'rangpurpedia.xyz' will expire on Mon 19 Apr 2021 07:02:10 GMT +0000. [14:28:46] RECOVERY - icclopedia.org - LetsEncrypt on sslhost is OK: OK - Certificate 'icclopedia.org' will expire on Fri 26 Mar 2021 01:09:09 GMT +0000. [14:28:49] RECOVERY - cp3 HTTPS on cp3 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 2206 bytes in 0.991 second response time [14:28:50] RECOVERY - wiki.zymonic.com - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.zymonic.com' will expire on Fri 26 Mar 2021 00:35:47 GMT +0000. [14:28:54] RECOVERY - wiki.tulpa.info - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.tulpa.info' will expire on Fri 26 Mar 2021 00:33:21 GMT +0000. [14:29:01] but how is that possible? [14:29:03] RECOVERY - wiki.xysspon.com - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.xysspon.com' will expire on Fri 26 Mar 2021 00:35:11 GMT +0000. [14:29:13] https://github.com/miraheze/puppet/blob/master/modules/ssl/manifests/hiera/certs.pp#L25 was correct [14:29:14] [ puppet/certs.pp at master · miraheze/puppet · GitHub ] - github.com [14:29:14] RECOVERY - wiki.helioss.co - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.helioss.co' will expire on Fri 26 Mar 2021 00:24:38 GMT +0000. [14:29:23] RECOVERY - wiki.ricochetuniverse.com - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.ricochetuniverse.com' will expire on Fri 26 Mar 2021 00:31:04 GMT +0000. [14:29:28] RECOVERY - www.thesimswiki.com - LetsEncrypt on sslhost is OK: OK - Certificate 'www.thesimswiki.com' will expire on Thu 25 Mar 2021 22:57:41 GMT +0000. [14:29:28] RECOVERY - wiki.macc.nyc - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.macc.nyc' will expire on Fri 26 Mar 2021 00:26:18 GMT +0000. [14:29:28] RECOVERY - enc.for.uz - LetsEncrypt on sslhost is OK: OK - Certificate 'enc.for.uz' will expire on Fri 26 Mar 2021 01:05:47 GMT +0000. [14:29:29] RECOVERY - www.fraterniterebellion.com - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.fraterniterebellion.com' will expire on Fri 26 Mar 2021 00:24:08 GMT +0000. [14:29:31] RECOVERY - vitriol.spoon.army - LetsEncrypt on sslhost is OK: OK - Certificate 'vitriol.spoon.army' will expire on Fri 26 Mar 2021 01:23:33 GMT +0000. [14:29:32] RECOVERY - wiki.qadrishattari.xyz - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.qadrishattari.xyz' will expire on Fri 26 Mar 2021 00:30:13 GMT +0000. [14:29:35] RECOVERY - wiki.ddr.red - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.ddr.red' will expire on Sat 29 May 2021 05:27:19 GMT +0000. [14:29:40] RECOVERY - meta.gyaanipedia.com - LetsEncrypt on sslhost is OK: OK - Certificate 'meta.gyaanipedia.com' will expire on Fri 26 Mar 2021 01:15:44 GMT +0000. [14:29:40] RECOVERY - bharatwiki.online - LetsEncrypt on sslhost is OK: OK - Certificate 'bharatwiki.online' will expire on Fri 26 Mar 2021 01:01:23 GMT +0000. [14:29:40] RECOVERY - meta.orain.org - LetsEncrypt on sslhost is OK: OK - Certificate 'orain.org' will expire on Fri 26 Mar 2021 02:34:11 GMT +0000. [14:29:43] RECOVERY - allthetropes.miraheze.org - Sectigo on sslhost is OK: OK - Certificate '*.miraheze.org' will expire on Sat 23 Oct 2021 23:59:59 GMT +0000. [14:29:43] RECOVERY - private.revi.wiki - Sectigo on sslhost is OK: OK - Certificate 'private.revi.wiki' will expire on Sat 06 Nov 2021 23:59:59 GMT +0000. [14:29:46] RECOVERY - lukaibrisimovic.org - LetsEncrypt on sslhost is OK: OK - Certificate 'www.lukaibrisimovic.org' will expire on Thu 25 Mar 2021 23:16:38 GMT +0000. [14:29:47] RECOVERY - pt.graalmilitary.com - LetsEncrypt on sslhost is OK: OK - Certificate 'pt.graalmilitary.com' will expire on Sat 24 Apr 2021 15:13:52 GMT +0000. [14:29:48] RECOVERY - bluevies.org - LetsEncrypt on sslhost is OK: OK - Certificate 'www.bluevies.org' will expire on Thu 25 Mar 2021 23:12:56 GMT +0000. [14:29:50] RECOVERY - wiki.mineland.eu - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.mineland.eu' will expire on Fri 26 Mar 2021 00:27:05 GMT +0000. [14:29:51] RECOVERY - hololive.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'hololive.wiki' will expire on Fri 26 Mar 2021 01:08:49 GMT +0000. [14:29:52] RECOVERY - piwik.miraheze.org - Sectigo on sslhost is OK: OK - Certificate '*.miraheze.org' will expire on Sat 23 Oct 2021 23:59:59 GMT +0000. [14:29:52] RECOVERY - marthaspeaks.miraheze.org - Sectigo on sslhost is OK: OK - Certificate '*.miraheze.org' will expire on Sat 23 Oct 2021 23:59:59 GMT +0000. [14:29:53] RECOVERY - wiki.worldsofweary.com - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.worldsofweary.com' will expire on Fri 26 Mar 2021 00:34:42 GMT +0000. [14:29:56] RECOVERY - www.mh142.com - reverse DNS on sslhost is OK: rDNS OK - www.mh142.com reverse DNS resolves to cp10.miraheze.org [14:29:57] RECOVERY - marinebiodiversitymatrix.org - reverse DNS on sslhost is OK: rDNS OK - marinebiodiversitymatrix.org reverse DNS resolves to cp10.miraheze.org [14:30:00] RECOVERY - wikien.wildterra2.com - reverse DNS on sslhost is OK: rDNS OK - wikien.wildterra2.com reverse DNS resolves to cp11.miraheze.org [14:30:06] RECOVERY - en.phgalaxy.xyz - LetsEncrypt on sslhost is OK: OK - Certificate 'en.phgalaxy.xyz' will expire on Fri 26 Mar 2021 01:05:20 GMT +0000. [14:30:06] RECOVERY - wiki.yunachannel.com - reverse DNS on sslhost is OK: rDNS OK - wiki.yunachannel.com reverse DNS resolves to cp11.miraheze.org [14:30:06] RECOVERY - wiki.etn.link - reverse DNS on sslhost is OK: rDNS OK - wiki.etn.link reverse DNS resolves to cp10.miraheze.org [14:30:08] RECOVERY - wiki.cdntennis.ca - reverse DNS on sslhost is OK: rDNS OK - wiki.cdntennis.ca reverse DNS resolves to cp11.miraheze.org [14:30:09] RECOVERY - wiki.counterculturelabs.org - reverse DNS on sslhost is OK: rDNS OK - wiki.counterculturelabs.org reverse DNS resolves to cp10.miraheze.org [14:30:09] RECOVERY - n64brew.dev - reverse DNS on sslhost is OK: rDNS OK - n64brew.dev reverse DNS resolves to cp11.miraheze.org [14:30:09] RECOVERY - ecole.science - reverse DNS on sslhost is OK: rDNS OK - ecole.science reverse DNS resolves to cp11.miraheze.org [14:30:10] RECOVERY - guia.esporo.net - reverse DNS on sslhost is OK: rDNS OK - guia.esporo.net reverse DNS resolves to cp11.miraheze.org [14:30:10] RECOVERY - enc.for.uz - reverse DNS on sslhost is OK: rDNS OK - enc.for.uz reverse DNS resolves to cp11.miraheze.org [14:30:11] RECOVERY - nonbinary.wiki - reverse DNS on sslhost is OK: rDNS OK - nonbinary.wiki reverse DNS resolves to cp10.miraheze.org [14:30:11] RECOVERY - wiki.make717.org - reverse DNS on sslhost is OK: rDNS OK - wiki.make717.org reverse DNS resolves to cp11.miraheze.org [14:30:16] RECOVERY - wiki.warframestat.us - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.warframestat.us' will expire on Fri 26 Mar 2021 00:34:29 GMT +0000. [14:30:17] RECOVERY - pubwiki.lab.co.pl - LetsEncrypt on sslhost is OK: OK - Certificate 'pubwiki.lab.co.pl' will expire on Fri 26 Mar 2021 01:20:09 GMT +0000. [14:30:17] RECOVERY - wiki.warframestat.us - reverse DNS on sslhost is OK: rDNS OK - wiki.warframestat.us reverse DNS resolves to cp11.miraheze.org [14:30:20] RECOVERY - ao90.pinho.org - reverse DNS on sslhost is OK: rDNS OK - ao90.pinho.org reverse DNS resolves to cp10.miraheze.org [14:30:20] breaking this down to the two natural stages it should have been done it from the start is problem the best way forward [14:30:20] RECOVERY - wiki.elgeis.com - reverse DNS on sslhost is OK: rDNS OK - wiki.elgeis.com reverse DNS resolves to cp11.miraheze.org [14:30:22] RECOVERY - wiki.dom64.xyz - reverse DNS on sslhost is OK: rDNS OK - wiki.dom64.xyz reverse DNS resolves to cp10.miraheze.org [14:30:23] RECOVERY - en.nocyclo.tk - reverse DNS on sslhost is OK: rDNS OK - en.nocyclo.tk reverse DNS resolves to cp11.miraheze.org [14:30:23] RECOVERY - wiki.jacksonheights.nyc - reverse DNS on sslhost is OK: rDNS OK - wiki.jacksonheights.nyc reverse DNS resolves to cp11.miraheze.org [14:30:25] RECOVERY - www.fraterniterebellion.com - reverse DNS on sslhost is OK: rDNS OK - www.fraterniterebellion.com reverse DNS resolves to cp11.miraheze.org [14:30:25] RECOVERY - www.highdeologywiki.com - reverse DNS on sslhost is OK: rDNS OK - www.highdeologywiki.com reverse DNS resolves to cp11.miraheze.org [14:30:27] RECOVERY - dc.miraheze.org - reverse DNS on sslhost is OK: rDNS OK - dc.miraheze.org reverse DNS resolves to cp11.miraheze.org [14:30:28] RECOVERY - tradwiki.org - reverse DNS on sslhost is OK: rDNS OK - tradwiki.org reverse DNS resolves to cp10.miraheze.org [14:30:28] RECOVERY - wiki.raghuveer.net - reverse DNS on sslhost is OK: rDNS OK - wiki.raghuveer.net reverse DNS resolves to cp10.miraheze.org [14:30:31] RECOVERY - fr.gyaanipedia.com - LetsEncrypt on sslhost is OK: OK - Certificate 'fr.gyaanipedia.com' will expire on Fri 26 Mar 2021 01:06:54 GMT +0000. [14:30:34] PROBLEM - cp12 Puppet on cp12 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 5 minutes ago with 1 failures [14:30:34] RECOVERY - wiki.caker18.com - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.caker18.com' will expire on Fri 28 May 2021 09:28:45 GMT +0000. [14:30:34] RECOVERY - infectowiki.com - reverse DNS on sslhost is OK: rDNS OK - infectowiki.com reverse DNS resolves to cp10.miraheze.org [14:30:34] PROBLEM - cp3 Puppet on cp3 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 3 minutes ago with 1 failures [14:30:34] RECOVERY - archive.a2b2.org - reverse DNS on sslhost is OK: rDNS OK - archive.a2b2.org reverse DNS resolves to cp10.miraheze.org [14:30:34] RECOVERY - datacrondatabase.com - reverse DNS on sslhost is OK: rDNS OK - datacrondatabase.com reverse DNS resolves to cp11.miraheze.org [14:30:35] RECOVERY - wiki.fraterniterebellion.com - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.fraterniterebellion.com' will expire on Fri 26 Mar 2021 00:24:08 GMT +0000. [14:30:35] RECOVERY - pastport.org - reverse DNS on sslhost is OK: rDNS OK - pastport.org reverse DNS resolves to cp11.miraheze.org [14:30:35] RECOVERY - miraheze.gq - reverse DNS on sslhost is OK: rDNS OK - miraheze.gq reverse DNS resolves to cp11.miraheze.org [14:30:36] RECOVERY - wiki.nevillepedia.eu - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.nevillepedia.eu' will expire on Mon 29 Mar 2021 18:51:58 GMT +0000. [14:30:37] RECOVERY - orain.org - reverse DNS on sslhost is OK: rDNS OK - orain.org reverse DNS resolves to cp11.miraheze.org [14:30:37] RECOVERY - wiki.teamrelectric.ca - reverse DNS on sslhost is OK: rDNS OK - wiki.teamrelectric.ca reverse DNS resolves to cp10.miraheze.org [14:30:48] PROBLEM - cp10 Puppet on cp10 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 6 minutes ago with 1 failures [14:30:48] PROBLEM - cp11 Puppet on cp11 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 6 minutes ago with 1 failures [14:30:49] RECOVERY - miraheze.wiki - reverse DNS on sslhost is OK: rDNS OK - miraheze.wiki reverse DNS resolves to cp11.miraheze.org [14:30:49] RECOVERY - wiki.responsibly.ai - reverse DNS on sslhost is OK: rDNS OK - wiki.responsibly.ai reverse DNS resolves to cp11.miraheze.org [14:30:51] RECOVERY - wiki.miraheze.org - reverse DNS on sslhost is OK: rDNS OK - wiki.miraheze.org reverse DNS resolves to cp10.miraheze.org [14:30:52] RECOVERY - wikipariksha.com - reverse DNS on sslhost is OK: rDNS OK - wikipariksha.com reverse DNS resolves to cp11.miraheze.org [14:30:52] RECOVERY - iesd.mobagenie.my.id - LetsEncrypt on sslhost is OK: OK - Certificate 'iesd.mobagenie.my.id' will expire on Fri 26 Mar 2021 01:09:38 GMT +0000. [14:30:55] RECOVERY - privacy-wiki.0x.no - reverse DNS on sslhost is OK: rDNS OK - privacy-wiki.0x.no reverse DNS resolves to cp11.miraheze.org [14:30:55] RECOVERY - wiki.shaazzz.ir - reverse DNS on sslhost is OK: rDNS OK - wiki.shaazzz.ir reverse DNS resolves to cp11.miraheze.org [14:30:58] RECOVERY - wiki.jafaron.net - reverse DNS on sslhost is OK: rDNS OK - wiki.jafaron.net reverse DNS resolves to cp10.miraheze.org [14:30:58] RECOVERY - pa.gyaanipedia.com - LetsEncrypt on sslhost is OK: OK - Certificate 'pa.gyaanipedia.com' will expire on Fri 26 Mar 2021 01:18:27 GMT +0000. [14:30:59] RECOVERY - wiki.mcpirevival.tk - reverse DNS on sslhost is OK: rDNS OK - wiki.mcpirevival.tk reverse DNS resolves to cp11.miraheze.org [14:30:59] RECOVERY - wiki.wysteria.ga - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.wysteria.ga' will expire on Fri 26 Mar 2021 00:34:57 GMT +0000. [14:30:59] RECOVERY - wiki.conworlds.org - reverse DNS on sslhost is OK: rDNS OK - wiki.conworlds.org reverse DNS resolves to cp10.miraheze.org [14:31:00] RECOVERY - es.publictestwiki.com - LetsEncrypt on sslhost is OK: OK - Certificate 'publictestwiki.com' will expire on Fri 26 Mar 2021 01:20:00 GMT +0000. [14:31:04] RECOVERY - www.lab612.at - reverse DNS on sslhost is OK: rDNS OK - www.lab612.at reverse DNS resolves to cp10.miraheze.org [14:31:05] RECOVERY - allthetropes.miraheze.org - reverse DNS on sslhost is OK: rDNS OK - allthetropes.miraheze.org reverse DNS resolves to cp10.miraheze.org [14:31:06] RECOVERY - data.nonbinary.wiki - reverse DNS on sslhost is OK: rDNS OK - data.nonbinary.wiki reverse DNS resolves to cp10.miraheze.org [14:31:06] RECOVERY - savagepedia.wiki - reverse DNS on sslhost is OK: rDNS OK - savagepedia.wiki reverse DNS resolves to cp10.miraheze.org [14:31:13] RECOVERY - dc-multiverse.dcwikis.com - LetsEncrypt on sslhost is OK: OK - Certificate 'dc-multiverse.dcwikis.com' will expire on Fri 26 Mar 2021 01:03:45 GMT +0000. [14:31:20] RECOVERY - pubwiki.lab.co.pl - reverse DNS on sslhost is OK: rDNS OK - pubwiki.lab.co.pl reverse DNS resolves to cp10.miraheze.org [14:31:20] RECOVERY - pa.gyaanipedia.com - reverse DNS on sslhost is OK: rDNS OK - pa.gyaanipedia.com reverse DNS resolves to cp10.miraheze.org [14:31:20] RECOVERY - viileapedia.ga - LetsEncrypt on sslhost is OK: OK - Certificate 'viileapedia.ga' will expire on Fri 26 Mar 2021 01:23:11 GMT +0000. [14:31:20] RECOVERY - wiki.invinciblewellbeing.com - reverse DNS on sslhost is OK: rDNS OK - wiki.invinciblewellbeing.com reverse DNS resolves to cp10.miraheze.org [14:31:21] RECOVERY - wiki.wysteria.ga - reverse DNS on sslhost is OK: rDNS OK - wiki.wysteria.ga reverse DNS resolves to cp11.miraheze.org [14:31:21] RECOVERY - mediawiki.ecfin.eu - reverse DNS on sslhost is OK: rDNS OK - mediawiki.ecfin.eu reverse DNS resolves to cp11.miraheze.org [14:31:22] RECOVERY - test2.miraheze.org - reverse DNS on sslhost is OK: rDNS OK - test2.miraheze.org reverse DNS resolves to cp11.miraheze.org [14:31:24] RECOVERY - wiki.ciptamedia.org - reverse DNS on sslhost is OK: rDNS OK - wiki.ciptamedia.org reverse DNS resolves to cp11.miraheze.org [14:31:25] RECOVERY - www.wikimicrofinanza.it - LetsEncrypt on sslhost is OK: OK - Certificate 'www.wikimicrofinanza.it' will expire on Thu 25 Mar 2021 23:12:18 GMT +0000. [14:31:25] RECOVERY - encyclopediaofastrobiology.org - reverse DNS on sslhost is OK: rDNS OK - encyclopediaofastrobiology.org reverse DNS resolves to cp10.miraheze.org [14:31:25] RECOVERY - wiki.twilightsignal.com - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.twilightsignal.com' will expire on Fri 26 Mar 2021 00:33:30 GMT +0000. [14:31:26] RECOVERY - runzeppelin.ru - reverse DNS on sslhost is OK: rDNS OK - runzeppelin.ru reverse DNS resolves to cp11.miraheze.org [14:31:28] RECOVERY - speleo.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'speleo.wiki' will expire on Fri 26 Mar 2021 01:21:37 GMT +0000. [14:31:29] RECOVERY - oecumene.org - LetsEncrypt on sslhost is OK: OK - Certificate 'oecumene.org' will expire on Fri 26 Mar 2021 01:18:12 GMT +0000. [14:31:31] RECOVERY - wiki.rebirthofthenight.com - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.rebirthofthenight.com' will expire on Fri 26 Mar 2021 00:30:39 GMT +0000. [14:31:33] RECOVERY - wiki.climatechange.ai - reverse DNS on sslhost is OK: rDNS OK - wiki.climatechange.ai reverse DNS resolves to cp10.miraheze.org [14:31:33] RECOVERY - en.phgalaxy.xyz - reverse DNS on sslhost is OK: rDNS OK - en.phgalaxy.xyz reverse DNS resolves to cp10.miraheze.org [14:31:34] RECOVERY - www.erikapedia.com - LetsEncrypt on sslhost is OK: OK - Certificate 'www.erikapedia.com' will expire on Thu 25 Mar 2021 23:15:05 GMT +0000. [14:31:34] RECOVERY - wiki.fraterniterebellion.com - reverse DNS on sslhost is OK: rDNS OK - wiki.fraterniterebellion.com reverse DNS resolves to cp11.miraheze.org [14:31:35] RECOVERY - allthetropes.orain.org - reverse DNS on sslhost is OK: rDNS OK - allthetropes.orain.org reverse DNS resolves to cp11.miraheze.org [14:31:35] RECOVERY - wiki.candela.digital - reverse DNS on sslhost is OK: rDNS OK - wiki.candela.digital reverse DNS resolves to cp11.miraheze.org [14:31:36] RECOVERY - wiki.thesimswiki.com - LetsEncrypt on sslhost is OK: OK - Certificate 'www.thesimswiki.com' will expire on Thu 25 Mar 2021 22:57:41 GMT +0000. [14:31:36] RECOVERY - meta.gyaanipedia.com - reverse DNS on sslhost is OK: rDNS OK - meta.gyaanipedia.com reverse DNS resolves to cp10.miraheze.org [14:31:36] RECOVERY - chouverse.cludisciples.cyou - LetsEncrypt on sslhost is OK: OK - Certificate 'chouverse.cludisciples.cyou' will expire on Fri 26 Mar 2021 01:13:19 GMT +0000. [14:31:38] RECOVERY - vise.dayid.org - LetsEncrypt on sslhost is OK: OK - Certificate 'vise.dayid.org' will expire on Fri 26 Mar 2021 01:23:27 GMT +0000. [14:31:45] RECOVERY - zw.fontainebleau-avon.fr - reverse DNS on sslhost is OK: rDNS OK - zw.fontainebleau-avon.fr reverse DNS resolves to cp10.miraheze.org [14:32:50] PROBLEM - db11 Puppet on db11 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 13 minutes ago with 0 failures [14:33:02] PROBLEM - db13 Puppet on db13 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 11 minutes ago with 0 failures [14:33:22] PROBLEM - dbbackup2 Puppet on dbbackup2 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 20 minutes ago with 0 failures [14:33:52] PROBLEM - dbbackup1 Puppet on dbbackup1 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 4 minutes ago with 0 failures [14:33:57] PROBLEM - mon2 Puppet on mon2 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 13 minutes ago with 0 failures [14:34:02] PROBLEM - gluster4 Puppet on gluster4 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 9 minutes ago with 0 failures [14:34:16] PROBLEM - gluster3 Puppet on gluster3 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 15 minutes ago with 0 failures [14:34:20] PROBLEM - db12 Puppet on db12 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 15 minutes ago with 0 failures [14:35:45] PROBLEM - mw9 Puppet on mw9 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[nginx-syntax] [14:36:05] PROBLEM - mw10 Puppet on mw10 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[nginx-syntax] [14:36:29] PROBLEM - mw8 Puppet on mw8 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[nginx-syntax] [14:36:37] that begs another question - why didn't the nginx-syntax safeguard detect this? [14:37:00] PROBLEM - jobrunner4 Puppet on jobrunner4 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[nginx-syntax] [14:37:41] PROBLEM - mw11 Puppet on mw11 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[nginx-syntax] [14:37:41] PROBLEM - jobrunner3 Puppet on jobrunner3 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[nginx-syntax] [14:38:26] JohnLewis found why [14:38:28] fixing [14:38:57] paladox: for which part? files not getting created, nginx restarting when it shouldn't have? [14:39:07] nginx restarting when it shouldn't have [14:39:24] Great [14:39:24] !log reception@jobrunner3:/srv/mediawiki/w/extensions/CentralAuth/maintenance$ sudo -u www-data php migrateAccount.php --wiki loginwiki --username Trafficcone3 --attachbroken [14:39:27] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [14:39:32] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JqkBA [14:39:34] [02miraheze/puppet] 07paladox 0300350b8 - varnish: Fix ssl::cert -> m.miraheze.org so it doesn't restart if there are errors [14:39:36] JohnLewis ^ [14:39:48] RECOVERY - services4 Puppet on services4 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [14:40:37] JohnLewis i suspect defined() is the cause of the certs not being installed. [14:41:16] Okay, you look at that while I do the incident report [14:42:03] RECOVERY - services3 Puppet on services3 is OK: OK: Puppet is currently enabled, last run 48 seconds ago with 0 failures [14:44:03] RECOVERY - mw10 Puppet on mw10 is OK: OK: Puppet is currently enabled, last run 20 seconds ago with 0 failures [14:44:21] I am wondering what caused the recent downtimes. Have they been investigated? [14:46:14] R4356th: https://github.com/miraheze/puppet/compare/e2bc175a502b...44ad3d509391 being merged, but https://git.io/JqkBA was what caused the actual incident, nginx restarted when it shouldn't have. That second commit fixes the actual incident cause, the primary cause of why nginx was failing is currently under investigation [14:46:15] [ Comparing e2bc175a502b...44ad3d509391 · miraheze/puppet · GitHub ] - github.com [14:46:16] [ Comparing 4f77255fe858...00350b8a8a03 · miraheze/puppet · GitHub ] - git.io [14:47:45] Oh, I see. What happened last night though? [14:48:20] There was an outage last night? [14:49:43] Yes. [14:49:52] A brief one but globally. [14:50:06] PROBLEM - mw10 Puppet on mw10 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [14:52:05] RECOVERY - mw10 Puppet on mw10 is OK: OK: Puppet is currently enabled, last run 16 seconds ago with 0 failures [14:52:21] Any more information on that? Icinga didn't report anything, there's nothing in my emails that suggests it and as far as I am aware there's been no user reports that I've seen or I've been told about from others [14:53:32] ok, so i've added a notice and i don't see civclassic.civwiki.org in ssl::hiera::hiera [14:53:51] *hiera::certs [14:55:19] I myself was not online when that happened but there were reports in #cvt. [14:55:35] [02miraheze/ssl] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JqkRa [14:55:37] [02miraheze/ssl] 07paladox 03cbd2299 - Fix whitespace [14:55:44] And https://status.miraheze.wiki/ seems to mention it. [14:55:45] [ Status Page - Miraheze ] - status.miraheze.wiki [14:56:41] (And I got an automated email.) [14:56:45] Oh, yesterday - I have no idea, I didn't look into that, but there wasn't one last night [14:57:36] Guess I should had made it clear that it was last night my time. ¯_(ツ)_/¯ [14:58:53] i think that could have been the outage i did the IR for JohnLewis [14:59:26] paladox: it was [15:01:59] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JqkRx [15:02:01] [02miraheze/puppet] 07paladox 03f2db289 - Remove duplication [15:02:58] RECOVERY - jobrunner4 Puppet on jobrunner4 is OK: OK: Puppet is currently enabled, last run 9 seconds ago with 0 failures [15:03:11] paladox: link if it has been published (I do not see anything on RC)? [15:03:41] RECOVERY - mw11 Puppet on mw11 is OK: OK: Puppet is currently enabled, last run 30 seconds ago with 0 failures [15:03:45] RECOVERY - mw9 Puppet on mw9 is OK: OK: Puppet is currently enabled, last run 16 seconds ago with 0 failures [15:03:48] it's waiting for JohnLewis to ok it [15:03:51] https://meta.miraheze.org/wiki/Special:IncidentReports/37/ [15:03:52] [ Permission error - Miraheze Meta ] - meta.miraheze.org [15:04:23] There's some comments, but I want to focus on this on-going incident than review and deal with a past incident [15:04:28] RECOVERY - mw8 Puppet on mw8 is OK: OK: Puppet is currently enabled, last run 58 seconds ago with 0 failures [15:04:38] 👍 [15:05:39] RECOVERY - jobrunner3 Puppet on jobrunner3 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [15:14:18] [02miraheze/ssl] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/Jqk08 [15:14:20] [02miraheze/ssl] 07paladox 0341b88cc - Fix indentation [15:14:21] [02ssl] 07paladox created branch 03paladox-patch-1 - 13https://git.io/vxP9L [15:14:23] [02ssl] 07paladox opened pull request 03#394: Fix indentation - 13https://git.io/Jqk04 [15:14:31] paladox: any advancement over why files were created yet? [15:14:53] No, but i have found that one of them is not being included in hiera::certs [15:15:04] i used notice() and i don't see it [15:15:09] but i see others [15:15:15] Strange [15:15:51] oh!!! [15:15:59] * paladox looks at Reception123 [15:16:30] [02miraheze/ssl] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Jqk01 [15:16:32] [02miraheze/ssl] 07paladox 03213b05d - Fix duplicate tag [15:16:37] JohnLewis ^ [15:17:01] mkay [15:17:08] [02ssl] 07paladox closed pull request 03#394: Fix indentation - 13https://git.io/Jqk04 [15:17:10] [02miraheze/ssl] 07paladox deleted branch 03paladox-patch-1 [15:17:11] [02ssl] 07paladox deleted branch 03paladox-patch-1 - 13https://git.io/vxP9L [15:17:24] [02miraheze/ssl] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/Jqk0M [15:17:25] [02miraheze/ssl] 07paladox 037de3613 - Fix indentation [15:17:27] [02ssl] 07paladox created branch 03paladox-patch-1 - 13https://git.io/vxP9L [15:17:28] [02ssl] 07paladox opened pull request 03#395: Fix indentation - 13https://git.io/Jqk0D [15:18:33] [02miraheze/ssl] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/Jqk0H [15:18:34] [02miraheze/ssl] 07paladox 03672cc66 - Update redirects.yaml [15:18:36] [02ssl] 07paladox synchronize pull request 03#395: Fix indentation - 13https://git.io/Jqk0D [15:18:59] Ah, one too many duplicate cert tags [15:19:12] [02miraheze/ssl] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/Jqk07 [15:19:14] [02miraheze/ssl] 07paladox 033ccb50c - Update redirects.yaml [15:19:15] [02ssl] 07paladox synchronize pull request 03#395: Fix indentation - 13https://git.io/Jqk0D [15:20:37] Ok, we should add yamllint to ci for that repo [15:22:10] Sounds like a good idea to do later [15:22:33] my bad :( [15:22:49] PROBLEM - cp10 Puppet on cp10 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[nginx-reload] [15:22:50] PROBLEM - cp11 Puppet on cp11 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[nginx-reload] [15:23:00] running puppet on cp* now JohnLewis [15:23:21] it works :D [15:24:32] RECOVERY - cp12 Puppet on cp12 is OK: OK: Puppet is currently enabled, last run 16 seconds ago with 0 failures [15:24:33] PROBLEM - cp3 Puppet on cp3 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[nginx-reload] [15:24:37] [02ssl] 07paladox edited pull request 03#395: Fix indentation and whitespace - 13https://git.io/Jqk0D [15:24:50] RECOVERY - cp10 Puppet on cp10 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [15:24:51] RECOVERY - cp11 Puppet on cp11 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [15:26:33] RECOVERY - cp3 Puppet on cp3 is OK: OK: Puppet is currently enabled, last run 32 seconds ago with 0 failures [15:26:52] RECOVERY - db11 Puppet on db11 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [15:27:04] RECOVERY - db13 Puppet on db13 is OK: OK: Puppet is currently enabled, last run 12 seconds ago with 0 failures [15:27:21] RECOVERY - dbbackup2 Puppet on dbbackup2 is OK: OK: Puppet is currently enabled, last run 19 seconds ago with 0 failures [15:27:55] RECOVERY - dbbackup1 Puppet on dbbackup1 is OK: OK: Puppet is currently enabled, last run 38 seconds ago with 0 failures [15:28:05] RECOVERY - gluster4 Puppet on gluster4 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [15:28:16] RECOVERY - gluster3 Puppet on gluster3 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [15:28:20] RECOVERY - db12 Puppet on db12 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [15:29:58] RECOVERY - mon2 Puppet on mon2 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [15:32:09] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JqkEr [15:32:10] [02miraheze/puppet] 07paladox 03213bf29 - openldap: Remove provider debian [15:45:29] [02miraheze/ssl] 07paladox pushed 031 commit to 03master [+0/-0/±2] 13https://git.io/Jqku0 [15:45:30] [02miraheze/ssl] 07paladox 0326e946f - Fix indentation and whitespace (#395) [15:45:32] [02ssl] 07paladox closed pull request 03#395: Fix indentation and whitespace - 13https://git.io/Jqk0D [15:45:33] [02ssl] 07paladox deleted branch 03paladox-patch-1 - 13https://git.io/vxP9L [15:45:35] [02miraheze/ssl] 07paladox deleted branch 03paladox-patch-1 [15:55:10] [02puppet] 07JohnFLewis closed pull request 03#1602: Add The Internet Archive to CSP whitelist - 13https://git.io/Jtn5V [15:55:52] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/Jqkzt [15:55:54] [02miraheze/puppet] 07paladox 03ca71219 - nginx: Use a systemd override to check for syntax [15:55:55] [02puppet] 07paladox created branch 03paladox-patch-1 - 13https://git.io/vbiAS [15:55:57] [02puppet] 07paladox opened pull request 03#1682: nginx: Use a systemd override to check for syntax - 13https://git.io/Jqkzq [15:59:27] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/Jqkzs [15:59:29] [02miraheze/puppet] 07paladox 03640b18a - Update site.pp [15:59:30] [02puppet] 07paladox synchronize pull request 03#1682: nginx: Use a systemd override to check for syntax - 13https://git.io/Jqkzq [16:01:06] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/JqkzB [16:01:07] [02miraheze/puppet] 07paladox 034438cf0 - Update init.pp [16:01:09] [02puppet] 07paladox synchronize pull request 03#1682: nginx: Use a systemd override to check for syntax - 13https://git.io/Jqkzq [16:02:08] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/Jqkz0 [16:02:10] [02miraheze/puppet] 07paladox 03362925c - Update elasticsearch.pp [16:02:11] [02puppet] 07paladox synchronize pull request 03#1682: nginx: Use a systemd override to check for syntax - 13https://git.io/Jqkzq [16:03:25] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/Jqkza [16:03:27] [02miraheze/puppet] 07paladox 03c11cb25 - Update init.pp [16:03:28] [02puppet] 07paladox synchronize pull request 03#1682: nginx: Use a systemd override to check for syntax - 13https://git.io/Jqkzq [16:03:51] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/JqkzV [16:03:53] [02miraheze/puppet] 07paladox 03b26a874 - Update nginx.pp [16:03:54] [02puppet] 07paladox synchronize pull request 03#1682: nginx: Use a systemd override to check for syntax - 13https://git.io/Jqkzq [16:05:30] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/JqkzP [16:05:31] [02miraheze/puppet] 07paladox 03598476d - Update nginx.pp [16:05:33] [02puppet] 07paladox synchronize pull request 03#1682: nginx: Use a systemd override to check for syntax - 13https://git.io/Jqkzq [16:06:03] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/JqkzX [16:06:04] [02miraheze/puppet] 07paladox 038a99409 - Update mathoid.pp [16:06:06] [02puppet] 07paladox synchronize pull request 03#1682: nginx: Use a systemd override to check for syntax - 13https://git.io/Jqkzq [16:06:57] !log reception@jobrunner3:~$ sudo -u www-data php /srv/mediawiki/w/maintenance/deleteBatch.php --wiki=teessidehackspacewiki -u "Dmehus" -r "Spam [[m:Stewards|Steward]] [[phab:T6930|request]] on [[m:Phabricator|Phabricator]]" /home/reception/teessidehackspace.txt [16:07:00] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/JqkzM [16:07:01] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:07:01] [02miraheze/puppet] 07paladox 038fd4f63 - Update init.pp [16:07:03] [02puppet] 07paladox synchronize pull request 03#1682: nginx: Use a systemd override to check for syntax - 13https://git.io/Jqkzq [16:07:26] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/JqkzD [16:07:28] [02miraheze/puppet] 07paladox 03175443e - Update restbase.pp [16:07:29] [02puppet] 07paladox synchronize pull request 03#1682: nginx: Use a systemd override to check for syntax - 13https://git.io/Jqkzq [16:07:58] [02ManageWiki] 07Universal-Omega closed pull request 03#255: Multiple improvements to ManageWikiNamespaces - 13https://git.io/JqTwL [16:08:00] [02miraheze/ManageWiki] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±4] 13https://git.io/Jqkz9 [16:08:01] [02miraheze/ManageWiki] 07Universal-Omega 0373d6750 - Multiple improvements to ManageWikiNamespaces (#255) [16:08:03] [02ManageWiki] 07Universal-Omega deleted branch 03Universal-Omega-patch-2 - 13https://git.io/vpSns [16:08:04] [02miraheze/ManageWiki] 07Universal-Omega deleted branch 03Universal-Omega-patch-2 [16:08:59] miraheze/ManageWiki - Universal-Omega the build passed. [16:09:12] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/JqkzN [16:09:14] [02miraheze/puppet] 07paladox 03318bba8 - Update init.pp [16:09:15] [02puppet] 07paladox synchronize pull request 03#1682: nginx: Use a systemd override to check for syntax - 13https://git.io/Jqkzq [16:11:53] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/JqkgI [16:11:54] [02miraheze/puppet] 07paladox 03fb37c14 - Update cert.pp [16:11:56] [02puppet] 07paladox synchronize pull request 03#1682: nginx: Use a systemd override to check for syntax - 13https://git.io/Jqkzq [16:13:10] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/JqkgZ [16:13:12] [02miraheze/puppet] 07paladox 033c84d1c - Update certs.pp [16:13:13] [02puppet] 07paladox synchronize pull request 03#1682: nginx: Use a systemd override to check for syntax - 13https://git.io/Jqkzq [16:13:50] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/Jqkgn [16:13:52] [02miraheze/puppet] 07paladox 03fb5e439 - Update wildcard.pp [16:13:53] [02puppet] 07paladox synchronize pull request 03#1682: nginx: Use a systemd override to check for syntax - 13https://git.io/Jqkzq [16:14:10] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/Jqkgc [16:14:11] [02miraheze/puppet] 07paladox 035c49b06 - Update nginx.pp [16:14:12] [02puppet] 07paladox synchronize pull request 03#1682: nginx: Use a systemd override to check for syntax - 13https://git.io/Jqkzq [16:15:32] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/Jqkg8 [16:15:34] [02miraheze/puppet] 07paladox 031df04c2 - Update init.pp [16:15:35] [02puppet] 07paladox synchronize pull request 03#1682: nginx: Use a systemd override to check for syntax - 13https://git.io/Jqkzq [16:18:06] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-1 [+1/-0/±0] 13https://git.io/Jqkgr [16:18:08] [02miraheze/puppet] 07paladox 0378aeee4 - Create nginx-systemd-override.conf.erb [16:18:09] [02puppet] 07paladox synchronize pull request 03#1682: nginx: Use a systemd override to check for syntax - 13https://git.io/Jqkzq [16:18:28] [02miraheze/ManageWiki] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Jqkgo [16:18:30] [02miraheze/ManageWiki] 07Universal-Omega 03de59dc4 - Move a['overridedefault'] above $configs [16:19:09] PROBLEM - gluster4 Puppet on gluster4 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 25 minutes ago with 0 failures [16:19:19] PROBLEM - services3 Puppet on services3 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 8 minutes ago with 0 failures [16:19:19] PROBLEM - ns2 Puppet on ns2 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 31 minutes ago with 0 failures [16:19:20] PROBLEM - ns1 Puppet on ns1 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 24 minutes ago with 0 failures [16:19:23] PROBLEM - dbbackup1 Puppet on dbbackup1 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 20 minutes ago with 0 failures [16:19:24] miraheze/ManageWiki - Universal-Omega the build passed. [16:19:30] PROBLEM - cloud3 Puppet on cloud3 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 31 minutes ago with 0 failures [16:19:31] PROBLEM - jobrunner4 Puppet on jobrunner4 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 16 minutes ago with 0 failures [16:19:32] PROBLEM - mon2 Puppet on mon2 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 28 minutes ago with 0 failures [16:19:43] PROBLEM - cp10 Puppet on cp10 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 27 minutes ago with 0 failures [16:19:51] PROBLEM - mail2 Puppet on mail2 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 22 minutes ago with 0 failures [16:19:52] PROBLEM - cloud5 Puppet on cloud5 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 25 minutes ago with 0 failures [16:19:53] PROBLEM - mem2 Puppet on mem2 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 30 minutes ago with 0 failures [16:19:58] PROBLEM - db13 Puppet on db13 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 28 minutes ago with 0 failures [16:19:58] PROBLEM - graylog2 Puppet on graylog2 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 19 minutes ago with 0 failures [16:20:02] PROBLEM - mw11 Puppet on mw11 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 16 minutes ago with 0 failures [16:20:10] PROBLEM - puppet3 Puppet on puppet3 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 27 minutes ago with 0 failures [16:20:13] PROBLEM - phab2 Puppet on phab2 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 18 minutes ago with 0 failures [16:20:14] PROBLEM - mw10 Puppet on mw10 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 16 minutes ago with 0 failures [16:20:16] PROBLEM - cp12 Puppet on cp12 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 27 minutes ago with 0 failures [16:20:18] PROBLEM - dbbackup2 Puppet on dbbackup2 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 7 minutes ago with 0 failures [16:20:19] PROBLEM - cp11 Puppet on cp11 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 14 minutes ago with 0 failures [16:20:22] PROBLEM - db12 Puppet on db12 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 31 minutes ago with 0 failures [16:20:26] PROBLEM - cp3 Puppet on cp3 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 6 minutes ago with 0 failures [16:20:27] PROBLEM - services4 Puppet on services4 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 11 minutes ago with 0 failures [16:20:34] PROBLEM - cloud4 Puppet on cloud4 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 7 minutes ago with 0 failures [16:20:35] PROBLEM - ping4 on dbbackup2 is WARNING: PING WARNING - Packet loss = 0%, RTA = 143.33 ms [16:20:38] PROBLEM - jobrunner3 Puppet on jobrunner3 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 17 minutes ago with 0 failures [16:20:47] PROBLEM - mem1 Puppet on mem1 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 16 minutes ago with 0 failures [16:20:48] PROBLEM - db11 Puppet on db11 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 31 minutes ago with 0 failures [16:20:48] PROBLEM - gluster3 Puppet on gluster3 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 31 minutes ago with 0 failures [16:20:49] PROBLEM - ldap2 Puppet on ldap2 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 14 minutes ago with 0 failures [16:21:13] PROBLEM - mw8 Puppet on mw8 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 17 minutes ago with 0 failures [16:21:27] PROBLEM - mw9 Puppet on mw9 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 17 minutes ago with 0 failures [16:21:51] PROBLEM - mail2 Puppet on mail2 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 1 minute ago with 1 failures. Failed resources (up to 3 shown): Exec[nginx-syntax-roundcubemail] [16:23:46] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/Jqkgb [16:23:48] [02miraheze/puppet] 07paladox 030fcdc4c - Update wildcard.pp [16:23:49] [02puppet] 07paladox synchronize pull request 03#1682: nginx: Use a systemd override to check for syntax - 13https://git.io/Jqkzq [16:23:51] RECOVERY - mail2 Puppet on mail2 is OK: OK: Puppet is currently enabled, last run 1 second ago with 0 failures [16:23:58] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/JqkgN [16:23:59] [02miraheze/puppet] 07paladox 0301959d4 - Update cert.pp [16:24:01] [02puppet] 07paladox synchronize pull request 03#1682: nginx: Use a systemd override to check for syntax - 13https://git.io/Jqkzq [16:24:37] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/JqkgA [16:24:38] [02miraheze/puppet] 07paladox 0353080f5 - Update certs.pp [16:24:40] [02puppet] 07paladox synchronize pull request 03#1682: nginx: Use a systemd override to check for syntax - 13https://git.io/Jqkzq [16:31:47] PROBLEM - mail2 HTTPS on mail2 is CRITICAL: connect to address 51.195.236.253 and port 443: Connection refusedHTTP CRITICAL - Unable to open TCP socket [16:31:53] PROBLEM - mail2 Puppet on mail2 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[nginx] [16:32:01] PROBLEM - mail2 webmail.miraheze.org HTTPS on mail2 is CRITICAL: connect to address 51.195.236.253 and port 443: Connection refusedHTTP CRITICAL - Unable to open TCP socket [16:40:47] PROBLEM - history.sdtef.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for history.sdtef.org could not be found [16:40:50] PROBLEM - wiki.triplescripts.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.triplescripts.org could not be found [16:40:51] PROBLEM - cyberlaw.ccdcoe.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for cyberlaw.ccdcoe.org could not be found [16:40:54] PROBLEM - ecole.science - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for ecole.science could not be found [16:47:30] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/JqkaQ [16:47:32] [02miraheze/puppet] 07paladox 03ab7c8a0 - Update init.pp [16:47:33] [02puppet] 07paladox synchronize pull request 03#1682: nginx: Use a systemd override to check for syntax - 13https://git.io/Jqkzq [16:47:37] RECOVERY - history.sdtef.org - reverse DNS on sslhost is OK: rDNS OK - history.sdtef.org reverse DNS resolves to cp10.miraheze.org [16:47:39] RECOVERY - ecole.science - reverse DNS on sslhost is OK: rDNS OK - ecole.science reverse DNS resolves to cp11.miraheze.org [16:47:43] RECOVERY - wiki.triplescripts.org - reverse DNS on sslhost is OK: rDNS OK - wiki.triplescripts.org reverse DNS resolves to cp10.miraheze.org [16:47:51] RECOVERY - cyberlaw.ccdcoe.org - reverse DNS on sslhost is OK: rDNS OK - cyberlaw.ccdcoe.org reverse DNS resolves to cp10.miraheze.org [16:48:02] RECOVERY - mail2 webmail.miraheze.org HTTPS on mail2 is OK: HTTP OK: HTTP/1.1 200 OK - 5664 bytes in 0.141 second response time [16:49:16] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/Jqkap [16:49:17] [02miraheze/puppet] 07paladox 03176093e - Update nginx-systemd-override.conf.erb [16:49:19] [02puppet] 07paladox synchronize pull request 03#1682: nginx: Use a systemd override to check for syntax - 13https://git.io/Jqkzq [16:49:47] RECOVERY - mail2 HTTPS on mail2 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 443 bytes in 0.005 second response time [16:49:51] RECOVERY - mail2 Puppet on mail2 is OK: OK: Puppet is currently enabled, last run 11 seconds ago with 0 failures [16:55:08] RECOVERY - ping4 on dbbackup2 is OK: PING OK - Packet loss = 0%, RTA = 118.84 ms [16:57:10] Her hair is Harlow gold Her lips, a sweet surprise Her hands are never cold She got Bette Davis eyes She'll turn the music on you You won't have to think twice She's pure as New York snow She got Bette Davis eyes And she'll tease you She'll unease you All the better just to please you She's precocious And she knows just what it takes to make a pro blush She got Greta Garbo stand-off sighs She's got Bette Davis eyes She'll [16:57:11] let you take her home It whets her appetite She'll lay you on a throne She got Bette Davis eyes She'll take a tumble on you Roll you like you were dice Until you come up blue She's got Bette Davis eyes She'll expose you When she snows you Off your feet with the crumbs she throws you She's ferocious And she knows just what it takes to make a pro blush All the boys think she's a spy She's got Bette Davis eyes She'll tease you She'll [16:57:11] unease you All the better just to please you She's precocious And she knows just what it takes to make a pro blush All the boys think she's a spy She's got Bette Davis eyes She'll tease you She'll unease you Just to please you She's got Bette Davis eyes She'll expose you When she snows you 'Cause she knows you She's got Bette Davis eyes [16:57:12] Her hair is Harlow gold Her lips, a sweet surprise Her hands are never cold She got Bette Davis eyes She'll turn the music on you You won't have to think twice She's pure as New York snow She got Bette Davis eyes And she'll tease you She'll unease you All the better just to please you She's precocious And she knows just what it takes to make a pro blush She got Greta Garbo stand-off sighs She's got Bette Davis eyes She'll [16:57:12] let you take her home It whets her appetite She'll lay you on a throne She got Bette Davis eyes She'll take a tumble on you Roll you like you were dice Until you come up blue She's got Bette Davis eyes She'll expose you When she snows you Off your feet with the crumbs she throws you She's ferocious And she knows just what it takes to make a pro blush All the boys think she's a spy She's got Bette Davis eyes She'll tease you She'll [16:57:13] unease you All the better just to please you She's precocious And she knows just what it takes to make a pro blush All the boys think she's a spy She's got Bette Davis eyes She'll tease you She'll unease you Just to please you She's got Bette Davis eyes She'll expose you When she snows you 'Cause she knows you She's got Bette Davis eyes [16:57:13] Her hair is Harlow gold Her lips, a sweet surprise Her hands are never cold She got Bette Davis eyes She'll turn the music on you You won't have to think twice She's pure as New York snow She got Bette Davis eyes And she'll tease you She'll unease you All the better just to please you She's precocious And she knows just what it takes to make a pro blush She got Greta Garbo stand-off sighs She's got Bette Davis eyes She'll [16:57:14] let you take her home It whets her appetite She'll lay you on a throne She got Bette Davis eyes She'll take a tumble on you Roll you like you were dice Until you come up blue She's got Bette Davis eyes She'll expose you When she snows you Off your feet with the crumbs she throws you She's ferocious And she knows just what it takes to make a pro blush All the boys think she's a spy She's got Bette Davis eyes She'll tease you She'll [16:57:14] unease you All the better just to please you She's precocious And she knows just what it takes to make a pro blush All the boys think she's a spy She's got Bette Davis eyes She'll tease you She'll unease you Just to please you She's got Bette Davis eyes She'll expose you When she snows you 'Cause she knows you She's got Bette Davis eyes [16:57:15] Her hair is Harlow gold Her lips, a sweet surprise Her hands are never cold She got Bette Davis eyes She'll turn the music on you You won't have to think twice She's pure as New York snow She got Bette Davis eyes And she'll tease you She'll unease you All the better just to please you She's precocious And she knows just what it takes to make a pro blush She got Greta Garbo stand-off sighs She's got Bette Davis eyes She'll [16:57:15] let you take her home It whets her appetite She'll lay you on a throne She got Bette Davis eyes She'll take a tumble on you Roll you like you were dice Until you come up blue She's got Bette Davis eyes She'll expose you When she snows you Off your feet with the crumbs she throws you She's ferocious And she knows just what it takes to make a pro blush All the boys think she's a spy She's got Bette Davis eyes She'll tease you She'll [16:57:16] unease you All the better just to please you She's precocious And she knows just what it takes to make a pro blush All the boys think she's a spy She's got Bette Davis eyes She'll tease you She'll unease you Just to please you She's got Bette Davis eyes She'll expose you When she snows you 'Cause she knows you She's got Bette Davis eyes [16:57:16] Her hair is Harlow gold Her lips, a sweet surprise Her hands are never cold She got Bette Davis eyes She'll turn the music on you You won't have to think twice She's pure as New York snow She got Bette Davis eyes And she'll tease you She'll unease you All the better just to please you She's precocious And she knows just what it takes to make a pro blush She got Greta Garbo stand-off sighs She's got Bette Davis eyes She'll [16:57:17] let you take her home It whets her appetite She'll lay you on a throne She got Bette Davis eyes She'll take a tumble on you Roll you like you were dice Until you come up blue She's got Bette Davis eyes She'll expose you When she snows you Off your feet with the crumbs she throws you She's ferocious And she knows just what it takes to make a pro blush All the boys think she's a spy She's got Bette Davis eyes She'll tease you She'll [16:57:17] unease you All the better just to please you She's precocious And she knows just what it takes to make a pro blush All the boys think she's a spy She's got Bette Davis eyes She'll tease you She'll unease you Just to please you She's got Bette Davis eyes She'll expose you When she snows you 'Cause she knows you She's got Bette Davis eyes [16:57:18] Her hair is Harlow gold Her lips, a sweet surprise Her hands are never cold She got Bette Davis eyes She'll turn the music on you You won't have to think twice She's pure as New York snow She got Bette Davis eyes And she'll tease you She'll unease you All the better just to please you She's precocious And she knows just what it takes to make a pro blush She got Greta Garbo stand-off sighs She's got Bette Davis eyes She'll [16:57:18] let you take her home It whets her appetite She'll lay you on a throne She got Bette Davis eyes She'll take a tumble on you Roll you like you were dice Until you come up blue She's got Bette Davis eyes She'll expose you When she snows you Off your feet with the crumbs she throws you She's ferocious And she knows just what it takes to make a pro blush All the boys think she's a spy She's got Bette Davis eyes She'll tease you She'll [16:57:19] unease you All the better just to please you She's precocious And she knows just what it takes to make a pro blush All the boys think she's a spy She's got Bette Davis eyes She'll tease you She'll unease you Just to please you She's got Bette Davis eyes She'll expose you When she snows you 'Cause she knows you She's got Bette Davis eyes [17:04:17] PROBLEM - ping4 on dbbackup2 is WARNING: PING WARNING - Packet loss = 0%, RTA = 136.52 ms [17:11:10] [02miraheze/puppet] 07JohnFLewis pushed 031 commit to 03master [+1/-0/±1] 13https://git.io/JqkVp [17:11:12] [02miraheze/puppet] 07JohnFLewis 03111cdc4 - add a script to run geoip updates and save to private git [17:13:34] [02puppet] 07paladox closed pull request 03#1682: nginx: Use a systemd override to check for syntax - 13https://git.io/Jqkzq [17:13:35] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+1/-0/±14] 13https://git.io/JqkwJ [17:13:37] [02miraheze/puppet] 07paladox 032fd8258 - nginx: Use a systemd override to check for syntax (#1682) [17:14:57] paladox, can we restore the Discord relay in #miraheze-sre now? [17:15:04] sure [17:15:09] Oh, nevermind, seems to be working, I think [17:15:13] PROBLEM - mw8 Puppet on mw8 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [17:15:15] PROBLEM - mw9 Puppet on mw9 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [17:15:19] but that's good too [17:15:21] thanks :) [17:15:46] [02puppet] 07paladox deleted branch 03paladox-patch-1 - 13https://git.io/vbiAS [17:15:47] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-1 [17:15:48] JohnLewis, paladox: both of you have just pushed to puppet [17:15:56] Can you check error? [17:16:02] PROBLEM - mw11 Puppet on mw11 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [17:16:07] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JqkwZ [17:16:08] [02miraheze/puppet] 07paladox 03c267acf - Fix duplicate notify [17:16:14] PROBLEM - mw10 Puppet on mw10 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [17:16:25] RhinosF1 check what? [17:16:37] paladox: Failed to apply catalog [17:16:51] The 4 critical [17:17:07] [02miraheze/puppet] 07JohnFLewis pushed 033 commits to 03master [+1/-0/±16] 13https://git.io/Jqkwc [17:17:08] [02miraheze/puppet] 07JohnFLewis 03b66bc7e - fetch geoip from private [17:17:10] [02miraheze/puppet] 07JohnFLewis 03a00a6dc - Merge branch 'master' of github.com:/miraheze/puppet [17:17:12] [02miraheze/puppet] 07JohnFLewis 035182fb3 - Merge branch 'master' of github.com:/miraheze/puppet [17:17:13] RECOVERY - mw8 Puppet on mw8 is OK: OK: Puppet is currently enabled, last run 3 seconds ago with 0 failures [17:17:14] RECOVERY - mw9 Puppet on mw9 is OK: OK: Puppet is currently enabled, last run 8 seconds ago with 0 failures [17:17:53] Oh what great timing [17:18:02] RECOVERY - mw11 Puppet on mw11 is OK: OK: Puppet is currently enabled, last run 52 seconds ago with 0 failures [17:18:13] RECOVERY - mw10 Puppet on mw10 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [17:18:16] RECOVERY - cp11 Puppet on cp11 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [17:18:25] RECOVERY - cp3 Puppet on cp3 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [17:18:30] RECOVERY - ping4 on dbbackup2 is OK: PING OK - Packet loss = 0%, RTA = 117.54 ms [17:19:12] looking at Phab, https://phabricator.miraheze.org/T4005 has been there for more than *2 years* and normal priority [17:19:14] [ ⚓ T4005 Execute external commands on MediaWiki servers inside sandboxes ] - phabricator.miraheze.org [17:19:18] RECOVERY - ns2 Puppet on ns2 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [17:19:19] RECOVERY - services3 Puppet on services3 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [17:19:24] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Jqkw0 [17:19:27] [02miraheze/puppet] 07paladox 0394cf3b3 - Remove use of nginx-syntax [17:20:59] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Jqkwa [17:21:00] [02miraheze/puppet] 07paladox 0317df42c - Fix [17:21:03] Universal_Omega, looks like you and I both got disconnected at the same time. Wonder if it was a Freenode or IRCCloud issue? [17:21:24] Freenode [17:21:31] net split yeah [17:21:41] RhinosF1, ack [17:21:41] Freenode split into multiple parts [17:21:54] I got disconnected and then reconnected as an op in all channels. [17:21:55] Reception123, ack I got hit by one of those. Well those are annoying :P [17:22:26] I got lucky this time [17:22:30] heh [17:23:17] and I got forwarded to #wikimedia-overflow instead of #wikimedia-stewards. Probably too many people tried reconnecting to the channel at the same time? [17:26:28] Reception123: 2 years is bad considering it's also security tagged - tbf though we've only been able to do it for a year in retrospect [17:27:25] yeah, though since it's assigned to SPF|Cloud I'll ask if he still plans on doing it himself [17:28:21] He won't [17:29:03] Just because it's assigned to him doesn't mean no one else can do it, in fact he recommends others do it because there's a lot of task assigned to him, quite a few he must do, quite a few (like this one) anyone can do [17:31:50] Maybe Universal_Omega you would be interested in having a look? [17:43:24] RhinosF1 https://phabricator.miraheze.org/T4425#136672 [17:43:25] [ ⚓ T4425 Fix all mysql tables that are using latin rather then utf8mb4 ] - phabricator.miraheze.org [17:49:50] [02mediawiki] 07Universal-Omega closed pull request 03#1215: Set Cargo to use REL1_35 branch - 13https://git.io/JqTxK [17:49:54] [02mediawiki] 07Universal-Omega deleted branch 03Universal-Omega-patch-1 - 13https://git.io/vbL5b [17:49:55] [02miraheze/mediawiki] 07Universal-Omega deleted branch 03Universal-Omega-patch-1 [17:53:27] paladox: they use the MediaWiki default [17:53:33] I thought I forwarded that on [17:53:36] ah ok [17:53:43] guess we should close it [17:53:46] He just said to read what's on the MW help oage [17:53:47] Page [17:56:20] !log reception@jobrunner3:/srv/mediawiki/w/extensions/ReplaceText/maintenance$ sudo -u www-data php replaceAll.php --wiki pokemundowiki --replacements /home/reception/movebatch.txt --rename [17:56:22] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:57:08] Reception123: https://phabricator.miraheze.org/T5976#136818 [17:57:09] [ ⚓ T5976 Add the EventStreams service ] - phabricator.miraheze.org [17:58:08] JohnLewis: ack, unfortunately yeah, from the start it did seem to require too much RAM that we can't allocate [17:58:09] As the JobQueue task is in your queue, I'll leave a decision regarding that with you [17:59:00] JohnLewis: iirc you said you were willing to have a look at that yourself [17:59:08] and if it would really eliminate the jobrunners the reward might be worth it [18:01:33] My concern is it's over 3 years since services/jobrunner was last committed too [18:03:23] I’ve had a look and I’m not really sure the memory requirement of the service would give us the outweigh in terms of benefits [18:04:19] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/JqkKf [18:04:20] [02miraheze/puppet] 07paladox 030351092 - mariadb: Add support for instances in mariadb icinga check [18:04:22] [02puppet] 07paladox created branch 03paladox-patch-1 - 13https://git.io/vbiAS [18:04:24] [02puppet] 07paladox opened pull request 03#1683: mariadb: Add support for instances in mariadb icinga check - 13https://git.io/JqkKJ [18:05:43] I’m not sure why 3 years of no commits is a big problem, it’s a simple bit of software that still works for our purposes [18:06:14] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/JqkKI [18:06:15] [02miraheze/puppet] 07paladox 031bc7d1b - Update config.pp [18:06:17] [02puppet] 07paladox synchronize pull request 03#1683: mariadb: Add support for instances in mariadb icinga check - 13https://git.io/JqkKJ [18:06:41] I might be able to investigate whether it works with 8GB and take it from there, but if it doesn’t then it’s not going to be a way forward [18:06:54] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/JqkKm [18:06:55] [02miraheze/puppet] 07paladox 0351803d6 - Update instance.pp [18:06:57] [02puppet] 07paladox synchronize pull request 03#1683: mariadb: Add support for instances in mariadb icinga check - 13https://git.io/JqkKJ [18:09:25] yeah, that sounds good, if it's not worth it then that's it [18:19:04] [02ssl] 07Universal-Omega opened pull request 03#396: Fix redirect - 13https://git.io/JqkK7 [18:21:30] Reception123: ^ [18:23:25] [02ssl] 07Reception123 closed pull request 03#396: Fix redirect - 13https://git.io/JqkK7 [18:23:26] [02miraheze/ssl] 07Reception123 pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Jqk6J [18:23:27] Universal_Omega: whoops, good catch :) [18:23:28] [02miraheze/ssl] 07Universal-Omega 03a60dba1 - Fix redirect (#396) [18:24:22] Reception123: thanks, but not me that did catch it... it was reported in the original task. [18:24:43] oh :( [18:30:46] [02puppet] 07Universal-Omega opened pull request 03#1684: Add docs.blender.org to CSP whitelist (T6916) - 13https://git.io/Jqk6E [18:39:54] Reception123: more and more I look and think about it, I'm going to have to decline it in consideration. I don't really see and understand why we need to invest heavily in a system which requires 8GB minimum to maybe be an improvement when we have a system that works at minimal cost in resources. If we really want a new system, we can build one based around a less expensive service [18:43:47] ok [18:50:18] Reception123: https://phabricator.miraheze.org/T6006#136839 I proposed 2 alternatives as well in that decline [18:50:19] [ ⚓ T6006 Migrate to a Kafka Job Queue ] - phabricator.miraheze.org [18:50:53] [02mw-config] 07Universal-Omega closed pull request 03#3754: Add wgVectorDefaultSkinVersion to ManageWikiSettings - 13https://git.io/JqUXw [18:50:55] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±2] 13https://git.io/JqkiE [18:50:56] [02miraheze/mw-config] 07Kelvs599 031622d8c - Add wgVectorDefaultSkinVersion to ManageWikiSettings (#3754) [18:51:24] ack, if a need arises we could look into the alternatives then [18:51:35] JohnLewis: would infra team be able to assist in implementing them alternatives? [18:51:59] miraheze/mw-config - Universal-Omega the build passed. [18:52:14] Memcache might be interesting [18:52:36] Infra can assist if it is explicitly asked to do something [18:53:53] JohnLewis: this is me asking [18:55:23] Yeah, but I'm saying we need to be asked explicitly what MediaWiki needs assistance with [18:56:12] JohnLewis: setting up memcache [18:56:32] JohnLewis: not related but I was wondering what the plans with https://phabricator.miraheze.org/T5397 were: do we plan on implementing such a system in the new future? [18:56:33] [ ⚓ T5397 Create a logbot for server actions ] - phabricator.miraheze.org [18:56:49] Hrm, both of John's ideas sound like reasonable options we could look into. Good, thoughtful analysis on closing that task [18:57:03] and which "certain actions" would it be, would it be specific commands, is that possible to set up? [18:57:17] It's fairly easy [18:57:35] Using a bash script [18:57:36] Reception123, I'm confused how T5397 is different than what MirahezeLogbot provides? [18:57:44] dmehus: it auto logging [18:57:47] what's it proposing to log? [18:57:51] oh [18:58:00] so sysadmins don't have to manually `!log`? [18:58:04] Maint scripts mainly [18:58:04] Yes [18:58:08] oh [18:58:56] how would that work, would a cron be set up on the jobrunner and other servers to parse the sysadmins' bash histories at regular intervals and generate log entries? [18:59:28] even though I'm not familiar with it I'd say no or else it would log everything even 'ls' and we definitely don't want that [18:59:29] hi all [18:59:31] UDP [18:59:34] hi SPF|Cloud :) [18:59:39] Hi SPF|Cloud! [19:00:26] Reception123, yeah, but I did say parse, so we'd have to code it to ignore all other commands issued [19:00:26] RhinosF1: I'm sure paladox can assist with setting up a memcache install on jobrunner* or alternative, using mem* as already existing may also be an appropriate solution depending stability of the service already [19:00:39] Reception123: a task on my radar, that I haven't looked at yet [19:00:47] Reception123: setup an mwscript bash script and have it be able to parse the input and then run the script and then send the script ran to a UDP instance like the RC bot does [19:00:57] dmehus: no no and no [19:00:57] hmm? [19:01:07] RhinosF1, ack [19:01:09] paladox: memcache for jobrunner, can you set up? [19:02:29] I see lots of activity, thanks all [19:03:00] tbh, getting rid of jobrunner* as a whole may also be a beneficial thing potentially [19:04:02] [02puppet] 07Reception123 closed pull request 03#1684: Add docs.blender.org to CSP whitelist (T6916) - 13https://git.io/Jqk6E [19:04:04] [02miraheze/puppet] 07Reception123 pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JqkPG [19:04:05] [02miraheze/puppet] 07Universal-Omega 03aff0b82 - Add docs.blender.org to CSP whitelist (T6916) (#1684) [19:05:41] question; why has the ssl_trusted_certificate been removed from nginx configuration? [19:11:37] ^ paladox [19:21:12] JohnLewis: and replacing with? [19:21:36] SPF|Cloud JohnLewis did that today [19:21:42] See the comment added to the task [19:22:09] RhinosF1 yes we can [19:22:56] JohnLewis: I see nothing after https://phabricator.miraheze.org/T6006#136839 [19:22:57] [ ⚓ T6006 Migrate to a Kafka Job Queue ] - phabricator.miraheze.org [19:23:15] "Alternatively, we could also drop dedicated runners, take our main cluster up to 8 and running jobs on all of them. If we allocate 1 runner per server, that would give us 8 instances, which is double what we have right now." [19:23:26] For the OCSP stapling to work, the certificate of the server certificate issuer should be known. If the ssl_certificate file does not contain intermediate certificates, the certificate of the server certificate issuer should be present in the ssl_trusted_certificate file. [19:23:48] nevermind, looks like the settings wasn't needed after all, since we set intermediate certificates in the ssl_certificate files [19:24:18] JohnLewis: so turn jobrunner* into mw* and make an mw server the jobchron server too [19:24:19] ^ which is why I removed it :P We were sending the cert again, so increasing the payload of requests [19:24:28] I mean I see a benefit [19:24:39] Might be a good idea to still do memcache though same time [19:24:46] Given were moving off redis for sessions [19:24:52] doing memcache would require a whole rewrite [19:24:58] RhinosF1: yeah, each server has 1 runner, we gain 2 new distributed servers, technically there's 8 job runners compared to our 4 now [19:25:01] jobrunner is based in redis. [19:25:13] I see John, thanks, good work [19:25:21] paladox: is it reasonable? [19:26:08] JohnLewis: I don't see why not. [19:26:24] if you are willing to rewrite it. But then wouldn't the service just be the same but using memcache rather than redis? [19:26:30] so would there be any benefits? [19:26:49] paladox: memcache is apparently better performance [19:27:10] Ok, i don't think performance matter much to the backlog. [19:27:15] Well, for sessions it has shown that, for jobs that would be more of a guess [19:27:23] i think it's more that jobs re-run due to errors [19:27:38] and then there's the fact some jobs get inserted for all wikis [19:27:38] Which is why ideally before we have any changes to jobrunner, we have solid stats so we can compare and understamd [19:27:40] I have just reviewed documents regarding the new group responsible for enforcing ToU [19:29:09] JohnLewis: what impact do you expect having 8 runners would have too? Effect on MW load? [19:31:13] paladox: well I know that will have an impact but we don't have a solution to that unless we want to work on upgrading the redis code [19:31:20] SPF|Cloud: great! [19:31:41] well actually i think increasing capacity would reduce our issues too. [19:31:49] i mean we have over 3,000 wikis [19:32:13] with like 2 jobrunners, running what 2 or 3 processes which in total equals 4 or 6. [19:33:20] paladox: as long as it won't harm MediaWiki performance then we can increase capacity [19:33:22] RhinosF1: yes, I hope this will lessen the workload on you [19:33:41] There's no reason not to explore both [19:33:45] I think it would harm mediawiki performance unless we use nice. [19:33:52] we did this with mw3 back in the day [19:33:59] before we had dedicated infra [19:34:16] [02mw-config] 07Universal-Omega commented on pull request 03#3675: Fix config name - 13https://git.io/Jqk1f [19:34:17] SPF|Cloud: hopefully, as long as we have enough people to man the team [19:34:21] we would need to increase the cores available to mw* [19:34:29] if we increase by one [19:34:45] we would reduce our total core usage of 2 accross both clouds [19:34:56] well 4 [19:35:12] oh! [19:35:16] We're not scrapping jobrunner* [19:35:21] They'd become an mw* [19:35:26] We wouldn't argubly need to increase cores on mw* [19:35:42] yes, I have some concerns regarding staffing the team, but even if one SRE member is going to join the group, it's still an improvement [19:35:45] why would they be mw* if they are only a jobrunner*? [19:35:46] oh [19:36:04] a discussion regarding a to-be take action won't keep the whole SRE team busy anymore [19:36:10] well do we have enough cpu power for that? we would need to give at least 4 cores to each jobrunner [19:36:49] each jobrunner right now has 3 [19:36:59] Why would we need to add 4 to eveyr mw server if we do that? [19:37:34] i'm presuming that we're just adding extra mw*s but also running a jobrunner on each mw*. So we need to match the specs of the other mw*s [19:37:43] unless i'm reading this proposal wrong? [19:39:48] we'd need 2 additional CPU to match mw* and two more GB to match mw* [19:40:03] yeh [19:41:46] across the system, right now we have 16-6 CPU and 16-6 RAM for traffic-jobs. Under the new system we'd have 16-8 CPU and 24GB (gen). CPU isn't dedicated, so we'd have 24-24 as a split [19:42:34] does that seem ok to you JohnLewis ? [19:42:42] Yes, I did propose it afterall? [19:42:48] ok [19:43:43] seems ok to me then! Increasing our mw* capacity whilst increasing jobrunners [19:44:10] as long as we use nice [19:44:10] But before I actually approve anything of the nature, I want data to justify it [19:44:16] we don't need to use nice... [19:45:00] Using nice actually defeats the whole point as we'd be better off keeping jobrunner* [19:47:14] fun fact; we have 36 tasks open currently - in a split of 22:11:3 for MW:Infra:SPF :P [19:48:01] right [19:48:16] if it's better if I hand over some tasks, let me know [19:48:33] SPF|Cloud: those 3 are all DSRE ones [19:48:49] aha [19:51:32] How hard is it to change a wiki's database name? [19:52:13] not hard [19:52:17] Sario, not too difficult. Just needs a Phabricator task and an SRE-level sysadmin runs a script [19:53:07] JohnLewis ah, ok [19:53:36] dmehus: Ok, thanks [19:56:06] Sario, cool. is this for your QuIRC Wiki or something else? [19:56:28] My private wiki [19:56:38] Sario, ah, ok, cool :) [19:57:24] I just need redundant 'wiki' dropped off the name [19:59:15] Sario: check MirahezeBots puppet for the status config after [19:59:56] As you have your wiki enabled [20:00:11] @Sario: replied on task. Do you also want the subdomain changed, if not I don't think that can be done. [20:00:22] Sario, ah, that should be easy enough [20:00:40] I think they're fine with the subdomain changing as well, Universal_Omega [20:00:52] as they're the only one that use that private wiki, I think? [20:00:58] Universal_Omega: I marked declined, it's not worth it [20:01:20] I'd have to change a bunch of links and other stuff [20:01:29] ah [20:01:49] Thanks for the quick response [20:02:13] You could also create a second `sariowiki` and when everything is migrated to your new wiki, all your links updated, `sariowikiwiki` could be deleted, too [20:02:18] so that's an option as well [20:03:10] Technically possible, but also more work than it's worth [20:15:39] Or ah [20:44:07] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/Jqkyr [20:44:09] [02miraheze/puppet] 07paladox 03c2c26ff - Update commands.conf [20:44:10] [02puppet] 07paladox synchronize pull request 03#1683: mariadb: Add support for instances in mariadb icinga check - 13https://git.io/JqkKJ [20:44:43] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/Jqkyi [20:44:44] [02miraheze/puppet] 07paladox 03a9156eb - Update nrpe.cfg.erb [20:44:46] [02puppet] 07paladox synchronize pull request 03#1683: mariadb: Add support for instances in mariadb icinga check - 13https://git.io/JqkKJ [20:45:00] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/Jqky1 [20:45:02] [02miraheze/puppet] 07paladox 03af821dd - Update monitoring.pp [20:45:03] [02puppet] 07paladox synchronize pull request 03#1683: mariadb: Add support for instances in mariadb icinga check - 13https://git.io/JqkKJ [20:51:43] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/JqkSf [20:51:45] [02miraheze/puppet] 07paladox 0366859d6 - Update dbbackup.pp [20:51:46] [02puppet] 07paladox synchronize pull request 03#1683: mariadb: Add support for instances in mariadb icinga check - 13https://git.io/JqkKJ [20:52:20] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/JqkST [20:52:21] [02miraheze/puppet] 07paladox 030385aa6 - Update instance.pp [20:52:23] [02puppet] 07paladox synchronize pull request 03#1683: mariadb: Add support for instances in mariadb icinga check - 13https://git.io/JqkKJ [20:52:33] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/JqkSI [20:52:34] [02miraheze/puppet] 07paladox 03db23e5b - Update monitoring.pp [20:52:35] [02puppet] 07paladox synchronize pull request 03#1683: mariadb: Add support for instances in mariadb icinga check - 13https://git.io/JqkKJ [20:54:04] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/JqkSZ [20:54:06] [02miraheze/puppet] 07paladox 037ab7ed1 - Update nrpe.cfg.erb [20:54:07] [02puppet] 07paladox synchronize pull request 03#1683: mariadb: Add support for instances in mariadb icinga check - 13https://git.io/JqkKJ [20:55:38] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/JqkSB [20:55:39] [02miraheze/puppet] 07paladox 0367c9256 - Update dbbackup.pp [20:55:41] [02puppet] 07paladox synchronize pull request 03#1683: mariadb: Add support for instances in mariadb icinga check - 13https://git.io/JqkKJ [20:56:09] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/JqkSR [20:56:10] [02miraheze/puppet] 07paladox 030c4b2e1 - Update commands.conf [20:56:12] [02puppet] 07paladox synchronize pull request 03#1683: mariadb: Add support for instances in mariadb icinga check - 13https://git.io/JqkKJ [21:00:47] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/JqkSP [21:00:49] [02miraheze/puppet] 07paladox 03cb6c80f - Update config.pp [21:00:50] [02puppet] 07paladox synchronize pull request 03#1683: mariadb: Add support for instances in mariadb icinga check - 13https://git.io/JqkKJ [21:01:00] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/JqkSX [21:01:02] [02miraheze/puppet] 07paladox 03770ef29 - Update instance.pp [21:01:03] [02puppet] 07paladox synchronize pull request 03#1683: mariadb: Add support for instances in mariadb icinga check - 13https://git.io/JqkKJ [21:06:01] RECOVERY - ns1 Puppet on ns1 is OK: OK: Puppet is currently enabled, last run 11 minutes ago with 0 failures [21:06:02] RECOVERY - db11 Puppet on db11 is OK: OK: Puppet is currently enabled, last run 17 minutes ago with 0 failures [21:06:14] RECOVERY - phab2 Puppet on phab2 is OK: OK: Puppet is currently enabled, last run 4 minutes ago with 0 failures [21:06:15] RECOVERY - cp11 Puppet on cp11 is OK: OK: Puppet is currently enabled, last run 50 seconds ago with 0 failures [21:06:25] RECOVERY - cloud3 Puppet on cloud3 is OK: OK: Puppet is currently enabled, last run 18 minutes ago with 0 failures [21:06:27] RECOVERY - dbbackup2 Puppet on dbbackup2 is OK: OK: Puppet is currently enabled, last run 3 minutes ago with 0 failures [21:06:29] RECOVERY - mem2 Puppet on mem2 is OK: OK: Puppet is currently enabled, last run 17 minutes ago with 0 failures [21:06:30] RECOVERY - services4 Puppet on services4 is OK: OK: Puppet is currently enabled, last run 27 minutes ago with 0 failures [21:06:30] RECOVERY - puppet3 Puppet on puppet3 is OK: OK: Puppet is currently enabled, last run 14 minutes ago with 0 failures [21:06:32] RECOVERY - db12 Puppet on db12 is OK: OK: Puppet is currently enabled, last run 17 minutes ago with 0 failures [21:06:41] RECOVERY - services3 Puppet on services3 is OK: OK: Puppet is currently enabled, last run 26 minutes ago with 0 failures [21:06:46] RECOVERY - mon2 Puppet on mon2 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [21:07:01] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/JqkSx [21:07:02] [02miraheze/puppet] 07paladox 031157406 - Update nrpe.cfg.erb [21:07:03] [02puppet] 07paladox synchronize pull request 03#1683: mariadb: Add support for instances in mariadb icinga check - 13https://git.io/JqkKJ [21:07:10] RECOVERY - mw10 Puppet on mw10 is OK: OK: Puppet is currently enabled, last run 4 minutes ago with 0 failures [21:07:13] RECOVERY - ns2 Puppet on ns2 is OK: OK: Puppet is currently enabled, last run 19 minutes ago with 0 failures [21:07:16] RECOVERY - cp3 Puppet on cp3 is OK: OK: Puppet is currently enabled, last run 24 minutes ago with 0 failures [21:07:17] RECOVERY - dbbackup1 Puppet on dbbackup1 is OK: OK: Puppet is currently enabled, last run 5 minutes ago with 0 failures [21:07:17] RECOVERY - mw8 Puppet on mw8 is OK: OK: Puppet is currently enabled, last run 4 minutes ago with 0 failures [21:07:21] RECOVERY - jobrunner3 Puppet on jobrunner3 is OK: OK: Puppet is currently enabled, last run 4 minutes ago with 0 failures [21:07:22] RECOVERY - cp10 Puppet on cp10 is OK: OK: Puppet is currently enabled, last run 15 minutes ago with 0 failures [21:07:23] RECOVERY - gluster4 Puppet on gluster4 is OK: OK: Puppet is currently enabled, last run 13 minutes ago with 0 failures [21:07:25] RECOVERY - cp12 Puppet on cp12 is OK: OK: Puppet is currently enabled, last run 15 minutes ago with 0 failures [21:07:26] RECOVERY - gluster3 Puppet on gluster3 is OK: OK: Puppet is currently enabled, last run 18 minutes ago with 0 failures [21:07:30] RECOVERY - cloud5 Puppet on cloud5 is OK: OK: Puppet is currently enabled, last run 12 minutes ago with 0 failures [21:07:31] RECOVERY - mem1 Puppet on mem1 is OK: OK: Puppet is currently enabled, last run 4 minutes ago with 0 failures [21:07:32] RECOVERY - graylog2 Puppet on graylog2 is OK: OK: Puppet is currently enabled, last run 7 minutes ago with 0 failures [21:07:34] RECOVERY - mw11 Puppet on mw11 is OK: OK: Puppet is currently enabled, last run 4 minutes ago with 0 failures [21:07:35] RECOVERY - ldap2 Puppet on ldap2 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [21:07:36] RECOVERY - ping4 on dbbackup2 is OK: PING OK - Packet loss = 0%, RTA = 105.40 ms [21:07:39] RECOVERY - db13 Puppet on db13 is OK: OK: Puppet is currently enabled, last run 16 minutes ago with 0 failures [21:07:41] RECOVERY - mw9 Puppet on mw9 is OK: OK: Puppet is currently enabled, last run 4 minutes ago with 0 failures [21:07:44] RECOVERY - cloud4 Puppet on cloud4 is OK: OK: Puppet is currently enabled, last run 25 minutes ago with 0 failures [21:07:53] RECOVERY - jobrunner4 Puppet on jobrunner4 is OK: OK: Puppet is currently enabled, last run 4 minutes ago with 0 failures [21:07:54] PROBLEM - dbbackup2 Check MariaDB Replication c3 on dbbackup2 is CRITICAL: MariaDB replication - both - CRITICAL - Slave IO State not correct, slave stopped or replication broken! [21:07:54] PROBLEM - dbbackup1 Check MariaDB Replication c4 on dbbackup1 is UNKNOWN: NRPE: Command 'check_mysql_replication_c4' not defined [21:08:14] PROBLEM - dbbackup1 Check MariaDB Replication c2 on dbbackup1 is UNKNOWN: NRPE: Command 'check_mysql_replication_c2' not defined [21:08:20] PROBLEM - dbbackup1 MariaDB c4 on dbbackup1 is CRITICAL: Can't connect to MySQL server on 'dbbackup1.miraheze.org' (111) [21:08:57] RECOVERY - dbbackup1 Check MariaDB Replication c2 on dbbackup1 is OK: MariaDB replication - both - OK - Slave_IO_Running state : Yes, Slave_SQL_Running state : Yes, Seconds_Behind_Master : 0s [21:12:15] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/Jqk9U [21:12:17] [02miraheze/puppet] 07paladox 03a5413d7 - Update instance.cnf.erb [21:12:18] [02puppet] 07paladox synchronize pull request 03#1683: mariadb: Add support for instances in mariadb icinga check - 13https://git.io/JqkKJ [21:13:46] PROBLEM - dbbackup1 MariaDB c2 on dbbackup1 is CRITICAL: Can't connect to MySQL server on 'dbbackup1.miraheze.org' (111) [21:13:53] oh [21:16:37] PROBLEM - dbbackup1 Check MariaDB Replication c2 on dbbackup1 is UNKNOWN: NRPE: Unable to read output [21:19:52] RECOVERY - dbbackup1 MariaDB c2 on dbbackup1 is OK: Uptime: 66 Threads: 8 Questions: 1 Slow queries: 0 Opens: 16 Flush tables: 1 Open tables: 10 Queries per second avg: 0.015 [21:20:26] PROBLEM - dbbackup1 Check MariaDB Replication c2 on dbbackup1 is CRITICAL: MariaDB replication - both - CRITICAL - Slave IO State not correct, slave stopped or replication broken! [21:24:42] PROBLEM - dbbackup1 Check MariaDB Replication c2 on dbbackup1 is WARNING: MariaDB replication - both - WARNING - Slave_IO_Running state : Yes, Slave_SQL_Running state : Yes, Seconds_Behind_Master : 10s [21:24:47] RECOVERY - dbbackup1 Check MariaDB Replication c2 on dbbackup1 is OK: MariaDB replication - both - OK - Slave_IO_Running state : Yes, Slave_SQL_Running state : Yes, Seconds_Behind_Master : 0s [21:27:00] [02puppet] 07paladox closed pull request 03#1683: mariadb: Add support for instances in mariadb icinga check - 13https://git.io/JqkKJ [21:27:01] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±5] 13https://git.io/Jqk9E [21:27:03] [02miraheze/puppet] 07paladox 03a69dce4 - mariadb: Add support for instances in mariadb icinga check (#1683) [21:27:04] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-1 [21:27:06] [02puppet] 07paladox deleted branch 03paladox-patch-1 - 13https://git.io/vbiAS [21:30:23] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/Jqk9a [21:30:25] [02miraheze/puppet] 07paladox 03f19f55a - postfix::dmarc: Switch to systemd::service [21:30:26] [02puppet] 07paladox created branch 03paladox-patch-1 - 13https://git.io/vbiAS [21:30:28] [02puppet] 07paladox opened pull request 03#1685: postfix::dmarc: Switch to systemd::service - 13https://git.io/Jqk9V [21:32:39] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-1 [+1/-1/±0] 13https://git.io/Jqk96 [21:32:41] [02miraheze/puppet] 07paladox 036f112d8 - Rename modules/postfix/files/opendmarc.systemd to modules/postfix/templates/initscripts/opendmarc.systemd.erb [21:32:42] [02puppet] 07paladox synchronize pull request 03#1685: postfix::dmarc: Switch to systemd::service - 13https://git.io/Jqk9V [21:35:42] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/Jqk9y [21:35:43] [02miraheze/puppet] 07paladox 035cefb2c - Update opendmarc.systemd.erb [21:35:45] [02puppet] 07paladox synchronize pull request 03#1685: postfix::dmarc: Switch to systemd::service - 13https://git.io/Jqk9V [21:35:52] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-1/±0] 13https://git.io/Jqk99 [21:35:53] [02miraheze/puppet] 07paladox 03a48f9a5 - Delete opendmarc [21:35:55] [02puppet] 07paladox synchronize pull request 03#1685: postfix::dmarc: Switch to systemd::service - 13https://git.io/Jqk9V [21:36:06] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/Jqk9H [21:36:07] [02miraheze/puppet] 07paladox 0390deb12 - Update dmarc.pp [21:36:09] [02puppet] 07paladox synchronize pull request 03#1685: postfix::dmarc: Switch to systemd::service - 13https://git.io/Jqk9V [21:36:33] [02puppet] 07paladox closed pull request 03#1685: postfix::dmarc: Switch to systemd::service - 13https://git.io/Jqk9V [21:36:35] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+1/-2/±1] 13https://git.io/Jqk97 [21:36:36] [02miraheze/puppet] 07paladox 03383d43c - postfix::dmarc: Switch to systemd::service (#1685) [21:36:38] [02puppet] 07paladox deleted branch 03paladox-patch-1 - 13https://git.io/vbiAS [21:36:39] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-1 [21:37:06] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-3 [+0/-0/±1] 13https://git.io/Jqk95 [21:37:07] [02miraheze/mw-config] 07Universal-Omega 03c7033f1 - add CannedResponses [21:37:09] [02mw-config] 07Universal-Omega synchronize pull request 03#3745: Set new CreateWiki configs - 13https://git.io/JtA6q [21:38:14] miraheze/mw-config - Universal-Omega the build passed. [21:39:52] [02mw-config] 07Universal-Omega commented on pull request 03#3745: Set new CreateWiki configs - 13https://git.io/Jqk9A [21:40:01] [02mw-config] 07Universal-Omega closed pull request 03#3745: Set new CreateWiki configs - 13https://git.io/JtA6q [21:40:02] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Jqk9x [21:40:04] [02miraheze/mw-config] 07Universal-Omega 03dccb45e - Set new CreateWiki configs (#3745) [21:40:06] [02mw-config] 07Universal-Omega deleted branch 03Universal-Omega-patch-3 - 13https://git.io/vbvb3 [21:40:07] [02miraheze/mw-config] 07Universal-Omega deleted branch 03Universal-Omega-patch-3 [21:40:52] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Jqk9p [21:40:53] [02miraheze/puppet] 07paladox 0311698cb - postfix::dmarc: Restart service if there are changes [21:41:00] miraheze/mw-config - Universal-Omega the build passed. [21:42:18] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/JqkHe [21:42:20] [02miraheze/puppet] 07paladox 032a49d63 - prometheus::varnish_prometheus_exporter: Switch to systemd::service [21:42:21] [02puppet] 07paladox created branch 03paladox-patch-1 - 13https://git.io/vbiAS [21:42:23] [02puppet] 07paladox opened pull request 03#1686: prometheus::varnish_prometheus_exporter: Switch to systemd::service - 13https://git.io/JqkHv [21:43:34] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-1 [+1/-1/±0] 13https://git.io/JqkHJ [21:43:35] [02miraheze/puppet] 07paladox 03cb565c2 - Rename modules/prometheus/templates/prometheus-varnish-exporter.systemd to modules/prometheus/templates/initscripts/prometheus-varnish-exporter.systemd.erb [21:43:37] [02puppet] 07paladox synchronize pull request 03#1686: prometheus::varnish_prometheus_exporter: Switch to systemd::service - 13https://git.io/JqkHv [21:43:56] [02puppet] 07paladox closed pull request 03#1686: prometheus::varnish_prometheus_exporter: Switch to systemd::service - 13https://git.io/JqkHv [21:43:57] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+1/-1/±1] 13https://git.io/JqkHU [21:43:59] [02miraheze/puppet] 07paladox 031b8b7f5 - prometheus::varnish_prometheus_exporter: Switch to systemd::service (#1686) [21:44:24] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JqkHT [21:44:25] [02miraheze/mw-config] 07Universal-Omega 039808bd3 - Fix wgCreateWikiPurposes [21:44:51] [02puppet] 07paladox deleted branch 03paladox-patch-1 - 13https://git.io/vbiAS [21:44:52] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-1 [21:45:25] miraheze/mw-config - Universal-Omega the build passed. [21:46:04] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JqkHk [21:46:06] [02miraheze/puppet] 07paladox 038ac630f - prometheus::varnish_prometheus_exporter: Add user to systemd script [21:49:25] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JqkHm [21:49:27] [02miraheze/puppet] 07paladox 0346e5875 - Update varnish_prometheus_exporter.pp [21:50:15] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/JqkHO [21:50:16] [02miraheze/puppet] 07paladox 03e1940a9 - letsencrypt::web: Switch to systemd::service [21:50:18] [02puppet] 07paladox created branch 03paladox-patch-1 - 13https://git.io/vbiAS [21:50:20] [02puppet] 07paladox opened pull request 03#1687: letsencrypt::web: Switch to systemd::service - 13https://git.io/JqkH3 [21:51:07] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-1 [+1/-1/±0] 13https://git.io/JqkHn [21:51:09] [02miraheze/puppet] 07paladox 03b1fa709 - Rename modules/letsencrypt/files/mirahezerenewssl.systemd to modules/letsencrypt/templates/initscripts/mirahezerenewssl.systemd.erb [21:51:10] [02puppet] 07paladox synchronize pull request 03#1687: letsencrypt::web: Switch to systemd::service - 13https://git.io/JqkH3 [21:52:12] [02puppet] 07paladox closed pull request 03#1687: letsencrypt::web: Switch to systemd::service - 13https://git.io/JqkH3 [21:52:13] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+1/-1/±1] 13https://git.io/JqkHl [21:52:15] [02miraheze/puppet] 07paladox 03c87ddde - letsencrypt::web: Switch to systemd::service (#1687) [21:52:16] [02puppet] 07paladox deleted branch 03paladox-patch-1 - 13https://git.io/vbiAS [21:52:18] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-1 [21:52:21] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-3 [+0/-0/±1] 13https://git.io/JqkH8 [21:52:22] [02miraheze/mw-config] 07Universal-Omega 038822df8 - add minimized display options for wgCreateWikiCannedResponses [21:52:24] [02mw-config] 07Universal-Omega created branch 03Universal-Omega-patch-3 - 13https://git.io/vbvb3 [21:52:27] [02mw-config] 07Universal-Omega opened pull request 03#3756: add minimized display options for wgCreateWikiCannedResponses - 13https://git.io/JqkH4 [21:53:30] miraheze/mw-config - Universal-Omega the build passed. [21:56:12] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/JqkHz [21:56:13] [02miraheze/puppet] 07paladox 03af1c35e - prometheus::php_fpm: Switch to systemd::service [21:56:15] [02puppet] 07paladox created branch 03paladox-patch-1 - 13https://git.io/vbiAS [21:56:17] [02mw-config] 07Universal-Omega closed pull request 03#3756: add minimized display options for wgCreateWikiCannedResponses - 13https://git.io/JqkH4 [21:56:18] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JqkHg [21:56:19] [02miraheze/mw-config] 07Universal-Omega 035421faa - add minimized display options for wgCreateWikiCannedResponses (#3756) [21:56:21] [02puppet] 07paladox opened pull request 03#1688: prometheus::php_fpm: Switch to systemd::service - 13https://git.io/JqkH2 [21:56:22] [02mw-config] 07Universal-Omega deleted branch 03Universal-Omega-patch-3 - 13https://git.io/vbvb3 [21:56:24] [02miraheze/mw-config] 07Universal-Omega deleted branch 03Universal-Omega-patch-3 [21:57:06] miraheze/mw-config - Universal-Omega the build passed. [21:57:21] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-1 [+1/-1/±0] 13https://git.io/JqkHK [21:57:22] [02miraheze/puppet] 07paladox 03c0751ae - Rename modules/prometheus/files/php-fpm/prometheus-php-fpm.systemd to modules/prometheus/templates/initscripts/prometheus-php-fpm.systemd.erb [21:57:24] [02puppet] 07paladox synchronize pull request 03#1688: prometheus::php_fpm: Switch to systemd::service - 13https://git.io/JqkH2 [21:57:51] [02puppet] 07paladox closed pull request 03#1688: prometheus::php_fpm: Switch to systemd::service - 13https://git.io/JqkH2 [21:57:52] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+1/-1/±1] 13https://git.io/JqkH6 [21:57:54] [02miraheze/puppet] 07paladox 0382609f8 - prometheus::php_fpm: Switch to systemd::service (#1688) [21:57:55] [02puppet] 07paladox deleted branch 03paladox-patch-1 - 13https://git.io/vbiAS [21:57:57] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-1 [22:13:38] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JqkQs [22:13:39] [02miraheze/puppet] 07paladox 03a29a626 - prometheus::php-fpm: upgrade exporter to 2.0.1 [22:28:22] [02miraheze/CreateWiki] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://git.io/JqkQX [22:28:24] [02miraheze/CreateWiki] 07Universal-Omega 03b9c305d - add cw_bio to save() [22:28:25] [02CreateWiki] 07Universal-Omega created branch 03Universal-Omega-patch-1 - 13https://git.io/vpJTL [22:28:27] [02CreateWiki] 07Universal-Omega opened pull request 03#196: add cw_bio to save() - 13https://git.io/JqkQM [22:29:28] miraheze/CreateWiki - Universal-Omega the build passed. [22:30:00] [02miraheze/puppet] 07paladox pushed 031 commit to 03patch1 [+0/-0/±3] 13https://git.io/JqkQS [22:30:01] [02miraheze/puppet] 07paladox 03c481bfa - prometheus::php-fpm: switch to Lusitaniae/phpfpm_exporter [22:30:03] [02puppet] 07paladox created branch 03patch1 - 13https://git.io/vbiAS [22:33:52] [02CreateWiki] 07Universal-Omega closed pull request 03#196: add cw_bio to save() - 13https://git.io/JqkQM [22:33:53] [02miraheze/CreateWiki] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JqkQN [22:33:55] [02miraheze/CreateWiki] 07Universal-Omega 039e83975 - add cw_bio to save() (#196) [22:33:56] [02miraheze/CreateWiki] 07Universal-Omega deleted branch 03Universal-Omega-patch-1 [22:33:58] [02CreateWiki] 07Universal-Omega deleted branch 03Universal-Omega-patch-1 - 13https://git.io/vpJTL [22:34:00] PROBLEM - phab2 Puppet on phab2 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[prometheus-php-fpm] [22:34:48] miraheze/CreateWiki - Universal-Omega the build passed. [22:34:56] [02miraheze/puppet] 07paladox pushed 031 commit to 03patch1 [+0/-0/±3] 13https://git.io/JqkQA [22:34:58] [02miraheze/puppet] 07paladox 037c99619 - prometheus::php-fpm: switch to Lusitaniae/phpfpm_exporter [22:36:19] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-3 [+0/-0/±1] 13https://git.io/JqkQx [22:36:20] [02miraheze/mw-config] 07Universal-Omega 0335e146d - Comment out wgCreateWikiPurposes temporarily [22:36:21] [02mw-config] 07Universal-Omega created branch 03Universal-Omega-patch-3 - 13https://git.io/vbvb3 [22:36:23] [02mw-config] 07Universal-Omega opened pull request 03#3757: Comment out wgCreateWikiPurposes temporarily - 13https://git.io/JqkQp [22:36:24] PROBLEM - mw8 Puppet on mw8 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[prometheus-php-fpm] [22:36:40] PROBLEM - mw9 Puppet on mw9 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[prometheus-php-fpm] [22:37:10] PROBLEM - mw11 Puppet on mw11 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[prometheus-php-fpm] [22:37:23] miraheze/mw-config - Universal-Omega the build passed. [22:37:32] I think you were doing something with prometheus-php-fpm, eh, paladox? [22:37:32] [02mw-config] 07Universal-Omega closed pull request 03#3757: Comment out wgCreateWikiPurposes temporarily - 13https://git.io/JqkQp [22:37:34] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Jqk7e [22:37:35] [02miraheze/mw-config] 07Universal-Omega 0344915fb - Comment out wgCreateWikiPurposes temporarily (#3757) [22:37:37] [02miraheze/mw-config] 07Universal-Omega deleted branch 03Universal-Omega-patch-3 [22:37:38] [02mw-config] 07Universal-Omega deleted branch 03Universal-Omega-patch-3 - 13https://git.io/vbvb3 [22:37:40] [02miraheze/puppet] 07Southparkfan pushed 031 commit to 03Southparkfan-patch-1 [+0/-0/±1] 13https://git.io/Jqk7v [22:37:41] [02miraheze/puppet] 07Southparkfan 0383e7d9e - matomo: use mailacceptinggeneralid field in LDAP for retrieving email address [22:37:43] [02puppet] 07Southparkfan created branch 03Southparkfan-patch-1 - 13https://git.io/vbiAS [22:37:44] [02puppet] 07Southparkfan opened pull request 03#1689: matomo: use mailacceptinggeneralid field in LDAP for retrieving email… - 13https://git.io/Jqk7f [22:38:10] [02puppet] 07paladox edited pull request 03#1689: matomo: use mailacceptinggeneralid field in LDAP for retrieving email address - 13https://git.io/Jqk7f [22:38:14] [02puppet] 07paladox edited pull request 03#1689: matomo: use mailacceptinggeneralid field in LDAP for retrieving email address - 13https://git.io/Jqk7f [22:38:17] [02puppet] 07paladox closed pull request 03#1689: matomo: use mailacceptinggeneralid field in LDAP for retrieving email address - 13https://git.io/Jqk7f [22:38:18] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Jqk7J [22:38:20] [02miraheze/puppet] 07Southparkfan 037a7bc40 - matomo: use mailacceptinggeneralid field in LDAP for retrieving email address (#1689) [22:38:21] [02puppet] 07paladox deleted branch 03Southparkfan-patch-1 - 13https://git.io/vbiAS [22:38:23] [02miraheze/puppet] 07paladox deleted branch 03Southparkfan-patch-1 [22:38:24] Alot going on from everyone today. [22:38:32] yeah [22:38:33] miraheze/mw-config - Universal-Omega the build passed. [22:40:40] [02miraheze/puppet] 07paladox pushed 031 commit to 03patch1 [+0/-0/±3] 13https://git.io/Jqk7q [22:40:42] [02miraheze/puppet] 07paladox 033517d33 - prometheus::php-fpm: switch to Lusitaniae/phpfpm_exporter [22:42:22] RECOVERY - mw8 Puppet on mw8 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [22:42:31] [02miraheze/mediawiki] 07Universal-Omega pushed 031 commit to 03REL1_35 [+0/-0/±1] 13https://git.io/Jqk73 [22:42:32] [02miraheze/mediawiki] 07Universal-Omega 03bbf70d4 - Update CreateWiki [22:44:59] [02miraheze/puppet] 07paladox pushed 031 commit to 03patch1 [+0/-0/±3] 13https://git.io/Jqk7G [22:45:00] [02miraheze/puppet] 07paladox 030c4f349 - prometheus::php-fpm: switch to Lusitaniae/phpfpm_exporter [22:50:32] RECOVERY - mw9 Puppet on mw9 is OK: OK: Puppet is currently enabled, last run 5 seconds ago with 0 failures [22:51:01] [02puppet] 07paladox deleted branch 03patch1 - 13https://git.io/vbiAS [22:51:03] [02miraheze/puppet] 07paladox deleted branch 03patch1 [22:51:05] [02puppet] 07paladox deleted branch 03paladox-patch-5 - 13https://git.io/vbiAS [22:51:07] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-5 [22:51:08] [02puppet] 07paladox deleted branch 03revert-1568-paladox-patch-3 - 13https://git.io/vbiAS [22:51:10] [02miraheze/puppet] 07paladox deleted branch 03revert-1568-paladox-patch-3 [22:51:10] RECOVERY - mw11 Puppet on mw11 is OK: OK: Puppet is currently enabled, last run 40 seconds ago with 0 failures [22:51:11] [02puppet] 07paladox deleted branch 03paladox-patch-4 - 13https://git.io/vbiAS [22:51:13] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-4 [22:51:14] [02puppet] 07paladox deleted branch 03paladox-patch-2 - 13https://git.io/vbiAS [22:51:16] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-2 [23:01:59] RECOVERY - phab2 Puppet on phab2 is OK: OK: Puppet is currently enabled, last run 27 seconds ago with 0 failures [23:07:50] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/Jqk7Q [23:07:51] [02miraheze/puppet] 07paladox 031222daf - prometheus::redis_exporter: Switch to systèmd::service [23:07:53] [02puppet] 07paladox created branch 03paladox-patch-1 - 13https://git.io/vbiAS [23:07:54] [02puppet] 07paladox opened pull request 03#1690: prometheus::redis_exporter: Switch to systèmd::service - 13https://git.io/Jqk77 [23:08:53] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-1 [+1/-1/±0] 13https://git.io/Jqk7A [23:08:54] [02miraheze/puppet] 07paladox 038a9e4ac - Rename modules/prometheus/files/redis/prometheus-redis.systemd to modules/prometheus/templates/initscripts/prometheus-redis-exporter.systemd.erb [23:08:56] [02puppet] 07paladox synchronize pull request 03#1690: prometheus::redis_exporter: Switch to systèmd::service - 13https://git.io/Jqk77 [23:10:32] [02puppet] 07paladox closed pull request 03#1690: prometheus::redis_exporter: Switch to systèmd::service - 13https://git.io/Jqk77 [23:10:33] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+1/-1/±1] 13https://git.io/Jqk5U [23:10:35] [02miraheze/puppet] 07paladox 03e2fae9c - prometheus::redis_exporter: Switch to systèmd::service (#1690) [23:10:36] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-1 [23:10:38] [02puppet] 07paladox deleted branch 03paladox-patch-1 - 13https://git.io/vbiAS [23:12:15] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Jqk5G [23:12:16] [02miraheze/puppet] 07paladox 0322a8ad6 - prometheus::redis_exporter: update to 1.17.1 [23:13:27] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Jqk5l [23:13:29] [02miraheze/puppet] 07paladox 0326fa84a - Fix notify for prometheus::redis_exporter [23:15:24] [02miraheze/CreateWiki] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://git.io/Jqk5w [23:15:26] [02miraheze/CreateWiki] 07Universal-Omega 0399e9d14 - RequestWikiRequestViewer: Add edit-bio and edit-purpose fields to submitForm() [23:15:27] [02CreateWiki] 07Universal-Omega created branch 03Universal-Omega-patch-1 - 13https://git.io/vpJTL [23:15:46] [02CreateWiki] 07Universal-Omega opened pull request 03#197: Add edit-bio and edit-purpose fields to submitForm() - 13https://git.io/Jqk56 [23:16:41] miraheze/CreateWiki - Universal-Omega the build passed. [23:18:08] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/Jqk5d [23:18:09] [02miraheze/puppet] 07paladox 033fbdcaa - redis: Switch to systemd::service [23:18:11] [02puppet] 07paladox created branch 03paladox-patch-1 - 13https://git.io/vbiAS [23:18:12] [02puppet] 07paladox opened pull request 03#1691: redis: Switch to systemd::service - 13https://git.io/Jqk5F [23:19:17] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-1 [+1/-1/±0] 13https://git.io/Jqk5h [23:19:18] [02miraheze/puppet] 07paladox 035aeb2c4 - Rename modules/redis/files/redis-server.systemd to modules/redis/templates/initscripts/redis.systemd.erb [23:19:20] [02puppet] 07paladox synchronize pull request 03#1691: redis: Switch to systemd::service - 13https://git.io/Jqk5F [23:19:55] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/Jqkdf [23:19:57] [02miraheze/puppet] 07paladox 0391f22c5 - Update init.pp [23:19:58] [02puppet] 07paladox synchronize pull request 03#1691: redis: Switch to systemd::service - 13https://git.io/Jqk5F [23:20:20] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/Jqkdk [23:20:22] [02miraheze/puppet] 07paladox 038476aed - Update init.pp [23:20:23] [02puppet] 07paladox synchronize pull request 03#1691: redis: Switch to systemd::service - 13https://git.io/Jqk5F [23:20:36] [02puppet] 07paladox closed pull request 03#1691: redis: Switch to systemd::service - 13https://git.io/Jqk5F [23:20:37] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+1/-1/±1] 13https://git.io/Jqkdt [23:20:39] [02miraheze/puppet] 07paladox 0331ea868 - redis: Switch to systemd::service (#1691) [23:20:41] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-1 [23:20:42] [02puppet] 07paladox deleted branch 03paladox-patch-1 - 13https://git.io/vbiAS [23:21:20] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+1/-1/±0] 13https://git.io/Jqkdm [23:21:21] [02miraheze/puppet] 07paladox 033267ecf - Fix [23:24:10] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/Jqkd4 [23:24:11] [02miraheze/puppet] 07paladox 03330171a - puppetserver: Switch to systemd::service [23:24:13] [02puppet] 07paladox created branch 03paladox-patch-1 - 13https://git.io/vbiAS [23:24:14] [02puppet] 07paladox opened pull request 03#1692: puppetserver: Switch to systemd::service - 13https://git.io/JqkdB [23:24:49] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-1 [+1/-1/±0] 13https://git.io/Jqkdu [23:24:51] [02miraheze/puppet] 07paladox 03bed188a - Rename modules/puppetserver/files/puppetserver.systemd to modules/puppetserver/templates/initscripts/puppetserver.systemd.erb [23:24:52] [02puppet] 07paladox synchronize pull request 03#1692: puppetserver: Switch to systemd::service - 13https://git.io/JqkdB [23:25:35] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/JqkdV [23:25:36] [02miraheze/puppet] 07paladox 037645967 - Update init.pp [23:25:38] [02puppet] 07paladox synchronize pull request 03#1692: puppetserver: Switch to systemd::service - 13https://git.io/JqkdB [23:25:59] [02puppet] 07paladox closed pull request 03#1692: puppetserver: Switch to systemd::service - 13https://git.io/JqkdB [23:26:01] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+1/-1/±1] 13https://git.io/Jqkd6 [23:26:03] [02miraheze/puppet] 07paladox 031840b26 - puppetserver: Switch to systemd::service (#1692) [23:26:04] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-1 [23:26:05] [02puppet] 07paladox deleted branch 03paladox-patch-1 - 13https://git.io/vbiAS [23:30:56] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/JqkFU [23:30:58] [02miraheze/puppet] 07paladox 035d84610 - mariadb::config: Switch to systemd::unit for override [23:30:59] [02puppet] 07paladox created branch 03paladox-patch-1 - 13https://git.io/vbiAS [23:31:01] [02puppet] 07paladox opened pull request 03#1693: mariadb::config: Switch to systemd::unit for override - 13https://git.io/JqkFT [23:31:23] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-1 [+1/-1/±0] 13https://git.io/JqkFt [23:31:24] [02miraheze/puppet] 07paladox 0366937c7 - Rename modules/mariadb/files/no-timeout-mariadb.conf to modules/mariadb/templates/mariadb-systemd-override.conf.erb [23:31:26] [02puppet] 07paladox synchronize pull request 03#1693: mariadb::config: Switch to systemd::unit for override - 13https://git.io/JqkFT [23:32:11] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/JqkF3 [23:32:12] [02miraheze/puppet] 07paladox 0357c3d4e - Update config.pp [23:32:14] [02puppet] 07paladox synchronize pull request 03#1693: mariadb::config: Switch to systemd::unit for override - 13https://git.io/JqkFT [23:32:23] [02puppet] 07paladox closed pull request 03#1693: mariadb::config: Switch to systemd::unit for override - 13https://git.io/JqkFT [23:32:25] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+1/-1/±1] 13https://git.io/JqkFG [23:32:26] [02miraheze/puppet] 07paladox 03f85a0d3 - mariadb::config: Switch to systemd::unit for override (#1693) [23:32:28] [02puppet] 07paladox deleted branch 03paladox-patch-1 - 13https://git.io/vbiAS [23:32:29] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-1 [23:35:44] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/JqkFa [23:35:45] [02miraheze/puppet] 07paladox 0325bb33e - mediawiki::jobrunner: Switch to systemd::service [23:35:47] [02puppet] 07paladox created branch 03paladox-patch-1 - 13https://git.io/vbiAS [23:35:48] [02puppet] 07paladox opened pull request 03#1694: mediawiki::jobrunner: Switch to systemd::service - 13https://git.io/JqkFV [23:36:08] [02CreateWiki] 07Universal-Omega closed pull request 03#197: Add edit-bio and edit-purpose fields to submitForm() - 13https://git.io/Jqk56 [23:36:09] [02miraheze/CreateWiki] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JqkFo [23:36:11] [02miraheze/CreateWiki] 07Universal-Omega 0345c8597 - RequestWikiRequestViewer: Add edit-bio and edit-purpose fields to submitForm() (#197) [23:36:12] [02CreateWiki] 07Universal-Omega deleted branch 03Universal-Omega-patch-1 - 13https://git.io/vpJTL [23:36:14] [02miraheze/CreateWiki] 07Universal-Omega deleted branch 03Universal-Omega-patch-1 [23:36:39] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-1 [+1/-1/±0] 13https://git.io/JqkFi [23:36:41] [02miraheze/puppet] 07paladox 03c74215c - Rename modules/mediawiki/files/jobrunner/jobchron.systemd to modules/mediawiki/templates/initscripts/jobchron.systemd.erb [23:36:42] [02puppet] 07paladox synchronize pull request 03#1694: mediawiki::jobrunner: Switch to systemd::service - 13https://git.io/JqkFV [23:36:57] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-1 [+1/-1/±0] 13https://git.io/JqkFM [23:36:59] [02miraheze/puppet] 07paladox 035b5db7d - Rename modules/mediawiki/files/jobrunner/jobrunner.systemd to modules/mediawiki/templates/initscripts/jobrunner.systemd.erb [23:37:00] [02puppet] 07paladox synchronize pull request 03#1694: mediawiki::jobrunner: Switch to systemd::service - 13https://git.io/JqkFV [23:37:14] miraheze/CreateWiki - Universal-Omega the build passed. [23:37:18] [02puppet] 07paladox closed pull request 03#1694: mediawiki::jobrunner: Switch to systemd::service - 13https://git.io/JqkFV [23:37:19] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+2/-2/±1] 13https://git.io/JqkFS [23:37:21] [02miraheze/puppet] 07paladox 03f4f7948 - mediawiki::jobrunner: Switch to systemd::service (#1694) [23:37:22] [02puppet] 07paladox deleted branch 03paladox-patch-1 - 13https://git.io/vbiAS [23:37:24] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-1 [23:41:12] [02miraheze/mediawiki] 07Universal-Omega pushed 031 commit to 03REL1_35 [+0/-0/±1] 13https://git.io/JqkbL [23:41:14] [02miraheze/mediawiki] 07Universal-Omega 038317738 - Update CreateWiki [23:51:33] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-3 [+0/-0/±1] 13https://git.io/Jqkbr [23:51:35] [02miraheze/puppet] 07paladox 03743cfdd - Update logging.pp [23:51:36] [02puppet] 07paladox synchronize pull request 03#1672: Introduce gluster::logging - 13https://git.io/JthIO [23:57:33] Sario around? [23:57:35] err [23:57:38] sorry, i meant SPF|Cloud [23:57:55] for a few mins, yes [23:58:03] I'm here too [23:58:42] SPF|Cloud i'm wondering if you can help me with parsing gluster glusterd.log with syslog_ng please? [23:58:45] i've got this: [23:58:50] https://github.com/miraheze/puppet/pull/1672/files#diff-8dee9a1f54a342ce8d75548f3679aa85d8f42d778a3e0146ae50e2a104079792 [23:58:51] [ Introduce gluster::logging by paladox · Pull Request #1672 · miraheze/puppet · GitHub ] - github.com [23:59:03] which i see graylog is reciving but, it's not parsing correctly [23:59:11] see https://graylog.miraheze.org/messages/graylog_112/56b7c0c4-7fa0-11eb-b6dc-0200001a24a4 [23:59:39] i haven't been able to figure out how to set a application name in syslog_ng (as you can see it's setting "The" as the app name) [23:59:44] * SPF|Cloud looks [23:59:59] message "I [MSGID: 106487] should also be removed