[00:07:59] PROBLEM - bacula1 Bacula Private Git on bacula1 is WARNING: WARNING: Full, 4092 files, 8.388MB, 2019-07-14 00:05:00 (1.1 weeks ago) [00:11:38] !log provision db5 [00:11:42] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [00:14:07] [02miraheze/dns] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fjM1m [00:14:08] [02miraheze/dns] 07paladox 032b7b873 - Add db5 to dns [00:23:16] PROBLEM - es2 Puppet on es2 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [00:24:49] [02puppet] 07paladox created branch 03paladox-patch-1 - 13https://git.io/vbiAS [00:24:50] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/fjM1s [00:24:52] [02miraheze/puppet] 07paladox 038b85038 - Add db5 to puppet [00:24:53] [02puppet] 07paladox opened pull request 03#1046: Add db5 to puppet - 13https://git.io/fjM1G [00:25:40] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-1 [+1/-0/±0] 13https://git.io/fjM1Z [00:25:41] [02miraheze/puppet] 07paladox 03fd2eedb - Create db5.yaml [00:25:43] [02puppet] 07paladox synchronize pull request 03#1046: Add db5 to puppet - 13https://git.io/fjM1G [00:25:49] [02puppet] 07paladox closed pull request 03#1046: Add db5 to puppet - 13https://git.io/fjM1G [00:25:50] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+1/-0/±1] 13https://git.io/fjM1n [00:25:52] [02miraheze/puppet] 07paladox 03c71893c - Add db5 to puppet (#1046) * Add db5 to puppet * Create db5.yaml [00:25:53] [02puppet] 07paladox deleted branch 03paladox-patch-1 - 13https://git.io/vbiAS [00:25:55] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-1 [00:34:05] PROBLEM - db5 MySQL on db5 is CRITICAL: Host '185.52.1.76' is not allowed to connect to this MariaDB server [00:42:32] !log db5: sudo fallocate -l 8G /swapfile ; sudo chmod 600 /swapfile ; sudo mkswap /swapfile ; sudo swapon /swapfile [00:42:37] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [00:44:53] [02mw-config] 07paladox created branch 03paladox-patch-3 - 13https://git.io/vbvb3 [00:44:54] [02miraheze/mw-config] 07paladox pushed 031 commit to 03paladox-patch-3 [+0/-0/±1] 13https://git.io/fjM1u [00:44:56] [02miraheze/mw-config] 07paladox 03d26dbbc - Add db5 to Database [00:44:57] [02mw-config] 07paladox opened pull request 03#2705: Add db5 to Database - 13https://git.io/fjM1z [00:45:47] [02puppet] 07paladox created branch 03paladox-patch-1 - 13https://git.io/vbiAS [00:45:48] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/fjM1g [00:45:50] [02miraheze/puppet] 07paladox 03b50775f - Add mediawiki-internal-db-master-db5.miraheze.org [00:46:20] [02puppet] 07paladox opened pull request 03#1047: Add mediawiki-internal-db-master-db5.miraheze.org - 13https://git.io/fjM12 [00:46:37] [02puppet] 07paladox closed pull request 03#1047: Add mediawiki-internal-db-master-db5.miraheze.org - 13https://git.io/fjM12 [00:46:38] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fjM1a [00:46:40] [02miraheze/puppet] 07paladox 03c6c8d95 - Add mediawiki-internal-db-master-db5.miraheze.org (#1047) [00:46:41] [02puppet] 07paladox deleted branch 03paladox-patch-1 - 13https://git.io/vbiAS [00:46:43] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-1 [01:02:36] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fjM1X [01:02:38] [02miraheze/puppet] 07paladox 03395b574 - db: Update dbcopy public key [01:10:16] [02puppet] 07paladox created branch 03paladox-patch-1 - 13https://git.io/vbiAS [01:10:18] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/fjM1y [01:10:19] [02miraheze/puppet] 07paladox 032cf0442 - matomo: Switch to db5.miraheze.org [01:10:21] [02puppet] 07paladox opened pull request 03#1048: matomo: Switch to db5.miraheze.org - 13https://git.io/fjM1S [01:46:41] [02puppet] 07paladox closed pull request 03#1048: matomo: Switch to db5.miraheze.org - 13https://git.io/fjM1S [01:46:42] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fjM1j [01:46:44] [02miraheze/puppet] 07paladox 0305e6502 - matomo: Switch to db5.miraheze.org (#1048) [01:46:45] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-1 [01:46:47] [02puppet] 07paladox deleted branch 03paladox-patch-1 - 13https://git.io/vbiAS [01:51:29] [02miraheze/dns] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fjMMv [01:51:30] [02miraheze/dns] 07paladox 03c63da1d - Update miraheze.org [01:52:02] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fjMMJ [01:52:03] [02miraheze/puppet] 07paladox 039fcdf1d - Update init.pp [01:52:46] [02miraheze/mw-config] 07paladox pushed 031 commit to 03paladox-patch-3 [+0/-0/±1] 13https://git.io/fjMMU [01:52:48] [02miraheze/mw-config] 07paladox 03a343de6 - Update Database.php [01:52:49] PROBLEM - db5 MySQL on db5 is UNKNOWN: [01:52:49] [02mw-config] 07paladox synchronize pull request 03#2705: Add db5 to Database - 13https://git.io/fjM1z [01:52:52] PROBLEM - db5 Disk Space on db5 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [01:52:54] [02mw-config] 07paladox closed pull request 03#2705: Add db5 to Database - 13https://git.io/fjM1z [01:52:55] [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fjMMT [01:52:57] [02miraheze/mw-config] 07paladox 03c40a504 - Add db5 to Database (#2705) * Add db5 to Database * Update Database.php [01:52:58] [02miraheze/mw-config] 07paladox deleted branch 03paladox-patch-3 [01:53:00] [02mw-config] 07paladox deleted branch 03paladox-patch-3 - 13https://git.io/vbvb3 [01:53:15] PROBLEM - db5 Puppet on db5 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [01:53:35] PROBLEM - misc2 HTTPS on misc2 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 372 bytes in 0.009 second response time [01:54:09] PROBLEM - db5 SSH on db5 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:54:26] miraheze/mw-config/paladox-patch-3/a343de6 - paladox The build has errored. https://travis-ci.org/miraheze/mw-config/builds/561879535 [01:54:38] PROBLEM - db5 Current Load on db5 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [01:55:01] PROBLEM - Host db5 is DOWN: PING CRITICAL - Packet loss = 100% [02:04:12] RECOVERY - Host db5 is UP: PING OK - Packet loss = 0%, RTA = 78.04 ms [02:04:29] RECOVERY - db5 SSH on db5 is OK: SSH OK - OpenSSH_7.4p1 Debian-10+deb9u6 (protocol 2.0) [02:21:02] RECOVERY - misc2 HTTPS on misc2 is OK: HTTP OK: HTTP/1.1 200 OK - 897 bytes in 8.793 second response time [02:39:41] [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fjMMl [02:39:42] [02miraheze/mw-config] 07paladox 033a94f03 - Revert "Add db5 to Database (#2705)" This reverts commit c40a504d33867b59be815ab597321cfc0e1d58b4. [02:43:39] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/fjMM4 [02:43:41] [02miraheze/puppet] 07paladox 039265039 - Add REPLICATION CLIENT to mediawiki grants [02:43:42] [02puppet] 07paladox created branch 03paladox-patch-1 - 13https://git.io/vbiAS [02:43:44] [02puppet] 07paladox opened pull request 03#1049: Add REPLICATION CLIENT to mediawiki grants - 13https://git.io/fjMMB [02:49:22] PROBLEM - misc2 HTTPS on misc2 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:01:43] RECOVERY - misc2 HTTPS on misc2 is OK: HTTP OK: HTTP/1.1 200 OK - 897 bytes in 6.315 second response time [03:03:20] PROBLEM - es1 Puppet on es1 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [03:03:36] !log GRANT REPLICATION CLIENT ON *.* TO 'wikiadmin'@'%'; - db4 [03:03:40] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [03:04:45] [02puppet] 07paladox created branch 03paladox-patch-4 - 13https://git.io/vbiAS [03:04:46] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-4 [+0/-0/±1] 13https://git.io/fjMMw [03:04:48] [02miraheze/puppet] 07paladox 036800605 - Add grant to add REPLICATION CLIENT to wikiadmin [03:04:49] [02puppet] 07paladox opened pull request 03#1050: Add grant to add REPLICATION CLIENT to wikiadmin - 13https://git.io/fjMMr [03:04:57] PROBLEM - es1 Current Load on es1 is CRITICAL: CRITICAL - load average: 7.06, 4.66, 2.35 [03:05:28] RECOVERY - es1 Puppet on es1 is OK: OK: Puppet is currently enabled, last run 11 minutes ago with 0 failures [03:06:00] PROBLEM - misc2 HTTPS on misc2 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:06:16] [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fjMMo [03:06:17] [02miraheze/mw-config] 07paladox 038665208 - Add db5 to Database [03:06:53] PROBLEM - es1 Current Load on es1 is WARNING: WARNING - load average: 2.60, 3.94, 2.37 [03:08:49] RECOVERY - es1 Current Load on es1 is OK: OK - load average: 0.54, 2.71, 2.10 [03:24:19] RECOVERY - misc2 HTTPS on misc2 is OK: HTTP OK: HTTP/1.1 200 OK - 41772 bytes in 6.606 second response time [03:26:10] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fjMMi [03:26:12] [02miraheze/puppet] 07paladox 03062db5f - Revert "matomo: Switch to db5.miraheze.org (#1048)" This reverts commit 05e6502e4e7ac9746f78bd6e61720b76a7d696f8. [05:26:02] I do not know if it just happens to me, but in Meta I get this error. [05:26:03] https://cdn.discordapp.com/attachments/435711390544560128/602733062652166144/Screenshot_20190722-012508.png [06:06:46] @System Administrators ^^ a report on Phab as well [06:09:02] Paladox, Reception123, SPF|Cloud: ^^^ [06:16:22] @System Administrators reports are pretty high — mysql on db5 went critical on icinga at 03:02 and there’s reports since about then [06:21:48] [02miraheze/mw-config] 07Reception123 pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fjMDw [06:21:50] [02miraheze/mw-config] 07Reception123 03c72c061 - remove db5 causing errors on wikis [06:21:55] @RhinosF1 not sure but will try ^ [06:23:01] Reception123: check the exception logs - were all getting exception so it should be there [06:23:24] Looks back though [06:27:17] Also, I think we might want an incident report for this one — looks like we had issues for at least 3-4 hours [06:30:17] I have checked, it is a database error caused by the adding of db5 [06:30:34] There will be an incident report [06:31:05] Reception123: Okay, let me know if you need any more information on finding it / where reports where [06:38:34] Sure [08:10:12] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fjMSk [08:10:14] [02miraheze/services] 07MirahezeSSLBot 0385d9449 - BOT: Updating services config for wikis [08:25:11] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fjMS8 [08:25:13] [02miraheze/services] 07MirahezeSSLBot 033e024d3 - BOT: Updating services config for wikis [08:35:21] Hello to all. Can somebody help? I have a question regarding 'Featured article' on the main page of my wiki: 'http://cnt.miraheze.org'. I included the tex-code in the main page and I checked if the 'Extension:RandomSelection' is actived, which is; but it still does not work. The wiki asks for a template back, which should be the site appearing in t [08:35:21] he featured articles. [08:41:26] Einstein: in Special:MangeWiki/settings can you turn on 'Allow slow parser functions' and 'Enable string function functionality' [08:49:07] Einstein: is that better? You just need to create something at the featured article pages now [08:49:27] RF1|pingonreply one moment. [08:56:03] RF1|pingonreply I thought I did create something at the featured article page as I entered the code from this site: 'https://www.mediawiki.org/wiki/Extension:RandomSelection' in 'Combinatorial Number Theory:Today's featured article/July 21, 2019'. [08:56:03] [ Extension:RandomSelection' - MediaWiki ] - www.mediawiki.org [08:57:58] RF1|pingonreply I exchanged only the article's name with some of my site to see if they appear. [08:59:27] Einstein: you need to put content now in pages like https://combinatorialnumbertheory.miraheze.org/w/index.php?title=Template:Featured_article/Complex_binomial_distribution&action=edit&redlink=1 [08:59:27] [ Creating Template:Featured article/Complex binomial distribution - Combinatorial Number Theory ] - combinatorialnumbertheory.miraheze.org [09:00:13] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fjMSH [09:00:14] [02miraheze/services] 07MirahezeSSLBot 038c3cdf7 - BOT: Updating services config for wikis [09:00:45] RF1|pingonreply thank you very much for the fast reply. Can I ask another question? Because I am not sure what you mean by "content"? [09:01:21] Einstein: like what will actually appear on the main page. [09:03:06] RF1|pingonreply I meant the tex-code, because I tought articles that I have already written can appear there. [09:04:17] Einstein: then on https://combinatorialnumbertheory.miraheze.org/w/index.php?title=Combinatorial_Number_Theory:Today%27s_featured_article/July_21,_2019&action=edit remove the Template:Featured Article/ bit and it will transclude the whole article [09:04:18] [ Editing Combinatorial Number Theory:Today's featured article/July 21, 2019 - Combinatorial Number Theory ] - combinatorialnumbertheory.miraheze.org [09:09:58] RF1|pingonreply it works!!! Mayor thanks! But I have a question. Now there are two pages; one time 'Template:Featured_article/Complex_binomial_distribution' and one time 'Featured article/Complex binomial distribution'. Can I delete the latter? [09:11:01] Einstein: yes, it translcudes the article direclty so you only need that [09:12:08] RF1|pingonreply and I have a follow up question. Do you know a way, so I do not have to include manually all the articles in the choose list? [09:12:31] RF1|pingonreply thank you very much. This was really great help. [09:13:18] Einstein: no, i'm not sure on that [09:14:25] RF1|pingonreply anyways, this was great help as I was struggling with that problem for two days now. [09:14:44] Einstein: no problem, ask any time [09:15:57] RF1|pingonreply that is good to read, but I feel as if my question were boring. [09:18:27] RF1|pingonreply can I ask, if you know a way to limit the size of the article appearing in the featured article section as I did not find any specific tex code on that. [09:20:05] Einstein: a noinclude tag would stop a part being translcuded [09:20:39] RF1|pingonreply I am sorry, but I do not know what that means. [09:21:42] RF1|pingonreply I found this though: 'https://www.mediawiki.org/wiki/Transclusion#Partial_transclusion'. [09:21:42] [ Transclusion - MediaWiki ] - www.mediawiki.org [09:25:52] Einstein: look in the Transclution Markup section [09:27:21] RF1|pingonreply so I have to go to each article and add the ... markup? [09:27:48] Einstein: to restrict what gets transcluded yes [09:29:02] RF1|pingonreply seems a bit excessive, but I will try it. Thank you very much for your fine help. [09:29:09] Einstein: np [09:30:38] RF1|pingonreply have a nice day. You others too. And thanks very much for the fast help. [09:30:46] np [09:30:51] thx [09:34:55] RF1|pingonreply you are very welcome. [10:24:44] PROBLEM - bacula1 Bacula Databases db4 on bacula1 is WARNING: WARNING: Full, 759946 files, 78.12GB, 2019-07-07 10:22:00 (2.1 weeks ago) [11:20:12] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fjMQF [11:20:13] [02miraheze/services] 07MirahezeSSLBot 032804d53 - BOT: Updating services config for wikis [12:56:14] PROBLEM - misc2 HTTPS on misc2 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 372 bytes in 0.009 second response time [12:57:07] PROBLEM - cp4 Varnish Backends on cp4 is CRITICAL: 3 backends are down. mw1 mw2 mw3 [12:57:15] PROBLEM - cp4 HTTP 4xx/5xx ERROR Rate on cp4 is CRITICAL: CRITICAL - NGINX Error Rate is 89% [12:57:20] PROBLEM - cp3 Varnish Backends on cp3 is CRITICAL: 3 backends are down. mw1 mw2 mw3 [12:57:23] PROBLEM - misc1 webmail.miraheze.org HTTPS on misc1 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:57:39] PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 6 datacenters are down: 107.191.126.23/cpweb, 2604:180:0:33b::2/cpweb, 128.199.139.216/cpweb, 2400:6180:0:d0::403:f001/cpweb, 81.4.109.133/cpweb, 2a00:d880:5:8ea::ebc7/cpweb [12:57:40] !log root@db4:/home/paladox# rm /srv/mariadb/mysql-bin.001412 [12:57:46] !log restart mysql on db4 [12:58:02] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 6 datacenters are down: 107.191.126.23/cpweb, 2604:180:0:33b::2/cpweb, 128.199.139.216/cpweb, 2400:6180:0:d0::403:f001/cpweb, 81.4.109.133/cpweb, 2a00:d880:5:8ea::ebc7/cpweb [12:58:16] PROBLEM - cp2 Varnish Backends on cp2 is CRITICAL: 3 backends are down. mw1 mw2 mw3 [12:58:36] paladox is that u breaking phab? [12:58:39] PROBLEM - cp2 HTTP 4xx/5xx ERROR Rate on cp2 is CRITICAL: CRITICAL - NGINX Error Rate is 65% [12:58:53] db4 ran out of space, so i'm cleaning the bin logs to gain space. [12:59:33] paladox: db5 broke everything as well, see scrollback and https://phabricator.miraheze.org/T4564 when it works [12:59:33] [ Phabricator Setup Error ] - phabricator.miraheze.org [12:59:42] yup [13:00:11] aware, and speaking to JohnLewis and Reception123 about it. That was my fault. [13:00:39] PROBLEM - cp2 HTTP 4xx/5xx ERROR Rate on cp2 is WARNING: WARNING - NGINX Error Rate is 57% [13:01:05] paladox: you set db5 up didn't u? Reception reverted it earlier but wasn't sure what u did [13:01:09] RF1|pingonreply phab should be fixed now. [13:01:11] RECOVERY - misc1 webmail.miraheze.org HTTPS on misc1 is OK: HTTP OK: Status line output matched "HTTP/1.1 401 Unauthorized" - 5799 bytes in 0.040 second response time [13:01:12] RF1|pingonreply yeh [13:01:15] RECOVERY - cp4 HTTP 4xx/5xx ERROR Rate on cp4 is OK: OK - NGINX Error Rate is 24% [13:01:19] RECOVERY - cp3 Varnish Backends on cp3 is OK: All 5 backends are healthy [13:01:39] RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online [13:01:44] paladox: yes, phab is, was only seeing if you'd noticed the mess from eralier [13:02:00] RECOVERY - misc2 HTTPS on misc2 is OK: HTTP OK: HTTP/1.1 200 OK - 41772 bytes in 0.134 second response time [13:02:02] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [13:02:16] RECOVERY - cp2 Varnish Backends on cp2 is OK: All 5 backends are healthy [13:02:39] RECOVERY - cp2 HTTP 4xx/5xx ERROR Rate on cp2 is OK: OK - NGINX Error Rate is 5% [13:03:07] RECOVERY - cp4 Varnish Backends on cp4 is OK: All 5 backends are healthy [13:10:14] PROBLEM - www.b1tcash.org - LetsEncrypt on sslhost is CRITICAL: connect to address www.b1tcash.org and port 443: Connection refusedHTTP CRITICAL - Unable to open TCP socket [13:14:04] RECOVERY - www.b1tcash.org - LetsEncrypt on sslhost is OK: OK - Certificate 'b1tcash.org' will expire on Mon 19 Aug 2019 08:46:49 PM GMT +0000. [13:19:21] PROBLEM - misc2 Puppet on misc2 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 6 minutes ago with 0 failures [13:21:49] PROBLEM - www.b1tcash.org - LetsEncrypt on sslhost is CRITICAL: connect to address www.b1tcash.org and port 443: Connection refusedHTTP CRITICAL - Unable to open TCP socket [13:31:06] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fjMd4 [13:31:08] [02miraheze/puppet] 07paladox 0381a4e2e - matomo: Switch to db5 [13:32:17] paladox: good luck - matomo was one of the down services earlier, i'm around if anything needs checking, don't forget to update phab [13:32:54] ok [13:35:31] RECOVERY - www.b1tcash.org - LetsEncrypt on sslhost is OK: OK - Certificate 'b1tcash.org' will expire on Mon 19 Aug 2019 08:46:49 PM GMT +0000. [13:36:24] PROBLEM - misc2 HTTPS on misc2 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:38:31] PROBLEM - test1 Current Load on test1 is CRITICAL: CRITICAL - load average: 2.31, 1.81, 1.22 [13:40:28] RECOVERY - test1 Current Load on test1 is OK: OK - load average: 0.96, 1.47, 1.16 [13:46:36] RECOVERY - misc2 HTTPS on misc2 is OK: HTTP OK: HTTP/1.1 200 OK - 897 bytes in 1.639 second response time [13:50:19] PROBLEM - www.b1tcash.org - LetsEncrypt on sslhost is CRITICAL: connect to address www.b1tcash.org and port 443: Connection refusedHTTP CRITICAL - Unable to open TCP socket [13:52:50] PROBLEM - misc2 HTTPS on misc2 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:54:04] PROBLEM - test1 Current Load on test1 is CRITICAL: CRITICAL - load average: 2.57, 2.29, 1.67 [13:56:02] PROBLEM - test1 Current Load on test1 is WARNING: WARNING - load average: 1.50, 1.88, 1.58 [13:57:59] PROBLEM - test1 Current Load on test1 is CRITICAL: CRITICAL - load average: 2.12, 2.05, 1.68 [13:58:00] RECOVERY - www.b1tcash.org - LetsEncrypt on sslhost is OK: OK - Certificate 'b1tcash.org' will expire on Mon 19 Aug 2019 08:46:49 PM GMT +0000. [13:59:58] RECOVERY - test1 Current Load on test1 is OK: OK - load average: 1.04, 1.64, 1.57 [14:05:11] RECOVERY - misc2 HTTPS on misc2 is OK: HTTP OK: HTTP/1.1 200 OK - 897 bytes in 7.872 second response time [14:06:42] PROBLEM - www.b1tcash.org - LetsEncrypt on sslhost is CRITICAL: connect to address www.b1tcash.org and port 443: Connection refusedHTTP CRITICAL - Unable to open TCP socket [14:06:49] PROBLEM - test1 Current Load on test1 is WARNING: WARNING - load average: 1.35, 1.82, 1.72 [14:08:37] RECOVERY - www.b1tcash.org - LetsEncrypt on sslhost is OK: OK - Certificate 'b1tcash.org' will expire on Mon 19 Aug 2019 08:46:49 PM GMT +0000. [14:08:47] PROBLEM - test1 Current Load on test1 is CRITICAL: CRITICAL - load average: 2.28, 1.98, 1.79 [14:09:29] PROBLEM - misc2 HTTPS on misc2 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:18:34] PROBLEM - test1 Current Load on test1 is WARNING: WARNING - load average: 1.14, 1.62, 1.77 [14:20:14] PROBLEM - www.b1tcash.org - LetsEncrypt on sslhost is CRITICAL: connect to address www.b1tcash.org and port 443: Connection refusedHTTP CRITICAL - Unable to open TCP socket [14:20:31] PROBLEM - test1 Current Load on test1 is CRITICAL: CRITICAL - load average: 2.09, 1.97, 1.88 [14:24:04] RECOVERY - www.b1tcash.org - LetsEncrypt on sslhost is OK: OK - Certificate 'b1tcash.org' will expire on Mon 19 Aug 2019 08:46:49 PM GMT +0000. [14:29:51] !log db4: MariaDB [piwik]> drop database piwik; [14:29:55] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [14:31:18] PROBLEM - db4 Disk Space on db4 is WARNING: DISK WARNING - free space: / 34751 MB (9% inode=96%); [14:37:36] PROBLEM - www.b1tcash.org - LetsEncrypt on sslhost is CRITICAL: connect to address www.b1tcash.org and port 443: Connection refusedHTTP CRITICAL - Unable to open TCP socket [14:38:07] PROBLEM - test1 Current Load on test1 is WARNING: WARNING - load average: 1.00, 1.73, 1.93 [14:39:31] RECOVERY - www.b1tcash.org - LetsEncrypt on sslhost is OK: OK - Certificate 'b1tcash.org' will expire on Mon 19 Aug 2019 08:46:49 PM GMT +0000. [14:45:58] PROBLEM - test1 Current Load on test1 is CRITICAL: CRITICAL - load average: 2.14, 1.55, 1.74 [14:49:58] PROBLEM - test1 Current Load on test1 is WARNING: WARNING - load average: 1.68, 1.82, 1.81 [14:51:58] RECOVERY - test1 Current Load on test1 is OK: OK - load average: 0.98, 1.51, 1.70 [14:57:04] PROBLEM - es3 Current Load on es3 is WARNING: WARNING - load average: 3.64, 2.45, 1.22 [14:57:11] PROBLEM - es4 Current Load on es4 is WARNING: WARNING - load average: 3.71, 2.79, 1.39 [14:59:04] RECOVERY - es3 Current Load on es3 is OK: OK - load average: 0.95, 1.95, 1.20 [14:59:09] RECOVERY - es4 Current Load on es4 is OK: OK - load average: 0.94, 2.17, 1.33 [14:59:21] RECOVERY - misc2 Puppet on misc2 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [15:00:51] PROBLEM - www.b1tcash.org - LetsEncrypt on sslhost is CRITICAL: connect to address www.b1tcash.org and port 443: Connection refusedHTTP CRITICAL - Unable to open TCP socket [15:08:33] RECOVERY - www.b1tcash.org - LetsEncrypt on sslhost is OK: OK - Certificate 'b1tcash.org' will expire on Mon 19 Aug 2019 08:46:49 PM GMT +0000. [15:20:08] PROBLEM - www.b1tcash.org - LetsEncrypt on sslhost is CRITICAL: connect to address www.b1tcash.org and port 443: Connection refusedHTTP CRITICAL - Unable to open TCP socket [15:29:45] RECOVERY - www.b1tcash.org - LetsEncrypt on sslhost is OK: OK - Certificate 'b1tcash.org' will expire on Mon 19 Aug 2019 08:46:49 PM GMT +0000. [15:39:32] PROBLEM - www.b1tcash.org - LetsEncrypt on sslhost is CRITICAL: connect to address www.b1tcash.org and port 443: Connection refusedHTTP CRITICAL - Unable to open TCP socket [15:41:31] RECOVERY - www.b1tcash.org - LetsEncrypt on sslhost is OK: OK - Certificate 'b1tcash.org' will expire on Mon 19 Aug 2019 08:46:49 PM GMT +0000. [15:44:15] is there any rule on meta about removing things from your TP [15:45:32] PROBLEM - www.b1tcash.org - LetsEncrypt on sslhost is CRITICAL: connect to address www.b1tcash.org and port 443: Connection refusedHTTP CRITICAL - Unable to open TCP socket [15:48:26] PROBLEM - mh142.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'mh142.com' expires in 15 day(s) (Wed 07 Aug 2019 03:46:16 PM GMT +0000). [15:48:50] PROBLEM - www.mh142.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'mh142.com' expires in 15 day(s) (Wed 07 Aug 2019 03:46:16 PM GMT +0000). [15:49:31] RECOVERY - www.b1tcash.org - LetsEncrypt on sslhost is OK: OK - Certificate 'b1tcash.org' will expire on Mon 19 Aug 2019 08:46:49 PM GMT +0000. [15:53:32] PROBLEM - www.b1tcash.org - LetsEncrypt on sslhost is CRITICAL: connect to address www.b1tcash.org and port 443: Connection refusedHTTP CRITICAL - Unable to open TCP socket [15:54:08] RF1|pingonreply: no, freedom of choice to the user whose tp is it [15:55:00] JohnLewis: i've reverted someone telling someone they can't do that [15:55:38] +1 [15:57:31] RECOVERY - www.b1tcash.org - LetsEncrypt on sslhost is OK: OK - Certificate 'b1tcash.org' will expire on Mon 19 Aug 2019 08:46:49 PM GMT +0000. [16:01:32] PROBLEM - www.b1tcash.org - LetsEncrypt on sslhost is CRITICAL: connect to address www.b1tcash.org and port 443: Connection refusedHTTP CRITICAL - Unable to open TCP socket [16:19:31] RECOVERY - www.b1tcash.org - LetsEncrypt on sslhost is OK: OK - Certificate 'b1tcash.org' will expire on Mon 19 Aug 2019 08:46:49 PM GMT +0000. [16:20:52] [02ssl] 07paladox created branch 03paladox-patch-1 - 13https://git.io/vxP9L [16:20:53] [02miraheze/ssl] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/fjMbX [16:20:55] [02miraheze/ssl] 07paladox 033dd4238 - Remove www.b1tcash.org ssl certificate Domain is not pointing at our nameserver/mw-lb [16:20:56] [02ssl] 07paladox opened pull request 03#203: Remove www.b1tcash.org ssl certificate - 13https://git.io/fjMb1 [16:21:12] [02miraheze/ssl] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/fjMbM [16:21:14] [02miraheze/ssl] 07paladox 03813ebd8 - Update redirects.yaml [16:21:15] [02ssl] 07paladox synchronize pull request 03#203: Remove www.b1tcash.org ssl certificate - 13https://git.io/fjMb1 [16:21:53] [02miraheze/ssl] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-1/±0] 13https://git.io/fjMbD [16:21:54] [02miraheze/ssl] 07paladox 03ce9b70a - Delete b1tcash.org.crt [16:21:56] [02ssl] 07paladox synchronize pull request 03#203: Remove www.b1tcash.org ssl certificate - 13https://git.io/fjMb1 [16:22:08] [02ssl] 07paladox closed pull request 03#203: Remove www.b1tcash.org ssl certificate - 13https://git.io/fjMb1 [16:22:09] [02miraheze/ssl] 07paladox pushed 031 commit to 03master [+0/-1/±2] 13https://git.io/fjMby [16:22:11] [02miraheze/ssl] 07paladox 031fb1032 - Remove www.b1tcash.org ssl certificate (#203) * Remove www.b1tcash.org ssl certificate Domain is not pointing at our nameserver/mw-lb * Update redirects.yaml * Delete b1tcash.org.crt [16:22:12] [02miraheze/ssl] 07paladox deleted branch 03paladox-patch-1 [16:22:14] [02ssl] 07paladox deleted branch 03paladox-patch-1 - 13https://git.io/vxP9L [16:23:32] PROBLEM - www.b1tcash.org - LetsEncrypt on sslhost is CRITICAL: connect to address www.b1tcash.org and port 443: Connection refusedHTTP CRITICAL - Unable to open TCP socket [16:24:37] PROBLEM - b1tcash.org - LetsEncrypt on sslhost is CRITICAL: connect to address b1tcash.org and port 443: Connection refusedHTTP CRITICAL - Unable to open TCP socket [16:25:20] PROBLEM - cp4 Puppet on cp4 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): File[b1tcash.org] [16:26:12] PROBLEM - mw3 Puppet on mw3 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): File[b1tcash.org] [16:26:17] PROBLEM - mw1 Puppet on mw1 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): File[b1tcash.org] [16:26:35] RECOVERY - b1tcash.org - LetsEncrypt on sslhost is OK: OK - Certificate '*.miraheze.org' will expire on Wed 23 Oct 2019 08:12:13 PM GMT +0000. [16:26:38] PROBLEM - mw2 Puppet on mw2 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): File[b1tcash.org] [16:27:19] RECOVERY - cp4 Puppet on cp4 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [16:27:21] PROBLEM - cp3 Puppet on cp3 is CRITICAL: CRITICAL: Puppet has 2 failures. Last run 2 minutes ago with 2 failures. Failed resources (up to 3 shown): File[b1tcash.org],File[b1tcash.org_private] [16:27:27] PROBLEM - cp2 Puppet on cp2 is CRITICAL: CRITICAL: Puppet has 2 failures. Last run 3 minutes ago with 2 failures. Failed resources (up to 3 shown): File[b1tcash.org],File[b1tcash.org_private] [16:27:31] RECOVERY - www.b1tcash.org - LetsEncrypt on sslhost is OK: OK - Certificate '*.miraheze.org' will expire on Wed 23 Oct 2019 08:12:13 PM GMT +0000. [16:30:32] PROBLEM - b1tcash.org - LetsEncrypt on sslhost is CRITICAL: connect to address b1tcash.org and port 443: Connection refusedHTTP CRITICAL - Unable to open TCP socket [16:32:30] RECOVERY - b1tcash.org - LetsEncrypt on sslhost is OK: OK - Certificate '*.miraheze.org' will expire on Wed 23 Oct 2019 08:12:13 PM GMT +0000. [16:34:11] RECOVERY - mw3 Puppet on mw3 is OK: OK: Puppet is currently enabled, last run 43 seconds ago with 0 failures [16:34:25] RECOVERY - mw1 Puppet on mw1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [16:34:32] RECOVERY - mw2 Puppet on mw2 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [16:34:59] RECOVERY - cp3 Puppet on cp3 is OK: OK: Puppet is currently enabled, last run 27 seconds ago with 0 failures [16:35:00] RECOVERY - cp2 Puppet on cp2 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [17:48:43] PROBLEM - es2 Disk Space on es2 is CRITICAL: connect to address 168.235.110.49 port 5666: Connection refusedconnect to host 168.235.110.49 port 5666: Connection refused [17:48:58] PROBLEM - es2 Current Load on es2 is CRITICAL: connect to address 168.235.110.49 port 5666: Connection refusedconnect to host 168.235.110.49 port 5666: Connection refused [17:54:39] RECOVERY - es2 Disk Space on es2 is OK: DISK OK - free space: / 36922 MB (95% inode=97%); [17:54:48] [02miraheze/dns] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fjMAK [17:54:49] [02miraheze/dns] 07paladox 03fe4040a - Update miraheze.org [17:54:59] RECOVERY - es2 Current Load on es2 is OK: OK - load average: 0.97, 0.39, 0.17 [17:55:22] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fjMAX [17:55:23] [02miraheze/puppet] 07paladox 031acc27d - Update elasticsearch.pp [17:56:02] PROBLEM - es2 Puppet on es2 is UNKNOWN: UNKNOWN: Failed to check. Reason is: no_summary_file [18:00:03] RECOVERY - es2 Puppet on es2 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [18:33:56] PROBLEM - russopedia.info - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:35:11] PROBLEM - es2 Puppet on es2 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 2 minutes ago with 0 failures [18:38:01] RECOVERY - russopedia.info - LetsEncrypt on sslhost is OK: OK - Certificate 'russopedia.info' will expire on Sat 31 Aug 2019 03:18:26 PM GMT +0000. [18:43:39] PROBLEM - es1 Puppet on es1 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 10 minutes ago with 0 failures [18:45:52] PROBLEM - es4 Current Load on es4 is CRITICAL: CRITICAL - load average: 4.59, 4.15, 2.16 [18:46:37] !log resetting es cluster [18:46:42] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [18:46:49] PROBLEM - es3 Current Load on es3 is CRITICAL: CRITICAL - load average: 2.43, 4.06, 2.60