[02:24:03] [02mw-config] 07The-Voidwalker opened pull request 03#2641: configure uncyclopedia2wiki per T4070 - 13https://git.io/fhNyr [02:28:18] [02mw-config] 07paladox closed pull request 03#2641: configure uncyclopedia2wiki per T4070 - 13https://git.io/fhNyr [02:28:19] [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fhNy6 [02:28:21] [02miraheze/mw-config] 07The-Voidwalker 03cabdc9c - configure uncyclopedia2wiki per T4070 (#2641) [03:30:12] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fhNSC [03:30:13] [02miraheze/services] 07MirahezeSSLBot 033d1bf21 - BOT: Updating services config for wikis [04:14:01] PROBLEM - antiguabarbudacalypso.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:16:01] RECOVERY - antiguabarbudacalypso.com - LetsEncrypt on sslhost is OK: OK - Certificate 'antiguabarbudacalypso.com' will expire on Wed 15 May 2019 09:02:45 PM GMT +0000. [07:09:31] PROBLEM - antiguabarbudacalypso.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:13:39] RECOVERY - antiguabarbudacalypso.com - LetsEncrypt on sslhost is OK: OK - Certificate 'antiguabarbudacalypso.com' will expire on Wed 15 May 2019 09:02:45 PM GMT +0000. [08:30:14] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fhNQz [08:30:15] [02miraheze/services] 07MirahezeSSLBot 03e700e02 - BOT: Updating services config for wikis [10:21:11] PROBLEM - antiguabarbudacalypso.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:25:20] RECOVERY - antiguabarbudacalypso.com - LetsEncrypt on sslhost is OK: OK - Certificate 'antiguabarbudacalypso.com' will expire on Wed 15 May 2019 09:02:45 PM GMT +0000. [11:15:53] PROBLEM - antiguabarbudacalypso.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:17:55] RECOVERY - antiguabarbudacalypso.com - LetsEncrypt on sslhost is OK: OK - Certificate 'antiguabarbudacalypso.com' will expire on Wed 15 May 2019 09:02:45 PM GMT +0000. [12:20:51] o/ [12:25:42] PROBLEM - antiguabarbudacalypso.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:27:41] RECOVERY - antiguabarbudacalypso.com - LetsEncrypt on sslhost is OK: OK - Certificate 'antiguabarbudacalypso.com' will expire on Wed 15 May 2019 09:02:45 PM GMT +0000. [12:46:34] .tell Voidwalker the wiki create notify userscript isnt working for me anymore again no matter what I do [12:46:34] Zppix|mobile: I'll pass that on when Voidwalker is around. [14:12:55] hi [14:13:50] Hispano76: Hello [14:25:45] hi Zppix :) [14:30:19] [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fhNx8 [14:30:20] [02miraheze/mw-config] 07paladox 0353062cd - uncyclopedia2wiki: Set $wgAutoConfirmCount and $wgAutoConfirmAge [14:42:20] Zppix: CWN is deprecated in favor of echo notifications [14:55:11] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fhNpW [14:55:12] [02miraheze/services] 07MirahezeSSLBot 03ee3ee5f - BOT: Updating services config for wikis [14:59:54] MacFan4000: i know that now :P [15:03:33] [02miraheze/WikiDiscover] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fhNp2 [15:03:34] [02miraheze/WikiDiscover] 07paladox 03b9e9a4a - Convert array() to [] [15:14:21] PROBLEM - cp2 Varnish Backends on cp2 is CRITICAL: 3 backends are down. mw1 mw2 mw3 [15:14:27] PROBLEM - misc2 HTTPS on misc2 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 372 bytes in 0.010 second response time [15:14:33] PROBLEM - cp2 HTTP 4xx/5xx ERROR Rate on cp2 is WARNING: WARNING - NGINX Error Rate is 53% [15:14:33] PROBLEM - cp5 Varnish Backends on cp5 is CRITICAL: 3 backends are down. mw1 mw2 mw3 [15:14:42] PROBLEM - misc1 webmail.miraheze.org HTTPS on misc1 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:14:44] PROBLEM - cp4 HTTP 4xx/5xx ERROR Rate on cp4 is CRITICAL: CRITICAL - NGINX Error Rate is 86% [15:14:55] that looks significant [15:14:55] Voidwalker: 2019-02-28 - 12:46:34UTC tell Voidwalker the wiki create notify userscript isnt working for me anymore again no matter what I do [15:14:58] PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 6 datacenters are down: 107.191.126.23/cpweb, 2604:180:0:33b::2/cpweb, 81.4.109.133/cpweb, 2a00:d880:5:8ea::ebc7/cpweb, 172.104.111.8/cpweb, 2400:8902::f03c:91ff:fe07:444e/cpweb [15:15:05] Voidwalker: ignore that msg [15:15:07] PROBLEM - cp5 HTTP 4xx/5xx ERROR Rate on cp5 is CRITICAL: CRITICAL - NGINX Error Rate is 65% [15:15:12] yeah, I read the logs :) [15:15:16] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 6 datacenters are down: 107.191.126.23/cpweb, 2604:180:0:33b::2/cpweb, 81.4.109.133/cpweb, 2a00:d880:5:8ea::ebc7/cpweb, 172.104.111.8/cpweb, 2400:8902::f03c:91ff:fe07:444e/cpweb [15:15:23] PROBLEM - cp2 HTTPS on cp2 is CRITICAL: HTTP CRITICAL: HTTP/1.1 503 Backend fetch failed - 3838 bytes in 0.398 second response time [15:15:37] PROBLEM - cp4 Varnish Backends on cp4 is CRITICAL: 3 backends are down. mw1 mw2 mw3 [15:15:40] PROBLEM - cp4 HTTPS on cp4 is CRITICAL: HTTP CRITICAL: HTTP/1.1 503 Backend fetch failed - 3838 bytes in 0.011 second response time [15:16:16] ruh roe [15:16:18] paladox, ^ [15:16:26] hmm [15:16:30] * paladox looks [15:16:33] PROBLEM - cp2 HTTP 4xx/5xx ERROR Rate on cp2 is CRITICAL: CRITICAL - NGINX Error Rate is 77% [15:17:39] PROBLEM - cp5 HTTPS on cp5 is CRITICAL: HTTP CRITICAL: HTTP/1.1 503 Backend fetch failed - 3851 bytes in 1.033 second response time [15:17:42] oh [15:17:44] it's the db [15:18:24] oh its the db he says calmly [15:18:59] the db ran out of space :( [15:19:36] paladox: pssh space is for amateurs [15:19:55] !log stopping mysql (to be able to remove the binary logs manually) [15:20:07] icingaweb is down because of disk space too [15:20:13] yup [15:20:29] though icinga should still be able to monitor, just the UI will be down [15:21:24] RECOVERY - cp2 HTTPS on cp2 is OK: HTTP OK: HTTP/1.1 200 OK - 23765 bytes in 0.498 second response time [15:22:08] PROBLEM - misc4 phabricator.miraheze.org HTTPS on misc4 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 4194 bytes in 0.029 second response time [15:22:33] RECOVERY - cp2 HTTP 4xx/5xx ERROR Rate on cp2 is OK: OK - NGINX Error Rate is 35% [15:23:23] PROBLEM - db4 MySQL on db4 is CRITICAL: Can't connect to MySQL server on '81.4.109.166' (111 "Connection refused") [15:23:49] PROBLEM - mw3 JobQueue on mw3 is CRITICAL: JOBQUEUE CRITICAL - check plugin (check_jobqueue) or PHP errors [15:23:53] PROBLEM - misc4 phab.miraheze.wiki HTTPS on misc4 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/1.1 500 Internal Server Error [15:24:38] PROBLEM - misc1 Puppet on misc1 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[idomysql-import-schema] [15:25:23] PROBLEM - cp2 HTTPS on cp2 is CRITICAL: HTTP CRITICAL: HTTP/1.1 503 Backend fetch failed - 3838 bytes in 0.397 second response time [15:25:56] !log rm mysql-bin.0* on db4 [15:26:33] PROBLEM - cp2 HTTP 4xx/5xx ERROR Rate on cp2 is CRITICAL: CRITICAL - NGINX Error Rate is 83% [15:26:41] PROBLEM - db4 Disk Space on db4 is WARNING: DISK WARNING - free space: / 49395 MB (13% inode=96%); [15:31:01] so, the logs were ~49 GB? [15:31:07] RECOVERY - cp5 HTTP 4xx/5xx ERROR Rate on cp5 is OK: OK - NGINX Error Rate is 35% [15:32:20] Voidwalker appears so [15:32:46] that's a bit much, doncha think? [15:33:34] nope, seeing as there was 5 days worth of bin logs (at 1.1gb each) [15:33:47] though we should cut it so we store less bin logs for now [15:34:27] * 6 days [15:35:07] PROBLEM - cp5 HTTP 4xx/5xx ERROR Rate on cp5 is CRITICAL: CRITICAL - NGINX Error Rate is 85% [15:41:07] PROBLEM - cp5 HTTP 4xx/5xx ERROR Rate on cp5 is WARNING: WARNING - NGINX Error Rate is 56% [15:41:23] RECOVERY - db4 MySQL on db4 is OK: Uptime: 107 Threads: 55 Questions: 8545 Slow queries: 0 Opens: 887 Flush tables: 1 Open tables: 800 Queries per second avg: 79.859 [15:41:24] RECOVERY - cp2 HTTPS on cp2 is OK: HTTP OK: HTTP/1.1 200 OK - 23754 bytes in 0.597 second response time [15:41:37] RECOVERY - cp4 Varnish Backends on cp4 is OK: All 5 backends are healthy [15:41:37] RECOVERY - cp5 HTTPS on cp5 is OK: HTTP OK: HTTP/1.1 200 OK - 23754 bytes in 2.194 second response time [15:41:40] RECOVERY - cp4 HTTPS on cp4 is OK: HTTP OK: HTTP/1.1 200 OK - 23731 bytes in 0.044 second response time [15:41:49] RECOVERY - mw3 JobQueue on mw3 is OK: JOBQUEUE OK - job queue below 300 jobs [15:41:54] RECOVERY - misc4 phab.miraheze.wiki HTTPS on misc4 is OK: HTTP OK: Status line output matched "HTTP/1.1 200" - 17648 bytes in 0.214 second response time [15:42:09] RECOVERY - misc4 phabricator.miraheze.org HTTPS on misc4 is OK: HTTP OK: HTTP/1.1 200 OK - 18361 bytes in 0.111 second response time [15:42:21] RECOVERY - cp2 Varnish Backends on cp2 is OK: All 5 backends are healthy [15:42:24] RECOVERY - misc2 HTTPS on misc2 is OK: HTTP OK: HTTP/1.1 200 OK - 45587 bytes in 0.147 second response time [15:42:29] !log [15:19:55] <+paladox> !log stopping mysql (to be able to remove the binary logs manually) [15:42:33] RECOVERY - cp2 HTTP 4xx/5xx ERROR Rate on cp2 is OK: OK - NGINX Error Rate is 16% [15:42:33] RECOVERY - cp5 Varnish Backends on cp5 is OK: All 5 backends are healthy [15:42:34] RECOVERY - misc1 webmail.miraheze.org HTTPS on misc1 is OK: HTTP OK: Status line output matched "HTTP/1.1 401 Unauthorized" - 5681 bytes in 0.029 second response time [15:42:36] !log [15:25:56] <+paladox> !log rm mysql-bin.0* on db4 [15:42:37] RECOVERY - misc1 Puppet on misc1 is OK: OK: Puppet is currently enabled, last run 56 seconds ago with 0 failures [15:42:44] RECOVERY - cp4 HTTP 4xx/5xx ERROR Rate on cp4 is OK: OK - NGINX Error Rate is 1% [15:42:58] RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online [15:43:08] RECOVERY - cp5 HTTP 4xx/5xx ERROR Rate on cp5 is OK: OK - NGINX Error Rate is 2% [15:43:16] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [15:46:16] 50 3 [15:46:33] work [15:50:26] !log started import for frikipediawiki [16:22:27] [02miraheze/IncidentReporting] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fhNjQ [16:22:29] [02miraheze/IncidentReporting] 07paladox 0360ff350 - Add missing $this->checkPermissions() Per @Void who noticed this. [16:23:40] [02miraheze/mediawiki] 07paladox pushed 031 commit to 03REL1_32 [+0/-0/±1] 13https://git.io/fhNjN [16:23:41] [02miraheze/mediawiki] 07paladox 0393f2618 - Update IncidentReporting [16:26:03] paladox: you should probs run LC, since that included i18n changes [16:26:10] ok [16:27:03] !log run lc on mw* [16:27:07] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [16:27:14] !log [15:19:55] <+paladox> !log stopping mysql (to be able to remove the binary logs manually) [16:27:17] !log [15:25:56] <+paladox> !log rm mysql-bin.0* on db4 [16:27:18] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [16:27:22] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [16:34:39] hi [16:34:47] hi [16:42:57] [02miraheze/IncidentReporting] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fhAeD [16:42:59] [02miraheze/IncidentReporting] 07paladox 0379f5d7d - fix "count(): Parameter must be an array or an object that implements Countable" [16:44:05] out of curiosity, there is news about ManageWiki [16:44:07] ?* [16:47:34] Hm? [16:49:53] !log upgrade phabricator [16:49:58] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [17:04:04] [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fhAvu [17:04:06] [02miraheze/mw-config] 07paladox 03ebc0ec2 - Compress beidipediawiki [17:05:10] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fhAvz [17:05:12] [02miraheze/services] 07MirahezeSSLBot 030c4c5d1 - BOT: Updating services config for wikis [17:06:37] !log import beidipedia dump [17:06:41] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [18:09:14] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fhAJT [18:09:15] [02miraheze/puppet] 07paladox 03a711edc - Backup pwikiwiki monthly per T4157 [21:27:05] PROBLEM - permanentfuturelab.wiki - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:29:00] RECOVERY - permanentfuturelab.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'permanentfuturelab.wiki' will expire on Tue 07 May 2019 10:15:37 PM GMT +0000. [23:06:58] PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 2 datacenters are down: 107.191.126.23/cpweb, 2604:180:0:33b::2/cpweb [23:08:58] RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online [23:23:48] [02miraheze/IncidentReporting] 07JohnFLewis pushed 031 commit to 03master [+0/-0/±4] 13https://git.io/fhALu [23:23:49] [02miraheze/IncidentReporting] 07JohnFLewis 03371db84 - push 1.0.2