[00:00:40] PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 6 datacenters are down: 107.191.126.23/cpweb, 2604:180:0:33b::2/cpweb, 81.4.109.133/cpweb, 2a00:d880:5:8ea::ebc7/cpweb, 172.104.111.8/cpweb, 2400:8902::f03c:91ff:fe07:444e/cpweb [00:01:06] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 6 datacenters are down: 107.191.126.23/cpweb, 2604:180:0:33b::2/cpweb, 81.4.109.133/cpweb, 2a00:d880:5:8ea::ebc7/cpweb, 172.104.111.8/cpweb, 2400:8902::f03c:91ff:fe07:444e/cpweb [00:01:14] 503 [00:01:16] PROBLEM - cp2 HTTP 4xx/5xx ERROR Rate on cp2 is CRITICAL: CRITICAL - NGINX Error Rate is 85% [00:01:56] Oh the db [00:02:00] JohnLewis: ^^ [00:02:33] Hmm not the db [00:02:36] cant you resolve? [00:02:37] 16gb left [00:02:42] huh [00:02:44] RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online [00:03:01] ooh hm [00:03:10] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [00:03:16] RECOVERY - cp2 HTTP 4xx/5xx ERROR Rate on cp2 is OK: OK - NGINX Error Rate is 13% [00:03:22] PROBLEM - puppet1 Puppet on puppet1 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [00:03:34] PROBLEM - misc1 Puppet on misc1 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [00:03:46] recovered on it's own JohnLewis [00:03:56] PROBLEM - cp5 HTTP 4xx/5xx ERROR Rate on cp5 is WARNING: WARNING - NGINX Error Rate is 46% [00:05:56] RECOVERY - cp5 HTTP 4xx/5xx ERROR Rate on cp5 is OK: OK - NGINX Error Rate is 0% [00:06:00] should look at why [00:06:13] because all three going down at the same time is bad [00:11:22] RECOVERY - puppet1 Puppet on puppet1 is OK: OK: Puppet is currently enabled, last run 9 seconds ago with 0 failures [00:13:34] RECOVERY - misc1 Puppet on misc1 is OK: OK: Puppet is currently enabled, last run 42 seconds ago with 0 failures [00:45:15] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fxiuv [00:45:17] [02miraheze/services] 07MirahezeSSLBot 0305ca661 - BOT: Updating services config for wikis [03:18:48] Hi! Here is the list of currently open high priority tasks on Phabricator [03:18:55] No updates for 28 days - https://phabricator.miraheze.org/T3649 - Fix varnish / nginx to be able to stream videos - authored by Paladox, assigned to None [06:34:21] PROBLEM - misc2 Puppet on misc2 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Package[php7.2-gettext] [06:42:21] RECOVERY - misc2 Puppet on misc2 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [06:53:30] PROBLEM - mw3 JobQueue on mw3 is CRITICAL: JOBQUEUE CRITICAL - job queue greater than 300 jobs. Current queue: 2915 [08:53:29] RECOVERY - mw3 JobQueue on mw3 is OK: JOBQUEUE OK - job queue below 300 jobs [09:07:46] PROBLEM - tensegritywiki.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'tensegritywiki.com' expires in 15 day(s) (Thu 08 Nov 2018 09:04:54 AM GMT +0000). [09:07:59] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fxipm [09:08:00] [02miraheze/ssl] 07MirahezeSSLBot 03f1fc417 - Bot: Update SSL cert for tensegritywiki.com [09:11:47] RECOVERY - tensegritywiki.com - LetsEncrypt on sslhost is OK: OK - Certificate 'tensegritywiki.com' will expire on Mon 21 Jan 2019 08:07:53 AM GMT +0000. [12:50:14] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fxPcu [12:50:15] [02miraheze/services] 07MirahezeSSLBot 03051172f - BOT: Updating services config for wikis [13:39:32] !log sudo -u www-data php /srv/mediawiki/w/maintenance/importDump.php --wiki=americangirldollswiki --report=1 American+Girl+Wiki-20181021003500.xml on mw1 ref T3716 [13:39:36] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [13:41:28] !log sudo -u www-data php /srv/mediawiki/w/maintenance/importDump.php --wiki=americangirldollswiki --report=1 American+Girl+Wiki-20181021003633.xml on mw1 ref T3716 [13:41:32] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [13:41:56] !log sudo -u www-data php /srv/mediawiki/w/maintenance/importDump.php --wiki=americangirldollswiki --report=1 American+Girl+Wiki-20181021003912.xml on mw1 ref T3716 [13:42:01] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [14:45:03] [02miraheze/mediawiki] 07paladox pushed 031 commit to 03REL1_32 [+0/-0/±1] 13https://git.io/fxPao [14:45:04] [02miraheze/mediawiki] 07paladox 03ed62d4e - Update AbuseFilter [14:46:31] [02miraheze/mediawiki] 07paladox pushed 031 commit to 03REL1_31 [+0/-0/±1] 13https://git.io/fxPaN [14:46:33] [02miraheze/mediawiki] 07paladox 03bd26268 - Update AbuseFilter [15:05:30] PROBLEM - mw3 JobQueue on mw3 is CRITICAL: JOBQUEUE CRITICAL - job queue greater than 300 jobs. Current queue: 3303 [15:22:11] PROBLEM - wiki.lbcomms.co.za - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:37:16] RECOVERY - wiki.lbcomms.co.za - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.lbcomms.co.za' will expire on Tue 04 Dec 2018 02:51:00 PM GMT +0000. [17:31:29] RECOVERY - mw3 JobQueue on mw3 is OK: JOBQUEUE OK - job queue below 300 jobs [18:05:31] PROBLEM - mw3 JobQueue on mw3 is CRITICAL: JOBQUEUE CRITICAL - job queue greater than 300 jobs. Current queue: 3018 [19:09:29] RECOVERY - mw3 JobQueue on mw3 is OK: JOBQUEUE OK - job queue below 300 jobs [22:11:31] PROBLEM - mw3 JobQueue on mw3 is CRITICAL: JOBQUEUE CRITICAL - job queue greater than 300 jobs. Current queue: 5924