[00:00:26] RECOVERY - misc3 Puppet on misc3 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [00:00:35] RECOVERY - misc4 HTTP on misc4 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 421 bytes in 0.002 second response time [00:01:16] PROBLEM - test1 HTTP on test1 is WARNING: HTTP WARNING: HTTP/1.1 404 Not Found - 319 bytes in 0.002 second response time [00:03:54] PROBLEM - bacula1 Puppet on bacula1 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [00:04:12] PROBLEM - cp2 Puppet on cp2 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [00:11:52] RECOVERY - bacula1 Puppet on bacula1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [00:11:58] RECOVERY - cp2 Puppet on cp2 is OK: OK: Puppet is currently enabled, last run 42 seconds ago with 0 failures [00:57:30] !log git checkout 632cafec8 due to a error on the master branch that needs fixing (phabricator (misc4)) [00:57:35] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [01:19:33] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fAnvk [01:19:34] [02miraheze/puppet] 07paladox 034575f36 - Add phabricator.miraheze.org http checks [01:20:21] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fAnvt [01:20:23] [02miraheze/puppet] 07paladox 03da2a312 - Add lizard.miraheze.org icinga http checks [01:26:07] PROBLEM - misc4 lizard.miraheze.org HTTPS on misc4 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 2280 bytes in 0.093 second response time [01:26:09] PROBLEM - misc4 phabricator.miraheze.org HTTPS on misc4 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 2280 bytes in 0.087 second response time [01:33:08] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fAnv2 [01:33:09] [02miraheze/puppet] 07paladox 03afff04b - Remove http icinga check [01:34:20] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fAnvV [01:34:22] [02miraheze/puppet] 07paladox 03c109a65 - Update init.pp [01:34:41] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fAnvr [01:34:43] [02miraheze/puppet] 07paladox 03a7bcd15 - Update web.pp [01:38:29] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fAnv9 [01:38:31] [02miraheze/puppet] 07paladox 034ec4b90 - httpd: Remove http icinga check [01:39:40] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fAnv7 [01:39:42] [02miraheze/puppet] 07paladox 0329459ae - Update web.pp [01:39:51] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fAnv5 [01:39:52] [02miraheze/puppet] 07paladox 03a68fb63 - Update web.pp [01:41:55] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fAnvF [01:41:57] [02miraheze/puppet] 07paladox 033488b71 - Update init.pp [01:46:38] PROBLEM - misc4 HTTPS on misc4 is WARNING: HTTP WARNING: HTTP/1.1 401 Unauthorized - 397 bytes in 0.018 second response time [04:03:01] PROBLEM - misc4 HTTPS on misc4 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:04:09] PROBLEM - cp4 Disk Space on cp4 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [04:04:29] PROBLEM - misc4 Puppet on misc4 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [04:04:35] PROBLEM - cp4 Varnish Backends on cp4 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [04:04:39] PROBLEM - cp5 Varnish Backends on cp5 is CRITICAL: 3 backends are down. mw1 mw2 mw3 [04:04:49] PROBLEM - cp4 Puppet on cp4 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [04:04:51] PROBLEM - cp4 Current Load on cp4 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [04:04:59] PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 6 datacenters are down: 107.191.126.23/cpweb, 2604:180:0:33b::2/cpweb, 81.4.109.133/cpweb, 2a00:d880:5:8ea::ebc7/cpweb, 172.104.111.8/cpweb, 2400:8902::f03c:91ff:fe07:444e/cpweb [04:05:07] PROBLEM - misc4 phd on misc4 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [04:05:17] PROBLEM - test1 MediaWiki Rendering on test1 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:06:13] PROBLEM - misc4 Current Load on misc4 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [04:06:21] PROBLEM - cp4 HTTP 4xx/5xx ERROR Rate on cp4 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [04:06:23] PROBLEM - cp4 HTTPS on cp4 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:06:43] PROBLEM - cp2 HTTP 4xx/5xx ERROR Rate on cp2 is CRITICAL: CRITICAL - NGINX Error Rate is 66% [04:06:53] PROBLEM - misc4 Disk Space on misc4 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [04:07:01] PROBLEM - studynotekr.miraheze.org - GlobalSign on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:07:15] RECOVERY - misc4 SSH on misc4 is OK: SSH OK - OpenSSH_7.4p1 Debian-10+deb9u4 (protocol 2.0) [04:07:22] JohnLewis: Miraheze is 503ing [04:07:47] PROBLEM - hellointernet.miraheze.org - GlobalSign on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:07:59] PROBLEM - cp5 HTTP 4xx/5xx ERROR Rate on cp5 is WARNING: WARNING - NGINX Error Rate is 52% [04:08:20] Any sysadmins about? [04:08:23] RECOVERY - cp4 HTTPS on cp4 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 1126 bytes in 4.178 second response time [04:08:57] PROBLEM - studynotekr.miraheze.org - GlobalSign on sslhost is WARNING: WARNING - Certificate '*.miraheze.org' expires in 23 day(s) (Sat 22 Sep 2018 08:12:13 PM GMT +0000). [04:09:27] PROBLEM - mw1 MediaWiki Rendering on mw1 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:09:45] PROBLEM - test1 Puppet on test1 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [04:09:49] PROBLEM - hellointernet.miraheze.org - GlobalSign on sslhost is WARNING: WARNING - Certificate '*.miraheze.org' expires in 23 day(s) (Sat 22 Sep 2018 08:12:13 PM GMT +0000). [04:10:09] PROBLEM - test1 Current Load on test1 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [04:10:15] PROBLEM - bacula1 Bacula Phabricator Static on bacula1 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [04:10:29] PROBLEM - mw2 MediaWiki Rendering on mw2 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:10:35] PROBLEM - wc.miraheze.org on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:10:49] RECOVERY - misc4 Puppet on misc4 is OK: OK: Puppet is currently enabled, last run 9 minutes ago with 0 failures [04:10:56] JohnLewis: MacFan4000: labster: paladox ^ [04:11:15] PROBLEM - test1 HTTPS on test1 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:11:49] RECOVERY - test1 Puppet on test1 is OK: OK: Puppet is currently enabled, last run 10 minutes ago with 0 failures [04:12:30] PROBLEM - wc.miraheze.org on sslhost is WARNING: WARNING - Certificate '*.miraheze.org' expires in 23 day(s) (Sat 22 Sep 2018 08:12:13 PM GMT +0000). [04:12:32] RECOVERY - mw2 MediaWiki Rendering on mw2 is OK: HTTP OK: HTTP/1.1 200 OK - 30846 bytes in 5.147 second response time [04:12:46] PROBLEM - mw3 MediaWiki Rendering on mw3 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:13:34] PROBLEM - misc4 SSH on misc4 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:13:58] RECOVERY - cp5 HTTP 4xx/5xx ERROR Rate on cp5 is OK: OK - NGINX Error Rate is 38% [04:14:12] RECOVERY - test1 Current Load on test1 is OK: OK - load average: 0.00, 0.05, 0.09 [04:14:20] RECOVERY - misc4 Current Load on misc4 is OK: OK - load average: 0.16, 0.11, 0.21 [04:14:32] RECOVERY - cp4 Disk Space on cp4 is OK: DISK OK - free space: / 22881 MB (55% inode=99%); [04:14:44] RECOVERY - mw3 MediaWiki Rendering on mw3 is OK: HTTP OK: HTTP/1.1 200 OK - 30847 bytes in 0.041 second response time [04:15:10] RECOVERY - misc4 Disk Space on misc4 is OK: DISK OK - free space: / 53492 MB (87% inode=99%); [04:15:12] RECOVERY - cp4 Current Load on cp4 is OK: OK - load average: 0.03, 0.06, 0.07 [04:15:16] RECOVERY - cp2 Varnish Backends on cp2 is OK: All 4 backends are healthy [04:15:18] PROBLEM - misc4 HTTPS on misc4 is WARNING: HTTP WARNING: HTTP/1.1 401 Unauthorized - 397 bytes in 0.016 second response time [04:15:20] RECOVERY - test1 HTTPS on test1 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 459 bytes in 0.024 second response time [04:15:30] RECOVERY - misc4 SSH on misc4 is OK: SSH OK - OpenSSH_7.4p1 Debian-10+deb9u4 (protocol 2.0) [04:15:40] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [04:15:44] RECOVERY - mw1 MediaWiki Rendering on mw1 is OK: HTTP OK: HTTP/1.1 200 OK - 30846 bytes in 0.029 second response time [04:15:52] RECOVERY - test1 MediaWiki Rendering on test1 is OK: HTTP OK: HTTP/1.1 200 OK - 30847 bytes in 0.032 second response time [04:16:16] Cyberpower678: physical server down it seemed [04:16:26] RECOVERY - cp4 HTTP 4xx/5xx ERROR Rate on cp4 is OK: OK - NGINX Error Rate is 20% [04:16:40] RECOVERY - cp5 Varnish Backends on cp5 is OK: All 4 backends are healthy [04:16:44] RECOVERY - cp2 HTTP 4xx/5xx ERROR Rate on cp2 is OK: OK - NGINX Error Rate is 3% [04:16:46] RECOVERY - cp4 Varnish Backends on cp4 is OK: All 4 backends are healthy [04:16:48] JohnLewis: ah [04:16:57] Running OAuth experiments on Miraheze [04:17:16] And noticed everything went boom on my end unexpectedly. :p [04:17:18] RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online [04:17:37] ironically on the only non redundant bit of service [04:21:10] RECOVERY - cp4 Puppet on cp4 is OK: OK: Puppet is currently enabled, last run 17 seconds ago with 0 failures [04:24:32] PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 6 datacenters are down: 107.191.126.23/cpweb, 2604:180:0:33b::2/cpweb, 81.4.109.133/cpweb, 2a00:d880:5:8ea::ebc7/cpweb, 172.104.111.8/cpweb, 2400:8902::f03c:91ff:fe07:444e/cpweb [04:25:16] PROBLEM - cp2 Varnish Backends on cp2 is CRITICAL: 1 backends are down. mw1 [04:25:26] PROBLEM - cp4 Varnish Backends on cp4 is CRITICAL: 2 backends are down. mw1 mw2 [04:25:28] PROBLEM - cp5 Varnish Backends on cp5 is CRITICAL: 3 backends are down. mw1 mw2 mw3 [04:26:14] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 6 datacenters are down: 107.191.126.23/cpweb, 2604:180:0:33b::2/cpweb, 81.4.109.133/cpweb, 2a00:d880:5:8ea::ebc7/cpweb, 172.104.111.8/cpweb, 2400:8902::f03c:91ff:fe07:444e/cpweb [04:31:11] RECOVERY - cp4 Varnish Backends on cp4 is OK: All 4 backends are healthy [04:31:17] RECOVERY - cp2 Varnish Backends on cp2 is OK: All 4 backends are healthy [04:31:21] RECOVERY - cp5 Varnish Backends on cp5 is OK: All 4 backends are healthy [04:32:29] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [04:32:45] RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online [06:25:14] !log reception@mw3:/srv/mediawiki/w/maintenance$ sudo -u www-data php importDump.php --wiki electowikiwiki /home/reception/Electowiki-20180826232629_ovr.xml [06:25:20] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [09:33:15] revi: hey https://phabricator.miraheze.org/T3533 [09:33:16] Title: [ ⚓ T3533 Add MirahezeMagic to translatewiki.net ] - phabricator.miraheze.org [10:04:11] paladox: hi. around? [10:06:48] Reception123: semi around? [10:06:57] paladox: good enough for https://phabricator.miraheze.org/T3530 I guess :P [10:06:58] Title: [ ⚓ T3530 PDFHandler doesn't appear to do anything ] - phabricator.miraheze.org [10:08:03] Hmm? [10:08:15] I saw that bot not sure why it is failing for them [10:09:23] paladox: I don't know why but I can't log into that wiki without clearing cookies [10:09:28] because it's a custom domain [10:10:06] Reception123: did you try relogging into it before clearing the cookies? [10:10:16] paladox: 3 times, it keeps denying me [10:10:47] Oh ok [10:11:01] Reception123: it should work I guess your just a special case heh :) [10:11:28] I'm afraid I'm probably not :( [10:12:09] Reception123: it works now logging in after clearing cookies right? [10:12:37] paladox: yes [10:12:48] Ok [10:12:59] paladox: Oh wait I thought it was but then [10:13:01] "Auto-creation of a local account failed: Automatic account creation is not allowed." [10:13:11] hmmmm [10:13:56] Hmm [10:25:26] PROBLEM - mw3 JobQueue on mw3 is CRITICAL: JOBQUEUE CRITICAL - job queue greater than 300 jobs. Current queue: 3005 [10:26:53] yay [10:27:18] !log sudo -u www-data foreachwikiindblist /srv/mediawiki/dblist/all.dblist /srv/mediawiki/w/maintenance/runJobs.php on mw1 [10:27:22] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [10:43:01] PROBLEM - studynotekr.miraheze.org - GlobalSign on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:43:53] PROBLEM - test1 Puppet on test1 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [10:44:05] PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 6 datacenters are down: 107.191.126.23/cpweb, 2604:180:0:33b::2/cpweb, 81.4.109.133/cpweb, 2a00:d880:5:8ea::ebc7/cpweb, 172.104.111.8/cpweb, 2400:8902::f03c:91ff:fe07:444e/cpweb [10:44:09] PROBLEM - cp4 Disk Space on cp4 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [10:44:29] PROBLEM - misc4 Puppet on misc4 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [10:44:35] PROBLEM - papelor.io - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:44:39] PROBLEM - cp5 Varnish Backends on cp5 is CRITICAL: 3 backends are down. mw1 mw2 mw3 [10:44:51] PROBLEM - misc4 Disk Space on misc4 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [10:45:01] PROBLEM - studynotekr.miraheze.org - GlobalSign on sslhost is WARNING: WARNING - Certificate '*.miraheze.org' expires in 23 day(s) (Sat 22 Sep 2018 08:12:13 PM GMT +0000). [10:45:07] PROBLEM - wiki.make717.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:45:09] PROBLEM - mw3 HTTPS on mw3 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:45:17] PROBLEM - misc4 SSH on misc4 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:45:49] PROBLEM - cp5 HTTP 4xx/5xx ERROR Rate on cp5 is CRITICAL: CRITICAL - NGINX Error Rate is 63% [10:46:11] PROBLEM - bacula1 Bacula Phabricator Static on bacula1 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [10:46:15] PROBLEM - mw1 MediaWiki Rendering on mw1 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:46:21] PROBLEM - cp4 HTTPS on cp4 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:46:25] PROBLEM - cp4 HTTP 4xx/5xx ERROR Rate on cp4 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [10:46:27] PROBLEM - bacula1 Bacula Misc4 Lizardfs Master on bacula1 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [10:46:35] RECOVERY - papelor.io - LetsEncrypt on sslhost is OK: OK - Certificate 'papelor.io' will expire on Fri 12 Oct 2018 06:42:54 PM GMT +0000. [10:46:43] PROBLEM - cp2 HTTP 4xx/5xx ERROR Rate on cp2 is CRITICAL: CRITICAL - NGINX Error Rate is 75% [10:47:01] RECOVERY - wiki.make717.org - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.make717.org' will expire on Sun 21 Oct 2018 10:01:39 AM GMT +0000. [10:47:15] RECOVERY - misc4 SSH on misc4 is OK: SSH OK - OpenSSH_7.4p1 Debian-10+deb9u4 (protocol 2.0) [10:47:23] PROBLEM - misc4 Current Load on misc4 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [10:47:27] PROBLEM - mw3 MediaWiki Rendering on mw3 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:47:29] PROBLEM - unmade.miraheze.org - GlobalSign on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:47:57] RECOVERY - test1 Puppet on test1 is OK: OK: Puppet is currently enabled, last run 6 seconds ago with 0 failures [10:48:17] PROBLEM - test1 MediaWiki Rendering on test1 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:48:21] PROBLEM - test1 HTTPS on test1 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:48:23] RECOVERY - mw1 MediaWiki Rendering on mw1 is OK: HTTP OK: HTTP/1.1 200 OK - 30847 bytes in 9.050 second response time [10:48:27] RECOVERY - bacula1 Bacula Misc4 Lizardfs Master on bacula1 is OK: OK: Full, 16 files, 120.1MB, 2018-08-26 00:05:00 (4.4 days ago) [10:48:37] RECOVERY - misc4 Puppet on misc4 is OK: OK: Puppet is currently enabled, last run 7 minutes ago with 0 failures [10:48:53] RECOVERY - cp4 Puppet on cp4 is OK: OK: Puppet is currently enabled, last run 7 minutes ago with 0 failures [10:48:55] RECOVERY - misc4 Disk Space on misc4 is OK: DISK OK - free space: / 53512 MB (87% inode=99%); [10:49:03] PROBLEM - misc4 HTTPS on misc4 is WARNING: HTTP WARNING: HTTP/1.1 401 Unauthorized - 397 bytes in 0.012 second response time [10:49:07] RECOVERY - mw3 HTTPS on mw3 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 459 bytes in 0.021 second response time [10:49:15] RECOVERY - misc4 Current Load on misc4 is OK: OK - load average: 0.09, 0.13, 0.20 [10:49:17] RECOVERY - cp2 Varnish Backends on cp2 is OK: All 4 backends are healthy [10:49:31] PROBLEM - unmade.miraheze.org - GlobalSign on sslhost is WARNING: WARNING - Certificate '*.miraheze.org' expires in 23 day(s) (Sat 22 Sep 2018 08:12:13 PM GMT +0000). [10:49:49] RECOVERY - cp5 HTTP 4xx/5xx ERROR Rate on cp5 is OK: OK - NGINX Error Rate is 39% [10:50:13] RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online [10:50:16] RECOVERY - cp4 HTTP 4xx/5xx ERROR Rate on cp4 is OK: OK - NGINX Error Rate is 17% [10:50:17] RECOVERY - test1 MediaWiki Rendering on test1 is OK: HTTP OK: HTTP/1.1 200 OK - 30848 bytes in 0.095 second response time [10:50:19] RECOVERY - cp4 Disk Space on cp4 is OK: DISK OK - free space: / 23165 MB (56% inode=99%); [10:50:23] what is going on [10:50:23] RECOVERY - cp4 HTTPS on cp4 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 1127 bytes in 0.031 second response time [10:50:31] RECOVERY - cp4 Varnish Backends on cp4 is OK: All 4 backends are healthy [10:50:39] RECOVERY - cp5 Varnish Backends on cp5 is OK: All 4 backends are healthy [10:50:43] RECOVERY - cp2 HTTP 4xx/5xx ERROR Rate on cp2 is OK: OK - NGINX Error Rate is 4% [10:51:15] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [10:51:31] RECOVERY - mw3 MediaWiki Rendering on mw3 is OK: HTTP OK: HTTP/1.1 200 OK - 30847 bytes in 0.044 second response time [11:19:45] RECOVERY - mw3 JobQueue on mw3 is OK: JOBQUEUE OK - job queue below 300 jobs [11:42:43] [02miraheze/MatomoAnalytics] 07MacFan4000 pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fAnKQ [11:42:45] [02miraheze/MatomoAnalytics] 07MacFan4000 03a095905 - When global ID is false it should be a string Otherwise it causes a JS error [11:43:18] Err [11:46:13] it sounds like nodes keep getting rebooted [11:46:39] and thus icinga goes crazy [11:47:53] Yep they are being rebooted for security reasons [11:55:14] [02mw-config] 07MacFan4000 closed pull request 03#2409: Update LocalSettings.php - 13https://git.io/fAZ2g [11:57:12] [02miraheze/MatomoAnalytics] 07MacFan4000 pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fAniJ [11:57:14] [02miraheze/MatomoAnalytics] 07MacFan4000 035c76ddd - revert [12:11:15] PROBLEM - misc4 HTTPS on misc4 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 2280 bytes in 0.092 second response time [12:14:00] paladox: why is misc4 https critical? [12:14:18] In icinga [12:14:20] Because https://misc4.miraheze.org [12:14:21] Title: [ Site Not Found ] - misc4.miraheze.org [12:14:22] is a 500 [12:14:50] So we have to remove the check? [12:14:58] Nope we have to fix it [12:15:04] How? [12:15:16] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fAnPD [12:15:18] [02miraheze/services] 07MirahezeSSLBot 03f860c71 - BOT: Updating services config for wikis [12:15:21] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fAnPy [12:15:22] [02miraheze/puppet] 07paladox 038093e5a - Update phabricator.miraheze.org.conf [12:16:59] PROBLEM - misc4 HTTPS on misc4 is WARNING: HTTP WARNING: HTTP/1.1 401 Unauthorized - 397 bytes in 0.019 second response time [12:17:44] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fAnXv [12:17:46] [02miraheze/puppet] 07paladox 031f73243 - Update site.pp [12:18:08] [02mw-config] 07paladox deleted branch 03MacFan4000-patch-1 - 13https://git.io/vbvb3 [12:18:10] [02miraheze/mw-config] 07paladox deleted branch 03MacFan4000-patch-1 [12:18:18] [02mw-config] 07paladox deleted branch 03Southparkfan-patch-1 - 13https://git.io/vbvb3 [12:18:20] [02miraheze/mw-config] 07paladox deleted branch 03Southparkfan-patch-1 [12:18:21] [02mw-config] 07paladox deleted branch 03revert-2410-Southparkfan-patch-1 - 13https://git.io/vbvb3 [12:18:23] [02miraheze/mw-config] 07paladox deleted branch 03revert-2410-Southparkfan-patch-1 [12:18:24] [02mw-config] 07paladox deleted branch 03revert-2411-revert-2410-Southparkfan-patch-1 - 13https://git.io/vbvb3 [12:18:26] [02miraheze/mw-config] 07paladox deleted branch 03revert-2411-revert-2410-Southparkfan-patch-1 [12:18:53] i think we are ready to remove icinga1 support now [12:20:16] Why is the https 401 unauthorized now? [12:20:31] becuase it's now defaulting to lizard.miraheze.org [12:20:37] seeing as l goes before p [12:20:53] Is there a way to make it use phab [12:23:15] ? [12:23:31] apparently not. it was failing when i did phabricator.miraheze.org [12:23:32] with a 500 [12:24:15] Ok [12:25:12] Btw I reverted my change to MA as it cause then invalid json error [12:25:22] caused* [12:25:22] ok [12:25:31] yeh it would have caused a invalid error [12:25:34] as it used '' [12:25:36] and not "" [12:26:04] what’s the correct way to do it? [12:26:12] Oh, nvm [12:26:31] There is a difference between “ and ‘ in php though [12:26:43] yep but in json it's a different lang [12:26:45] and not php [12:29:46] PROBLEM - misc4 phabricator.miraheze.org HTTPS on misc4 is WARNING: HTTP WARNING: HTTP/1.1 401 Unauthorized - 397 bytes in 0.025 second response time [12:34:40] RECOVERY - misc4 phabricator.miraheze.org HTTPS on misc4 is OK: HTTP OK: HTTP/1.1 200 OK - 17264 bytes in 0.244 second response time [12:34:49] fixed :D [12:37:32] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fAnM0 [12:37:33] [02miraheze/puppet] 07paladox 034882735 - add icinga http check for phabricator.miraheze.org and phab.miraheze.wiki [12:37:59] [02MatomoAnalytics] 07JohnFLewis commented on commit 03a09590535da81b29a826a8605a4373c351b08a86 - 13https://git.io/fAnMg [12:40:49] [02MatomoAnalytics] 07MacFan4000 commented on commit 03a09590535da81b29a826a8605a4373c351b08a86 - 13https://git.io/fAnMF [12:42:19] PROBLEM - misc4 phab.miraheze.wiki HTTPS on misc4 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 9702 bytes in 0.108 second response time [12:43:34] [02MatomoAnalytics] 07JohnFLewis commented on commit 03a09590535da81b29a826a8605a4373c351b08a86 - 13https://git.io/fAnDY [12:44:30] [02MatomoAnalytics] 07JohnFLewis commented on commit 03a09590535da81b29a826a8605a4373c351b08a86 - 13https://git.io/fAnDs [12:44:55] JohnLewis which is me :D [12:46:31] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fAnDB [12:46:33] [02miraheze/puppet] 07paladox 033ac69cd - Expect phab.miraheze.wiki to return 500 [12:51:34] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fAnDp [12:51:35] [02miraheze/puppet] 07paladox 03b4ee0ff - fix http_expect var [12:55:23] [02mw-config] 07Amanda-Catherine opened pull request 03#2414: Userrights configuration for igrovyesistemywiki T3536 - 13https://git.io/fAny4 [12:55:51] RECOVERY - misc4 phab.miraheze.wiki HTTPS on misc4 is OK: HTTP OK: Status line output matched "HTTP/1.1 500 Internal Server Error" - 9702 bytes in 0.174 second response time [12:56:34] [02mw-config] 07paladox closed pull request 03#2414: Userrights configuration for igrovyesistemywiki T3536 - 13https://git.io/fAny4 [12:56:35] [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fAnyz [12:56:37] [02miraheze/mw-config] 07Amanda-Catherine 039c8197e - Userrights configuration for igrovyesistemywiki T3536 (#2414) [12:57:03] paladox: MacFan4000 I recommend that we wait for Travis CI checks before merging [12:57:16] There is always the small chance that there's a minor not noticeable syntax error [12:57:32] [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fAnyo [12:57:32] Reception123 ok [12:57:32] (pinging MacFan since you've also merged a few PRs before the checks) [12:57:34] [02miraheze/mw-config] 07paladox 03915ec66 - fix syntax [12:57:49] paladox: See! You've just completely proven my point! [12:57:59] Reception123 yep lol [12:58:08] paladox: so you don't need anymore proof now that you should wait :P [12:58:14] LOL [12:59:59] Hallo [13:00:03] Wiki-1776: hi [13:00:41] JohnLewis: the other day, I was referring to https://github.com/miraheze/MatomoAnalytics/blob/master/includes/SpecialAnalytics.php#L16 [13:00:41] Title: [ MatomoAnalytics/SpecialAnalytics.php at master · miraheze/MatomoAnalytics · GitHub ] - github.com [13:00:48] is it not possible to make them i18n so we can translate? [13:03:30] PROBLEM - misc4 lizard.miraheze.org HTTPS on misc4 is WARNING: HTTP WARNING: HTTP/1.1 401 Unauthorized - 397 bytes in 0.014 second response time [13:03:37] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fAnSm [13:03:38] [02miraheze/puppet] 07paladox 038d0b6c6 - Add lizard.miraheze.org http icinga check [13:03:39] Reception123: oh those, yeah, standard MW i18n method [13:03:56] ok, great :) [13:04:03] I will move them then [13:04:14] [02MatomoAnalytics] 07MacFan4000 commented on commit 03a09590535da81b29a826a8605a4373c351b08a86 - 13https://git.io/fAnSs [13:05:33] [02MatomoAnalytics] 07paladox commented on commit 03a09590535da81b29a826a8605a4373c351b08a86 - 13https://git.io/fAnSl [13:05:49] [02MatomoAnalytics] 07paladox commented on commit 03a09590535da81b29a826a8605a4373c351b08a86 - 13https://git.io/fAnSR [13:06:03] [02MatomoAnalytics] 07JohnFLewis commented on commit 03a09590535da81b29a826a8605a4373c351b08a86 - 13https://git.io/fAnSu [13:08:01] [02miraheze/MatomoAnalytics] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fAnSr [13:08:02] [02miraheze/MatomoAnalytics] 07paladox 03f50dd98 - Fix false not being set for globalId (which is the default value) [13:08:04] [02MatomoAnalytics] 07JohnFLewis commented on commit 03a09590535da81b29a826a8605a4373c351b08a86 - 13https://git.io/fAnSo [13:08:05] MacFan4000 JohnLewis ^^ [13:08:21] [02MatomoAnalytics] 07MacFan4000 commented on commit 03a09590535da81b29a826a8605a4373c351b08a86 - 13https://git.io/fAnSi [13:08:59] [02miraheze/mediawiki] 07paladox pushed 031 commit to 03REL1_31 [+0/-0/±1] 13https://git.io/fAnSD [13:09:01] [02miraheze/mediawiki] 07paladox 0337da145 - Update MatomoAnalytics [13:09:32] [02MatomoAnalytics] 07paladox commented on commit 03a09590535da81b29a826a8605a4373c351b08a86 - 13https://git.io/fAnS9 [13:16:40] meh still fails [13:16:45] i see js syntax error [13:17:05] change the value to null [13:17:09] would that do JohnLewis ? [13:17:32] [02mw-config] 07Amanda-Catherine opened pull request 03#2415: igrovyesistemywiki: ceo user group = sysop user group (T3536) - 13https://git.io/fAn91 [13:18:15] [02mw-config] 07MacFan4000 closed pull request 03#2415: igrovyesistemywiki: ceo user group = sysop user group (T3536) - 13https://git.io/fAn91 [13:18:17] [02miraheze/mw-config] 07MacFan4000 pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fAn9S [13:18:18] [02miraheze/mw-config] 07Amanda-Catherine 0345c07d7 - igrovyesistemywiki: ceo user group = sysop user group (T3536) (#2415) [13:18:28] 2 out of 3 php checks had passed [13:18:43] ok [13:18:43] PROBLEM - test1 Puppet on test1 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki core] [13:18:49] ^ that must be me [13:21:07] DONT RM ON TEST1 [13:21:19] Heh [13:21:20] PLEASE FOR THR LOVE OF GOD [13:21:26] JohnLewis: Lol [13:21:28] Reception123: ^^ [13:21:31] JohnLewis: Yeah, I know about ManageWiki [13:21:34] JohnLewis: also see my question above [13:21:42] Thank god [13:21:46] I just tested somethign with MatomoAnalytics so I don't mess up [13:22:05] JohnLewis writing in all caps... [13:22:06] Heh [13:22:14] Paladox, maybe. Could just string it if you wanted to [13:22:34] I guess null and then go to string if it dosent work [13:22:39] and to be safe, I copied the ManageWiki dir to my homedir too just to be extra sure :) [13:22:42] so no need to to worry [13:22:56] AmandaCatherine: I had to get a very urgent and important point across okay :P [13:23:03] Reception123: mine setting null in MA [13:23:15] For the config MacFan4000 tryed to change please [13:23:31] paladox: why are you a "permission" in Special:MWPermissions? [13:23:34] are you that speical? [13:23:36] paladox: hm? [13:23:51] Reception123: seems I am [13:24:02] I am deferring that question to JohnLewis [13:24:09] Who did that per the email I got :D [13:25:21] Reception123: cause I thought I had it working, paladox was around and I was going to demonstrate the fact it worked to him [13:25:27] but it didn’t xD [13:25:37] [02mw-config] 07Amanda-Catherine opened pull request 03#2416: Additional userrights for igrovyesistemywiki T3536 - 13https://git.io/fAnHR [13:25:50] JohnLewis: Permissions looking really nice by the way :) [13:26:06] reminded me of CentralAuth, and from the code looks like I was right :D [13:26:27] AmandaCatherine: what's all this "UserType" stuff? [13:26:31] Good that it looks nice [13:26:39] because that’s all it is rn xD [13:26:46] lol [13:27:00] Reception123: https://phabricator.miraheze.org/T3536#67353 [13:27:01] Title: [ ⚓ T3536 Change mediawiki config on igrovyesistemywiki ] - phabricator.miraheze.org [13:28:23] ok [13:28:26] [02miraheze/MatomoAnalytics] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fAnHb [13:28:28] [02miraheze/MatomoAnalytics] 07paladox 03b8f0b1b - Set MatomoAnalyticsGlobalID Default value to null [13:28:37] Reception123: please deploy ^^ [13:28:56] o [13:28:56] k [13:33:38] [02miraheze/mediawiki] 07Reception123 pushed 031 commit to 03REL1_31 [+0/-0/±1] 13https://git.io/fAnQg [13:33:39] [02miraheze/mediawiki] 07Reception123 032305861 - Update MatomoAnalytics [13:34:39] [02mw-config] 07Amanda-Catherine opened pull request 03#2417: Autopromotion settings for igrovyesistemywiki T3536 - 13https://git.io/fAnQK [13:34:53] ^ ...and one more still to go! [13:35:03] The number of changes being requested here is huge [13:39:28] There’s a sand and ocean for you JohnLewis [13:39:33] [02mw-config] 07Amanda-Catherine opened pull request 03#2418: Restriction levels for igrovyesistemywiki T3536 - 13https://git.io/fAn7U [13:39:35] Meh it’s not uploading [13:40:04] And... whew! [13:40:22] I think ManageWiki will be very helpful for tasks like this https://phabricator.miraheze.org/T3536 [13:40:23] Title: [ ⚓ T3536 Change mediawiki config on igrovyesistemywiki ] - phabricator.miraheze.org [13:40:41] AmandaCatherine; when I roll out permissions management, I want it to be a small deployment first. Care to be involved? [13:41:03] JohnLewis: what do you mean “be involved”? [13:41:27] Your wiki is one that gets it before any others [13:41:35] Sure [13:41:59] I don’t mind being a guinea pig for things like this [13:42:44] paladox: still the same js error [13:42:57] Meta will also get it and maybe testwiki (Reception123?) [13:43:14] Yeah :) [13:43:19] Okay now, why are https://github.com/miraheze/mw-config/pull/2416 and https://github.com/miraheze/mw-config/pull/2417 both failing Travis? [13:43:21] Title: [ Additional userrights for igrovyesistemywiki T3536 by Amanda-Catherine · Pull Request #2416 · miraheze/mw-config · GitHub ] - github.com [13:43:24] Title: [ Autopromotion settings for igrovyesistemywiki T3536 by Amanda-Catherine · Pull Request #2417 · miraheze/mw-config · GitHub ] - github.com [13:43:39] Great, three wikis sounds good for now when I fix stuff [13:43:57] AmandaCathrine looking [13:43:59] Hm [13:44:06] Maybe a string then [13:44:16] “false” [13:44:20] AmandaCatherine, use of '=' instead of '=>' [13:44:35] Then !== ‘false’ in the php code [13:44:44] Voidwalker: oh [13:45:43] Paladox: you have an if essentially already. [13:46:13] youre issue is if you, if $var not true, then set $var is false [13:46:23] aka, you’ve changed no behaviour [13:47:04] [02mw-config] 07Amanda-Catherine synchronize pull request 03#2416: Additional userrights for igrovyesistemywiki T3536 - 13https://git.io/fAnHR [13:47:35] JohnLewis: how do I get it to set false? [13:48:49] * paladox is on the beach [13:48:54] Near the North Sea [13:49:40] The issue is it IS false [13:50:00] [02mw-config] 07Amanda-Catherine synchronize pull request 03#2416: Additional userrights for igrovyesistemywiki T3536 - 13https://git.io/fAnHR [13:50:25] JohnLewis: yep [13:50:35] So any idea how to get it to say false [13:50:41] In the value [13:51:10] $var ?? ‘false’? [13:51:25] aka what you’ve done, just ‘false’ not false [13:51:40] Also you should probably int check it [13:52:27] MacFan4000: ^^ [13:52:33] So “false” in the Jason [13:52:38] *json [13:52:44] No... [13:53:28] Ack WTF did I do to these PRs [13:53:38] So many requests and I screw stuff up [13:54:07] * AmandaCatherine slaps herself around with a large trout [13:55:14] paladox: I bet JSON comes from the name "Jason" :P [13:55:22] or maybe it just stands for Javascript Object Notation [13:57:40] Lol [13:57:40] [02mw-config] 07Amanda-Catherine synchronize pull request 03#2416: Additional userrights for igrovyesistemywiki T3536 - 13https://git.io/fAnHR [13:59:40] [02mw-config] 07Amanda-Catherine synchronize pull request 03#2416: Additional userrights for igrovyesistemywiki T3536 - 13https://git.io/fAnHR [14:01:09] [02miraheze/MatomoAnalytics] 07MacFan4000 pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fAnF2 [14:01:11] [02miraheze/MatomoAnalytics] 07MacFan4000 03a5c535b - Update extension.json [14:01:23] [02mw-config] 07Amanda-Catherine commented on pull request 03#2416: Additional userrights for igrovyesistemywiki T3536 - 13https://git.io/fAnFw [14:01:38] It finally passed ^ [14:02:28]