[00:15:17] <icinga-miraheze>	 RECOVERY - JobQueue on mw3 is OK: JOBQUEUE OK - job queue below 300 jobs  
[00:15:27] <icinga-miraheze>	 RECOVERY - JobQueue on mw2 is OK: JOBQUEUE OK - job queue below 300 jobs  
[04:34:28] <icinga-miraheze>	 PROBLEM - JobQueue on mw2 is CRITICAL: JOBQUEUE CRITICAL - job queue greater than 300 jobs. Current queue: 2517  
[04:43:36] <icinga-miraheze>	 PROBLEM - HTTP 4xx/5xx ERROR Rate on cp4 is CRITICAL: CRITICAL - NGINX Error Rate is 82%  
[04:45:36] <icinga-miraheze>	 RECOVERY - HTTP 4xx/5xx ERROR Rate on cp4 is OK: OK - NGINX Error Rate is 36%  
[04:54:37] <icinga-miraheze>	 PROBLEM - Puppet on misc2 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Apt_key[Add key: F86AA916A2195E121AEDB11437BBEE3F7AD95B3F from Apt::Source grafana_apt]  
[05:00:27] <icinga-miraheze>	 RECOVERY - JobQueue on mw3 is OK: JOBQUEUE OK - job queue below 300 jobs  
[05:00:29] <icinga-miraheze>	 RECOVERY - JobQueue on mw2 is OK: JOBQUEUE OK - job queue below 300 jobs  
[05:02:37] <icinga-miraheze>	 RECOVERY - Puppet on misc2 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures  
[05:12:26] <icinga-miraheze>	 PROBLEM - JobQueue on mw3 is CRITICAL: JOBQUEUE CRITICAL - job queue greater than 300 jobs. Current queue: 5262  
[05:12:28] <icinga-miraheze>	 PROBLEM - JobQueue on mw2 is CRITICAL: JOBQUEUE CRITICAL - job queue greater than 300 jobs. Current queue: 5259  
[05:36:10] <Reception123>	 !log reception@mw2:/srv/mediawiki/w/maintenance$ sudo -u www-data php importDump.php --wiki jordepediawiki /home/reception/jorde_pages_full.xml                              
[05:36:14] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master
[05:39:02] <Not-683e>	 [02miraheze/services] 07Reception123 pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/f45Lu
[05:39:04] <Not-683e>	 [02miraheze/services] 07Reception123 03931c26a - add "jordepedia" to services
[05:40:58] <Not-683e>	 [02miraheze/services] 07Reception123 pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/f45La
[05:40:59] <Not-683e>	 [02miraheze/services] 07Reception123 0331250b1 - add "unab" to services
[05:44:46] <Not-683e>	 [02miraheze/mw-config] 07Reception123 pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/f45LQ
[05:44:48] <Not-683e>	 [02miraheze/mw-config] 07Reception123 03e3fea31 - enable filetypes on unabwiki per T3276
[05:54:18] <Not-683e>	 [02miraheze/mw-config] 07Reception123 pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/f45tE
[05:54:19] <Not-683e>	 [02miraheze/mw-config] 07Reception123 03fac0186 - extra ns for wikibookwiki per T3188 resolves T3188
[06:02:17] <icinga-miraheze>	 PROBLEM - JobQueue on mw3 is UNKNOWN: NRPE: Unable to read output  
[06:02:27] <icinga-miraheze>	 PROBLEM - JobQueue on mw2 is UNKNOWN: NRPE: Unable to read output  
[06:03:10] <Not-683e>	 [02miraheze/ssl] 07Reception123 pushed 031 commit to 03master [+1/-0/±1] 13https://git.io/f45ty
[06:03:11] <Not-683e>	 [02miraheze/ssl] 07Reception123 03e9a09af - add wiki.bonessdiving.club cert
[06:03:17] <icinga-miraheze>	 PROBLEM - MediaWiki Rendering on mw2 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 222 bytes in 0.062 second response time  
[06:03:27] <icinga-miraheze>	 PROBLEM - Varnish Backends on cp4 is CRITICAL: 3 backends are down.  mw1 mw2 mw3  
[06:03:37] <icinga-miraheze>	 PROBLEM - MediaWiki Rendering on mw3 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 222 bytes in 0.030 second response time  
[06:03:39] <icinga-miraheze>	 PROBLEM - Varnish Backends on cp5 is CRITICAL: 3 backends are down.  mw1 mw2 mw3  
[06:03:47] <icinga-miraheze>	 PROBLEM - GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 6 datacenters are down: 107.191.126.23/cpweb, 2604:180:0:33b::2/cpweb, 81.4.109.133/cpweb, 2a00:d880:5:8ea::ebc7/cpweb, 172.104.111.8/cpweb, 2400:8902::f03c:91ff:fe07:444e/cpweb  
[06:04:37] <icinga-miraheze>	 PROBLEM - MediaWiki Rendering on mw1 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 222 bytes in 0.090 second response time  
[06:04:39] <icinga-miraheze>	 PROBLEM - HTTP 4xx/5xx ERROR Rate on cp2 is CRITICAL: CRITICAL - NGINX Error Rate is 85%  
[06:07:31] <Reception123>	 paladox: why did travis not say anything?
[06:07:37] <icinga-miraheze>	 PROBLEM - HTTP 4xx/5xx ERROR Rate on cp5 is CRITICAL: CRITICAL - NGINX Error Rate is 66%  
[06:08:10] <Not-683e>	 [02miraheze/mw-config] 07Reception123 pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/f45tp
[06:08:12] <Not-683e>	 [02miraheze/mw-config] 07Reception123 03c8c471a - syntax
[06:09:18] <icinga-miraheze>	 RECOVERY - MediaWiki Rendering on mw2 is OK: HTTP OK: HTTP/1.1 200 OK - 30033 bytes in 0.198 second response time  
[06:09:28] <icinga-miraheze>	 RECOVERY - Varnish Backends on cp4 is OK: All 3 backends are healthy  
[06:09:38] <icinga-miraheze>	 RECOVERY - MediaWiki Rendering on mw3 is OK: HTTP OK: HTTP/1.1 200 OK - 30033 bytes in 0.254 second response time  
[06:09:40] <icinga-miraheze>	 RECOVERY - Varnish Backends on cp5 is OK: All 3 backends are healthy  
[06:09:54] <travis-ci>	 miraheze/mw-config/master/c8c471a - Reception123 The build was fixed. https://travis-ci.org/miraheze/mw-config/builds/397200728
[06:10:28] <icinga-miraheze>	 PROBLEM - JobQueue on mw3 is CRITICAL: JOBQUEUE CRITICAL - job queue greater than 300 jobs. Current queue: 1495  
[06:10:30] <icinga-miraheze>	 PROBLEM - JobQueue on mw2 is CRITICAL: JOBQUEUE CRITICAL - job queue greater than 300 jobs. Current queue: 1493  
[06:10:38] <icinga-miraheze>	 RECOVERY - MediaWiki Rendering on mw1 is OK: HTTP OK: HTTP/1.1 200 OK - 30034 bytes in 0.189 second response time  
[06:10:40] <icinga-miraheze>	 RECOVERY - HTTP 4xx/5xx ERROR Rate on cp2 is OK: OK - NGINX Error Rate is 6%  
[06:11:30] <icinga-miraheze>	 RECOVERY - GDNSD Datacenters on misc1 is OK: OK - all datacenters are online  
[06:11:40] <icinga-miraheze>	 RECOVERY - GDNSD Datacenters on ns1 is OK: OK - all datacenters are online  
[06:13:40] <icinga-miraheze>	 RECOVERY - HTTP 4xx/5xx ERROR Rate on cp5 is OK: OK - NGINX Error Rate is 17%  
[06:16:20] <icinga-miraheze>	 RECOVERY - JobQueue on mw2 is OK: JOBQUEUE OK - job queue below 300 jobs  
[06:17:48] <Not-683e>	 [02miraheze/mw-config] 07Reception123 pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/f45q2
[06:17:49] <Not-683e>	 [02miraheze/mw-config] 07Reception123 03f8e4186 - add custom domain wiki.bonessdiving.club per T3243
[08:13:00] <icinga-miraheze>	 PROBLEM - Current Load on misc1 is WARNING: WARNING - load average: 1.88, 1.66, 1.23  
[08:15:00] <icinga-miraheze>	 RECOVERY - Current Load on misc1 is OK: OK - load average: 1.45, 1.59, 1.26  
[09:14:40] <icinga-miraheze>	 PROBLEM - Puppet on misc2 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Apt_key[Add key: F86AA916A2195E121AEDB11437BBEE3F7AD95B3F from Apt::Source grafana_apt]  
[09:22:40] <icinga-miraheze>	 RECOVERY - Puppet on misc2 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures  
[10:03:09] <Reception123>	 JohnLewis: earlier today I got a "the build was fixed" from Travis but I never got a broken build message 
[10:06:25] <JohnLewis>	 Yep
[10:06:42] <JohnLewis>	 welcome to UDP protocol bots
[10:44:40] <icinga-miraheze>	 PROBLEM - Puppet on misc2 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Apt_key[Add key: F86AA916A2195E121AEDB11437BBEE3F7AD95B3F from Apt::Source grafana_apt]  
[10:52:40] <icinga-miraheze>	 RECOVERY - Puppet on misc2 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures  
[11:15:58] <JohnLewis>	 Why does that keep happening
[11:23:30] <paladox>	 JohnLewis i presume rate limiting?
[11:23:47] <JohnLewis>	 find out
[11:25:27] <paladox>	 Jun 27 10:40:51 misc2 puppet-agent[14921]: Execution of '/usr/bin/apt-key adv --keyserver keyserver.ubuntu.com --recv-keys F86AA916A2195E121AEDB11437BBEE3F7AD95B3F' returned 2: Warning: apt-key output should not be parsed (stdout is not a terminal)
[11:25:32] <paladox>	 Jun 27 10:40:51 misc2 puppet-agent[14921]: Executing: /tmp/apt-key-gpghome.ymclguVHxd/gpg.1.sh --keyserver keyserver.ubuntu.com --recv-keys F86AA916A2195E121AEDB11437BBEE3F7AD95B3F
[11:25:32] <paladox>	 Jun 27 10:40:51 misc2 puppet-agent[14921]: gpg: keyserver receive failed: No data
[11:25:34] <paladox>	 JohnLewis ^^
[11:27:58] <JohnLewis>	 mkay
[11:28:07] <JohnLewis>	 its happening regularly
[11:38:29] <icinga-miraheze>	 PROBLEM - wiki.svenskabriardklubben.se - LetsEncrypt on sslhost is WARNING: WARNING - Certificate wiki.svenskabriardklubben.se expires in 15 day(s) (Fri 13 Jul 2018 11:35:45 AM GMT +0000).  
[11:38:44] <Not-683e>	 [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/f45Xo
[11:38:45] <Not-683e>	 [02miraheze/ssl] 07MirahezeSSLBot 033090cef - Bot: Update SSL cert for wiki.svenskabriardklubben.se
[12:58:04] <icinga-miraheze>	 PROBLEM - www.splat-teams.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate www.splat-teams.com expires in 15 day(s) (Fri 13 Jul 2018 12:54:27 PM GMT +0000).  
[12:58:17] <Not-683e>	 [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/f45Q6
[12:58:18] <Not-683e>	 [02miraheze/ssl] 07MirahezeSSLBot 03765514b - Bot: Update SSL cert for www.splat-teams.com
[13:02:07] <icinga-miraheze>	 RECOVERY - www.splat-teams.com - LetsEncrypt on sslhost is OK: OK - Certificate www.splat-teams.com will expire on Tue 25 Sep 2018 11:58:12 AM GMT +0000.  
[13:02:27] <icinga-miraheze>	 RECOVERY - wiki.svenskabriardklubben.se - LetsEncrypt on sslhost is OK: OK - Certificate wiki.svenskabriardklubben.se will expire on Tue 25 Sep 2018 10:38:39 AM GMT +0000.  
[13:07:37] <icinga-miraheze>	 PROBLEM - www.schulwiki.de - LetsEncrypt on sslhost is WARNING: WARNING - Certificate www.schulwiki.de expires in 15 day(s) (Fri 13 Jul 2018 01:04:17 PM GMT +0000).  
[13:07:48] <Not-683e>	 [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/f4579
[13:07:50] <Not-683e>	 [02miraheze/ssl] 07MirahezeSSLBot 03aa31e38 - Bot: Update SSL cert for www.schulwiki.de
[13:11:37] <icinga-miraheze>	 RECOVERY - www.schulwiki.de - LetsEncrypt on sslhost is OK: OK - Certificate www.schulwiki.de will expire on Tue 25 Sep 2018 12:07:44 PM GMT +0000.  
[13:22:48] <icinga-miraheze>	 PROBLEM - holonet.pw - LetsEncrypt on sslhost is WARNING: WARNING - Certificate holonet.pw expires in 15 day(s) (Fri 13 Jul 2018 01:19:59 PM GMT +0000).  
[13:22:59] <Not-683e>	 [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/f455A
[13:23:00] <Not-683e>	 [02miraheze/ssl] 07MirahezeSSLBot 03057a2a8 - Bot: Update SSL cert for holonet.pw
[13:30:51] <icinga-miraheze>	 RECOVERY - holonet.pw - LetsEncrypt on sslhost is OK: OK - Certificate holonet.pw will expire on Tue 25 Sep 2018 12:22:54 PM GMT +0000.  
[13:48:17] <ZppixBot>	 Hello Alvgjerdka! If you have any questions feel free to ask and someone should answer soon.
[14:34:49] <icinga-miraheze>	 PROBLEM - Puppet on misc2 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Apt_key[Add key: F86AA916A2195E121AEDB11437BBEE3F7AD95B3F from Apt::Source grafana_apt]  
[14:39:56] <paladox>	 !log upgrading grafana to 5.2
[14:40:01] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master
[14:40:30] <paladox>	 !log upgrading phabricator
[14:40:36] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master
[14:42:49] <icinga-miraheze>	 RECOVERY - Puppet on misc2 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures  
[14:44:56] <paladox>	 !log upgrading mariadb on bacula1
[14:45:00] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master
[14:46:03] <JohnLewis>	 k
[14:46:18] <MacFan4000>	 So why does icinga say that bacula1 mariaDB is unknown?
[14:47:42] <paladox>	 because i think it keeps crashing
[14:49:40] <icinga-miraheze>	 PROBLEM - Bacula Daemon on bacula1 is CRITICAL: PROCS CRITICAL: 1 process with UID = 110 (bacula)  
[14:49:42] <paladox>	 180627 14:46:01 [ERROR] mysqld got signal 6 ;
[14:49:45] <paladox>	 uh
[14:49:50] <icinga-miraheze>	 PROBLEM - Bacula - Phabricator Static on bacula1 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds.  
[14:50:06] <paladox>	 Lets remove mysql from bacula
[14:50:10] <paladox>	 it dosen't work
[14:50:21] <paladox>	 JohnLewis ^^ is that ok with you?
[14:50:31] <JohnLewis>	 yes
[14:50:54] <paladox>	 ok
[14:51:40] <icinga-miraheze>	 RECOVERY - Bacula - Databases - db4 on bacula1 is OK: OK: Diff, 45541 files, 52.58GB, 2018-06-17 01:33:00 (1.5 weeks ago)  
[14:52:05] <Not-683e>	 [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/f4dv5
[14:52:07] <Not-683e>	 [02miraheze/puppet] 07paladox 03a8dda64 - Update site.pp
[14:55:38] <paladox>	 !log removed mariadb from bacula1 as it kept crashing
[14:55:43] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master
[14:55:54] <paladox>	 !log rm -rf /srv/mariadb on bacula1
[14:55:59] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master
[14:56:06] * paladox double checks before hitting enter
[15:05:46] <MacFan4000>	 SPF|Cloud: ^^ CC as you are the one that installed it I belive
[15:07:10] <JohnLewis>	 he did
[15:36:38] <icinga-miraheze>	 PROBLEM - Varnish Backends on cp2 is CRITICAL: 1 backends are down.  mw1  
[15:36:40] <icinga-miraheze>	 PROBLEM - Varnish Backends on cp5 is CRITICAL: 1 backends are down.  mw1  
[15:38:38] <icinga-miraheze>	 RECOVERY - Varnish Backends on cp5 is OK: All 3 backends are healthy  
[16:13:56] <Not-683e>	 [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/f4dq9
[16:13:58] <Not-683e>	 [02miraheze/mw-config] 07paladox 03a49c8a8 - TEST: Make wgNoticeProject a string
[16:15:01] <Wiki-1776>	 HTTP ERROR 500.
[16:15:18] <paladox>	 hmm
[16:15:45] <Wiki-1776>	 ...503 
[16:16:04] <paladox>	 should be workign now
[16:16:23] <Wiki-1776>	 no
[16:16:34] <Wiki-1776>	 https://union.miraheze.org/wiki/a
[16:16:44] <paladox>	 oh
[16:16:51] <paladox>	 fixing
[16:17:09] <Not-683e>	 [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/f4dmI
[16:17:11] <Not-683e>	 [02miraheze/mw-config] 07paladox 033d50bb4 - Revert "TEST: Make wgNoticeProject a string" This reverts commit a49c8a8b078a0fe345e822e2a9c71927e8fb792a.
[16:18:07] <paladox>	 works now Wiki-1776 
[16:18:23] <Wiki-1776>	 Thanks paladox 
[16:18:28] <paladox>	 your welcome :)
[16:19:29] <MacFan4000>	 paladox: Can you please remove the bacula mysql Check from icinga?
[16:19:38] <paladox>	 yep
[16:20:27] <MacFan4000>	 Thanks
[16:24:47] <Not-683e>	 [02mw-config] 07paladox created branch 03paladox-patch-1 - 13https://git.io/vbvb3
[16:24:48] <Not-683e>	 [02miraheze/mw-config] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/f4dmy
[16:24:50] <Not-683e>	 [02miraheze/mw-config] 07paladox 034f18c2f - Set wgNoticeProject to be a string and not a array per docs at https://www.mediawiki.org/wiki/Extension:CentralNotice#Quick_developer_setup it shows wgNoticeProject as a string and not a array
[16:24:50] <ZppixBot>	 Title: [ Extension:CentralNotice - MediaWiki ] - www.mediawiki.org
[16:24:51] <Not-683e>	 [02mw-config] 07paladox opened pull request 03#2272: Set wgNoticeProject to be a string and not a array - 13https://git.io/f4dmS
[16:25:21] <Not-683e>	 [02miraheze/mw-config] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/f4dmQ
[16:25:22] <Not-683e>	 [02miraheze/mw-config] 07paladox 034ffc02a - Update LocalSettings.php
[16:25:24] <Not-683e>	 [02mw-config] 07paladox synchronize pull request 03#2272: Set wgNoticeProject to be a string and not a array - 13https://git.io/f4dmS
[16:25:30] <Not-683e>	 [02mw-config] 07paladox closed pull request 03#2272: Set wgNoticeProject to be a string and not a array - 13https://git.io/f4dmS
[16:25:32] <Not-683e>	 [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±2] 13https://git.io/f4dm7
[16:25:33] <Not-683e>	 [02miraheze/mw-config] 07paladox 032905f0d - Set wgNoticeProject to be a string and not a array (#2272) * Set wgNoticeProject to be a string and not a array per docs at https://www.mediawiki.org/wiki/Extension:CentralNotice#Quick_developer_setup it shows wgNoticeProject as a string and not a array * Update LocalSettings.php
[16:25:33] <ZppixBot>	 Title: [ Extension:CentralNotice - MediaWiki ] - www.mediawiki.org
[16:32:52] <Not-683e>	 [02miraheze/mediawiki] 07paladox pushed 031 commit to 03REL1_31 [+0/-0/±1] 13https://git.io/f4dYa
[16:32:53] <Not-683e>	 [02miraheze/mediawiki] 07paladox 03aa359d8 - Update DynamicPageList3 to upstream
[16:37:00] <JohnLewis>	 shouldn't be 'all' now because it's not 'all'
[16:37:53] <Not-683e>	 [02miraheze/mediawiki] 07paladox pushed 031 commit to 03REL1_31 [+0/-0/±1] 13https://git.io/f4dOI
[16:37:54] <Not-683e>	 [02miraheze/mediawiki] 07paladox 03ade8936 - Update DynamicPageList3 to a511bf2eda2fea3cd0b2699dad2f62f0c713eebd
[16:38:21] <paladox>	 JohnLewis that var dosen't take array
[16:38:24] <paladox>	 the error has gone
[16:38:26] <paladox>	 i think
[16:38:50] <JohnLewis>	 that's not relevant to what I said
[16:39:08] <paladox>	 JohnLewis ok, then what should it be ? :)
[16:39:15] <JohnLewis>	 open?
[16:40:26] <paladox>	 JohnLewis open?
[16:40:35] <JohnLewis>	 yes open
[16:40:37] <paladox>	 wgNoticeProjects has all
[16:40:50] <JohnLewis>	 change it to opej
[16:40:53] <JohnLewis>	 *open
[16:41:00] <paladox>	 ok
[16:41:03] <paladox>	 JohnLewis including in wgNoticeProjects ?
[16:41:20] <JohnLewis>	 yes because wgNoticeProjects should be a set of wgNoticeProject
[16:41:42] <Not-683e>	 [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/f4dOS
[16:41:44] <Not-683e>	 [02miraheze/mw-config] 07paladox 03c634949 - Change wgNoticeProjects and wgNoticeProject to open
[16:41:48] <paladox>	 JohnLewis done ^^
[16:42:14] <JohnLewis>	 okay
[16:42:28] <icinga-miraheze>	 PROBLEM - JobQueue on mw3 is CRITICAL: JOBQUEUE CRITICAL - job queue greater than 300 jobs. Current queue: 2541  
[16:53:29] <icinga-miraheze>	 PROBLEM - Varnish Backends on cp4 is CRITICAL: 1 backends are down.  mw2  
[16:53:37] <icinga-miraheze>	 PROBLEM - Varnish Backends on cp2 is CRITICAL: 1 backends are down.  mw2  
[16:53:58] <paladox>	 huh
[16:55:28] <paladox>	 i see nothing in the log that would indicate why it's down.
[16:55:30] <paladox>	 JohnLewis ^^
[16:56:02] <paladox>	 did someone click on visualeditor on metawiki main page?
[16:57:27] <icinga-miraheze>	 RECOVERY - GDNSD Datacenters on misc1 is OK: OK - all datacenters are online  
[16:59:27] <icinga-miraheze>	 RECOVERY - Varnish Backends on cp4 is OK: All 3 backends are healthy  
[16:59:37] <icinga-miraheze>	 RECOVERY - Varnish Backends on cp2 is OK: All 3 backends are healthy  
[16:59:52] <JohnLewis>	 No? Nginx should give something 
[17:00:05] <paladox>	 JohnLewis on cp4 on mw2?
[17:00:29] <JohnLewis>	 both I guess
[17:00:34] <JohnLewis>	 mw2 first
[17:01:01] <paladox>	 ok
[17:12:28] <icinga-miraheze>	 RECOVERY - JobQueue on mw3 is OK: JOBQUEUE OK - job queue below 300 jobs  
[18:12:28] <icinga-miraheze>	 PROBLEM - JobQueue on mw2 is CRITICAL: JOBQUEUE CRITICAL - job queue greater than 300 jobs. Current queue: 325  
[18:12:30] <icinga-miraheze>	 PROBLEM - JobQueue on mw3 is CRITICAL: JOBQUEUE CRITICAL - job queue greater than 300 jobs. Current queue: 325  
[18:16:28] <icinga-miraheze>	 RECOVERY - JobQueue on mw3 is OK: JOBQUEUE OK - job queue below 300 jobs  
[18:23:33] <ZppixBot>	 Hi! Just contacting you to tell that here are no high priority open tasks with last update time > 3 days. Nice job!
[19:34:49] <icinga-miraheze>	 PROBLEM - Puppet on misc2 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Apt_key[Add key: F86AA916A2195E121AEDB11437BBEE3F7AD95B3F from Apt::Source grafana_apt]  
[19:40:49] <icinga-miraheze>	 RECOVERY - Puppet on misc2 is OK: OK: Puppet is currently enabled, last run 16 seconds ago with 0 failures  
[20:05:56] <Not-683e>	 [02miraheze/mediawiki] 07paladox pushed 031 commit to 03REL1_31 [+0/-0/±1] 13https://git.io/f4da3
[20:05:58] <Not-683e>	 [02miraheze/mediawiki] 07paladox 0381e552e - Update Comments to upstream
[20:11:10] <Not-683e>	 [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/f4daQ
[20:11:12] <Not-683e>	 [02miraheze/mw-config] 07paladox 032a5156b - Make timeless a global skin (not default skin)
[20:11:20] <paladox>	 uh that was ment to be in one comment
[20:11:28] <paladox>	 oh well cleaning up it in other places
[20:12:08] <Not-683e>	 [02miraheze/mw-config] 07paladox pushed 031 commit to 03paladox-patch-2 [+0/-0/±1] 13https://git.io/f4daN
[20:12:10] <Not-683e>	 [02miraheze/mw-config] 07paladox 03bfd7969 - Timeless is now a global skin
[20:12:11] <Not-683e>	 [02mw-config] 07paladox created branch 03paladox-patch-2 - 13https://git.io/vbvb3
[20:12:13] <Not-683e>	 [02mw-config] 07paladox opened pull request 03#2273: Timeless is now a global skin - 13https://git.io/f4daA
[20:12:59] <Not-683e>	 [02miraheze/mw-config] 07paladox pushed 031 commit to 03paladox-patch-2 [+0/-0/±1] 13https://git.io/f4dap
[20:13:00] <Not-683e>	 [02miraheze/mw-config] 07paladox 03791645c - Update LocalExtensions.php
[20:13:02] <Not-683e>	 [02mw-config] 07paladox synchronize pull request 03#2273: Timeless is now a global skin - 13https://git.io/f4daA
[20:13:32] <Not-683e>	 [02miraheze/mw-config] 07paladox pushed 031 commit to 03paladox-patch-2 [+0/-0/±1] 13https://git.io/f4dVf
[20:13:34] <Not-683e>	 [02miraheze/mw-config] 07paladox 0357b3197 - Update LocalSettings.php
[20:13:35] <Not-683e>	 [02mw-config] 07paladox synchronize pull request 03#2273: Timeless is now a global skin - 13https://git.io/f4daA
[20:13:45] <Not-683e>	 [02mw-config] 07paladox closed pull request 03#2273: Timeless is now a global skin - 13https://git.io/f4daA
[20:13:47] <Not-683e>	 [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±3] 13https://git.io/f4dVT
[20:13:48] <Not-683e>	 [02miraheze/mw-config] 07paladox 03f17d92b - Timeless is now a global skin (#2273) * Timeless is now a global skin * Update LocalExtensions.php * Update LocalSettings.php
[20:14:01] <Not-683e>	 [02mw-config] 07paladox deleted branch 03paladox-patch-2 - 13https://git.io/vbvb3
[20:14:02] <Not-683e>	 [02miraheze/mw-config] 07paladox deleted branch 03paladox-patch-2
[20:15:54] <travis-ci>	 miraheze/mw-config/paladox-patch-2/791645c - paladox The build was broken. https://travis-ci.org/miraheze/mw-config/builds/397514660
[20:16:04] <paladox>	 that's not broken
[20:16:09] <paladox>	 just i deleted the pull branch
[20:17:32] <travis-ci>	 miraheze/mw-config/paladox-patch-2/57b3197 - paladox The build was broken. https://travis-ci.org/miraheze/mw-config/builds/397514964
[20:24:48] <icinga-miraheze>	 PROBLEM - Puppet on misc2 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Apt_key[Add key: F86AA916A2195E121AEDB11437BBEE3F7AD95B3F from Apt::Source grafana_apt]  
[20:30:48] <icinga-miraheze>	 RECOVERY - Puppet on misc2 is OK: OK: Puppet is currently enabled, last run 10 seconds ago with 0 failures  
[21:26:28] <icinga-miraheze>	 PROBLEM - JobQueue on mw3 is CRITICAL: JOBQUEUE CRITICAL - job queue greater than 300 jobs. Current queue: 5401  
[21:29:58] <JohnLewis>	 seriously GUP is annoying now
[21:30:27] <NeoBogan>	 GUP?
[21:30:55] <paladox>	 NeoBogan GlobalUserPage
[21:30:55] <JohnLewis>	 GlobalUserPage
[21:30:59] <NeoBogan>	 right
[21:31:02] <NeoBogan>	 (x2)
[21:31:03] <paladox>	 JohnLewis :)
[21:31:13] <NeoBogan>	 at least it wasn’t Grand User Paladox
[21:31:19] <paladox>	 lol
[21:38:19] <NeoBogan>	 or giving up porn...
[21:38:33] <NeoBogan>	 that would be annoying :P
[22:14:43] <JohnLewis>	 70 open issues on Phab, fun
[22:15:09] <paladox>	 and you get the fun of triaging them / tagging them JohnLewis!
[22:15:42] <JohnLewis>	 they're all triaged and tagged
[22:15:47] <NeoBogan>	 and you get the fun of resolving all of them, paladox
[22:15:54] <paladox>	 lol
[22:15:54] <NeoBogan>	 call it a field training exercise
[22:16:11] <NeoBogan>	 john can call it a patience exercise
[22:16:22] <paladox>	 huh?
[22:16:28] <paladox>	 why patience exercise?
[22:16:34] <NeoBogan>	 cleaning up after you
[22:16:35] <NeoBogan>	 :P
[22:16:46] <paladox>	 Also that would be nearly impossible for me to resolve all 70 open tasks on my own :)
[22:16:57] <NeoBogan>	 do them one at a time
[22:17:01] <paladox>	 as i am already working on another project for #operations!
[22:17:05] <NeoBogan>	 by the end of the week you should have made a fair dent in the backlog
[22:17:41] <paladox>	 and NeoBogan gets the fun for paying for it all :)
[22:17:43] <JohnLewis>	 22 of them are extension review tasks
[22:17:56] <NeoBogan>	 sure
[22:18:00] <NeoBogan>	 I’ll even give you a payrise
[22:18:04] <NeoBogan>	 what’s 20% of 0?
[22:18:10] <paladox>	 0
[22:18:17] <NeoBogan>	 ^ your new pay rate :)
[22:18:46] <paladox>	 NeoBogan what's 20% take aray £1,000 from 0?
[22:19:40] <NeoBogan>	 -one thousand quid
[22:19:48] <NeoBogan>	 rephrase... negative one thousand quid
[22:19:55] <paladox>	 NeoBogan then you need to pay up :)
[22:20:04] <paladox>	 you owe use £1,000!
[22:20:05] * NeoBogan looks around for “up”?
[22:20:10] <paladox>	 use = us
[22:21:56] * paladox gives JohnLewis notice that NeoBogan owes us £1,000!
[22:23:42] <NeoBogan>	 good luck collecting it
[22:23:47] <paladox>	 lol
[22:30:15] <JohnLewis>	 get ready for more jobqueue warnings :P
[22:35:07] <paladox>	 JohnLewis heh, i know :) and why?
[22:35:14] * paladox thinks JohnLewis just did something!
[22:35:25] <JohnLewis>	 nah, someone edited their user page
[22:36:39] <paladox>	 oh
[22:46:10] * NeoBogan echofloods paladox across the entire WMF farm
[22:46:19] <NeoBogan>	 now *that* would be a jobqueue!
[22:46:42] <paladox>	 lol
[22:58:38] <icinga-miraheze>	 PROBLEM - HTTP 4xx/5xx ERROR Rate on cp4 is WARNING: WARNING - NGINX Error Rate is 59%  
[22:59:49] <paladox>	 well.......
[23:00:38] <icinga-miraheze>	 RECOVERY - HTTP 4xx/5xx ERROR Rate on cp4 is OK: OK - NGINX Error Rate is 5%  
[23:00:51] <paladox>	 recovered
[23:38:28] <icinga-miraheze>	 RECOVERY - JobQueue on mw3 is OK: JOBQUEUE OK - job queue below 300 jobs  
[23:50:47] <Not-683e>	 [02miraheze/dns] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/f4dFV
[23:50:49] <Not-683e>	 [02miraheze/dns] 07paladox 035c3b1cd - Depool cp4 for upgrade T3269
[23:52:59] <icinga-miraheze>	 PROBLEM - Current Load on misc1 is WARNING: WARNING - load average: 1.85, 1.35, 1.09  
[23:54:57] <icinga-miraheze>	 RECOVERY - Current Load on misc1 is OK: OK - load average: 0.95, 1.27, 1.10