[00:11:06] <icinga-miraheze>	 PROBLEM - MediaWiki Rendering on test1 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 222 bytes in 0.018 second response time  
[00:13:04] <icinga-miraheze>	 RECOVERY - MediaWiki Rendering on test1 is OK: HTTP OK: HTTP/1.1 200 OK - 30105 bytes in 0.934 second response time  
[00:13:36] <JohnLewis>	 mkay
[00:17:05] <icinga-miraheze>	 PROBLEM - MediaWiki Rendering on test1 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 222 bytes in 0.024 second response time  
[00:51:35] <icinga-miraheze>	 PROBLEM - Current Load on swift1 is WARNING: WARNING - load average: 3.77, 3.36, 3.44  
[01:01:35] <icinga-miraheze>	 RECOVERY - Current Load on swift1 is OK: OK - load average: 2.61, 3.08, 3.30  
[01:25:05] <icinga-miraheze>	 PROBLEM - Puppet on misc4 is WARNING: WARNING: Puppet is currently disabled, message: reason not specified, last run 4 minutes ago with 0 failures  
[01:43:04] <icinga-miraheze>	 RECOVERY - Puppet on misc4 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures  
[02:34:25] <icinga-miraheze>	 PROBLEM - JobQueue on mw3 is CRITICAL: JOBQUEUE CRITICAL - job queue greater than 300 jobs. Current queue: 5506  
[02:39:02] <samwilson>	 morning all
[02:39:31] <paladox>	 Hi
[02:39:37] <paladox>	 3:39am here!
[02:42:11] <samwilson>	 ah well that's still morning! :) sort of
[02:42:48] <paladox>	 Heh
[02:42:55] <paladox>	 Already getting bright out
[02:45:04] <samwilson>	 are you a phabricator admin by any chance?
[02:45:13] <samwilson>	 i'm locked out
[02:45:28] <paladox>	 Yep I am
[02:45:51] <samwilson>	 https://meta.miraheze.org/wiki/Talk:Phabricator#Removing_2FA
[02:45:52] <ZppixBot>	 Title: [ Talk:Phabricator - Miraheze Meta ] - meta.miraheze.org
[02:46:02] <paladox>	 Ah I see
[02:46:04] * paladox looks
[02:46:14] <samwilson>	 yeah, is tricky I think
[02:46:41] <paladox>	 Hmm proving your identity 
[02:46:53] <paladox>	 samwilson: you have an account on meta?
[02:47:10] <paladox>	 Can you still login there? If yes then edit something under your user page
[02:47:25] <samwilson>	 yep, i've just added a committed identity too
[02:47:35] <paladox>	 Ah ok
[02:47:55] <samwilson>	 the rule on that other wiki place is that the committed identity has to have been in place for a month... is there a similar thing in miraheze?
[02:48:24] <paladox>	 Not that I’ve seen at least
[02:48:33] <samwilson>	 oh cool
[02:48:56] <paladox>	 Any confirmed that it is you by going to your user page
[02:49:02] <paladox>	 Resetting 2fa on phab now
[02:49:09] <samwilson>	 oh cool thanks :)
[02:50:03] <samwilson>	 yay, i'm in
[02:50:09] <paladox>	 !log sudo /srv/phab/phabricator/bin/auth strip --all-types --user Samwilson on misc4
[02:50:13] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master
[02:54:33] <samwilson>	 paladox: the australian wikimedia chapter is interested in moving its wikis to miraheze
[02:54:42] <paladox>	 samwilson ah :)
[02:55:03] <samwilson>	 :) money will come along with em hehe
[02:55:12] <paladox>	 oh :)
[02:57:28] <paladox>	 samwilson JohnLewis has done alot of work with ManageWiki :)
[02:57:51] <samwilson>	 really? cool. i haven't checked it out for ages
[02:57:53] <paladox>	 to allow wiki's to decide which ext they want without having to wait for someone to merge a patch to enable that ext on a wiki.
[02:58:14] <paladox>	 only some ext require super user to enable
[02:58:23] <paladox>	 as they require us to do extra things
[02:58:38] <paladox>	 We have restbase, mathoid, we have just migrated to swift!
[02:59:26] <samwilson>	 wow, sounds really good!
[02:59:53] <samwilson>	 i've been super impressed by miraheze... been wanting to help more. now WMAU is shifting, i might have more impetus
[03:00:05] <paladox>	 :)
[03:00:23] <paladox>	 samwilson we also upgraded to mw 1.31, php7.2 and stretch
[03:00:45] <samwilson>	 7.2?! already?
[03:00:50] <samwilson>	 any issues?
[03:01:13] <samwilson>	 I found one the other day: https://gerrit.wikimedia.org/r/#/c/mediawiki/core/+/443206/
[03:01:13] <ZppixBot>	 Title: [ Gerrit Code Review ] - gerrit.wikimedia.org
[03:01:28] <paladox>	 we got that in the log
[03:01:40] <paladox>	 i cannot reproduce that through the ui though
[03:01:55] <paladox>	 i did fix php7.2 issues in dynamicpagelist3 though
[03:02:07] <paladox>	 and a few other exts too. (some still open).
[03:02:12] <paladox>	 but mostly seems stable now
[03:04:34] <samwilson>	 no, i think it's only when importing certain exif data that it fails
[03:04:42] <paladox>	 oh i see
[03:04:59] <samwilson>	 the existing tests fail with ExifTest.php on php7.2
[03:05:15] <paladox>	 oh
[03:05:49] <samwilson>	 i don't think that patch is going to get any attention though
[03:06:13] <samwilson>	 does miraheze run mediawiki core tests?
[03:06:20] <samwilson>	 (I assume not; no need to)
[03:06:53] <paladox>	 nope
[03:13:34] <Not-683e>	 [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fNUtU
[03:13:35] <Not-683e>	 [02miraheze/mw-config] 07paladox 03cde1d48 - Set wgRandomGameDisplay['random_picturegame'] to false This extension that this config enables is not installed.
[03:16:28] <Not-683e>	 [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fNUtq
[03:16:29] <Not-683e>	 [02miraheze/mw-config] 07paladox 0361caeb3 - Update LocalSettings.php
[03:45:16] <PuppyKun>	 samwilson: Hi! good to see ya again 
[03:45:32] <samwilson>	 hi PuppyKun :)
[03:45:40] <samwilson>	 how's it going?
[03:45:56] <PuppyKun>	 Pretty swell. Not super active around these parts.
[03:46:03] <PuppyKun>	 Hoping I don't get called in to work tomorrow :)
[03:46:34] <samwilson>	 hehe yeah
[03:46:55] <samwilson>	 i'm sick at the mo', but one of the joys of working from home is that it doesn't really mean i don't work
[04:34:25] <icinga-miraheze>	 RECOVERY - JobQueue on mw3 is OK: JOBQUEUE OK - job queue below 300 jobs  
[04:56:25] <icinga-miraheze>	 PROBLEM - JobQueue on mw3 is CRITICAL: JOBQUEUE CRITICAL - job queue greater than 300 jobs. Current queue: 2633  
[05:16:05] <Reception123>	 paladox: what's wrong with https://meta.miraheze.org/wiki/Tech:Timelines? I can't see anything on that page
[05:16:06] <ZppixBot>	 Title: [ Tech:Timelines - Miraheze Meta ] - meta.miraheze.org
[05:48:24] <icinga-miraheze>	 RECOVERY - JobQueue on mw3 is OK: JOBQUEUE OK - job queue below 300 jobs  
[08:24:35] <icinga-miraheze>	 PROBLEM - Puppet on misc1 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Apt_key[Add key: F86AA916A2195E121AEDB11437BBEE3F7AD95B3F from Apt::Source grafana_apt]  
[08:32:26] <icinga-miraheze>	 RECOVERY - Puppet on misc1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures  
[08:54:55] <icinga-miraheze>	 PROBLEM - wiki.ngscott.net - LetsEncrypt on sslhost is WARNING: WARNING - Certificate wiki.ngscott.net expires in 15 day(s) (Thu 26 Jul 2018 08:51:22 AM GMT +0000).  
[08:55:12] <Not-683e>	 [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fNU4e
[08:55:13] <Not-683e>	 [02miraheze/ssl] 07MirahezeSSLBot 03ce1da6e - Bot: Update SSL cert for wiki.ngscott.net
[09:03:01] <icinga-miraheze>	 RECOVERY - wiki.ngscott.net - LetsEncrypt on sslhost is OK: OK - Certificate wiki.ngscott.net will expire on Mon 08 Oct 2018 07:55:01 AM GMT +0000.  
[09:14:32] <icinga-miraheze>	 PROBLEM - Puppet on misc1 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Apt_key[Add key: F86AA916A2195E121AEDB11437BBEE3F7AD95B3F from Apt::Source grafana_apt]  
[09:22:33] <icinga-miraheze>	 RECOVERY - Puppet on misc1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures  
[09:34:31] <icinga-miraheze>	 PROBLEM - Puppet on misc1 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Apt_key[Add key: F86AA916A2195E121AEDB11437BBEE3F7AD95B3F from Apt::Source grafana_apt]  
[10:42:42] <icinga-miraheze>	 PROBLEM - Current Load on swift1 is CRITICAL: CRITICAL - load average: 4.03, 3.37, 3.00  
[10:44:42] <icinga-miraheze>	 PROBLEM - Current Load on swift1 is WARNING: WARNING - load average: 3.45, 3.32, 3.02  
[10:45:37] <paladox>	 Reception123: known, it’s created the timeline in a separate container. I’m trying to fix this.
[10:46:42] <icinga-miraheze>	 RECOVERY - Current Load on swift1 is OK: OK - load average: 3.01, 3.20, 3.01  
[10:47:25] <Reception123>	 paladox: oh, ok
[11:43:41] <icinga-miraheze>	 PROBLEM - Current Load on swift1 is WARNING: WARNING - load average: 3.58, 3.48, 3.05  
[11:45:42] <icinga-miraheze>	 RECOVERY - Current Load on swift1 is OK: OK - load average: 3.11, 3.35, 3.05  
[11:48:24] <Not-683e>	 [02miraheze/services] 07Reception123 pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fNUrV
[11:48:26] <Not-683e>	 [02miraheze/services] 07Reception123 03f48216f - add "xenharmonic" to services
[12:02:43] <icinga-miraheze>	 PROBLEM - Current Load on swift1 is CRITICAL: CRITICAL - load average: 4.08, 3.37, 3.02  
[12:21:32] <icinga-miraheze>	 PROBLEM - Varnish Backends on cp5 is CRITICAL: 1 backends are down.  mw2  
[12:21:42] <icinga-miraheze>	 PROBLEM - GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 2 datacenters are down: 172.104.111.8/cpweb, 2400:8902::f03c:91ff:fe07:444e/cpweb  
[12:23:32] <icinga-miraheze>	 RECOVERY - Varnish Backends on cp5 is OK: All 3 backends are healthy  
[12:23:42] <icinga-miraheze>	 RECOVERY - GDNSD Datacenters on misc1 is OK: OK - all datacenters are online  
[12:59:42] <icinga-miraheze>	 PROBLEM - Current Load on swift1 is WARNING: WARNING - load average: 3.41, 3.07, 2.93  
[13:05:43] <icinga-miraheze>	 RECOVERY - Current Load on swift1 is OK: OK - load average: 3.24, 3.34, 3.11  
[13:13:21] <icinga-miraheze>	 PROBLEM - Puppet on misc4 is WARNING: WARNING: Puppet is currently disabled, message: reason not specified, last run 2 minutes ago with 0 failures  
[13:43:41] <icinga-miraheze>	 PROBLEM - Current Load on swift1 is WARNING: WARNING - load average: 3.44, 3.33, 2.96  
[13:44:31] <icinga-miraheze>	 PROBLEM - Puppet on misc1 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Apt_key[Add key: F86AA916A2195E121AEDB11437BBEE3F7AD95B3F from Apt::Source grafana_apt]  
[13:47:42] <icinga-miraheze>	 RECOVERY - Current Load on swift1 is OK: OK - load average: 3.25, 3.34, 3.05  
[13:52:32] <icinga-miraheze>	 RECOVERY - Puppet on misc1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures  
[14:07:43] <Not-683e>	 [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fNUd4
[14:07:45] <Not-683e>	 [02miraheze/puppet] 07paladox 03405a7aa - Fix rewrite.py
[14:09:23] <icinga-miraheze>	 RECOVERY - Puppet on misc4 is OK: OK: Puppet is currently enabled, last run 24 seconds ago with 0 failures  
[14:25:22] <icinga-miraheze>	 PROBLEM - Puppet on misc4 is WARNING: WARNING: Puppet is currently disabled, message: reason not specified, last run 4 minutes ago with 0 failures  
[14:52:32] <icinga-miraheze>	 PROBLEM - JobQueue on mw3 is CRITICAL: JOBQUEUE CRITICAL - job queue greater than 300 jobs. Current queue: 2674  
[15:06:13] <Not-683e>	 [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fNUjg
[15:06:14] <Not-683e>	 [02miraheze/puppet] 07paladox 03e2f5d49 - More fixes to rewrite.py
[17:13:23] <icinga-miraheze>	 RECOVERY - Puppet on misc4 is OK: OK: Puppet is currently enabled, last run 6 seconds ago with 0 failures  
[18:01:42] <icinga-miraheze>	 PROBLEM - Current Load on swift1 is WARNING: WARNING - load average: 3.50, 3.07, 2.74  
[18:05:42] <icinga-miraheze>	 RECOVERY - Current Load on swift1 is OK: OK - load average: 2.89, 3.15, 2.86  
[18:07:22] <icinga-miraheze>	 PROBLEM - Puppet on misc4 is WARNING: WARNING: Puppet is currently disabled, message: reason not specified, last run 6 minutes ago with 0 failures  
[18:42:32] <icinga-miraheze>	 RECOVERY - JobQueue on mw3 is OK: JOBQUEUE OK - job queue below 300 jobs  
[18:43:42] <icinga-miraheze>	 PROBLEM - Current Load on swift1 is WARNING: WARNING - load average: 3.47, 3.17, 3.05  
[19:10:11] <Not-683e>	 [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fNTRZ
[19:10:12] <Not-683e>	 [02miraheze/puppet] 07paladox 03ec81248 - Some more fixes
[19:51:03] <icinga-miraheze>	 PROBLEM - wiki.ombre.io - LetsEncrypt on sslhost is WARNING: WARNING - Certificate wiki.ombre.io expires in 15 day(s) (Thu 26 Jul 2018 07:47:29 PM GMT +0000).  
[19:51:16] <Not-683e>	 [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fNTuK
[19:51:18] <Not-683e>	 [02miraheze/ssl] 07MirahezeSSLBot 0332adf6d - Bot: Update SSL cert for wiki.ombre.io
[19:53:37] <icinga-miraheze>	 PROBLEM - JobQueue on mw3 is CRITICAL: JOBQUEUE CRITICAL - job queue greater than 300 jobs. Current queue: 2631  
[20:02:56] <icinga-miraheze>	 RECOVERY - wiki.ombre.io - LetsEncrypt on sslhost is OK: OK - Certificate wiki.ombre.io will expire on Mon 08 Oct 2018 06:51:11 PM GMT +0000.  
[20:19:27] <icinga-miraheze>	 PROBLEM - wikiescola.com.br - LetsEncrypt on sslhost is WARNING: WARNING - Certificate wikiescola.com.br expires in 15 day(s) (Thu 26 Jul 2018 08:16:43 PM GMT +0000).  
[20:19:37] <Not-683e>	 [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fNT2u
[20:19:38] <Not-683e>	 [02miraheze/ssl] 07MirahezeSSLBot 03f8e2659 - Bot: Update SSL cert for wikiescola.com.br
[20:21:27] <icinga-miraheze>	 RECOVERY - wikiescola.com.br - LetsEncrypt on sslhost is OK: OK - Certificate wikiescola.com.br will expire on Mon 08 Oct 2018 07:19:31 PM GMT +0000.  
[20:49:27] <icinga-miraheze>	 RECOVERY - JobQueue on mw3 is OK: JOBQUEUE OK - job queue below 300 jobs  
[21:37:28] <icinga-miraheze>	 PROBLEM - JobQueue on mw3 is CRITICAL: JOBQUEUE CRITICAL - job queue greater than 300 jobs. Current queue: 5509  
[22:01:47] <icinga-miraheze>	 PROBLEM - Current Load on swift1 is WARNING: WARNING - load average: 3.75, 3.38, 3.07  
[22:03:47] <icinga-miraheze>	 RECOVERY - Current Load on swift1 is OK: OK - load average: 3.17, 3.37, 3.10  
[22:17:46] <icinga-miraheze>	 PROBLEM - Current Load on swift1 is WARNING: WARNING - load average: 3.91, 3.51, 3.21  
[22:59:10] * paladox has spent the whole day adding swift support in SocialProfile :)
[23:01:16] <Not-683e>	 [02mw-config] 07The-Voidwalker opened pull request 03#2281: add DP exemption to marketingspecialswiki - 13https://git.io/fNTSX
[23:01:47] <icinga-miraheze>	 PROBLEM - Current Load on swift1 is CRITICAL: CRITICAL - load average: 4.21, 4.05, 4.25  
[23:02:13] <Not-683e>	 [02mw-config] 07paladox closed pull request 03#2281: add DP exemption to marketingspecialswiki - 13https://git.io/fNTSX
[23:02:15] <Not-683e>	 [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fNTSD
[23:02:16] <Not-683e>	 [02miraheze/mw-config] 07The-Voidwalker 03f5c75eb - add DP exemption to marketingspecialswiki (#2281) per email to stewards@
[23:19:48] <icinga-miraheze>	 PROBLEM - Current Load on swift1 is WARNING: WARNING - load average: 2.50, 3.34, 3.87  
[23:27:47] <icinga-miraheze>	 PROBLEM - Current Load on swift1 is CRITICAL: CRITICAL - load average: 4.12, 3.62, 3.74  
[23:29:47] <icinga-miraheze>	 PROBLEM - Current Load on swift1 is WARNING: WARNING - load average: 3.53, 3.56, 3.70  
[23:35:27] <icinga-miraheze>	 RECOVERY - JobQueue on mw3 is OK: JOBQUEUE OK - job queue below 300 jobs  
[23:41:40] <Not-683e>	 [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fNTQp
[23:41:41] <Not-683e>	 [02miraheze/mw-config] 07paladox 031655b01 - Update LocalSettings.php
[23:43:04] <Not-683e>	 [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fNT7J
[23:43:05] <Not-683e>	 [02miraheze/mw-config] 07paladox 038b75fba - Set $wgSocialProfileFileBackend to miraheze-swift This variable is not yet in SocialProfile but will be soon if my change gets merged. https://gerrit.wikimedia.org/r/#/c/mediawiki/extensions/SocialProfile/+/444441/
[23:43:06] <ZppixBot>	 Title: [ Gerrit Code Review ] - gerrit.wikimedia.org
[23:58:09] <Not-683e>	 [02mw-config] 07paladox created branch 03paladox-patch-1 - 13https://git.io/vbvb3
[23:58:10] <Not-683e>	 [02miraheze/mw-config] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/fNT5L
[23:58:12] <Not-683e>	 [02miraheze/mw-config] 07paladox 03fb1b04b - Move file backend extension configs to FIleBackend.php
[23:58:13] <Not-683e>	 [02mw-config] 07paladox opened pull request 03#2282: Move file backend extension configs to FIleBackend.php - 13https://git.io/fNT5t
[23:58:50] <Not-683e>	 [02miraheze/mw-config] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/fNT5O
[23:58:52] <Not-683e>	 [02miraheze/mw-config] 07paladox 035f3a9e9 - Update FileBackend.php
[23:58:53] <Not-683e>	 [02mw-config] 07paladox synchronize pull request 03#2282: Move file backend extension configs to FIleBackend.php - 13https://git.io/fNT5t
[23:58:58] <Not-683e>	 [02mw-config] 07paladox closed pull request 03#2282: Move file backend extension configs to FIleBackend.php - 13https://git.io/fNT5t
[23:59:00] <Not-683e>	 [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±2] 13https://git.io/fNT53
[23:59:01] <Not-683e>	 [02miraheze/mw-config] 07paladox 0362e0f58 - Move file backend extension configs to FIleBackend.php (#2282) * Move file backend extension configs to FIleBackend.php * Update FileBackend.php
[23:59:03] <Not-683e>	 [02mw-config] 07paladox deleted branch 03paladox-patch-1 - 13https://git.io/vbvb3
[23:59:04] <Not-683e>	 [02miraheze/mw-config] 07paladox deleted branch 03paladox-patch-1
[23:59:42] <travis-ci>	 miraheze/mw-config/paladox-patch-1/5f3a9e9 - paladox The build was broken. https://travis-ci.org/miraheze/mw-config/builds/402456113