[00:11:48] <icinga-miraheze>	 RECOVERY - Puppet on test1 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures  
[01:00:48] <NeoBogan>	 .tell labster I was about to say welcome back, but then you quit you bloody quitter!
[01:00:48] <ZppixBot>	 NeoBogan: I'll pass that on when labster is around.
[01:33:09] <ZppixBot>	 Hello Voidwalker|away! If you have any questions feel free to ask and someone should answer soon.
[04:33:39] <icinga-miraheze>	 PROBLEM - Current Load on misc1 is WARNING: WARNING - load average: 1.98, 1.48, 1.01  
[04:37:37] <icinga-miraheze>	 RECOVERY - Current Load on misc1 is OK: OK - load average: 1.50, 1.42, 1.08  
[04:43:38] <icinga-miraheze>	 PROBLEM - Current Load on misc1 is WARNING: WARNING - load average: 1.94, 1.68, 1.29  
[10:48:08] <icinga-miraheze>	 PROBLEM - JobQueue on mw3 is CRITICAL: JOBQUEUE CRITICAL - job queue greater than 300 jobs. Current queue: 1049  
[11:06:08] <icinga-miraheze>	 RECOVERY - JobQueue on mw3 is OK: JOBQUEUE OK - job queue below 300 jobs  
[12:44:08] <icinga-miraheze>	 PROBLEM - Puppet on misc2 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Apt_key[Add key: F86AA916A2195E121AEDB11437BBEE3F7AD95B3F from Apt::Source grafana_apt]  
[12:52:08] <icinga-miraheze>	 RECOVERY - Puppet on misc2 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures  
[12:55:09] <MacFan4000>	 I may try GlobalUserPage again today
[13:04:12] <Not-683e>	 [02dns] 07paladox created branch 03paladox-patch-1 - 13https://git.io/vbQXl
[13:04:13] <Not-683e>	 [02miraheze/dns] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/f4DXE
[13:04:15] <Not-683e>	 [02miraheze/dns] 07paladox 0341f08f6 - Add swift-lb will be used to proxy over https to a swift frontend
[13:04:16] <Not-683e>	 [02dns] 07paladox opened pull request 03#91: Add swift-lb will be used to proxy over https to a swift frontend - 13https://git.io/f4DXu
[13:05:42] <travis-ci>	 miraheze/dns/paladox-patch-1/41f08f6 - paladox The build failed. https://travis-ci.org/miraheze/dns/builds/395457302
[13:05:54] <Not-683e>	 [02miraheze/dns] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/f4DXo
[13:05:56] <Not-683e>	 [02miraheze/dns] 07paladox 031da12d4 - Update config
[13:05:57] <Not-683e>	 [02dns] 07paladox synchronize pull request 03#91: Add swift-lb will be used to proxy over https to a swift frontend - 13https://git.io/f4DXu
[13:09:02] <travis-ci>	 miraheze/dns/paladox-patch-1/1da12d4 - paladox The build was fixed. https://travis-ci.org/miraheze/dns/builds/395457879
[13:09:17] <Not-683e>	 [02dns] 07paladox closed pull request 03#91: Add swift-lb will be used to proxy over https to a swift frontend - 13https://git.io/f4DXu
[13:09:18] <Not-683e>	 [02miraheze/dns] 07paladox pushed 031 commit to 03master [+0/-0/±2] 13https://git.io/f4DXb
[13:09:20] <Not-683e>	 [02miraheze/dns] 07paladox 038b5b8d4 - Add swift-lb will be used to proxy over https to a swift frontend (#91) * Add swift-lb will be used to proxy over https to a swift frontend * Update config
[13:09:21] <Not-683e>	 [02dns] 07paladox deleted branch 03paladox-patch-1 - 13https://git.io/vbQXl
[13:09:23] <Not-683e>	 [02miraheze/dns] 07paladox deleted branch 03paladox-patch-1
[13:50:10] <icinga-miraheze>	 PROBLEM - JobQueue on mw3 is CRITICAL: JOBQUEUE CRITICAL - job queue greater than 300 jobs. Current queue: 552  
[13:54:08] <icinga-miraheze>	 RECOVERY - JobQueue on mw3 is OK: JOBQUEUE OK - job queue below 300 jobs  
[14:07:04] <Not-683e>	 [02miraheze/mediawiki] 07MacFan4000 pushed 032 commits to 03REL1_31 [+1/-0/±7] 13https://git.io/f4DHg
[14:07:06] <Not-683e>	 [02miraheze/mediawiki] 07MacFan4000 034b1df63 - Update GUP to my form
[14:07:07] <Not-683e>	 [02miraheze/mediawiki] 07MacFan4000 03c3d8c0d - Merge branch 'REL1_31' of https://github.com/miraheze/mediawiki into REL1_31
[14:07:08] <ZppixBot>	 Title: [ GitHub - miraheze/mediawiki: The collaborative editing software that runs Wikipedia. ] - github.com
[14:07:25] <Not-683e>	 [02mediawiki] 07MacFan4000 commented on commit 034b1df632f84dd50e5db71a11d04defdea43809c7 - 13https://git.io/f4DHw
[14:09:33] <Not-683e>	 [02miraheze/mediawiki] 07MacFan4000 pushed 031 commit to 03REL1_31 [+0/-0/±1] 13https://git.io/f4DH5
[14:09:34] <Not-683e>	 [02miraheze/mediawiki] 07MacFan4000 03a65b183 - Auctually update
[14:09:43] <paladox>	 That is going to break submodule a
[14:09:47] <paladox>	 Submodules 
[14:15:20] <icinga-miraheze>	 PROBLEM - Puppet on mw1 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki core]  
[14:15:38] <icinga-miraheze>	 PROBLEM - Puppet on mw3 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 25 seconds ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki core]  
[14:16:08] <icinga-miraheze>	 PROBLEM - Puppet on mw2 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 29 seconds ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki core]  
[14:16:19] <paladox>	 MacFan4000: ^^
[14:16:36] <MacFan4000>	 ack, running git pull manually
[14:17:06] <paladox>	 It’s the submodule that’s broken 
[14:17:12] <paladox>	 You changed the url
[14:17:37] <paladox>	 So you will have to remove the w folder and reclone it
[14:17:44] <paladox>	 MacFan4000: ^^
[14:17:58] <icinga-miraheze>	 PROBLEM - Puppet on test1 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki core]  
[14:18:14] <MacFan4000>	 Im running git pull manually also with --recurse-submodules and its not having any issues
[14:18:48] <paladox>	 git submodule update
[14:19:49] <MacFan4000>	 Though right now I'm only running these commands on mw3
[14:20:50] <paladox>	 Run puppet?
[14:21:09] <paladox>	 Will Lilkly fail with it cannot find commits as it would still be using wikimedia
[14:21:51] <paladox>	 MacFan4000: ^^
[14:22:28] <MacFan4000>	 I think we should probs just reclone
[14:22:41] <MacFan4000>	 Though I can't depot servers
[14:22:58] <MacFan4000>	 s/depot/depool
[14:22:58] <ZppixBot>	 MacFan4000 meant to say: Though I can't depool servers
[14:23:01] <paladox>	 You will need to wait for an op
[14:23:04] <paladox>	 To do that
[14:23:07] <MacFan4000>	 like you
[14:23:27] <paladox>	 I carnt at the moment
[14:23:30] <paladox>	 At college 
[14:23:33] <MacFan4000>	 ok
[14:24:52] <MacFan4000>	 hough doesn't varnish auto-depool when there is an error?
[14:25:01] <MacFan4000>	 s/hough/though
[14:25:01] <ZppixBot>	 MacFan4000 meant to say: though doesn't varnish auto-depool when there is an error?
[14:25:28] <paladox>	 MacFan4000: doint try it without ops please :)
[14:25:52] <paladox>	 And yes it does but can take a few secs, so users will see the error then get switched to another mw
[14:26:53] <paladox>	 MacFan4000: have you tested this ext to make sure it dosent break?
[14:27:14] <MacFan4000>	 no, but I have no way to test it
[14:27:51] <paladox>	 MacFan4000: on your local install? Or test1.
[14:28:37] <MacFan4000>	 I was preparing to test it on wikis such as testwiki, test1wiki
[14:29:06] <paladox>	 Yeh but switching submodules is a long task :)
[14:29:11] <paladox>	 Adding them is easy
[14:29:23] <paladox>	 But changing them or removing them requires more work
[14:29:34] <Wiki-1776>	 Hello paladox MacFan4000 :)
[14:29:42] <paladox>	 Hi
[14:30:19] <paladox>	 MacFan4000: is it disabled by default?
[14:30:27] <paladox>	 If not it has to be for stability reasons
[14:30:42] <MacFan4000>	 Its not enabled at all anywhere
[14:31:22] <paladox>	 Ok
[14:31:37] <MacFan4000>	 I would ask JohnLewis to help but he's not here ATM
[14:33:38] <paladox>	 Ok
[14:40:36] <MacFan4000>	 https://meta.miraheze.org/wiki/Talk:FAQ#How_to_set_a_new_logo_and_favicon.2C_exactly.3F
[14:40:37] <ZppixBot>	 Title: [ Talk:FAQ - Miraheze Meta ] - meta.miraheze.org
[14:40:51] <MacFan4000>	 There’s a who
[14:41:22] <MacFan4000>	 s/who/whole page documenting that :P
[14:41:22] <ZppixBot>	 MacFan4000 meant to say: There’s a whole page documenting that :P
[15:03:39] <icinga-miraheze>	 RECOVERY - Puppet on mw3 is OK: OK: Puppet is currently enabled, last run 49 seconds ago with 0 failures  
[15:03:54] <MacFan4000>	 I fixed it :)
[15:04:03] <MacFan4000>	 I did git submodule sync
[15:11:19] <icinga-miraheze>	 RECOVERY - Puppet on mw1 is OK: OK: Puppet is currently enabled, last run 38 seconds ago with 0 failures  
[15:11:47] <icinga-miraheze>	 RECOVERY - Puppet on test1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures  
[15:12:07] <icinga-miraheze>	 RECOVERY - Puppet on mw2 is OK: OK: Puppet is currently enabled, last run 23 seconds ago with 0 failures  
[15:12:12] <MacFan4000>	 yay
[15:12:27] <MacFan4000>	 paladox:^^
[15:13:21] <MacFan4000>	 I did git submodule sync
[15:13:31] <MacFan4000>	 then git submodule update
[15:13:37] <MacFan4000>	 than ran puppet
[15:13:42] <MacFan4000>	 and bam, error fixed
[15:19:16] <paladox>	 Ok
[15:20:40] <MacFan4000>	 this proves that recloning is not necessary when updating submodule urls
[15:21:00] <paladox>	 What if
[15:21:09] <paladox>	 What if you removed a submodule?
[15:21:16] <paladox>	 Docs only says it updates the url
[15:22:12] <MacFan4000>	 Im sure that there is a way
[15:22:25] <MacFan4000>	 though I could be wrong
[15:22:55] <paladox>	 They were discussing it @ on a wmf change
[15:23:04] <paladox>	 As they want to remove a submodule from puppet
[15:23:18] <paladox>	 But it will break all local puppet masters in the cloud
[15:23:31] <paladox>	 Since git will fail
[15:23:48] <paladox>	 By git I mean git submodule 
[15:24:44] <Not-683e>	 [02mw-config] 07MacFan4000 created branch 03REL1_31 - 13https://git.io/vbvb3
[15:24:45] <Not-683e>	 [02miraheze/mw-config] 07MacFan4000 pushed 0399 commits to 03REL1_31 [+2/-0/±141] 13https://git.io/f4DhX
[15:24:47] <Not-683e>	 [02miraheze/mw-config] 07MacFan4000 035303d11 - Update
[15:24:48] <Not-683e>	 [02miraheze/mw-config] 07paladox 035955b2d - Fix extension name
[15:24:50] <Not-683e>	 [02miraheze/mw-config] 07MacFan4000 035d3e580 - Update LocalSettings.php
[15:24:51] <Not-683e>	 [02miraheze/mw-config] ... and 96 more commits.
[15:25:02] <MacFan4000>	 grrrr
[15:25:15] <MacFan4000>	 I was on the wrong branch
[15:25:58] <Not-683e>	 [02mw-config] 07MacFan4000 deleted branch 03REL1_31 - 13https://git.io/vbvb3
[15:25:59] <Not-683e>	 [02miraheze/mw-config] 07MacFan4000 deleted branch 03REL1_31
[15:26:59] <travis-ci>	 miraheze/mw-config/REL1_31/877a1be - MacFan4000 The build was broken. https://travis-ci.org/miraheze/mw-config/builds/395514428
[15:31:21] <Not-683e>	 [02miraheze/mw-config] 07MacFan4000 pushed 031 commit to 03master [+0/-0/±2] 13https://git.io/f4Dj2
[15:31:22] <Not-683e>	 [02miraheze/mw-config] 07MacFan4000 0316ffb18 - Enable GUP on select wikis
[15:31:54] <Not-683e>	 [02miraheze/mw-config] 07MacFan4000 pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/f4DjP
[15:31:56] <Not-683e>	 [02miraheze/mw-config] 07MacFan4000 031b8ce1d - Update LocalExtensions.php
[15:31:59] <paladox>	 MacFan4000 no metawiki until it works on test wiki's please!
[15:32:47] <MacFan4000>	 that would require another config change as metawiki is currently set as the central wiki
[15:33:11] <paladox>	 ok
[15:33:43] <MacFan4000>	 and nothing broke
[15:35:44] <MacFan4000>	 and the GlobalPrefs integration works
[15:49:09] <Not-683e>	 [02miraheze/mw-config] 07MacFan4000 pushed 033 commits to 03master [+0/-0/±4] 13https://git.io/f4yv1
[15:49:11] <Not-683e>	 [02miraheze/mw-config] 07MacFan4000 03e6e561d - Revert "Update LocalExtensions.php" This reverts commit 1b8ce1d4e130d4eb8a2327368895c266b7503d49.
[15:49:12] <Not-683e>	 [02miraheze/mw-config] 07MacFan4000 0390b316e - Revert "Enable GUP on select wikis" This reverts commit 16ffb18aed3e3a5cbbe48e5fe0102c8582e36514.
[15:49:14] <Not-683e>	 [02miraheze/mw-config] 07MacFan4000 0373fb8c2 - Push to global
[15:54:05] <MacFan4000>	 !log sudo -u www-data php rebuildLocalisationCache.php --wiki test1wiki on mw* and test1
[15:54:10] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master
[16:02:39] <MacFan4000>	 A big phab task now resolved. :)
[16:04:28] <icinga-miraheze>	 PROBLEM - Current Load on test1 is CRITICAL: CRITICAL - load average: 2.09, 1.73, 1.18  
[16:05:04] <MacFan4000>	 Thats to be expected when running LC on test1
[16:05:35] <MacFan4000>	 Since 1 core is slower then 2
[16:08:29] <icinga-miraheze>	 PROBLEM - Current Load on test1 is WARNING: WARNING - load average: 1.50, 1.74, 1.32  
[16:09:59] <icinga-miraheze>	 PROBLEM - Puppet on test1 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki core]  
[16:14:29] <icinga-miraheze>	 PROBLEM - Current Load on test1 is CRITICAL: CRITICAL - load average: 2.38, 2.05, 1.56  
[16:17:49] <icinga-miraheze>	 PROBLEM - Puppet on test1 is WARNING: WARNING: Puppet is currently disabled, message: swift, last run 1 minute ago with 1 failures  
[16:45:19] <icinga-miraheze>	 PROBLEM - JobQueue on mw3 is CRITICAL: JOBQUEUE CRITICAL - job queue greater than 300 jobs. Current queue: 2638  
[16:46:59] <paladox>	 2638?
[16:48:15] <MacFan4000>	 I wonder which wiki?
[16:50:15] <paladox>	 maybe greenleftwiki ?
[16:52:06] <MacFan4000>	 That had like 1-2 jobs
[16:52:16] <MacFan4000>	 Just ran them
[17:00:42] <MacFan4000>	 !log restarted mw3 JobRunner and running jobs on all wikis
[17:00:53] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master
[17:10:38] <icinga-miraheze>	 PROBLEM - Current Load on misc1 is CRITICAL: CRITICAL - load average: 2.16, 1.65, 1.23  
[17:12:38] <icinga-miraheze>	 RECOVERY - Current Load on misc1 is OK: OK - load average: 1.40, 1.57, 1.25  
[17:49:09] <icinga-miraheze>	 RECOVERY - JobQueue on mw3 is OK: JOBQUEUE OK - job queue below 300 jobs  
[17:51:10] <MacFan4000>	 That’s better ^
[17:53:33] <MacFan4000>	 And done running jobs
[18:04:08] <icinga-miraheze>	 PROBLEM - Puppet on misc2 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Apt_key[Add key: F86AA916A2195E121AEDB11437BBEE3F7AD95B3F from Apt::Source grafana_apt]  
[18:12:08] <icinga-miraheze>	 RECOVERY - Puppet on misc2 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures  
[18:22:39] <icinga-miraheze>	 PROBLEM - Current Load on misc1 is WARNING: WARNING - load average: 1.94, 1.52, 1.06  
[18:24:39] <icinga-miraheze>	 PROBLEM - Current Load on misc1 is CRITICAL: CRITICAL - load average: 2.17, 1.68, 1.16  
[18:28:37] <icinga-miraheze>	 PROBLEM - Current Load on misc1 is WARNING: WARNING - load average: 1.77, 1.81, 1.33  
[20:21:13] <Not-683e>	 [02miraheze/mw-config] 07Reception123 pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/f4yuY
[20:21:14] <Not-683e>	 [02miraheze/mw-config] 07Reception123 03c561cdd - add linenwiki to DP exemptions
[20:31:53] <Not-683e>	 [02miraheze/mediawiki] 07paladox pushed 031 commit to 03REL1_31 [+0/-0/±1] 13https://git.io/f4yzE
[20:31:54] <Not-683e>	 [02miraheze/mediawiki] 07paladox 0380374d2 - Update CookieWarning to 67c7aa56c32520ecc5c46beeffa754a63a01a06c
[20:36:46] <Not-683e>	 [02miraheze/mediawiki] 07paladox pushed 031 commit to 03REL1_31 [+0/-0/±1] 13https://git.io/f4ygL
[20:36:47] <Not-683e>	 [02miraheze/mediawiki] 07paladox 031703964 - Update Duplicator to 9893d8c81ea22fadb4080c245c302ffde811f9ac
[20:44:07] <icinga-miraheze>	 PROBLEM - Puppet on misc2 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Apt_key[Add key: F86AA916A2195E121AEDB11437BBEE3F7AD95B3F from Apt::Source grafana_apt]  
[20:52:08] <icinga-miraheze>	 RECOVERY - Puppet on misc2 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures  
[21:32:18] <Not-683e>	 [02miraheze/mediawiki] 07paladox pushed 031 commit to 03REL1_31 [+0/-0/±1] 13https://git.io/f4ywp
[21:32:19] <Not-683e>	 [02miraheze/mediawiki] 07paladox 03685284b - Update News to 3081a02faf848313abf9290601e4e57a615a1127
[21:50:50] <Not-683e>	 [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-2 [+0/-0/±1] 13https://git.io/f4yoa
[21:50:51] <Not-683e>	 [02miraheze/puppet] 07paladox 0345563c6 - Update static.miraheze.org to use swift instead of nfs
[21:50:53] <Not-683e>	 [02puppet] 07paladox created branch 03paladox-patch-2 - 13https://git.io/vbiAS
[21:50:54] <Not-683e>	 [02puppet] 07paladox opened pull request 03#759: Update static.miraheze.org to use swift instead of nfs - 13https://git.io/f4yoV
[21:51:54] <Not-683e>	 [02puppet] 07paladox synchronize pull request 03#759: Update static.miraheze.org to use swift instead of nfs - 13https://git.io/f4yoV
[21:51:55] <Not-683e>	 [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-2 [+0/-0/±1] 13https://git.io/f4yoP
[21:51:57] <Not-683e>	 [02miraheze/puppet] 07paladox 0384c40f7 - Update mediawiki.conf
[21:54:21] <Not-683e>	 [02puppet] 07paladox synchronize pull request 03#759: Update static.miraheze.org to use swift instead of nfs - 13https://git.io/f4yoV
[21:54:22] <Not-683e>	 [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-2 [+0/-0/±1] 13https://git.io/f4yox
[21:54:24] <Not-683e>	 [02miraheze/puppet] 07paladox 03ac1e5d1 - Update mediawiki.conf
[22:42:39] <icinga-miraheze>	 PROBLEM - Current Load on misc1 is WARNING: WARNING - load average: 1.88, 1.40, 1.03  
[22:44:37] <icinga-miraheze>	 RECOVERY - Current Load on misc1 is OK: OK - load average: 1.18, 1.26, 1.02  
[23:34:08] <icinga-miraheze>	 PROBLEM - Puppet on misc2 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Apt_key[Add key: F86AA916A2195E121AEDB11437BBEE3F7AD95B3F from Apt::Source grafana_apt]  
[23:42:09] <icinga-miraheze>	 RECOVERY - Puppet on misc2 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures