[00:04:03] <icinga-miraheze>	 PROBLEM - Puppet on db4 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues  
[00:12:02] <icinga-miraheze>	 RECOVERY - Puppet on db4 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures  
[00:23:51] <ZppixBot>	 Hello OlegCinema! If you have any questions feel free to ask and someone should answer soon.
[00:39:23] <icinga-miraheze>	 PROBLEM - Varnish Backends on cp2 is CRITICAL: 1 backends are down.  mw2  
[00:40:13] <icinga-miraheze>	 PROBLEM - GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 6 datacenters are down: 107.191.126.23/cpweb, 2604:180:0:33b::2/cpweb, 81.4.109.133/cpweb, 2a00:d880:5:8ea::ebc7/cpweb, 172.104.111.8/cpweb, 2400:8902::f03c:91ff:fe07:444e/cpweb  
[00:40:23] <icinga-miraheze>	 PROBLEM - Varnish Backends on cp5 is CRITICAL: 2 backends are down.  mw1 mw3  
[00:41:36] <Voidwalker>	 ^no bueno
[00:42:12] <icinga-miraheze>	 RECOVERY - GDNSD Datacenters on misc1 is OK: OK - all datacenters are online  
[00:43:22] <icinga-miraheze>	 RECOVERY - Varnish Backends on cp2 is OK: All 3 backends are healthy  
[00:44:22] <icinga-miraheze>	 RECOVERY - Varnish Backends on cp5 is OK: All 3 backends are healthy  
[01:25:37] <Not-683e>	 [02miraheze/mediawiki] 07paladox pushed 031 commit to 03REL1_31 [+0/-0/±1] 13https://git.io/f4S00
[01:25:38] <Not-683e>	 [02miraheze/mediawiki] 07Brad Jorsch 0385b5b1d - PHPSessionHandler: Suppress warnings in initialize() PHP 7.2 has gotten strict about calling various session-related methods after headers were sent. Even in CLI mode where there are no headers to send in the first place. Silence these warnings. Bug: T197030 Change-Id: Idaabf1320c56e0d6c26387f03af05f32e1496a1c (cherry picked from commit
[01:25:38] <Not-683e>	 701854b3ebc3c2b06067823395a7d95e8984cfda)
[01:33:33] <icinga-miraheze>	 PROBLEM - HTTP 4xx/5xx ERROR Rate on cp4 is WARNING: WARNING - NGINX Error Rate is 44%  
[01:35:33] <icinga-miraheze>	 RECOVERY - HTTP 4xx/5xx ERROR Rate on cp4 is OK: OK - NGINX Error Rate is 6%  
[01:42:42] <Not-683e>	 [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/f4S09
[01:42:44] <Not-683e>	 [02miraheze/mw-config] 07paladox 03408688b - Fix "Undefined index: editor"
[02:05:39] <Not-683e>	 [02miraheze/mediawiki] 07paladox pushed 031 commit to 03REL1_31 [+0/-0/±1] 13https://git.io/f4SEm
[02:05:40] <Not-683e>	 [02miraheze/mediawiki] 07paladox 03025a227 - Update DynamicPageList3
[03:53:22] <icinga-miraheze>	 PROBLEM - Puppet on cp4 is WARNING: WARNING: Puppet last ran 1 hour ago  
[04:01:22] <icinga-miraheze>	 RECOVERY - Puppet on cp4 is OK: OK: Puppet is currently enabled, last run 30 seconds ago with 0 failures  
[07:40:46] <ZppixBot>	 Hello The! If you have any questions feel free to ask and someone should answer soon.
[07:42:16] <The>	 Something is going wrong and I can't edit miraheze wikis anymore, with a message saying "You do not have permission to edit this page, for the following reason:  The action you have requested is limited to users in one of the groups: Administrators, Bureaucrats, Editors.  Are you sure that you're logged in?" displayed on screen. Can anyone tell me what's going on?
[07:42:16] <ZppixBot>	 The: 2018-04-17 - 01:27:37 <paladox> tell The bot can send as many messages too it seems! This is not spam.
[07:44:37] <The>	 I am not locked/blocked, but I can't even edit my user talk page.
[07:45:02] <The>	 Something must be going wrong.
[08:07:02] <videojeux4>	 oh
[08:15:29] <videojeux4>	 Reception123 ^ read the message of The
[08:15:56] <Reception123>	 The: Hello
[08:16:12] <Reception123>	 The: Which wiki is this on? What is your userpage?
[08:16:15] <Reception123>	 videojeux4: thanks for the ping
[08:16:37] <videojeux4>	 :)
[08:17:57] <The>	 meta, and any other wikis, except for those I have the permission as a bureaucrat or an editor.
[08:18:15] <The>	 My user page is User:開拓者
[08:18:34] <videojeux4>	 Reception123: https://github.com/miraheze/mw-config/commit/408688b924a2f1313f9f56bb7ed519230800d59e
[08:18:35] <ZppixBot>	 Title: [ Fix "Undefined index: editor" · miraheze/mw-config@408688b · GitHub ] - github.com
[08:19:24] <Reception123>	 checking now
[08:19:56] <Reception123>	 I can confirm that it doesn't work
[08:20:10] <Reception123>	 Will revert that change
[08:21:03] <The>	 thanks :)
[08:21:43] <Reception123>	 paladox: please make sure to check your changes on test1 first, or if not at least test to see if they worked as intended
[08:22:21] <Not-683e>	 [02miraheze/mw-config] 07Reception123 pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/f4Sw1
[08:22:22] <Not-683e>	 [02miraheze/mw-config] 07Reception123 03dbd9818 - Revert "Fix "Undefined index: editor"" This reverts commit 408688b924a2f1313f9f56bb7ed519230800d59e.
[08:22:36] <Reception123>	 The: you should be able to edit again in a few minutes
[08:26:18] <The>	 Confirmed. Thanks :)
[08:31:58] <paladox>	 Reception123: that causes a undefined error in the jobqueue
[08:32:16] <Reception123>	 how come it didn't before?
[08:32:26] <Reception123>	 + that's better than no edits for non-editors/crats/admins
[08:32:33] <Reception123>	 We need to find a solution that doesn't break editing
[08:32:54] <paladox>	 Reception123: it seems the error was showing in the jobqueue but not the debug logs
[08:33:11] <paladox>	 Which is probaly why we didn’t catch it :)
[08:33:20] <Reception123>	 but does the error cause the jobqueue not to work properly?
[08:33:40] <paladox>	 That I’m not sure of :)
[08:34:02] <paladox>	 I just wanted to fix the error so the jobqueue was clean
[08:34:47] <Reception123>	 ok
[08:34:57] <Reception123>	 then you should try to find a fix that doesn't break anything else :D
[08:36:11] <paladox>	 Heh, I tested that the wikis didn’t go down :), but I thought using isset was he same thing
[08:36:18] <paladox>	 But apparently not
[08:38:00] <Reception123>	 yeah, some groups could still edit but not user
[09:12:12] <icinga-miraheze>	 RECOVERY - JobQueue on mw3 is OK: JOBQUEUE OK - job queue below 300 jobs  
[10:20:23] <icinga-miraheze>	 PROBLEM - JobQueue on mw3 is CRITICAL: JOBQUEUE CRITICAL - job queue greater than 300 jobs. Current queue: 5411  
[10:50:37] <AudiHugo>	 Hello
[10:50:52] <AudiHugo>	 Thanks for putting our wiki in, and sorry if I disturbed you :/
[11:30:21] <MH-Discord>	 <sau226> You're always welcome. No problem.
[11:35:43] <MH-Discord>	 <sau226> Requesting steward: Grant wikicreator to https://meta.miraheze.org/wiki/Special:UserRights/OlegCinema as per https://meta.miraheze.org/wiki/Stewards%27_noticeboard
[11:35:44] <ZppixBot>	 Title: [ User rights - Miraheze Meta ] - meta.miraheze.org
[11:35:45] <ZppixBot>	 Title: [ Stewards' noticeboard - Miraheze Meta ] - meta.miraheze.org
[12:04:13] <icinga-miraheze>	 PROBLEM - GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 2 datacenters are down: 2604:180:0:33b::2/cpweb, 81.4.109.133/cpweb  
[12:06:13] <icinga-miraheze>	 RECOVERY - GDNSD Datacenters on ns1 is OK: OK - all datacenters are online  
[12:23:46] <paladox>	 Reception123 hmm jobqueue went up again
[12:24:31] <paladox>	  #011PHP Notice:  Undefined index: editor in /srv/mediawiki/config/LocalWiki.php on line 439
[12:31:46] <paladox>	 Reception123 im thinking that is what is causing the high queue
[12:35:08] <paladox>	 !log sudo -u www-data php /srv/mediawiki/w/maint*/rebuildLocalisationCache.php --wiki test1wiki on mw3
[12:35:15] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master
[12:39:20] <paladox>	 !log sudo -u www-data php /srv/mediawiki/w/maint*/rebuildLocalisationCache.php --wiki test1wiki on mw*
[12:39:26] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master
[13:17:53] <Not-683e>	 [02miraheze/mediawiki] 07paladox pushed 031 commit to 03REL1_31 [+0/-0/±1] 13https://git.io/f4S18
[13:17:55] <Not-683e>	 [02miraheze/mediawiki] 07paladox 0326e4ffa - Update Comments to c305e4d7a48affde0edb698dfd240a89201bb33b
[13:23:17] <Not-683e>	 [02miraheze/mediawiki] 07paladox pushed 031 commit to 03REL1_31 [+0/-0/±1] 13https://git.io/f4S1a
[13:23:19] <Not-683e>	 [02miraheze/mediawiki] 07paladox 03491a014 - Update Comments to f27ca4eb68e3d92b9af5f92f76477221ad1079ed
[13:37:51] <Not-683e>	 [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/f4S1p
[13:37:53] <Not-683e>	 [02miraheze/mw-config] 07paladox 03ea75c6d - Fix "Undefined index: editor"
[13:39:24] <paladox>	 Reception123 ^^
[13:45:09] <Not-683e>	 [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/f4SMs
[13:45:11] <Not-683e>	 [02miraheze/puppet] 07paladox 03868fc57 - Switch to Restart=always
[13:45:18] <Not-683e>	 [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/f4SMZ
[13:45:20] <Not-683e>	 [02miraheze/puppet] 07paladox 0370c6d13 - Switch to Restart=always
[14:02:01] <Not-683e>	 [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/f4SMD
[14:02:03] <Not-683e>	 [02miraheze/puppet] 07paladox 03912d951 - Switch mw2 back into a jobrunner T3260
[14:03:03] <icinga-miraheze>	 PROBLEM - Current Load on misc1 is WARNING: WARNING - load average: 1.71, 1.51, 1.19  
[14:04:26] <Reception123>	 paladox: can you confirm that it works?
[14:04:32] <paladox>	 Reception123 yep
[14:04:38] <paladox>	 i tryed it as a anon
[14:04:41] <paladox>	 on metawiki
[14:05:07] <icinga-miraheze>	 RECOVERY - Current Load on misc1 is OK: OK - load average: 1.68, 1.50, 1.22  
[14:05:15] <Reception123>	 ok
[14:05:23] <Reception123>	 paladox: what about as a user?
[14:05:37] <Reception123>	 I have a test account which I use for testing such things, as my main account of course has sysadmin permissions
[14:05:52] <paladox>	 Reception123 could you test with your test account please?
[14:05:57] <Reception123>	 sure
[14:06:15] <icinga-miraheze>	 PROBLEM - JobChron Service on mw2 is CRITICAL: PROCS CRITICAL: 0 processes with args redisJobChronService  
[14:06:45] <icinga-miraheze>	 RECOVERY - JobChron Service on mw2 is OK: PROCS OK: 1 process with args redisJobChronService  
[14:06:46] <Reception123>	 ok, no problems
[14:06:47] <icinga-miraheze>	 PROBLEM - JobQueue on mw2 is CRITICAL: JOBQUEUE CRITICAL - job queue greater than 300 jobs. Current queue: 13458  
[14:06:55] <Reception123>	 except those ^ :P
[14:06:58] <Reception123>	 13458 what
[14:07:17] <paladox>	 Reception123 yep
[14:07:21] <Reception123>	 !log restarted mw2 jobrunner
[14:07:24] <paladox>	 i am trying to deal with the high jobqueue
[14:07:25] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master
[14:07:48] <paladox>	 Reception123 i just turned mw2 into a jobrunner ;D
[14:07:49] <paladox>	 see:
[14:07:50] <paladox>	 https://phabricator.miraheze.org/T3260
[14:07:51] <ZppixBot>	 Title: [ ⚓ T3260 JobQueue needs looking at ] - phabricator.miraheze.org
[14:08:09] <Reception123>	 !log manually run sudo -u www-data foreachwikiindblist /srv/mediawiki/dblist/all.dblist /srv/mediawiki/w/maintenance/runJobs.php on mw1 to alleviate queue
[14:08:13] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master
[14:08:18] <Reception123>	 ok
[14:33:42] <Not-683e>	 [02miraheze/mediawiki] 07paladox pushed 031 commit to 03REL1_31 [+0/-0/±1] 13https://git.io/f4SyT
[14:33:44] <Not-683e>	 [02miraheze/mediawiki] 07paladox 0342216db - Update CreateRedirect to eb29c1c5610477d847690cb2399e67da39a1340d
[15:01:33] <Not-683e>	 [02miraheze/services] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/f4SS6
[15:01:35] <Not-683e>	 [02miraheze/services] 07paladox 03333511c - Enable services for uzencwiki fixes T3267
[15:25:16] <icinga-miraheze>	 PROBLEM - studentwiki.ddns.net - LetsEncrypt on sslhost is WARNING: WARNING - Certificate studentwiki.ddns.net expires in 15 day(s) (Tue 10 Jul 2018 03:21:19 PM GMT +0000).  
[15:25:26] <Not-683e>	 [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/f4S91
[15:25:27] <Not-683e>	 [02miraheze/ssl] 07MirahezeSSLBot 030e86557 - Bot: Update SSL cert for studentwiki.ddns.net
[15:34:16] <Reception123>	 paladox: too bad we can't make it 5 days 
[15:34:27] <paladox>	 Reception123 hmm why?
[15:34:30] <paladox>	 best to keep it 15
[15:34:36] <paladox>	 in case needs a intervention
[15:34:57] <Reception123>	 yeah but 15 is a lot, 5 days is usually enough
[15:35:07] <Reception123>	 + that means we're in reality getting 2m15d certs instead of 3m
[15:35:20] <paladox>	 yeh
[16:04:26] <icinga-miraheze>	 PROBLEM - Puppet on misc2 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Apt_key[Add key: F86AA916A2195E121AEDB11437BBEE3F7AD95B3F from Apt::Source grafana_apt]  
[16:11:38] <Reception123>	 ^ paladox 
[16:12:27] <icinga-miraheze>	 RECOVERY - Puppet on misc2 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures  
[16:12:29] <paladox>	 Reception123 that will recover
[16:12:32] <paladox>	 keeps happening
[16:13:33] <Reception123>	 ok
[16:27:56] <icinga-miraheze>	 PROBLEM - Current Load on misc1 is WARNING: WARNING - load average: 1.49, 1.73, 1.53  
[16:29:58] <icinga-miraheze>	 RECOVERY - Current Load on misc1 is OK: OK - load average: 1.63, 1.70, 1.54  
[16:32:46] <icinga-miraheze>	 RECOVERY - studentwiki.ddns.net - LetsEncrypt on sslhost is OK: OK - Certificate studentwiki.ddns.net will expire on Sat 22 Sep 2018 02:25:21 PM GMT +0000.  
[16:33:56] <icinga-miraheze>	 PROBLEM - Current Load on misc1 is WARNING: WARNING - load average: 1.53, 1.91, 1.69  
[17:07:08] <Not-683e>	 [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/f4S72
[17:07:10] <Not-683e>	 [02miraheze/mw-config] 07paladox 03d52fc89 - Add + in front of wishwiki Fix is by @Void
[17:31:56] <icinga-miraheze>	 PROBLEM - Current Load on misc1 is WARNING: WARNING - load average: 1.91, 1.63, 1.29  
[17:33:56] <icinga-miraheze>	 RECOVERY - Current Load on misc1 is OK: OK - load average: 1.27, 1.48, 1.27  
[17:43:57] <icinga-miraheze>	 PROBLEM - Current Load on misc1 is WARNING: WARNING - load average: 1.61, 1.82, 1.52  
[20:23:56] <Yatalu>	 hi
[20:24:08] <Yatalu>	 question: where do I find how to put a wordmark again?
[21:27:34] <Not-683e>	 [02mw-config] 07paladox created branch 03paladox-patch-1 - 13https://git.io/vbvb3
[21:27:35] <Not-683e>	 [02miraheze/mw-config] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/f4SjU
[21:27:37] <Not-683e>	 [02miraheze/mw-config] 07paladox 0321cb8b6 - Switch to Swift for file uploads / viewing files
[21:27:38] <Not-683e>	 [02mw-config] 07paladox opened pull request 03#2270: Switch to Swift for file uploads / viewing files - 13https://git.io/f4SjT
[21:28:20] <travis-ci>	 miraheze/mw-config/paladox-patch-1/21cb8b6 - paladox The build was fixed. https://travis-ci.org/miraheze/mw-config/builds/396174041
[21:29:17] <Not-683e>	 [02miraheze/mw-config] 07paladox pushed 031 commit to 03paladox-patch-1 [+1/-0/±0] 13https://git.io/f4SjI
[21:29:19] <Not-683e>	 [02miraheze/mw-config] 07paladox 036324c22 - Create FileBackend.php
[21:29:20] <Not-683e>	 [02mw-config] 07paladox synchronize pull request 03#2270: Switch to Swift for file uploads / viewing files - 13https://git.io/f4SjT
[21:46:23] <MH-Discord>	 <GOTILON> https://pirategalaxy.miraheze.org/wiki/User:GOTILON
[21:46:23] <ZppixBot>	 Title: [ User:GOTILON - Pirate Galaxy Wiki ] - pirategalaxy.miraheze.org
[21:46:31] <MH-Discord>	 <GOTILON> Wikipedia
[22:33:37] <icinga-miraheze>	 PROBLEM - Puppet on cp4 is WARNING: WARNING: Puppet last ran 1 hour ago  
[22:33:58] <paladox>	 !log reboot cp4 due to OOM
[22:34:01] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master
[22:41:35] <icinga-miraheze>	 RECOVERY - Puppet on cp4 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures  
[23:13:05] <icinga-miraheze>	 RECOVERY - JobQueue on mw2 is OK: JOBQUEUE OK - job queue below 300 jobs  
[23:13:07] <icinga-miraheze>	 RECOVERY - JobQueue on mw3 is OK: JOBQUEUE OK - job queue below 300 jobs  
[23:24:26] <icinga-miraheze>	 PROBLEM - Puppet on misc2 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Apt_key[Add key: F86AA916A2195E121AEDB11437BBEE3F7AD95B3F from Apt::Source grafana_apt]  
[23:28:49] <Not-683e>	 [02mw-config] 07paladox synchronize pull request 03#2270: Switch to Swift for file uploads / viewing files - 13https://git.io/f4SjT
[23:28:51] <Not-683e>	 [02miraheze/mw-config] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/f49Ju
[23:28:52] <Not-683e>	 [02miraheze/mw-config] 07paladox 03367d17a - Update FileBackend.php
[23:32:26] <icinga-miraheze>	 RECOVERY - Puppet on misc2 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures  
[23:58:18] <Not-683e>	 [02puppet] 07paladox synchronize pull request 03#759: Update static.miraheze.org to use swift instead of nfs - 13https://git.io/f4yoV
[23:58:19] <Not-683e>	 [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-2 [+0/-0/±1] 13https://git.io/f49UV
[23:58:21] <Not-683e>	 [02miraheze/puppet] 07paladox 03884d977 - Update mediawiki.conf