[00:09:49] <MacFan4000>	 So, I sent a MassMessage for site updates, but after I realized I forgot to change the month
[00:10:14] <MacFan4000>	 so instead of september it says july
[00:10:58] <Voidwalker>	 rip
[00:11:41] <MacFan4000>	 i am not going to try to edit the mass amount of talk pages on the list
[00:11:51] <icinga-miraheze>	 PROBLEM - db4 Disk Space on db4 is WARNING: DISK WARNING - free space: / 76924 MB (20% inode=94%);  
[00:12:12] <AmandaCatherine>	 This is where MassEditRegex or a similar extension would come in handy
[00:12:51] <MacFan4000>	 issues the pages would have another entry for july that we want to keep intact
[00:13:00] <Voidwalker>	 rip Jobs "metawiki * [[User:ChroniclesXII]]"
[00:14:11] <MacFan4000>	 !log running jobs everywhere in a screen
[00:14:15] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master
[00:14:16] <AmandaCatherine>	 Voidwalker: ? They created a user page
[00:14:23] <AmandaCatherine>	 Why is that a problem
[00:14:44] <MacFan4000>	 because any edit to a user page on meta creates globaluserpage jobs
[00:14:48] <Voidwalker>	 it creates around 2-3 thousand jobs
[00:15:02] <Voidwalker>	 JobQueue goes critical at 300
[00:15:11] <MacFan4000>	 but that is not the current issue just to clarify
[00:15:12] <AmandaCatherine>	 Depends on how many wikis are attached to the global account, though?
[00:15:45] <Voidwalker>	 Current queue: 6626, up 6000 from before that edit
[00:15:55] <AmandaCatherine>	 meh
[00:16:18] <AmandaCatherine>	 I would think that the real issue would be users who flood Meta RC by making 30, sometimes even 40+ edits to their user page all within a matter of minutes
[00:16:36] <AmandaCatherine>	 (Which sometimes I wonder if it is being done to game autoconfirmed or the edit filters)
[00:16:36] <Voidwalker>	 that's how you get upwards of 90k jobs
[00:16:58] <MacFan4000>	 anyway I am manually running jobs currently useing the script
[00:17:12] <AmandaCatherine>	 I highly doubt that people don’t know how to use preview to the point where they need 40+ minor edits
[00:17:19] <MacFan4000>	 the 90k jobs is due to a lot of htmlcache jobs
[00:17:21] <AmandaCatherine>	 I think that its gaming the system in some way
[00:37:33] <AmandaCatherine>	 !night
[00:37:33] <wm-bot5>	 Good night, sweet dreams and dream of the miraheze servers without errors!
[00:38:06] <Voidwalker>	 there are three wiki requests for the same wiki by three different users
[00:38:24] <AmandaCatherine>	 Voidwalker: likely socks then
[00:38:34] <Voidwalker>	 nope
[00:38:37] <AmandaCatherine>	 Highly unlikely that three unrelated users would request the exact same wiki
[00:39:01] <Voidwalker>	 it is, when they are all ex wikia users interested in migrating the same wiki
[00:39:13] <AmandaCatherine>	 Oh haha
[00:39:27] <AmandaCatherine>	 Wikia really kicks people out for questionable reasons
[00:39:50] <icinga-miraheze>	 RECOVERY - db4 Disk Space on db4 is OK: DISK OK - free space: / 76934 MB (21% inode=94%);  
[00:40:42] <AmandaCatherine>	 !night
[00:40:42] <wm-bot5>	 Good night, sweet dreams and dream of the miraheze servers without errors!
[00:43:51] <icinga-miraheze>	 PROBLEM - db4 Disk Space on db4 is WARNING: DISK WARNING - free space: / 76914 MB (20% inode=94%);  
[01:03:57] <MacFan4000>	 https://meta.miraheze.org/wiki/Requests_for_Stewardship#MacFan4000.27s_Request_for_Stewardship
[01:03:58] <ZppixBot>	 Title: [ Requests for Stewardship - Miraheze Meta ] - meta.miraheze.org
[01:04:12] <MacFan4000>	 I was considering doing that for a while and now I’ve done it
[01:50:59] <icinga-miraheze>	 RECOVERY - mw3 JobQueue on mw3 is OK: JOBQUEUE OK - job queue below 300 jobs  
[01:56:59] <icinga-miraheze>	 PROBLEM - mw3 JobQueue on mw3 is CRITICAL: JOBQUEUE CRITICAL - job queue greater than 300 jobs. Current queue: 6368  
[01:57:25] <Voidwalker>	 heh, looks like the rename killed itself by doing that
[02:13:10] <IMSAIKI>	 I’m sexy and I know it. I’m sexy and you know it. So how bout this: I’ll take my shirt off and wear a sexy spaghetti strap tank top. Then I’ll slide the straps off my shoulders and expose my bra. And then maybe I’ll take my bra off and show you my sexy perky boobs. And THEN maybe I’ll lie down on top of you and press my perky sexy boobs up against 
[02:13:10] <IMSAIKI>	 you. How does that sound??
[02:13:11] <IMSAIKI>	 I’m sexy and I know it. I’m sexy and you know it. So how bout this: I’ll take my shirt off and wear a sexy spaghetti strap tank top. Then I’ll slide the straps off my shoulders and expose my bra. And then maybe I’ll take my bra off and show you my sexy perky boobs. And THEN maybe I’ll lie down on top of you and press my perky sexy boobs up against 
[02:13:11] <IMSAIKI>	 you. How does that sound??
[02:13:14] <IMSAIKI>	 I’m sexy and I know it. I’m sexy and you know it. So how bout this: I’ll take my shirt off and wear a sexy spaghetti strap tank top. Then I’ll slide the straps off my shoulders and expose my bra. And then maybe I’ll take my bra off and show you my sexy perky boobs. And THEN maybe I’ll lie down on top of you and press my perky sexy boobs up against 
[02:13:14] <IMSAIKI>	 you. How does that sound??
[03:40:14] <Not-a634>	 [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fAw8A
[03:40:15] <Not-a634>	 [02miraheze/services] 07MirahezeSSLBot 03f47d254 - BOT: Updating services config for wikis
[06:52:40] <icinga-miraheze>	 PROBLEM - misc2 Puppet on misc2 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Package[php7.2-gettext]  
[07:00:39] <icinga-miraheze>	 RECOVERY - misc2 Puppet on misc2 is OK: OK: Puppet is currently enabled, last run 4 seconds ago with 0 failures  
[08:40:15] <Not-a634>	 [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fAwKv
[08:40:17] <Not-a634>	 [02miraheze/services] 07MirahezeSSLBot 038434b3e - BOT: Updating services config for wikis
[08:43:00] <icinga-miraheze>	 RECOVERY - mw3 JobQueue on mw3 is OK: JOBQUEUE OK - job queue below 300 jobs  
[11:57:01] <icinga-miraheze>	 PROBLEM - mw3 JobQueue on mw3 is CRITICAL: JOBQUEUE CRITICAL - job queue greater than 300 jobs. Current queue: 3181  
[12:26:34] <icinga-miraheze>	 PROBLEM - misc4 Puppet on misc4 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_clone_phabricator-extensions]  
[12:32:33] <icinga-miraheze>	 RECOVERY - misc4 Puppet on misc4 is OK: OK: Puppet is currently enabled, last run 20 seconds ago with 0 failures  
[12:44:34] <icinga-miraheze>	 PROBLEM - misc4 Puppet on misc4 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_clone_phabricator-extensions]  
[12:57:19] <Islami>	 Hi! Sir!
[12:57:45] <Islami>	 Are you here? 
[12:58:59] <Islami>	 We are a community, want to create a encyclopedia.    
[13:00:26] <Islami>	 I proposed to my community, hosting with you...   
[13:01:17] <Wiki-1776>	 Hello, ok. You can request it at https://meta.miraheze.org/wiki/Special:RequestWiki .  And be descriptive Islami 
[13:01:18] <ZppixBot>	 Title: [ Request a wiki - Miraheze Meta ] - meta.miraheze.org
[13:01:51] <Islami>	 But, your interwiki system is include all your hosted wiki.   
[13:02:44] <Islami>	 Is there any system to remove it?  
[13:03:58] <Wiki-1776>	 hmm, you can customize interwikis but if you don't need them, don't use them Islami 
[13:05:40] <Wiki-1776>	 Islami: ?
[13:05:50] <Islami>	 Is there footer customizing possible?   
[13:06:59] <icinga-miraheze>	 RECOVERY - mw3 JobQueue on mw3 is OK: JOBQUEUE OK - job queue below 300 jobs  
[13:08:56] <Wiki-1776>	 sorry, can you explain what you want? Islami 
[13:10:21] <Islami>	 I want to add our community logo on footer and remove cc logo. Is it possible?  
[13:12:09] <Wiki-1776>	 aah, yes see https://meta.miraheze.org/wiki/ManageWiki Islami 
[13:12:09] <ZppixBot>	 Title: [ ManageWiki - Miraheze Meta ] - meta.miraheze.org
[13:14:50] <Wiki-1776>	 hello Islami_ 
[13:15:41] <Islami_>	 Thank you! @Wiki-1776
[13:15:55] <Wiki-1776>	 Ok Islami_ 
[13:15:58] <Islami_>	 I'll try with you.... 
[13:16:56] <Wiki-1776>	 ok
[13:24:32] <icinga-miraheze>	 RECOVERY - misc4 Puppet on misc4 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures  
[13:36:33] <icinga-miraheze>	 PROBLEM - misc4 Puppet on misc4 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_phabricator-extensions]  
[13:51:35] <MacFan4000>	 Why does misc4 puppet keep failing?
[14:28:17] <MacFan4000>	 @Operations icingabot needs a restart
[14:51:46] <MH-Discord>	 <NDKilla> I don't have shell for like 4 hours
[15:16:59] <Reception123>	 paladox: ^
[15:17:12] <Reception123>	 If not I'll get to it but I can't currently 
[15:17:38] <icinga-miraheze>	 RECOVERY - misc4 Puppet on misc4 is OK: OK: Puppet is currently enabled, last run 56 seconds ago with 0 failures  
[15:17:56] <paladox>	 I can do it 10-20
[15:18:00] <paladox>	 Oh
[15:18:07] <paladox>	 Reception123: did it :P
[15:18:12] <JohnLewis>	 it's like you guys don't know I exist
[15:20:23] <paladox>	 JohnLewis: oh you did it?
[15:20:35] <JohnLewis>	 ofc :P
[15:20:40] <paladox>	 Add !ops to your stalk word
[15:20:46] <JohnLewis>	 !no
[15:22:52] <Reception123>	 Even if he did, MacFan did @ops
[15:23:08] <paladox>	 JohnLewis: Lol what
[15:26:33] <icinga-miraheze>	 PROBLEM - misc4 Puppet on misc4 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_clone_phabricator-extensions]  
[15:32:33] <icinga-miraheze>	 RECOVERY - misc4 Puppet on misc4 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures  
[15:39:27] <Wiki-1776>	 https://meta.miraheze.org/wiki/Community_noticeboard#About_Copyrighted_Images_.28and_that_they_use_wgAllowExternalImages.29 comments?
[15:39:27] <ZppixBot>	 Title: [ Community noticeboard - Miraheze Meta ] - meta.miraheze.org
[15:44:34] <icinga-miraheze>	 PROBLEM - misc4 Puppet on misc4 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_clone_phabricator-extensions]  
[15:44:41] <paladox>	 meh
[15:56:34] <icinga-miraheze>	 RECOVERY - misc4 Puppet on misc4 is OK: OK: Puppet is currently enabled, last run 15 seconds ago with 0 failures  
[16:14:35] <icinga-miraheze>	 PROBLEM - misc4 Puppet on misc4 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_clone_phabricator-extensions]  
[16:17:36] <MacFan4000>	 Anyone have any idea why misc4 is having constant puppet fails?
[16:17:46] <paladox>	 yep see staff channel
[16:22:34] <icinga-miraheze>	 RECOVERY - misc4 Puppet on misc4 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures  
[16:48:46] <Wiki-1776>	 Hallo
[16:57:00] <icinga-miraheze>	 PROBLEM - mw3 JobQueue on mw3 is CRITICAL: JOBQUEUE CRITICAL - job queue greater than 300 jobs. Current queue: 6318  
[17:14:12] <Not-a634>	 [02ManageWiki] 07paladox created branch 03paladox-patch-1 - 13https://git.io/vpSns
[17:14:14] <Not-a634>	 [02miraheze/ManageWiki] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/fArzE
[17:14:15] <Not-a634>	 [02miraheze/ManageWiki] 07paladox 030876074 - Fix variable
[17:14:17] <Not-a634>	 [02ManageWiki] 07paladox opened pull request 03#23: Fix variable - 13https://git.io/fArzu
[17:15:13] <Not-a634>	 [02ManageWiki] 07paladox synchronize pull request 03#23: Fix variable - 13https://git.io/fArzu
[17:15:14] <Not-a634>	 [02miraheze/ManageWiki] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/fArza
[17:15:16] <Not-a634>	 [02miraheze/ManageWiki] 07paladox 03b4cf20f - Update SpecialManageWikiPermissions.php
[17:19:16] <Not-a634>	 [02ManageWiki] 07paladox closed pull request 03#23: Fix variable - 13https://git.io/fArzu
[17:19:18] <Not-a634>	 [02miraheze/ManageWiki] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fArzA
[17:19:19] <Not-a634>	 [02miraheze/ManageWiki] 07paladox 03de59a6e - Fix variable (#23) * Fix variable * Update SpecialManageWikiPermissions.php
[17:19:35] <Not-a634>	 [02ManageWiki] 07paladox deleted branch 03paladox-patch-1 - 13https://git.io/vpSns
[17:19:36] <Not-a634>	 [02miraheze/ManageWiki] 07paladox deleted branch 03paladox-patch-1
[17:19:58] <Not-a634>	 [02miraheze/mediawiki] 07paladox pushed 031 commit to 03REL1_31 [+0/-0/±1] 13https://git.io/fArge
[17:19:59] <Not-a634>	 [02miraheze/mediawiki] 07paladox 036db240f - Update MW
[17:21:11] <paladox>	 !log sudo -u www-data php /srv/mediawiki/w/maint*/rebuildLocalisationCache.php --wiki test1wiki on mw*
[17:21:15] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master
[17:26:10] <Not-a634>	 [02ManageWiki] 07paladox created branch 03paladox-patch-1 - 13https://git.io/vpSns
[17:26:11] <Not-a634>	 [02miraheze/ManageWiki] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/fArgX
[17:26:13] <Not-a634>	 [02miraheze/ManageWiki] 07paladox 035c73e7c - Change add and remove to addgroups and removegroups
[17:26:14] <Not-a634>	 [02ManageWiki] 07paladox opened pull request 03#24: Change add and remove to addgroups and removegroups - 13https://git.io/fArg1
[17:27:28] <Not-a634>	 [02ManageWiki] 07paladox synchronize pull request 03#24: Change add and remove to addgroups and removegroups - 13https://git.io/fArg1
[17:27:30] <Not-a634>	 [02miraheze/ManageWiki] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/fArgF
[17:27:31] <Not-a634>	 [02miraheze/ManageWiki] 07paladox 034798640 - Update ManageWikiHooks.php
[17:27:50] <Not-a634>	 [02ManageWiki] 07paladox closed pull request 03#24: Change add and remove to addgroups and removegroups - 13https://git.io/fArg1
[17:27:52] <Not-a634>	 [02miraheze/ManageWiki] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fArgp
[17:27:53] <Not-a634>	 [02miraheze/ManageWiki] 07paladox 0346aa09f - Change add and remove to addgroups and removegroups (#24) * Change add and remove to addgroups and removegroups * Update ManageWikiHooks.php
[17:28:02] <Not-a634>	 [02ManageWiki] 07paladox deleted branch 03paladox-patch-1 - 13https://git.io/vpSns
[17:28:04] <Not-a634>	 [02miraheze/ManageWiki] 07paladox deleted branch 03paladox-patch-1
[17:29:04] <icinga-miraheze>	 PROBLEM - test1 Puppet on test1 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki core]  
[17:38:23] <travis-ci>	 miraheze/ManageWiki/paladox-patch-1/4798640 - paladox The build has errored. https://travis-ci.org/miraheze/ManageWiki/builds/427310476
[19:51:59] <icinga-miraheze>	 PROBLEM - cp5 HTTP 4xx/5xx ERROR Rate on cp5 is WARNING: WARNING - NGINX Error Rate is 54%  
[19:53:59] <icinga-miraheze>	 RECOVERY - cp5 HTTP 4xx/5xx ERROR Rate on cp5 is OK: OK - NGINX Error Rate is 37%  
[20:41:53] <MH-Discord>	 <ezidi 21> I have a problem with https://phabricator.miraheze.org/. I wrote the wrong E-Mail and  i can't change the E-Mail anymore.
[20:41:54] <ZppixBot>	 Title: [ Login ] - phabricator.miraheze.org
[20:44:05] <paladox>	 @ezidi 21 hi, what's your username on phab?
[20:44:56] <paladox>	 @ezidi 21 i can delete the user and you can recreate the user with the correct email.
[20:45:58] <MH-Discord>	 <ezidi 21> wirtschaftsinformatik hbs
[20:46:19] <Not-a634>	 [02miraheze/ManageWiki] 07JohnFLewis pushed 031 commit to 03master [+0/-0/±4] 13https://git.io/fAr5H
[20:46:20] <Not-a634>	 [02miraheze/ManageWiki] 07JohnFLewis 032939c07 - convert caching to CDB
[20:46:25] <JohnLewis>	 paladox: ^ pls review
[20:46:30] * paladox looks
[20:48:42] <MacFan4000>	 paladox: they also responded with a username
[20:48:52] <paladox>	 ah o
[20:48:54] <paladox>	 ok
[20:49:00] * paladox does both things at the same time
[20:49:43] <paladox>	 @ezidi 21 try now
[20:49:49] <paladox>	 creating the account with the correct email
[20:50:19] <paladox>	 JohnLewis codes look lgtm now i have to say very good!!!!!
[20:50:51] <JohnLewis>	 uh
[20:50:55] <JohnLewis>	 who protected the mw-config branh
[20:51:10] * paladox wonders that too
[20:53:29] <JohnLewis>	 !log removed master branch restrictions on mw-config, wasn't approved and I was heavily against it
[20:53:33] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master
[20:53:46] <Not-a634>	 [02miraheze/mw-config] 07JohnFLewis pushed 032 commits to 03master [+0/-0/±5] 13https://git.io/fArd2
[20:53:47] <Not-a634>	 [02miraheze/mw-config] 07JohnFLewis 03bbc86c8 - add directory var
[20:53:49] <Not-a634>	 [02miraheze/mw-config] 07JohnFLewis 03c862df0 - Merge branch 'master' of github.com:miraheze/mw-config
[20:53:53] <Wiki-1776>	 https://meta.miraheze.org/wiki/Community_noticeboard#About_Copyrighted_Images_.28and_that_they_use_wgAllowExternalImages.29 comments?
[20:53:54] <ZppixBot>	 Title: [ Community noticeboard - Miraheze Meta ] - meta.miraheze.org
[20:54:05] <paladox>	 JohnLewis ^^
[20:54:15] <JohnLewis>	 ask me a simple question
[20:54:24] <MH-Discord>	 <ezidi 21> thanks it works
[20:54:40] <paladox>	 JohnLewis see Wiki-1776 link.
[20:54:41] <paladox>	 please
[20:54:51] <JohnLewis>	 paladox: I know, I said ask me a simple question
[20:55:01] <paladox>	 oh
[20:55:05] <JohnLewis>	 I looked at it and I didn't know what it was about when I had time so I don't when I don't have tie
[20:55:15] <paladox>	 ah ok
[20:55:31] <paladox>	 Wiki-1776 would you be able to explain to JohnLewis what you want to know please?
[20:59:09] <Wiki-1776>	 I will try. No problem with my wikis using wgAllowExternalImages and linking images with copyright or unknown license from government websites, wikia, etc.? JohnLewis  
[21:00:04] <Wiki-1776>	 But my wikis are public and seen in practice by anyone.
[21:02:04] <Not-a634>	 [02miraheze/puppet] 07JohnFLewis pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fArFu
[21:02:05] <Not-a634>	 [02miraheze/puppet] 07JohnFLewis 03ccc729c - create managewiki directory in w/cache
[21:02:29] <JohnLewis>	 Wiki-1776: all I'm going to say is respect the licenses
[21:02:50] <JohnLewis>	 if they say you can't use them or there's no appropriate clause in US or EU law (e.g. fair use) then you can't use them
[21:04:32] <Not-a634>	 [02miraheze/mediawiki] 07JohnFLewis pushed 031 commit to 03REL1_31 [+0/-0/±1] 13https://git.io/fArFQ
[21:04:33] <Not-a634>	 [02miraheze/mediawiki] 07JohnFLewis 0382c7b81 - pull MW changes
[21:06:26] <AmandaCatherine>	 Hello
[21:06:31] <paladox>	 hi
[21:07:02] <icinga-miraheze>	 RECOVERY - test1 Puppet on test1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures  
[21:07:10] <AmandaCatherine>	 Hmm.. BotBot is being discontinued...
[21:07:14] <AmandaCatherine>	 Huh
[21:09:42] <AmandaCatherine>	 paladox: 503
[21:09:48] <AmandaCatherine>	 Miraheze is down
[21:09:48] <ZppixBot>	 That sounds bad! A sysadmin should be here shortly to investigate. If you haven't already, please file a Phabricator ticket to facilitate the process!
[21:09:58] <AmandaCatherine>	 Lol ^
[21:10:02] <paladox>	 Hi yep
[21:10:06] <icinga-miraheze>	 PROBLEM - mw1 Puppet on mw1 is WARNING: WARNING: Puppet is currently disabled, message: John, last run 1 minute ago with 0 failures  
[21:10:11] <paladox>	 JohnLewis and me are aware and currrently working on fixing it
[21:10:16] <AmandaCatherine>	 Ok
[21:10:37] <Wiki-1776>	 Ok 
[21:11:00] <icinga-miraheze>	 PROBLEM - mw2 Puppet on mw2 is WARNING: WARNING: Puppet is currently disabled, message: John, last run 2 minutes ago with 0 failures  
[21:11:06] <icinga-miraheze>	 PROBLEM - cp2 Varnish Backends on cp2 is CRITICAL: 3 backends are down.  mw1 mw2 mw3  
[21:11:06] <JohnLewis>	 I pushed the MW change at 4 minutes past
[21:11:08] <icinga-miraheze>	 PROBLEM - cp2 HTTP 4xx/5xx ERROR Rate on cp2 is CRITICAL: CRITICAL - NGINX Error Rate is 95%  
[21:11:11] <JohnLewis>	 and I disabled puppet at 6 minutes past
[21:11:17] <JohnLewis>	 the change should NOT have deployed
[21:11:25] <paladox>	 JohnLewis revert it on mw*
[21:11:27] <JohnLewis>	 no
[21:11:31] <JohnLewis>	 I've made the dirs
[21:11:48] <icinga-miraheze>	 PROBLEM - cp4 Varnish Backends on cp4 is CRITICAL: 3 backends are down.  mw1 mw2 mw3  
[21:11:49] <paladox>	 done
[21:11:54] <icinga-miraheze>	 PROBLEM - mw3 Puppet on mw3 is WARNING: WARNING: Puppet is currently disabled, message: John, last run 2 minutes ago with 0 failures  
[21:11:55] <paladox>	 JohnLewis yeh but see logs
[21:12:14] <SPF|Cloud>	 [11-Sep-2018 21:12:06 UTC] PHP Notice:  Undefined variable: cach in /srv/mediawiki/w/extensions/ManageWiki/includes/ManageWikiHooks.php on line 84
[21:12:17] <SPF|Cloud>	 has nothing to do with dirs
[21:12:20] <icinga-miraheze>	 PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 6 datacenters are down: 107.191.126.23/cpweb, 2604:180:0:33b::2/cpweb, 81.4.109.133/cpweb, 2a00:d880:5:8ea::ebc7/cpweb, 172.104.111.8/cpweb, 2400:8902::f03c:91ff:fe07:444e/cpweb  
[21:12:31] <paladox>	 meta back
[21:12:34] <JohnLewis>	 grr
[21:12:35] <paladox>	 i reverted on mw3 
[21:12:42] <icinga-miraheze>	 PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 6 datacenters are down: 107.191.126.23/cpweb, 2604:180:0:33b::2/cpweb, 81.4.109.133/cpweb, 2a00:d880:5:8ea::ebc7/cpweb, 172.104.111.8/cpweb, 2400:8902::f03c:91ff:fe07:444e/cpweb  
[21:12:45] <JohnLewis>	 how the fuck did this change deploy outside of puppet runs?
[21:13:03] <Wiki-1776>	 As far as I know, where I live there's no such thing as Fair Use. But there are some sites with license CC BY NC ND but they are few and for images of 20 or 50 years ago.
[21:13:08] <icinga-miraheze>	 RECOVERY - cp2 HTTP 4xx/5xx ERROR Rate on cp2 is OK: OK - NGINX Error Rate is 13%  
[21:14:24] <icinga-miraheze>	 RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online  
[21:14:46] <icinga-miraheze>	 RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online  
[21:15:04] <icinga-miraheze>	 PROBLEM - cp5 HTTP 4xx/5xx ERROR Rate on cp5 is WARNING: WARNING - NGINX Error Rate is 42%  
[21:15:13] <Not-a634>	 [02miraheze/ManageWiki] 07JohnFLewis pushed 031 commit to 03master [+1/-0/±1] 13https://git.io/fArb5
[21:15:14] <Not-a634>	 [02miraheze/ManageWiki] 07JohnFLewis 0374ca2dc - fix cach->cache
[21:15:28] <Wiki-1776>	 I will assume that the license BY NC ND is not affected because the wiki is public
[21:15:42] <Not-a634>	 [02miraheze/ManageWiki] 07JohnFLewis pushed 031 commit to 03master [+0/-1/±0] 13https://git.io/fArbb
[21:15:43] <Not-a634>	 [02miraheze/ManageWiki] 07JohnFLewis 03e4f6c46 - rm misc file
[21:16:23] <Not-a634>	 [02miraheze/mediawiki] 07JohnFLewis pushed 031 commit to 03REL1_31 [+0/-0/±1] 13https://git.io/fArbh
[21:16:24] <Not-a634>	 [02miraheze/mediawiki] 07JohnFLewis 03d0d2108 - pull MW
[21:17:04] <icinga-miraheze>	 RECOVERY - cp5 HTTP 4xx/5xx ERROR Rate on cp5 is OK: OK - NGINX Error Rate is 34%  
[21:18:18] <icinga-miraheze>	 RECOVERY - cp5 Varnish Backends on cp5 is OK: All 5 backends are healthy  
[21:18:47] <Wiki-1776>	 Thank you JohnLewis , I'm going to investigate the laws of the country. Maybe they will allow it under BY NC ND. 
[21:19:04] <icinga-miraheze>	 RECOVERY - cp2 Varnish Backends on cp2 is OK: All 5 backends are healthy  
[21:19:49] <icinga-miraheze>	 RECOVERY - cp4 Varnish Backends on cp4 is OK: All 5 backends are healthy  
[21:21:55] <icinga-miraheze>	 RECOVERY - mw3 Puppet on mw3 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures  
[21:22:05] <icinga-miraheze>	 RECOVERY - mw1 Puppet on mw1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures  
[21:22:17] <AmandaCatherine>	 .tell Voidwalker please check your PM as soon as possible
[21:22:18] <ZppixBot>	 AmandaCatherine: I'll pass that on when Voidwalker is around.
[21:22:28] <AmandaCatherine>	 Thanks ZppixBot
[21:22:29] <ZppixBot>	 You're welcome AmandaCatherine
[21:22:59] <icinga-miraheze>	 RECOVERY - mw2 Puppet on mw2 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures  
[21:26:26] <Voidwalker>	 i've only got about five minutes, but from a quick look, it doesn't seem too urgent
[21:26:27] <ZppixBot>	 Voidwalker: 2018-09-11 - 21:22:17 <AmandaCatherine> tell Voidwalker please check your PM as soon as possible
[21:28:02] <JohnLewis>	 AmandaCatherine: can you go to your wiki please and test if deleting a group works?
[21:28:26] <JohnLewis>	 because if so, then I may have code that I can deploy to every wiki
[21:29:08] <AmandaCatherine>	 JohnLewis: my wiki is down
[21:30:01] <MacFan4000>	 Re botbot being shutdown in november, we just have to change the link in the topic to point to the wm-bot logs
[21:30:08] <AmandaCatherine>	 So no I can’t because I can’t even access my wiki
[21:30:11] <JohnLewis>	 uh
[21:30:35] <AmandaCatherine>	 JohnLewis: “[9732e7703d144a43b59f584f] 2018-09-11 21:28:59: Fatal exception of type "Error"”
[21:30:40] <Wiki-1776>	 my wiki works
[21:30:46] <JohnLewis>	 yeah I know...
[21:30:58] <JohnLewis>	 I don't know how four wikis with the same code, only 1 doesn't work
[21:31:19] <AmandaCatherine>	 Ok, now the wiki is loading
[21:31:26] <JohnLewis>	 oh? now it works?
[21:31:26] <AmandaCatherine>	 Let me test deleting the empty group
[21:31:41] <AmandaCatherine>	 Yes, it loads, but slowly
[21:31:44] <Voidwalker>	 metawiki * [[Special:Log/newusers]] create  * Anamda Catherine *  New user account
[21:31:56] <Voidwalker>	 best keep an eye on that
[21:31:57] <AmandaCatherine>	 Uhh
[21:32:04] <AmandaCatherine>	 Lock immediately for impersonation IMHO
[21:32:06] <JohnLewis>	 AmandaCatherine: it will for the first couple of times + every time you change permissions
[21:32:19] <AmandaCatherine>	 Ok
[21:32:24] <Wiki-1776>	 Definitely, you will have to ask permission from the authors or the website...  :(
[21:35:54] <AmandaCatherine>	 JohnLewis: looks like group deletion worked
[21:37:23] <AmandaCatherine>	 Yep it worked
[21:37:42] <JohnLewis>	 great
[21:38:03] <AmandaCatherine>	 Although the steward group has mysteriously disappeared
[21:39:41] <JohnLewis>	 it has? hm
[21:40:10] <AmandaCatherine>	 Yeah, there’s no steward group in ListGroupRights or UserRights, and now I don’t have a “Groups you cannot change” column
[21:40:15] <JohnLewis>	 heh
[21:40:17] <Not-a634>	 [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fArxt
[21:40:18] <Not-a634>	 [02miraheze/services] 07MirahezeSSLBot 037a69f76 - BOT: Updating services config for wikis
[21:40:30] <JohnLewis>	 that needs to be fix now then
[21:40:54] <AmandaCatherine>	 JohnLewis: the only thing it’s used for is local CUs
[21:41:04] <JohnLewis>	 but it's on meta too
[21:41:20] <AmandaCatherine>	 Oh, it disappeared from Meta too?
[21:44:03] <Not-a634>	 [02miraheze/ManageWiki] 07JohnFLewis pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fArxX
[21:44:05] <Not-a634>	 [02miraheze/ManageWiki] 07JohnFLewis 034a6ec03 - build and append arrays
[21:44:38] <Not-a634>	 [02miraheze/mediawiki] 07JohnFLewis pushed 031 commit to 03REL1_31 [+0/-0/±1] 13https://git.io/fArx9
[21:44:39] <Not-a634>	 [02miraheze/mediawiki] 07JohnFLewis 03cb576ea - pull MW
[21:47:51] <AmandaCatherine>	 ok, someone lock that account now
[21:47:57] <AmandaCatherine>	 JohnLewis Voidwalker|away
[21:48:05] <AmandaCatherine>	 They are attempting to impersonate me
[21:48:09] <JohnLewis>	 would love to but we cant rn
[21:48:26] <AmandaCatherine>	 Ack
[21:48:35] <JohnLewis>	 now we can
[21:52:21] <Not-a634>	 [02miraheze/puppet] 07JohnFLewis pushed 031 commit to 03master [+0/-0/±2] 13https://git.io/fArpR
[21:52:23] <Not-a634>	 [02miraheze/puppet] 07JohnFLewis 039fcdd0d - meta->login for all monitoring uses
[21:52:43] <JohnLewis>	 paladox: ^ can you deploy and watch that for me while I deal with the Amanda issue please?
[21:52:50] <paladox>	 yep
[21:53:04] <AmandaCatherine>	 JohnLewis: it’s already dealt with :)
[21:53:04] <JohnLewis>	 emphasis on watch :D
[21:53:11] <JohnLewis>	 AmandaCatherine: it's been CU'd?
[21:53:15] <AmandaCatherine>	 Oh, no
[21:53:20] <AmandaCatherine>	 But locked and banned
[21:53:22] <JohnLewis>	 so then it's not :P
[21:53:27] <JohnLewis>	 ik I locked it ;)
[21:56:08] <AmandaCatherine>	 ...and, 503 again
[21:56:19] <paladox>	 JohnLewis you forgot https://github.com/miraheze/dns/blob/master/config#L16
[21:56:20] <ZppixBot>	 Title: [ dns/config at master · miraheze/dns · GitHub ] - github.com
[21:56:35] <JohnLewis>	 503
[21:56:36] <paladox>	 AmandaCatherine it's back
[21:56:39] <Wiki-1776>	 Botbot :/
[21:56:39] <JohnLewis>	 cool
[21:56:48] <paladox>	 JohnLewis should i do that change
[21:56:49] <paladox>	 to dns?
[21:56:49] <JohnLewis>	 paladox: I didn't forget it, I just didn't care :P
[21:56:51] <JohnLewis>	 yeah
[21:56:52] <paladox>	 oh
[21:57:11] <Not-a634>	 [02miraheze/dns] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fArpN
[21:57:13] <Not-a634>	 [02miraheze/dns] 07paladox 036d6d70f - meta -> login
[21:57:58] <JohnLewis>	 heh
[21:58:06] <JohnLewis>	 actually that kinda didn't matter anyway
[21:58:18] <JohnLewis>	 varnish proxies /healthcheck using vhost login anyway
[22:02:07] <icinga-miraheze>	 PROBLEM - mw1 MediaWiki Rendering on mw1 is WARNING: HTTP WARNING: HTTP/1.1 404 Not Found - 15260 bytes in 0.180 second response time   
[22:02:21] <icinga-miraheze>	 PROBLEM - mw2 MediaWiki Rendering on mw2 is WARNING: HTTP WARNING: HTTP/1.1 404 Not Found - 15260 bytes in 0.143 second response time   
[22:02:31] <icinga-miraheze>	 PROBLEM - mw3 MediaWiki Rendering on mw3 is WARNING: HTTP WARNING: HTTP/1.1 404 Not Found - 15259 bytes in 0.206 second response time   
[22:03:03] <paladox>	 hmm
[22:03:04] <paladox>	 err
[22:04:33] <Not-a634>	 [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fArhK
[22:04:35] <Not-a634>	 [02miraheze/puppet] 07paladox 03d40b7df - Fix path
[22:05:40] <Not-a634>	 [02miraheze/ManageWiki] 07JohnFLewis pushed 032 commits to 03master [+0/-0/±2] 13https://git.io/fArhM
[22:05:42] <Not-a634>	 [02miraheze/ManageWiki] 07JohnFLewis 03ee924da - avoid dupes
[22:05:43] <Not-a634>	 [02miraheze/ManageWiki] 07JohnFLewis 03bdde72f - Merge branch 'master' of github.com:miraheze/ManageWiki
[22:05:49] <icinga-miraheze>	 PROBLEM - mw1 MediaWiki Rendering on mw1 is WARNING: HTTP WARNING: HTTP/1.1 404 Not Found - 15260 bytes in 0.327 second response time   
[22:06:03] <icinga-miraheze>	 PROBLEM - test1 MediaWiki Rendering on test1 is WARNING: HTTP WARNING: HTTP/1.1 404 Not Found - 15260 bytes in 0.217 second response time   
[22:06:07] <icinga-miraheze>	 PROBLEM - mw2 MediaWiki Rendering on mw2 is WARNING: HTTP WARNING: HTTP/1.1 404 Not Found - 15260 bytes in 0.260 second response time   
[22:06:15] <icinga-miraheze>	 PROBLEM - mw3 MediaWiki Rendering on mw3 is WARNING: HTTP WARNING: HTTP/1.1 404 Not Found - 15256 bytes in 0.211 second response time   
[22:06:20] <Not-a634>	 [02miraheze/mediawiki] 07JohnFLewis pushed 031 commit to 03REL1_31 [+0/-0/±1] 13https://git.io/fArhS
[22:06:21] <Not-a634>	 [02miraheze/mediawiki] 07JohnFLewis 03ef1e2ba - pull MW
[22:06:25] <paladox>	 JohnLewis yay!!!!
[22:07:11] <paladox>	 err
[22:08:46] <paladox>	 woo recovered.
[22:09:49] <JohnLewis>	 if running the script on test1 presents no issues, I can globally deploy MWP
[22:10:52] <paladox>	 !log sudo -u www-data php /srv/mediawiki/w/maint*/rebuildLocalisationCache.php --wiki test1wiki on test1
[22:10:57] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master
[22:17:26] <icinga-miraheze>	 PROBLEM - test1 Puppet on test1 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki core]  
[22:20:15] <Not-a634>	 [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fAoef
[22:20:16] <Not-a634>	 [02miraheze/services] 07MirahezeSSLBot 03244de90 - BOT: Updating services config for wikis
[22:21:27] <icinga-miraheze>	 RECOVERY - test1 Puppet on test1 is OK: OK: Puppet is currently enabled, last run 15 seconds ago with 0 failures  
[22:24:57] <Not-a634>	 [02miraheze/ManageWiki] 07JohnFLewis pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fAoer
[22:24:59] <Not-a634>	 [02miraheze/ManageWiki] 07JohnFLewis 03e8e36a8 - cast type array
[22:25:31] <Not-a634>	 [02miraheze/mediawiki] 07JohnFLewis pushed 031 commit to 03REL1_31 [+0/-0/±1] 13https://git.io/fAoeP
[22:25:32] <Not-a634>	 [02miraheze/mediawiki] 07JohnFLewis 038c05fbe - update MW
[22:27:35] <JohnLewis>	 !log populatingGroupPermissions for all wikis
[22:27:39] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master
[22:33:06] <JohnLewis>	 so when I push the revert change, all non already visited wikis will be slow on first visits and db4 load may go high for a little bit but will soften out
[22:35:03] <paladox>	 Ok great :)
[22:39:34] <icinga-miraheze>	 PROBLEM - test1 Puppet on test1 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki core]  
[22:51:15] <Not-a634>	 [02miraheze/mw-config] 07JohnFLewis pushed 031 commit to 03master [+0/-0/±3] 13https://git.io/fAofH
[22:51:17] <Not-a634>	 [02miraheze/mw-config] 07JohnFLewis 031eb0b01 - Revert "Revert "Revert "Revert "convert GroupPermissions to ManageWiki"""" This reverts commit 15668ca116c4323c11e4a0c4bd6d6e6911d1a159.
[22:52:12] <JohnLewis>	 paladox: ^ I've gotta go mobile now, can you deploy that? ensuring it's all good please
[22:52:39] <paladox>	 Ok
[22:54:56] <paladox>	 JohnLewis: deployed now
[23:03:38] <JohnLewis>	 Checked the logs?
[23:04:29] <JohnLewis>	 So
[23:04:46] <JohnLewis>	 Some wikis work sometimes and sometimes don’t 
[23:04:59] <JohnLewis>	 has to be a deployment discrepancy 
[23:05:33] <AmandaCatherine>	 JohnLewis: did we re-re-re-enable MWP globally?
[23:05:59] <AmandaCatherine>	 (and no, the multiple “re” is not a typo :P)
[23:07:47] <paladox>	 JohnLewis it was deployed
[23:07:53] <paladox>	 AmandaCatherine yes
[23:08:07] <paladox>	 JohnLewis so looking at https://grafana.miraheze.org/d/W9MIkA7iz/miraheze-cluster?orgId=1&var-job=node&var-node=db4.miraheze.org&var-port=9100
[23:08:08] <ZppixBot>	 Title: [ Grafana ] - grafana.miraheze.org
[23:08:11] <paladox>	 its not going down
[23:09:41] <JohnLewis>	 Paladox: please see my last message 
[23:09:50] <paladox>	 Checked the logs?
[23:09:56] <JohnLewis>	 it is both urgent and critical and directly related to you’re issue 
[23:10:11] <paladox>	 yesoh
[23:10:16] <paladox>	 JohnLewis PHP Notice:  Undefined variable: cache in /srv/mediawiki/w/extensions/ManageWiki/includes/ManageWikiHooks.php on line 84
[23:10:17] <JohnLewis>	 Wikis work sometimes and sometimes don’t 
[23:10:26] <JohnLewis>	 a server can’t be deployed right 
[23:10:44] <JohnLewis>	 Ha my stupidity then 
[23:11:26] <paladox>	 $cache is not set
[23:12:14] <Not-a634>	 [02miraheze/ManageWiki] 07JohnFLewis pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fAoUW
[23:12:15] <Not-a634>	 [02miraheze/ManageWiki] 07JohnFLewis 030f44dcb - Define cache when not using cache
[23:12:20] * paladox deploys
[23:12:59] <Not-a634>	 [02miraheze/mediawiki] 07paladox pushed 031 commit to 03REL1_31 [+0/-0/±1] 13https://git.io/fAoU8
[23:13:01] <Not-a634>	 [02miraheze/mediawiki] 07paladox 0320199ae - Update MW
[23:14:50] <paladox>	 JohnLewis deployed now
[23:32:19] <icinga-miraheze>	 RECOVERY - test1 Puppet on test1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures