[03:50:00] <wm-bot>	 !log tools.bridgebot <bd808> Trying out a custom remote nick format for #wikipedia-abstract
[03:50:02] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.bridgebot/SAL
[08:45:04] <arturo>	 !log admin [codfw1dev] enabled account creation in labtestwiki briefly for jbond42 to create an account
[08:45:06] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL
[08:48:23] <arturo>	 !log admin [codfw1dev] add jbond as user in the bastion-codfw1dev and cloudinfra-codfw1dev projects
[08:48:24] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL
[08:54:13] <jbond42>	 !log [codfw1dev] upgrading hiera to version5 
[08:54:14] <stashbot>	 jbond42: Unknown project "[codfw1dev]"
[08:54:41] <arturo>	 we use the admin project to log stuff related to codfw1dev
[08:55:00] <arturo>	 basically, our current SAL setup is not multi-deployment aware
[08:55:20] <arturo>	 so jbond42 try `!log admin [codfw1dev] upgrading hiera to version5`
[08:55:47] <jbond42>	 !log admin [codfw1dev] upgrading hiera to version5 
[08:55:48] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL
[08:55:49] <jbond42>	 ack thanks
[09:09:19] <yuvipanda>	 I miss stashbot
[09:09:27] <yuvipanda>	 need to build something like that for my current situation
[09:20:45] <arturo>	 fork it!
[14:03:33] <Amir1>	 !log codesearch restart codesearch to pick up new config (adding mediawiki/vagrant)
[14:03:34] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Codesearch/SAL
[16:19:03] <Krinkle>	 I noticed that my redirect tool has stopped working. I can guess why but just wanted to double check that this was anticipated.
[16:19:03] <Krinkle>	 https://github.com/Krinkle/toolserver-misc/blob/master/krinkle-redirect/.lighttpd.conf
[16:19:12] <Krinkle>	 	"^/krinkle-redirect/getWMML\.php(\?.*)?$" => "https://tools.wmflabs.org/list/$1",
[16:20:53] <bd808>	 Krinkle: that sounds like something that needs updated for the toolforge.org changes? The leading tool name in that lighttpd.conf probably needs to be removed?
[16:21:16] <Krinkle>	 bd808: yeah, exactly.
[16:21:37] <Krinkle>	 bd808: was there a way this could have worked previously without harcoding that? I did wonder whether I was doing that correctly
[16:22:24] <bd808>	 The old subpath routing needed that prefix for sure
[16:29:21] <Krinkle>	 perfect, it's working again now, e.g. bit.ly/wikitechLatest
[16:32:32] <Krinkle>	 https://phabricator.wikimedia.org/P12019
[16:32:34] <Krinkle>	 quite a chain :)
[16:32:37] <Krinkle>	 Thanks!
[16:38:33] <bd808>	 Krinkle: The redirects for the old toolserver.org URLs are in ops/puppet.git at modules/toolserver_legacy/templates/www.toolserver.org.erb if you want to propose direct redirects :)
[18:30:02] <SpainDist>	 Hello.
[18:31:17] <SpainDist>	 !kb SpainDist
[18:31:47] <SpainDist>	 @kb @MadriCR
[18:31:47] <wm-bot>	 Permission denied
[18:31:54] <SpainDist>	 @kb SpainDist
[18:31:54] <wm-bot>	 Permission denied
[18:32:21] <Wiki13>	 bd808: ^^^
[18:32:42] <bd808>	 @kb SpainDist 
[18:32:54] <Wiki13>	 thanks :)
[19:15:36] <Wikijoaquinito01>	 @kb bd808
[19:15:36] <wm-bot>	 Permission denied
[19:15:45] <Wikijoaquinito01>	 @kb McFan400
[19:15:45] <wm-bot>	 Permission denied
[19:15:45] <bd808>	 @kb Wikijoaquinito01 
[20:10:39] <Urbanecm>	 !log tools.stewardbots tools.stewardbots@tools-sgebastion-07:~$ restart_stewardbot.sh
[20:10:42] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.stewardbots/SAL
[21:18:49] <wm-bot>	 User was deleted from access list
[21:18:49] <bd808>	 @trustdel .*@wikimedia/.*
[21:18:58] <wm-bot>	 User was deleted from access list
[21:18:58] <bd808>	 @trustdel .*@mediawiki/.*
[21:19:05] <wm-bot>	 User was deleted from access list
[21:19:05] <bd808>	 @trustdel .*@wikipedia/.*
[21:19:58] <wm-bot>	 I trust:  bd808!.*@wikimedia/BDavis-WMF (2admin), .*@wikimedia/andrew-bogott (2admin), arturo!*arturo@* (2trusted), bstorm!*bstorm_@* (2trusted),
[21:19:58] <Texas>	 @trusted
[21:20:21] <wm-bot>	 Wrong number of parameters, go fix it - example @trustadd regex (admin|trusted)
[21:20:21] <bd808>	 @trustadd .*@wikimedia/(?!bot/).*
[21:20:36] <wm-bot>	 Successfully added .*@wikimedia/(?!bot/).*
[21:20:36] <bd808>	 @trustadd .*@wikimedia/(?!bot/).* trusted
[21:20:49] <Texas>	 bd808: I think that the access entries have to be in valid regex
[21:21:02] <bd808>	 that's a valid regex :)
[21:21:10] <Texas>	 so arturo!*arturo@* should be arturo!.*arturo@.*
[21:21:20] <bd808>	 ah. ack
[21:21:29] <Texas>	 :)
[21:22:00] <bd808>	 It's annoying that you can't do this bit of management outside the channel
[21:22:06] <Texas>	 Yeah
[21:22:10] <Texas>	 a little spammy
[21:22:16] <bd808>	 you can show the list in a PM with the bot, but not change it
[21:22:33] <Texas>	 I think you can change it from #wm-bot
[21:22:44] <bd808>	 and no, I'm not learning C# to fix petan's code :)
[21:22:50] <Texas>	 :P
[21:23:12] <wm-bot>	 User was deleted from access list
[21:23:12] <bd808>	 @trustdel arturo!*arturo@*
[21:23:30] <wm-bot>	 Unknown user level!
[21:23:30] <bd808>	 @trustadd arturo!.*@.* 
[21:23:37] <wm-bot>	 Successfully added arturo!.*@.*
[21:23:37] <bd808>	 @trustadd arturo!.*@.* trusted
[21:23:43] <Texas>	 ...and I think @kb and stuff requires admin
[21:24:22] <wm-bot>	 Unable to add user because this user is already in a list
[21:24:22] <bd808>	 @trustadd arturo!.*@.* admin
[21:24:36] <wm-bot>	 User was deleted from access list
[21:24:36] <bd808>	 @trustdel arturo!.*@.*
[21:24:40] <wm-bot>	 Successfully added arturo!.*@.*
[21:24:40] <bd808>	 @trustadd arturo!.*@.* admin
[21:25:12] <wm-bot>	 User was deleted from access list
[21:25:12] <bd808>	 @trustdel bstorm!*bstorm_@*
[21:25:39] <wm-bot>	 Successfully added bstorm!.*@.*
[21:25:39] <bd808>	 @trustadd bstorm!.*@.* admin
[21:26:09] <wm-bot>	 Successfully added .*@mediawiki/(?!bot/).*
[21:26:09] <bd808>	 @trustadd .*@mediawiki/(?!bot/).* trusted
[21:26:16] <wm-bot>	 Successfully added .*@wikipedia/(?!bot/).*
[21:26:16] <bd808>	 @trustadd .*@wikipedia/(?!bot/).* trusted
[21:26:22] * bd808 stops piddling
[21:27:58] <joaquinrufog>	 Are you secure?
[21:28:20] <joaquinrufog>	 Reedy.
[21:28:23] <MacFan4000>	 @kb joaquinrufog
[21:34:21] <bd808>	 @kb Toolserver 
[21:34:26] <JohnMarkOckerblo>	 I hope I'min the right place.  Does anyone have any idea about why my script on Tools.wmflabs.org is now no longer running, but instead returning the source code of the script?  (I have not changed the source code recently)
[21:35:07] <Reedy>	 You are... Which tool etc?
[21:35:13] <bd808>	 JohnMarkOckerblo: what's the tool name? My first guess is that it died during the final cuttover to the toolforge.org naming scheme
[21:35:30] <bd808>	 https://wikitech.wikimedia.org/wiki/News/Toolforge.org
[21:36:07] <wm-bot>	 !log tools.wd-image-positions <lucaswerkmeister> deployed 6b34c5bb7b (editing regions)
[21:36:09] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.wd-image-positions/SAL
[21:36:15] <JohnMarkOckerblo>	 ftl  . The URL is https://tools.wmflabs.org/ftl/cgi-bin/ftl
[21:37:25] <JohnMarkOckerblo>	 Looks like it should instead have something like ftl.toolforge.org in it, if I'm reading that notice right?
[21:37:55] <bd808>	 JohnMarkOckerblo: yeah, it's related to the naming change. You need to update your tools' $HOME/.lighttpd.conf and remove the "/ftl" patch prefix. Related to https://wikitech.wikimedia.org/wiki/News/Toolforge.org#Rewrites_in_$HOME/.lighttpd.conf_not_working
[21:38:07] <bd808>	 *path prefix
[21:39:03] <bd808>	 Texas: if I kick you will joaquino leave us alone? ;)
[21:39:12] <Texas>	 lol ;)
[21:39:16] <Wiki13>	 rofl
[21:39:22] <Texas>	 I was grouping the nick so I can ghost him
[21:39:26] <JohnMarkOckerblo>	 OK.  I'll check.  https://ftl.toolforge.org/ currently gives me a 403 Forbidden; will the right lighttpd.conf change fix that?
[21:40:59] <bd808>	 JohnMarkOckerblo: your tool doesn't have a $HOME/public_html/index.html page nor indexing turned on, so you will need to do one of those as well. The old  https://tools.wmflabs.org/ftl/ url would have returned the same 403 response
[21:41:17] <Wiki13>	 lol
[21:41:18] <Texas>	 oh boy
[21:41:24] <Wiki13>	 hahaha
[21:41:58] <Toolforge>	 Hi Examknow
[21:42:00] <Wiki13>	 hm
[21:42:10] <bd808>	 @kb Toolforge 
[21:43:05] <JohnMarkOckerblo>	 OK.   There are no static HTML files to serve, just the Perl script.  I have a remap of $HTTP["url"] =~ "^/ftl/cgi-bin" {        cgi.assign = ( "" => "/usr/bin/perl" )} but it sounds like I should just reassign everything to that, since it has its own domain now?
[21:43:30] <bd808>	 JohnMarkOckerblo: yeah, something like that might work for you
[21:44:25] <bd808>	 JohnMarkOckerblo: if this was all a surprise to you, you probably should subscribe to https://lists.wikimedia.org/mailman/listinfo/cloud-announce
[21:46:03] <JohnMarkOckerblo>	 Thanks.  I'll subscribe and check the archive.  Looks like my first quick-fix attempt didn't work, so I'll look through the announcement and docs.  I'll also have to change some templates on Wikipedia later on to point to the new URL, it sounds like, once I get the script working again.
[21:49:21] <bd808>	 JohnMarkOckerblo: I think your current problem is that your tool is now using Kubernetes rather than the grid engine backend
[21:49:39] <JohnMarkOckerblo>	 So Kubernetes is no longer supported?
[21:49:48] <bd808>	 the Kubernetes platform does not have a perl container.
[21:49:57] <JohnMarkOckerblo>	 OK.  The grid engine does?
[21:50:04] <bd808>	 your perl cgi would have never worked on our kubernetes system
[21:50:23] <bd808>	 yes, the grid engine backend has the perl libraries your script needs
[21:51:02] <bd808>	 something like `webservice stop; webservice --backend=gridengine lighttpd start` should switch you over
[21:51:38] <bd808>	 I'm guessing you ran `webservice start` and got the new defaults which moved you to kubernetes and a php runtime
[21:52:30] <JohnMarkOckerblo>	 Yes. I'll try the webservice start with the gridengine argument.
[21:53:20] <JohnMarkOckerblo>	 OK, at least now I get a more informative error message.
[22:06:57] <wm-bot>	 !log tools.wd-image-positions <lucaswerkmeister> deployed aa97ea1589 (Esc for editing regions)
[22:06:59] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.wd-image-positions/SAL
[22:07:18] <bstorm>	 !log tools set the tools-k8s-haproxy-1 (main load balancer for toolforge) to have an egress limit of 800MB per sec instead of the same as all the other servers
[22:07:20] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL
[22:15:06] <bstorm>	 !log tools set the tools-k8s-control nodes to also use 800MBps to prevent issues with toolforge ingress and api system
[22:15:09] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL
[22:16:53] <WMF-Office>	 Consistent with the Terms of Use, Joaquinito01 has been banned by the Wikimedia Foundation from editing Wikimedia sites.
[22:17:13] <WMF-Office>	 Consistent with the Terms of Use, bd808 has been banned by the Wikimedia Foundation from editing Wikimedia sites.
[22:17:30] <WMF-Office>	 Consistent with the Terms of Use, Reedy has been banned by the Wikimedia Foundation from editing Wikimedia sites.
[22:17:43] <bd808>	 @kb WMF-Office 
[22:17:43] <WMF-Office>	 Consistent with the Terms of Use, Waggie has been banned by the Wikimedia Foundation from editing Wikimedia sites.
[22:17:44] <Reedy>	 @kb WMF-Office
[22:17:44] <wm-bot>	 Permission denied
[22:17:56] <Seddon>	 lol
[22:18:12] <bd808>	 consitent with being a troll the troll is trolling today
[22:18:22] <Seddon>	 Well I mean at least we got two global bans right
[22:18:26] * Seddon hides
[22:19:10] <bd808>	 there are silver linings I guess :)
[22:19:24] <bd808>	 does this mean that I can stop writing tech docs?
[22:20:36] <bd808>	 Seddon: I had a nice chat with Lane yesterday. Thanks for sending him to find me :)
[22:26:18] <wm-bot>	 !log tools.wd-image-positions <lucaswerkmeister> deployed 9eb2aa216d (no edit region without regions)
[22:26:28] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.wd-image-positions/SAL
[22:28:32] <bstorm>	 !log tools setting tools-k8s-control prefix to mount NFS v4.2 T257945
[22:28:35] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL
[22:28:35] <stashbot>	 T257945: NFS v4.1/2 as possible fix for elevated load and lock contention on our NFS servers - https://phabricator.wikimedia.org/T257945
[22:30:57] <Seddon>	 bd808: No worries :) I'm trying to get both Ryan and Lane in front of as many smart people and we can manage (whilst actually managing to get some work done).
[22:31:51] <bd808>	 Mostly I told him "good luck" and to reach out if he hits something that feels like I could help understand
[22:38:25] <JohnMarkOckerblo>	 OK, I got the script working again on ftl.toolforge.org . Thanks for your help!
[22:39:19] <JohnMarkOckerblo>	 (Next up is to change the templates using it to point directly there instead of to wmflabs.org.  I'll try to get that done as soon as possible, though I don't think I have the power to change my own templates at present.
[22:42:46] <Reedy>	 If you link the templates, someone can probably help
[22:56:49] <JohnMarkOckerblo>	 Thanks.  I've changed Template:Library link by, but Template:Library link about is locked hard enough that I can't change it myself.  I just need to have "//tools.wmflabs.org/ftl/cgi-bin/ftl" replaced with "//ftl.toolforge.org/cgi-bin/ftl" (as I've now done for Template:Library link by)
[22:59:32] <JohnMarkOckerblo>	 (the template still works due to the auto-redirect, but I'd guess the redirect from tools.wmflabs.org will eventually ctop working)
[22:59:38] <Reedy>	 fwiw, there's not much value in having the protocol relative links these days. All wikipedia is https
[23:00:17] <Reedy>	 https://en.wikipedia.org/w/index.php?title=Template%3ALibrary_link_about&type=revision&diff=969022836&oldid=827250930
[23:01:38] <legoktm>	 JohnMarkOckerblo: it's expected that tools.wmflabs.org redirects will work indefinitely.
[23:07:28] <bstorm>	 !log tools disabling puppet on k8s workers to reduce the effect of changing the NFS mount version all at once T257945
[23:07:32] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL
[23:07:32] <stashbot>	 T257945: NFS v4.1/2 as possible fix for elevated load and lock contention on our NFS servers - https://phabricator.wikimedia.org/T257945
[23:11:19] <bstorm>	 !log tools running puppet and NFS remount on tools-k8s-worker-[1-15] T257945
[23:11:22] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL
[23:14:39] <bstorm>	 !log tools running puppet and NFS 4.2 remount on tools-k8s-worker-[21-40] T257945
[23:14:42] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL
[23:14:42] <stashbot>	 T257945: NFS v4.1/2 as possible fix for elevated load and lock contention on our NFS servers - https://phabricator.wikimedia.org/T257945
[23:17:46] <bstorm>	 !log tools running puppet and NFS 4.2 remount on tools-k8s-worker-[41-55] T257945
[23:17:48] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL
[23:22:49] <bstorm>	 !log tools running puppet and NFS 4.2 remount on tools-k8s-worker-[56-60] T257945
[23:22:52] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL
[23:22:52] <stashbot>	 T257945: NFS v4.1/2 as possible fix for elevated load and lock contention on our NFS servers - https://phabricator.wikimedia.org/T257945
[23:24:55] <bstorm>	 !log tools created server group 'tools-k8s-worker' to create any new worker nodes in so that they have a low chance of being scheduled together by openstack unless it is necessary T258663
[23:24:57] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL
[23:24:58] <stashbot>	 T258663: Add some more k8s worker nodes to Toolforge at the xlarge size - https://phabricator.wikimedia.org/T258663
[23:32:10] <bstorm>	 !log paws setting the default NFS version to 4.2 while excepting the two stretch servers T257945
[23:32:13] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Paws/SAL
[23:32:13] <stashbot>	 T257945: NFS v4.1/2 as possible fix for elevated load and lock contention on our NFS servers - https://phabricator.wikimedia.org/T257945