[00:50:05] <bd808>	 !log admin-monitoring Removed Yuvipanda at user request (T186289)
[00:50:07] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin-monitoring/SAL
[00:50:07] <stashbot>	 T186289: Remove cloud-admin rights from YuviPanda - https://phabricator.wikimedia.org/T186289
[00:53:19] <bd808>	 !log admin-monitoring Added Arturo Borrero Gonzalez, BryanDavis, and Bstorm as admins
[00:53:20] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin-monitoring/SAL
[00:54:19] <bd808>	 !log admin Removed Yuvipanda at user request (T186289)
[00:54:22] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL
[00:55:27] <bd808>	 !log admin Added Arturo Borrero Gonzalez and Bstorm as project members
[00:55:29] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL
[00:55:37] <Oshwah>	 wat
[00:56:22] <bd808>	 !log chasetestproject Removed Yuvipanda at user request (T186289)
[00:56:24] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Chasetestproject/SAL
[00:56:24] <stashbot>	 T186289: Remove cloud-admin rights from YuviPanda - https://phabricator.wikimedia.org/T186289
[00:57:09] <bd808>	 !log chasetestproject Removed Yuvipanda at user request (T186289)
[00:57:12] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Chasetestproject/SAL
[00:57:23] <bd808>	 !log ci-staging Removed Yuvipanda at user request (T186289)
[00:57:25] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Ci-staging/SAL
[00:57:25] <bd808>	 paste fail
[00:58:14] <bd808>	 !log contintcloud Removed Yuvipanda at user request (T186289)
[00:58:22] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Contintcloud/SAL
[00:59:17] <bd808>	 !log deployment-prep Removed Yuvipanda at user request (T186289)
[00:59:26] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Deployment-prep/SAL
[01:00:02] <bd808>	 !log discourse Removed Yuvipanda at user request (T186289)
[01:00:04] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Discourse/SAL
[01:00:40] <bd808>	 !log etcd Removed Yuvipanda at user request (T186289)
[01:00:43] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Etcd/SAL
[01:01:16] <bd808>	 !log extdist Removed Yuvipanda at user request (T186289)
[01:01:18] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Extdist/SAL
[01:01:57] <bd808>	 !log graphite Removed Yuvipanda at user request (T186289)
[01:02:01] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Graphite/SAL
[01:02:01] <stashbot>	 T186289: Remove cloud-admin rights from YuviPanda - https://phabricator.wikimedia.org/T186289
[01:02:34] <bd808>	 !log integration Removed Yuvipanda at user request (T186289)
[01:02:37] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Integration/SAL
[01:03:14] <bd808>	 !log kubernetes-testing Removed Yuvipanda at user request (T186289)
[01:03:17] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Kubernetes-testing/SAL
[01:03:56] <bd808>	 !log librarybase Removed Yuvipanda at user request (T186289)
[01:03:58] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Librarybase/SAL
[01:04:37] <bd808>	 !log project-proxy Removed Yuvipanda at user request (T186289)
[01:04:40] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Project-proxy/SAL
[01:05:18] <bd808>	 !log project-proxy Removed inactive users Ryan Lane and Coren
[01:05:20] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Project-proxy/SAL
[01:06:01] <bd808>	 !log quarry Removed Yuvipanda at user request (T186289)
[01:06:05] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Quarry/SAL
[01:06:57] <bd808>	 !log quarry Removed TestingAccount2 at user request (T186289)
[01:06:59] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Quarry/SAL
[01:08:03] <bd808>	 !log redirects Removed Yuvipanda at user request (T186289)
[01:08:06] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Redirects/SAL
[01:08:06] <stashbot>	 T186289: Remove cloud-admin rights from YuviPanda - https://phabricator.wikimedia.org/T186289
[01:09:27] <bd808>	 !log shinken Removed Yuvipanda at user request (T186289)
[01:09:29] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Shinken/SAL
[01:11:13] <bd808>	 !log testlabs Removed Yuvipanda at user request (T186289)
[01:11:16] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Testlabs/SAL
[01:11:53] <bd808>	 !log toolsbeta Removed Yuvipanda at user request (T186289)
[01:11:56] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL
[01:12:28] <bd808>	 !log wdq-mm Removed Yuvipanda at user request (T186289)
[01:12:31] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Wdq-mm/SAL
[02:01:15] <bd808>	 !log mwv-apt Removed Yuvipanda at user request (T186289)
[02:01:19] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Mwv-apt/SAL
[02:01:19] <stashbot>	 T186289: Remove cloud-admin rights from YuviPanda - https://phabricator.wikimedia.org/T186289
[02:17:29] <mutante>	 i have an instance that still responds to ping but not to ssh anymore (planet-hotdog), Action Log doesnt show anything besides the last start on Jan 16. status is shown as "Active". another instance in the same project is fine. i'm trying hard restart via horizon
[02:25:48] <mutante>	 yes, it came back 
[02:26:28] <bd808>	 mutante: lucky you :)
[03:19:36] <MarioFinale>	 !help
[03:19:36] <wm-bot5>	 MarioFinale: If you don't get a response in 15-30 minutes, please create a phabricator task -- https://phabricator.wikimedia.org/maniphest/task/edit/form/1/?projects=wmcs-team
[03:20:24] <chicocvenancio>	 MarioFinale: what do you need? 
[03:22:02] <MarioFinale>	 hi, chicocvenancio. I have a problem running my tool/thing in my Project instance
[03:22:43] <MarioFinale>	 My tool is a BOT written in .net, its compatible with MONO but the instance doesnt have the package mono-vbnc
[03:23:24] <MarioFinale>	 And that specific package is essential to my bot
[03:23:29] <chicocvenancio>	 Cloud vps, right? 
[03:23:38] <MarioFinale>	 yes
[03:27:02] <chicocvenancio>	 MarioFinale: I think it is best to go with a phabricator task on this one. I'm not familiar enough with cloud vps and Mono to help you directly 
[03:27:38] <MarioFinale>	 I made a phabricator task about that (T186846), but i don´t really know if i made it right. Im new on phabricator...
[03:27:38] <stashbot>	 T186846: Request creation of PeriodiBOT VPS project - https://phabricator.wikimedia.org/T186846
[03:29:53] <MarioFinale>	 chicocvenancio: is that task ok? or i need to change something?
[03:31:21] <chicocvenancio>	 MarioFinale, so you have access to toolserver
[03:32:39] <chicocvenancio>	 That task seems OK, but it'd be good to have more details about CPU, ram, and storage needs; as well as what the bot does
[03:34:48] <chicocvenancio>	 I don't think we have any other .net bot running. But if there is a .deb package for all dependencies there is a good chance it is possible to run as a VPS project 
[03:35:15] <chicocvenancio>	 MarioFinale: ^
[03:35:40] <chicocvenancio>	 *meant Toolforge, not toolserver
[03:38:22] <MarioFinale>	 chicocvenancio: I just need that specific package, my bot only uses between 20-100Mb of ram, 20Mb of disk for Log storage and isn't CPU intensive at all, i should specify that?
[03:39:13] <MarioFinale>	 should i*
[03:39:16] <chicocvenancio>	 Well, not strictly needed for low values, but it'd be nice 
[03:39:48] <chicocvenancio>	 Definitely mention the package and the purpose of the bot, though 
[03:53:24] <MarioFinale>	 chicocvenancio: I made some changes, what about now?
[03:54:13] <chicocvenancio>	 It's good now, thanks 
[03:54:53] <chicocvenancio>	 I can't promise, but I will poke the right people tomorrow 
[03:54:59] <MarioFinale>	 chicocvenancio: no, thanks you!
[03:55:07] <MarioFinale>	 ok, thanks
[03:55:45] <MarioFinale>	 }i have to go, bye.
[03:55:49] <MarioFinale>	 thanks
[03:55:50] <MarioFinale>	 :)
[04:39:28] <bd808>	 !log tools Killed 4 orphan php-fcgi processes from jembot that were running on tools-webgrid-lighttpd-1417 and pegging the cpu there
[04:39:33] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL
[04:56:42] <bd808>	 !log tools Rescheduled 30 of the 60 tools running on tools-webgrid-lighttpd-1421 (T186830)
[04:56:48] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL
[04:56:48] <stashbot>	 T186830: tools/meta/accounteligibility: notice of heavy use till the end of February - https://phabricator.wikimedia.org/T186830
[05:01:17] <bd808>	 why are there orphan php-cgi processes for jembot all over the grid?
[05:02:01] <bd808>	 !log tools Killed 4 orphan php-fcgi processes from jembot that were running on tools-webgrid-lighttpd-1421 and pegging the cpu there
[05:02:07] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL
[05:05:24] <bd808>	 !log tools Killed 1 orphan php-fcgi process from jembot that were running on tools-webgrid-lighttpd-1409
[05:05:28] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL
[05:06:27] <bd808>	 !log tools Killed 4 orphan php-fcgi processes from jembot that were running on tools-webgrid-lighttpd-1411
[05:06:32] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL
[05:07:40] <bd808>	 !log tools Killed 4 orphan php-fcgi processes from jembot that were running on tools-webgrid-lighttpd-1426
[05:07:45] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL
[05:23:09] <zhuyifei1999_>	 bd808: https://phabricator.wikimedia.org/T182070 ?
[05:40:43] <bd808>	 argh. iabot has the orphan procs scattered around too
[06:15:14] <bd808>	 !log tools Killed orphan processes owned by iabot, dupdet, and wsexport scattered across the webgrid nodes
[06:15:20] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL
[10:14:39] <cocaine->	 (_)                                               | |
[10:14:39] <cocaine->	 (_)                                               | |
[10:14:39] <cocaine->	 (_)                                               | |
[10:14:39] <cocaine->	 (_)                                               | |
[10:14:42] <cocaine->	 _ _ __ ___   ___ _   _ _ __   ___ _ __ _ __   ___| |_ ___   ___  _ __ __ _
[10:14:42] <cocaine->	 _ _ __ ___   ___ _   _ _ __   ___ _ __ _ __   ___| |_ ___   ___  _ __ __ _
[10:14:42] <cocaine->	 _ _ __ ___   ___ _   _ _ __   ___ _ __ _ __   ___| |_ ___   ___  _ __ __ _
[10:14:42] <cocaine->	 _ _ __ ___   ___ _   _ _ __   ___ _ __ _ __   ___| |_ ___   ___  _ __ __ _
[10:14:46] <cocaine->	 | | '__/ __| / __| | | | '_ \ / _ \ '__| '_ \ / _ \ __/ __| / _ \| '__/ _` |
[10:14:46] <cocaine->	 | | '__/ __| / __| | | | '_ \ / _ \ '__| '_ \ / _ \ __/ __| / _ \| '__/ _` |
[10:14:46] <cocaine->	 | | '__/ __| / __| | | | '_ \ / _ \ '__| '_ \ / _ \ __/ __| / _ \| '__/ _` |
[10:14:46] <cocaine->	 | | '__/ __| / __| | | | '_ \ / _ \ '__| '_ \ / _ \ __/ __| / _ \| '__/ _` |
[10:14:50] <cocaine->	 | | | | (__ _\__ \ |_| | |_) |  __/ |  | | | |  __/ |_\__ \| (_) | | | (_| |
[10:14:50] <cocaine->	 | | | | (__ _\__ \ |_| | |_) |  __/ |  | | | |  __/ |_\__ \| (_) | | | (_| |
[10:14:50] <cocaine->	 | | | | (__ _\__ \ |_| | |_) |  __/ |  | | | |  __/ |_\__ \| (_) | | | (_| |
[10:14:50] <cocaine->	 | | | | (__ _\__ \ |_| | |_) |  __/ |  | | | |  __/ |_\__ \| (_) | | | (_| |
[10:14:54] <cocaine->	 |_|_|  \___(_)___/\__,_| .__/ \___|_|  |_| |_|\___|\__|___(_)___/|_|  \__, |
[10:14:54] <cocaine->	 |_|_|  \___(_)___/\__,_| .__/ \___|_|  |_| |_|\___|\__|___(_)___/|_|  \__, |
[10:14:54] <cocaine->	 |_|_|  \___(_)___/\__,_| .__/ \___|_|  |_| |_|\___|\__|___(_)___/|_|  \__, |
[10:14:54] <cocaine->	 |_|_|  \___(_)___/\__,_| .__/ \___|_|  |_| |_|\___|\__|___(_)___/|_|  \__, |
[10:35:03] <arturo>	 !log tools deploy https://gerrit.wikimedia.org/r/#/c/409226/ T179343 T182562 T186846
[10:35:11] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL
[10:35:11] <stashbot>	 T179343: Add xhprof PHP package - https://phabricator.wikimedia.org/T179343
[10:35:11] <stashbot>	 T186846: Install mono-vbnc on Toolforge hosts - https://phabricator.wikimedia.org/T186846
[10:35:12] <stashbot>	 T182562: Please install python3-tk on toolforge nodes - https://phabricator.wikimedia.org/T182562
[12:01:47] <addshore>	 Hi all!
[12:02:00] <addshore>	 Is it possible to add an ldap group as a maintainer of a tool?
[12:02:09] <addshore>	 or add an ldap group to a cloud project?
[12:02:14] <addshore>	 rather than adding users individually?
[12:03:23] <arturo>	 addshore: I would say no. Only individual users. But I'm not sure and this should be confirmed with other team members
[12:03:44] <addshore>	 cool, thanks for the initial reply :)
[12:04:04] <addshore>	 Yeh, i just realised wmde have a bunch of tools, and an ldap account, but right now we have to keep the lists of maintainers upt o date by hand
[12:04:14] <addshore>	 and regularly have people missing off of some toold etc
[12:04:16] <addshore>	 *tools
[12:05:10] <arturo>	 yeah I understand the use case, but I can't really speak with confidence about whether we support that or not
[16:49:44] <paladox>	 !log git testing some gitiles changes on gerrit-test3 (so disabling puppet)
[16:49:47] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Git/SAL
[17:05:36] <paladox>	 bd808 hi, when i rebooted my instance and ran mwvgrant up, i got
[17:05:38] <paladox>	 ==> default: There was a problem while downloading the metadata for your box
[17:06:04] <paladox>	 https://phabricator.wikimedia.org/P6674
[17:12:23] <bd808>	 paladox: I don't think I can do anything to fix the Vagrant metadata service
[17:12:28] <paladox>	 ok
[17:12:48] <chicocvenancio>	 paladox: does it fail? 
[17:12:56] <paladox>	 chicocvenancio nope, it still runs
[17:14:58] <bd808>	 This is the box image that the Vagrantfile should be referencing -- https://app.vagrantup.com/debian/boxes/stretch64
[17:15:25] <bd808>	 hmmm... they stopped building it for lxc at v9.1.0
[17:17:02] <bd808>	 its the most downloads ranked stretch + lxc image still though. Hopefully their metadata server just had a hisccup
[17:17:07] <bd808>	 *hiccup
[17:19:27] <paladox>	 yep
[17:25:18] <arturo>	 chicocvenancio: perhaps do you have any more information available for addshore question?
[17:26:19] <chicocvenancio>	 arturo: nope
[17:26:30] <arturo>	 thx
[17:26:47] * chicocvenancio check phabricator if there is a task about ldap and tool membership 
[17:27:24] <arturo>	 addshore: could you please open a phab task requesting the feature?
[17:30:48] <bd808>	 I'll close it as declined :)
[17:31:34] <chicocvenancio>	 Looking at previous tickets, this was the behavior previously. 
[17:31:34] <bd808>	 addshore: there is a hacky way to do that today. A tool (say wmde-staff) can be a maintainer of another tool (say wmde-cool-stuff)
[17:32:16] <arturo>	 :-) ouch
[17:32:21] <bd808>	 I *think* that when you do that the workflow ends up being "become the member tool and then become the desired tool"
[17:33:18] <bd808>	 there was no time in the Tool Labs/Toolforge era that arbitrary LDAP groups could be attached to service accounts that I know of
[17:34:31] <bd808>	 there is also no facility for managing arbitrary LDAP groups that we expose to people
[17:34:57] <bd808>	 so adding that as a feature brings a whole lot of questions about how to do it
[17:35:48] <bd808>	 like "who can add/remove people from groups" and "what constraints are there on creating groups" and "where will there be a UI to this from"
[17:36:42] <bd808>	 coding a feature into Striker that only benefits WMDE is not going to make my priority list
[17:53:59] <DMaza>	 when querying the replicas, does it make a difference what cluster to use? meaning, does it matter if I connect to enwiki.analytics.db.svc.eqiad.wmflabs but then I query itwiki_p ? 
[17:55:34] <DMaza>	 what's the point of the project name in front of the cluster alias?
[17:58:22] <bd808>	 DMaza: the project name part makes sure you get to a database server that has the data for that project
[17:59:20] <chicocvenancio>	 DMaza: from what I understand, at this point all hosts have all information, but it will change in the future
[17:59:24] <bd808>	 the difference between "web" and "analytics" is that the web cluster is tuned for short, low latency responses and the analytics cluster is tuned for big, high latency responses
[18:00:16] <bd808>	 chicocvenancio is correct. Today you can hop from one project db to another without switching to a new connection, but that will not be the case at some point in the future
[18:00:20] <DMaza>	 the docs explain web vs analytics, but it was not clear to the project name.. and yes chicocvenancio, all hosts have all the information right now
[18:00:47] <chicocvenancio>	 that is mainly because this is unsuported
[18:00:58] <DMaza>	 any idea when that will change?
[18:01:28] <chicocvenancio>	 when one of the wikis becomes large enough so that it is needed, I suppose
[18:01:47] <bd808>	 yeah, it will be in response to data size or traffic
[18:02:05] <DMaza>	 thank you guys
[18:02:26] <bd808>	 right now we are on servers that are *much* more powerful than the last generation, but we will outgrow putting all of the data on each of them at some point
[18:04:13] <chicocvenancio>	 DMaza: per DBA comment on list "(we are currently at 7.8TB used out of 12TB)"
[18:05:41] <DMaza>	 the reason I'm asking is because I'm working on a tool that needs access to multiple project dbs.. I guess I'll have to factor this when creating the connection so it won't break in the future
[18:05:48] <chicocvenancio>	 (as of last month)
[18:05:53] <chicocvenancio>	 yes
[18:06:03] <DMaza>	 chicocvenancio: that's past 50% :S
[18:06:11] <chicocvenancio>	 there is some good work on GUC by Krinkle recently that maybe relevant
[18:06:25] <DMaza>	 GUC ?
[18:06:57] <chicocvenancio>	 its a tool to get Global User Contributions
[18:07:14] <chicocvenancio>	 so it hits all Wikis for the users contributions
[18:07:24] * chicocvenancio tries to find the relevant task
[18:07:32] <DMaza>	 this one ? https://github.com/wikimedia/labs-tools-guc
[18:08:08] <chicocvenancio>	 yes
[18:09:26] <chicocvenancio>	 T182916 and T186436 are the tasks I was looking for
[18:09:26] <stashbot>	 T186436: Improve GUC database handling - https://phabricator.wikimedia.org/T186436
[18:09:26] <stashbot>	 T182916: Database error: Unable to connect to s7.web.db.svc.eqiad.wmflabs - https://phabricator.wikimedia.org/T182916
[18:09:30] <DMaza>	 chicocvenancio: thanks, I'll take a look
[18:49:34] <MarioFinale>	 chicocvenancio: Hi, i saw your message on the wiki, so i readed how to use the grid engine. Just a little question: My bot read/write on some files on his parent directory, that should work OK or i need to do some config?
[18:50:23] <chicocvenancio>	 MarioFinale: that's fine, either Grid or k8s will have the same $home folder for your tool
[18:51:10] <chicocvenancio>	 in your specific case, please reach out if any problems occur since Mono was just installed this morning.
[18:51:47] <chicocvenancio>	 I think it should be on all machines by now, but if you find any problems ping me and I'll help you sort it out
[18:54:36] <MarioFinale>	 chicocvenancio: oh, there are some details with mono, the default install of the package doesnt come with any CA certificate, so all SSL/Https connections fail. I just installed a bunch of CA Root certificates using the "mozroot" tools that comes with mono. That should be reflected on the instance on my bot running on the grid?
[18:55:37] <chicocvenancio>	 what is the path to the certificates?
[18:56:01] <MarioFinale>	 hmm.. give me a minute, let me see...
[18:56:17] <chicocvenancio>	 unless they live in your tool folder, probably not
[19:02:02] <chicocvenancio>	 ok, I see now
[19:03:20] <chicocvenancio>	 you ran everything with your user `mariofinale`, the certs are in `/home/mariofinale/.config/.mono/certs`
[19:05:36] <chicocvenancio>	 MarioFinale: you there?
[19:05:52] <MarioFinale>	 chicocvenancio: Ok, they are stored on /.config/.mono/certs/Trust
[19:06:01] <chicocvenancio>	 yeah
[19:06:05] <chicocvenancio>	 but on your home
[19:06:14] <MarioFinale>	 yes
[19:06:21] <MarioFinale>	 so, it wont work...}
[19:06:26] <chicocvenancio>	 you should become your tool first
[19:06:27] <MarioFinale>	 so, it wont work...?
[19:08:05] <MarioFinale>	 chicocvenancio: Where can i read about that
[19:08:13] <chicocvenancio>	 literally `become toolname`
[19:08:23] * chicocvenancio checks the page
[19:10:21] <chicocvenancio>	 https://wikitech.wikimedia.org/wiki/Help:Toolforge#Using_a_Tool_account
[19:12:31] <chicocvenancio>	 so, the steps are, 1.  become your tool; 2. run `mozroot` as your tool; 3. send the job to the grid (or k8s)
[19:13:03] <MarioFinale>	 i'll try
[19:13:14] <MarioFinale>	 give me 3-5 min
[19:13:30] <chicocvenancio>	 the rationale for this is that the bastion host is for login and development, if all the users run tools directly there it becomes slow
[19:13:43] <chicocvenancio>	 and degrades experience for everyone
[19:13:55] <chicocvenancio>	 in fact it happens often
[19:14:06] <wikibugs>	 (03PS1) 10Bstorm: tools-cdnjs-beta: switching to a reverse proxy solution [labs/tools/cdnjs-index] (beta) - 10https://gerrit.wikimedia.org/r/409416 (https://phabricator.wikimedia.org/T182604)
[19:16:25] <MarioFinale>	 i see
[19:16:34] <MarioFinale>	 makes sense
[19:18:53] <MarioFinale>	 can i run my tool on a screen? so can i attach/detach when i need to enter some commands?
[19:19:29] <chicocvenancio>	 is it interactive?
[19:20:27] <MarioFinale>	 yes but only for the initial config, then doesn't need any direct interaction.
[19:21:53] <zhuyifei1999_>	 you might want to make the bot load config from a file instead of loading from interactive input
[19:22:14] <MarioFinale>	 zhuyifei1999_: It does
[19:22:38] <zhuyifei1999_>	 so the bot can start in a schedule without any human intervention
[19:23:05] <zhuyifei1999_>	 if it can run without interactive input, why do you need screen?
[19:23:42] <zhuyifei1999_>	 non-interactive bot jobs should be submitted to https://wikitech.wikimedia.org/wiki/Help:Toolforge/Grid
[19:24:03] <MarioFinale>	 now than i think about that, i should just copy the config file from the old server
[19:24:57] <chicocvenancio>	 if you need to run it quickly once to build the config file in bastion it would be fine... but not for continuous work
[19:24:58] <MarioFinale>	 zhuyifei1999_: The bot Only needs human intervention when the config file is missing or damaged
[19:25:24] <zhuyifei1999_>	 heh why would it get damaged?
[19:25:39] <MarioFinale>	 my old server was pretty crappy :(
[19:26:43] <zhuyifei1999_>	 well, we never had files just mysteriously go missing or gone
[19:26:55] <MarioFinale>	 that's good to hear
[19:27:14] <chicocvenancio>	 yeah, but do have backup of important data
[19:27:44] <annika>	 what would be needed to scp between two instances of a project?
[19:29:17] <zhuyifei1999_>	 the only time that I know of that is remotely similar is when some tools marked their directories as world-writable (so deleteable by everyone) without knowing what that 'mark' actually meant, and then someone accidentally tried to delete everything that can be deleted. We restored the files back anyhow
[19:30:14] <mutante>	 annika: there is a puppet class called rsync::quickdatacopy.  it sets up syncing of data between 2 machines using rsync and we use it in production. it might be useful and make it easy for you.. if you already use puppet by any chance
[19:30:36] <mutante>	 (and maybe security groups need to be edited to open port)
[19:30:47] <mutante>	 but that is possible for a project admin
[19:31:20] <annika>	 so when port 22 is open (ssh/scp) this is not enough for scp?
[19:31:51] <mutante>	 for the key-based login to work you also need to forward your agent
[19:31:57] <annika>	 aaah
[19:31:58] <mutante>	 with ssh -A
[19:32:05] <mutante>	 in prod this isnt allowed
[19:32:11] <mutante>	 for security reasons 
[19:33:54] <mutante>	 if you are the only root it wouldn't be an issue
[19:34:11] <zhuyifei1999_>	 annika: you might also want to see https://superuser.com/questions/686394/scp-between-two-remote-hosts-from-my-third-pc
[19:35:23] <annika>	 nice
[19:40:58] <MarioFinale>	 chicocvenancio: Ok, is this 
[19:41:02] <MarioFinale>	 command ok?
[19:41:04] <MarioFinale>	 jstart -stderr -continuous -cwd -N BOT-NAME -M My@email -m beas mono [exe-file]
[19:42:09] <chicocvenancio>	 looks good
[19:42:25] <MarioFinale>	 ok, i'll try
[19:42:30] <chicocvenancio>	 assuming BOT-NAME and My@email will be replaced by correct values
[19:42:36] <MarioFinale>	 yep
[19:43:02] <chicocvenancio>	 also, cwd will change the working directory
[19:43:13] <chicocvenancio>	 not sure how mono will deal with that
[19:44:11] <MarioFinale>	 hmm.. let's see
[19:49:07] <chicocvenancio>	 its getting a network error...
[19:49:16] <MarioFinale>	 yep
[19:50:36] <MarioFinale>	 There is some network/proxy setting that i need to do?
[19:51:43] <chicocvenancio>	 let me check things
[19:52:21] <chicocvenancio>	 did you kill it?
[19:52:34] <MarioFinale>	 yes
[19:53:14] <MarioFinale>	 do you need me to start it again?
[19:53:55] <chicocvenancio>	 I can do it, but repeating the command with all your options is better
[19:54:03] <MarioFinale>	 ok
[19:54:23] <MarioFinale>	 done
[19:55:54] <chicocvenancio>	 now you used a different directory, it can't find the .exe
[19:58:10] <MarioFinale>	 What about now?
[19:58:32] <chicocvenancio>	 its going, probably will give the network error again
[19:58:42] <chicocvenancio>	 I'll check whats going on now
[19:59:53] <MarioFinale>	 yep, network error again
[20:00:06] <MarioFinale>	 Timeour
[20:00:09] <MarioFinale>	 Timeout*
[20:00:38] <Krinkle>	 bd808: Looks like stashbot isn't specifying the project in its phab message - https://phabricator.wikimedia.org/T186289#3957206
[20:01:11] <bd808>	 Krinkle: T186845 :)
[20:01:11] <stashbot>	 T186845: Display project name in SAL messages posted to Phabricator - https://phabricator.wikimedia.org/T186845
[20:03:03] <Krinkle>	 bd808: Hehe, right. 
[20:03:04] <Krinkle>	 Th
[20:03:06] <Krinkle>	 x
[20:13:02] <bd808>	 SQL: can you change the "cron-28" job for aivanalysis to run less often than every minute? Running anything that often from cron is a wasteful pattern. The cost of setting up and tearing down a grid job is probably a lot higher than you think. :)
[20:14:16] <bd808>	 running every 5 minutes is ok, but if you for some reason need a tighter loop than that I would suggest rewriting it as a continuous job that sleeps before looping back to the start
[20:29:46] <chicocvenancio>	 huh
[20:30:04] <chicocvenancio>	 MarioFinale: it works on the same host it is being run if I run it there...
[20:31:35] <MarioFinale>	 maybe the -cwd parameter is giving problems...
[20:31:44] <chicocvenancio>	 thats my guess
[20:31:53] <chicocvenancio>	 just stopped and resent without it
[20:31:58] <MarioFinale>	 ok
[20:32:05] <MarioFinale>	 give me a minute
[20:33:59] <MarioFinale>	 done
[20:36:44] <chicocvenancio>	 again failed
[20:36:50] <MarioFinale>	 yep, same
[20:36:58] <chicocvenancio>	 could you start a new phabricator task about this
[20:37:04] <chicocvenancio>	 ?
[20:37:48] <chicocvenancio>	 I'll fill in everything I've done to debug and the more knowledgeable folk from the team can find the reason for this weird behaviour 
[20:39:04] <wikibugs>	 (03CR) 10Bstorm: [C: 032] "Merging since this is a separate branch for the beta project" [labs/tools/cdnjs-index] (beta) - 10https://gerrit.wikimedia.org/r/409416 (https://phabricator.wikimedia.org/T182604) (owner: 10Bstorm)
[20:39:09] <MarioFinale>	 Ok, what would be a suitable title for the task?
[20:39:38] <wikibugs>	 (03CR) 10Bstorm: [V: 032 C: 032] tools-cdnjs-beta: switching to a reverse proxy solution [labs/tools/cdnjs-index] (beta) - 10https://gerrit.wikimedia.org/r/409416 (https://phabricator.wikimedia.org/T182604) (owner: 10Bstorm)
[20:39:46] <chicocvenancio>	 "Network error from grid with mono bot", or something like that
[20:40:00] <MarioFinale>	 thanks
[20:41:16] <wikibugs>	 (03CR) 10Zhuyifei1999: "I wonder if we can also get T173996 fixed." (039 comments) [labs/tools/cdnjs-index] (beta) - 10https://gerrit.wikimedia.org/r/409416 (https://phabricator.wikimedia.org/T182604) (owner: 10Bstorm)
[20:45:41] <wikibugs>	 (03CR) 10Bstorm: [V: 032 C: 032] "Yup, looking at it, that token is extremely restricted to viewing public repos and cannot really do anything, but I'm resetting and starti" [labs/tools/cdnjs-index] (beta) - 10https://gerrit.wikimedia.org/r/409416 (https://phabricator.wikimedia.org/T182604) (owner: 10Bstorm)
[20:52:48] <MarioFinale>	 chicocvenancio: How it is? T186924
[20:52:48] <stashbot>	 T186924: Network error from grid with mono bot - https://phabricator.wikimedia.org/T186924
[20:59:10] <chicocvenancio>	 MarioFinale: thats great, I'm adding details there now
[20:59:17] <MarioFinale>	 thanks
[21:08:36] <zhuyifei1999_>	 MarioFinale: I'm wondering, why much be the program arguments be concealed?
[21:11:06] <MarioFinale>	 zhuyifei1999_: my habit of hide personal data, but my toolforge email is public so...
[21:11:25] <MarioFinale>	 zhuyifei1999_: just an habit
[21:14:26] <chicocvenancio>	 bd808: the tools has no `.profile`or `.bashrc` files 
[21:15:22] <bd808>	 weird
[21:15:37] <zhuyifei1999_>	 so direct run on grid is okay but not with jsub?
[21:16:27] <zhuyifei1999_>	 probably environment vars (eg. locale), or expecting tty / stdin/out/err
[21:17:01] <chicocvenancio>	 weird indeed
[21:17:49] <zhuyifei1999_>	 where is the source code of the bot? I don't think I see it
[21:18:24] <chicocvenancio>	 MarioFinale: ^
[21:18:25] <zhuyifei1999_>	 never mind found it on github
[21:18:37] * chicocvenancio shrugs
[21:24:04] <wm-bot5>	 I will let you know when I see MarioFinale and I will deliver that message to them
[21:24:04] <zhuyifei1999_>	 @notify MarioFinale FYI: I chmod your Config.cfg as non-world-readable (o-r) due to it containing passwords
[21:28:15] <wikibugs>	 (03PS1) 10Bstorm: tools-cdnjs-beta: switching to a reverse proxy solution [labs/tools/cdnjs-index] (beta) - 10https://gerrit.wikimedia.org/r/409448 (https://phabricator.wikimedia.org/T182604)
[21:33:26] <wikibugs>	 (03CR) 10Bstorm: "I haven't changed how I get the paths since I fell that is the correct way to do that when messing with paths in python.  However, this ve" [labs/tools/cdnjs-index] (beta) - 10https://gerrit.wikimedia.org/r/409448 (https://phabricator.wikimedia.org/T182604) (owner: 10Bstorm)
[21:38:16] <chicocvenancio>	 zhuyifei1999_: you make me look bad
[21:38:25] <zhuyifei1999_>	 :(
[21:38:35] <chicocvenancio>	 no, thanks
[21:39:28] <chicocvenancio>	 -mem 1g makes it work
[21:40:21] <chicocvenancio>	 though it still consumes a lot more cpu than when lauched interactively
[21:43:38] <chicocvenancio>	 `-mem 2g` fixes that as well
[21:44:13] <chicocvenancio>	 it then consumes 1.2gb ram, explaining everything
[21:44:40] <chicocvenancio>	 MarioFinale: zhuyifei1999_ solved it
[21:45:01] <chicocvenancio>	 add `-mem 2g`to your jstart call
[21:45:13] <MarioFinale>	 oh
[21:45:15] <MarioFinale>	 thanks
[21:45:40] <MarioFinale>	 sorry i got disconnected from IRC and i dind't noticed
[21:45:55] <chicocvenancio>	 it was now, don't worry
[21:46:07] <MarioFinale>	 zhuyifei1999_: Ok, i'll do it now
[21:46:14] <MarioFinale>	 zhuyifei1999_: i forgot about that
[21:47:16] <zhuyifei1999_>	 np
[21:48:59] <MarioFinale>	 zhuyifei1999_: i'ts ok now? 
[21:49:27] <MarioFinale>	 i ran chmod a-r [file] then chmod u=rxw [file]
[21:53:15] * zhuyifei1999_ is outside now
[21:53:32] <chicocvenancio>	 MarioFinale: yeah, the permissions are ok now
[21:53:42] <zhuyifei1999_>	 I chmodded it already though
[21:53:49] <MarioFinale>	 oh
[21:53:54] <MarioFinale>	 thanks anyway
[21:54:34] <MarioFinale>	 i created the task, but it appearts that it's not running...
[21:55:15] <chicocvenancio>	 it is not finding the .exe
[21:55:21] <chicocvenancio>	 see the .err file
[21:55:45] <MarioFinale>	 oh
[21:55:48] <MarioFinale>	 the directory
[21:56:10] <wikibugs>	 (03PS2) 10Bstorm: tools-cdnjs-beta: switching to a reverse proxy solution [labs/tools/cdnjs-index] (beta) - 10https://gerrit.wikimedia.org/r/409448 (https://phabricator.wikimedia.org/T182604)
[21:58:16] <chicocvenancio>	 MarioFinale: do `jstop PeriodiBOT`
[21:58:26] <chicocvenancio>	 then resend the job to grid
[21:58:36] <MarioFinale>	 maybe doesn't have permission to read the file
[21:59:18] <chicocvenancio>	 how did you send it?
[21:59:33] <MarioFinale>	  jstart -stderr -continuous -N PeriodiBOT -M mariofinale@hotmail.com -m beas -mem 2g mono PeriodiBOT-IRC.exe
[21:59:51] <chicocvenancio>	 so, the working directory is the toolhome
[22:00:22] <MarioFinale>	 it needs the -cwd parameter then?
[22:00:42] <chicocvenancio>	 from there PeriodiBOT-IRC.exe is in the PeriodiBOT/ directory
[22:00:53] <MarioFinale>	 yep
[22:01:50] <chicocvenancio>	 well, that is one way, `cd PeriodiBOT` then use `-cwd`.  I would just do  jstart -stderr -continuous -N PeriodiBOT -M mariofinale@hotmail.com -m beas -mem 2g mono ~/PeriodiBOT/PeriodiBOT-IRC.exe` however
[22:02:00] <MarioFinale>	 btw i don't know why is consuming that mucho memory, on ubuntu xenial only uses 50-90mb
[22:02:15] <MarioFinale>	 much*
[22:02:25] <chicocvenancio>	 `jstart -stderr -continuous -N PeriodiBOT -M mariofinale@hotmail.com -m beas -mem 2g mono ~/PeriodiBOT/PeriodiBOT-IRC.exe`
[22:04:00] <MarioFinale>	 yep
[22:04:04] <MarioFinale>	 now it works
[22:04:35] <MarioFinale>	 But idk why it's using so mucho memory
[22:04:37] <MarioFinale>	 much*
[22:05:00] <MarioFinale>	 i've never saw my bot using 1.2 GB of RAM
[22:05:33] <chicocvenancio>	 mention that on the task, I don't know how else to debug it, someone else might
[22:06:08] <zhuyifei1999_>	 (back)
[22:06:50] <zhuyifei1999_>	 oh grid memory calculation is super weird
[22:07:56] <MarioFinale>	 maybe it loads the entire mono lib on ram?
[22:08:00] <MarioFinale>	 idk
[22:08:05] <MarioFinale>	 weird
[22:10:41] <zhuyifei1999_>	 yeah it allocated 1209m, but only uses 47m
[22:11:25] * chicocvenancio shrugs
[22:11:37] <MarioFinale>	 oh no!
[22:11:50] <zhuyifei1999_>	 don't worry about over-allocation
[22:12:01] <MarioFinale>	 well i have to go, thanks for your help!
[22:12:03] <zhuyifei1999_>	 the kernel handles them quite well
[22:12:10] <MarioFinale>	 i'll try to dont
[22:12:43] * zhuyifei1999_ digging history about that memory weirdness
[22:12:58] <MarioFinale>	 thanks for the help and patience
[22:13:00] <MarioFinale>	 bye