[01:40:41] <Reedy>	 !log commonshelper restarted webservice due to error 500
[01:40:42] <stashbot>	 Reedy: Unknown project "commonshelper"
[01:40:42] <stashbot>	 Reedy: Did you mean to say "tools.commonshelper" instead?
[01:40:50] <Reedy>	 !log tools.commonshelper restarted webservice due to error 500
[01:40:53] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.commonshelper/SAL
[16:42:08] <ebernhardson>	 seems a silly question i should be able to answer, but how do i add a user to a group in cloud? i ran `sudo adduser ebernhardson docker`, logged out and logged back in, but my group membership doesn't include docker (although it is in /etc/group).  nsswitch.conf defaults to 'files ldap' so i would have expected it to merge the two (being somewait naive about nsswitch)
[16:42:42] <ebernhardson>	 its a new debian stretch based instance, fwiw
[16:46:26] <bd808>	 ebernhardson: hmm... I would think that would work
[16:47:14] <bd808>	 you can always use the `sudo` club instead of adding rights to your user, but it would be nice to know how to do this sort of thing
[16:48:19] <ebernhardson>	 yea i'll probably just sudo it. Was trying to do things the right way :)
[17:10:08] <bd808>	 ebernhardson: this is probably worth opening a phab task about. We can have someone try to figure it out, or Andrew or Chase may know what the deal is
[17:11:20] <ebernhardson>	 ok i'll get a ticket going
[17:23:50] <paravoid>	 ebernhardson: did you try after a while? there's some caching involved that may affect this
[17:32:26] <bd808>	 Yeah, nscd would cache some group stuff I think. In theory it has inotify watches on the files, but maybe that doesn't always work
[17:32:58] <bd808>	 sudo nscd -i /etc/passwd -- that would be worth trying
[17:34:18] <ebernhardson>	 lemme check now
[17:34:47] <ebernhardson>	 id still doesn't report docker, but /etc/group has `docker:x:498:ebernhardson`
[17:35:19] <bd808>	 maybe /etc/group is cached directly too, but I'm not sure
[17:57:40] <jem>	 Hi, tools-bastion-03 isn't responsive at the moment, is this known?
[17:59:09] <zhuyifei1999_>	 sigh tools-bastion-03 is "/always/" unresponsive
[18:04:16] <jem>	 Ok, but this is way out of normal
[18:05:18] <bd808>	 jem: I'll see if I can get in there an kill some processes. There have a been a *lot* of people running pywikibot directly on that host recently
[18:05:50] <jem>	 Ok, bd808, thanks
[18:06:26] <bd808>	 jem: as work around, you can try using tools-dev.wmflabs.org instead
[18:07:08] <jem>	 I take note... I knew about that but I usually forget
[18:10:13] <bd808>	 !log tools.legobot Eating a lot of io on tools-bastion-03 (tools-login) via a grep of "goodarticles.out"
[18:10:16] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.legobot/SAL
[18:12:38] <bd808>	 !log tools.legobot Killed grep of goodarticles.out that was consuming 99% of io on tools-bastion-03
[18:12:39] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.legobot/SAL
[18:14:46] <bd808>	 !log tools.congressedits Killed /data/project/congressedits/congresseditors/congresseditors.js process running on tools-bastion-03. See https://wikitech.wikimedia.org/wiki/Help:Toolforge/Grid for instructions on running jobs on the grid.
[18:14:48] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.congressedits/SAL
[18:17:26] <legoktm>	 bd808: ah crap, my bad
[18:21:52] <bd808>	 legoktm: it happens :/
[18:27:55] <bd808>	 !log tools.tedbot Killed /data/project/tedbot/bot_data/wiki.pl process running on tools-bastion-03. See https://wikitech.wikimedia.org/wiki/Help:Toolforge/Grid for instructions on running jobs on the grid.
[18:27:57] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.tedbot/SAL
[18:30:02] <bd808>	 !log tools Load on tools-bastion-03 down to 0.72 from 17.47 after killing a bunch of local processes that should have been running on the job grid instead
[18:30:07] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL
[19:35:37] <musikanimal>	 where can I find the logs for when a k8s pod is restarted?
[19:35:53] <musikanimal>	 I see I can use `kubectl logs <pod-name>` but that only gives me a fresh log since the pod was last booted
[19:36:10] <musikanimal>	 I'm seeing that mine keeps restarting, so I assume it's somehow dying, but I don't know where to look for errors
[19:37:03] <madhuvishy>	 musikanimal: which tool is this?
[19:37:34] <musikanimal>	 tools.community-tech-tools
[19:37:48] <musikanimal>	 the deployment name is community-tech-tools.bot
[19:38:40] <musikanimal>	 and currently the pod is at community-tech-tools.bot-1903210647-32uhg
[19:42:10] <madhuvishy>	 musikanimal: i don't think there
[19:42:20] <madhuvishy>	 i dont there is a straight forward way
[19:44:02] <madhuvishy>	 musikanimal: I was watching the logs on the pod
[19:44:04] <madhuvishy>	 https://www.irccloud.com/pastebin/Lgqnhcvs/
[19:44:16] <musikanimal>	 yeah that looks right
[19:44:22] <musikanimal>	 it's restarting every 10 minutes, it seems
[19:44:54] <musikanimal>	 there haven't been any recent changes to the pages it looks for, so I'm not sure why it keeps dying
[19:45:21] <musikanimal>	 it's certainly nice that it automatically reboots, though!
[19:45:42] <madhuvishy>	 https://www.irccloud.com/pastebin/pprYk0hY/
[19:45:52] <madhuvishy>	 the pod itself has been up for 14h though
[19:46:06] <madhuvishy>	 but yeah 84 restarts
[19:46:11] <musikanimal>	 haha yeah
[19:46:13] <musikanimal>	 that's a lot!
[19:46:39] <madhuvishy>	 you might wanna add more logging from your service to see what's happening
[19:46:45] <musikanimal>	 I'm inclined to believe it's not the code erroring out, but something else that's making it restart
[19:46:51] <musikanimal>	 there's a lot of logging
[19:46:56] <musikanimal>	 everything it does
[19:47:17] <musikanimal>	 but like I said whenever it restarts it apparently loses the logs from the previous run
[19:47:31] <musikanimal>	 I guess I could make it manually write to a log file?
[19:47:56] <madhuvishy>	 yeah you can do that, we do that with webservices
[19:48:07] <madhuvishy>	 or you can watch with kubectl logs -f
[19:48:15] <musikanimal>	 oh nice, let me try that
[19:48:31] <madhuvishy>	 that's what I did, and I pasted whatever it threw
[19:48:45] <musikanimal>	 because actual errors should show up in the logs, too
[19:49:09] <musikanimal>	 I was just hoping there was a k8s restart log or something
[19:50:42] <madhuvishy>	 musikanimal: no, i doubt things outside the pod have visibility to what went on inside. I was looking at https://kubernetes.io/docs/tasks/debug-application-cluster/determine-reason-pod-failure/
[19:52:04] <musikanimal>	 oh neat-o, so I might be able to set up a termination log file
[19:53:02] <musikanimal>	 saw the error!
[19:53:07] <musikanimal>	 using -f
[19:53:19] <bd808>	 that trace that madhuvishy caught looks like your app is dying from MySQL connection hiccups
[19:53:20] <musikanimal>	 there error was "Error: Connection lost: The server closed the connection."
[19:53:32] <musikanimal>	 oh hmm, it was mysql, actually
[19:53:37] <musikanimal>	 dammit
[19:53:42] <bd808>	 musikanimal: same as https://www.irccloud.com/pastebin/Lgqnhcvs/ ?
[19:53:43] <madhuvishy>	 yeah, mysql may be just timing out and causing the script to die?
[19:54:01] <musikanimal>	 looks like it
[19:54:08] <musikanimal>	 that's lame that it would do that, cause it to error out
[19:54:13] <bd808>	 is the bot trying to hold an idle connection open?
[19:54:16] <musikanimal>	 because it's not making any queries
[19:54:24] <musikanimal>	 bd808: yeah I guess so, hah
[19:54:53] <musikanimal>	 I originally had it establish the connection, then do the actual conenction.connect() and connection.end() before/after each query
[19:55:07] <musikanimal>	 but that npm module won't let you re-connect with .connect after you end a session
[19:55:16] <musikanimal>	 I think I'm just doing this the wrong way
[19:56:35] <bd808>	 looks like that may be the library's designed behavior -- https://github.com/mysqljs/mysql#error-handling
[19:57:03] <bd808>	 "Note: 'error' events are special in node. If they occur without an attached listener, a stack trace is printed and your process is killed."
[19:57:46] <musikanimal>	 yeah, so I guess I'm going to create a new connection whenever a change is detected, and end the session after it's done processing
[19:58:04] <musikanimal>	 as opposed to creating the connection at the very beginning, before a change is detected
[19:59:29] <musikanimal>	 that seems like the correct way to do it, anyway!
[20:00:35] <bd808>	 it will make the DBAs happier if you aren't holding idle connections open :)
[20:03:26] <bd808>	 Our friends in CommTech have the 2017 survey open -- https://meta.wikimedia.org/wiki/2017_Community_Wishlist_Survey -- If there is a thing you think really needs to be fixed go and tell them about it.
[20:16:28] <musikanimal>	 okay I think I fixed it. Thank you for the help!
[21:11:17] <Amir1>	 !log wikilabels u_wikilabels=> update campaign set info_url = 'https://www.mediawiki.org/wiki/ORES#Edit_quality' where id = 41;
[21:11:20] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Wikilabels/SAL
[21:11:49] <Amir1>	 awight: Looks nice: http://labels.wmflabs.org/ui/enwiki/
[21:11:56] <Amir1>	 We should do it for more wikis
[21:12:32] <awight>	 Amir1: The OAuth2 flow was nice.  What am I looking at exactly, though?
[21:12:47] <Amir1>	 The "more info" thing
[21:12:53] <Amir1>	 it's coming from the database
[21:13:30] <awight>	 Amir1: cool!
[21:49:01] <tgr>	 wikilabels has OAuth 2?
[21:49:13] <Zppix>	 Yes