[12:59:44] !log tools.heritage Deploy latest from Git master: 52b2f4d [12:59:47] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.heritage/SAL [19:15:32] btw [19:15:42] how long will tools creation takes? [19:16:16] I created some tools twenty minutes ago and it's still not found through 'become' [19:28:41] how long will tool creation take? [19:28:45] hi viztor_ [19:28:48] saw your messages [19:29:00] hi sorry, i think the connection wasn't stable [19:29:04] np [19:29:15] it should be pretty quick, I expect that system will just be writing to LDAP and become should pick it up straight away? [19:29:45] it has been twenty minutes since I created the tools and its still not available [19:29:45] is this vizbot, am, or amstats? [19:29:50] am and amstats [19:29:57] interesting, they do appear [19:30:00] like inside LDAP [19:30:07] hmmm. [19:30:14] although become thinks it does not exist... [19:30:15] * Krenair digs [19:30:34] running become vizbot is ok, but it's created ages ago [19:30:46] neither am nor amstats is working [19:30:46] oh right [19:30:51] if ! id "$prefix.$tool" >/dev/null 2>&1 || ! [ -d "/data/project/$tool" ]; then [19:30:52] echo "$(basename $0): no such tool '$tool'" >&2 [19:30:52] exit 1 [19:31:12] it doesn't just need to exist in LDAP - that bit is working fine and I've confirmed with `id` - it needs to also have a dir under /data/project/ [19:31:23] have a funny feeling I might know what creates those... [19:31:36] should i create the data folder manually? [19:31:36] is it one of the credentials-managing processes? [19:31:42] you won't be able to I wouldn't think [19:31:58] i suppose [19:32:14] maybe it's done by one of those maintenance script? [19:32:31] should be on cron without need for sysadmin manual intervention [19:32:44] so I think this dir probably gets created when the user is issued credentials to do things like connect to the mysql replicas [19:32:47] why couldn't it just create the folder automatically when a ldap user is present? [19:33:22] not really sure how to proceed [19:34:45] no point setting that up if something else is already responsible for creating it [19:34:54] in this case maybe the thing responsible is not running properly... [19:35:08] maintain-kubeusers is supposed to create this dir if it doesn't already exist [19:36:33] (it looks like we did indeed used to do it in maintain-dbusers which would be handling replica credentials, seems that got changed at some point due to race conditions) [19:36:44] i suppose someone can create the folder manually if the script is not working? [19:37:00] it's all blackbox to me [19:37:47] The only people who would be able to do that would probably fix the script first, at which point there'd be no need to touch an individual tool [19:38:48] that's probably true [19:39:36] Looks like the host that's supposed to run maintain-kubeusers is the tools k8s master (that's logical), however I am not able to log in there (not being a tools admin) [19:45:33] hmmmm [19:45:40] who are the tools admin [19:47:04] I just found one [19:47:23] (it's the people listed as members inside `ldapsearch -x cn=tools.admin`) [19:54:05] i suppose they can fix this relatively quickly [20:46:38] Krenair, viztor_, it's https://phabricator.wikimedia.org/T194859. Restarting the script will un-stick it for a bit and create the new tools, but the underlying problem hasn't been fixed yet. [20:51:34] !log `sudo service maintain-kubeusers restart` on tools-k8s-master-01 [20:51:35] Reedy: Unknown project "`sudo" [20:51:42] !log tools `sudo service maintain-kubeusers restart` on tools-k8s-master-01 [20:51:43] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [21:00:50] viztor_, ^ try now [21:00:57] thanks! [21:01:02] it seems to be working [21:01:15] though from the log it appears to be a chronical issue. [21:01:33] That's why there's a bug that's not fixed for it ;) [21:02:56] perhaps a cron job could be set up to restart it every few hours? [21:03:08] just as a temporary fix [21:04:44] temporary fixes have an awful habit of becoming permanent around here [21:05:20] hahahaha [21:05:40] a lot of TLS cert rotations that wikimedia misc services rely on currently are triggered by a cronjob that sends a SIGHUP :) [21:10:28] lol [21:10:52] cron rules