[08:34:28] !log wikispeech !log wikispeech Deploy latest from Git master: 85fa89a (T244344, T244345), f79fe03, 956e507, 66da277, 18c20c7 (T246085), 044eba0 (T249207), 2f98bef (T203161), 7aa7556 (T245823), 43e26a2 [08:34:38] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Wikispeech/SAL [12:55:58] السلام عليكم [12:57:56] helo [12:58:16] hi [13:01:33] I have some question about toolforg [13:03:25] Should the account be a bot or the main account? [13:07:51] Clayology, you should make an account for yourself and then you create a 'tool' to represent the bot [13:08:57] ok Thank you<3 [19:54:35] hello, is there any issue with tools-dev? I can only ssh through tools-login, otherwise I get 'Connection closed by remote host' errors, same key as usual [19:55:42] PeterBowman: we had a bunch of LDAP related problems yesterday, but we thought we had fixed them [19:56:06] PeterBowman: what's your shell name? [19:56:25] andrewbogott it's peterbowman [19:56:58] This is a long shot, but, try now? [19:57:25] fine, it works now :) thanks [19:57:36] that's mysterious but good I guess [19:58:03] this feels like we have a negative cache problem for the SSSD/NSS layer :/ [20:00:38] !log tools sss_cache -E on tools-sgebastion-07 [20:00:41] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [20:01:32] that didn't break my login at least [20:01:39] think I should do the same on login? [20:02:12] andrewbogott: seems like it wouldn't hurt [20:02:27] oh, it's already installed there too [20:02:58] um, wait [20:03:03] dev and login are apparently the same host :) [20:03:07] * andrewbogott finds the other ones [20:04:07] hmm... login should be sgebastion-07 and dev -08 [20:04:54] yeah, my ssh config was stupid [20:06:00] so that means that I ran a command on -07 and it fixed peter's login on 08 [20:06:29] !log tools sss_cache -E on tools-sgebastion-08 and tools-sgebastion-09 [20:06:31] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [20:25:52] andrewbogott sorry, it's broken again [20:26:04] yeah, for me too! weird [20:27:03] and it works now? [20:27:26] do we have a split brain in the LDAP backends maybe? [20:29:40] I'll compare [20:29:51] we're still thinking this would be in the 'groups' ou right? [20:30:21] andrewbogott: yeah, I think it would be the project-* groups [20:33:04] ugh, this output is unsorted so the diff is enormous [20:33:53] huh [20:35:04] so, I dumped and sorted the groups ou for both ldap servers. (Sorting means I can't get context for the diffs, just detect how large it is) [20:35:10] there is no difference in records [20:35:24] but serpens uses camel case all over the place, and seaborgium doesn't [20:35:32] this is upsetting but /probably/ not related to our issue... [20:35:55] Imagine this diff but for 260+ lines: [20:35:58] https://www.irccloud.com/pastebin/9U26iytY/ [20:38:51] andrewbogott: that is a really interesting issue [20:57:17] !log admin service sssd stop; rm -rf /var/lib/sss/db*; service sssd start on tools-sgebastion-08 [20:57:18] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [20:57:41] PeterBowman: it is working again, for me at least. Please ping if/when it fails for you again [20:58:27] andrewbogott okay, thanks! it works for me, too [20:59:40] bd808: that last rm -rf may have fixed it. I need to go pick up my groceries; if the issue recurs on a different host that's a good thing to try. If it recurs on the same host then we're nowhere [21:00:01] thanks andrewbogott [23:17:13] !log toolsbeta.test remove the .webservicerc file [23:17:14] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta.test/SAL