[09:41:45] <arturo>	 !log tools T221225 disable puppet agent in the bastions
[09:41:49] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL
[09:41:49] <stashbot>	 T221225: sssd integration needs to be updated to include sudo config from LDAP support - https://phabricator.wikimedia.org/T221225
[09:43:20] <arturo>	 !log tools T221225 use `profile::ldap::client::labs::client_stack: sssd` in the puppet bastion prefix
[09:43:23] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL
[09:49:22] <arturo>	 !log tools T221225 run puppet agent in the bastions and reboot them with sssd
[09:49:26] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL
[09:49:27] <stashbot>	 T221225: sssd integration needs to be updated to include sudo config from LDAP support - https://phabricator.wikimedia.org/T221225
[10:10:09] <Eugene233>	 arturo: Are there any such changes like last time ? :)
[10:10:30] <arturo>	 Eugene233: yes, is there anything not working for you?
[10:10:36] <Eugene233>	 I still get "sudo: a password is required" when i type 'become <tool_name>
[10:10:45] <arturo>	 sigh...
[10:10:57] <arturo>	 the problem is that I can't properly reproduce the issue
[10:11:17] <arturo>	 see https://phabricator.wikimedia.org/T221225#5129870
[10:12:09] <arturo>	 what is your tool name Eugene233 ?
[10:12:31] <Eugene233>	 arturo: eugene233
[10:12:52] <arturo>	 the tool name I mean, not your username
[10:14:30] <arturo>	 Eugene233: ?
[10:14:33] <Eugene233>	 arturo: isa
[10:15:14] <arturo>	 ok I can see the issue now
[10:15:16] <arturo>	 https://www.irccloud.com/pastebin/hTqmj4LX/
[10:16:32] <arturo>	 !log tools T221225 disable puppet in tools-sgebastion-08 for sssd testing
[10:16:34] <Eugene233>	 ^^ could this be 'eugene233 is not in the sudoers file'
[10:16:36] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL
[10:16:36] <stashbot>	 T221225: sssd integration needs to be updated to include sudo config from LDAP support - https://phabricator.wikimedia.org/T221225
[10:17:24] <arturo>	 Eugene233: ?
[10:18:15] <Eugene233>	 arturo: Could it be because user 'eugene233' is not in the sudoers file?
[10:18:55] <arturo>	 Eugene233: is likely because sssd is not properly configured to work with sudo in our environment
[10:19:30] <arturo>	 let's do something: I will switch tools-sgebastion-07 back to nscd/nslcd and leave tools-sgebastion-08 with sssd so we can do testing
[10:19:47] <Eugene233>	 arturo: ok nice 
[10:20:04] <Krenair>	 Is it set up to handle the servicegroups OU?
[10:22:00] <Krenair>	 also, um:
[10:22:01] <Krenair>	 $ ssh login.tools.wmflabs.org
[10:22:01] <Krenair>	 krenair@login.tools.wmflabs.org: Permission denied (publickey,hostbased).
[10:22:41] <Krenair>	 that's tools-sgebastion-07
[10:23:08] <arturo>	 yes, I'm switching back to nscd/nslcd and is in an inconsistent state right now
[10:26:24] <Krenair>	 actually forget the servicegroups OU, it should be this object which sssd should be picking up with that config... https://phabricator.wikimedia.org/P8428
[10:27:41] <Krenair>	 interesting that it appeared to pick up your tools.admin sudo abilities but not that
[10:27:50] <arturo>	 !log tools T221225 rebooting tools-sgebastion-07 to clean sssd confiuration
[10:27:54] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL
[10:27:55] <stashbot>	 T221225: sssd integration needs to be updated to include sudo config from LDAP support - https://phabricator.wikimedia.org/T221225
[10:28:02] <arturo>	 !log tools T221225 use `profile::ldap::client::labs::client_stack: classic` in the puppet bastion prefix
[10:28:05] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL
[10:34:08] <Krenair>	 maybe it doesn't like the lack of sudoRunAsUser, maybe it doesn't like sudoUser being a group denoted by a percent sign
[10:34:16] <Krenair>	 sorry
[10:34:21] <Krenair>	 I mean maybe it doesn't like sudoRunAsUser*
[10:42:07] <arturo>	 I'm currently trying to investigate what this means
[10:42:09] <arturo>	 https://www.irccloud.com/pastebin/wP7zISdT/
[10:48:12] <Krenair>	 arturo, interesting, perhaps it fails to find the tools.isa group?
[10:49:40] <Krenair>	 That would explain it allowing you to sudo as root but failing to become the tools user from Eugene233 
[10:49:50] <Krenair>	 I tried to log into tools-sgebastion-08 to take a look but that fails now instead :/
[10:50:11] <Krenair>	 oh it's back
[10:50:42] <Krenair>	 it finds the group for 'getent group tools.isa'...
[10:50:44] <arturo>	 I'm live hacing tools-sgebastion-08
[10:50:50] <arturo>	 live hacking*
[10:51:10] <arturo>	 expect many changes in the sssd config and daemon restarts
[10:51:28] <Krenair>	 ok
[10:55:28] <arturo>	 Krenair: do you know what `ou=servicegroups,dc=wikimedia,dc=org` is supposed to hold?
[10:56:42] <Krenair>	 a servicegroup is basically a tool
[10:57:11] <Krenair>	 it contains groups directly under the OU
[10:57:12] <Krenair>	 e.g.
[10:57:17] <Krenair>	 dn: cn=tools.isa,ou=servicegroups,dc=wikimedia,dc=org
[10:57:26] <Krenair>	 gidNumber: 54010
[10:57:26] <Krenair>	 cn: tools.isa
[10:57:26] <Krenair>	 member: uid=eugene233,ou=people,dc=wikimedia,dc=org
[10:57:27] <Krenair>	 etc.
[10:57:40] <arturo>	 that is probably the missing bit in our sssd config
[10:57:50] <Krenair>	 well I thought that
[10:57:59] <Krenair>	 but then I went on sgebastion-08 and it found the group just fine
[10:58:50] <Krenair>	 there's also ou=people,ou=servicegroups,dc=wikimedia,dc=org which holds the tools users themselves, e.g.:
[10:58:52] <Krenair>	 dn: uid=tools.isa,ou=people,ou=servicegroups,dc=wikimedia,dc=org
[10:59:10] <Krenair>	 homeDirectory: /data/project/isa
[10:59:10] <Krenair>	 loginShell: /bin/bash
[10:59:18] <Krenair>	 uidNumber: 54010
[10:59:23] <Krenair>	 cn: tools.isa
[10:59:25] <Krenair>	 and so on
[11:00:42] <Krenair>	 'Service Groups' is a historical, broader idea than a tool - they could exist in any project but I don't think they get used outside tools anymore
[11:03:34] <arturo>	 anyway this is clearly a misconfiguration
[11:03:51] <Krenair>	 for some reason the sssd config is not world-readable
[11:04:20] <arturo>	 Krenair: for some good reasons actually. sssd consider it to contain sensitive information
[11:04:30] <arturo>	 and will refuse to start if the file is world-readable
[11:05:06] <arturo>	 sssd has many many configuration options and our toolforge environment is not simple. Bad combination perhaps :-P
[11:05:07] * Krenair eyeroll
[11:05:35] <Krenair>	 I guess they assume our anonymous LDAP viewer password will be private?
[11:06:00] <arturo>	 you can configure sssd in many ways, not only with LDAP, but with kerberos, etc
[11:06:09] <arturo>	 you can check the template in puppet
[11:07:39] <Krenair>	 the template looks ok, not that I know anything about sssd
[11:08:24] <Krenair>	 maybe it needs ldap_group_search_base or something set
[11:09:32] <Krenair>	 thing is though I'd expect `getent group tools.isa` to fail if it can't find that group?
[11:17:23] * arturo nods
[11:20:10] <Paucabot>	 hi there
[11:20:28] <Paucabot>	 can somebody help me installing pywikibot in toolforge?
[11:27:50] <arturo>	 Paucabot: try this https://wikitech.wikimedia.org/wiki/Help:Toolforge/Pywikibot
[11:28:52] <Paucabot>	 Thanks Arturo
[11:28:58] <Paucabot>	 I've tried to follow
[11:29:00] <Paucabot>	 https://wikitech.wikimedia.org/wiki/Help:Toolforge/Pywikibot#Using_the_shared_Pywikibot_files_(recommended_setup)
[11:29:07] <Paucabot>	 but I'm stuck
[11:29:25] <Paucabot>	 (I'm a newbie)
[11:30:18] <Paucabot>	 I don't get the same output at step six
[11:33:06] <Paucabot>	 i must be going
[11:33:17] <Paucabot>	 I'll be back later
[11:33:19] <Paucabot>	 thanks
[11:33:28] <arturo>	 Krenair: it turns ou libsss-sudo wasn't installed
[11:33:43] <arturo>	 installing it doesn't solve the issue though
[12:57:37] <arturo>	 !log tools T221225 use `profile::ldap::client::labs::client_stack: sssd` in the puppet bastion prefix, try again with sssd in the bastions, reboot them
[12:57:41] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL
[12:57:41] <stashbot>	 T221225: sssd integration needs to be updated to include sudo config from LDAP support - https://phabricator.wikimedia.org/T221225
[13:06:00] <arturo>	 !log tools T221225 use `profile::ldap::client::labs::client_stack: classic` in the puppet bastion prefix, again. Rollback again.
[13:06:05] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL
[13:06:05] <stashbot>	 T221225: sssd integration needs to be updated to include sudo config from LDAP support - https://phabricator.wikimedia.org/T221225
[14:30:12] <mdholloway>	 !log mobile destroyed android-builder instance
[14:30:14] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Mobile/SAL
[14:32:03] <andrewbogott>	 !log wikifactmine rebooting elasticsearch-01, unreachable
[14:32:04] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Wikifactmine/SAL
[14:34:05] <andrewbogott>	 !log traffic rebooting all elasticsearch nodes as they are unreachable
[14:34:06] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Traffic/SAL
[14:34:43] <andrewbogott>	 !log wikifactmine rebooting all elasticsearch nodes as they are unreachable
[14:34:43] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Wikifactmine/SAL
[14:34:58] <andrewbogott>	 !log traffic ignore last log message — wrong project
[14:34:58] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Traffic/SAL
[14:44:04] <mdholloway>	 !log mobile destroyed edit-counts instance
[14:44:07] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Mobile/SAL
[14:55:27] <andrewbogott>	 !log wikifactmine fixing ferm rules by hand on elasticsearch nodes.  Ferm is installed but seems to not be puppetized :(
[14:55:38] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Wikifactmine/SAL
[14:58:46] <musikanimal>	 hi, I get "sudo: a password is required" when I try to `become` one of my tools. Is this related to the scheduled K8s outage, or otherwise a known issue?
[14:59:10] <Krenair>	 arturo, ^
[14:59:43] <Krenair>	 musikanimal, that is most likely related to the nslcd -> sssd migration that arturo has been working on today
[15:00:05] <musikanimal>	 okay, just making note of it. Thanks!
[15:00:08] <Krenair>	 which is the thing that sits on our instances and makes them look up stuff like users, groups, and sudo rules against LDAP instead of just locally.
[15:00:10] <bd808>	 musikanimal: I think that things are messed up on stretch-dev.tools.wmflabs.org (tools-sgebastion-08). login.tools.wmflabs.org should work as expected
[15:00:24] <Krenair>	 (sudo rules used by `become` to get into your tools are stored in LDAP)
[15:00:39] <musikanimal>	 ah. I was using tools-dev.wmflabs.org
[15:01:30] <musikanimal>	 login.tools.wmflabs.org works indeed, thank you :)
[15:02:11] <bd808>	 T221225 is the issue we are working through on tools-sgebastion-08
[15:02:36] <stashbot>	 T221225: sssd integration needs to be updated to include sudo config from LDAP support - https://phabricator.wikimedia.org/T221225
[15:19:17] <arturo>	 !log tools T221225 creating tools-sgebastion-09 for testing sssd stuff
[15:19:21] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL
[15:19:24] <stashbot>	 T221225: sssd integration needs to be updated to include sudo config from LDAP support - https://phabricator.wikimedia.org/T221225
[15:21:35] <xSavitar>	 !log tools.awmd-stats Restarting to use latest merged and deployed changes
[15:21:36] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.awmd-stats/SAL
[15:26:47] <arturo>	 !log tools T221225 rebooting tools-sgebastion-08 to cleanup sssd
[15:26:57] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL
[15:26:57] <stashbot>	 T221225: sssd integration needs to be updated to include sudo config from LDAP support - https://phabricator.wikimedia.org/T221225