[05:57:00] <vgutierrez>	 arturo: so.. I've been trying to use Kren.air script with the traffic.wmflabs.org. DNS zone and the user I've just created (traffic-cloud-dns-manager), and I'm getting this error, keystoneauth1.exceptions.http.Unauthorized: The request you have made requires authentication. (HTTP 401) (Request-ID: req-c6f30f48-c91b-4681-a263-40df8df101df)
[05:57:10] <vgutierrez>	 of course the script works and the credentials apparently are valid
[05:57:21] <vgutierrez>	 could you check on your side what's going on?
[06:27:36] <vgutierrez>	 arturo: to rule out the custom script, I've been trying with the designate CLI and I'm getting the same error as with the acme chief designate sync client
[06:34:50] <vgutierrez>	 arturo: I'm wondering if the issue could be related with the default MFA requirements
[08:17:42] <arturo>	 o/
[08:18:40] <arturo>	 vgutierrez: I would say the issue may be related to the user dont having enough perms on the project?
[08:19:13] <vgutierrez>	 the user is a projectadmin right now
[08:19:18] <vgutierrez>	 so I doubt is a permissions issue
[08:19:47] <vgutierrez>	 checking the debug output of the designate CLI client it's failing to get the API token
[08:20:15] <vgutierrez>	 DEBUG: http://cloudcontrol1003.wikimedia.org:5000 "POST /v3/auth/tokens HTTP/1.1" 401 114
[08:21:36] <arturo>	 mmm
[08:23:09] <vgutierrez>	 also the first request that fails on the designate script that we are using is listing the DNS zones
[08:23:21] <vgutierrez>	 and via the web interface I can list them using the same user
[08:23:49] <arturo>	 where is the source code?
[08:24:07] <arturo>	 may I take a look?
[08:24:33] <vgutierrez>	 sure
[08:24:34] <vgutierrez>	 https://github.com/wikimedia/puppet/blob/production/modules/acme_chief/files/designate-sync.py
[08:24:51] <vgutierrez>	 let me say again that designate-sync.py is working as expected for deployment-prep
[08:25:28] <vgutierrez>	 for me, it's failing on line 48, https://github.com/wikimedia/puppet/blob/production/modules/acme_chief/files/designate-sync.py#L48 a pure read only operation
[08:27:54] <arturo>	 honestly I dont know what is going on. I could try looking at server side logs
[08:51:21] <vgutierrez>	 yeah.. that's what I've asked a while ago :)
[08:56:01] <arturo>	 there is nothing in the logs apparently
[08:57:19] <vgutierrez>	 so from what I've read in the documentation, the tokens endpoint can return a 401 if the MFA requirements are not satisfied
[08:57:31] <vgutierrez>	 obviously the designate script doesn't use MFA
[08:57:33] <arturo>	 what does MFA means? :-P
[08:57:45] <vgutierrez>	 Multiple Factor Authentication?
[08:57:47] <vgutierrez>	 aka 2FA?
[08:58:02] <arturo>	 also, where is your token stored? I would like to see it
[08:58:24] <vgutierrez>	 as you'd seen on the source code, it uses the horizon user & password
[08:58:51] <arturo>	 I mean, you have a config file with the credentials, right? where is that, in which VM?
[08:59:45] <vgutierrez>	 dunno why that's important, I'm getting the same issue with the designate CLI setting all the credentials/parameters with env variables
[08:59:56] <arturo>	 I see some entries like this in the logs:
[09:00:00] <arturo>	 (keystone.middleware.auth): 2019-08-07 08:57:55,636 WARNING RBAC: Invalid token
[09:00:00] <arturo>	 (keystone.common.wsgi): 2019-08-07 08:57:55,639 WARNING The request you have made requires authentication.
[09:00:19] <arturo>	 (wmfkeystoneauth.password_whitelist): 2019-08-07 09:00:02,466 WARNING Password auth not allowed for traffic-cloud-dns-manager from 172.16.7.184
[09:00:19] <arturo>	 (keystone.common.wsgi): 2019-08-07 09:00:02,467 WARNING Authorization failed. The request you have made requires authentication. from 172.16.7.184
[09:00:34] <vgutierrez>	 yeah.. that's the problem... the "Password auth not allowed"
[09:00:46] <vgutierrez>	 I guess that somebody changed that for deployment-dns-manager :)
[09:02:42] <vgutierrez>	 otherwise the setup in deployment-prep wouldn't work either
[09:03:08] <arturo>	 found it
[09:04:07] <arturo>	 creating a patch
[09:07:37] <arturo>	 https://gerrit.wikimedia.org/r/c/operations/puppet/+/528720
[09:08:02] <vgutierrez>	 nice :)
[09:11:03] <arturo>	 vgutierrez: what is your gerrit username? trying to mark you as reviewer
[09:11:25] <vgutierrez>	 vgutierrez
[09:11:37] <vgutierrez>	 or Vgutierrez
[09:11:53] <vgutierrez>	 err I don't know if I can review that CR
[09:12:08] <arturo>	 sigh, gerrit very very slow
[09:12:14] <vgutierrez>	 the username is ok, but I'm not aware of the keystone subtleties 
[09:18:53] <arturo>	 vgutierrez: try again your script
[09:20:31] <vgutierrez>	 hmmm slightly better :)
[09:20:36] <vgutierrez>	 I need to debug things on my side now
[09:20:37] <vgutierrez>	 thx
[09:20:42] <arturo>	 \o/
[09:22:48] <vgutierrez>	 hmmm
[09:22:56] <vgutierrez>	 something is wrong with the DNS zone apparently
[09:23:15] <vgutierrez>	 I'm seeing some TXT records created there
[09:23:21] <vgutierrez>	 (on the web UI)
[09:23:37] <vgutierrez>	 but if I try to get those via a DNS lookup, it doesn't work
[09:23:58] <vgutierrez>	 hmm now it works..... wow that was pretty slow
[09:24:26] <vgutierrez>	 more than 2 minutes later after the tokens have been injected /o\
[09:58:01] <arturo>	 vgutierrez: I would try adjusting the role of the user
[10:00:53] <arturo>	 I just deleted the projectadmin role and added desginateadmin
[10:01:16] <vgutierrez>	 ack
[10:01:53] <arturo>	 !log traffic remove projectadmin and added designateadmin role to the `traffic-cloud-dns-manager` user T229786
[10:01:56] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Traffic/SAL
[10:01:56] <stashbot>	 T229786: Create a service account to manage traffic.wmflabs.org. from acme-chief - https://phabricator.wikimedia.org/T229786
[14:01:03] <wm-bot>	  Technical Advice IRC meeting starting in 60 minutes in channel #wikimedia-tech, hosts: @Lucas_WMDE & @tgr - all questions welcome, more infos: https://www.mediawiki.org/wiki/Technical_Advice_IRC_Meeting
[14:29:34] <jeh>	 !log remove invalid neutron role assignment T230003
[14:29:35] <stashbot>	 jeh: Unknown project "remove"
[14:29:36] <stashbot>	 T230003: openstack: cleanup neutron user - https://phabricator.wikimedia.org/T230003
[14:29:48] <jeh>	 !log openstack remove invalid neutron role assignment T230003
[14:29:50] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Openstack/SAL
[14:50:58] <wm-bot>	  Technical Advice IRC meeting starting in 10 minutes in channel #wikimedia-tech, hosts: @Lucas_WMDE & @tgr - all questions welcome, more infos: https://www.mediawiki.org/wiki/Technical_Advice_IRC_Meeting
[19:07:20] <bd808>	 !log tools Disassociated SUL and Phabricator accounts from user Lophi (T229713)
[19:07:23] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL
[19:07:23] <stashbot>	 T229713: Transfer linked SUL and Phab accounts from user 'Lophi' to user 'Lofhi' - https://phabricator.wikimedia.org/T229713
[20:04:38] <legoktm>	 !log codesearch switched over to using gerrit-replica
[20:04:41] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Codesearch/SAL
[20:04:51] <paladox>	 \o/
[20:05:05] <paladox>	 https://gerrit-replica.wikimedia.org/r/monitoring?part=graph&graph=httpHitsRate :D
[20:08:14] <mutante>	 paladox: are we still slower ?:o
[20:08:36] <paladox>	 for gerrit2001? Yeh accross the atlantic but in the cloud it's very fast
[20:09:40] <mutante>	 so it's just the distance? alrighty
[20:09:49] <paladox>	 mutante i'm not too sure
[20:09:59] <paladox>	 there's a massive drop compared to cobalt
[20:10:02] <legoktm>	 !log codesearch deleting now unneeded ggmirror-02 instance
[20:10:04] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Codesearch/SAL
[20:15:56] <legoktm>	 paladox: do I need to log in on gerrit-replica to access the monitoring URL? I'm getting forbidden
[20:16:28] <paladox>	 yeh, you need to go to /login/ then you have to type in the replica url (as it redirects you to gerrit.w.org)
[20:18:16] <legoktm>	 got it
[20:18:17] <legoktm>	 woah
[20:18:19] <legoktm>	 that's a giant spike
[20:18:46] <paladox>	 yup
[20:18:59] <legoktm>	 I also switched libup last night
[20:19:20] <paladox>	 great!
[20:22:23] <legoktm>	 https://gerrit.wikimedia.org/r/q/topic:%22gerrit-replica%22+(status:open%20OR%20status:merged)
[20:24:21] <paladox>	 :)
[20:29:00] <legoktm>	 paladox: do you know the answer to https://phabricator.wikimedia.org/T226240#5395373 ?
[20:29:45] <paladox>	 legoktm i think it's updated regularly (all repos are updated on startup)  otherwise it's updated when ever a ref update event is triggered
[20:30:14] <paladox>	 i know gerrit.w.org has a bad replication config atm, which there's a patch to fix it.
[20:30:29] <paladox>	 https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/528769/
[20:31:19] <legoktm>	 I think the question he was asking is what's the expected delay between gerrit and gerrit-replica
[20:43:28] <paladox>	 legoktm ah, i think possibly a few mins delay
[20:47:00] <mutante>	 i am fixing the gerrit replication config now
[20:47:05] <mutante>	 by merging tyler's change finally
[20:47:36] <mutante>	 legoktm: one gerrit restart is needed and it should pick it up again
[20:48:15] <legoktm>	 woot
[20:48:20] <legoktm>	 mutante: I could also use a +2 on https://gerrit.wikimedia.org/r/528919
[20:50:56] <mutante>	 yes, i saw that
[20:51:06] <mutante>	 i am waiting for puppet-merge right now though
[21:27:23] <bd808>	 !log tools.stashbot Restarted bot for config change; added to #wikimedia-cpt (T230074)
[21:27:27] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.stashbot/SAL
[21:27:28] <stashbot>	 T230074: Add stashbot to #wikimedia-cpt - https://phabricator.wikimedia.org/T230074
[21:40:44] <bd808>	 !log tools.stashbot Restarted bot for config change; added to #wikimedia-dev-africa (T223289)
[21:40:48] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.stashbot/SAL
[21:40:48] <stashbot>	 T223289: Add stashbot to #wikimedia-dev-africa - https://phabricator.wikimedia.org/T223289
[21:44:20] <mutante>	 legoktm: merged