[02:09:19] !log clouddb-services clouddb1002 is now the replica for clouddb1001/toolsdb and is catching up slowly [02:09:23] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Clouddb-services/SAL [08:49:51] chicocvenancio: can you remind me which task? [10:19:28] pintoch: https://phabricator.wikimedia.org/T217238 but I've commented there [10:20:11] thanks a lot! [18:55:14] !log tools start building jobutils 1.36 T217297 [18:55:18] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [18:55:18] T217297: Manual page of jsub has two defaults, but only one can be default - https://phabricator.wikimedia.org/T217297 [19:08:04] !log tools test failures during build, see ticket [19:08:06] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [19:36:09] !log tool built with debuild instead T217297 [19:36:09] zhuyifei1999_: Unknown project "tool" [19:36:09] zhuyifei1999_: Did you mean to say "tools.tool" instead? [19:36:10] T217297: Manual page of jsub has two defaults, but only one can be default - https://phabricator.wikimedia.org/T217297 [19:36:16] !log tools built with debuild instead T217297 [19:36:18] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [23:06:49] what does 'Rr' mean on a qstat job ? [23:06:55] r from running I guess [23:09:11] R is restarted [23:09:59] thank you! [23:10:17] I guess my "bigbrother" file restarted the job then [23:25:00] ebernhardson: Are you around? I don't have good ideas re: your login issues but I'd like to watch the logs while you try again. (But also I have no idea what TZ you're in) [23:29:18] andrewbogott: yea i'm here, i'm roughly based in SF [23:29:29] cool, let me get some things set up here... [23:29:39] then I will ask you to try to log in again and again without success :( [23:30:31] yea :) FWIW this all started happening after i reset 2FA on wikitech, but dunno if that helps ... basically i could log into horizon but not wikitech, then i reset the 2fa on wikitech (manually, in the database via wikitech instructions), and then i can only log into wikitech and not horizon [23:31:23] huh [23:33:32] bd808: if looking for random 2fa rejections, which central log would you look in on mwlog1001? (There are a lot!) [23:34:02] andrewbogott: CentralAuth probably [23:34:09] ebernhardson: can you try now? Hopefully I can get some context at least. [23:34:18] ebernhardson: does that include usernames in the log messages? It's a very noisy log [23:34:47] andrewbogott: done [23:35:46] andrewbogott: not sure, i just grepped the extension we use to see what logging channels it uses [23:37:54] ebernhardson: try again with mixed case, EBernhardson? [23:38:42] andrewbogott: done [23:39:06] poking around to see if i can find the 2fa implementation in mediawiki... [23:39:22] hm, ok — I see you failing in the Horizon code but no mention of anything hitting MW [23:40:22] let's see if I can turn up some log levels... [23:43:44] andrewbogott: found the logs, its in `authentication.log` coming from the OATHAuth extension. And it does log usernames [23:45:22] ok, now I have many more windows open and many more logging channels enabled. Try again with EBernhardson? [23:45:52] andrewbogott: done [23:47:20] andrewbogott: theory: When i log into wikitech it says 'Ebernhardson' in authentication.log. When i login via horizon and provide 'Ebernhardson' the failure log says 'EBernhardson' [23:47:42] that's… interesting [23:48:04] I see "OATHAuth user EBernhardson failed OTP/scratch token" in the auth log [23:48:09] which suggests that it knows who you are at least [23:48:17] hm… let me check something in the db [23:50:31] yeah, ok, I think I found it [23:51:59] here is the hole you have fallen into: https://phabricator.wikimedia.org/T165795 [23:52:34] oh, fun! [23:52:53] so i had two separate 2fa tokens, which is why the wierdness [23:52:58] s/had/have/ [23:53:20] one account has a token and one doesn't [23:53:39] For the moment, I propose to murder one of your accounts, and then you have to be compulsive about how you CamelCase your login name to wikitech [23:53:43] which do you want to keep? [23:54:07] (I'll probably have you reset 2fa in either case) [23:54:08] keep Ebernhardson, mediawiki will auto upper-case the first letter [23:54:13] oh gawd. that bug sucks :/ [23:54:14] then i can type without capitals [23:54:19] bd808: yes it does! [23:54:21] ebernhardson: ok :) [23:54:57] andrewbogott: obviously I should have fixed that 2 years ago when it popped up :/ [23:55:44] I still think Brad should do it :) [23:57:09] ebernhardson: ok, now if you log in on wikitech I think you'll find that 2fa is disabled. You can re-enable it and re-sync with your phone and then… maybe it'll be fixed? [23:57:41] yup, no 2fa. Adding one now [23:59:26] Hey, when did #wikimedia-labs become invite only? [23:59:35] andrewbogott: added 2fa to wikitech, now logging into horizon fails and no failure message in authentication.log :S [23:59:52] !help I cannot connect to login-stretch [23:59:52] TParis: If you don't get a response in 15-30 minutes, please create a phabricator task -- https://phabricator.wikimedia.org/maniphest/task/edit/form/1/?projects=wmcs-team [23:59:59] TParis: I think it sends you a redirect message saying to come here instead