[00:15:40] !log planet moving renamed Hiera values to Prefix puppet for planet-* after https://gerrit.wikimedia.org/r/#/c/397729 - fixing puppet run on planet-hotdog [00:15:42] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Planet/SAL [00:17:40] a reboot of phabricator server is imminent [13:13:24] !log tools detected low space in tools-worker-1020, big files in /var/log due to kubelet issue. Opened T184604 [13:13:28] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [13:13:29] T184604: tools: out of disk: tools-worker-1020 - https://phabricator.wikimedia.org/T184604 [13:18:21] !log tools aborrero@tools-k8s-master-01:~$ sudo kubectl cordon tools-worker-1020.tools.eqiad.wmflabs for T184604 [13:18:26] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [13:20:34] !log tools aborrero@tools-worker-1020:~$ sudo service kubelet restart [13:20:39] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [13:22:52] !log tools empty by hand syslog and daemon.log files. They are so big that logrotate won't handle them [13:22:57] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [13:26:39] !log tools sudo kubectl drain tools-worker-1020.tools.eqiad.wmflabs [13:26:43] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [13:45:12] !log tools T184604 aborrero@tools-worker-1020:/var/log$ sudo mkdir /var/lib/kubelet/pods/bcb36fe1-7d3d-11e7-9b1a-fa163edef48a/volumes [13:45:18] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [13:45:18] T184604: tools: out of disk: tools-worker-1020 - https://phabricator.wikimedia.org/T184604 [13:46:25] !log tools T184604 aborrero@tools-k8s-master-01:~$ sudo kubectl uncordon tools-worker-1020.tools.eqiad.wmflabs [13:46:30] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [13:57:33] !log tolls T184604 cleaned stalled log files that prevented logrotate from working. Triggered a couple of logrorate runs by hand in tools-worker-1020.tools.eqiad.wmflabs [13:57:34] arturo: Unknown project "tolls" [13:57:34] T184604: tools: out of disk: tools-worker-1020 - https://phabricator.wikimedia.org/T184604 [13:57:44] !log tools T184604 cleaned stalled log files that prevented logrotate from working. Triggered a couple of logrorate runs by hand in tools-worker-1020.tools.eqiad.wmflabs [13:57:50] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [14:01:44] !log tools tools-k8s-master-01:~# kubectl uncordon tools-worker-1001.tools.eqiad.wmflabs [14:01:49] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [14:41:41] !log tools tools-clushmaster-01:~$ clush -w @k8s-worker "sudo puppet agent --disable 'chase rollout'" [14:41:45] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [15:02:57] Technical Advice IRC meeting starting in 60 minutes in channel #wikimedia-tech, hosts: @addshore & @CFisch_WMDE - all questions welcome, more infos: https://www.mediawiki.org/wiki/Technical_Advice_IRC_Meeting [15:03:18] !log tools tools-k8s-master-01:~# for n in `kubectl get nodes | awk '{print $1}' | grep -v -e tools-worker-1001 -e tools-worker-1016 -e tools-worker-1016`; do kubectl cordon $n; done [15:03:23] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [15:14:33] !log tools tools-clushmaster-01:~$ clush -f 1 -w @k8s-worker "sudo puppet agent --enable && sudo puppet agent --test" [15:14:37] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [16:01:25] Technical Advice IRC meeting starting now in channel #wikimedia-tech, hosts: @addshore & @Christoph_Jauera_(WMDE) - all questions welcome, more infos: https://www.mediawiki.org/wiki/Technical_Advice_IRC_Meeting [17:10:12] petan: please look at https://phabricator.wikimedia.org/T184613 [17:10:40] oh and Matthew_ thanks for the pointer [17:10:56] zhuyifei1999_: No problem :) [17:12:53] If there's no response within the next day or so I'll reach out other ways to try to get it fixed. I don't have access to the XMLRc information so... [17:13:04] *XmlRcs [17:17:59] Matthew_: maybe addshore? https://tools.wmflabs.org/openstack-browser/project/huggle [17:18:36] Maybe. [17:23:01] * Matthew_ gently pokes addshore wondering if they can fix the XmlRcs instance for Huggle [17:24:11] wahahaa, im super buried right now [17:25:55] That's fine. [17:25:58] I understand. [19:02:46] Hi! Can someone help me with ssh -i id_rsa user@login.tools.wmflabs.org? Suddenly it throws me the error "Connection to login.tools.wmflabs.org closed by remote host. Connection to login.tools.wmflabs.org closed." [19:03:18] anna___: Are you a member of "Shell user" group on Wikitech? [19:03:26] No [19:03:33] Check your preferences [19:03:59] If you don't are not in that group then maybe bd808 should add you on-wiki. Or did you file a request? [19:04:58] I filed a request a month ago and got approved and succeeded to connect, but now it doesn't work [19:05:37] Hmm.... Then that should be something else but did you check preferences? [19:05:48] And you are not part of the "Shell user" group? [19:06:00] I checked and I'm not [19:06:37] anna___: Okay, I think you need to be added [19:07:04] How do I do that? [19:07:55] Someone with rights need to add you to the group [19:08:11] Maybe someone here will see our chat and then add you? What's your Wikitech username? [19:08:23] Anna.d [19:08:28] anna___: can you link to your toolforge membership request? [19:08:57] https://toolsadmin.wikimedia.org/tools/membership/status/193 [19:13:11] anna___: Are you literally using "user@" in your ssh command? [19:13:20] you should be using annad92@ [19:14:50] Yes, I'm using mu username. Now on annad92 I get Permission denied (publickey,hostbased). [19:16:47] anna___: in the logs on the server I'm seeing "Failed publickey for annad92" which means that the ssh key that you are using is not the ssh key that the server expects [19:18:54] The key is listed both in wikitech and toolforge.. [19:20:39] anna___: the key you are presenting has the fingerprint "SHA256:VSttyWeJSyftipxX4nMlZwPaAu8sYAireXARPjGeVv0" [19:21:07] does that match what you see at https://toolsadmin.wikimedia.org/profile/settings/ssh-keys [19:23:19] Yes [19:23:57] Does it has something to do with the fact that I'm Anna.d on toolforge? [19:24:57] your shell account name and your wiki account name are related, but I am seeing the correct "annad92" name in the ssh auth failure log [19:31:01] Why does it says "Wikimedia account "Anna.d92" is already attached to another LDAP account." I dont' have another LDAP account. [19:32:14] anna___: per https://toolsadmin.wikimedia.org/tools/membership/status/193 -- the SUL account Anna.d92 is linked to the LDAP/Wikitech account Anna.d92 [19:32:26] you do not need to make a new LDAP/Wikitech account [19:32:49] But my LDAP account is Anna.d [19:34:17] anna___: ah ha! I think we may be getting somewhere here. [19:34:28] you have 2 LDAP accounts [19:35:45] This is weird because the last time I tried to ssh connect it worked, I did nothing new, tried again today, and it stopped working [19:35:51] Anna.d92/annad92 which is a toolforge member. Anna.d/anna-d which is not [19:36:46] are you maybe using an ssh key that you setup for the "anna-d" account as the "annad92" user? [19:37:05] On wikitech user Anna.d92 does not exist [19:37:20] https://wikitech.wikimedia.org/wiki/User_talk:Anna.d92 [19:37:40] the account is not attached on wikitech, but it exists in the LDAP directory [19:38:02] this means the account was created using https://toolsadmin.wikimedia.org/ and has never logged into Wikitech [19:39:46] How do I reset password on LDAP? [19:41:24] If the account has never been attached on Wikitech, right now we do not have a way to reset the password unfortunately [19:42:11] So can I file a new request? with Anna.d [19:42:34] If you make a phab task I can attach it and then you can reset the password via the normal login screen [19:42:55] What to write on the task? [19:43:24] anna___: something like "please attach Anna.d92 on wikitech so I can get a password reset" [19:44:18] OK, thank you [19:47:06] Done [19:48:06] https://phabricator.wikimedia.org/T184644 [19:48:23] T184644 [19:48:23] T184644: Attaching LDAP account on Wikitech - https://phabricator.wikimedia.org/T184644 [22:08:02] Nemo_bis, Nikerabbit, is the 'ttmserver' project long since defunct? [22:08:23] I'm repairing some things there but… it doesn't look like anyone else has been here in a while [22:23:31] i was wondering what to do if my tokens no longer allow me to log in in wikitech [22:23:37] anyone here has nay idea? [22:23:41] *any [22:23:47] using two factor [22:24:28] nuria_: when you say 'my tokens' you mean the codes from your authenticator app? [22:26:25] andrewbogott: yes [22:26:36] nuria_: if so, sometimes that's caused by clock drift on your phone and can be resolved by restarting your phone or (maybe) turning airplane mode off and on [22:26:51] I can't think of what else would break it, other than an account mixup [22:26:59] andrewbogott: i had to use one of the emergency tokens this one time and from that moment on they ones from the phone no longer work [22:27:05] oh, huh [22:27:12] that's not supposed to happen, but I guess... [22:27:20] it could right? [22:27:37] do you have other emergency tokens? It might be worth you just disabling 2fa and then re-enabling and resynching with your phone. [22:28:25] i can try that, is disabling an option anymore? [22:29:57] nuria_: Horizon requires 2fa. So if you disable and leave it off you'll be limited to other tools. Basically read-only things. [22:30:12] andrewbogott: ok, let me try [22:30:16] andrewbogott: thank you [22:30:39] nuria_: I can also disable 2fa on my end but that will require some id-confirmation rigamarole [22:31:35] andrewbogott: ok, done, let me retry again to set it up [22:33:05] nuria_: forget 2fa stuff? Dont worry ive done.. once.. okay maybe more :P I use authy its pretty cross platform and even on mobile and chrome extension ive been fine since just giving you some advice :) [22:33:54] Zppix: I don't think nuria forget anything, it's some kind of bug probably on the server side [22:34:10] Ah i musta misread :P [22:37:31] andrewbogott: reset it all up, hopefully it works or i will just be able to log in wikitech ONE LAST TIME [22:38:06] nuria_: well your old emergency codes are reset if you reset things… so you should have a new set [22:38:18] for FIVE LAST TIMES (or however many) [22:40:12] nuria_: that sounds like it is worth filing a bug against our OATH extension too. Using an emergency token should not change anything that would prevent using the original htop seed generated values later [22:40:45] we might be able to nerd snipe Reedy into looking into reproducing that ;) [22:41:12] bd808: Do you mean, lock me in a room next week? :P [22:41:29] I'll sneak you pastrami and beer Reedy [22:41:44] oh... well yes I will have physical coercive techniques at my disposal [22:42:02] he lives on yogurt and pineapple while at my house [22:42:16] bd808: i think the issue was my authenticator , i changed phones and it upgraded (i used emergency token in between) and something went bad, [22:42:17] pineapple, the only fruit that eats you [22:42:37] bd808: makes Reedy sound like he is 3 years old but hey, WHATEVER [22:42:53] The truth hurts [22:43:02] How did you re-setup it on the new phone? [22:43:08] Did you have the seed saved? [22:43:11] I pictured three air snaps at that 'WHATEVER' [22:44:26] Reedy: let me think how did i do it , part of my xmas was spent in a blurry updates on various family members iphones [22:44:39] Ok np [22:44:47] Reedy: wouldn't a backup of your iphone save your seed? it must, right? [22:44:59] iOS? [22:45:06] yes [22:46:08] Reedy: is that how it works? seed is local to your device and server correct? [22:46:43] yeah. that's what it uses as a common part (along with the current time) when generating/confirming the codes [22:46:57] bd808: if you really want reedy to suffer you could force him to work on T165795 [22:46:58] T165795: Ldap auth extension vs. ldap vs. username Case - https://phabricator.wikimedia.org/T165795 [22:47:02] https://github.com/google/google-authenticator/issues/193#issuecomment-354603453 [22:47:16] Reedy: i did a backup on my iphone, clean my new one and restore backup [22:47:18] https://github.com/google/google-authenticator/issues/193#issuecomment-353806856 [22:48:21] andrewbogott: let's just get LDAP off of wikitech ;) [22:49:06] Reedy: mmm i see not a seamless process but i backup to local [22:49:28] Reedy: and restore from local [22:49:47] Reedy: but still something broke there yeah [22:50:43] It'd be interesting to see if it's replicable for sure