[06:53:18] (03PS2) 10Lokal Profil: Capture task id even if first line [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/465007 [07:19:43] bismilah: If you've got the IP address I can take a look [07:19:49] ccppuu, letsencrypt could save cacert though [13:10:48] (03PS1) 10Urbanecm: Fix W191 - indentation contains tabs [labs/tools/map-of-monuments] - 10https://gerrit.wikimedia.org/r/465028 [13:11:03] (03CR) 10Urbanecm: [C: 032] Fix W191 - indentation contains tabs [labs/tools/map-of-monuments] - 10https://gerrit.wikimedia.org/r/465028 (owner: 10Urbanecm) [13:11:27] (03Merged) 10jenkins-bot: Fix W191 - indentation contains tabs [labs/tools/map-of-monuments] - 10https://gerrit.wikimedia.org/r/465028 (owner: 10Urbanecm) [13:24:32] (03PS1) 10Urbanecm: Fix W191 indentation contains tabs [labs/tools/commons-mass-description] - 10https://gerrit.wikimedia.org/r/465029 [13:24:43] (03CR) 10Urbanecm: [C: 032] Fix W191 indentation contains tabs [labs/tools/commons-mass-description] - 10https://gerrit.wikimedia.org/r/465029 (owner: 10Urbanecm) [13:25:08] (03Merged) 10jenkins-bot: Fix W191 indentation contains tabs [labs/tools/commons-mass-description] - 10https://gerrit.wikimedia.org/r/465029 (owner: 10Urbanecm) [13:36:40] (03PS1) 10Urbanecm: Remove dupes from tox.ini [labs/tools/harvesting-data-refinery] - 10https://gerrit.wikimedia.org/r/465031 [13:36:57] (03CR) 10Urbanecm: [C: 032] Remove dupes from tox.ini [labs/tools/harvesting-data-refinery] - 10https://gerrit.wikimedia.org/r/465031 (owner: 10Urbanecm) [13:37:19] (03Merged) 10jenkins-bot: Remove dupes from tox.ini [labs/tools/harvesting-data-refinery] - 10https://gerrit.wikimedia.org/r/465031 (owner: 10Urbanecm) [13:43:38] (03PS1) 10Urbanecm: Fix W191 indentation contains tabs [labs/tools/weapon-of-mass-description] - 10https://gerrit.wikimedia.org/r/465032 [13:43:51] (03CR) 10Urbanecm: [C: 032] Fix W191 indentation contains tabs [labs/tools/weapon-of-mass-description] - 10https://gerrit.wikimedia.org/r/465032 (owner: 10Urbanecm) [13:44:14] (03Merged) 10jenkins-bot: Fix W191 indentation contains tabs [labs/tools/weapon-of-mass-description] - 10https://gerrit.wikimedia.org/r/465032 (owner: 10Urbanecm) [13:47:39] (03PS1) 10Urbanecm: Fix W191 - indentation contains tabs [labs/tools/wikinity] - 10https://gerrit.wikimedia.org/r/465033 [13:48:12] (03CR) 10Urbanecm: [C: 032] Fix W191 - indentation contains tabs [labs/tools/wikinity] - 10https://gerrit.wikimedia.org/r/465033 (owner: 10Urbanecm) [13:48:35] (03Merged) 10jenkins-bot: Fix W191 - indentation contains tabs [labs/tools/wikinity] - 10https://gerrit.wikimedia.org/r/465033 (owner: 10Urbanecm) [13:53:47] (03PS1) 10Urbanecm: Remove do not fix from tox.ini error [labs/tools/commons-mass-description] - 10https://gerrit.wikimedia.org/r/465034 [13:53:53] (03CR) 10jerkins-bot: [V: 04-1] Remove do not fix from tox.ini error [labs/tools/commons-mass-description] - 10https://gerrit.wikimedia.org/r/465034 (owner: 10Urbanecm) [13:54:00] (03CR) 10Urbanecm: [C: 032] Remove do not fix from tox.ini error [labs/tools/commons-mass-description] - 10https://gerrit.wikimedia.org/r/465034 (owner: 10Urbanecm) [13:54:06] (03CR) 10jerkins-bot: [V: 04-1] Remove do not fix from tox.ini error [labs/tools/commons-mass-description] - 10https://gerrit.wikimedia.org/r/465034 (owner: 10Urbanecm) [13:54:45] (03Abandoned) 10Urbanecm: Remove do not fix from tox.ini error [labs/tools/commons-mass-description] - 10https://gerrit.wikimedia.org/r/465034 (owner: 10Urbanecm) [13:56:25] (03PS1) 10Urbanecm: Add do not fix note to E302 [labs/tools/commons-mass-description] - 10https://gerrit.wikimedia.org/r/465035 [13:56:40] (03CR) 10Urbanecm: [C: 032] Add do not fix note to E302 [labs/tools/commons-mass-description] - 10https://gerrit.wikimedia.org/r/465035 (owner: 10Urbanecm) [13:57:02] (03Merged) 10jenkins-bot: Add do not fix note to E302 [labs/tools/commons-mass-description] - 10https://gerrit.wikimedia.org/r/465035 (owner: 10Urbanecm) [14:12:38] (03PS1) 10Urbanecm: Remove fixed lint violation [labs/tools/wikinity] - 10https://gerrit.wikimedia.org/r/465037 [14:12:54] (03CR) 10Urbanecm: [C: 032] Remove fixed lint violation [labs/tools/wikinity] - 10https://gerrit.wikimedia.org/r/465037 (owner: 10Urbanecm) [14:13:18] (03Merged) 10jenkins-bot: Remove fixed lint violation [labs/tools/wikinity] - 10https://gerrit.wikimedia.org/r/465037 (owner: 10Urbanecm) [14:18:30] (03PS1) 10D3r1ck01: [IMPR][cleanup] Date picker improvements plus cleanup [labs/tools/awmd-stats] - 10https://gerrit.wikimedia.org/r/465038 [14:20:05] (03PS2) 10D3r1ck01: [IMPR][cleanup] Date picker improvements plus cleanup [labs/tools/awmd-stats] - 10https://gerrit.wikimedia.org/r/465038 [15:13:37] (03PS3) 10D3r1ck01: [IMPR][cleanup] Date picker improvements plus cleanup [labs/tools/awmd-stats] - 10https://gerrit.wikimedia.org/r/465038 [15:18:13] (03PS4) 10D3r1ck01: [IMPR][cleanup] Date picker improvements plus cleanup [labs/tools/awmd-stats] - 10https://gerrit.wikimedia.org/r/465038 [15:25:03] (03PS5) 10D3r1ck01: [IMPR][cleanup] Date picker improvements plus cleanup [labs/tools/awmd-stats] - 10https://gerrit.wikimedia.org/r/465038 [15:37:02] (03PS6) 10D3r1ck01: [IMPR][cleanup] Date picker improvements plus cleanup [labs/tools/awmd-stats] - 10https://gerrit.wikimedia.org/r/465038 [15:38:00] (03CR) 10jerkins-bot: [V: 04-1] [IMPR][cleanup] Date picker improvements plus cleanup [labs/tools/awmd-stats] - 10https://gerrit.wikimedia.org/r/465038 (owner: 10D3r1ck01) [15:40:03] (03PS7) 10D3r1ck01: [IMPR][cleanup] Date picker improvements plus cleanup [labs/tools/awmd-stats] - 10https://gerrit.wikimedia.org/r/465038 [15:41:14] (03CR) 10jerkins-bot: [V: 04-1] [IMPR][cleanup] Date picker improvements plus cleanup [labs/tools/awmd-stats] - 10https://gerrit.wikimedia.org/r/465038 (owner: 10D3r1ck01) [15:42:49] (03PS8) 10D3r1ck01: [IMPR][cleanup] Date picker improvements plus cleanup [labs/tools/awmd-stats] - 10https://gerrit.wikimedia.org/r/465038 [16:05:31] (03PS9) 10D3r1ck01: [IMPR][cleanup] Date picker improvements plus cleanup [labs/tools/awmd-stats] - 10https://gerrit.wikimedia.org/r/465038 [16:06:32] (03CR) 10jerkins-bot: [V: 04-1] [IMPR][cleanup] Date picker improvements plus cleanup [labs/tools/awmd-stats] - 10https://gerrit.wikimedia.org/r/465038 (owner: 10D3r1ck01) [16:07:57] (03PS10) 10D3r1ck01: [IMPR][cleanup] Date picker improvements plus cleanup [labs/tools/awmd-stats] - 10https://gerrit.wikimedia.org/r/465038 [16:28:04] (03PS11) 10D3r1ck01: [IMPR][cleanup] Date picker improvements plus cleanup [labs/tools/awmd-stats] - 10https://gerrit.wikimedia.org/r/465038 [16:31:39] (03PS12) 10D3r1ck01: [IMPR][cleanup] Date picker, UI/UX improvements plus cleanup [labs/tools/awmd-stats] - 10https://gerrit.wikimedia.org/r/465038 [16:33:56] (03PS13) 10D3r1ck01: [IMPR][cleanup] Date picker, UI/UX improvements plus cleanup [labs/tools/awmd-stats] - 10https://gerrit.wikimedia.org/r/465038 [16:50:28] (03PS14) 10D3r1ck01: [IMPR][cleanup] Date picker, UI/UX improvements plus cleanup [labs/tools/awmd-stats] - 10https://gerrit.wikimedia.org/r/465038 [16:52:28] (03PS15) 10D3r1ck01: [IMPR][cleanup] Date picker, UI/UX improvements plus cleanup [labs/tools/awmd-stats] - 10https://gerrit.wikimedia.org/r/465038 [18:39:31] I'm getting 'no such tool' errors for https://toolsadmin.wikimedia.org/tools/id/gerrit-avatar-uploader [18:39:44] any troubleshooting steps other than waiting a lot (which I did)? [18:40:10] https://toolsadmin.wikimedia.org/tools/id/gerrit-avatar-uploader loads for me [18:50:09] loads for me too :) [18:58:15] tgr nice tool too! [20:15:37] I mean 'become gerrit-avatar-uploader' fails [20:20:06] tgr: the user exists [20:20:11] but /data/project/gerrit-avatar-uploader does not [20:20:28] that's why become fails [20:26:26] can I do anything about that? [20:58:14] tgr, Platonides: isn't there some bot running on an nfs server that's supposed to set up mysql credentials etc. for tools, and in doing so create their home dirs? [20:59:06] modules/role/manifests/labs/nfs/secondary.pp: include role::labs::db::maintain_dbusers [20:59:33] oh, nope [20:59:37] modules/role/files/labs/db/maintain-dbusers.py has this [20:59:48] # if a homedir for this account does not exist yet, just ignore it [20:59:48] # home directory creation (for tools) is currently handled by maintain-kubeusers, [20:59:48] # and we do not want to race. Tool accounts that get passed over like this will be [20:59:48] # picked up on the next round [21:03:46] yeah you'll have to ask ops tomorrow, sorry tgr [21:04:28] thanks for looking into it [21:38:09] tgr: I'll see what's going on with maintain-kubeusers [21:42:14] etcd on tools-k8s-master-01 is failing a lot. should I be concerned? [21:44:07] !log tools journal on tools-k8s-master-01 is full of etcd failures, did a puppet run, nothing interesting happens [21:44:11] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [21:48:42] !log tools maintain-kubeusers on tools-k8s-master-01 seems to be in an infinite loop of 10 seconds. installed python3-dbg [21:48:45] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [21:57:45] !log tools restarted maintain-kubeusers on tools-k8s-master-01 T194859 [21:57:49] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [21:57:50] T194859: Toolforge maintain-kubeusers stauck in infinite sleeps of 10 seconds - https://phabricator.wikimedia.org/T194859 [22:07:22] !log tools.ldap Restarted, then stopped and started webservice to attempt to fix gateway timeout errors. Failures continue. Will investigate further [22:07:24] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.ldap/SAL [22:09:50] tgr, Platonides: https://phabricator.wikimedia.org/T194859 [22:11:36] "Ah yes. It wasn't running at all" xDD [22:16:13] !log tools.ldap Got webservice to connect to gateway properly with: webservice stop; rm $HOME/service.manifest; webservice --backend=kubernetes python start [22:16:15] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.ldap/SAL [22:21:29] bstorm_: I see. Thanks. [22:22:16] Noisy damned logs [22:22:22] They cloud up the whole mess. [22:22:28] :) [22:22:45] Took me a while to even realize that it leaves logs lol [22:23:12] I queried journalctl and got nothing [22:23:36] sometimes, I forget syslog exists [22:24:07] It doesnt have the python systemd module loaded [22:24:15] So it probably doesn't use journald right [22:24:38] That and this is Jessie, which has kind of dreadful systemd [22:24:44] :-D [22:24:45] I'm pretty sure journal is just too flooded and git rotated [22:24:54] *got [22:25:14] so is etcd failing a concern? [22:25:29] Well there's a journald logger with the python systemd module. I added it to one of our scripts only to find out that it doesn't work well with jessie anyway :-p [22:25:38] ok [22:26:30] Personally, it might be better if this script failed harder when it loses LDAP. I see it connects to LDAP within the infinite loop, but because it pools the connection, it probably doesn't reconnect well after all servers fail? [22:26:51] We might want to change that task to re-open connections when they are all not working. :) [22:27:05] Or maybe to make the service fail harder [22:27:07] makes sense [22:27:11] yeah [22:27:17] etcd? I have no idea :) [22:28:14] it's complaining Oct 07 22:27:41 tools-k8s-master-01 etcd[13778]: open /var/lib/etcd/ssl/certs/cert.pem: no such file or directory [22:28:31] and then systemd restarts it over and over again [22:32:03] Ick [22:32:12] That doesn't sound good [22:33:46] `/var/lib/etcd/ssl/` is empty... [22:36:03] It has been doing that since at least Sept 30th [22:36:17] and considering k8s seems to be fine, makes me wonder if etcd is supposed to be on this host at all [22:38:09] yeah. The very old logs that are saved there don't mention this issue. Maybe andre.w or bry.an know. I can ask them on Tuesday...if this has been the state since the 30th, it doesn't seem to be hurting anyone just now [22:38:24] ok [22:39:09] If you have a moment to make a task for it, that would help me not forget, though (I'll try to remember later if you don't). I need to get back to things here. [22:39:21] Thanks for finding that problem with maintain-kubeusers! [22:40:47] ok, np [23:42:14] thanks for the quick fix! [23:42:39] is it possible to sftp into a tool account? [23:53:00] tgr: are you trying to copy files into tools? [23:53:08] scp should work for you [23:54:33] my IDE does not know scp [23:57:23] Doint you have a bash prompt? [23:57:31] Or what os do you use?