[02:13:35] I'm having trouble finding a guide to pushing to diffusion - anyone have a place I could start? [02:14:23] SQL: pushing code? [02:14:31] chicocvenancio: yeah [02:14:49] To what repo are you trying to push? [02:15:09] tool-ipcheck [02:15:37] Let me check that, give me a minute [02:15:42] tyvm [02:16:58] chicocvenancio SQL hi, i think you need arcanist [02:17:20] or you can just push straight into the diffusion repo using git push. [02:17:46] https://www.mediawiki.org/wiki/Phabricator/Arcanist [02:17:58] https://www.mediawiki.org/wiki/Phabricator/Differential [02:18:10] Thanks! [02:18:13] I think I got it now [02:19:06] SQL : great! If you still have problems let me know [02:19:14] Will do, thanks again [02:19:16] paladox: thanks [02:19:25] Your welcome :) [10:18:03] !log tools T181948 deploy misctools 1.27 in the cluster [10:18:09] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [10:18:09] T181948: Toolforge crontab command cannot handle unicode crontab when the locale is non-utf8 - https://phabricator.wikimedia.org/T181948 [10:31:26] !log tools aborrero@tools-clushmaster-01:~$ clush -w @all 'sudo puppet agent -t -v' <--- check again how is the cluster with puppet [10:31:31] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [11:19:25] zhuyifei1999_: could you please take a look at https://phabricator.wikimedia.org/T181948#3916790 ? I would like to know your opinion before I perform more steps [11:21:09] !log tools puppet in the cluster is mostly fine, except for a couple of deprecation warnings, a conn timeout to services-01 and https://phabricator.wikimedia.org/T181948#3916790 [11:21:14] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [13:19:44] hey !help, could a discord bot that searches wikipedia be hosted here? [13:26:08] theFanoulis: is discord open source? [13:26:11] what is discord? [13:26:32] if it is https://en.wikipedia.org/wiki/Discord_(software) than: No. [13:26:34] this https://en.wikipedia.org/wiki/Discord_(software) ?? <-- ni idea actually [13:29:25] the gaming chat service, the links arent working [13:29:34] discordapp.com [13:35:07] theFanoulis: is what you're describing a bot that searches wikipedia for a miscellaneous chats? and if so are those chats related to the wikimedia projects themselves? [13:35:45] misc. chats. Does wikimedia have a discord server? [13:36:55] *not having a wikipedia theme [13:37:14] theFanoulis: not to my knowledge, in general (if I understand) this would not be a valid use case as it would not be in service of the wikimedia projects themselves but rather for consumption of them. which is a much bigger and nearly global use case we can't host. [13:37:29] but opening a task to discuss if that seems an untrue characterization is reasonable [13:37:51] what do you mean by opening a task? [13:38:12] theFanoulis: we have a ticketing system https://phabricator.wikimedia.org/ [13:38:12] phab? [13:38:26] ok, I am opening one [13:38:37] cheers [13:39:01] theFanoulis: many/most of us are traveling this week so it may take a week or so to discuss indepth [13:40:00] I mean, it will be open for discussion, and it is not needed to complete the talk now [13:41:07] I have to leave now, will drop a link to the task soon [13:44:50] we need to know what a "discord bot" is and what exactly it is supposed to do and how that is related to Wikimedia, I'd say. [13:45:12] hmmm that sounds correct [14:13:33] (03CR) 10jenkins-bot: Localisation updates from https://translatewiki.net. [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/405707 (owner: 10L10n-bot) [14:27:17] arturo: what's the command that was sent to crontab? [14:28:49] I can't really figure out why the parameter that was sent to crontab was a `str` not `bytes` [14:53:08] zhuyifei1999_: fwiw I know he's shuffling about packing for travel in between irc pings so it may be a few minutes :) [14:53:49] ok [14:54:00] zhuyifei1999_: not sure, something in puppet? [14:54:24] I mean, that output was from a puppet run in the cluster [14:55:27] bd808: i do not understand why i must follow your format for a quota increase as I already have stated all requested information from the template (T185386) [14:55:27] T185386: Increase quota for dwl - https://phabricator.wikimedia.org/T185386 [15:06:01] arturo: I’ll do more testing. It would be great if it can be reproduced outside of puppet [15:08:54] oh does puppet set any locale environment variables? just in case it matters... [15:17:21] zhuyifei1999_: I'm not sure. Perahps you could try exploring the environment inside the script and then calling it using puppet agent in either bastion-02 or bastion-05 [15:18:29] * zhuyifei1999_ can't run agent [15:19:04] the error happens on all three bastion hosts right? [15:22:45] annika: do you still have doubts about that request? [15:26:04] (from the timestamps it seems you acknowledged the task as unneeded after requesting help here; and bd808 is traveling this week, he might be unavailable for longer periods aa [15:43:42] annika: if "because I asked nicely" isn't a good enough reason, then because the ability to read and follow policies is important in Cloud VPS and Toolforge. [15:59:44] Hi, i carn't seem to access puppet-paladox [15:59:47] its failing with [15:59:55] ssh puppet-paladox [15:59:55] Connection closed by UNKNOWN port 65535 [16:00:05] But i can access gerrit-test [16:13:47] Hmm some how ssh is not allowing me to connect to that host [16:13:55] fails on one of the labs instance too [16:13:59] Connection closed by 10.68.20.181 [16:17:02] andrewbogott hi, i see your on call, im wondering could you try ssh into puppet-paladox please? :) [16:17:54] paladox: works for me, puppet running fine there [16:18:00] Hmm [16:18:34] Failed publickey for paladox from 10.68.17.232 port 44984 ssh2: RSA c6:52:75:38:30:92:c9:36:e0:dc:56:a5:bd:3f:0b:1b [16:18:43] ah thanks. Hmm [16:18:54] i guess i will update my key as it works for my other instances. [16:21:55] andrewbogott hmm openstackmanager is failing to import my new ed25519 key. [16:25:20] paladox: I don't think creating a new problem is likely to help with the old one :) [16:25:29] oh [16:25:30] paladox: use the UI at https://toolsadmin.wikimedia.org/profile/settings/ssh-keys [16:25:36] ah [16:25:55] thanks [16:25:59] I'm not sure we ever got OSM to handle all the key formats [16:33:35] I doin't think it's my key at fault [16:33:39] as it works with gerrit-test [16:37:15] andrewbogott bd808 hmm the problem is only affecting puppet masters [16:37:29] i carn't sign into puppet-phabricator and puppet-paladox [16:37:38] * paladox wonders was there a recent puppet change? [16:38:27] (03PS1) 10Zhuyifei1999: crontab: Make the input file is binary when it is sys.stdin [labs/toollabs] - 10https://gerrit.wikimedia.org/r/405734 (https://phabricator.wikimedia.org/T181948) [16:39:59] paladox: I would try rebooting the instance, and if that doesn't help file a ticket [16:40:05] Ok [16:40:21] I will file a ticket as i rebooted through horizion after i coulden't sign in :) [16:41:21] Hi everyone, I'm hitting paladox's issue as well [16:41:31] andrewbogott ^^ [16:41:55] Connection closed by UNKNOWN port 65535 [16:41:57] hjfocs is that instance a puppetmaster? [16:42:13] * zhuyifei1999_ 's instinct tells me that ^ is an ldap issue on that host [16:42:58] I don't have puppetmasters to play with so can't tell :) [16:43:25] paladox: VPS project = puppet master? [16:43:42] nope [16:43:54] but what i mean is, is that instance a puppetmaster [16:44:02] ie the puppetmaster class applied [16:44:50] checking, just a sec [16:45:42] no [16:46:20] arturo: ^ patch should workaround it. python's own bugs is sometimes annoying and unexpected :( [16:46:48] i am wondering if there is a way to write .out/.err logs of the grid engine to the subdirectory by default O_O [16:48:06] ssh access was working just fine a few hours ago [16:49:22] paladox: thanks for the phab ticket, will paste ssh debug output there [16:50:38] Your welcome :) [16:53:23] ok this is happening everywhere [16:53:49] hm, ok, this is probably something that I broke then. I will look... [16:54:37] andrewbogott: the ticket is https://phabricator.wikimedia.org/T185507 [16:54:54] andrewbogott could it possibly be https://gerrit.wikimedia.org/r/#/c/405671/? [16:54:59] i see that as your last change [16:55:06] and it affects ldap. [16:55:12] probably [16:55:16] paladox: hjfocs are those instances jessie? [16:55:20] Yep [16:55:33] andrewbogott: I think I get teh same behavior from jessie potentially but not trusty? [16:56:01] chasemp: yes [16:56:11] andrewbogott: yeah which seems to mean [16:56:24] 22 16:56:00 tools-k8s-master-01 sshd[9425]: Failed publickey for rush from 10.68.18.66 port 54094 ssh2: RSA d6:15:c3:89:3f:f3:21:00:b4:02:f3:3d:55:d5:cc:fd [16:56:47] Jan 22 16:56:00 tools-k8s-master-01 sshd[9425]: pam_access(sshd:account): access denied for user `rush' from `bastion-restricted-01.bastion.eqiad.wmflabs' [16:56:48] huh [16:56:48] there's some secret global variable that I can't locate but that is affecting everything [16:56:55] this does work tho [16:56:56] tools-k8s-master-01:~# ssh-key-ldap-lookup rush [16:57:04] that or [] resolves to 'true' in puppet [16:57:11] anyway, paladox, is puppet-paladox better now? [16:57:31] andrewbogott yep thanks :) [16:57:32] it works now [16:57:40] ok, that change will roll out over the next few minutes. [16:57:46] zhuyifei1999_: what instance? [16:58:04] andrewbogott: yeah to the best of my knowledge it effects jessie but not trusty atm [16:58:36] chasemp andrewbogott: all three bastions of toolforge [16:58:44] all trusty [16:58:56] zhuyifei1999_: ah, tx [16:59:03] maybe I hit it at the wrong time there [16:59:06] seems so [16:59:09] well that's a bummer [16:59:52] andrewbogott: what prompted https://gerrit.wikimedia.org/r/#/c/405671/ ? [17:00:04] wondering if we are reverting to a previously known issue to fix this [17:00:05] also tested 5 hosts outside of toolforge [17:00:06] tools-bastion-03 should be better, the other bastions will be fixed in a few minutes [17:00:20] chasemp: I reverted the patch, everything will be better shortly [17:00:26] and I need to go to the Ops meeting [17:00:49] andrewbogott: cheers I'll do a mass puppet run for Tools [17:01:08] the tools puppetmaster only merges every 10 minutes so everything is slower there [17:01:13] I'm updating the puppet repo there [17:01:49] thanks a lot folks for quickly digging into this! [17:02:33] andrewbogott wondering could you git fetch origin && git rebase origin on puppet-phabricator please. Some how the agent is connected to itself so it was not as fast as on puppet-paladox. [17:02:35] please [17:03:04] paladox: done [17:03:13] thanks :) [17:03:48] Steinsplitter: if we could find a good syntax for that we could add a flag to jsub to do it. Right now I think you have to do something like `jsub -o dir/dir/file.out -e dir/dir/file.err ...` [17:07:38] bd808: thanks. i use this for a few jobs yet, but it is a pain if there are x-jobs - because i cannot pass $jobname.out etc. [17:08:14] Steinsplitter: *nod* I can see how that would be annoying [17:09:00] It would be possible to add some templating features to the processing of -o/-e by jsub [17:10:06] the code for that lives in https://gerrit.wikimedia.org/r/#/admin/projects/labs/toollabs [17:10:57] https://phabricator.wikimedia.org/source/toollabs/browse/master/jobutils/bin/jsub [17:14:06] zhuyifei1999_: I will have a look as soon as possible :-) thanks for your quick response BTW! It's really appreciated :) [17:14:17] np [17:16:34] (03CR) 10Arturo Borrero Gonzalez: [C: 032] crontab: Make the input file is binary when it is sys.stdin [labs/toollabs] - 10https://gerrit.wikimedia.org/r/405734 (https://phabricator.wikimedia.org/T181948) (owner: 10Zhuyifei1999) [17:17:52] (03Merged) 10jenkins-bot: crontab: Make the input file is binary when it is sys.stdin [labs/toollabs] - 10https://gerrit.wikimedia.org/r/405734 (https://phabricator.wikimedia.org/T181948) (owner: 10Zhuyifei1999) [17:20:22] arturo: if there are still issues please ping me :) [18:32:22] !log tools T181948 T185314 deploying jobutils and misctools v1.28 in the cluster [18:32:28] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [18:32:29] T185314: Toolforge crontab command needs to validate output - https://phabricator.wikimedia.org/T185314 [18:32:29] T181948: Toolforge crontab command cannot handle unicode crontab when the locale is non-utf8 - https://phabricator.wikimedia.org/T181948 [18:43:06] zhuyifei1999_: ^^^ [18:49:41] arturo: k. is puppet fixed now? [18:49:52] * arturo testing [18:51:25] this is tools-bastion-02: [18:51:29] https://www.irccloud.com/pastebin/LY5W4HTp/ [18:51:48] the ssh issue is probably a different thing, an issue I already discussed with chasemp [18:51:57] yeah [18:52:12] but why is user Prometheus on ldap? [18:52:23] shouldn't it be a local user? [18:52:33] that's an error based on a user taking that name before the id range ban was in place [18:52:40] and I think it's just a huge mess to cleanup [18:52:54] name ban that is [18:53:03] that. its a wreck [18:53:22] well, we could whitelist that uid in the crontab command [18:53:43] tools-bastion-05 is similar: [18:53:47] https://www.irccloud.com/pastebin/Ifqe0aEO/ [18:53:48] yeah, its probably going to come to that [18:54:30] that's not an unreasonable short term solution yep [18:55:05] https://phabricator.wikimedia.org/T179386#3724674 -- ideally the check for UID 0 would work [18:55:23] but somehow that is broken I guess [18:55:48] https://phabricator.wikimedia.org/source/toollabs/browse/master/misctools/oge-crontab;ef661cd32d699a7897671ac9780dc8f578c60676$267 [18:56:26] the check for uid 0 is only for the case when you -u [18:56:36] and denies -u if uid is not 0 [18:57:15] service users have a uid < 500 and they are allowed direct crontab access with execv [18:57:34] (that 500 is defined somewhere is puppet) [18:57:40] *is => in [18:58:03] * arturo out [18:59:43] when a root crontab -u I'd expect it to load the remote crontab, and not requiring roots to ssh into cron host to read/write the remote crontab [19:00:40] at least that was how iirc the old perl script worked [19:08:24] zhuyifei1999_: makes sense yes [19:12:04] https://phabricator.wikimedia.org/T45795 <= where that 500 is defined. https://phabricator.wikimedia.org/source/toollabs/browse/master/misctools/oge-crontab;ef661cd32d699a7897671ac9780dc8f578c60676$267 <= where it's used [22:57:29] (03Draft3) 10MacFan4000: Html syntax fixes [labs/tools/ZppixBot] - 10https://gerrit.wikimedia.org/r/405811 [22:57:54] (03CR) 10MacFan4000: [C: 032] Html syntax fixes [labs/tools/ZppixBot] - 10https://gerrit.wikimedia.org/r/405811 (owner: 10MacFan4000) [22:58:02] (03Merged) 10jenkins-bot: Html syntax fixes [labs/tools/ZppixBot] - 10https://gerrit.wikimedia.org/r/405811 (owner: 10MacFan4000) [22:58:10] zppixbot? [22:59:04] irc bot running sopel.