[00:08:20] 10Tool-Labs-tools-stewardbots: Automatically start the irc bots - https://phabricator.wikimedia.org/T144461#2600503 (10Platonides) [00:54:18] (03PS1) 10Mattflaschen: New naming scheme for Edit-Review-Improvements projects [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/307892 [00:58:49] (03CR) 10Catrope: [C: 032] New naming scheme for Edit-Review-Improvements projects [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/307892 (owner: 10Mattflaschen) [00:59:06] (03Merged) 10jenkins-bot: New naming scheme for Edit-Review-Improvements projects [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/307892 (owner: 10Mattflaschen) [01:05:13] !log tools.wikibugs Updated channels.yaml to: 9315b267bc6dea02ee1a10bca37b0bc562d9e514 New naming scheme for Edit-Review-Improvements projects [01:05:20] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.wikibugs/SAL, Master [01:15:24] 06Labs, 10DBA, 06Editing-Analysis, 05Security: Replicate editor_month table from analytics-store to Labs - https://phabricator.wikimedia.org/T143955#2600611 (10Neil_P._Quinn_WMF) >>! In T143955#2599417, @jcrespo wrote: > I think it would be easier if you created a public dump, then created a tool with data... [01:47:02] RECOVERY - Host secgroup-lag-102 is UP: PING OK - Packet loss = 0%, RTA = 0.83 ms [01:55:58] RECOVERY - Host tools-secgroup-test-102 is UP: PING OK - Packet loss = 0%, RTA = 1.28 ms [02:13:42] RECOVERY - Host tools-secgroup-test-103 is UP: PING OK - Packet loss = 0%, RTA = 0.61 ms [02:17:00] PROBLEM - Host secgroup-lag-102 is DOWN: CRITICAL - Host Unreachable (10.68.17.218) [02:18:40] PROBLEM - Host tools-secgroup-test-103 is DOWN: CRITICAL - Host Unreachable (10.68.21.22) [02:23:29] PROBLEM - Host tools-secgroup-test-102 is DOWN: CRITICAL - Host Unreachable (10.68.21.170) [02:50:05] (03PS1) 10Krinkle: Only query for edits and new pages from recent changes [labs/tools/guc] - 10https://gerrit.wikimedia.org/r/307907 [02:51:21] (03PS2) 10Krinkle: Only query for edits and new pages from recent changes [labs/tools/guc] - 10https://gerrit.wikimedia.org/r/307907 [02:52:31] (03PS3) 10Krinkle: Only query for edits and new pages from recent changes [labs/tools/guc] - 10https://gerrit.wikimedia.org/r/307907 [02:57:48] (03PS4) 10Krinkle: Only query for edits and new pages from recent changes [labs/tools/guc] - 10https://gerrit.wikimedia.org/r/307907 [02:58:11] (03CR) 10Krinkle: [C: 032 V: 032] Only query for edits and new pages from recent changes [labs/tools/guc] - 10https://gerrit.wikimedia.org/r/307907 (owner: 10Krinkle) [03:03:08] (03PS1) 10Krinkle: build: Fix phpcs coding style violations [labs/tools/guc] - 10https://gerrit.wikimedia.org/r/307908 [03:04:00] I'm trying to kill a cron job but it won't die; suggestions? [03:07:48] (03PS2) 10Krinkle: build: Fix phpcs coding style violations [labs/tools/guc] - 10https://gerrit.wikimedia.org/r/307908 [03:09:35] (03CR) 10Krinkle: "recheck" [labs/tools/guc] - 10https://gerrit.wikimedia.org/r/307908 (owner: 10Krinkle) [03:09:51] 06Labs, 10DBA, 06Editing-Analysis, 05Security: Replicate editor_month table from analytics-store to Labs - https://phabricator.wikimedia.org/T143955#2600665 (10Bawolff) Well gender being private would kind of defeat the point. The original use case required it to be public [03:10:09] (03CR) 10Krinkle: [C: 032] build: Fix phpcs coding style violations [labs/tools/guc] - 10https://gerrit.wikimedia.org/r/307908 (owner: 10Krinkle) [03:10:27] (03Merged) 10jenkins-bot: build: Fix phpcs coding style violations [labs/tools/guc] - 10https://gerrit.wikimedia.org/r/307908 (owner: 10Krinkle) [03:12:41] enterprisey: I can use sudo powers to kill it for you if its still stuck [03:15:08] bd808: yep, it's still stuck - the job id is 310739, thanks! [03:16:31] enterprisey: /data/project/apersonbot/bot/botreq-status/botreq-status.sh ? [03:16:40] yeah, that's the one [03:17:09] should be dead now [03:17:25] qdel -f $JOB is the magic [03:17:33] very deep magic, I see [03:17:42] but the force often doesn't work for non-root [03:17:48] wonderful! it's dead. thanks again! [03:17:55] no problem [05:06:59] 10Striker: Tool Maintainers badly overcounted - https://phabricator.wikimedia.org/T144064#2600700 (10bd808) And a third number: ``` tools-bastion-02.tools:~ bd808$ ldapsearch -LLL -x cn=project-tools|grep 'member: '|wc -l 1225 ``` I guess the next mystery to solve here is which of `cn=project-tools,ou=groups,dc... [05:19:54] !log tools restart maintain-kubeusers on tools-k8s-master-01, was stuck [05:20:02] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL, Master [05:21:16] bd808 apparently all ldap things are screwed, there is no hope. ^ was 'stuck' for like ~14h before I noticed [05:21:36] I'm just going to try to find a way to make it be restarted with monitoring rather than expect things to work the way they claim they do [05:46:46] 10Striker: Tool Maintainers badly overcounted - https://phabricator.wikimedia.org/T144064#2587654 (10AlexMonk-WMF) >>! In T144064#2600700, @bd808 wrote: > And a third number: > ``` > tools-bastion-02.tools:~ > bd808$ ldapsearch -LLL -x cn=project-tools|grep 'member: '|wc -l > 1225 > ``` > > I guess the next mys... [05:48:23] 06Labs, 10Tool-Labs: Rethink use of floating IPs in tools - https://phabricator.wikimedia.org/T141445#2499007 (10valhallasw) Iirc the main reason for this is to prevent the NAT exit node to become ratelimited or banned on freenode and our own wikis. [05:52:48] 06Labs, 10Tool-Labs: Rethink use of floating IPs in tools - https://phabricator.wikimedia.org/T141445#2499007 (10yuvipanda) yep, this is why we have them only on the exec nodes and not the web nodes. [05:54:49] 10Striker, 06Project-Admins: Allow self-service creation of Maniphest projects for Tools - https://phabricator.wikimedia.org/T144111#2588823 (10Peachey88) > There are some existing Maniphest projects for tools (e.g. `Tool-Labs-tools-Xtools`) that use a different naming convention of `#Tool-Labs-tools- 06Labs, 10DBA, 06Editing-Analysis, 05Security: Replicate editor_month table from analytics-store to Labs - https://phabricator.wikimedia.org/T143955#2600731 (10jcrespo) > Huh, are you saying that that Labs is no longer being developed? That's surprising to me. No, I am saying that it needs urgent maintena... [06:40:44] PROBLEM - Puppet run on tools-proxy-02 is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0] [06:59:48] is irssi supposed to be installed at tools labs? [07:02:01] 06Labs, 10Labs-Infrastructure, 07LDAP: Remove shell user "80686" - https://phabricator.wikimedia.org/T63967#2600758 (10MoritzMuehlenhoff) validnames is a configuration setting of nslcd and configured via a regex in puppet. There's a comment that the regex must be kept in sync with OSM. [07:15:43] RECOVERY - Puppet run on tools-proxy-02 is OK: OK: Less than 1.00% above the threshold [0.0] [07:32:43] 10Tool-Labs-tools-wikiloves: Create http://tools.wmflabs.org/wikiloves/monuments/2016 - https://phabricator.wikimedia.org/T144472#2600816 (10Ata) [07:35:49] PROBLEM - Free space - all mounts on tools-services-01 is CRITICAL: CRITICAL: tools.tools-services-01.diskspace.root.byte_percentfree (<11.11%) [08:39:32] Hello. I need help to understand how to configure `~/.bigbrotherrc`. Thank you. [09:04:42] 06Labs, 13Patch-For-Review: Kill ldapsupportlib.py - https://phabricator.wikimedia.org/T114063#1683588 (10jcrespo) I arrived here by looking at outdated documentation found on wikitech. I found the new one, too. But please before killing it, make sure you redirect on all places to the right command (for casual... [09:13:25] !log tools.stewardbots Restarted SULWatcher via jstart [09:13:29] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.stewardbots/SAL, Master [09:23:03] !log tools.stewardbots Restarted StewardBot, set to run via jstart [09:23:07] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.stewardbots/SAL, Master [09:33:12] 10Tool-Labs-tools-stewardbots: Automatically start the irc bots - https://phabricator.wikimedia.org/T144461#2600979 (10MarcoAurelio) a:03MarcoAurelio This can be done with the bigbrother option for jstart. I've just restarted both bots via jstart (not jsub) so they should be running in continuous mode now. I'l... [09:35:16] 10Tool-Labs-tools-stewardbots: Automatically start the irc bots - https://phabricator.wikimedia.org/T144461#2600981 (10MarcoAurelio) p:05Triage>03Low [09:36:29] 10Tool-Labs-tools-stewardbots: Automatically start the irc bots - https://phabricator.wikimedia.org/T144461#2600503 (10MarcoAurelio) Also, they were not using the trusty release. They're doing it now, however IIRC they intended to phase-out non-trusty instances. I can't find the link to that however. [09:44:23] 06Labs, 10Labs-Infrastructure, 10Tool-Labs: jsub/jstart take 60 s due to /usr/local/bin/log-command-invocation CPU hunger - https://phabricator.wikimedia.org/T131700#2600992 (10Nemo_bis) I'm now near 100 seconds wait, during which the machine had a significant iowait. ``` $ /usr/bin/time -v jsub -once -cont... [09:54:16] (03PS1) 10MarcoAurelio: Updating HTML docs for StewardBot [labs/tools/stewardbots] - 10https://gerrit.wikimedia.org/r/307923 [09:55:18] (03CR) 10MarcoAurelio: [C: 032] Updating HTML docs for StewardBot [labs/tools/stewardbots] - 10https://gerrit.wikimedia.org/r/307923 (owner: 10MarcoAurelio) [09:55:38] (03Merged) 10jenkins-bot: Updating HTML docs for StewardBot [labs/tools/stewardbots] - 10https://gerrit.wikimedia.org/r/307923 (owner: 10MarcoAurelio) [10:03:40] !log tools.stewardbot [[gerrit:307923|Updated HTML docs for StewardBot]] [10:03:41] tools.stewardbot is not a valid project. [10:03:49] !log tools.stewardbots [[gerrit:307923|Updated HTML docs for StewardBot]] [10:03:52] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.stewardbots/SAL, Master [10:34:09] 10Tool-Labs-tools-stewardbots: Automatically start the irc bots - https://phabricator.wikimedia.org/T144461#2601106 (10MarcoAurelio) Okay so I've created a `.bigbrotherrc' file with the following content: ``` jstart -N stewardbot -mem 2G -l release=trusty -j y python /data/project/stewardbots/StewardBot/Steward... [10:36:23] !log tools.stewardbots Created a .bigbrotherrc file to autorestart bots (T144461) [10:36:24] T144461: Automatically start the irc bots - https://phabricator.wikimedia.org/T144461 [10:36:27] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.stewardbots/SAL, Master [10:41:06] 10Tool-Labs-tools-wikiloves: Create http://tools.wmflabs.org/wikiloves/monuments/2016 - https://phabricator.wikimedia.org/T144472#2601118 (10Base) I think Jean-Frédéric has [https://commons.wikimedia.org/w/index.php?title=Module%3AWL_data&action=historysubmit&type=revision&diff=205318158&oldid=205314632 done it] [10:50:42] 06Labs, 13Patch-For-Review: Drop ldapsupportlib.py - https://phabricator.wikimedia.org/T114063#2601138 (10hashar) [10:51:44] Quick question - can meta.wikipedia.org be accessed via API, just like any language version of Wikipedia? And is this https://www.mediawiki.org/wiki/API:Main_page the documentation I should point our developers to? [10:56:15] Spinster: but it's meta.wiki*m*edia [11:05:45] True - I mean meta.wikimedia.org of course [11:49:20] hi there [11:50:18] i get a 502 on my tools webservice [11:50:58] how can i fix that? [11:53:21] freddy2001: webservice release=trusty start ? [11:54:12] sorry, webservice --release=trusty start [11:54:36] try to stop it first [11:54:38] webservice stop [11:54:46] and then webservice --release=trusty start [11:57:03] 06Labs, 10Labs-Infrastructure: Default source group (security group) allowances do not update properly - https://phabricator.wikimedia.org/T142165#2601227 (10Paladox) [11:57:10] still a 502... [12:15:59] freddy2001: does it work now? [12:16:13] Which tool is this? [12:16:36] it is https://tools.wmflabs.org/freddy2001/ [12:21:03] freddy2001: strange, my tools webservice is working [12:21:11] can you try webservice stop ? [12:24:47] mafk, now i have stopped the webservice, the only difference is that webservice status says, that it is stopped... nothing else [12:25:14] it itn't listed on https://tools.wmflabs.org/?status too [12:25:24] freddy2001: can you try now webservice --release=trusty start ? [12:26:39] tools.freddy2001@tools-bastion-03:~$ webservice --release=trusty start [12:26:39] --release is deprecated and has no effect, using trusty... Starting webservice............... [12:27:20] but the tools' page is still "502 Bad Gateway"... [12:27:47] then I have no idea what might be wrong [12:28:04] did you changed something in public_html recently? [12:31:43] no, i didn't [12:36:54] I have no idea then. Maybe you should fill a ticket in Phabricator [12:54:30] 10Tool-Labs-tools-Erwin's-tools: Unknown Error/MySQL errors - https://phabricator.wikimedia.org/T140421#2601290 (10Supernino) It's happening again. [12:58:43] 10Tool-Labs-tools-stewardbots, 06Stewards-and-global-tools: Cleanup of the project stewardbots - https://phabricator.wikimedia.org/T130031#2601303 (10MarcoAurelio) [13:00:56] okay, thank you anyway :) [13:02:47] 10Tool-Labs-tools-stewardbots, 06Stewards-and-global-tools: Cleanup of the project stewardbots - https://phabricator.wikimedia.org/T130031#2601307 (10MarcoAurelio) p:05Triage>03Low [13:30:56] hi, I have a problem with the grid engine. there is a task, which can not deleted/killed. when I use "qdel jobxxx", I got: job 310525 is already in deletion [13:30:57] . [13:31:04] is there any solution? [13:48:52] FNDE: I think there's docs on wikitech on how to deal with stuck jobs [13:49:00] what is the job number? [13:49:06] 310525 [13:49:21] yes, I found it, but I have no ssh-access [13:50:35] 06Labs, 10Gerrit: fix puppet issues when applying role::gerrit::server in labs - https://phabricator.wikimedia.org/T141803#2601400 (10Paladox) Adding labs project since were stuck until it is fixed. [13:51:12] FNDE: you want 310525 forceably remoed? [13:51:14] removed even [13:51:28] yes, exactly [13:51:31] I don't have access to delete the job sorry [13:52:44] chasemp i wonder could you help us with https://phabricator.wikimedia.org/T141803#2600576 please? Per Krenair comment that it dosent allow us to connect remotly to mysql on another instance from gerrit-test3 and alex-test [13:52:52] but does with gerrit-test and jenkins-slave-01 [13:52:56] FNDE: done [13:53:09] great! [13:53:14] thank u! [13:53:19] but, how it works? :) [13:53:40] FNDE: how did I do it? [13:53:45] force delete [13:53:47] qdel -f [13:54:07] paladox: I don't have time at the moment but I plan to ask Krenair about it later [13:54:15] Ok [13:54:16] thanks [13:55:12] okay. didn't worked for me before, but u solved it:) [13:55:33] sometimes if it's really stuck it takes a root [13:57:31] ah ok :) thank U! [14:43:19] 06Labs: Replace debian-8.5-jessie image with debian-8.5-jessie (testing) image - https://phabricator.wikimedia.org/T144439#2601485 (10Andrew) 05Open>03Resolved a:03Andrew [14:48:27] 10PAWS, 10Jupyter-Hub: I can't login my bot in JUPYTER - https://phabricator.wikimedia.org/T135306#2601495 (10Maathavan) @yuvipanda No!!!! [14:49:12] 06Labs, 10Labs-Infrastructure: DB operations in Labs hitting sqlalchemy limits - https://phabricator.wikimedia.org/T144339#2601497 (10chasemp) [14:54:05] 06Labs: New instances carnt connect to remote mysql setup by another instance - https://phabricator.wikimedia.org/T144494#2601528 (10Paladox) [14:54:33] 06Labs: New instances carnt connect to remote mysql setup by another instance - https://phabricator.wikimedia.org/T144494#2601542 (10Paladox) [14:54:36] 06Labs, 10Gerrit: fix puppet issues when applying role::gerrit::server in labs - https://phabricator.wikimedia.org/T141803#2601541 (10Paladox) [14:57:58] 06Labs: New instances carnt connect to remote mysql setup by another instance - https://phabricator.wikimedia.org/T144494#2601545 (10Paladox) Difference is I used the debian-jessie testing image that was created by @Krenair on gerrit-test3 and I used the normal debian-jessie image on gerrit-mysql which looks lik... [14:58:25] chasemp: hi [14:58:29] what would you like to know? [14:58:39] 06Labs: New instances carnt connect to remote mysql setup by another instance - https://phabricator.wikimedia.org/T144494#2601548 (10Paladox) Woops wrong person [14:59:10] caught in the middle of something, give me a few [15:03:08] k [15:03:41] 06Labs, 10Wikimedia-Site-requests, 10wikitech.wikimedia.org: Determine file strategy for wikitech - https://phabricator.wikimedia.org/T144495#2601575 (10Dereckson) [15:03:43] 06Labs, 10Labs-Infrastructure: DB operations in Labs hitting sqlalchemy limits - https://phabricator.wikimedia.org/T144339#2596823 (10chasemp) I looked into both instances and have an explanation as to this round of issues :) As part of T143938 I bumped up the nodepool load /a small bit/ with https://gerrit.w... [15:06:46] chasemp: ahh that sqlalchemy looks like a side effect of nodepool spam isn't it ? [15:07:53] it's a limit on concurrent behavior using the DB, and nodepool is a big offender there yes [15:08:51] I am also wonder how often quota_usages ends up being out of sync [15:09:52] often but not for long and not by much [15:09:57] is the most correct answer I think [15:10:10] fixed_ips, tracked usage: 26 [15:10:24] looks strange since the instance quota is 15 [15:10:58] I'm not entirely sure about what's going on there atm [15:13:01] Krenair: ok I'm tryign to make heads or tails of the mass of updates onT141803. iiuc the current issue of note is that gerrit-test3.git.eqiad.wmflabs cannot get to mysql on gerrit-mysql? [15:13:19] yes [15:18:20] chasemp, T144494 was also created for it [15:18:20] T144494: New instances carnt connect to remote mysql setup by another instance - https://phabricator.wikimedia.org/T144494 [15:35:30] apparently it fails at this project: createprojecttest10 [15:36:10] 10Striker: Tool Maintainers badly overcounted - https://phabricator.wikimedia.org/T144064#2601690 (10bd808) >>! In T144064#2600704, @AlexMonk-WMF wrote: > > I think it's the former, see T138150 `cn=project-tools,ou=groups,dc=wikimedia,dc=org` is the one I had picked at some point during Striker development as... [15:37:50] 06Labs, 06Operations, 06Release-Engineering-Team, 10wikitech.wikimedia.org, 07LDAP: Rename specific account in LDAP, Wikitech and Gerrit - https://phabricator.wikimedia.org/T133968#2601696 (10Sophivorus) Thanks! So @chasemp or @Andrew told you it's not ok to change a user's UID in LDAP? [15:37:51] wonder why novaadmin can't list servers in that project [15:38:35] oh, wrong channel [15:38:39] Krenair: I can just delete the project :) [15:38:49] Probably novaadmin isn't a member [15:38:55] although I don't know why that would've changed yesterday [15:38:56] going back to the right channel [15:39:05] 'k [15:41:11] (03PS1) 10BryanDavis: Count members of tools project, not all posix users [labs/striker] - 10https://gerrit.wikimedia.org/r/307960 (https://phabricator.wikimedia.org/T144064) [15:43:18] 10Striker, 13Patch-For-Review: Tool Maintainers badly overcounted - https://phabricator.wikimedia.org/T144064#2601734 (10AlexMonk-WMF) >>! In T144064#2601690, @bd808 wrote: > I'll be documenting the LDAP layout as maintained by OpenStackManager soon as I start figuring out how account creation works. Great, t... [15:43:20] 06Labs: New instances carnt connect to remote mysql setup by another instance - https://phabricator.wikimedia.org/T144494#2601737 (10chasemp) 05Open>03Resolved a:03chasemp this works now, there was an issue with the instance being stuck in hard reboot state which never completed successfully and the rule... [15:43:37] 06Labs, 10Gerrit: fix puppet issues when applying role::gerrit::server in labs - https://phabricator.wikimedia.org/T141803#2601743 (10chasemp) {T144494} should be fine now [15:47:26] 06Labs: New instances carnt connect to remote mysql setup by another instance - https://phabricator.wikimedia.org/T144494#2601757 (10Paladox) @chasemp thankyou for fixing the problem :) [15:47:37] 06Labs: New instances carnt connect to remote mysql setup by another instance - https://phabricator.wikimedia.org/T144494#2601761 (10Paladox) [15:47:39] 06Labs, 10Gerrit: fix puppet issues when applying role::gerrit::server in labs - https://phabricator.wikimedia.org/T141803#2601760 (10Paladox) 05stalled>03Open [15:58:27] bd808, hey [15:58:46] o/ Krenair [15:59:01] bd808, I may have forgotten some sudo magic or something [15:59:14] but I can't seem to scap deploy from krenair@striker-deploy03:/srv/deployment/striker/deploy$ [15:59:23] 15:58:20 ['/usr/bin/scap', 'deploy-local', '-v', '--repo', 'striker/deploy', '-g', 'default', 'fetch'] on striker-uwsgi02.striker.eqiad.wmflabs returned [255]: Permission denied (publickey,keyboard-interactive). [15:59:47] oh. you need to manually accept the ssh keys as *your* account [15:59:50] it's really annoying [16:00:05] wonder what I ran this as before [16:01:06] bd808, what do you mean accept them as my account? [16:01:51] you need the keys in ~/.ssh/hosts_allowed [16:02:25] so `ssh striker-uwsgi02.striker.eqiad.wmflabs` and accept the host key even though you won't actually auth due to no forwared ssh key [16:02:30] this is host keys? [16:02:36] I already dealt with host keys by doing that [16:02:47] this error is permission denied [16:02:52] oh... maybe the keyholder is unarmed? [16:03:13] there was some puppet patch for that merged I think [16:03:25] Identity added: /etc/keyholder.d/deploy_service (/etc/keyholder.d/deploy_service) [16:03:25] Identity added: /etc/keyholder.d/mwdeploy (/etc/keyholder.d/mwdeploy) [16:03:26] uh huh. [16:03:39] oh, damn irc [16:03:50] /etc/keyholder.d/deploy_service is not an acceptable key. Does it have a passphrase? [16:03:50] /etc/keyholder.d/mwdeploy is not an acceptable key. Does it have a passphrase? [16:03:50] Identity added: /etc/keyholder.d/deploy_service (/etc/keyholder.d/deploy_service) [16:03:51] Identity added: /etc/keyholder.d/mwdeploy (/etc/keyholder.d/mwdeploy) [16:04:01] bd808, it works now [16:04:02] thanks [16:04:08] awesome [16:04:23] next time I'll read the error message more closely :) [16:05:05] (03CR) 10Alex Monk: Count members of tools project, not all posix users (031 comment) [labs/striker] - 10https://gerrit.wikimedia.org/r/307960 (https://phabricator.wikimedia.org/T144064) (owner: 10BryanDavis) [16:31:03] (03CR) 10BryanDavis: Count members of tools project, not all posix users (031 comment) [labs/striker] - 10https://gerrit.wikimedia.org/r/307960 (https://phabricator.wikimedia.org/T144064) (owner: 10BryanDavis) [16:33:52] (03CR) 10Alex Monk: [C: 032] Count members of tools project, not all posix users [labs/striker] - 10https://gerrit.wikimedia.org/r/307960 (https://phabricator.wikimedia.org/T144064) (owner: 10BryanDavis) [16:34:52] (03Merged) 10jenkins-bot: Count members of tools project, not all posix users [labs/striker] - 10https://gerrit.wikimedia.org/r/307960 (https://phabricator.wikimedia.org/T144064) (owner: 10BryanDavis) [16:41:01] 10Striker, 06Project-Admins: Allow self-service creation of Maniphest projects for Tools - https://phabricator.wikimedia.org/T144111#2601969 (10bd808) >>! In T144111#2600712, @Peachey88 wrote: >> There are some existing Maniphest projects for tools (e.g. `Tool-Labs-tools-Xtools`) that use a different naming co... [16:42:55] (03PS1) 10BryanDavis: Bump striker submodule [labs/striker/deploy] - 10https://gerrit.wikimedia.org/r/307974 [16:43:07] (03CR) 10BryanDavis: [C: 032] Bump striker submodule [labs/striker/deploy] - 10https://gerrit.wikimedia.org/r/307974 (owner: 10BryanDavis) [16:43:13] (03Merged) 10jenkins-bot: Bump striker submodule [labs/striker/deploy] - 10https://gerrit.wikimedia.org/r/307974 (owner: 10BryanDavis) [16:45:21] (03PS3) 10Andrew Bogott: Don't attempt to set root user password [labs/private] - 10https://gerrit.wikimedia.org/r/304321 (owner: 10Yuvipanda) [16:48:42] (03CR) 10Andrew Bogott: [C: 032 V: 032] Don't attempt to set root user password [labs/private] - 10https://gerrit.wikimedia.org/r/304321 (owner: 10Yuvipanda) [16:48:54] 10Tool-Labs-tools-Erwin's-tools: Delete excess temporary files in ~erwin85/public_html/tmp - https://phabricator.wikimedia.org/T144506#2601981 (10Nemo_bis) [16:49:05] 10Tool-Labs-tools-Erwin's-tools: Delete excess temporary files in ~erwin85/public_html/tmp - https://phabricator.wikimedia.org/T144506#2601994 (10Nemo_bis) 05Open>03Resolved p:05Triage>03Normal [17:11:08] 10Striker, 13Patch-For-Review, 15User-bd808: Tool Maintainers badly overcounted - https://phabricator.wikimedia.org/T144064#2602052 (10bd808) 05Open>03Resolved a:03bd808 https://toolsadmin.wikimedia.org/ is now reporting 1225 maintainers which matches the output of `ldapsearch -LLL -x cn=project-tools|... [17:29:45] How can I authenticate users of my service with Labs LDAP? [17:29:57] you can't [17:30:07] bd808 but gerrit can [17:30:17] but gerrit is in prod, not labs [17:30:28] bd808 no i tested in labs [17:30:29] too [17:30:37] i had a gerrit-test up an running [17:30:41] and it worked [17:30:42] it is technically possible, but very much against TOU [17:30:49] Oh [17:31:01] You can use puppet and it does that [17:31:08] The LDAP directory is the same in prod and labs [17:31:21] users shouldn't be putting their ldap credentials through labs instances [17:31:22] taking a password that works in prod via Labs is against TOU [17:31:30] (03PS1) 10Yuvipanda: Temp. Hack to get tools up and running [labs/private] - 10https://gerrit.wikimedia.org/r/307989 [17:31:38] chasemp madhuvishy ^ [17:31:46] snooping on traffic in Labs is too easy to allow it [17:31:51] Oh i didnt know that since we setup a replica of gerrit in prod to test things [17:31:55] before there deployed [17:31:59] to prod gerrit [17:32:02] yuvipanda: one question, why use secret for the public key at all? [17:32:05] is it secret? [17:32:13] chasemp madhuvishy I think the right way to fix it is to not use secret for public key at all [17:32:16] hah [17:32:17] jinx etc [17:32:18] :) [17:32:20] tom29739: what are you wanting to do that SUL OAuth won't work for? [17:32:46] In theory we could get wikitech setup as an OAuth provider [17:32:47] 10Tool-Labs-tools-Erwin's-tools: Unknown Error/MySQL errors - https://phabricator.wikimedia.org/T140421#2602177 (10Nemo_bis) I've done `webservice lighttpd-precise start` again, then started some work to switch to `mysqli` but I don't have time to finish (in particular to give `mysqli_real_escape_string()` the c... [17:33:00] (03CR) 10Rush: [C: 031] Temp. Hack to get tools up and running [labs/private] - 10https://gerrit.wikimedia.org/r/307989 (owner: 10Yuvipanda) [17:33:07] chasemp yeah, I'm going to do that after, trying to maintain a more strict fifo so I can finish the previous thing I started. [17:33:17] sure thing [17:33:20] (looking at the wikidata-query project now) [17:33:34] bd808, my service is for labs developers. Figured it might be easier to authenticate them with that, because LDAP is more widely supported as an authentication mechanism. [17:33:40] (03CR) 10Yuvipanda: [C: 032 V: 032] Temp. Hack to get tools up and running [labs/private] - 10https://gerrit.wikimedia.org/r/307989 (owner: 10Yuvipanda) [17:33:46] + I just wondered whether it could be done :) [17:34:05] it takes... 5seconds for the 'submit' button to show up after I hit 'post' [17:34:06] fuck you gerrit [17:34:08] tom29739: *nod* this is part of the reason that the striker project is in prod and not in tool labs itself [17:34:24] yuvipanda lol, it dosent take me that long [17:34:41] just a +2 and i think the button shows [17:34:52] clearly it's my fault for attempting to use it. [17:35:07] (I use a glorified chromebook) [17:35:15] tom29739: so we could either (a) get wikitech setup as a OAuth provider, or (b) make toolsadmin (striker) into an OAuth provider [17:35:26] LOL, /me hates chromebook even though i never used one [17:35:37] bd808: that'd be helpful. [17:35:38] using wikitech should be just some configuration I think [17:36:26] bd808: b would be neat [17:36:40] What you wouldn't get from wikitech is whether or not an authed user was a tools project member [17:36:44] there is another option I dare not speak of so releng doesn't kill me [17:36:52] phab? [17:36:55] yes [17:37:08] *nod* [17:37:17] ....it makes some sense to me [17:37:20] but idk [17:37:45] OAuth 2 would be easier if that can be provided. [17:37:58] not via wikitech for sure [17:38:21] Our security team hates oauth2 [17:38:38] (or hated it before Chris left) [17:39:16] tom29739: I think you should write up a phab task with the requirements you have and we can discuss options [17:39:38] it's a reasonable use case I think [17:39:43] I think it's a reasonable idea but it probably won't be simple [17:39:45] but I'm not sure what teh reasonable outcome is [17:39:48] eheh [17:40:28] turing on the oauth extension for wikitech should be easy, but I'm not sure if it will meet your needs [17:40:54] we don't really expose authz data via our oauth, just authn [17:41:25] authz: authorization; what can be done [17:41:39] authn: authentication; who are you [17:42:16] something like SAML could carry role/group data [17:42:57] there have been a lot of usecases I've had in the past that really could have used authn/authz from prod in Labs [17:43:06] yeah any role or priv based reasoning would be local [17:43:33] this could also long term fall under the wing of the "authentication service" I keep hearing dreams about [17:43:45] yeah, in theory [17:44:01] that's pretty far out there for practical terms I think [17:44:08] but it wouldn't fix the "don't collect passwords in Labs" problem [17:44:09] you could also write a thing that provides a fake LDAP interface based on associations between LDAP and SUL accounts in striker [17:44:23] that's hilarious [17:44:31] I"m not saying it wouldn't work at all though [17:44:38] it's just going to centralize password hash storage/comparison and related things [17:45:01] writing an LDAP server from scratch, how could that possibly cause any problems for us in the long run [17:45:14] actually if you can authn with and get the uid of the account you can lookup authz in ldap yourself [17:45:27] inside Labs you don't need auth to browse the directory [17:45:33] true [17:45:48] but I Guess this is the same as making striker an oauth provider [17:45:55] pretty close [17:46:00] can we just use Microsoft Passport instead? [17:46:10] err, .NET Passport [17:46:14] litterly lol'd [17:46:38] let's write our own .NET passport alt instead [17:46:45] Live login you mean [17:46:47] simplest solution [17:46:51] err, MSN Live Login [17:47:14] firefox persona! [17:47:21] oh wait... dead already [17:47:34] what's that thing, openid or openidentity [17:47:37] federated authn/authz is a tarpit [17:47:44] yeah ipenid [17:47:49] *openid [17:47:53] also mostly dead now [17:48:14] there is a bug somewhere to make wikitech an openid provider [17:48:25] and another for enwiki I think [17:48:33] it was all the rage and then silence [17:48:44] it went open core pretty much [17:48:51] ahhh [17:48:55] the main backers started wanting $$$ [17:49:08] and then *boom* it imploded [17:49:12] all of my instincts said 'this will fail' so I stayed far away [17:49:19] I was kind of active with their project for a couple of years [17:49:31] Microsoft accunt? [17:49:33] account? [17:49:35] PROBLEM - Host tools-static-05 is DOWN: CRITICAL - Host Unreachable (10.68.22.235) [17:49:51] github, google, FB, ... [17:49:59] everyone wants to own the accounts [17:50:07] so they can sell the data [17:50:21] twitter, yahoo, pinterest? [17:50:39] you can login to yahoo with a google account and then login to other things with the yahoo account that was logged into with a google account, I think? [17:50:45] speaking of things yuvipanda and bd808 https://phabricator.wikimedia.org/T144290 [17:50:56] woudl like to understand that a bit more [17:51:07] sure. [17:51:07] it sure seems....possibly bad [17:51:23] the google api account credentials require a fixed list of origin IPs [17:51:42] so we made a proxy that allows things inside Labs to talk to google [17:51:55] from outside it tells you to go away [17:52:37] it's analogous to the http proxy used in prod to talk to the outside world [17:53:11] is it running in labs or tool labs? [17:53:17] Labs [17:53:30] but open to relaying for tools [17:53:45] ok is it moving to tool labs or is that ticket title wrong? [17:54:04] right, probably a grey area maybe but I don't think bad news necessarily [17:54:07] title is wrong [17:54:09] PROBLEM - Host tools-static-06 is DOWN: CRITICAL - Host Unreachable (10.68.22.41) [17:54:17] bd808: ok thanks taht was my main source of interest [17:54:58] I figured w/ https://google-api-proxy.wmflabs.org/ [17:55:10] retitled to "Allow the Labs Google API proxy to handle multiple Google APIs" [17:55:16] thanks man [17:56:39] this proxy came to be because the WMF has an API key that allows some copyvio tools to search google at higher rates that a "normal" person would [17:57:21] gotcha, all good, i was mainly worried about the questions for running it in Tools itself [17:57:38] *nod* [17:58:17] the new usecase is Indic wikisources [17:58:30] I was wondering why it wasn't allowed, because you can lookup stuff in LDAP without authentication. [17:58:38] T120788 [17:58:38] T120788: Tool to use Google OCRs in Indic language Wikisource - https://phabricator.wikimedia.org/T120788 [17:58:56] tom29739: yeah, lookup is fine. It's authentication that isn't [18:00:08] we obviously can't stop you from asking someone for a password with a technical block, but if we find a tool/labs project that is asking for prod passwords we will shut it down as fast as we can [18:00:28] the labs/tools network should never be assumed to be secure [18:00:37] even with ssl/tls etc [18:01:26] I was about to make a nodejs joke but will desist [18:01:43] bd808: so if I used SSH keys or something that's fine? [18:01:51] They're stored in LDAP. [18:01:59] !log wikidata-query cherry pick https://gerrit.wikimedia.org/r/#/c/307656/ [18:02:04] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Wikidata-query/SAL, Master [18:02:11] the public key is, yes. not the private key [18:02:48] I don't think setting up an ssh server as a tool would be a good idea, no. [18:03:08] for a project, you use them by adding people to your project [18:03:26] which then adds them to a group that is allowed to ssh to your project's instances [18:03:51] the only prod auth available in Labs/Tools today is SUL OAuth [18:04:24] that's ok because the user secrets are handled in the WMF production network [18:04:39] and only a signed token is exposed to the consumer app [18:05:41] PROBLEM - Puppet run on tools-static-07 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [18:08:19] bd808: that's kinda a blocker for T143939 [18:08:19] T143939: Request creation of labs-vpn labs project - https://phabricator.wikimedia.org/T143939 [18:09:02] It'd mean to auth with LDAP, the VPN would need to be in the production network. [18:09:22] Or some other authentication method would need to be found [18:09:36] Which wouldn't be too hard. [18:09:44] probably dedicated certs that only work to auth to the VPN [18:09:49] certs/passwords [18:10:08] you can take passwords in Labs, just not prod passwords [18:10:17] That's the standard way to authenticate to VPNs. [18:10:22] Special certs. [18:10:26] *nod* [18:10:37] x509 [18:12:26] tom29739: so one way to go about provisioning x509 certs for a VPN would be to make a MW extension to do it [18:12:38] and get that deployed on wikitech [18:12:56] another way would be to add that to toolsadmin [18:13:25] there are lots of not simple parts for both [18:13:33] bd808, that's what I was thinking [18:13:38] CAs, revocation, etc [18:14:14] If OAuth was added to wikitech, a special interface could be made that allows people to auth with oauth, then add/generate openvpn certs [18:14:20] *VPN cers [18:14:22] *certs [18:14:25] *nod* [18:17:15] 06Labs, 10Gerrit: fix puppet issues when applying role::gerrit::server in labs - https://phabricator.wikimedia.org/T141803#2602247 (10Dzahn) @chasemp @paladox confirmed, working now (yea, Access denied = "works" in this case, before it was all timeout) thank you! ``` dzahn@gerrit-test3:~$ mysql -h gerrit-mys... [18:17:21] It could even generate special .ovpn files with the key in. [18:17:44] So all the user has do to is open that file in their client, and press connect. [18:20:21] 06Labs: New instances carnt connect to remote mysql setup by another instance - https://phabricator.wikimedia.org/T144494#2601528 (10Dzahn) @chasemp thank you [18:26:10] tom29739: phab ticket and/or wiki page :) this needs planning and discussion [18:26:48] There's already a phab ticket of sorts [18:56:05] 06Labs, 10Labs-Infrastructure: Build new Precise base image - https://phabricator.wikimedia.org/T144335#2602425 (10Andrew) 05Open>03Resolved [18:56:39] 06Labs, 10Beta-Cluster-Infrastructure: puppet::self hosts now have two servers set - https://phabricator.wikimedia.org/T144108#2602426 (10Andrew) 05Open>03declined I guess I don't care about this for now. [18:57:16] 06Labs, 10Beta-Cluster-Infrastructure: puppet::self hosts now have two servers set - https://phabricator.wikimedia.org/T144108#2602442 (10yuvipanda) They shouldn't anymore, I fixed 'em all on tuesday. [18:58:03] 06Labs, 06Operations: Enable root passwords on Labs VMs - https://phabricator.wikimedia.org/T142216#2602444 (10Andrew) 05Open>03Resolved This is done now. Passwords are per-project, and located in var/local/labs-root-passwords/ on the labs puppetmaster (currently labcontrol1001). [19:37:19] PROBLEM - Puppet run on tools-docker-builder-01 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [19:46:10] bd808, how hard would it be to set up an LDAP server specially for the VPN thing? That would work, because it'd be inside labs, and not prod credentials. [19:46:35] And then perhaps connect it to Striker. [19:46:54] 10Tool-Labs-tools-Erwin's-tools: Unknown Error/MySQL errors - https://phabricator.wikimedia.org/T140421#2602757 (10Supernino) Thanks nemo, it would be nice if someone finds the time to give you an hand updating the code. :) [19:48:50] Nemo_bis : ping [19:49:10] bd808, or there's something like Duo: https://duo.com/docs/openvpn, but that's outside of the WMF entirely. [19:49:32] (willing to provide some help with preventing erwin's tools from using mysql_* functions ^^ ) [19:49:47] bd808, just seems like a bit of a waste of time building custom interfaces for everything. :/ [19:49:59] Alphos: I can add you to the project if you want [19:50:07] tom29739: what exactly do you need an LDAP server for? [19:50:13] is it core openvpn functionality? [19:50:30] Nemo_bis sure [19:50:59] tom29739: I've seen a lot of asks but no concrete plan of what is being built [19:51:00] bd808, it's one of the easiest ways to authenticate people. It's not core openvpn, it's a plugin [19:51:04] currently running a long query, hopefully won't hit the time limit, got some time to spare, and it wouldn't be the first time i've updated code in that way :) [19:51:32] chasemp hi, how long would it take instances to pick up security group changes [19:51:42] bd808, I requested that project to test out ways of authenticating people to the VPN, etc [19:51:52] since i just added the test security group to one of my instances and it is still not taking any effects [19:51:54] when it should [19:51:57] ? [19:52:14] right, but why do you need a project right off the bat? Can't you research and describe an implementation? [19:52:17] * https://www.toolserver.org/browse/erwin85 | SVN repository /// oh dear :D [19:52:23] paladox: give it 5m or so [19:52:29] Ok [19:52:30] thanks [19:52:58] bd808, I've been researching [19:53:04] chasemp strange it keeps removing my security groups from the instances [19:53:06] * tom29739 brain dumps into a document [19:53:14] by the way, checking : how long is the hard limit for queries ? i seem to remember it's 25 minutes, but not sure [19:53:20] i added test to gerrit-mysql today again and looking now it is removed from it [19:53:39] (granted, lots of my time will be spent in php and I/O, but still) [19:53:41] running a vpn out of Labs is going to be very prickly afa terms of use and security depending on what the intention is [19:54:07] chasemp: the idea is a vpn *into* labs [19:54:16] chasemp, not to outside hosts, just for accessing inside labs hosts [19:54:20] to make accessing the replicas etc easier [19:54:21] sure that's what I mean [19:54:37] there is a difference between into labs, into a labs project, and into tools [19:54:44] it really wouldn't be much different than the bastion project [19:54:56] Yay it worked thanks [19:55:03] "5. Using Wikimedia Labs as a network proxy: Do not use Wikimedia Labs servers or projects to proxy or relay traffic for other servers. Examples of such activities include running Tor nodes, peer-to-peer network services, or VPNs to other networks. In other words, all network connections must originate from or terminate at Wikimedia Labs." [19:55:11] "VPNs to other networks" [19:55:30] So a VPN into labs would terminate at Wikimedia Labs. [19:55:37] Nemo_bis : ping me when added :) [19:55:46] (also, name of the tool would be helpful) [19:55:47] chasemp i think there is a bug in horozon, since yesturday i added the test security group to gerrit-mysql and saw earley this morning it was removed and so i re added it and tonight looking at it it was removed again [19:55:52] in the technically precise definition yeah it's a violation at all but that's mostly becaues probably on one ever asked or seriously thought about it [19:55:53] strange [19:57:05] It might be something to ask Legal about, but it's like the bastion in that it allows people access into the Labs network. [19:57:14] Alphos: the tool is erwin85 [19:57:46] ok [19:57:53] paladox, I messed around with security groups in there [19:57:54] the utility functions are in inc/database,class.php [19:57:59] Oh ok [19:58:01] while we were attempting to debug that issue [19:58:05] Krenair thanks for explaning [19:58:13] Just was wondering [19:58:17] Thanks [19:58:18] tom29739: I think what both chasemp and I are saying however is that what is needed before a project is a straw dog proposal for what would be built, who it would help, and what the risks are [19:58:21] :) [19:58:22] Alphos: but there are a few dozens mysql_* calls in the other files [19:58:32] Nemo_bis darn, need to disconnect from my other session, i can't just now :-( [19:58:41] i promise i will as soon as my script is done though :) [19:59:08] new networking access methods need to be reviewed by quite a few people before we can turn them on [19:59:31] Alphos: There is also an openDb() function which uses mysqli IIRC, but commented out; dunno if it is a past attempt [19:59:38] and a pilot project is a pretty late stage part of those decisions [19:59:53] Nemo_bis for your sake, i strongly recommend PDO [20:00:00] from my perspective it's a bad idea but then again I don't know what hte motivators are really so that's probably premature [20:00:08] if anything, prepared statements are going to be much easier to use [20:00:23] binding with mysqli is nightmarish [20:00:56] chasemp: primary use case is replacing this -- https://wikitech.wikimedia.org/wiki/Help:Tool_Labs/Database#Connecting_to_the_database_replicas_from_your_own_computer -- with something that is actually doable by a normal windows user [20:01:27] Alphos: +1 to PDO and prepared statements [20:01:33] * Alphos hi5 bd808 [20:02:23] right, gotcha [20:02:25] there is a decent helper class for such things at https://github.com/wikimedia/wikimedia-slimapp/blob/master/src/Dao/AbstractDao.php [20:03:27] on the long list of reservations I would have is the first which is time to manage a VPN solution clients and endpoints, my perspective on that is it's very time consuming [20:03:50] even aside from the security concerns and user data concerns etc [20:03:55] but best to make a ticket I think [20:04:06] it's a long tailed discussion [20:04:08] *nod* [20:05:36] Nemo_bis : is the code still available publicly somewhere ? process is taking longer than expected, i'd rather get started now without closing my ssh session :) [20:06:15] Alphos: can't you just open a new ssh session elsewhere? :) [20:06:19] in a new window I mean [20:06:37] tried a few times already. m'afraid it won't let me until i kill the other session [20:06:46] strange, works for me [20:06:53] but I just put the code on http://github.com/erwin85/erwin85 [20:07:09] $ become erwin85 # You are not a member of the group tools.erwin85.\nAny existing member of the tool's group can add you to that. [20:08:18] ah I see what you mean, group assignments aren't applied until logout + login [20:08:31] Yeah I don't know of a solution for that [20:09:30] how often are the requests for new labs projects processed? [20:09:50] once a year ;) [20:13:14] Nikerabbit: weekly :) [20:13:17] usually [20:14:16] sounds good [20:34:56] 06Labs, 10Wikimedia-Labs-General, 10DBA, 06Operations, 07Tracking: Database replication services (tracking) - https://phabricator.wikimedia.org/T50930#2603094 (10Neil_P._Quinn_WMF) [20:34:59] 06Labs, 10DBA, 06Editing-Analysis, 05Security: Replicate editor_month table from analytics-store to Labs - https://phabricator.wikimedia.org/T143955#2603090 (10Neil_P._Quinn_WMF) 05stalled>03declined @jcrespo, thank you for the further explanation; I've [documented some of it on Wikitech](https://wikit... [20:41:07] Nemo_bis : took me a while to get netbeans to behave. which project are you interested in, in particular ? [20:41:12] uh, which tool, i mean [20:45:01] man, the whole thing needs a complete overhaul, RIGHT NOW >_> [20:45:49] singletons, singletons everywhere ! class wrapping around native functions ! trigger_error() as a way to report back to the user ! \o/ [20:54:38] 06Labs, 06Operations, 13Patch-For-Review: Phase out the 'puppet' module with fire, make self hosted puppetmasters use the puppetmaster module - https://phabricator.wikimedia.org/T120159#2603156 (10yuvipanda) wikidata-query done. [21:02:28] PROBLEM - Puppet run on tools-static-10 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [21:42:27] RECOVERY - Puppet run on tools-static-10 is OK: OK: Less than 1.00% above the threshold [0.0] [21:48:26] (03PS4) 10EdouardHue: Importing daily-uploads [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/303933 (https://phabricator.wikimedia.org/T142570) [21:56:19] 06Labs, 10Tool-Labs: AttributeError: 'module' object has no attribute 'python_2_unicode_compatible' - https://phabricator.wikimedia.org/T144554#2603370 (10Nemo_bis) [22:40:46] 06Labs, 10Gerrit: fix puppet issues when applying role::gerrit::server in labs - https://phabricator.wikimedia.org/T141803#2603572 (10Dzahn) We went through the instructions one more time and edited them slightly. It's done now, we have instructions how to get puppet up and running with just puppet that are r... [22:45:15] PROBLEM - Puppet run on tools-static-11 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [22:48:29] 06Labs, 10Gerrit: fix puppet issues when applying role::gerrit::server in labs - https://phabricator.wikimedia.org/T141803#2603582 (10Paladox) This is all fixed now yay. Thanks @Dzahn and @demon Link to the guide on the wiki is https://wikitech.wikimedia.org/wiki/How_to_setup_Gerrit_in_Labs [22:48:50] 06Labs: New instances carnt connect to remote mysql setup by another instance - https://phabricator.wikimedia.org/T144494#2603586 (10Paladox) [22:48:52] 06Labs, 10Gerrit: fix puppet issues when applying role::gerrit::server in labs - https://phabricator.wikimedia.org/T141803#2603583 (10Paladox) 05Open>03Resolved a:03Paladox Closing as resolved now. [22:51:08] Nemo_bis : working on blockfinder, getting along well, but really rewriting from scratch :/ [22:53:26] blockfinder? [22:54:31] https://github.com/erwin85/erwin85/blob/master/blockfinder.php finds blocks pertaining to a given IP address [22:54:45] either global+local, or global+everywhere [22:55:14] RECOVERY - Puppet run on tools-static-11 is OK: OK: Less than 1.00% above the threshold [0.0] [22:55:41] does it also do ranges? [22:56:31] it did, albeit poorly [22:56:36] it will, sanely [22:57:07] wiki\'s → shouldn't that be wikis? [22:57:25] nah, escaping the single quote in a single quoted string [22:57:30] i'd expect procent to be named percent [22:57:38] re https://wikitech.wikimedia.org/wiki/Purge_2016 my project on tool labs is not even listed, should I be concerned? [22:57:38] again, rewriting from scratch ^^' [22:57:44] Alphos: I mean that the text should say wikis instead of wiki's [22:57:48] work first, output last [22:58:04] Platonides oh, it probably should. i don't really care, that's UI ! :p [22:58:11] xD [22:58:25] right now working on actually performing the work before displaying it, AS THINGS SHOULD BE DONE [22:58:34] do your trigger_error generate exceptions? [22:58:40] definitely gonna hack a few things though [22:58:42] Platonides : well duh [22:59:06] throw new \Exception( 'Invalid IP address' ); [22:59:08] through an error handler [22:59:37] tbh haven't handled exceptions yet, that's for last, ain't the hardest, ain't the most rewarding [22:59:38] these are different codes [22:59:38] trigger_error("Please provide a valid IP address.", E_USER_ERROR); [22:59:41] https://github.com/erwin85/erwin85/blob/master/blockfinder.php#L41 [22:59:43] just throwing them like mad :p [22:59:50] yeah, that's gone ;p [23:00:10] getting to a PoC before committing [23:00:15] :) [23:00:32] believe it or not, i'm an experienced php dev ;p [23:00:39] x [23:00:40] xD [23:00:47] * Alphos has encountered these things and screamed ! [23:01:08] i even have a cute little idea for the list of wikis [23:01:26] you'll love it : $wikiList = \json_decode( `cat sitematrix.json | /data/project/erwin85/jq '[([.sitematrix | to_entries | .[] | select( .key != "count") | .value | select(type=="object") | .site[] ][] ),(.sitematrix | to_entries | .[] | select( .key != "count") | .value | select(type=="array") | .[] )]'`, true ); [23:01:45] and if you don't know what jq is, you're missing very dearly out ! [23:02:15] ough [23:02:16] https://stedolan.github.io/jq/ [23:02:27] what's wrong with reading the db list from mysql? [23:02:48] having to do it every time ? [23:03:03] sitematrix doesn't change every hour or even every day, does it ? :) [23:03:33] besides, might give a few people the idea of using jq in their projects ^^ [23:03:39] ok, keep a cache in-file if you want [23:03:59] fwiw javascript has its use cases but it doesn't have to be that unreadable [23:04:12] Gryllida : what's that javascript you're talking about ? [23:04:14] but it's even worse to spawn two programs plus a shell [23:04:30] Alphos: that line you pasted... if it's php, then php doesn't have to be that unreadable :) [23:04:39] cat ain't the most intensive thing there is. neither is jq [23:05:02] Gryllida : the "unreadable" part is jq ;) [23:05:08] (not javascript though :p ) [23:05:11] jq is a language now ? [23:05:12] re https://wikitech.wikimedia.org/wiki/Purge_2016 my project on tool labs is not even listed, should I be concerned? [23:05:17] (or you mean jquery) [23:05:24] i don't mean jquery, i mean jq :) https://stedolan.github.io/jq/ [23:05:29] jq !== jquery [23:05:31] Projects don't exist inside tool labs... 'tools' *is* your project [23:06:00] ah it's already marked as in use so i assume no action is required [23:06:08] thanks for catching that question by the way Krenair [23:07:02] Gryllida if you ever have to handle json from the command line at any point, or to rebase some json with weird data structure such as the sitematrix, i strongly urge you to start using jq ^_^ [23:07:20] i handle json from command line using perl scripts, Alphos :) [23:07:30] Gryllida : so not from the command line :p [23:11:14] PROBLEM - Puppet run on tools-static-11 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [23:13:47] Alphos: thanks for the link [23:14:24] enjoy :) [23:14:44] Gryllida you can thank dereckson later for giving me the idea of writing that puppy up there [23:15:10] jq is your puppy ? [23:15:24] nah, the puppy line up there ^^ [23:15:48] technically he wanted me to find dewiktionary in the sitematrix and i came up with this after a few iterations : jq -c '"dewiktionary" as $dbname | ([.sitematrix | to_entries | .[] | select( .key != "count") | .value | select(type=="object") | .site[] ][] | select (.dbname ==$dbname)),(.sitematrix | to_entries | .[] | select( .key != "count") | .value | select(type=="array") | .[] | select(.dbname==$dbname) )' [23:16:12] ok, i think we could move it to a bash script with shorter lines [23:17:07] not sure about that : it's composed of two separate queries joined into one (see the comma after the first select( .dbname==$dbname ) [23:17:38] so i'm afraid the best you could do is to \ it [23:19:31] !log etcd bring puppetmaster on 'master' up to current setup [23:19:34] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Etcd/SAL, Master