[01:53:01] Any PetScan experts whom can help me with a query? I'm trying to convert a Pagepile full of Wikidata IDs to their parralel articles on the English Wikipedia. The relevant page pile is 32498. Is it possible to use Petscan for this purpose? [01:59:14] Amir1: just filed a few codesearch tickets for you to look over :) [09:12:04] legoktm: sure, thanks! [16:37:31] Hi, i'm trying to ssh into one of my instance but it's failing. [16:37:46] it seems i can ssh into bastion but using Proxy/ProxyJump fails. [16:37:53] i get "patrickmulhall@bastion.wmcloud.org: Permission denied (publickey)." [16:38:00] oh! [16:38:25] paladox: wrong user name? [16:38:30] hmm, still fails: [16:38:30] paladox@bastion.wmcloud.org: Permission denied (publickey). [16:39:30] paladox: run it with -vvv and paste maybe? [16:40:19] mutante https://phabricator.wikimedia.org/P13337 [16:41:41] paladox: any changes in your config recently? looking [16:42:08] My config looks like: [16:42:12] Host *.wmflabs *.wikimedia.cloud [16:42:12] IdentityFile /Users/patrickmulhall/.ssh/id_ed25519_3 [16:42:13] ProxyCommand ssh -a -W %h:%p paladox@bastion.wmcloud.org [16:42:13] User paladox [16:43:06] so first of all, I am on that same bastion now and your user exists [16:43:27] the error looks like it's just the wrong key being sent [16:43:42] yeh, i can ssh into the bastion if i used: ssh -i /Users/patrickmulhall/.ssh/id_ed25519_3 paladox@bastion.wmcloud.org [16:44:15] do you normally use "ssh-add" to load keys into agent in the background? [16:44:34] nope [16:44:35] or an agent in your OS GUI? [16:44:42] nope [16:44:48] adding -vvv to ssh will also show you the config lines it matches (might help debug) [16:45:32] (note that it's very verbose 😉) [16:45:34] paladox: out of curiosity. wanna try if "ssh-add /Users/patrickmulhall/.ssh/id_ed25519_3" works for you? [16:45:38] debug1: /Users/patrickmulhall/.ssh/config line 5: Applying options for *.wikimedia.cloud [16:45:43] sure [16:46:00] oh [16:46:02] that worked [16:46:08] it should ask you for passphrase and then confirm "Identity added" [16:46:09] though i have IdentityFile in the config [16:46:13] and then just ssh again [16:46:14] shouldn't it have used that? [16:47:06] paladox: I have IdentityFile line as well but I still load my key with ssh-add to be able to SSH without being asked for passphrase every single time [16:47:13] ah [16:47:15] ok [16:47:16] not saying you need to remove it though [16:48:16] mutante thanks! [16:48:22] you can add this to the top of the ssh config file to add it automatically (AddKeysToAgent yes) [16:49:29] paladox: keeping the IdentityFile lines makes sure it uses the right key on the first try instead of trying all that are currently loaded in agent.. in some cases there can be so many keys that it reaches the max attempts before it gets to try the right one [18:10:28] is there a standard or easy way in a cloud vps project to set up services that use mTLS among a handful of instances? i can certainly do it by hand but i'm wondering if there are any puppet profiles that help automate the generation of a ca, server cert/key, and client cert/key [18:12:30] seems i could probably just reuse the puppet ca and host key/cert for the server side but it doesn't seem like i'd be able to get a client key/cert from the puppet ca without setting up my own puppetmaster for the project [18:13:01] marxarelli: nothing cloud vps specific. there may be some puppet profile that does the things you want, but I don't know what it would be [18:14:06] marxarelli: are you trying to POC something for a prod deployment later, or would this be a thing that stays in a VPS project indefinitely? [18:14:52] it's just for experimenting with buildkitd. nothing prod ready or targeting prod with any definitive timeline [18:15:56] i have 4 buildkitd servers running and i want to expose the daemons over tcp so i can run buildctl against them, but it needs mTLS to function safely [18:16:36] i don't expect them to run for more than a few weeks at the most [18:16:59] and they will be idle 99.9% of the time :) [18:17:43] so, yeah, POC is the short answer [18:18:59] ok, and what you really need is certs to do the tls with? Or do you need a proxy that does the TLS termination too? [18:19:52] it's for mutual auth, not just for transport encryption, so i need a pair of certs/keys for the server(s) and a pair for the client [18:20:49] so a self-signed ca should suffice, which i can set up no prob. i was just curious if there was something in puppet already for automating that within a cloud project [18:21:41] I think from the things I'm seeing in puppet that the puppet ca is the thing most often used for sort of similar things [18:23:05] without a puppet db instance I think the potentially tricky part of that is sharing the public keys around to the cluster memebers [18:23:52] marxarelli: you might ask the folks in #wikimedia-sre if there is magic that they know of hidding in ops/puppet for this [18:23:57] right. that makes sense. that would give me a ca cert and cert/key pair for the server, but it's not clear to me how i'd get another cert for the client signed by the same ca, unless i set up my own puppetmaster [18:24:34] cool. thank you! i'm probably making this too complicated :) [18:25:00] each puppet client makes is on cert pair and then gets its public cert signed by the puppet ca. there is stuff in ops/puppet for adding extra aliases to that cert [18:25:57] in prod the client public certs are collected by puppet db which lets you then query them back on other nodes, but we don't have a puppet db for the shared cloud vps puppetmaster because pupet does not really have multi-tenant isolation [18:26:23] *makes it's own (typing is hard) [18:29:07] haha no worries. i am fluent in typo. thanks for the explanation, bd808 [21:06:56] I'm trying a query to commonswiki and I'm not sure why it never resolves, https://quarry.wmflabs.org/query/49944 , the same query on enwiki works https://quarry.wmflabs.org/query/49947 . What dumb thing am I doing? Are there so many images added to commons in one second that it is impossible to count them in time? [21:24:54] joakino: I think its more likely that the 1 second your query looked at had no data at the time you queried. Any replag would keep what your wrote from working. [21:46:15] there is no integer x where x < n and x > n-1 [21:48:35] bd808, I think joakino's question was more about why it takes a query on commonswiki_p 1114 seconds to figure that out, while the same query against enwiki_p takes 1.2 seconds [21:49:44] I would guess that "INNER JOIN page ON page_title = img_name AND page_namespace = 6" is the big difference. [21:49:57] lots and lots more File: pages on commons [21:52:16] the query plan is ok, but the page index traversal hits 42,641,018 rows [21:57:31] https://quarry.wmflabs.org/query/49953 [21:58:31] considering that there is (supposed to be) an index on img_timestamp, I'd expect that to be much quicker, even with 65mil files [22:03:26] adding just the join brings it from 261s to 837s https://quarry.wmflabs.org/query/49950 [23:33:24] AntiComposite: part of the issue is the size of the index and the free ram on the sql server. The current wiki replicas just do not have enough ram to keep all the "hot" indexes in memory. Scanning the index from disk makes things slower. And contention with everything else happening in the same instance--including replication--makes that worse. [23:33:54] that'll do it