[08:23:49] godog: when you have a minute, the patch to add the option to prometheus is ready for review :) https://gerrit.wikimedia.org/r/c/operations/puppet/+/681107 [08:26:56] dcaro: ack! will take a look [08:29:31] dcaro: there's a "reprepro import" on apt1001 for ceph-octopus which holds the lock on current repo actions, but it's from 07:53Z, did that crash or is it maybe waiting for user input? [08:30:20] moritzm: I'm running it, it seems to be working (aptmethod got 'https://download.ceph.com/debian-octopus/pool/main/c/ceph/ceph-common_15.2.11-1~bpo10+1_amd64.deb') after (Getting packages...) [08:30:36] there's another three packages before that last one [08:30:52] ah, ok, so it's just very big then :-) [08:31:19] not sure it should be though xd, is there a way to see progress or similar? [08:31:39] that file is 18Mb, so not huge [08:34:09] I think the only way to assess progress is by checking the filesystem under /srv/wikimedia/pool/thirdparty/ceph-octopus [08:34:35] 👍 [08:34:50] there's 4.3G in there, so maybe it's actually still WIP [08:35:28] ceph-test-dbg alone is 1.3G [08:35:48] yep, that's the one it's currently pulling [08:35:54] (version 15.2.11) [08:36:15] ok, then I'll just be patient :-) [08:36:20] wait no, it's ceph-common-dbg [08:37:02] :/, so there's a few of those big packages [08:37:24] I might want to skip them (in the future), that can be done right? [08:39:33] yeah, we can. the ListShellHook option in modules/aptrepo/files/updates for given component allows that [08:39:54] it's a bit cryptic, but there's existing entries which should provide some examples [08:40:41] ack, I will send a patch, if we get it in faster than it finished I'll just stop and restart :) [08:41:05] btw. if you are waiting, I don't mind stopping right now and letting you go first [08:42:15] no, no, this can wait! I just wasn't sure if it wasn't a case of a stalled import or so [08:42:38] ack [08:45:51] moritzm: sent a patch, let me know if I messed something up :) https://gerrit.wikimedia.org/r/c/operations/puppet/+/681296 [08:50:10] ack, I'll have a look later the day [08:51:11] `No ECDSA host key is known for restricted.bastion.wmcloud.org and you have requested strict checking.` [08:51:20] is ^ a known issue, or a just-me one? [08:52:54] i've re-run `wmf-update-known-hosts-production .` on an updated operations/dns checkout, but no change [08:53:14] kormat: i am also missing the key for that host after running wmf-update-known-hosts-production, will take a look [08:54:05] kormat: for now https://wikitech.wikimedia.org/wiki/Help:SSH_Fingerprints/restricted.bastion.wmcloud.org [08:57:10] jbond42: this is odd. ~/.ssh/known_hosts.d/wmf-cloud hasn't been updated since jan 19th on my machine, but it _does_ contain an entry for the bastion [08:57:16] restricted.bastion.wmflabs.org,185.15.56.14 ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBBENix3RJ8b0bANemSWCTQYK7OwkWik4vKpJ1phA3MOqffx310EsIT3iUvzfZB+bbQ30+oTwe4eT8G09HYdhmY8= [08:57:32] kormat: that's not prod [08:57:41] not part of the hosts production script at all [08:57:53] volans: was just lkooking through the old scripts did you have something to update bastion,restricted? [08:58:15] volans: ohh. right :/ [08:58:26] nope, IIRC there wasn't anything specific about that [08:58:29] ah, and it's also the old name. wmflabs.org vs wmcloud.org [08:58:35] if there is a canonical place where to get that info we coul dadd it though [08:59:15] ahh ok cool as to canonical place im not sure as cloud dosn;t have puppetdb and therefore exported resources [09:02:44] fyi all im about to push a change to purge unmanaged sudo rules. In theory this should be a no-op however its possible old rules, that where never cleaned up are enabling some functionality https://gerrit.wikimedia.org/r/c/operations/puppet/+/681026 [09:03:42] ok, so the answer is to do: `ssh -o StrictHostKeyChecking=ask restricted.bastion.wmcloud.org`, and check the key against the wiki page [09:07:32] kormat: I'd say so, but what changed? did you change the neame from old/new in your ssh config? [09:07:33] kormat: the wmf-sre-laptop config has disables host checking for cloud (arguably ask is a better optin then no here) https://gerrit.wikimedia.org/r/plugins/gitiles/operations/debs/wmf-sre-laptop/+/refs/heads/master/configs/ssh-client-config#80 [09:07:40] I can ssh without issues [09:07:59] volans: i'm just back from a few days of holiday, and it was broken for me. i hadn't had time to change anything yet [09:08:00] jbond42: but the bastion is checked, see line 2 [09:08:15] ack [09:12:27] jbond42: sent a CR updating the wmf-laptop-sre docs (https://gerrit.wikimedia.org/r/681303) [09:14:22] thanks kormat i have merged that, ill see if there are any minor things to fix/add/merge and do a release later in the week [09:14:31] 👍 [09:55:07] XioNoX: o/ is it ok if we deploy https://gerrit.wikimedia.org/r/c/operations/homer/public/+/681059 to unblock Analytics? [09:56:07] for sure [09:56:31] at the barber right now so if you need my +1 you will have to wait a bit :) [09:57:40] XioNoX: nono it's ok thanks :) [15:44:50] XioNoX: any blockers to https://gerrit.wikimedia.org/r/c/operations/homer/public/+/681315 ? I'm looking forward to the work that it will unblock :) [16:10:35] andrewbogott: all good! [16:10:51] thanks! [17:58:50] XioNoX: are the current router interface down alerts on cr1 & cr2-codfw expected? [18:00:30] ah I'm guessing that this is just because the new FPC doesn't have any descriptions on its interfaces [18:02:34] cdanis: it's weird because they don't show up on the cli [18:02:51] nevermind, they do [18:03:11] yep, it's all normal [20:11:42] if you had to set up 1 million static redirects, how would you do it? I'm assuming that's too many to stuff into apache rewrites. Context is https://phabricator.wikimedia.org/T280731 (mailman2 pipermail -> mailman3 hyperkitty) [20:21:51] Maybe https://httpd.apache.org/docs/current/rewrite/rewritemap.html [20:22:07] Can use dbm or shell out to an external program. [20:24:26] ugh [20:25:30] there will be no regular way to transform urls for an email or an attachment once the base url to the mailing list is known? [20:28:07] dpifke: ooh, thanks, I'll give that a shot [20:28:32] apergos: actually it's very straightforward in mm3/hyperkitty, the problem is that the old pipermail URLs are not predictable / easily mappable [20:29:44] grrrrrr I see [20:29:50] it's all keyed on message-id going forward, but pipermail was just an incrementing counter, except for when it wasn't [20:29:59] hahahaha wonderful [20:39:12] I just had a flashback to my ISP days, when we measured Apache startup time in minutes because we had >100,000 virtual hosts and old Sun machines. :) [20:39:55] Although I may be conflating the runtime of the Perl script that produced that config file monstrosity with the time it took Apache to read it. [20:40:10] :-D [20:52:13] https://grafana.com/blog/2021/04/20/grafana-loki-tempo-relicensing-to-agplv3/ - starting to observe a pattern lol [20:52:42] although grafana appears to have been more prudent in taking steps to avoid an incoming amazon fork via a "strategic partnership" with AWS [20:53:32] AGPL is a great license [20:56:25] Anyone know how to change the timeout of a project in Jenkins, and if so who has permissions to do that? Need to increase the timeout of https://integration.wikimedia.org/ci/job/wikidata-query-rdf-maven-release-docker/ but it's not clear where in the UI the configuration lives (or if I just can't see it cause of perms or something) [20:59:54] ryankemper: it's configured in jenkins-job-builder, looks like its set at https://gerrit.wikimedia.org/g/integration/config/+/07b0a1396517af03b0cc7d641d323d3a0f926f4a/jjb/job-templates.yaml#347 [21:00:34] legoktm: perfect, thanks! [21:01:26] someone in #wikimedia-releng can help you deploy it (and if no one bites, I can help too)