[13:16:08] I'm hitting a really weird issue which is almost certainly a pebkac but I'm wondering if it's come up before - I've enabled tls in my helm chart, and I see the `similar-users-main-tls-proxy` container being created, but I'm not seeing the configmaps for the certs being created [13:16:34] Given that section is gated on tls.enabled just like the container itself I'm a bit confused as to what I'm missing [13:19:59] hnowlan: in staging? [13:20:54] yeah, but a diff makes it seem like it's going to do similar in codfw [13:23:09] hnowlan: I'll have a look in a fev [13:23:12] thanks! [13:23:25] I'm gonna take lunch so no rush [13:23:30] ack [13:50:10] hnowlan: thats because your're missing an "{{ include "tls.config" . }}" in configmap.yaml [13:51:17] in the similar-users chart ofc [14:12:28] hi, I've just filed https://phabricator.wikimedia.org/T272370 for a logging conflict in linkrecommendation, although I'm not sure which project is most appropriate, any suggestions ? [14:13:04] or should I say, recommendation ? [14:13:08] * godog grabs coat [14:13:13] godog: https://phabricator.wikimedia.org/tag/add-link/ ? [14:13:32] thank you Majavah [14:19:39] jayme: ah, d'oh. Thanks! [14:31:58] hnowlan: np, could have spotted in review as well :) [14:33:46] <_joe_> helm is so much the wrong way to express this stuff [14:47:21] yeah, we should have a fancy mutating webhook that is controlled via annotations :-) [14:48:04] what the kubernetes scene needs is more abstractions [14:48:11] but this time in HCL instead of YAML for some reason [14:54:45] <_joe_> hnowlan: no I seriously considered cdk8s, the stuff aws built [14:54:49] <_joe_> I even did a stub [14:55:16] <_joe_> basically you'd be able to create your new stuff as a subclass of whatever main classes we'd create [14:55:38] <_joe_> I think it'd be a much, much better way to abstract that yaml than templating [14:55:58] yeah, the demos of it I saw looked really nice [15:16:17] <_joe_> like https://github.com/lavagetto/cdk8s-stub/blob/master/blubberoid/main.py :} [15:25:17] https://media4.giphy.com/media/NsERytTcLshXZvOSYB/giphy.gif [15:25:31] config management and migration of it would be kinda complex though I'm guessing [15:25:49] like the helm hierarchical config stuff I mean [15:26:07] it wouldn't even be a huge amount of work to make that testable either I imagine [15:35:48] <_joe_> hnowlan: yeah the testability would be great, and yes we'd have to probably reimplement some stuff [15:35:54] <_joe_> like the hierarchical config [16:24:31] 10serviceops, 10SRE, 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO, 10User-jijiki: Upgrade MediaWiki clusters to Debian Buster (debian 10) - https://phabricator.wikimedia.org/T245757 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by dzahn on cumin1001.eqia... [16:24:45] 10serviceops, 10SRE, 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO, 10User-jijiki: Upgrade MediaWiki clusters to Debian Buster (debian 10) - https://phabricator.wikimedia.org/T245757 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by dzahn on cumin1001.eqia... [16:26:05] 10serviceops, 10SRE, 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO, 10User-jijiki: Upgrade MediaWiki clusters to Debian Buster (debian 10) - https://phabricator.wikimedia.org/T245757 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by dzahn on cumin1001.eqia... [16:26:21] 10serviceops, 10SRE, 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO, 10User-jijiki: Upgrade MediaWiki clusters to Debian Buster (debian 10) - https://phabricator.wikimedia.org/T245757 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by dzahn on cumin1001.eqia... [17:05:26] 10serviceops, 10SRE, 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO, 10User-jijiki: Upgrade MediaWiki clusters to Debian Buster (debian 10) - https://phabricator.wikimedia.org/T245757 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['mw2312.codfw.wmnet'] ` an... [17:06:10] 10serviceops, 10SRE, 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO, 10User-jijiki: Upgrade MediaWiki clusters to Debian Buster (debian 10) - https://phabricator.wikimedia.org/T245757 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['mw2313.codfw.wmnet'] ` an... [17:07:21] 10serviceops, 10SRE, 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO, 10User-jijiki: Upgrade MediaWiki clusters to Debian Buster (debian 10) - https://phabricator.wikimedia.org/T245757 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['mw2315.codfw.wmnet'] ` an... [17:07:51] 10serviceops, 10SRE, 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO, 10User-jijiki: Upgrade MediaWiki clusters to Debian Buster (debian 10) - https://phabricator.wikimedia.org/T245757 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['mw2314.codfw.wmnet'] ` an... [17:22:19] 10serviceops, 10Analytics, 10Analytics-Kanban, 10Patch-For-Review, 10User-jijiki: Mechanism to flag webrequests as "debug" - https://phabricator.wikimedia.org/T263683 (10Milimetric) ping @jijiki on the above question ^. In the meantime we have another request to add client source port to this data, so I... [19:17:02] 10serviceops, 10MW-on-K8s, 10Release Pipeline, 10Release-Engineering-Team (Pipeline), 10Release-Engineering-Team-TODO (2021-01-01 to 2021-03-31 (Q3)): Request volume for Docker images and container filesystems on releases machines - https://phabricator.wikimedia.org/T272092 (10dduvall) >>! In T272092#674... [19:30:01] 10serviceops, 10MW-on-K8s, 10Release Pipeline, 10Release-Engineering-Team (Pipeline), 10Release-Engineering-Team-TODO (2021-01-01 to 2021-03-31 (Q3)): Request volume for Docker images and container filesystems on releases machines - https://phabricator.wikimedia.org/T272092 (10dancy) +1 on the estimate. [20:15:34] 10serviceops, 10Wikibase Release Strategy: Create WMDE Namespace on docker-registry.wikimedia.org - https://phabricator.wikimedia.org/T268984 (10WMDE-leszek) 05Open→03Resolved a:03WMDE-leszek Seems to be resolved on the course of T269777 - thanks @toan! See https://releases.wikimedia.org/wikibase/ [21:18:18] 10serviceops, 10WMF-JobQueue, 10MW-1.36-notes (1.36.0-wmf.28; 2021-01-26), 10Patch-For-Review, and 3 others: Could not enqueue jobs: "Unable to deliver all events: 503: Service Unavailable" - https://phabricator.wikimedia.org/T249745 (10brennen) Noticed a spike of these around 2021-01-19 20:55 UTC: ` Coul... [22:59:31] turns out the docker-registry hosts are still stretch :( can't use python3.7 features [23:19:22] legoktm: I think upgrading them to buster should be a ticket. Tried to check if there is one and first found https://phabricator.wikimedia.org/T261920 now [23:19:55] but that is not this [23:28:49] my hardware is starting to break, now i lost the CursorUp key and it simply stopped working and all seems like a physical issue.. meh.. I am really missing it a lot immediately because of bash history [23:35:55] :| [23:36:05] remap it elsewhere? [23:39:03] yea..or an external keyboard but it's not very convenient. and I _just_ had opened a ticket with ITS for replacing the power adapter [23:40:04] is it old? [23:40:11] which leads to "is covered by the general stipend we got and when it's part of the laptop and not accessories" [23:40:21] ugh [23:40:33] old enough to start to be old but not old enough to replace it :) [23:40:49] I will use the stipend money to buy a new battery [23:42:31] 2017 5th gen X1 afaict [23:42:45] :( [23:45:31] actually looking up replacement policy.. so it is "> 4 years old" AND one of 'warranty expired' OR ' work is hindered by constant repair'. fair enough. it's close but not quite there yet [23:48:28] I'm currently building a new old version of python3-docker-report to import to stretch-wikimedia. trying to build 0.0.4 because that's the last version with python 3.5 support [23:49:17] I'm not sure how it built before because the version of python3-setuptools-scm in stretch-backports is broken, I'll tell the maintainer in a bit [23:49:56] ...or I won't because stretch-backports is closed but sigh