[08:39:18] 10serviceops, 10Operations: Chaos Engineering - Stop for x hours one or more mc10xx memcached shards - https://phabricator.wikimedia.org/T251378 (10Joe) 05Open→03Resolved a:03Joe We ran this test, and it passed with flying colors: - A transient peak of memcached errors, lasting less than 1 minute - The g... [08:39:23] 10serviceops, 10Operations, 10Patch-For-Review: Upgrade and improve our application object caching service (memcached) - https://phabricator.wikimedia.org/T244852 (10Joe) [10:23:32] 10serviceops, 10Operations: Reimage one memcached shard to Buster - https://phabricator.wikimedia.org/T252391 (10elukey) [12:16:36] 10serviceops, 10Release-Engineering-Team: decommission phab1003.eqiad.wmnet - https://phabricator.wikimedia.org/T238957 (10Cmjohnson) [13:01:18] 10serviceops, 10Core Platform Team, 10MediaWiki-General, 10Operations, 10Sustainability (Incident Prevention): Revisit timeouts, concurrency limits in remote HTTP calls from MediaWiki - https://phabricator.wikimedia.org/T245170 (10AMooney) a:03tstarling [13:07:52] paladox: thanks! [13:10:09] bd808: paladox: You got it right regarding what we try to do. Probably also regarding what we need to do. I'm rather new at WMSE, is this something I should request as access, or can I grant it myself? [13:10:29] you can grant your self if your the owner of the repo [13:10:49] I'm owner. [13:12:18] Or... am I? I'm "Karl Wettin (WMSE)" [13:27:58] appears the group is emptied. [14:31:27] 10serviceops, 10Release-Engineering-Team, 10decommission: decommission phab1003.eqiad.wmnet - https://phabricator.wikimedia.org/T238957 (10RobH) [14:42:58] <_joe_> ottomata: around? [14:43:38] <_joe_> I want to connect to kafka-main via TLS; how do I do that? Do I need to generate client certs for that? or the puppet certs are enough? [14:53:05] hiya [14:53:30] for TLS the puppet cert is enough, if you wanted to authenticate you'd need custom client certs [14:53:40] elukey is working on potentially changing that to kerberos [14:53:43] but ya just for TLS [14:54:27] lemme try to find you config example... [14:55:53] _joe_: here are librdkafka configs that eventgate-main uses [14:55:58] https://www.irccloud.com/pastebin/YdWnUAcb/ [14:56:05] /etc/eventgate/kafka_ca.crt.pem [14:56:11] is just the puppet ca cert [14:56:32] <_joe_> ok [14:56:36] <_joe_> perfect, thanks [14:56:40] yw :) [15:50:34] 10serviceops, 10Operations, 10Release Pipeline, 10Release-Engineering-Team-TODO, and 2 others: TEC3:O3:O3.1:Q4 Goal - Move cpjobqueue, Wikidata Termbox SSR (new service), Kask (session storage service) and ORES (partially) through the production CD Pipeline - https://phabricator.wikimedia.org/T220398 (10thc... [15:50:38] 10serviceops, 10ChangeProp, 10Operations, 10Release Pipeline, and 7 others: Migrate cpjobqueue to kubernetes - https://phabricator.wikimedia.org/T220399 (10thcipriani) [15:54:39] 10serviceops, 10Operations, 10Release Pipeline, 10Release-Engineering-Team-TODO, and 2 others: TEC3:O3:O3.1:Q4 Goal - Move cpjobqueue, Wikidata Termbox SSR (new service), Kask (session storage service) and ORES (partially) through the production CD Pipeline - https://phabricator.wikimedia.org/T220398 (10thc... [15:54:47] 10serviceops, 10Operations, 10Release Pipeline, 10Release-Engineering-Team, and 4 others: Introduce kask session storage service to kubernetes - https://phabricator.wikimedia.org/T220401 (10thcipriani) [15:55:01] 10serviceops, 10Operations, 10Release Pipeline, 10Release-Engineering-Team-TODO, and 2 others: TEC3:O3:O3.1:Q4 Goal - Move cpjobqueue, Wikidata Termbox SSR (new service), Kask (session storage service) and ORES (partially) through the production CD Pipeline - https://phabricator.wikimedia.org/T220398 (10thc... [15:57:19] 10serviceops, 10Operations, 10Release Pipeline, 10Release-Engineering-Team-TODO, and 2 others: TEC3:O3:O3.1:Q4 Goal - Move cpjobqueue, Wikidata Termbox SSR (new service), Kask (session storage service) and ORES (partially) through the production CD Pipeline - https://phabricator.wikimedia.org/T220398 (10thc... [15:57:22] 10serviceops, 10Operations: TEC3:Q4 Tracking task - https://phabricator.wikimedia.org/T220403 (10thcipriani) [16:21:29] 10serviceops, 10Operations: TEC3:Q4 Tracking task - https://phabricator.wikimedia.org/T220403 (10thcipriani) [16:28:05] 10serviceops, 10Operations: TEC3:05:05.1:Q4 Services and the deployment pipeline are hosted on production-level infrastructure - https://phabricator.wikimedia.org/T220405 (10thcipriani) [16:28:08] 10serviceops, 10Operations: TEC3:Q4 Tracking task - https://phabricator.wikimedia.org/T220403 (10thcipriani) [16:28:17] 10serviceops, 10Operations: Services and the deployment pipeline are hosted on production-level infrastructure - https://phabricator.wikimedia.org/T220405 (10thcipriani) [16:36:57] 10serviceops, 10Operations: TEC3:Q4 Tracking task - https://phabricator.wikimedia.org/T220403 (10thcipriani) 05Open→03Invalid Untangling task trees for completed quarters: separated open subtasks, closed completed subtasks. [16:50:28] 10serviceops, 10Kubernetes: Make helm upgrades atomic - https://phabricator.wikimedia.org/T252428 (10JMeybohm) [17:53:16] 10serviceops, 10Release-Engineering-Team-TODO, 10Release-Engineering-Team (Deployment services): upgrade MediaWiki appservers to Debian 10 (buster) - https://phabricator.wikimedia.org/T245757 (10Jdforrester-WMF) [20:12:19] 10serviceops, 10DC-Ops, 10Operations, 10ops-eqiad: scb1001: Memory correctable errors -EDAC- - https://phabricator.wikimedia.org/T250482 (10Cmjohnson) 05Stalled→03Declined Since we are not replacing, just setup a decom task when ready [20:35:34] 10serviceops, 10Operations, 10ops-eqiad: mw1280 correctable memory errors logged in getsel - https://phabricator.wikimedia.org/T251077 (10wiki_willy) Hi @elukey or @Dzahn - just wanted to follow up on this, to see if it's worth buying parts to keep this server online, especially with all the previous issues...