[00:41:13] 10serviceops, 10Operations, 10Phabricator, 10hardware-requests, 10Release-Engineering-Team (Development services): The phabricator server, WMF7426, was given to us temporarily, we would like to make it permanent - https://phabricator.wikimedia.org/T232887 (10Dzahn) a:05faidon→03Dzahn [11:34:31] 10serviceops, 10Wikidata, 10Wikidata-Termbox, 10Release, and 3 others: 1.34.0-wmf.21 cause termbox to emit: Test get rendered termbox returned the unexpected status 500 - https://phabricator.wikimedia.org/T232035 (10DannyS712) [batch] remove patch for review tag from resolved tasks [12:07:16] 10serviceops, 10Operations, 10Patch-For-Review, 10Performance-Team (Radar), and 2 others: Upgrade memcached for Debian Stretch/Buster - https://phabricator.wikimedia.org/T213089 (10elukey) >>! In T213089#5655528, @elukey wrote: > Not all new slab metrics are rendered, opened an issue upstream: https://gith... [15:26:04] 10serviceops, 10Deployments, 10Release-Engineering-Team, 10Performance-Team (Radar): Cache of wmf-config/InitialiseSettings often 1 step behind - https://phabricator.wikimedia.org/T236104 (10Anomie) Happened to me today when deploying a config change for T198312. Touching InitialiseSettings.php and re-sync... [18:20:15] hm, i'm looking into setting up ssl kafka producer for eventgate stuff [18:20:29] pretty easy to do, I just need the Puppet CA crt available on the pods [18:20:48] how can I get it there? put it in my helm chart? [18:21:37] hmm i guess for envoyproxy they are in the private values as text [18:22:11] hmm, config map i guess just like that. [18:22:12] hm [19:02:41] 10serviceops, 10Analytics-Kanban, 10Better Use Of Data, 10Event-Platform, and 5 others: Set up eventgate-logging-external in production - https://phabricator.wikimedia.org/T236386 (10Ottomata) [19:03:01] 10serviceops, 10Analytics-Kanban, 10Better Use Of Data, 10Event-Platform, and 8 others: Set up eventgate-logging-external in production - https://phabricator.wikimedia.org/T236386 (10Ottomata) [19:09:07] ottomata: here is how to create certs for envoy: https://wikitech.wikimedia.org/wiki/Cergen#Cheatsheet [19:09:47] 10serviceops, 10Analytics-Kanban, 10Better Use Of Data, 10Event-Platform, and 8 others: Set up eventgate-logging-external in production - https://phabricator.wikimedia.org/T236386 (10Ottomata) @Joe @akosiaris @ema I'd like to move forward with these patches this week, hopefully sooner rather than later. C... [19:11:19] ottomata: and here an example how then to add envoy on a role and use the cert name in Hiera , similar to https://gerrit.wikimedia.org/r/c/operations/puppet/+/534597 [19:11:37] aye mutante cool [19:11:40] will probably need that for schema.svc [19:11:42] this stuff is in k8s [19:11:57] mostly have that patch ready to go. I was asking about puppet ca for kafka producer [19:12:26] ah. ack! [19:14:27] 10serviceops, 10Analytics-Kanban, 10Better Use Of Data, 10Event-Platform, and 8 others: Set up eventgate-logging-external in production - https://phabricator.wikimedia.org/T236386 (10Ottomata) [19:16:45] 10serviceops, 10Analytics-Kanban, 10Better Use Of Data, 10Event-Platform, and 8 others: Set up eventgate-logging-external in production - https://phabricator.wikimedia.org/T236386 (10Ottomata) [19:17:30] 10serviceops, 10Analytics-Kanban, 10Better Use Of Data, 10Event-Platform, and 8 others: Set up eventgate-logging-external in production - https://phabricator.wikimedia.org/T236386 (10Ottomata) [19:20:45] 10serviceops, 10Analytics-Kanban, 10Better Use Of Data, 10Event-Platform, and 8 others: Set up eventgate-logging-external in production - https://phabricator.wikimedia.org/T236386 (10Ottomata) [19:54:58] <_joe_> ottomata: I'll be around on wednesday, we can work on eventgate-logging-external then [19:55:59] ok thanks _joe_ [19:56:38] we'll need to do the same thing for eventgate-analytics-external too, we aren't really ready to use that, but i'll prep the patches so we can do get all the ssl stuff ready [20:00:10] <_joe_> it makes sense to start ssl-only there too I guess [20:04:24] yah for LVS [20:47:29] 10serviceops, 10Phabricator: Phabricator downtime due to aphlict and websockets (aphlict current disabled) - https://phabricator.wikimedia.org/T238593 (10Dzahn) [22:31:18] 10serviceops, 10Operations, 10Phabricator, 10Release-Engineering-Team-TODO, and 2 others: Reimage both phab1001 and phab2001 to stretch / buster - https://phabricator.wikimedia.org/T190568 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by dzahn on cumin1001.eqiad.wmnet for hosts: ` phab2001.co... [22:38:02] 10serviceops, 10Operations: envoyproxy does not automatically reload certificates - https://phabricator.wikimedia.org/T238597 (10Dzahn) [23:20:38] 10serviceops, 10Operations, 10Phabricator, 10Release-Engineering-Team-TODO, and 2 others: Reimage both phab1001 and phab2001 to stretch / buster - https://phabricator.wikimedia.org/T190568 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['phab2001.codfw.wmnet'] ` and were **ALL** successful.