[06:52:28] 10serviceops, 10Analytics, 10ChangeProp, 10Community-Tech, and 6 others: Provide the ability to have time-delayed or time-offset jobs in the job queue - https://phabricator.wikimedia.org/T218812 (10Joe) @Mooeypoo thanks for clarifying requirements of the current project better, it's much clearer now. @aez... [11:28:45] 10serviceops, 10Analytics, 10ChangeProp, 10Community-Tech, and 6 others: Provide the ability to have time-delayed or time-offset jobs in the job queue - https://phabricator.wikimedia.org/T218812 (10Pchelolo) [14:06:02] 10serviceops, 10Operations, 10User-jijiki: create IRC channel for the Service Operations SRE subteam - https://phabricator.wikimedia.org/T211902 (10ArielGlenn) [14:58:59] 10serviceops, 10Operations, 10User-jijiki: create IRC channel for the Service Operations SRE subteam - https://phabricator.wikimedia.org/T211902 (10Dzahn) a:03mark Hi Mark, this ticket is resolved since a while except that one check box to setup an access list and add admins. I see currently you are on... [15:34:17] 10serviceops, 10Analytics, 10ChangeProp, 10Community-Tech, and 6 others: Provide the ability to have time-delayed or time-offset jobs in the job queue - https://phabricator.wikimedia.org/T218812 (10aezell) Thanks for the insight @Joe. I appreciate your giving some thought to this. I'll share some of this... [16:45:34] akosiaris: if I want to tcpdump traffic on kafka brokers from my eventgate pods [16:45:39] what hostname can I filter for? [16:45:45] how do I find it? [16:46:21] is it the cluster ip? [16:48:24] hm welp i found it by tcpdumping on kubestage1002 [16:48:25] :) [16:48:31] woudl be nice to get it from cli tho [16:50:31] hmm perhaps net 10.64.75.0/24 [16:50:32] will do it :) [16:50:33] ? [16:52:33] <_joe_> ottomata: I think it should, yes, you can look at calico's configuration to know precisely [16:52:56] _joe_: in puppet? [16:53:08] <_joe_> ottomata: probably [16:53:11] k danke [16:53:21] <_joe_> ottomata: you can also see the range of ips reserved for pods in dns [16:53:53] for a given services or for all pods? [16:59:00] 10serviceops, 10ChangeProp, 10Release Pipeline, 10Patch-For-Review, and 2 others: Migrate changeprop to kubernetes - https://phabricator.wikimedia.org/T213193 (10Ottomata) BTW, I'm currently having a node-rdkafka + k8s related issue in EventGate: {T218268}. It is possible that there is some EventGate spec... [17:24:07] 10serviceops, 10Analytics, 10Analytics-Kanban, 10EventBus, and 3 others: eventgate-analytics k8s pods occasionally can't produce to kafka - https://phabricator.wikimedia.org/T218268 (10Ottomata) I tcpdumped while reproducing this on kubestage1001 and the kafka brokers. I could see the looped metadata requ... [17:25:27] <_joe_> for all pods [17:25:42] <_joe_> if you want the ips of pods from your services [17:26:01] <_joe_> some kubectl show should give you that [17:38:26] 10serviceops, 10Analytics, 10Analytics-Kanban, 10EventBus, and 3 others: eventgate-analytics k8s pods occasionally can't produce to kafka - https://phabricator.wikimedia.org/T218268 (10Ottomata) Interesting fact: kafka-jumbo1003 and kafka-jumbo1006 are the only brokers in this cluster that are not in Row... [18:31:09] I'm onto something! [18:31:12] 10serviceops, 10Analytics, 10Analytics-Kanban, 10EventBus, and 3 others: eventgate-analytics k8s pods occasionally can't produce to kafka - https://phabricator.wikimedia.org/T218268 (10Ottomata) Ohooohooooo! Could this be related to IPv6? On a [[ https://logstash.wikimedia.org/goto/1a4ba12d778db9ae95d75db... [18:31:17] any body left around here? [18:31:17] [18:31:18] ^^ [19:03:00] 10serviceops, 10Analytics, 10Analytics-Kanban, 10EventBus, and 3 others: eventgate-analytics k8s pods occasionally can't produce to kafka - https://phabricator.wikimedia.org/T218268 (10Ottomata) From https://github.com/edenhill/librdkafka/wiki/FAQ > librdkafka will use the system resolver to resolve the... [20:25:19] 10serviceops, 10Gerrit, 10Operations, 10Release-Engineering-Team: Deploy multi-site plugin to cobalt and gerrit2001 - https://phabricator.wikimedia.org/T217174 (10Paladox) [20:34:30] 10serviceops, 10TechCom-RFC (TechCom-Approved): RfC: Standards for external services in the Wikimedia infrastructure. - https://phabricator.wikimedia.org/T208524 (10kchapman) [20:35:09] 10serviceops, 10TechCom-RFC (TechCom-Approved): RfC: Standards for external services in the Wikimedia infrastructure. - https://phabricator.wikimedia.org/T208524 (10kchapman) TechCom has approved this