[02:12:35] 10serviceops, 10Operations, 10ops-codfw: rack/setup/install new codfw mw systems - https://phabricator.wikimedia.org/T241852 (10Papaul) 25 servers in row B racked and Netbox updated mw2310-mw2335 [05:11:04] 10serviceops, 10Operations, 10Phabricator, 10Traffic, 10Release-Engineering-Team-TODO (2020-01 to 2020-03 (Q3)): Phabricator downtime due to aphlict and websockets (aphlict current disabled) - https://phabricator.wikimedia.org/T238593 (10DannyS712) a:03mmodell [08:23:31] 10serviceops, 10Operations, 10Wikimedia-Etherpad: vm request for etherpad1002 - https://phabricator.wikimedia.org/T243475 (10MoritzMuehlenhoff) p:05Triage→03Normal [08:47:23] 10serviceops, 10Operations, 10ops-eqiad: (Need By Dec 20) rack/setup/install mw13[49-84].eqiad.wmnet - https://phabricator.wikimedia.org/T236437 (10Jclark-ctr) [08:48:48] 10serviceops, 10Operations, 10ops-eqiad: (Need By Dec 20) rack/setup/install mw13[49-84].eqiad.wmnet - https://phabricator.wikimedia.org/T236437 (10Jclark-ctr) a:05Jclark-ctr→03Cmjohnson Host racked bios, ip , and password set. Needs dns server ip asset tag rack switch port mw1349 10.65.1.24 WMF5291 D1... [10:18:40] 10serviceops, 10Operations, 10Phabricator, 10Traffic, 10Release-Engineering-Team-TODO (2020-01 to 2020-03 (Q3)): Phabricator downtime due to aphlict and websockets (aphlict current disabled) - https://phabricator.wikimedia.org/T238593 (10Aklapper) [11:58:10] 10serviceops, 10Core Platform Team, 10MediaWiki-Cache, 10Performance-Team (Radar): Ensure apcu incr/decr are atomic (Upgrade php-apcu) - https://phabricator.wikimedia.org/T236800 (10jijiki) @Krinkle I can roll that out to the canaries, but finish production rollout until after all hands, does that sound go... [14:16:16] 10serviceops, 10Analytics, 10Analytics-Kanban: Clarify multi-service instance concepts in helm charts and enable canary releases - https://phabricator.wikimedia.org/T242861 (10Ottomata) > K cool, let's figure out a different name. I like service.name best, just don't want to confuse it with k8s Service. Sho... [14:43:57] 10serviceops, 10Operations, 10Patch-For-Review, 10User-jijiki: Create a mediawiki::cronjob define - https://phabricator.wikimedia.org/T211250 (10Joe) [14:56:25] akosiaris: maybe we can merge and deploy eventstreams to staging? [14:56:32] i'd like to benchmark as you suggested [14:56:44] it'll work as is, and we can incorporate the service.name stuff later [14:56:48] when we settle it? [14:58:10] ottomata: sure [14:58:14] sound fine to me [14:58:27] k [14:58:32] merging then! [14:58:39] I haven't yet commented on that btw, cause I am constantly through rabbitholes and wondering about it [14:58:45] :) [14:58:57] can't wait to hear what's in those rabbitholes [14:58:57] I 'd like to craft a patch to try that approach and see issues [14:59:01] k [14:59:08] otrs? etherpad? citoid/zotero [14:59:14] you really want me to continue? [14:59:29] hehe, well i want to hear what's at the end of those rabbitholes [14:59:51] ah .. mostly incidents, jessie deprecation and security upgrades [15:00:06] oh oh, you mean unrelated rabbitholes [15:00:20] yeah sorry, I should have said distracted [15:00:22] :) [15:00:57] akosiaris do you need to create the eventstreams namepace?, [15:01:15] ah yes indeed. Lemme do that now, before I am drawn into meetings [15:01:19] ok danke [15:04:56] ottomata: task? [15:05:16] https://phabricator.wikimedia.org/T238658 [15:09:59] 10serviceops, 10Operations: Upgrade and improve our application object caching service (memcached) - https://phabricator.wikimedia.org/T240684 (10jijiki) Gutter pool has been initially tested in Beta and looks well. To make this test work, we deployed the a config to mcrouter (attached at the bottom) running o... [15:23:28] 10serviceops, 10Operations: Upgrade and improve our application object caching service (memcached) - https://phabricator.wikimedia.org/T240684 (10jijiki) [16:11:19] akosiaris: how goes, can I deploy to staging? [16:13:02] ottomata: minor rabbithole to make my life easier [16:13:03] https://gerrit.wikimedia.org/r/#/c/operations/deployment-charts/+/566781/ [16:13:22] just checking it, but if this is a noop, evenstreams changes will be like 6 lines [16:13:29] btw, what does this need to talk to? [16:13:33] kafka hosts? [16:13:37] nice [16:13:39] I guess it's in the config, right ? [16:14:02] hm, yes kafka hosts. that should be it. aside from ingress from lvs eventually [16:14:13] kafka main hosts [16:14:15] just like eventgate main [16:14:33] oh, we should use kafka tls though i think [16:14:39] put both ports there [16:14:44] i think eventagte main has that [16:14:45] 9092 and 9093 [16:16:59] ok, will copy it from that then [16:17:00] thanks [16:18:47] 10serviceops, 10Operations: Move debugging symbols and tools to a new class - https://phabricator.wikimedia.org/T236048 (10Jdforrester-WMF) Boldly unlinking this from the parent as it can't block it if that's Resolved. [16:18:52] 10serviceops, 10Operations, 10ops-eqiad: (Need By: Jan 10) rack/setup/install mc-gp100[123].eqiad.wmnet - https://phabricator.wikimedia.org/T241795 (10Jclark-ctr) a:05Jclark-ctr→03Cmjohnson [16:18:55] 10serviceops, 10Operations: Move debugging symbols and tools to a new class - https://phabricator.wikimedia.org/T236048 (10Jdforrester-WMF) [16:18:58] 10serviceops, 10Operations, 10MW-1.35-notes (1.35.0-wmf.3; 2019-10-22), 10Patch-For-Review, 10Performance-Team (Radar): Remove HHVM from production - https://phabricator.wikimedia.org/T229792 (10Jdforrester-WMF) [16:22:49] <_joe_> do we have things to discuss today? [16:23:08] <_joe_> rlazarus and I are doing a thorny production transition [16:23:11] <_joe_> and we're not doen [16:23:33] I don't have anything [16:28:47] <_joe_> ok let's meet and if nothing is here that can't wait for monday, we will talk [17:18:55] ottomata: changes merged and deployed. Token generated and deployed, you should be good to go on all 3 clusters [17:19:05] ok great! tahnks [17:19:11] will deploy to staging shortly and do some benching [17:33:25] 10serviceops, 10Operations, 10Scap: Make canary wait time configurable - https://phabricator.wikimedia.org/T217924 (10jijiki) If it is a lot of work to limit when `--canary-wait-time` is available, we could do a graceful rollout, by asking deployers, via `utils.ask/utils.confirm`), to try this flag on, say,... [18:47:38] 10serviceops, 10Operations, 10ops-eqiad: (Need By: Jan 10) rack/setup/install mc-gp100[123].eqiad.wmnet - https://phabricator.wikimedia.org/T241795 (10Cmjohnson) @elukey @jijiki I am going as fast as I can ...there are several racking tasks that need to be completed. John updated switch ports this morning. [18:50:05] 10serviceops, 10Operations, 10ops-eqiad: (Need By: Jan 10) rack/setup/install mc-gp100[123].eqiad.wmnet - https://phabricator.wikimedia.org/T241795 (10Cmjohnson) not hitting the installer...still working on them [19:02:14] 10serviceops, 10Operations, 10Wikimedia-Etherpad: vm request for etherpad1002 - https://phabricator.wikimedia.org/T243475 (10Dzahn) a:03Dzahn [20:05:41] hmmm [20:05:57] how can I test the prometheus metrics coming from k8s staging? [20:06:49] the port is not exported in a service, i think our k8s stuff somehow queries and presents it? [20:12:01] ottomata: puppet modules/profile/manifests/prometheus/k8s/staging.pp might be helpful [20:26:32] hmmm [20:34:06] i have found a lot of ports...i have a feeling there would be a port on e.g. kubestage1001 i could hit to get prometheus metrics [20:34:08] not sure though [23:13:08] 10serviceops, 10Operations, 10ops-codfw: rack/setup/install new codfw mw systems - https://phabricator.wikimedia.org/T241852 (10Papaul) 19 servers in row A racked and Netnox updated mw2291-mw2309 [23:16:36] 10serviceops, 10Operations, 10Wikimedia-Etherpad: vm request for etherpad1002 - https://phabricator.wikimedia.org/T243475 (10Dzahn) 05Open→03Resolved [23:16:40] 10serviceops, 10Operations, 10Wikimedia-Etherpad, 10Patch-For-Review: Migrate etherpad1001 to Buster - https://phabricator.wikimedia.org/T224580 (10Dzahn) [23:17:12] 10serviceops, 10Operations, 10Wikimedia-Etherpad, 10Patch-For-Review: Migrate etherpad1001 to Buster - https://phabricator.wikimedia.org/T224580 (10Dzahn) >>! In T224580#5823156, @akosiaris wrote: >> * prometheus-etherpad-exporter >> * etherpad-lite > > I think both are done now. Wow that was so quick an... [23:41:50] 10serviceops, 10Operations, 10ops-codfw: rack/setup/install new codfw mw systems - https://phabricator.wikimedia.org/T241852 (10Papaul) switch port information for mw servers in row B rack B3 mw2310-mw2335 : ge-3/0/[26-40]