[00:16:08] 10serviceops, 10Deployments, 10Parsing-Team: Add a parse* host to the scap canary checks? - https://phabricator.wikimedia.org/T246728 (10Jdforrester-WMF) [00:54:51] 10serviceops, 10Parsing-critical-path, 10Parsoid-PHP, 10MW-1.35-notes (1.35.0-wmf.20; 2020-02-18), 10Patch-For-Review: Craft a deployment strategy to transition Parsoid/PHP from a faux extension to a composer library without breaking incoming requests - https://phabricator.wikimedia.org/T240055 (10Jdforre... [01:02:45] 10serviceops, 10Operations, 10Parsoid-PHP, 10SRE-Access-Requests: Give all members of the Parsing team production `deployment` access ( add arlolra to deployers) - https://phabricator.wikimedia.org/T245877 (10cscott) Ok, tested deploy rights today. We depooled wtp1025 temporarily for testing. On wtp1025... [12:13:44] 10serviceops, 10Operations, 10ops-eqiad: (Need by: 2020-02-28) rack/setup/install mw[1385-1413].eqiad.wmnet - https://phabricator.wikimedia.org/T241849 (10Cmjohnson) [13:27:12] 10serviceops, 10Operations, 10ops-eqiad: (Need by: 2020-02-28) rack/setup/install mw[1385-1413].eqiad.wmnet - https://phabricator.wikimedia.org/T241849 (10Cmjohnson) [13:30:28] 10serviceops, 10Operations, 10ops-eqiad: (Need by: 2020-02-28) rack/setup/install mw[1385-1413].eqiad.wmnet - https://phabricator.wikimedia.org/T241849 (10Cmjohnson) Everything but the initial puppet run has been completed. Did the puppet certification process change? This fails now cmjohnson@puppetmaster... [14:02:05] _joe_: hiya, is there a reason why https://wikitech.wikimedia.org/wiki/LVS#Add_a_new_load_balanced_service has adding DNS .svc entries after adding the etcd data? [14:02:27] i'd expect to just do all .svc DNS stuff first [14:02:30] just curious [14:03:06] <_joe_> you could do it either way [14:03:43] k [14:03:48] wasn't sure if there was a reason, danke [14:04:51] am prepping to add eventgate-analytics-external LVS, should I have a guide or can I just try to follow the instructions? [14:05:44] <_joe_> try to follow the instructions I guess [14:08:40] _joe_: q: "(Instead of a dedicated conftool service name, just use kubesvc as the service name instead.)" [14:08:49] i'm not sure where it is telling me to to use kubesvc [14:09:03] <_joe_> the conftool stanza of the config [14:09:22] AH in service:catalog [14:09:44] <_joe_> yeah you can skip the adding conftool step i guess for the pools [14:10:36] <_joe_> yes [14:11:06] 10serviceops, 10Operations, 10Parsoid-PHP, 10SRE-Access-Requests: Give all members of the Parsing team production `deployment` access ( add arlolra to deployers) - https://phabricator.wikimedia.org/T245877 (10Dzahn) 05Resolved→03Open [14:11:09] 10serviceops, 10Parsing-critical-path, 10Parsoid-PHP, 10MW-1.35-notes (1.35.0-wmf.20; 2020-02-18), 10Patch-For-Review: Craft a deployment strategy to transition Parsoid/PHP from a faux extension to a composer library without breaking incoming requests - https://phabricator.wikimedia.org/T240055 (10Dzahn) [14:41:21] hm, so, i'm changing some LVS services ports, and as far as I know changed all producers that used the services. [14:41:28] however I still see some traffic to the old ports [14:41:37] i'm not sure ehow I can find out where that traffic is coming from. [14:41:47] if not k8s, i'd tcpdump the port on the service node [14:41:59] i guess i can maybe do the same on one of the kubernetes nodes where my pod runs? [14:42:06] not having luck with that though [14:56:50] btw got some tcpdumps by finding the if name in docker [15:02:37] 10serviceops, 10Analytics, 10Analytics-Kanban, 10Event-Platform, 10Patch-For-Review: Create production and canary releases for existent eventgate helmfile services - https://phabricator.wikimedia.org/T245203 (10Ottomata) [15:13:02] 10serviceops, 10Analytics, 10Analytics-Kanban, 10Event-Platform: Create production and canary releases for existent eventgate helmfile services - https://phabricator.wikimedia.org/T245203 (10Ottomata) [15:21:46] 10serviceops, 10Operations, 10Parsoid-PHP, 10SRE-Access-Requests: Give all members of the Parsing team production `deployment` access ( add arlolra to deployers) - https://phabricator.wikimedia.org/T245877 (10Dzahn) Running scap from the deployment server (what James does) and running scap pull on a single... [16:35:13] let parsoid-admins also run scap pull on wtp servers: https://gerrit.wikimedia.org/r/c/operations/puppet/+/576383 reasonable? [16:43:42] 10serviceops, 10Operations, 10Parsoid-PHP, 10SRE-Access-Requests, 10Patch-For-Review: Give all members of the Parsing team production `deployment` access ( add arlolra to deployers) - https://phabricator.wikimedia.org/T245877 (10Dzahn) The change above would let parsoid-admins run "scap pull" AS user "mw... [16:44:51] 10serviceops, 10Operations, 10ops-eqiad: (Need by: 2020-02-28) rack/setup/install mw[1385-1413].eqiad.wmnet - https://phabricator.wikimedia.org/T241849 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by cmjohnson on cumin1001.eqiad.wmnet for hosts: ` mw1385.eqiad.wmnet ` The log can be found in `... [16:48:11] 10serviceops, 10Operations, 10ops-eqiad: (Need by: 2020-02-28) rack/setup/install mw[1385-1413].eqiad.wmnet - https://phabricator.wikimedia.org/T241849 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by cmjohnson on cumin1001.eqiad.wmnet for hosts: ` mw1385.eqiad.wmnet ` The log can be found in `... [16:49:26] 10serviceops, 10Operations, 10ops-eqiad: (Need by: 2020-02-28) rack/setup/install mw[1385-1413].eqiad.wmnet - https://phabricator.wikimedia.org/T241849 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by cmjohnson on cumin1001.eqiad.wmnet for hosts: ` mw1386.eqiad.wmnet ` The log can be found in `... [16:50:02] 10serviceops, 10Operations, 10ops-eqiad: (Need by: 2020-02-28) rack/setup/install mw[1385-1413].eqiad.wmnet - https://phabricator.wikimedia.org/T241849 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by cmjohnson on cumin1001.eqiad.wmnet for hosts: ` mw1387.eqiad.wmnet ` The log can be found in `... [17:00:08] 10serviceops, 10Operations, 10ops-eqiad: (Need by: 2020-02-28) rack/setup/install mw[1385-1413].eqiad.wmnet - https://phabricator.wikimedia.org/T241849 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by cmjohnson on cumin1001.eqiad.wmnet for hosts: ` mw1388.eqiad.wmnet ` The log can be found in `... [17:01:21] 10serviceops, 10Operations, 10ops-eqiad: (Need by: 2020-02-28) rack/setup/install mw[1385-1413].eqiad.wmnet - https://phabricator.wikimedia.org/T241849 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by cmjohnson on cumin1001.eqiad.wmnet for hosts: ` mw1389.eqiad.wmnet ` The log can be found in `... [17:04:12] 10serviceops, 10Operations, 10ops-eqiad: (Need by: 2020-02-28) rack/setup/install mw[1385-1413].eqiad.wmnet - https://phabricator.wikimedia.org/T241849 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by cmjohnson on cumin1001.eqiad.wmnet for hosts: ` mw1391.eqiad.wmnet ` The log can be found in `... [17:04:53] 10serviceops, 10Operations, 10ops-eqiad: (Need by: 2020-02-28) rack/setup/install mw[1385-1413].eqiad.wmnet - https://phabricator.wikimedia.org/T241849 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by cmjohnson on cumin1001.eqiad.wmnet for hosts: ` mw1392.eqiad.wmnet ` The log can be found in `... [17:14:38] 10serviceops, 10Operations, 10ops-eqiad: (Need by: 2020-02-28) rack/setup/install mw[1385-1413].eqiad.wmnet - https://phabricator.wikimedia.org/T241849 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by cmjohnson on cumin1001.eqiad.wmnet for hosts: ` mw1393.eqiad.wmnet ` The log can be found in `... [17:15:20] 10serviceops, 10Operations, 10ops-eqiad: (Need by: 2020-02-28) rack/setup/install mw[1385-1413].eqiad.wmnet - https://phabricator.wikimedia.org/T241849 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by cmjohnson on cumin1001.eqiad.wmnet for hosts: ` mw1394.eqiad.wmnet ` The log can be found in `... [17:16:10] 10serviceops, 10Operations, 10ops-eqiad: (Need by: 2020-02-28) rack/setup/install mw[1385-1413].eqiad.wmnet - https://phabricator.wikimedia.org/T241849 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by cmjohnson on cumin1001.eqiad.wmnet for hosts: ` mw1395.eqiad.wmnet ` The log can be found in `... [17:19:37] 10serviceops, 10Analytics, 10Analytics-Kanban, 10Event-Platform: Create production and canary releases for existent eventgate helmfile services - https://phabricator.wikimedia.org/T245203 (10Ottomata) [17:19:56] 10serviceops, 10Operations, 10ops-eqiad: (Need by: 2020-02-28) rack/setup/install mw[1385-1413].eqiad.wmnet - https://phabricator.wikimedia.org/T241849 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by cmjohnson on cumin1001.eqiad.wmnet for hosts: ` mw1396.eqiad.wmnet ` The log can be found in `... [17:23:24] 10serviceops, 10Operations, 10Parsoid-PHP, 10SRE-Access-Requests, 10Patch-For-Review: Give all members of the Parsing team production `deployment` access ( add arlolra to deployers) - https://phabricator.wikimedia.org/T245877 (10Dzahn) @cscott @ssastry After the merge above and the next puppet run all ex... [17:23:33] 10serviceops, 10Operations, 10ops-eqiad: (Need by: 2020-02-28) rack/setup/install mw[1385-1413].eqiad.wmnet - https://phabricator.wikimedia.org/T241849 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by cmjohnson on cumin1001.eqiad.wmnet for hosts: ` mw1397.eqiad.wmnet ` The log can be found in `... [17:27:55] 10serviceops, 10Operations, 10ops-eqiad: (Need by: 2020-02-28) rack/setup/install mw[1385-1413].eqiad.wmnet - https://phabricator.wikimedia.org/T241849 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by cmjohnson on cumin1001.eqiad.wmnet for hosts: ` mw1399.eqiad.wmnet ` The log can be found in `... [17:31:33] 10serviceops, 10Operations, 10ops-eqiad: (Need by: 2020-02-28) rack/setup/install mw[1385-1413].eqiad.wmnet - https://phabricator.wikimedia.org/T241849 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by cmjohnson on cumin1001.eqiad.wmnet for hosts: ` mw1400.eqiad.wmnet ` The log can be found in `... [17:32:11] 10serviceops, 10Operations, 10ops-eqiad: (Need by: 2020-02-28) rack/setup/install mw[1385-1413].eqiad.wmnet - https://phabricator.wikimedia.org/T241849 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by cmjohnson on cumin1001.eqiad.wmnet for hosts: ` mw1401.eqiad.wmnet ` The log can be found in `... [17:33:06] 10serviceops, 10Operations, 10ops-eqiad: (Need by: 2020-02-28) rack/setup/install mw[1385-1413].eqiad.wmnet - https://phabricator.wikimedia.org/T241849 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by cmjohnson on cumin1001.eqiad.wmnet for hosts: ` mw1402.eqiad.wmnet ` The log can be found in `... [17:35:34] 10serviceops, 10Operations, 10ops-eqiad: (Need by: 2020-02-28) rack/setup/install mw[1385-1413].eqiad.wmnet - https://phabricator.wikimedia.org/T241849 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by cmjohnson on cumin1001.eqiad.wmnet for hosts: ` mw1403.eqiad.wmnet ` The log can be found in `... [17:38:32] 10serviceops, 10Operations, 10ops-eqiad: (Need by: 2020-02-28) rack/setup/install mw[1385-1413].eqiad.wmnet - https://phabricator.wikimedia.org/T241849 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by cmjohnson on cumin1001.eqiad.wmnet for hosts: ` mw1398.eqiad.wmnet ` The log can be found in `... [17:42:42] 10serviceops, 10Operations, 10ops-eqiad: (Need by: 2020-02-28) rack/setup/install mw[1385-1413].eqiad.wmnet - https://phabricator.wikimedia.org/T241849 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by cmjohnson on cumin1001.eqiad.wmnet for hosts: ` mw1404.eqiad.wmnet ` The log can be found in `... [17:45:39] akosiaris: this look ok? [17:45:39] https://gerrit.wikimedia.org/r/c/operations/puppet/+/576402 [17:45:48] am following https://wikitech.wikimedia.org/wiki/LVS#Remove_a_load_balanced_service [17:48:55] 10serviceops, 10Operations, 10ops-eqiad: (Need by: 2020-02-28) rack/setup/install mw[1385-1413].eqiad.wmnet - https://phabricator.wikimedia.org/T241849 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by cmjohnson on cumin1001.eqiad.wmnet for hosts: ` mw1405.eqiad.wmnet ` The log can be found in `... [17:49:18] 10serviceops, 10Operations, 10ops-eqiad: (Need by: 2020-02-28) rack/setup/install mw[1385-1413].eqiad.wmnet - https://phabricator.wikimedia.org/T241849 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by cmjohnson on cumin1001.eqiad.wmnet for hosts: ` mw1406.eqiad.wmnet ` The log can be found in `... [17:49:37] 10serviceops, 10Operations, 10ops-eqiad: (Need by: 2020-02-28) rack/setup/install mw[1385-1413].eqiad.wmnet - https://phabricator.wikimedia.org/T241849 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by cmjohnson on cumin1001.eqiad.wmnet for hosts: ` mw1407.eqiad.wmnet ` The log can be found in `... [17:53:23] 10serviceops, 10Operations, 10ops-eqiad: (Need by: 2020-02-28) rack/setup/install mw[1385-1413].eqiad.wmnet - https://phabricator.wikimedia.org/T241849 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by cmjohnson on cumin1001.eqiad.wmnet for hosts: ` mw1403.eqiad.wmnet ` The log can be found in `... [17:53:39] 10serviceops, 10ChangeProp, 10Release Pipeline, 10Release-Engineering-Team-TODO, and 3 others: Migrate changeprop to kubernetes - https://phabricator.wikimedia.org/T213193 (10hnowlan) [17:53:51] 10serviceops, 10Operations, 10ops-eqiad: (Need by: 2020-02-28) rack/setup/install mw[1385-1413].eqiad.wmnet - https://phabricator.wikimedia.org/T241849 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by cmjohnson on cumin1001.eqiad.wmnet for hosts: ` mw1408.eqiad.wmnet ` The log can be found in `... [17:54:52] 10serviceops, 10ChangeProp, 10Release Pipeline, 10Release-Engineering-Team-TODO, and 3 others: Migrate changeprop to kubernetes - https://phabricator.wikimedia.org/T213193 (10hnowlan) changeprop now deploys to staging and creates pods successfully. Errors out with this, as expected. Nutcracker sidecar will... [17:57:46] 10serviceops, 10Operations, 10ops-eqiad: (Need by: 2020-02-28) rack/setup/install mw[1385-1413].eqiad.wmnet - https://phabricator.wikimedia.org/T241849 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by cmjohnson on cumin1001.eqiad.wmnet for hosts: ` mw1409.eqiad.wmnet ` The log can be found in `... [18:03:50] 10serviceops, 10Operations, 10ops-eqiad: (Need by: 2020-02-28) rack/setup/install mw[1385-1413].eqiad.wmnet - https://phabricator.wikimedia.org/T241849 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by cmjohnson on cumin1001.eqiad.wmnet for hosts: ` mw1410.eqiad.wmnet ` The log can be found in `... [18:04:18] 10serviceops, 10Operations, 10ops-eqiad: (Need by: 2020-02-28) rack/setup/install mw[1385-1413].eqiad.wmnet - https://phabricator.wikimedia.org/T241849 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by cmjohnson on cumin1001.eqiad.wmnet for hosts: ` mw1411.eqiad.wmnet ` The log can be found in `... [18:04:32] 10serviceops, 10Operations, 10ops-eqiad: (Need by: 2020-02-28) rack/setup/install mw[1385-1413].eqiad.wmnet - https://phabricator.wikimedia.org/T241849 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by cmjohnson on cumin1001.eqiad.wmnet for hosts: ` mw1412.eqiad.wmnet ` The log can be found in `... [18:04:50] 10serviceops, 10Operations, 10ops-eqiad: (Need by: 2020-02-28) rack/setup/install mw[1385-1413].eqiad.wmnet - https://phabricator.wikimedia.org/T241849 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by cmjohnson on cumin1001.eqiad.wmnet for hosts: ` mw1413.eqiad.wmnet ` The log can be found in `... [18:06:03] 10serviceops, 10Release Pipeline, 10Wikimedia-Portals, 10Release-Engineering-Team (Pipeline): Migrate www.wikimedia.org (the portal) to be hosted as a service - https://phabricator.wikimedia.org/T238747 (10Jdforrester-WMF) [18:07:36] 10serviceops, 10Operations, 10Parsoid-PHP, 10SRE-Access-Requests, 10Patch-For-Review: Give all members of the Parsing team production `deployment` access ( add arlolra to deployers) - https://phabricator.wikimedia.org/T245877 (10Dzahn) 05Open→03Resolved [18:07:39] 10serviceops, 10Parsing-critical-path, 10Parsoid-PHP, 10MW-1.35-notes (1.35.0-wmf.20; 2020-02-18), 10Patch-For-Review: Craft a deployment strategy to transition Parsoid/PHP from a faux extension to a composer library without breaking incoming requests - https://phabricator.wikimedia.org/T240055 (10Dzahn) [18:17:13] 10serviceops, 10Operations, 10ops-eqiad: (Need by: 2020-02-28) rack/setup/install mw[1385-1413].eqiad.wmnet - https://phabricator.wikimedia.org/T241849 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['mw1408.eqiad.wmnet'] ` and were **ALL** successful. [18:19:40] 10serviceops, 10Operations, 10ops-eqiad: (Need by: 2020-02-28) rack/setup/install mw[1385-1413].eqiad.wmnet - https://phabricator.wikimedia.org/T241849 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['mw1409.eqiad.wmnet'] ` and were **ALL** successful. [18:26:05] 10serviceops, 10Operations, 10ops-eqiad: (Need by: 2020-02-28) rack/setup/install mw[1385-1413].eqiad.wmnet - https://phabricator.wikimedia.org/T241849 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['mw1410.eqiad.wmnet'] ` and were **ALL** successful. [18:26:16] 10serviceops, 10Operations, 10ops-eqiad: (Need by: 2020-02-28) rack/setup/install mw[1385-1413].eqiad.wmnet - https://phabricator.wikimedia.org/T241849 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['mw1411.eqiad.wmnet'] ` and were **ALL** successful. [18:26:22] 10serviceops, 10Operations, 10ops-eqiad: (Need by: 2020-02-28) rack/setup/install mw[1385-1413].eqiad.wmnet - https://phabricator.wikimedia.org/T241849 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['mw1412.eqiad.wmnet'] ` and were **ALL** successful. [18:26:47] 10serviceops, 10Operations, 10ops-eqiad: (Need by: 2020-02-28) rack/setup/install mw[1385-1413].eqiad.wmnet - https://phabricator.wikimedia.org/T241849 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['mw1413.eqiad.wmnet'] ` and were **ALL** successful. [18:33:45] 10serviceops, 10Operations, 10Traffic, 10Patch-For-Review: Applayer services without TLS - https://phabricator.wikimedia.org/T210411 (10Dzahn) [18:34:20] 10serviceops, 10Operations, 10Traffic, 10Patch-For-Review: Applayer services without TLS - https://phabricator.wikimedia.org/T210411 (10Dzahn) grafana-labs-admin.wikimedia.org has been removed from DNS in https://gerrit.wikimedia.org/r/c/operations/dns/+/576408 therefore also removed here [18:37:58] 10serviceops, 10Operations, 10Traffic, 10Patch-For-Review: Applayer services without TLS - https://phabricator.wikimedia.org/T210411 (10Dzahn) [18:38:50] 10serviceops, 10Operations, 10Traffic, 10Patch-For-Review: Applayer services without TLS - https://phabricator.wikimedia.org/T210411 (10Dzahn) labmon1001 has been replaced by cloudmetrics1002 and is still hosting grafana-labs and graphite-labs. [18:46:38] I see this in the cergen wiki page (now): "By setting a key.password, cergen will output encrypted private key files. If you need an unencrypted private key file (e.g. for envoyproxy), you can omit the key.password. " But i always used passwords and the certs were for envoyproxy and it worked? [18:46:59] was that a special usecase for analytics maybe [20:56:42] 10serviceops, 10Operations, 10Traffic, 10Patch-For-Review: Applayer services without TLS - https://phabricator.wikimedia.org/T210411 (10Dzahn) [20:57:13] 10serviceops, 10Operations, 10Traffic, 10Patch-For-Review: Applayer services without TLS - https://phabricator.wikimedia.org/T210411 (10Dzahn) grafana-labs and graphite-labs have switched to TLS now. [x] cloudmetrics1002.eqiad.wmnet - http://grafana-labs.wikimedia.org http://graphite-labs.wikimedia.org [20:59:53] more envoy ^