[08:52:32] 10serviceops, 10Prod-Kubernetes, 10Kubernetes: Flag --admission-control has been deprecated - https://phabricator.wikimedia.org/T270063 (10JMeybohm) [09:59:33] _joe_, akosiaris: FYI I've opened https://phabricator.wikimedia.org/T270071 for the SVC DNS stuff we discussed, I've used serviceops-radar feel free to add/modify it into serviceops :) [10:21:49] 10serviceops, 10Prod-Kubernetes, 10Kubernetes: kube-apiserver errors for calico CRDs - https://phabricator.wikimedia.org/T270081 (10JMeybohm) [10:22:09] 10serviceops, 10Prod-Kubernetes, 10Kubernetes: kube-apiserver errors for calico CRDs - https://phabricator.wikimedia.org/T270081 (10JMeybohm) [10:22:11] 10serviceops, 10Prod-Kubernetes, 10Kubernetes, 10Patch-For-Review: Upgrade kubernetes clusters to a security supported (LTS) version - https://phabricator.wikimedia.org/T244335 (10JMeybohm) [10:22:13] 10serviceops, 10Operations, 10Prod-Kubernetes, 10Kubernetes, 10User-fsero: Upgrade Calico - https://phabricator.wikimedia.org/T207804 (10JMeybohm) [10:55:09] 10serviceops, 10Prod-Kubernetes, 10Kubernetes: kube-apiserver errors for calico CRDs - https://phabricator.wikimedia.org/T270081 (10JMeybohm) 05Open→03Resolved a:03JMeybohm "[...] has invalid property: anyOf" seems like a bug in apiserver and can be ignored IIUC: https://github.com/kubernetes/kubernete... [10:55:12] 10serviceops, 10Operations, 10Prod-Kubernetes, 10Kubernetes, 10User-fsero: Upgrade Calico - https://phabricator.wikimedia.org/T207804 (10JMeybohm) [13:14:19] 10serviceops, 10Operations, 10Platform Engineering, 10Patch-For-Review, 10Performance-Team (Radar): Phasing out "redis_sessions" MediaWiki cluster - https://phabricator.wikimedia.org/T267581 (10jijiki) [14:38:16] 10serviceops, 10Performance-Team, 10Platform Engineering, 10Wikimedia-Rdbms, and 2 others: Determine multi-dc strategy for ChronologyProtector - https://phabricator.wikimedia.org/T254634 (10Joe) >>! In T254634#6687682, @Krinkle wrote: >>>! In T254634#6681828, @jijiki wrote: >> @Krinkle do you think it woul... [15:08:23] 10serviceops, 10WMF-JobQueue, 10Patch-For-Review, 10Platform Team Workboards (Clinic Duty Team), and 2 others: Could not enqueue jobs: "Unable to deliver all events: 503: Service Unavailable" - https://phabricator.wikimedia.org/T249745 (10Ottomata) So, I just encountered a 503 service unavailable from even... [15:23:32] chaomodus: I will have a look at the config again, this slipped my mind, sorry [15:56:48] I will be a bit late to the meeting [16:00:58] 10serviceops, 10Operations, 10Release-Engineering-Team-TODO, 10Patch-For-Review, and 2 others: Upgrade MediaWiki clusters to Debian Buster (debian 10) - https://phabricator.wikimedia.org/T245757 (10jijiki) [16:04:09] 10serviceops, 10Operations, 10ops-codfw, 10Patch-For-Review, 10Sustainability (Incident Followup): (Need by: TBD) rack/setup/install kubernetes20[07-14].codfw.wmnet and kubestage200[1-2].codfw.wmnet. - https://phabricator.wikimedia.org/T252185 (10akosiaris) [16:06:12] 10serviceops, 10Operations, 10ops-codfw, 10Patch-For-Review, 10Sustainability (Incident Followup): (Need by: TBD) rack/setup/install kubernetes20[07-14].codfw.wmnet and kubestage200[1-2].codfw.wmnet. - https://phabricator.wikimedia.org/T252185 (10akosiaris) [16:06:43] 10serviceops, 10Operations, 10ops-codfw: (Need by: TBD) rack/setup/install 86 new codfw mw systems - https://phabricator.wikimedia.org/T241852 (10akosiaris) [16:07:24] 10serviceops, 10Operations, 10ops-codfw, 10Patch-For-Review, 10Sustainability (Incident Followup): (Need by: TBD) rack/setup/install kubernetes20[07-14].codfw.wmnet and kubestage200[1-2].codfw.wmnet. - https://phabricator.wikimedia.org/T252185 (10akosiaris) 05Open→03Resolved And finally being now use... [16:15:32] effie: no worries, i still need to do a bit of a rejigger, and test that the config does what i want, so i'll nudge you after that. [16:28:38] ahoyhoy - I have a chart for the new sockpuppet-api service in review. Would anyone have a few minutes to look it over? It's a very basic chart for the most part [16:30:59] https://gerrit.wikimedia.org/r/643721 [17:08:54] 10serviceops, 10WMF-JobQueue, 10Patch-For-Review, 10Platform Team Workboards (Clinic Duty Team), and 2 others: Could not enqueue jobs: "Unable to deliver all events: 503: Service Unavailable" - https://phabricator.wikimedia.org/T249745 (10akosiaris) https://grafana-rw.wikimedia.org/d/g0GUXaJMk/t249745?orgI... [17:20:36] 10serviceops, 10WMF-JobQueue, 10Patch-For-Review, 10Platform Team Workboards (Clinic Duty Team), and 2 others: Could not enqueue jobs: "Unable to deliver all events: 503: Service Unavailable" - https://phabricator.wikimedia.org/T249745 (10Ottomata) Huh interesting. Bumped to from 300Mi to 600Mi in https:/... [17:59:00] chaomodus: cool cool [19:55:15] 10serviceops, 10Operations, 10Platform Engineering, 10Wikidata, and 4 others: Upgrade memcached cluster to Debian Buster - https://phabricator.wikimedia.org/T213089 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by jiji on cumin1001.eqiad.wmnet for hosts: ` mc2031.codfw.wmnet ` The log can be... [19:55:23] 10serviceops, 10Operations, 10Platform Engineering, 10Wikidata, and 4 others: Upgrade memcached cluster to Debian Buster - https://phabricator.wikimedia.org/T213089 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by jiji on cumin1001.eqiad.wmnet for hosts: ` mc1031.eqiad.wmnet ` The log can be... [20:33:55] 10serviceops, 10Operations, 10Platform Engineering, 10Wikidata, and 4 others: Upgrade memcached cluster to Debian Buster - https://phabricator.wikimedia.org/T213089 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['mc2031.codfw.wmnet'] ` and were **ALL** successful. [20:44:37] 10serviceops, 10Operations, 10Platform Engineering, 10Wikidata, and 4 others: Upgrade memcached cluster to Debian Buster - https://phabricator.wikimedia.org/T213089 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['mc1031.eqiad.wmnet'] ` and were **ALL** successful. [22:15:54] 10serviceops, 10Operations, 10Parsoid (Tracking), 10Patch-For-Review: Upgrade Parsoid servers to buster - https://phabricator.wikimedia.org/T268524 (10Dzahn) @jijiki @ssastry I wrote the following httpbb tests based on that: ` # tests for parsoid appservers # hosts: parse*, wtp* http://en.wikipedia.org:... [22:16:59] the tests for parsoid we talked about in meeting: https://gerrit.wikimedia.org/r/c/operations/puppet/+/648383/2/modules/profile/files/httpbb/parse/test_parse.yaml [22:26:11] 10serviceops, 10Operations, 10Parsoid (Tracking), 10Patch-For-Review: Upgrade Parsoid servers to buster - https://phabricator.wikimedia.org/T268524 (10Dzahn) [22:27:34] hmm, ifup@eno1.service failed on both mc[1031,2031] which we reimaged today [22:27:55] log says Cannot find device "eno1/64" [22:28:06] cc effie for tomorrow