[00:16:45] 10serviceops, 10SRE, 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO, 10User-jijiki: Upgrade MediaWiki clusters to Debian Buster (debian 10) - https://phabricator.wikimedia.org/T245757 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['mw1278.eqiad.wmnet'] ` an... [07:03:09] 10serviceops, 10Analytics, 10Analytics-Kanban, 10Event-Platform, and 5 others: Set up internal eventstreams instance exposing all streams declared in stream config (and in kafka jumbo) - https://phabricator.wikimedia.org/T269160 (10elukey) 05Open→03Resolved [07:45:55] _joe_: should we push the resharding patch today or on Monday early morning ? [07:55:50] 10serviceops, 10Push-Notification-Service, 10Product-Infrastructure-Team-Backlog (Kanban), 10User-jijiki: Set up secrets for Token clean-up - https://phabricator.wikimedia.org/T262957 (10jijiki) 05Open→03Resolved [07:55:53] 10serviceops, 10Push-Notification-Service, 10Patch-For-Review, 10Product-Infrastructure-Team-Backlog (Kanban): Push notification service should make deletion requests to MediaWiki for invalid or expired subscriptions - https://phabricator.wikimedia.org/T260247 (10jijiki) [07:58:03] 10serviceops, 10Excimer, 10Performance-Team, 10Sustainability (Incident Followup), 10User-jijiki: Verify Excimer's performance footprint - https://phabricator.wikimedia.org/T270152 (10Joe) > We feel pretty good about it being low overhead, but I don't think we've ever validated this emperically. This is... [08:00:47] <_joe_> effie: monday [08:01:06] ok, I will push it in the early eu morning [08:22:27] 10serviceops, 10SRE, 10User-jijiki: Modernise memcached systemd unit / sync to current buster setup - https://phabricator.wikimedia.org/T273950 (10jijiki) p:05Triage→03Medium [08:42:56] 10serviceops, 10SRE: Kubernetes hosts raid check make facter fail - https://phabricator.wikimedia.org/T237197 (10JMeybohm) 05Open→03Resolved a:03JMeybohm I'm going to close this as we don't have this particular problem anymore (AFAICT) with mdadm checks being spread out across the first week of each month. [11:41:24] 10serviceops, 10MW-on-K8s, 10SRE, 10Shellbox, and 4 others: RFC: PHP microservice for containerized shell execution - https://phabricator.wikimedia.org/T260330 (10Daimona) [14:25:24] _joe_: i think i rember you saying recently that the ipresolv function didn;t actully add anything as the cache wasn't working as expected. 1) does that ring a bell; 2) if so was it fixed; 3) if not what was the original expecation 4) if not is it worth spending time on this since we have lived without it working for some time? [14:26:08] i.e. its easier to create something that just dose DNS and dosn;t try to do the cache part [14:26:33] <_joe_> so, it does add value in it's called everywhere, and it does allow caching over the single puppet compiler run [14:26:37] <_joe_> so it does something [14:26:57] <_joe_> we should fix the fact that puppet-via-passenger apparently doesn't have the concept of "global instances" [14:28:24] <_joe_> so I would not throw it away atm, but yeah I don't have time to fix it either right now [14:28:25] so was/is the intent/desire that the cache should exist for the life of the passanger process as apose to the compilation run [14:28:35] <_joe_> yes [14:28:39] ac [14:28:41] ack [14:28:44] <_joe_> as the cache was using TTLs [14:28:58] <_joe_> so it would've served to reduce the amount of dns queries from compiling puppet [14:31:06] ack i can have an explore on that however i wonder if the globa state is more a limitation in passanger then puppet with passanger. [15:07:38] <_joe_> jbond42: yeah [15:07:53] <_joe_> but even persisting within the same passenger process would be great [15:12:28] ack cheers [15:36:59] 10serviceops, 10Gerrit, 10SRE, 10cloud-services-team: Change /r/p/ to /r/ on all hosts (where https://gerrit.wikimedia.org/r/p/ exists) - https://phabricator.wikimedia.org/T222093 (10hashar) 05Open→03Resolved a:03Paladox AFAIK @Paladox has handled the whole migration. We might have some rewrite rule... [15:48:50] https://www.docker.com/blog/donating-docker-distribution-to-the-cncf/ [15:49:58] <_joe_> akosiaris: the url is wrong, it should say "dumping onto" [15:50:14] lol [16:52:11] 10serviceops, 10SRE, 10User-jijiki: ifup@eno1.service fails on mc* hosts after 4.19.171-2 upgrade - https://phabricator.wikimedia.org/T273918 (10elukey) ` elukey@mc2029:~$ python3 Python 3.7.3 (default, Jul 25 2020, 13:03:44) [GCC 8.3.0] on linux Type "help", "copyright", "credits" or "license" for more inf... [17:12:50] 10serviceops, 10Prod-Kubernetes: Kubernetes prod and staging share the same cluster key in hiera - https://phabricator.wikimedia.org/T273866 (10akosiaris) > We should probably use kubernetes-staging for staging kubernetes and corresponding etcd, but I'm unsure about the consequences a change like this would ha... [19:33:34] 10serviceops, 10SRE, 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO, 10User-jijiki: Upgrade MediaWiki clusters to Debian Buster (debian 10) - https://phabricator.wikimedia.org/T245757 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by dzahn on cumin1001.eqia... [19:34:41] 10serviceops, 10SRE, 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO, 10User-jijiki: Upgrade MediaWiki clusters to Debian Buster (debian 10) - https://phabricator.wikimedia.org/T245757 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by dzahn on cumin1001.eqia... [19:35:53] 10serviceops, 10SRE, 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO, 10User-jijiki: Upgrade MediaWiki clusters to Debian Buster (debian 10) - https://phabricator.wikimedia.org/T245757 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by dzahn on cumin1001.eqia... [19:38:05] 10serviceops, 10SRE, 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO, 10User-jijiki: Upgrade MediaWiki clusters to Debian Buster (debian 10) - https://phabricator.wikimedia.org/T245757 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by dzahn on cumin1001.eqia... [19:39:12] 10serviceops, 10SRE, 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO, 10User-jijiki: Upgrade MediaWiki clusters to Debian Buster (debian 10) - https://phabricator.wikimedia.org/T245757 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by dzahn on cumin1001.eqia... [20:15:28] 10serviceops, 10SRE, 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO, 10User-jijiki: Upgrade MediaWiki clusters to Debian Buster (debian 10) - https://phabricator.wikimedia.org/T245757 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['mw1395.eqiad.wmnet'] ` an... [20:16:37] 10serviceops, 10SRE, 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO, 10User-jijiki: Upgrade MediaWiki clusters to Debian Buster (debian 10) - https://phabricator.wikimedia.org/T245757 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['mw1394.eqiad.wmnet'] ` an... [20:23:55] 10serviceops, 10SRE, 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO, 10User-jijiki: Upgrade MediaWiki clusters to Debian Buster (debian 10) - https://phabricator.wikimedia.org/T245757 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['mw2289.codfw.wmnet'] ` an... [20:33:43] 10serviceops, 10SRE, 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO, 10User-jijiki: Upgrade MediaWiki clusters to Debian Buster (debian 10) - https://phabricator.wikimedia.org/T245757 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by dzahn on cumin1001.eqia... [20:38:42] 10serviceops, 10SRE, 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO, 10User-jijiki: Upgrade MediaWiki clusters to Debian Buster (debian 10) - https://phabricator.wikimedia.org/T245757 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by dzahn on cumin1001.eqia... [20:44:25] 10serviceops, 10SRE, 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO, 10User-jijiki: Upgrade MediaWiki clusters to Debian Buster (debian 10) - https://phabricator.wikimedia.org/T245757 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by dzahn on cumin1001.eqia... [20:52:38] 10serviceops, 10SRE, 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO, 10User-jijiki: Upgrade MediaWiki clusters to Debian Buster (debian 10) - https://phabricator.wikimedia.org/T245757 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['mw1307.eqiad.wmnet'] ` an... [21:00:08] 10serviceops, 10SRE, 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO, 10User-jijiki: Upgrade MediaWiki clusters to Debian Buster (debian 10) - https://phabricator.wikimedia.org/T245757 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by dzahn on cumin1001.eqia... [21:02:13] 10serviceops, 10SRE, 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO, 10User-jijiki: Upgrade MediaWiki clusters to Debian Buster (debian 10) - https://phabricator.wikimedia.org/T245757 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['mw2254.codfw.wmnet'] ` an... [21:11:10] 10serviceops, 10SRE, 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO, 10User-jijiki: Upgrade MediaWiki clusters to Debian Buster (debian 10) - https://phabricator.wikimedia.org/T245757 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by dzahn on cumin1001.eqia... [21:15:48] 10serviceops, 10SRE, 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO, 10User-jijiki: Upgrade MediaWiki clusters to Debian Buster (debian 10) - https://phabricator.wikimedia.org/T245757 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['mw1393.eqiad.wmnet'] ` an... [21:21:36] 10serviceops, 10SRE, 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO, 10User-jijiki: Upgrade MediaWiki clusters to Debian Buster (debian 10) - https://phabricator.wikimedia.org/T245757 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['mw1392.eqiad.wmnet'] ` an... [21:26:31] 10serviceops, 10SRE, 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO, 10User-jijiki: Upgrade MediaWiki clusters to Debian Buster (debian 10) - https://phabricator.wikimedia.org/T245757 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['mw2266.codfw.wmnet'] ` an... [21:33:25] 10serviceops, 10SRE, 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO, 10User-jijiki: Upgrade MediaWiki clusters to Debian Buster (debian 10) - https://phabricator.wikimedia.org/T245757 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by legoktm on cumin1001.eq... [21:46:30] 10serviceops, 10MediaWiki-Cache, 10Performance-Team, 10MW-1.36-notes (1.36.0-wmf.27; 2021-01-19), and 2 others: Let WANObjectCache store "sister keys" on the same backend as the main value key - https://phabricator.wikimedia.org/T252564 (10Krinkle) 05Open→03Resolved [22:15:54] 10serviceops, 10SRE, 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO, 10User-jijiki: Upgrade MediaWiki clusters to Debian Buster (debian 10) - https://phabricator.wikimedia.org/T245757 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['mw1306.eqiad.wmnet'] ` an... [22:24:47] 10serviceops, 10SRE, 10Patch-For-Review: Upgrade and improve our application object caching service (memcached) - https://phabricator.wikimedia.org/T244852 (10Krinkle) [22:30:05] 10serviceops, 10MW-on-K8s, 10Release Pipeline, 10Release-Engineering-Team-TODO: Create restricted docker-registry namespace for security patched images - https://phabricator.wikimedia.org/T273521 (10Legoktm) a:03Legoktm Should the existence of these images also be secret? Or is it okay if they're listed... [22:31:42] 10serviceops, 10SRE, 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO, 10User-jijiki: Upgrade MediaWiki clusters to Debian Buster (debian 10) - https://phabricator.wikimedia.org/T245757 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['mw1269.eqiad.wmnet'] ` an... [22:53:22] 10serviceops, 10SRE, 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO, 10User-jijiki: Upgrade MediaWiki clusters to Debian Buster (debian 10) - https://phabricator.wikimedia.org/T245757 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['mw1285.eqiad.wmnet'] ` an... [22:57:07] 10serviceops, 10MW-on-K8s, 10Release Pipeline, 10Release-Engineering-Team-TODO: Create restricted docker-registry namespace for security patched images - https://phabricator.wikimedia.org/T273521 (10Legoktm) Also, is there a chance developers will ask to be able to pull images locally for debugging? I supp... [23:04:08] 10serviceops, 10MW-on-K8s, 10Release Pipeline, 10Release-Engineering-Team-TODO: Create restricted docker-registry namespace for security patched images - https://phabricator.wikimedia.org/T273521 (10dduvall) >>! In T273521#6807963, @Legoktm wrote: > Also, is there a chance developers will ask to be able to... [23:24:19] 10serviceops: New Service Request - Calculator Service - https://phabricator.wikimedia.org/T273807 (10wkandek) p:05Triage→03Low [23:54:37] 10serviceops, 10SRE, 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO, 10User-jijiki: Upgrade MediaWiki clusters to Debian Buster (debian 10) - https://phabricator.wikimedia.org/T245757 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by dzahn on cumin1001.eqia... [23:56:58] 10serviceops, 10SRE, 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO, 10User-jijiki: Upgrade MediaWiki clusters to Debian Buster (debian 10) - https://phabricator.wikimedia.org/T245757 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by dzahn on cumin1001.eqia... [23:58:56] 10serviceops, 10SRE, 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO, 10User-jijiki: Upgrade MediaWiki clusters to Debian Buster (debian 10) - https://phabricator.wikimedia.org/T245757 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by dzahn on cumin1001.eqia...