[09:47:48] 10serviceops, 10SRE, 10ops-eqsin: ganeti5002 was down / powered off, machine check entries in SEL - https://phabricator.wikimedia.org/T261130 (10MoritzMuehlenhoff) @RobH, what's the status here? Was the IPMI error reproducible on a second attempt? [10:12:04] 10serviceops, 10Prod-Kubernetes, 10Kubernetes: kubestage200* change on every puppet run - https://phabricator.wikimedia.org/T271702 (10akosiaris) I 've had a look into it. The culprit is https://github.com/projectcalico/felix/pull/2424. The reason for the change itself is to honor kube-proxy rules in case of... [10:48:46] 10serviceops, 10SRE, 10docker-pkg: Duplicate image name in docker-images/production-images - https://phabricator.wikimedia.org/T271901 (10Joe) [10:49:02] 10serviceops, 10SRE, 10docker-pkg: Duplicate image name in docker-images/production-images - https://phabricator.wikimedia.org/T271901 (10Joe) p:05Triage→03Medium a:03Joe [11:12:12] 10serviceops, 10Analytics, 10Analytics-Kanban, 10Event-Platform, and 5 others: Set up internal eventstreams instance exposing all streams declared in stream config (and in kafka jumbo) - https://phabricator.wikimedia.org/T269160 (10elukey) Reserved port 4992 in https://wikitech.wikimedia.org/wiki/Service_p... [11:34:01] 10serviceops, 10Platform Engineering, 10SRE, 10Wikidata, and 4 others: Upgrade memcached cluster to Debian Buster - https://phabricator.wikimedia.org/T213089 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by jiji on cumin1001.eqiad.wmnet for hosts: ` mc1029.eqiad.wmnet ` The log can be found i... [11:34:17] 10serviceops, 10Platform Engineering, 10SRE, 10Wikidata, and 4 others: Upgrade memcached cluster to Debian Buster - https://phabricator.wikimedia.org/T213089 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by jiji on cumin1001.eqiad.wmnet for hosts: ` mc2029.codfw.wmnet ` The log can be found i... [11:35:35] 10serviceops, 10Analytics, 10Analytics-Kanban, 10Event-Platform, and 5 others: Set up internal eventstreams instance exposing all streams declared in stream config (and in kafka jumbo) - https://phabricator.wikimedia.org/T269160 (10elukey) I am following https://wikitech.wikimedia.org/wiki/Kubernetes#Add_a... [12:14:02] 10serviceops, 10Platform Engineering, 10SRE, 10Wikidata, and 4 others: Upgrade memcached cluster to Debian Buster - https://phabricator.wikimedia.org/T213089 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['mc1029.eqiad.wmnet'] ` Of which those **FAILED**: ` ['mc1029.eqiad.wmnet'] ` [12:36:15] 10serviceops, 10Platform Engineering, 10SRE, 10Wikidata, and 4 others: Upgrade memcached cluster to Debian Buster - https://phabricator.wikimedia.org/T213089 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['mc2029.codfw.wmnet'] ` Of which those **FAILED**: ` ['mc2029.codfw.wmnet'] ` [15:34:46] 10serviceops, 10SRE, 10Patch-For-Review: Upgrade and improve our application object caching service (memcached) - https://phabricator.wikimedia.org/T244852 (10jijiki) [15:35:17] 10serviceops, 10Platform Engineering, 10SRE, 10Wikidata, and 4 others: Upgrade memcached cluster to Debian Buster - https://phabricator.wikimedia.org/T213089 (10jijiki) [15:35:50] 10serviceops, 10Platform Engineering, 10SRE, 10Wikidata, and 4 others: Upgrade memcached cluster to Debian Buster - https://phabricator.wikimedia.org/T213089 (10jijiki) 05Open→03Resolved a:03jijiki Despite of what the above messages say, mc2029 and mc1029 were properly reimaged 🎉 [15:38:01] 10serviceops, 10Platform Engineering, 10SRE, 10Patch-For-Review, 10Performance-Team (Radar): Phasing out "redis_sessions" MediaWiki cluster - https://phabricator.wikimedia.org/T267581 (10jijiki) [15:38:09] 10serviceops, 10Platform Engineering, 10SRE, 10Wikidata, and 4 others: Upgrade memcached cluster to Debian Buster - https://phabricator.wikimedia.org/T213089 (10jijiki) [15:38:43] 10serviceops, 10Platform Engineering, 10SRE, 10Patch-For-Review, 10User-jijiki: Upgrade MediaWiki's Redis cluster to Debian Buster - https://phabricator.wikimedia.org/T265643 (10jijiki) 05Open→03Resolved a:03jijiki We ported version 2.8 to Buster, and all servers were upgraded as part of T213089. [15:41:04] 10serviceops, 10SRE, 10Patch-For-Review: Upgrade and improve our application object caching service (memcached) - https://phabricator.wikimedia.org/T244852 (10jijiki) [16:22:26] https://integration.wikimedia.org/ci/job/helm-lint/3297/console does the failure here happen because the chart I'm creating doesn't exist in the index yet or am I messing something else up? [16:32:27] hnowlan: yeah, thats the reason. The pipeline is just considering charts already in the repo [16:33:27] you could split the helmfile.d part into a second CR to have the pipeline properly check your stuff as soon as the chart CR is merged [16:37:01] ack, sounds good [16:54:44] 10serviceops, 10SRE, 10Patch-For-Review, 10Release-Engineering-Team (Deployment services), and 2 others: Upgrade MediaWiki clusters to Debian Buster (debian 10) - https://phabricator.wikimedia.org/T245757 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by dzahn on cumin1001.eqiad.wmnet for host... [16:56:41] 10serviceops, 10SRE, 10Patch-For-Review, 10Release-Engineering-Team (Deployment services), and 2 others: Upgrade MediaWiki clusters to Debian Buster (debian 10) - https://phabricator.wikimedia.org/T245757 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by dzahn on cumin1001.eqiad.wmnet for host... [16:59:31] 10serviceops, 10SRE, 10Patch-For-Review, 10Release-Engineering-Team (Deployment services), and 2 others: Upgrade MediaWiki clusters to Debian Buster (debian 10) - https://phabricator.wikimedia.org/T245757 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by dzahn on cumin1001.eqiad.wmnet for host... [17:13:51] 10serviceops, 10SRE, 10Patch-For-Review, 10Release-Engineering-Team (Deployment services), and 2 others: Upgrade MediaWiki clusters to Debian Buster (debian 10) - https://phabricator.wikimedia.org/T245757 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by dzahn on cumin1001.eqiad.wmnet for host... [18:07:09] 10serviceops, 10SRE, 10User-jijiki: Upgrade memcached to version 1.6.x - https://phabricator.wikimedia.org/T270315 (10jijiki) [18:14:12] 10serviceops, 10SRE, 10Patch-For-Review, 10Release-Engineering-Team (Deployment services), and 2 others: Upgrade MediaWiki clusters to Debian Buster (debian 10) - https://phabricator.wikimedia.org/T245757 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['mw2227.codfw.wmnet'] ` Of which those **F... [18:14:52] 10serviceops, 10SRE, 10Patch-For-Review, 10Release-Engineering-Team (Deployment services), and 2 others: Upgrade MediaWiki clusters to Debian Buster (debian 10) - https://phabricator.wikimedia.org/T245757 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['mw2229.codfw.wmnet'] ` Of which those **F... [18:17:40] 10serviceops, 10SRE, 10Patch-For-Review, 10Release-Engineering-Team (Deployment services), and 2 others: Upgrade MediaWiki clusters to Debian Buster (debian 10) - https://phabricator.wikimedia.org/T245757 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by dzahn on cumin1001.eqiad.wmnet for host... [18:17:49] 10serviceops, 10SRE, 10Patch-For-Review, 10Release-Engineering-Team (Deployment services), and 2 others: Upgrade MediaWiki clusters to Debian Buster (debian 10) - https://phabricator.wikimedia.org/T245757 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by dzahn on cumin1001.eqiad.wmnet for host... [18:18:04] 10serviceops, 10SRE, 10Patch-For-Review, 10Release-Engineering-Team (Deployment services), and 2 others: Upgrade MediaWiki clusters to Debian Buster (debian 10) - https://phabricator.wikimedia.org/T245757 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['mw2230.codfw.wmnet'] ` Of which those **F... [18:19:27] 10serviceops, 10SRE, 10User-jijiki: Enable TLS on memcached - https://phabricator.wikimedia.org/T271967 (10jijiki) [18:19:46] 10serviceops, 10SRE, 10User-jijiki: Enable TLS on memcached - https://phabricator.wikimedia.org/T271967 (10jijiki) [18:19:49] 10serviceops, 10SRE, 10Patch-For-Review: Upgrade and improve our application object caching service (memcached) - https://phabricator.wikimedia.org/T244852 (10jijiki) [18:20:19] 10serviceops, 10SRE, 10User-jijiki: Enable TLS on memcached - https://phabricator.wikimedia.org/T271967 (10jijiki) [18:31:06] 10serviceops, 10SRE, 10Patch-For-Review, 10Release-Engineering-Team (Deployment services), and 2 others: Upgrade MediaWiki clusters to Debian Buster (debian 10) - https://phabricator.wikimedia.org/T245757 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by dzahn on cumin1001.eqiad.wmnet for host... [18:34:30] 10serviceops, 10SRE, 10Patch-For-Review, 10Release-Engineering-Team (Deployment services), and 2 others: Upgrade MediaWiki clusters to Debian Buster (debian 10) - https://phabricator.wikimedia.org/T245757 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['mw2228.codfw.wmnet'] ` Of which those **F... [18:34:45] 10serviceops, 10SRE, 10Traffic, 10HTTPS, 10User-jijiki: Enable TLS on memcached - https://phabricator.wikimedia.org/T271967 (10Nintendofan885) [18:35:12] 10serviceops, 10Prod-Kubernetes, 10Traffic, 10HTTPS, and 2 others: Move termbox to use TLS only - https://phabricator.wikimedia.org/T254581 (10Nintendofan885) [18:36:19] 10serviceops, 10SRE, 10Traffic, 10HTTPS, and 3 others: Move blubberoid to use TLS only. - https://phabricator.wikimedia.org/T236017 (10Nintendofan885) [18:53:54] 10serviceops, 10SRE, 10User-jijiki: Enable TLS on memcached - https://phabricator.wikimedia.org/T271967 (10Aklapper) @Nintendofan885 This is unrelated to #HTTPS [18:57:47] 10serviceops, 10SRE, 10Kubernetes, 10Patch-For-Review, 10Release Pipeline (Blubber): Move blubberoid to use TLS only. - https://phabricator.wikimedia.org/T236017 (10Aklapper) @Nintendofan885 This is unrelated to HTTPS [19:15:01] 10serviceops, 10SRE, 10Patch-For-Review, 10Release-Engineering-Team (Deployment services), and 2 others: Upgrade MediaWiki clusters to Debian Buster (debian 10) - https://phabricator.wikimedia.org/T245757 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by dzahn on cumin1001.eqiad.wmnet for host... [19:20:30] 10serviceops, 10SRE, 10User-jijiki: Enable TLS on memcached - https://phabricator.wikimedia.org/T271967 (10elukey) My 2c: I'd vote for 1.6.x since it is close to what upstream is currently supporting, plus I don't think that it would be less stable than the last 1.5.x version.. In 1.6 a lot of new things wer... [19:26:08] 10serviceops, 10SRE, 10User-jijiki: Enable TLS on memcached - https://phabricator.wikimedia.org/T271967 (10MoritzMuehlenhoff) Also, memcached 1.6.6 is already used on the IDPs and available in a component. [19:27:45] 10serviceops, 10SRE, 10Wikimedia-production-error: PHP7 corruption reports in 2020-2021 (Call on wrong object, etc.) - https://phabricator.wikimedia.org/T245183 (10Krinkle) I'm seeing this several times a week and have for several months. I haven't reported it before since it's not essential prod code and we... [19:34:29] 10serviceops, 10SRE, 10Patch-For-Review, 10Release-Engineering-Team (Deployment services), and 2 others: Upgrade MediaWiki clusters to Debian Buster (debian 10) - https://phabricator.wikimedia.org/T245757 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['mw2231.codfw.wmnet'] ` Of which those **F... [19:35:45] 10serviceops, 10SRE, 10Patch-For-Review, 10Release-Engineering-Team (Deployment services), and 2 others: Upgrade MediaWiki clusters to Debian Buster (debian 10) - https://phabricator.wikimedia.org/T245757 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by dzahn on cumin1001.eqiad.wmnet for host... [19:36:26] 10serviceops, 10SRE, 10Patch-For-Review, 10Release-Engineering-Team (Deployment services), and 2 others: Upgrade MediaWiki clusters to Debian Buster (debian 10) - https://phabricator.wikimedia.org/T245757 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['mw2232.codfw.wmnet'] ` Of which those **F... [19:48:51] 10serviceops, 10SRE, 10Patch-For-Review, 10Release-Engineering-Team (Deployment services), and 2 others: Upgrade MediaWiki clusters to Debian Buster (debian 10) - https://phabricator.wikimedia.org/T245757 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['mw2233.codfw.wmnet'] ` Of which those **F... [20:32:16] 10serviceops, 10SRE, 10Patch-For-Review, 10Release-Engineering-Team (Deployment services), and 2 others: Upgrade MediaWiki clusters to Debian Buster (debian 10) - https://phabricator.wikimedia.org/T245757 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['mw2234.codfw.wmnet'] ` Of which those **F... [20:54:02] 10serviceops, 10SRE, 10Patch-For-Review, 10Release-Engineering-Team (Deployment services), and 2 others: Upgrade MediaWiki clusters to Debian Buster (debian 10) - https://phabricator.wikimedia.org/T245757 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['mw2235.codfw.wmnet'] ` Of which those **F... [21:09:05] 10serviceops, 10SRE, 10Patch-For-Review, 10Release-Engineering-Team (Deployment services), and 2 others: Upgrade MediaWiki clusters to Debian Buster (debian 10) - https://phabricator.wikimedia.org/T245757 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by dzahn on cumin1001.eqiad.wmnet for host... [21:09:35] 10serviceops, 10SRE, 10Patch-For-Review, 10Release-Engineering-Team (Deployment services), and 2 others: Upgrade MediaWiki clusters to Debian Buster (debian 10) - https://phabricator.wikimedia.org/T245757 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by dzahn on cumin1001.eqiad.wmnet for host... [21:10:52] 10serviceops, 10SRE, 10Patch-For-Review, 10Release-Engineering-Team (Deployment services), and 2 others: Upgrade MediaWiki clusters to Debian Buster (debian 10) - https://phabricator.wikimedia.org/T245757 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by dzahn on cumin1001.eqiad.wmnet for host... [21:11:13] 10serviceops, 10SRE, 10Patch-For-Review, 10Release-Engineering-Team (Deployment services), and 2 others: Upgrade MediaWiki clusters to Debian Buster (debian 10) - https://phabricator.wikimedia.org/T245757 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by dzahn on cumin1001.eqiad.wmnet for host... [22:08:37] 10serviceops, 10MW-on-K8s, 10SRE, 10Release Pipeline (Blubber), 10Release-Engineering-Team (Pipeline): Deployment infrastructure for PHP microservices - https://phabricator.wikimedia.org/T261369 (10sbassett) >>! In T261369#6695002, @akosiaris wrote: >> As I understand it, there's a halt on that npm appro... [22:26:25] 10serviceops, 10SRE, 10Patch-For-Review, 10Release-Engineering-Team (Deployment services), and 2 others: Upgrade MediaWiki clusters to Debian Buster (debian 10) - https://phabricator.wikimedia.org/T245757 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['mw2237.codfw.wmnet'] ` Of which those **F... [22:28:45] 10serviceops, 10SRE, 10Patch-For-Review, 10Release-Engineering-Team (Deployment services), and 2 others: Upgrade MediaWiki clusters to Debian Buster (debian 10) - https://phabricator.wikimedia.org/T245757 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['mw2239.codfw.wmnet'] ` Of which those **F... [22:29:39] 10serviceops, 10SRE, 10Patch-For-Review, 10Release-Engineering-Team (Deployment services), and 2 others: Upgrade MediaWiki clusters to Debian Buster (debian 10) - https://phabricator.wikimedia.org/T245757 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['mw2238.codfw.wmnet'] ` Of which those **F... [22:30:13] 10serviceops, 10SRE, 10Patch-For-Review, 10Release-Engineering-Team (Deployment services), and 2 others: Upgrade MediaWiki clusters to Debian Buster (debian 10) - https://phabricator.wikimedia.org/T245757 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['mw2240.codfw.wmnet'] ` Of which those **F... [22:54:54] 10serviceops, 10MW-on-K8s, 10SRE, 10Release Pipeline (Blubber), 10Release-Engineering-Team (Pipeline): Deployment infrastructure for PHP microservices - https://phabricator.wikimedia.org/T261369 (10bd808) >>! In T261369#6746098, @sbassett wrote: > That being said, npm install shouldn't be run on any prod...