[07:25:18] 10serviceops, 10Prod-Kubernetes, 10Kubernetes, 10Patch-For-Review: Make helm upgrades atomic - https://phabricator.wikimedia.org/T252428 (10JMeybohm) [07:25:31] 10serviceops, 10Operations, 10Prod-Kubernetes, 10Kubernetes: Add TLS termination to services running on kubernetes - https://phabricator.wikimedia.org/T235411 (10JMeybohm) [07:44:44] 10serviceops, 10Operations, 10Continuous-Integration-Config: docker-reporter-releng-images failed on deneb - https://phabricator.wikimedia.org/T251918 (10ayounsi) ACKing the alert https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=deneb&service=Check+systemd+state with this task. ` ayounsi@d... [08:13:18] 10serviceops, 10Prod-Kubernetes, 10Kubernetes: Upgrade all TLS enabled charts to v0.2 tls_helper - https://phabricator.wikimedia.org/T253396 (10JMeybohm) [09:02:50] 10serviceops, 10Operations, 10Patch-For-Review: decom people1001 - https://phabricator.wikimedia.org/T253296 (10ops-monitoring-bot) cookbooks.sre.hosts.decommission executed by dzahn@cumin1001 for hosts: `people1001.eqiad.wmnet` - people1001.eqiad.wmnet (**PASS**) - Downtimed host on Icinga - Found Gane... [09:31:02] 10serviceops, 10Operations, 10Patch-For-Review: decom people1001 - https://phabricator.wikimedia.org/T253296 (10Dzahn) 05Open→03Resolved [09:31:05] 10serviceops, 10Operations: upgrade people.wikimedia.org backend to buster - https://phabricator.wikimedia.org/T247649 (10Dzahn) [09:49:03] 10serviceops, 10Operations: docker-reporter-releng-images failed on deneb - https://phabricator.wikimedia.org/T251918 (10hashar) [11:40:06] _joe_: I assume it's mediawiki itself that generates updateBetaFeaturesUserCounts events in kafka, right? [12:21:10] <_joe_> hnowlan: yes [12:22:46] ah, cool. I don't know enough about this to say with any authority but I'm not certain that events are being generated for the topic atm. I haven't seen any events on the topics in kafka in a few hours and the numbers didn't update on my profile overnight afaict [12:25:34] Petr's on later hopefully, I'll check with him then [12:29:34] <_joe_> I don't remember when they are generated tbh [12:29:41] <_joe_> you'd need to read the code [12:39:24] yeah [12:57:02] 10serviceops, 10Operations, 10Patch-For-Review: No mw canary servers in codfw - https://phabricator.wikimedia.org/T242606 (10Dzahn) mw2187, mw2188 are new canary appservers, replacing mw2271, mw2272 mw2249, mw2250 are new jobrunner canaries that we did not have in codfw. Now we have 13 canaries in eqiad an... [12:57:16] 10serviceops, 10Operations, 10Patch-For-Review: No mw canary servers in codfw - https://phabricator.wikimedia.org/T242606 (10Dzahn) 05Open→03Resolved [14:31:49] 10serviceops, 10Prod-Kubernetes, 10Kubernetes, 10Patch-For-Review: Make helm upgrades atomic - https://phabricator.wikimedia.org/T252428 (10JMeybohm) [15:10:15] 10serviceops, 10Operations, 10Security-Team, 10vm-requests, 10PM: Eqiad: 1VM request for Peek (PM service in use by Security Team) - https://phabricator.wikimedia.org/T252210 (10chasemp) @MoritzMuehlenhoff could you revisit this when you have a minute? I'd like to get this off my plate and @wiki_willy a... [15:20:39] 10serviceops, 10Operations, 10Security-Team, 10vm-requests, 10PM: Eqiad: 1VM request for Peek (PM service in use by Security Team) - https://phabricator.wikimedia.org/T252210 (10MoritzMuehlenhoff) @chasemp I think it's a little overblown, but if it helps unblocking existing tests, feel free go ahead. Our... [15:25:23] 10serviceops, 10Operations, 10Traffic, 10Patch-For-Review: Certificate *.wikipedia.org valid until 2020-06-20 - https://phabricator.wikimedia.org/T251726 (10Dzahn) >>! In T251726#6109023, @Vgutierrez wrote: > IMHO 7 / 3 is not enough for the unified cert even when LE is the issuer considering our anti cloc... [15:27:07] 10serviceops, 10Operations, 10Security-Team, 10vm-requests, 10PM: Eqiad: 1VM request for Peek (PM service in use by Security Team) - https://phabricator.wikimedia.org/T252210 (10chasemp) >>! In T252210#6165524, @MoritzMuehlenhoff wrote: > @chasemp I think it's a little overblown, but if it helps unblocki... [15:41:31] 10serviceops, 10Prod-Kubernetes, 10Kubernetes, 10Patch-For-Review: Upgrade all TLS enabled charts to v0.2 tls_helper - https://phabricator.wikimedia.org/T253396 (10JMeybohm) I've used something like this to update the charts: `lang=bash function update_helper() { CHART=$(basename $1) git checkout... [18:21:42] 10serviceops, 10Performance-Team (Radar): Avoid php-opcache corruption in WMF production - https://phabricator.wikimedia.org/T253673 (10Krinkle) [18:29:01] 10serviceops, 10Performance-Team (Radar): Avoid php-opcache corruption in WMF production - https://phabricator.wikimedia.org/T253673 (10Krinkle) > 1) Do a restart for all deploys. Take the hit on deploy time and/or focus on ways to reduce it. The current estimate for the Scap rolling restart is 15 minutes. Is... [18:29:41] 10serviceops, 10Performance-Team (Radar): Avoid php-opcache corruption in WMF production - https://phabricator.wikimedia.org/T253673 (10Krinkle) [18:29:49] 10serviceops, 10Performance-Team (Radar): Avoid php-opcache corruption in WMF production - https://phabricator.wikimedia.org/T253673 (10Krinkle) [19:04:20] 10serviceops, 10Performance-Team (Radar): Remove mod_unique_id from app servers - https://phabricator.wikimedia.org/T253675 (10Krinkle) [19:04:41] 10serviceops, 10Performance-Team (Radar): Remove mod_unique_id from app servers - https://phabricator.wikimedia.org/T253675 (10Krinkle) p:05Triage→03Low [19:13:10] 10serviceops, 10Performance-Team (Radar): Remove mod_unique_id from app servers - https://phabricator.wikimedia.org/T253675 (10Krinkle) [19:47:41] 10serviceops, 10Performance-Team (Radar): Avoid php-opcache corruption in WMF production - https://phabricator.wikimedia.org/T253673 (10thcipriani) >>! In T253673#6166500, @Krinkle wrote: >> 1) Do a restart for all deploys. Take the hit on deploy time and/or focus on ways to reduce it. I would like to do this...