[01:18:40] 10serviceops, 10Operations, 10ops-eqiad: (Need By Dec 20) rack/setup/install mw13[49-84].eqiad.wmnet - https://phabricator.wikimedia.org/T236437 (10Papaul) @Jclark-ctr Please see below for available mgmt IP's that you can use for those servers. Once you have the asset tags please update the table with the... [09:05:13] 10serviceops, 10Operations: Reimage all mediawiki servers - https://phabricator.wikimedia.org/T239054 (10jijiki) [09:08:44] 10serviceops, 10Operations, 10HHVM, 10MW-1.35-notes (1.35.0-wmf.3; 2019-10-22), 10Performance-Team (Radar): Remove HHVM from production - https://phabricator.wikimedia.org/T229792 (10jijiki) [09:20:45] 10serviceops, 10Operations, 10Packaging: Build and upload envoy 1.12.0 package. - https://phabricator.wikimedia.org/T237235 (10Joe) 05Open→03Resolved [09:21:21] 10serviceops, 10Operations, 10Kubernetes: Add TLS termination to services running on kubernetes - https://phabricator.wikimedia.org/T235411 (10Joe) [11:40:36] 10serviceops, 10Release-Engineering-Team: switch prod Phabricator from phab1003 to phab1001 - https://phabricator.wikimedia.org/T238956 (10Lucas_Werkmeister_WMDE) >>! In T238956#5716923, @mmodell wrote: > I see the "related gerrit patches" box on this very page. Yup, it’s back now. Thanks! [14:08:32] 10serviceops, 10Services: mobileapps/aqs/recommendation-api (nodejs services) improve resilience against short network outages - https://phabricator.wikimedia.org/T233660 (10BBlack) [14:09:05] 10serviceops, 10DNS, 10Operations, 10Traffic, and 2 others: icinga alerts on nodejs services when a recdns server is depooled - https://phabricator.wikimedia.org/T162818 (10BBlack) [14:11:46] 10serviceops, 10DNS, 10Operations, 10Traffic, and 2 others: nodejs / restbase services (mobileapps, aqs, recommendation-api, etc?) fail persistently after short windows of DNS unavailability - https://phabricator.wikimedia.org/T162818 (10BBlack) [14:13:30] 10serviceops, 10DNS, 10Operations, 10Traffic, and 2 others: nodejs / restbase services (mobileapps, aqs, recommendation-api, etc?) fail persistently after short windows of DNS unavailability - https://phabricator.wikimedia.org/T162818 (10BBlack) While we'll work on improvements that make this less-likely i... [20:37:39] 10serviceops, 10Phabricator, 10Documentation, 10Release-Engineering-Team (Development services), and 2 others: Make PHD run on the backup phabricator server (phab2001, currently) - https://phabricator.wikimedia.org/T232883 (10Dzahn) @20after4 per recent talks we had.. is it still our goal? [21:15:31] 10serviceops, 10Release-Engineering-Team: switch prod Phabricator from phab1003 to phab1001 - https://phabricator.wikimedia.org/T238956 (10Volans) 05Resolved→03Open I've noticed that Phabricator emails are failing the SPF check, re-opening to add details, feel free to move it to a separate task if needed.... [21:33:30] 10serviceops, 10Release-Engineering-Team, 10Patch-For-Review: switch prod Phabricator from phab1003 to phab1001 - https://phabricator.wikimedia.org/T238956 (10Dzahn) @Volans Thanks for reporting! The SPF record has been updated now in DNS. This should go away now. [21:36:49] 10serviceops, 10Core Platform Team, 10Growth-Team, 10MediaWiki-Change-tagging, and 3 others: Change Tag increased database contention (deadlocks) by the jobqueue (InjectRCRecordsJob) - https://phabricator.wikimedia.org/T232081 (10Umherirrender) Deletion on `change_tag` is not done by the primary key in `Ch... [21:58:11] 10serviceops, 10Parsoid-PHP: Craft a deployment strategy to transition Parsoid/PHP from a faux extension to a composer library without breaking incoming requests - https://phabricator.wikimedia.org/T240055 (10ssastry) p:05Triage→03Normal [22:14:32] 10serviceops, 10Parsoid-PHP: Craft a deployment strategy to transition Parsoid/PHP from a faux extension to a composer library without breaking incoming requests - https://phabricator.wikimedia.org/T240055 (10Jdforrester-WMF) "Just" have the version of MW-core where the library is introduced hard-code the poin... [22:15:30] 10serviceops, 10DC-Ops, 10Operations: mw1252 - Memory correctable errors -EDAC- - https://phabricator.wikimedia.org/T236190 (10Dzahn) 05Open→03Resolved a:03Dzahn I don't know why but the alert in Icinga has cleared since 14 days.