[01:21:44] 06serviceops, 13Patch-For-Review: MediaWiki on PHP 8.3 production workload migration - https://phabricator.wikimedia.org/T405955#11361655 (10Scott_French) The next planned migration step will happen on Wednesday the 12th of November, when mw-api-ext and mw-web are fully migrated to PHP 8.3. During the UTC-lat... [08:34:19] 06serviceops: Memcached Cluster Capacity Planning for eqiad - https://phabricator.wikimedia.org/T409181#11362011 (10jijiki) 05Open→03Resolved Yesterday we decided to go for option B, and assess if an expansion will be needed sometime in the next quarters [09:01:45] zabe: can I merge it anytime ? do we have to do anything before/after? [09:10:03] effie: Yes. After merging you could make sure that the failed job (see T409212) is in a healthy state. [09:14:34] 06serviceops, 10CXServer, 10LPL Projects (Other), 07Unplanned-Sprint-Work: cxserver: Remove Yandex MT key from production - https://phabricator.wikimedia.org/T408138#11362071 (10KartikMistry) Can we remove the key? Possibly [[ https://logstash.wikimedia.org/app/discover#/doc/0fade920-6712-11eb-8327-370b46f... [09:19:10] zabe: I am looking at something else as well, the job started failing on the 31st of Oct? [09:19:19] is that correct? [09:20:32] No. It ahould have started failing with last weeks train. [09:22:46] Tue, Nov 4th [09:23:45] 06serviceops, 10MediaWiki-extensions-Score, 10Wikimedia-SVG-rendering, 07Upstream: Deploy Lilypond 2.24 with cairo support to shellbox containers - https://phabricator.wikimedia.org/T385404#11362124 (10akosiaris) >>! In T385404#11356077, @ori wrote: > @akosiaris Do you have any suggestions for getting this... [09:31:51] zabe: I am working on visualising this on grafana, thus my question [10:03:34] 06serviceops, 10MediaWiki-extensions-Score, 10Wikimedia-SVG-rendering, 07Upstream: Deploy Lilypond 2.24 with cairo support to shellbox containers - https://phabricator.wikimedia.org/T385404#11362285 (10MoritzMuehlenhoff) >>! In T385404#11060924, @TasteOfIT wrote: > What is the status of using Lilypond 2.24... [10:27:02] zabe: https://grafana.wikimedia.org/d/f8cb4b3b-8db9-4446-afa7-c293839964b8/jobs-cronjobs-mediawiki-mw-cron [10:28:12] the failed ones should clear up at some point [11:12:56] 06serviceops, 07sre-alert-triage: Alert in need of triage: SystemdUnitFailed (instance registry1005:9100) - https://phabricator.wikimedia.org/T409817 (10LSobanski) 03NEW [11:13:18] 06serviceops, 07sre-alert-triage: Alert in need of triage: SystemdUnitFailed (instance registry1005:9100) - https://phabricator.wikimedia.org/T409817#11362530 (10LSobanski) Same alert is also active for registry2004 and registry2005. [11:20:38] 06serviceops, 07sre-alert-triage: Alert in need of triage: SystemdUnitFailed (instance registry1005:9100) - https://phabricator.wikimedia.org/T409817#11362573 (10MoritzMuehlenhoff) That's for the unused stub service, the actual registries are served by docker_registry::instance. I think replacing the service... [11:29:20] 06serviceops, 06MediaWiki-Platform-Team, 07OKR-Work: api rate limiting: Create a helper service for exposing per-user (per-bucket) metrics - https://phabricator.wikimedia.org/T407999#11362632 (10daniel) [14:02:28] 06serviceops, 06MediaWiki-Platform-Team, 07OKR-Work: api-gateway chart: make emission of x-ratelimit headers configurable - https://phabricator.wikimedia.org/T408839#11363010 (10daniel) 05In progress→03Resolved [14:02:54] 06serviceops, 06MediaWiki-Platform-Team, 07OKR-Work: Test api rate limiting on production cluster - https://phabricator.wikimedia.org/T406498#11363014 (10daniel) [15:48:37] 06serviceops, 06MediaWiki-Platform-Team, 07OKR-Work: Test api rate limiting on production cluster - https://phabricator.wikimedia.org/T406498#11363529 (10daniel) [16:08:52] 06serviceops, 06MediaWiki-Platform-Team, 07OKR-Work: Test api rate limiting on production cluster - https://phabricator.wikimedia.org/T406498#11363565 (10daniel)