[07:22:39] 06Machine-Learning-Team, 05Goal, 07OKR-Work: Q1 FY2025-26 Goal: Make article topic data available at scale and within SLOs for Year in Review - https://phabricator.wikimedia.org/T392833#11532357 (10BWojtowicz-WMF) **Small Weekly Update** 1. Change adding `revision_id` support to Article Topics is close to b... [08:04:31] 06Machine-Learning-Team, 10Add-Link-Structured-Task, 06Growth-Team, 13Patch-For-Review: Add a Link: Remove Country and Continent names in suggestions - https://phabricator.wikimedia.org/T414297#11532388 (10OKarakaya-WMF) a:03OKarakaya-WMF [08:22:53] 06Machine-Learning-Team, 10Add-Link-Structured-Task, 06Growth-Team, 13Patch-For-Review: Add a Link: Remove Country and Continent names in suggestions - https://phabricator.wikimedia.org/T414297#11532415 (10OKarakaya-WMF) https://gitlab.wikimedia.org/repos/machine-learning/ml-pipelines/-/merge_requests/90 [12:00:58] 06Machine-Learning-Team, 05Goal, 07OKR-Work: Q1 FY2025-26 Goal: Task generation engine for Revise Tone task - https://phabricator.wikimedia.org/T408341#11533384 (10achou) **Weekly Report** Progress update on the hypothesis for the week, including if something has shipped: - [Post-delivery optimization] Upda... [12:11:47] 06Machine-Learning-Team, 10Add-Link-Structured-Task, 06Growth-Team, 13Patch-For-Review: Add a Link: Remove Country and Continent names in suggestions - https://phabricator.wikimedia.org/T414297#11533436 (10OKarakaya-WMF) I've trained a model without countries and continents for zhwiki. We get similar f1 sc... [12:47:10] hello, I have a small MR https://gitlab.wikimedia.org/repos/data-engineering/airflow-dags/-/merge_requests/1932/diffs related to the task https://phabricator.wikimedia.org/T414297#11533436 . Can you take a look when you're available? @kevinbazira [12:50:25] ozge_: LGTM. approved. [12:50:43] 🙌 [13:31:24] 06Machine-Learning-Team, 10Add-Link-Structured-Task, 06Growth-Team, 13Patch-For-Review: Add a Link: Remove Country and Continent names in suggestions - https://phabricator.wikimedia.org/T414297#11533729 (10OKarakaya-WMF) zhwiki v2 model checksum: `c4796c3c193d983980a445bb2a76f65def9f2459599fa6df055984bd85... [13:47:11] 06Machine-Learning-Team, 07sre-alert-triage: Alert in need of triage: SmartNotHealthy (instance ml-serve1001:9100) - https://phabricator.wikimedia.org/T414969 (10LSobanski) 03NEW [13:51:00] 06Machine-Learning-Team, 07sre-alert-triage: Alert in need of triage: HelmfileAdminNGPendingChanges (instance deploy1003:9100) - https://phabricator.wikimedia.org/T414971 (10LSobanski) 03NEW [14:49:49] FIRING: KubernetesDeploymentUnavailableReplicas: ... [14:49:55] Deployment revertrisk-wikidata-predictor-00002-deployment in revertrisk at codfw has persistently unavailable replicas - https://wikitech.wikimedia.org/wiki/Kubernetes/Troubleshooting#Troubleshooting_a_deployment - https://grafana.wikimedia.org/d/a260da06-259a-4ee4-9540-5cab01a246c8/kubernetes-deployment-details?var-site=codfw&var-cluster=k8s-mlserve&var-namespace=revertrisk&var-deployment=revertrisk-wikidata-predictor-00002-deployment - ... [14:49:55] https://alerts.wikimedia.org/?q=alertname%3DKubernetesDeploymentUnavailableReplicas [14:54:49] FIRING: [2x] KubernetesDeploymentUnavailableReplicas: Deployment revertrisk-wikidata-predictor-00002-deployment in revertrisk at codfw has persistently unavailable replicas - https://wikitech.wikimedia.org/wiki/Kubernetes/Troubleshooting#Troubleshooting_a_deployment - https://alerts.wikimedia.org/?q=alertname%3DKubernetesDeploymentUnavailableReplicas [15:17:35] here's a patch to fix this --^: https://gerrit.wikimedia.org/r/1228527 [15:17:37] whoever has a minute, please have a look [15:39:49] FIRING: [2x] KubernetesDeploymentUnavailableReplicas: Deployment revertrisk-wikidata-predictor-00002-deployment in revertrisk at codfw has persistently unavailable replicas - https://wikitech.wikimedia.org/wiki/Kubernetes/Troubleshooting#Troubleshooting_a_deployment - https://alerts.wikimedia.org/?q=alertname%3DKubernetesDeploymentUnavailableReplicas [15:44:49] RESOLVED: [2x] KubernetesDeploymentUnavailableReplicas: Deployment revertrisk-wikidata-predictor-00002-deployment in revertrisk at codfw has persistently unavailable replicas - https://wikitech.wikimedia.org/wiki/Kubernetes/Troubleshooting#Troubleshooting_a_deployment - https://alerts.wikimedia.org/?q=alertname%3DKubernetesDeploymentUnavailableReplicas [16:09:49] FIRING: KubernetesDeploymentUnavailableReplicas: ... [16:09:49] Deployment revertrisk-wikidata-predictor-00003-deployment in revertrisk at eqiad has persistently unavailable replicas - https://wikitech.wikimedia.org/wiki/Kubernetes/Troubleshooting#Troubleshooting_a_deployment - https://grafana.wikimedia.org/d/a260da06-259a-4ee4-9540-5cab01a246c8/kubernetes-deployment-details?var-site=eqiad&var-cluster=k8s-mlserve&var-namespace=revertrisk&var-deployment=revertrisk-wikidata-predictor-00003-deployment - ... [16:09:49] https://alerts.wikimedia.org/?q=alertname%3DKubernetesDeploymentUnavailableReplicas [16:14:49] FIRING: [2x] KubernetesDeploymentUnavailableReplicas: Deployment revertrisk-wikidata-predictor-00003-deployment in revertrisk at codfw has persistently unavailable replicas - https://wikitech.wikimedia.org/wiki/Kubernetes/Troubleshooting#Troubleshooting_a_deployment - https://alerts.wikimedia.org/?q=alertname%3DKubernetesDeploymentUnavailableReplicas [16:44:46] another patch to fix this --^: https://gerrit.wikimedia.org/r/1228548 [16:44:46] whoever has a minute, please have a look [17:04:49] RESOLVED: [2x] KubernetesDeploymentUnavailableReplicas: Deployment revertrisk-wikidata-predictor-00003-deployment in revertrisk at codfw has persistently unavailable replicas - https://wikitech.wikimedia.org/wiki/Kubernetes/Troubleshooting#Troubleshooting_a_deployment - https://alerts.wikimedia.org/?q=alertname%3DKubernetesDeploymentUnavailableReplicas [17:06:22] \o/