[03:52:03] 06Machine-Learning-Team, 10Add-Link-Structured-Task, 06Growth-Team: Explore retraining add-a-link machine learning model using only higher-quality articles - https://phabricator.wikimedia.org/T415621 (10Sdkb) 03NEW [03:54:54] 06Machine-Learning-Team, 10Add-Link-Structured-Task, 06Growth-Team: Explore retraining add-a-link machine learning model using only higher-quality articles - https://phabricator.wikimedia.org/T415621#11556410 (10Sdkb) [04:27:22] 06Machine-Learning-Team, 10Add-Link-Structured-Task, 06Growth-Team: Reduce tendency of Add a Link to suggest overlinks - https://phabricator.wikimedia.org/T415622 (10Sdkb) 03NEW [04:28:11] 06Machine-Learning-Team, 10Add-Link-Structured-Task, 06Growth-Team: Reduce tendency of Add a Link to suggest overlinks - https://phabricator.wikimedia.org/T415622#11556443 (10Sdkb) [04:28:18] 06Machine-Learning-Team, 10Add-Link-Structured-Task, 06Growth-Team, 13Patch-For-Review: Add a Link: Remove Country and Continent names in suggestions - https://phabricator.wikimedia.org/T414297#11556444 (10Sdkb) [04:44:49] 06Machine-Learning-Team, 10Add-Link-Structured-Task, 06Growth-Team: Explore retraining add-a-link machine learning model using only higher-quality articles - https://phabricator.wikimedia.org/T415621#11556462 (10santhosh) The current link suggestion system has an algorithmic issue that I had pointed in 2025.... [04:58:25] 06Machine-Learning-Team, 10Add-Link-Structured-Task, 06Growth-Team: Explore retraining add-a-link model using first sentence after the lead - https://phabricator.wikimedia.org/T415623 (10Sdkb) 03NEW [05:00:10] 06Machine-Learning-Team, 10Add-Link-Structured-Task, 06Growth-Team: Explore retraining add-a-link model using first sentence after the lead - https://phabricator.wikimedia.org/T415623#11556480 (10Sdkb) Noting here that it might be helpful to check in with folks more familiar with linking on non-English wikis... [05:01:14] 06Machine-Learning-Team, 10Add-Link-Structured-Task, 06Growth-Team: Explore retraining add-a-link machine learning model using only higher-quality articles - https://phabricator.wikimedia.org/T415621#11556482 (10Sdkb) Following up on the challenge point, do we have enough scale with the task that it'd be lik... [06:50:52] 10Lift-Wing, 06Machine-Learning-Team: Update WMF Debian vLLM image to support latest upstream software stack - https://phabricator.wikimedia.org/T415627 (10kevinbazira) 03NEW [06:55:04] 10Lift-Wing, 06Machine-Learning-Team, 10Wikidata, 07OKR-Work: Optimize revertrisk-wikidata inference service to achieve ~500ms latency target - https://phabricator.wikimedia.org/T414060#11556617 (10kevinbazira) a:03kevinbazira [06:57:07] 10Lift-Wing, 06Machine-Learning-Team, 10Wikidata, 07OKR-Work: Optimize revertrisk-wikidata inference service to achieve ~500ms latency target - https://phabricator.wikimedia.org/T414060#11556618 (10kevinbazira) [06:59:06] 10Lift-Wing, 06Machine-Learning-Team, 07Essential-Work: Update WMF Debian vLLM image to support latest upstream software stack - https://phabricator.wikimedia.org/T415627#11556620 (10kevinbazira) [07:42:33] 10Lift-Wing, 06Machine-Learning-Team, 07Essential-Work: Update WMF Debian vLLM image to support latest upstream software stack - https://phabricator.wikimedia.org/T415627#11556643 (10kevinbazira) The Ubuntu vLLM docker image [[ https://github.com/vllm-project/vllm/blob/6c006457123f802d78e0570471ee8ea2d2a87df... [08:56:09] 06Machine-Learning-Team, 13Patch-For-Review: Export retrained Tone-check model to an S3 bucket - https://phabricator.wikimedia.org/T406217#11556765 (10gkyziridis) ==== Update ==== I am facing some difficulties to test the final component: `move_model_to_s3` since the pipeline fails after the `copy_hdfs_to_pvc`... [10:35:06] 06Machine-Learning-Team, 10Add-Link-Structured-Task, 06Growth-Team, 13Patch-For-Review: Add a Link: Remove Country and Continent names in suggestions - https://phabricator.wikimedia.org/T414297#11556968 (10OKarakaya-WMF) Following wikis are deployed to prod and the others are in the queue. I see large wiki... [11:54:57] 10Lift-Wing, 06Machine-Learning-Team: Add LiftWing streams data to event_sanitized (increase data retention) - https://phabricator.wikimedia.org/T405358#11557401 (10kostajh) 05Resolved→03Open @gkyziridis I'm testing this out today but only seeing `revertrisk-language-agnostic` for an example revision on en... [12:27:49] FIRING: KubernetesDeploymentUnavailableReplicas: ... [12:27:55] Deployment revertrisk-wikidata-predictor-00005-deployment in revertrisk at eqiad has persistently unavailable replicas - https://wikitech.wikimedia.org/wiki/Kubernetes/Troubleshooting#Troubleshooting_a_deployment - https://grafana.wikimedia.org/d/a260da06-259a-4ee4-9540-5cab01a246c8/kubernetes-deployment-details?var-site=eqiad&var-cluster=k8s-mlserve&var-namespace=revertrisk&var-deployment=revertrisk-wikidata-predictor-00005-deployment - ... [12:27:55] https://alerts.wikimedia.org/?q=alertname%3DKubernetesDeploymentUnavailableReplicas [13:17:49] RESOLVED: KubernetesDeploymentUnavailableReplicas: ... [13:17:49] Deployment revertrisk-wikidata-predictor-00005-deployment in revertrisk at eqiad has persistently unavailable replicas - https://wikitech.wikimedia.org/wiki/Kubernetes/Troubleshooting#Troubleshooting_a_deployment - https://grafana.wikimedia.org/d/a260da06-259a-4ee4-9540-5cab01a246c8/kubernetes-deployment-details?var-site=eqiad&var-cluster=k8s-mlserve&var-namespace=revertrisk&var-deployment=revertrisk-wikidata-predictor-00005-deployment - ... [13:17:49] https://alerts.wikimedia.org/?q=alertname%3DKubernetesDeploymentUnavailableReplicas [13:43:49] FIRING: KubernetesDeploymentUnavailableReplicas: ... [13:43:49] Deployment reference-need-predictor-00012-deployment in revision-models at eqiad has persistently unavailable replicas - https://wikitech.wikimedia.org/wiki/Kubernetes/Troubleshooting#Troubleshooting_a_deployment - https://grafana.wikimedia.org/d/a260da06-259a-4ee4-9540-5cab01a246c8/kubernetes-deployment-details?var-site=eqiad&var-cluster=k8s-mlserve&var-namespace=revision-models&var-deployment=reference-need-predictor-00012-deployment - ... [13:43:49] https://alerts.wikimedia.org/?q=alertname%3DKubernetesDeploymentUnavailableReplicas [13:48:49] RESOLVED: KubernetesDeploymentUnavailableReplicas: ... [13:48:49] Deployment reference-need-predictor-00012-deployment in revision-models at eqiad has persistently unavailable replicas - https://wikitech.wikimedia.org/wiki/Kubernetes/Troubleshooting#Troubleshooting_a_deployment - https://grafana.wikimedia.org/d/a260da06-259a-4ee4-9540-5cab01a246c8/kubernetes-deployment-details?var-site=eqiad&var-cluster=k8s-mlserve&var-namespace=revision-models&var-deployment=reference-need-predictor-00012-deployment - ... [13:48:49] https://alerts.wikimedia.org/?q=alertname%3DKubernetesDeploymentUnavailableReplicas [16:41:49] FIRING: KubernetesDeploymentUnavailableReplicas: ... [16:41:49] Deployment revertrisk-wikidata-predictor-00005-deployment in revertrisk at eqiad has persistently unavailable replicas - https://wikitech.wikimedia.org/wiki/Kubernetes/Troubleshooting#Troubleshooting_a_deployment - https://grafana.wikimedia.org/d/a260da06-259a-4ee4-9540-5cab01a246c8/kubernetes-deployment-details?var-site=eqiad&var-cluster=k8s-mlserve&var-namespace=revertrisk&var-deployment=revertrisk-wikidata-predictor-00005-deployment - ... [16:41:49] https://alerts.wikimedia.org/?q=alertname%3DKubernetesDeploymentUnavailableReplicas [17:46:49] RESOLVED: KubernetesDeploymentUnavailableReplicas: ... [17:46:49] Deployment revertrisk-wikidata-predictor-00005-deployment in revertrisk at eqiad has persistently unavailable replicas - https://wikitech.wikimedia.org/wiki/Kubernetes/Troubleshooting#Troubleshooting_a_deployment - https://grafana.wikimedia.org/d/a260da06-259a-4ee4-9540-5cab01a246c8/kubernetes-deployment-details?var-site=eqiad&var-cluster=k8s-mlserve&var-namespace=revertrisk&var-deployment=revertrisk-wikidata-predictor-00005-deployment - ... [17:46:49] https://alerts.wikimedia.org/?q=alertname%3DKubernetesDeploymentUnavailableReplicas