[06:23:28] <wikibugs>	 06Machine-Learning-Team, 10Semantic Search: Semantic Search - Embeddings Service for MVP - https://phabricator.wikimedia.org/T412338#11482782 (10kevinbazira) >>! In T412338#11482301, @dcausse wrote: > @kevinbazira @OKarakaya-WMF thanks! is there a way to call this API in a way that is compatible with the [[htt...
[09:23:01] <wikibugs>	 06Machine-Learning-Team, 10Semantic Search: Semantic Search - Embeddings Service for MVP - https://phabricator.wikimedia.org/T412338#11482910 (10dcausse) @kevinbazira thanks, yes this seems like a format that opensearch would be able to work with (P86755 is what is working at the moment, we don't pass the `mod...
[09:32:56] <wikibugs>	 06Machine-Learning-Team, 10Semantic Search: Semantic Search - Embeddings Service for MVP - https://phabricator.wikimedia.org/T412338#11482923 (10dcausse) >>! In T412338#11482782, @kevinbazira wrote: > 2. Add a `usage` object that reports token counts. Is this field a hard requirement for your use case? I don't...
[12:10:49] <jinxer-wm>	 FIRING: KubernetesDeploymentUnavailableReplicas: ...
[12:10:49] <jinxer-wm>	 Deployment revertrisk-wikidata-predictor-00001-deployment in revertrisk at eqiad has persistently unavailable replicas - https://wikitech.wikimedia.org/wiki/Kubernetes/Troubleshooting#Troubleshooting_a_deployment - https://grafana.wikimedia.org/d/a260da06-259a-4ee4-9540-5cab01a246c8/kubernetes-deployment-details?var-site=eqiad&var-cluster=k8s-mlserve&var-namespace=revertrisk&var-deployment=revertrisk-wikidata-predictor-00001-deployment - ...
[12:10:49] <jinxer-wm>	 https://alerts.wikimedia.org/?q=alertname%3DKubernetesDeploymentUnavailableReplicas
[12:55:49] <jinxer-wm>	 RESOLVED: KubernetesDeploymentUnavailableReplicas: ...
[12:55:49] <jinxer-wm>	 Deployment revertrisk-wikidata-predictor-00001-deployment in revertrisk at eqiad has persistently unavailable replicas - https://wikitech.wikimedia.org/wiki/Kubernetes/Troubleshooting#Troubleshooting_a_deployment - https://grafana.wikimedia.org/d/a260da06-259a-4ee4-9540-5cab01a246c8/kubernetes-deployment-details?var-site=eqiad&var-cluster=k8s-mlserve&var-namespace=revertrisk&var-deployment=revertrisk-wikidata-predictor-00001-deployment - ...
[12:55:49] <jinxer-wm>	 https://alerts.wikimedia.org/?q=alertname%3DKubernetesDeploymentUnavailableReplicas
[13:31:49] <jinxer-wm>	 FIRING: KubernetesDeploymentUnavailableReplicas: ...
[13:31:49] <jinxer-wm>	 Deployment revertrisk-wikidata-predictor-00001-deployment in revertrisk at eqiad has persistently unavailable replicas - https://wikitech.wikimedia.org/wiki/Kubernetes/Troubleshooting#Troubleshooting_a_deployment - https://grafana.wikimedia.org/d/a260da06-259a-4ee4-9540-5cab01a246c8/kubernetes-deployment-details?var-site=eqiad&var-cluster=k8s-mlserve&var-namespace=revertrisk&var-deployment=revertrisk-wikidata-predictor-00001-deployment - ...
[13:31:49] <jinxer-wm>	 https://alerts.wikimedia.org/?q=alertname%3DKubernetesDeploymentUnavailableReplicas
[13:47:06] <aiko>	 revertrisk-wikidata... looks like Wikimedia Enterprise is starting their load tests
[13:48:50] <aiko>	 https://grafana.wikimedia.org/d/n3LJdTGIk/kserve-inference-services?orgId=1&var-cluster=aWotKxQMz&var-namespace=revertrisk&var-component=predictor&var-model_name=revertrisk-wikidata&from=now-3h&to=now&timezone=utc
[13:53:37] <aiko>	 https://grafana.wikimedia.org/d/zsdYRV7Vk/istio-sidecar?orgId=1&var-cluster=aWotKxQMz&var-namespace=revertrisk&var-backend=revertrisk-wikidata-predictor-00001-private&var-response_code=$__all&var-quantile=0.5&var-quantile=0.95&var-quantile=0.99&from=now-3h&to=now&timezone=utc
[13:53:42] <aiko>	 https://grafana.wikimedia.org/d/a260da06-259a-4ee4-9540-5cab01a246c8/kubernetes-deployment-details?var-site=eqiad&var-cluster=k8s-mlserve&var-namespace=revertrisk&var-deployment=revertrisk-wikidata-predictor-00001-deployment&orgId=1&from=now-3h&to=now&timezone=utc
[13:58:16] <aiko>	 traffic is around 30 req/s
[13:58:31] <aiko>	  k8s has scaled the model up to the max 15 replicas
[14:02:26] <aiko>	 from the istio dashboard,  latency is ~800ms at p0.5, ~15s at p0.95, and ~25s in p0.99
[14:03:50] <aiko>	 these are for successful requests (200)
[14:21:49] <jinxer-wm>	 RESOLVED: KubernetesDeploymentUnavailableReplicas: ...
[14:21:49] <jinxer-wm>	 Deployment revertrisk-wikidata-predictor-00001-deployment in revertrisk at eqiad has persistently unavailable replicas - https://wikitech.wikimedia.org/wiki/Kubernetes/Troubleshooting#Troubleshooting_a_deployment - https://grafana.wikimedia.org/d/a260da06-259a-4ee4-9540-5cab01a246c8/kubernetes-deployment-details?var-site=eqiad&var-cluster=k8s-mlserve&var-namespace=revertrisk&var-deployment=revertrisk-wikidata-predictor-00001-deployment - ...
[14:21:49] <jinxer-wm>	 https://alerts.wikimedia.org/?q=alertname%3DKubernetesDeploymentUnavailableReplicas
[17:01:07] <wikibugs>	 06Machine-Learning-Team, 06Wikimedia Enterprise (WME Kanban): Test liftwing wikidata revert risk API for scale and latency - https://phabricator.wikimedia.org/T409388#11483570 (10FNavas-foundation) Results from @SGupta-WMF 's test --   In short - success rate is slightly under what we'd want but can live with....