[05:03:28] <wikibugs>	 10serviceops, 10DBA, 10Operations, 10Wikidata: Hourly read spikes against s8 resulting in occasional user-visible latency & error spikes - https://phabricator.wikimedia.org/T264821 (10Marostegui) Adding @Addshore and @Ladsgroup as they have lots of cool dashboards (that I am unable to find) where they mayb...
[08:07:25] <wikibugs>	 10serviceops, 10DBA, 10Operations, 10Wikidata: Hourly read spikes against s8 resulting in occasional user-visible latency & error spikes - https://phabricator.wikimedia.org/T264821 (10Marostegui) I forgot to paste: ` root@mwmaint2001:~# crontab -l -uwww-data | grep -w wikidatawiki */3 * * * * echo "$$: Sta...
[08:16:57] <wikibugs>	 10serviceops, 10DBA, 10Operations, 10Wikidata: Hourly read spikes against s8 resulting in occasional user-visible latency & error spikes - https://phabricator.wikimedia.org/T264821 (10Marostegui) The only common query I have found that executes on all hosts is: ` SELECT /* Wikibase\Lib\Store\Sql\Terms\Data...
[11:37:46] <wikibugs>	 10serviceops, 10DBA, 10Operations, 10Wikidata: Hourly read spikes against s8 resulting in occasional user-visible latency & error spikes - https://phabricator.wikimedia.org/T264821 (10Addshore) One of the places to look that should track all access to this term related storage is https://grafana.wikimedia....
[12:19:31] <wikibugs>	 10serviceops, 10DBA, 10Operations, 10Wikidata: Hourly read spikes against s8 resulting in occasional user-visible latency & error spikes - https://phabricator.wikimedia.org/T264821 (10Addshore) Flagging {T263999} up here as it could be related (not a currently prooven link)
[12:28:09] <wikibugs>	 10serviceops, 10DBA, 10Operations, 10Wikidata: Hourly read spikes against s8 resulting in occasional user-visible latency & error spikes - https://phabricator.wikimedia.org/T264821 (10Addshore) ` 1:23 PM <addshore> I also notice this? https://grafana.wikimedia.org/d/000000202/api-frontend-summary?orgId=1&f...
[12:45:04] <wikibugs>	 10serviceops, 10DBA, 10Operations, 10Wikidata: Hourly read spikes against s8 resulting in occasional user-visible latency & error spikes - https://phabricator.wikimedia.org/T264821 (10Ladsgroup) An update from IRC, it seems the culprit is wikifeeds: https://grafana.wikimedia.org/d/b1jttnFMz/envoy-telemetry...
[12:56:41] <wikibugs>	 10serviceops, 10Kubernetes, 10Patch-For-Review: Support TLS for service-to-service communication in k8s staging - https://phabricator.wikimedia.org/T260917 (10akosiaris) >>! In T260917#6518755, @JMeybohm wrote: > I could use a pair of eyes on https://gerrit.wikimedia.org/r/q/bug:T260917 > The PCC full diff (...
[13:01:38] <wikibugs>	 10serviceops, 10Kubernetes, 10Patch-For-Review: Support TLS for service-to-service communication in k8s staging - https://phabricator.wikimedia.org/T260917 (10JMeybohm) >>! In T260917#6525081, @akosiaris wrote: > What's peculiar, is that https://puppet-compiler.wmflabs.org/compiler1002/25621/deploy1001.eqiad...
[13:03:15] <wikibugs>	 10serviceops, 10DBA, 10Operations, 10Wikidata: Hourly read spikes against s8 resulting in occasional user-visible latency & error spikes - https://phabricator.wikimedia.org/T264821 (10Addshore) ` 1:57 PM <addshore> While i was browsing around I also saw spikes in action=query in grafana, but couldn't dive...
[13:04:08] <wikibugs>	 10serviceops, 10Kubernetes, 10Patch-For-Review: Support TLS for service-to-service communication in k8s staging - https://phabricator.wikimedia.org/T260917 (10akosiaris) >>! In T260917#6525089, @JMeybohm wrote: >>>! In T260917#6525081, @akosiaris wrote: >> What's peculiar, is that https://puppet-compiler.wmf...
[13:11:48] <wikibugs>	 10serviceops, 10DBA, 10Operations, 10Wikidata: Hourly read spikes against s8 resulting in occasional user-visible latency & error spikes - https://phabricator.wikimedia.org/T264821 (10Joe) To clarify a bit - restbase has hourly spikes of requests for the `feed` endpoint, which go back to wikifeeds, which c...