[06:52:43] <_joe_>	 mutante: \o/
[10:38:19] <_joe_>	 fsero: registry1001 has a systemd alert
[10:38:23] <_joe_>	 can you check please?
[10:40:39] <jynus>	 he is afk, said in another channel
[10:44:02] <fsero>	 _joe_ yep, I'll downtime it is because it's trying to read from swift on eqiad
[10:44:28] <fsero>	 After I come back in some mins
[10:44:40] <_joe_>	 yeah no rush
[10:46:17] <wikibugs>	 10serviceops, 10MediaWiki-Cache, 10Operations, 10Patch-For-Review, and 3 others: Apply -R 200 to all the memcached mw object cache instances running in eqiad/codfw - https://phabricator.wikimedia.org/T208844 (10jijiki)
[11:22:48] <jijiki>	 anyone knows why we have a spike, I believe every day at ~9:50 UTC on our api backends?
[11:22:53] <jijiki>	 https://grafana.wikimedia.org/d/000000002/api-backend-summary?refresh=5m&orgId=1&from=now-2d&to=now
[11:23:12] <jijiki>	 looks like a job obviously
[11:23:30] <jynus>	 I mean, it is api, it is the jungle out there!
[11:23:43] <jynus>	 or do you think it is internal?
[11:24:06] <jijiki>	 it could be internal yes
[11:24:19] <jynus>	 oh, I see it is parsoid
[11:24:20] <jijiki>	 I don't know what to assume reall 
[11:24:45] <jynus>	 ignore my comment, I tought it was the external api
[11:25:03] <jynus>	 (which sometimes has small spikes of requests)
[11:33:55] <_joe_>	 jijiki: you can go look at api.log on mwlog1002 (IIRC)
[11:34:11] <_joe_>	 around the time of the spike
[11:34:16] <_joe_>	 and sample IPs/UAs
[11:34:43] <_joe_>	 or just the apache log (other_vhost_access.log) on a random api server, too
[11:35:08] <_joe_>	 but yes, it's parsoid-batch
[11:35:37] <_joe_>	 I would bet it's some job that updates some template used everywhere
[11:36:05] <_joe_>	 jijiki: so next step is looking at what calls parsoid at that time, I guess changeprop but better to check that
[11:39:21] <wikibugs>	 10serviceops, 10MediaWiki-Cache, 10Operations, 10Patch-For-Review, and 3 others: Apply -R 200 to all the memcached mw object cache instances running in eqiad/codfw - https://phabricator.wikimedia.org/T208844 (10jijiki)
[11:39:45] <jijiki>	 _joe_: ok 
[11:52:38] <wikibugs>	 10serviceops, 10Operations, 10Performance-Team (Radar), 10User-Elukey, 10User-jijiki: Upgrade memcached for Debian Stretch/Buster - https://phabricator.wikimedia.org/T213089 (10jijiki)
[13:51:04] * jijiki errand
[14:57:41] <apergos>	 reminder that kube v3 is coming up n a couple minutes
[14:57:48] <apergos>	 (I always forget and am late)
[15:01:13] <apergos>	 reminder that kubernetes vol 3 talk is starting any moment now (see your calendar for hangout link)
[15:01:21] <apergos>	 woops, wrong window grrrr