[00:16:40] 10serviceops, 10Gerrit, 10Release-Engineering-Team (Development services), 10Release-Engineering-Team-TODO (2020-01 to 2020-03 (Q3)): Add second virtual hard disk to ganeti gerrit test instance - https://phabricator.wikimedia.org/T243983 (10Dzahn) VM is back up now and has an additional 10GB drive (/dev/vd... [00:16:45] 10serviceops, 10Gerrit, 10Release-Engineering-Team (Development services), 10Release-Engineering-Team-TODO (2020-01 to 2020-03 (Q3)): Add second virtual hard disk to ganeti gerrit test instance - https://phabricator.wikimedia.org/T243983 (10Dzahn) 05Open→03Resolved [00:26:10] 10serviceops, 10Core Platform Team Workboards (Clinic Duty Team): restrouter.svc.{eqiad,codfw}.wmnet in a failed state - https://phabricator.wikimedia.org/T242461 (10Eevans) I believe we have consensus around de-deploying restrouter from k8s, @WDoranWMF can you confirm? [04:16:05] 10serviceops, 10User-brennen, 10Wikimedia-production-error: Opcache hit ratio dropped after 22/1 train on appeservers - https://phabricator.wikimedia.org/T243601 (10brennen) p:05Unbreak!→03Triage We're definitely not treating this as a true blocker, given that wmf.16 is fully deployed at this point. Aft... [04:16:07] 10serviceops, 10User-brennen, 10Wikimedia-production-error: Opcache hit ratio dropped after 22/1 train on appeservers - https://phabricator.wikimedia.org/T243601 (10brennen) [07:53:53] 10serviceops, 10Operations, 10ops-eqiad: (Need By Dec 20) rack/setup/install mw13[49-84].eqiad.wmnet - https://phabricator.wikimedia.org/T236437 (10Joe) p:05Normal→03High Can we please expedite this? We really need these servers to join rotation. [08:08:03] 10serviceops, 10Core Platform Team, 10MediaWiki-General, 10Operations: siteinfo api calls should be cached for N minutes on the caching layer - https://phabricator.wikimedia.org/T244204 (10Joe) p:05Triage→03High [10:16:56] 10serviceops, 10Core Platform Team, 10MediaWiki-General, 10Operations: siteinfo api calls should be cached for N minutes on the caching layer - https://phabricator.wikimedia.org/T244204 (10Schnark) I see a `"time": ""` in the output by `siteinfo`. I don't know if and how anyone uses that... [11:47:00] 10serviceops, 10User-brennen, 10Wikimedia-production-error: Opcache hit ratio dropped after 22/1 train on appeservers - https://phabricator.wikimedia.org/T243601 (10jijiki) 05Open→03Resolved a:03jijiki @greg we have not observed this issue before, apart from when we restart a php-fpm instance, so I do... [11:49:22] 10serviceops, 10Core Platform Team, 10MediaWiki-Cache, 10Performance-Team (Radar): Ensure apcu incr/decr are atomic (Upgrade php-apcu) - https://phabricator.wikimedia.org/T236800 (10jijiki) @Krinkle we can push the new packaged to the canaries this week if you are ok with it [12:00:37] 10serviceops, 10Operations, 10Performance-Team, 10Wikimedia-production-error: Page takes over 15s to load: https://en.wikipedia.org/w/index.php?title=European_Union&type=revision&diff=938561921&oldid=938557616 - https://phabricator.wikimedia.org/T244058 (10jijiki) [12:12:46] 10serviceops, 10Operations, 10Performance-Team, 10Wikimedia-production-error: Wiki diffs take over 15s to load - https://phabricator.wikimedia.org/T244058 (10jcrespo) [12:46:08] 10serviceops, 10User-brennen, 10Wikimedia-production-error: Opcache hit ratio dropped after 2020-01-22 train on appeservers - https://phabricator.wikimedia.org/T243601 (10Aklapper) [13:47:13] akosiaris: helloooooOoO if/when you are back and human again after traveling, let's settle the service.name+canary stuff eh!? https://phabricator.wikimedia.org/T242861#5819686 [13:48:23] back I am indeed. The human part... that will take a while [13:48:40] but yeah, let's wrap this up this week [13:51:43] ottomata: that sounded almost like 'let's settle the score' [13:52:25] akosiaris: I think he dares you on a bike polo duel [13:52:55] ha heck no my lateral epicondylitis would be a severe handicap for me [13:53:10] and I would still be crashed regardless [13:54:57] * effie puts the popcorn down [13:55:07] also a q for you about eventstreams benchmarking at https://phabricator.wikimedia.org/T238658#5830499 [13:59:21] 10serviceops, 10Operations: decom debug proxies (was: Migrate debug proxies to Stretch/Buster) - https://phabricator.wikimedia.org/T224567 (10ema) [15:08:36] 10serviceops, 10Operations, 10Performance-Team, 10Wikimedia-production-error: Wiki diffs take over 15s to load - https://phabricator.wikimedia.org/T244058 (10thcipriani) > Both are running 1.35.0-wmf.16 at the moment. When this task was filled on 2020-02-01 the wikipedia wikis were on 1.35.0-wmf.15 due to... [15:09:37] so after this week I won't be able to do our usual Thursday slot for service ops meeting anymore, for a while :( [15:09:40] annual planning meetings [15:09:50] so I propose to move it to Wednesday instead, same time for now [15:09:53] would that work for everyone? [15:10:25] it's ok for me [15:10:43] lgtm [15:14:43] <_joe_> not sure it does for me [15:15:13] <_joe_> I have sometimes techcom at the same time - once a month or so [15:15:38] <_joe_> I can skip that meeting once a month in case [15:43:50] 10serviceops, 10Operations, 10Performance-Team (Radar): decom debug proxies (was: Migrate debug proxies to Stretch/Buster) - https://phabricator.wikimedia.org/T224567 (10Krinkle) [15:44:14] 10serviceops, 10Operations, 10Performance-Team (Radar): decom debug proxies (was: Migrate debug proxies to Stretch/Buster) - https://phabricator.wikimedia.org/T224567 (10Krinkle) [15:44:29] 10serviceops, 10Operations, 10Performance-Team (Radar): decom debug proxies (was: Migrate debug proxies to Stretch/Buster) - https://phabricator.wikimedia.org/T224567 (10Krinkle) Thanks! Less is more :) [16:33:34] 10serviceops, 10Operations, 10Performance-Team, 10Wikimedia-production-error: Wiki diffs take over 15s to load - https://phabricator.wikimedia.org/T244058 (10RhinosF1) Noticing alot of slow wikis + report of downtime on mediawiki.org discord - Both pages get a wikimedia timeout mentioned above. [16:35:12] 10serviceops, 10Operations, 10Performance-Team, 10Wikimedia-production-error: Wiki diffs take over 15s to load - https://phabricator.wikimedia.org/T244058 (10Marostegui) >>! In T244058#5848681, @RhinosF1 wrote: > Noticing alot of slow wikis + report of downtime on mediawiki.org discord - Both pages get a w... [16:36:55] 10serviceops, 10Operations, 10Performance-Team, 10Wikimedia-production-error: Wiki diffs take over 15s to load - https://phabricator.wikimedia.org/T244058 (10RhinosF1) >>! In T244058#5848683, @Marostegui wrote: >>>! In T244058#5848681, @RhinosF1 wrote: >> Noticing alot of slow wikis + report of downtime on... [16:37:26] 10serviceops, 10Operations, 10Performance-Team, 10Wikimedia-production-error: Wiki diffs take over 15s to load - https://phabricator.wikimedia.org/T244058 (10Marostegui) >>! In T244058#5848686, @RhinosF1 wrote: >>>! In T244058#5848683, @Marostegui wrote: >>>>! In T244058#5848681, @RhinosF1 wrote: >>> Notic... [17:24:49] 10serviceops, 10Core Platform Team, 10MediaWiki-General, 10Operations: siteinfo api calls should be cached for N minutes on the caching layer - https://phabricator.wikimedia.org/T244204 (10Anomie) Caching in the caching layer might have been helpful for these specific requests, but would that merely have d... [17:30:59] 10serviceops, 10Operations, 10ops-eqiad: (Need By Dec 20) rack/setup/install mw13[49-84].eqiad.wmnet - https://phabricator.wikimedia.org/T236437 (10Cmjohnson) @joe working on these next...will have to you in next day or 2. [19:33:31] 10serviceops, 10Operations, 10ops-codfw: rack/setup/install new codfw mw systems - https://phabricator.wikimedia.org/T241852 (10Papaul) [19:38:36] 10serviceops, 10Operations, 10Performance-Team, 10Wikimedia-production-error: Wiki diffs take over 15s to load - https://phabricator.wikimedia.org/T244058 (10aaron) Links to old (non-current) versions due not use the parser cache. This means that rendering will always require a full parse. Profiling info... [20:09:53] 10serviceops, 10Core Platform Team, 10MediaWiki-Cache, 10Performance-Team (Radar): Ensure apcu incr/decr are atomic (Upgrade php-apcu) - https://phabricator.wikimedia.org/T236800 (10aaron) >>! In T236800#5847650, @jijiki wrote: > @Krinkle we can push the new package to the canaries this week if you are ok... [21:08:43] 10serviceops, 10Operations, 10Performance-Team, 10Wikimedia-production-error: Wiki diffs take over 15s to load - https://phabricator.wikimedia.org/T244058 (10Daimona) I forgot to say that the second example in the task description is unrelated. It was discussed with ops earlier today, and my comment can be... [22:56:06] 10serviceops, 10Performance-Team, 10Release-Engineering-Team: Create warmup procedure for MediaWiki app servers - https://phabricator.wikimedia.org/T230037 (10Krinkle) During the first three switch overs the impact and gains was imho quite clearly proven given that on a cold server (at the time HHVM), latenc...