[04:49:23] 10serviceops, 10Operations, 10PHP 7.2 support, 10Performance-Team (Radar), 10Wikimedia-production-error: PHP 7 corruption during deployment (was: PHP 7 fatals on mw1262) - https://phabricator.wikimedia.org/T224491 (10Joe) So, I am disabling the opcache resets for now, and putting timestamp validation to... [09:48:33] 10serviceops, 10ORES, 10Operations, 10Scoring-platform-team: Migrate ORES Redis servers to Stretch/Buster - https://phabricator.wikimedia.org/T224569 (10akosiaris) We might be able to get away with reusing the redis misc servers (rdb1005/rdb1009). That should give us more memory and allow us to use the emp... [10:08:58] 10serviceops, 10MediaWiki-Cache, 10Operations, 10Patch-For-Review, and 2 others: Apply -R 200 to all the memcached mw object cache instances running in eqiad/codfw - https://phabricator.wikimedia.org/T208844 (10elukey) [10:21:46] 10serviceops, 10Operations, 10Release Pipeline, 10Release-Engineering-Team, and 5 others: Introduce kask session storage service to kubernetes - https://phabricator.wikimedia.org/T220401 (10akosiaris) One thing that I just met is that kask stops accepting HTTP connections if kask cert/key pair is configure... [10:47:21] 10serviceops, 10Operations, 10PHP 7.2 support, 10Performance-Team (Radar), 10Wikimedia-production-error: PHP 7 corruption during deployment (was: PHP 7 fatals on mw1262) - https://phabricator.wikimedia.org/T224491 (10Joe) FWIW - the percentage of errors coming from php-fpm servers was including timeouts,... [13:35:04] <_joe_> ok, by my very professional and absolutely reliable measurements on the api.log data [13:35:22] <_joe_> php is on average 1 ms faster than hhvm on equally busy servers [14:44:08] sorry I was late, it was the security thing, I totally lost track of the hour [14:44:42] do hmu if you want tendril slides (depends what exactly you need, I have good coverage for some things but I don't know what angle you're coming at it from) [15:02:58] apergos: can you email me links or the slides themselves? [15:03:01] 10serviceops, 10Operations, 10HHVM, 10Performance-Team (Radar), 10User-Marostegui: Increased instability in MediaWiki backends (according to load balancers) - https://phabricator.wikimedia.org/T223952 (10Marostegui) [15:03:12] I can give you a link but [15:09:14] 10serviceops, 10Operations, 10Release Pipeline, 10Release-Engineering-Team, and 5 others: Introduce kask session storage service to kubernetes - https://phabricator.wikimedia.org/T220401 (10Eevans) >>! In T220401#5225728, @akosiaris wrote: > One thing that I just met is that kask stops accepting HTTP conne... [15:09:59] (done off channel) [15:18:01] 10serviceops, 10Operations, 10Release Pipeline, 10Release-Engineering-Team, and 5 others: Introduce kask session storage service to kubernetes - https://phabricator.wikimedia.org/T220401 (10fgiunchedi) >>! In T220401#5226287, @Eevans wrote: >> 1. We use an Exec probe that executes something like `curl http... [15:36:20] 10serviceops, 10Operations, 10Release Pipeline, 10Release-Engineering-Team, and 5 others: Introduce kask session storage service to kubernetes - https://phabricator.wikimedia.org/T220401 (10Eevans) >>! In T220401#5226287, @Eevans wrote: >>>! In T220401#5225728, @akosiaris wrote: >> One thing that I just me... [15:37:26] 10serviceops, 10Operations, 10Release Pipeline, 10Release-Engineering-Team, and 5 others: Introduce kask session storage service to kubernetes - https://phabricator.wikimedia.org/T220401 (10akosiaris) >>! In T220401#5226287, @Eevans wrote: >>>! In T220401#5225728, @akosiaris wrote: >> One thing that I just... [16:25:20] 10serviceops, 10Operations, 10Release Pipeline, 10Release-Engineering-Team, and 5 others: Introduce kask session storage service to kubernetes - https://phabricator.wikimedia.org/T220401 (10akosiaris) > And this uncovered now that prometheus can't talk to it (cause it expects HTTP I guess?). /me looking in... [16:50:18] 10serviceops, 10Operations, 10Release Pipeline, 10Release-Engineering-Team, and 5 others: Introduce kask session storage service to kubernetes - https://phabricator.wikimedia.org/T220401 (10akosiaris) One minor question. Given per T220401#5128786 1 kask instance is able to handle ~300req/s, how many instan... [17:26:13] 10serviceops, 10Operations, 10Release Pipeline, 10Release-Engineering-Team, and 5 others: Introduce kask session storage service to kubernetes - https://phabricator.wikimedia.org/T220401 (10Eevans) >>! In T220401#5226531, @akosiaris wrote: > One minor question. Given per T220401#5128786 1 kask instance is... [20:01:36] 10serviceops, 10Gerrit, 10Release-Engineering-Team: Gerrit http threads stuck behind sendemail thread - https://phabricator.wikimedia.org/T224448 (10Paladox) I think https://groups.google.com/forum/#!topic/repo-discuss/0yej8sQDcPo may be related. Also im leaning towards (based on the user reply in that threa... [20:17:21] 10serviceops, 10Gerrit, 10Release-Engineering-Team: Gerrit http threads stuck behind sendemail thread - https://phabricator.wikimedia.org/T224448 (10Paladox) I can reproduce this locally now (after adding the web cache to my config locally and set memory to 10, the threads sky rocketed to over 140)