[03:06:15] 10serviceops, 10Continuous-Integration-Config, 10Epic, 10Release-Engineering-Team (Kanban): Define variant Wikimedia production config in compiled, static files - https://phabricator.wikimedia.org/T223602 (10ArielGlenn) @Ladsgroup what sort of issues have you had with yaml in python? How complicated are th... [05:17:58] 10serviceops, 10Operations, 10PHP 7.2 support, 10Performance-Team (Radar), and 2 others: PHP 7 corruption during deployment (was: PHP 7 fatals on mw1262) - https://phabricator.wikimedia.org/T224491 (10Joe) >>! In T224491#5234675, @pmiazga wrote: > @Niedzielski noticed another, pretty similar issue: {T22501... [08:22:11] 10serviceops, 10CX-cxserver, 10Citoid, 10Graphoid, and 10 others: Make services swagger specs standard compliant - https://phabricator.wikimedia.org/T218217 (10Pchelolo) [08:28:29] 10serviceops, 10Release-Engineering-Team, 10Scap, 10PHP 7.2 support, 10User-jijiki: Enhance MediaWiki deployments for support of php7.x - https://phabricator.wikimedia.org/T224857 (10Joe) >! In T224857#5233723, @thcipriani wrote: > > The speed of deployment directly relates to platform stability. Sacrif... [08:59:42] 10serviceops, 10Release-Engineering-Team, 10Scap, 10PHP 7.2 support, 10User-jijiki: Enhance MediaWiki deployments for support of php7.x - https://phabricator.wikimedia.org/T224857 (10Joe) Ok... found another limitation. Pybal, after reading a change to etcd, waits one second before reading the next one (... [13:37:27] little heads up that me and jijiki are going to merge and deploy https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/468363/ on mw canaries [15:23:18] 10serviceops, 10Operations, 10Reading-Infrastructure-Team-Backlog, 10Release Pipeline, and 2 others: Machine vision image metadata service - https://phabricator.wikimedia.org/T224917 (10Mholloway) p:05Triage→03Normal [18:02:43] 10serviceops, 10Wikimedia-Hackathon-2019, 10Wikimedia-Site-requests, 10Patch-For-Review, 10Performance-Team (Radar): Enlarging the default thumb size on Dutch Wikipedia - https://phabricator.wikimedia.org/T215106 (10Ciell) What's the status on this task? [20:10:51] _joe_: fsero: For your pleasure tomorrow morning, or how alex messed up and yet we survived https://wikitech.wikimedia.org/wiki/Incident_documentation/20190604-blubberoid [20:11:03] I may have to give my chaos monkey badge back :( [20:16:29] akosiaris: I have one additional suggestion, which is run more than one pod of blubberoid, or really anything that serves traffic [20:16:42] (even if the traffic is nominally epsilon qps) [20:16:58] yeah, that's doable [20:17:07] good point [20:19:05] cdanis: added under conclusions [20:20:38] <_joe_> it's incredible it survived at all... [20:20:53] ottomata: btw, eventgate-analytics and eventgate-main and schema won't page on LVS failures per https://gerrit.wikimedia.org/r/plugins/gitiles/operations/puppet/+/production/hieradata/common/lvs/configuration.yaml#1560 et al [20:21:21] <_joe_> maybe they should now [20:21:26] they == ? [20:21:33] <_joe_> those services [20:21:49] <_joe_> they're in production [20:22:15] bear with me, I am tired and probably don't understand [20:22:35] and yes I don't... [20:22:43] I was reading know instead of now [20:22:51] for some bizzare reason. Probably fatigue [20:23:22] anyway, yes ottomata you probably want to review the LVS paging settings for eventgate-analytics,eventgate-main and schema LVS services [20:23:24] yeah they should yeah [20:23:30] i'll make a patch [20:23:55] anyway https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/514574 for blubberoid LVS [20:24:06] I 'll not merge right now however [20:25:19] k, patch made, added you as reviewer, don't worry about it now [20:25:25] do when you are not tired :) [20:25:35] thank you! [20:26:35] cdanis: suggestion implemented [20:26:45] \o/ [20:33:29] Next time instead of cordoning uncordoning you can scale to 0 deployments [20:33:41] Give you more control about what is going to be restarted [20:34:05] Scale to 0 reset to current replicas [20:34:07] I mean [20:34:35] In any case an story worthy of being included in the Zalando summary :)