[06:23:31] <_joe_> subbu: definitely [06:42:03] 10serviceops, 10Operations, 10Parsoid-PHP, 10Patch-For-Review, 10Wikimedia-production-error: wt2html: Out of memory crashers - https://phabricator.wikimedia.org/T236833 (10jcrespo) This is ongoing, so adding production error tag: ` PHP Fatal error: Allowed memory size of 692060160 bytes exhausted (trie... [11:52:18] 10serviceops, 10Operations: Appservers rising GET latency might have triggered LVS pages - https://phabricator.wikimedia.org/T238973 (10jbond) p:05Triage→03Normal [14:37:58] 10serviceops, 10Operations, 10Prod-Kubernetes, 10Pybal, 10Traffic: Proposal: simplify set up of a new load-balanced service on kubernetes - https://phabricator.wikimedia.org/T238909 (10fgiunchedi) First thank you for getting the ball rolling on this proposal! A question: are all approaches proposed targe... [14:46:14] 10serviceops, 10Operations, 10Prod-Kubernetes, 10Pybal, 10Traffic: Proposal: simplify set up of a new load-balanced service on kubernetes - https://phabricator.wikimedia.org/T238909 (10Joe) >>! In T238909#5693597, @fgiunchedi wrote: > From my POV there's great value in having a single solution for load b... [14:59:33] 10serviceops, 10Operations, 10Parsoid-PHP, 10Patch-For-Review, 10Wikimedia-production-error: wt2html: Out of memory crashers - https://phabricator.wikimedia.org/T236833 (10Joe) Patches are merged and the memory limit is now at 760 MB, as confirmed by the current OOMs. [15:02:33] _joe_, ty [16:34:06] 10serviceops, 10Parsoid-PHP, 10Core Platform Team Workboards (Clinic Duty Team): php-fpm isn't restarted when deploys are rolled back - https://phabricator.wikimedia.org/T238685 (10ssastry) 05Open→03Resolved @mobrovac verified this worked. But, he found that servers aren't depooled during rollback. Will... [18:50:52] i am going to add envoy to mwmaint. so that we can do TLS between caching layer and noc.wikimedia.org. it's one of the few misc things left on that big ticket about services without TLS [18:51:14] also using mwmaint.discovery.wmnet [18:52:16] cert already created a while ago [20:35:15] 10serviceops, 10Core Platform Team, 10Performance-Team, 10Scap, and 5 others: Define variant Wikimedia production config in compiled, static files - https://phabricator.wikimedia.org/T223602 (10Jdforrester-WMF) [21:48:02] 10serviceops, 10MediaWiki-General, 10Core Platform Team Workboards (Clinic Duty Team), 10Language-Team (Language-2019-October-December), and 4 others: Preemptive refresh in getMultiWithSetCallback() and getMultiWithUnionSetCallback() pollutes cache - https://phabricator.wikimedia.org/T235188 (10Krinkle) >>... [21:48:20] 10serviceops, 10Core Platform Team, 10MediaWiki-General, 10Core Platform Team Workboards (Clinic Duty Team), and 5 others: Preemptive refresh in getMultiWithSetCallback() and getMultiWithUnionSetCallback() pollutes cache - https://phabricator.wikimedia.org/T235188 (10Krinkle)