[07:20:50] <wikibugs>	 10serviceops, 10Release Pipeline, 10Release-Engineering-Team: Provide the official production base images for Wikimedia use - https://phabricator.wikimedia.org/T238774 (10Joe) This task is generic enough, in this form, that I'm not sure wether it's a good idea.  What should this base image set up? A LAMP app...
[07:22:36] <wikibugs>	 10serviceops, 10Release Pipeline, 10Release-Engineering-Team: Provide the official production base images for Wikimedia use - https://phabricator.wikimedia.org/T238774 (10Joe) p:05Triage→03Normal
[07:24:17] <wikibugs>	 10serviceops, 10Operations, 10Kubernetes: New Deployment charts should allow exposing services via TLS - https://phabricator.wikimedia.org/T236008 (10Joe) p:05Triage→03Normal
[07:29:01] <wikibugs>	 10serviceops, 10Performance-Team, 10Release-Engineering-Team: Create warmup procedure for MediaWiki app servers - https://phabricator.wikimedia.org/T230037 (10Joe) I am not convinced this is a great idea.  Warmup at restart makes all operating procedures more complex, and frankly the perfomance gain seems pr...
[08:32:06] <wikibugs>	 10serviceops, 10Cleanup, 10Repository-Admins: Archive operations/debs/hhvm repository - https://phabricator.wikimedia.org/T237038 (10Joe) 05Stalled→03Open
[08:33:00] <wikibugs>	 10serviceops, 10Cleanup, 10Repository-Admins: Archive operations/debs/hhvm repository - https://phabricator.wikimedia.org/T237038 (10Joe) We should also remove all the stale docker images.
[08:36:09] <wikibugs>	 10serviceops, 10Operations, 10Wikimedia-Apache-configuration: Build a black-box httpd testing framework - https://phabricator.wikimedia.org/T236699 (10Joe) p:05Triage→03Normal
[08:37:11] <wikibugs>	 10serviceops, 10Prod-Kubernetes, 10decommission, 10Kubernetes: etcd1004-1006 is unused and idle, use the cluster or kill it. - https://phabricator.wikimedia.org/T212934 (10Joe) @akosiaris didn't we just dismiss this cluster last week?
[08:53:27] <wikibugs>	 10serviceops, 10Prod-Kubernetes, 10decommission, 10Kubernetes: etcd1004-1006 is unused and idle, use the cluster or kill it. - https://phabricator.wikimedia.org/T212934 (10akosiaris) 05Open→03Resolved a:03akosiaris >>! In T212934#5812094, @Joe wrote: > @akosiaris didn't we just dismiss this cluster l...
[10:50:34] <wikibugs>	 10serviceops, 10Proton, 10Patch-For-Review, 10Product-Infrastructure-Team-Backlog (Kanban): Profile proton memory usage for Helm chart - https://phabricator.wikimedia.org/T238830 (10akosiaris) >>! In T238830#5810385, @MSantos wrote: >>>! In T238830#5810194, @akosiaris wrote: >> @MSantos where were the test...
[11:34:22] <wikibugs>	 10serviceops, 10Operations, 10Patch-For-Review, 10User-jijiki: Create a  mediawiki::cronjob define - https://phabricator.wikimedia.org/T211250 (10Joe)
[11:37:47] <wikibugs>	 10serviceops, 10Operations, 10Patch-For-Review, 10User-jijiki: Create a  mediawiki::cronjob define - https://phabricator.wikimedia.org/T211250 (10Joe) We should complete this work before we perform the MediaWiki switchover.
[12:02:49] <wikibugs>	 10serviceops, 10Proton, 10Patch-For-Review, 10Product-Infrastructure-Team-Backlog (Kanban): Profile proton memory usage for Helm chart - https://phabricator.wikimedia.org/T238830 (10akosiaris) @Joe pointed out to me that in my paste there is also `"Unexpected error: Error: spawn ENOMEM"` (which I should ha...
[12:17:44] <wikibugs>	 10serviceops, 10Proton, 10Patch-For-Review, 10Product-Infrastructure-Team-Backlog (Kanban): Profile proton memory usage for Helm chart - https://phabricator.wikimedia.org/T238830 (10akosiaris) FWIW, increasing the locust "users" to 100, bumped the failures to 15%, response times increased and RPS dropped b...
[12:47:15] <wikibugs>	 10serviceops, 10Core Platform Team, 10MediaWiki-Cache, 10Performance-Team (Radar): Ensure apcu incr/decr are atomic (Upgrade php-apcu) - https://phabricator.wikimedia.org/T236800 (10jijiki) @Krinkle I backported https://github.com/krakjoe/apcu/pull/384/commits/c8849b0894712d0ede63552fb99ca8dc4d3d884f and p...
[13:13:03] <wikibugs>	 10serviceops, 10Release-Engineering-Team, 10Release-Engineering-Team-TODO: Update Wikimedia production to PHP 7.2.26 - https://phabricator.wikimedia.org/T241222 (10jijiki) 05Open→03Resolved a:03jijiki
[13:29:49] <wikibugs>	 10serviceops, 10Proton, 10Patch-For-Review, 10Product-Infrastructure-Team-Backlog (Kanban): Profile proton memory usage for Helm chart - https://phabricator.wikimedia.org/T238830 (10MSantos) @akosiaris interesting! Thanks for testing it, maybe the better path for the benchmark would be to deploy proton alo...
[16:14:27] <wikibugs>	 10serviceops, 10Proton, 10Patch-For-Review, 10Product-Infrastructure-Team-Backlog (Kanban): Profile proton memory usage for Helm chart - https://phabricator.wikimedia.org/T238830 (10akosiaris) >>! In T238830#5812800, @MSantos wrote: > @akosiaris interesting! Thanks for testing it, maybe the better path for...
[16:19:55] <wikibugs>	 10serviceops, 10Proton, 10Patch-For-Review, 10Product-Infrastructure-Team-Backlog (Kanban): Profile proton memory usage for Helm chart - https://phabricator.wikimedia.org/T238830 (10akosiaris) I just noticed that `num_workers: ncpu` in the chart. Sigh, this probably makes all CPU calculations wrong as it i...
[19:00:51] <wikibugs>	 10serviceops, 10Cleanup, 10Repository-Admins: Archive operations/debs/hhvm repository - https://phabricator.wikimedia.org/T237038 (10Dzahn)
[19:00:55] <wikibugs>	 10serviceops, 10Cleanup, 10Repository-Admins: Archive operations/debs/hhvm repository - https://phabricator.wikimedia.org/T237038 (10Dzahn)
[19:01:38] <wikibugs>	 10serviceops, 10Cleanup, 10Phabricator, 10Repository-Admins: Archive operations/debs/hhvm repository - https://phabricator.wikimedia.org/T237038 (10Dzahn)
[19:07:27] <wikibugs>	 10serviceops, 10Cleanup, 10Phabricator, 10Repository-Admins: Archive operations/debs/hhvm repository - https://phabricator.wikimedia.org/T237038 (10Dzahn)
[19:08:35] <wikibugs>	 10serviceops, 10Cleanup, 10Phabricator, 10Repository-Admins: Archive operations/debs/hhvm repository - https://phabricator.wikimedia.org/T237038 (10Dzahn)
[19:13:23] <wikibugs>	 10serviceops, 10Cleanup, 10Phabricator, 10Repository-Admins, 10Patch-For-Review: Archive operations/debs/hhvm repository - https://phabricator.wikimedia.org/T237038 (10Jdforrester-WMF)
[19:15:18] <wikibugs>	 10serviceops, 10Cleanup, 10Phabricator, 10Repository-Admins, 10Patch-For-Review: Archive operations/debs/hhvm repository - https://phabricator.wikimedia.org/T237038 (10Dzahn)
[19:25:07] <wikibugs>	 10serviceops, 10Cleanup, 10Phabricator, 10Repository-Admins, 10Patch-For-Review: Archive operations/debs/hhvm repository - https://phabricator.wikimedia.org/T237038 (10Dzahn)
[19:25:33] <wikibugs>	 10serviceops, 10Cleanup, 10Phabricator, 10Repository-Admins, 10Patch-For-Review: Archive operations/debs/hhvm repository - https://phabricator.wikimedia.org/T237038 (10Dzahn)
[19:26:17] <wikibugs>	 10serviceops, 10Cleanup, 10Phabricator, 10Repository-Admins, 10Patch-For-Review: Archive operations/debs/hhvm repository - https://phabricator.wikimedia.org/T237038 (10Jdforrester-WMF)
[19:39:51] <wikibugs>	 10serviceops, 10WMF-JobQueue, 10Core Platform Team Workboards (Clinic Duty Team): Jobrunner monitoring still calles /rpc/runJobs.php - https://phabricator.wikimedia.org/T243096 (10Pchelolo)
[20:31:56] <wikibugs>	 10serviceops, 10Release Pipeline, 10Release-Engineering-Team: Provide the official production base images for Wikimedia use - https://phabricator.wikimedia.org/T238774 (10Jdforrester-WMF) >>! In T238774#5812017, @Joe wrote: > This task is generic enough, in this form, that I'm not sure wether it's a good ide...
[21:29:05] <wikibugs>	 10serviceops, 10Proton, 10Patch-For-Review, 10Product-Infrastructure-Team-Backlog (Kanban): Profile proton memory usage for Helm chart - https://phabricator.wikimedia.org/T238830 (10MSantos) >>! In T238830#5813108, @akosiaris wrote: > I just noticed that `num_workers: ncpu` in the chart. Sigh, this probabl...