[06:36:01] 10serviceops, 10Operations, 10Performance-Team, 10Sustainability (Incident Prevention): Test gutter pool failover in production and memcached 1.5.x - https://phabricator.wikimedia.org/T240684 (10Joe) >>! In T240684#6193541, @elukey wrote: > We don't, and doing it should require a big change in the mcrout... [07:19:51] 10serviceops, 10Operations, 10decommission, 10ops-codfw, 10Patch-For-Review: codfw: decom at least 15 appservers in codfw rack C3 to make room for new servers - https://phabricator.wikimedia.org/T247018 (10Dzahn) >>! In T247018#6204125, @Papaul wrote: > switch ports removed for mw2154 through mw2186 @P... [07:38:47] i am trying to add a second envoy listener on phabricator for aphlict.. but build-envoy-config says there are issues which are ultimately " caused by DataSourceValidationError.Filename: ["value length must be at least " '\x01' " bytes"]". I don't see why why but config is invalid somehow. hmm [09:21:32] 10serviceops, 10Operations, 10Patch-For-Review: rack/setup/install ganeti10([09]|1[0-8]).eqiad.wmnet - https://phabricator.wikimedia.org/T228924 (10akosiaris) [09:22:59] 10serviceops, 10Operations, 10Patch-For-Review: rack/setup/install ganeti10([09]|1[0-8]).eqiad.wmnet - https://phabricator.wikimedia.org/T228924 (10akosiaris) 05Open→03Resolved The hardware machines are now in full production mode, ready to receive VMs. In fact, the row C machines already have VMs as the... [10:30:28] 10serviceops, 10ChangeProp, 10Operations, 10Release Pipeline, and 7 others: Migrate cpjobqueue to kubernetes - https://phabricator.wikimedia.org/T220399 (10hnowlan) cpjobqueue is now running fully on Kubernetes. The instance running on scb has no rules enabled. All that remains to be done is to remove the... [12:32:32] <_joe_> hnowlan: \o/ [14:13:45] <_joe_> everyone - I created a very bare gdoc where I started writing down what I think we should do on various things we own in the future [14:14:20] <_joe_> please add stuff and things I didn't pen down [14:14:45] <_joe_> take the time to think about what you think should be the direction for the component you are writing about [14:16:52] awesome, would love to see that in some way or form as well :) [14:17:46] <_joe_> once it's not just a bad bulleted list, sure :) [14:27:17] where's the doc then? [14:34:23] <_joe_> I sent an invite your way [16:08:24] 10serviceops, 10Prod-Kubernetes, 10Kubernetes, 10Patch-For-Review: Move termbox to use TLS only - https://phabricator.wikimedia.org/T254581 (10JMeybohm) [16:35:29] https://grafana.wikimedia.org/d/ifM0GzjWk/cdanis-xxx-php-worker-threads?panelId=2&edit&fullscreen&orgId=1&from=now-24h&to=now seems like the statustext exporter is working correctly [16:35:40] the slight skew between variables is correct :) [17:01:58] 10serviceops, 10Operations, 10decommission, 10ops-codfw, 10Patch-For-Review: codfw: decom at least 15 appservers in codfw rack C3 to make room for new servers - https://phabricator.wikimedia.org/T247018 (10Papaul) @Dzahn yes you can merge that. Thabks [17:09:40] Pchelolo: puppet is currently in status "no bueno" on deployment-prep, I guess no chance of seeing the config change in action for me today as I've gotta go now. Will take a closer look tomorrow though! [17:10:43] ema: thank you [18:05:22] 10serviceops, 10Operations, 10ops-codfw, 10Patch-For-Review: (Need by: TBD) rack/setup/install 86 new codfw mw systems - https://phabricator.wikimedia.org/T241852 (10Papaul) [18:28:03] thanks joe [19:20:36] <_joe_> cdanis: well done [19:21:16] _joe_: oh, I think I forgot to mention, my idiotic pull request to php-fpm upstream might have nerdsniped the maintainer into implementing a special status-request-only worker for each worker pool [19:21:25] (the setproctitle hack) [19:22:01] <_joe_> oh very nice [19:22:27] <_joe_> we should send hacky patch upstream to convince maintainers to do things the right way [19:26:33] yeah I was impressed, it's basically the "Linux sucks because it can't do X!!" strategy but high-effort [19:26:43] it's right in cdanis's groove [20:09:50] 10serviceops, 10Core Platform Team, 10Performance-Team, 10Wikimedia-Rdbms: Determine multi-dc strategy for ChronologyProtector - https://phabricator.wikimedia.org/T254634 (10eprodromou) Glad to see movement on this and we'll be monitoring the progress.