[00:56:15] 10serviceops, 10Operations: decom cobalt - https://phabricator.wikimedia.org/T236187 (10RobH) [00:56:39] 10serviceops, 10Operations: decom cobalt - https://phabricator.wikimedia.org/T236187 (10RobH) [01:33:22] 10serviceops, 10Operations, 10Parsoid-PHP, 10Patch-For-Review: wt2html: Out of memory crashers - https://phabricator.wikimedia.org/T236833 (10Dzahn) I uploaded 2 changes. One to simply change it on all and the second one is an attempt to allow different settings for parsoid-php servers vs. the other appser... [02:24:54] 10serviceops, 10Operations, 10Parsoid-PHP, 10Patch-For-Review: wt2html: Out of memory crashers - https://phabricator.wikimedia.org/T236833 (10Krinkle) >>! @ssastry > How should we handle this asymmetry so that when we eventually switch everything to Parsoid, we don't have previously rendered pages stop ren... [08:34:09] 10serviceops, 10Operations, 10observability: basic prometheus monitoring for PoolCounter - https://phabricator.wikimedia.org/T237407 (10MoritzMuehlenhoff) p:05Triage→03Normal [09:39:00] 10serviceops, 10Operations: decom cobalt - https://phabricator.wikimedia.org/T236187 (10hashar) Thank you @Dzahn for all the clean up tasks! [10:07:08] 10serviceops, 10Gerrit, 10Release-Engineering-Team: Gerrit crashed due to out of Heap - https://phabricator.wikimedia.org/T225166 (10hashar) 05Open→03Resolved The heap size has been grown from 20G to 32G ( https://gerrit.wikimedia.org/r/545381 ) We have a new server which is way more powerful We did a... [15:34:51] akosiaris: i can deploy the eventgate-logging-external stuff now [15:34:58] blocker of creating namespaces? [15:37:04] ottomata: if you try, you will fail, there are no namespaces yet [15:39:00] ok [15:39:25] akosiaris: what to do then? :) [17:21:06] ottomata: tell us you want to do it and when ? [17:27:06] akosiaris: today?! [17:27:09] should i just merge that patch? [17:27:15] lol [17:27:22] you should say yesterday [17:27:25] yesterday! [17:27:26] I thought you knew that [17:27:27] :P [17:27:29] haha [17:27:38] you can merge ofc [17:27:43] but you will fail to deploy [17:27:58] aye [17:28:00] lemme see if I can whiff up the patch needed [17:28:04] k danke [17:28:21] I 'll add you to it. will need a couple of hours. I am on the move from the conf to dinner [17:33:13] ok np ty [17:39:37] 10serviceops, 10Machine vision, 10Operations, 10Product-Infrastructure-Team-Backlog (Kanban): Configure Google Cloud Vision credentials in production - https://phabricator.wikimedia.org/T236426 (10Joe) Hi, the credentials file should be stored, as any secret that needs to be accessed by MediaWiki, within i... [18:31:43] _joe_, subbu: re our convo yday about deployment freeze, now i see in SoS notes: This week train last train (1.35.0-wmf.5 - [[phab:T233853]]) is the last train before two week break [18:32:47] hmm ... given that we don't ride the train ... what are your thoughts on that? [18:33:09] i'm not sure if that means there are no deploys at all or what [18:33:41] and wait, does parsoid/php not ride the train? [18:33:57] if it's otherwise an appserver it should [18:34:36] mobrovac, no. we use parsoid scap deploy .. parsoid/php code lives in the parsoid/js repos (main and deploy) [18:35:10] ah yes, indeed [18:35:12] it is a custom extension for now .. till we move api & config wrappers from parsoid repo to core. [18:35:18] sorry, brain fart [18:35:27] its okay. lot of moving pieces. [18:35:32] they are in scap groups though, so they get mw train [18:35:34] akosiaris: oh fyi we have another one coming too [18:35:49] https://phabricator.wikimedia.org/T233629 [18:35:50] this will be [18:35:54] eventgate-analytics-external [18:36:09] can be done anytime so [18:36:16] if you can create the namespace for that at the same time, would be much obliged :) [19:08:24] 10serviceops, 10Machine vision, 10Operations, 10Product-Infrastructure-Team-Backlog (Kanban): Configure Google Cloud Vision credentials in production - https://phabricator.wikimedia.org/T236426 (10Mholloway) @Joe Thanks, that's helpful. I found the private repo you're referring to, and the credentials can... [19:50:27] 10serviceops, 10Core Platform Team (Needs Cleaning - Security, stability, performance, and scalability (TEC1)): Allow service-checker to run multiple domains for RESTBase - https://phabricator.wikimedia.org/T227198 (10CCicalese_WMF) 05Open→03Declined [20:59:58] 10serviceops, 10Operations, 10observability: basic prometheus monitoring for PoolCounter - https://phabricator.wikimedia.org/T237407 (10RLazarus) a:03RLazarus [22:05:24] 10serviceops, 10Operations, 10ops-eqiad: Can't SSH to mw1290.mgmt - https://phabricator.wikimedia.org/T234153 (10Dzahn) [22:23:55] 10serviceops, 10Operations, 10ops-eqiad: Can't SSH to mw1290.mgmt - https://phabricator.wikimedia.org/T234153 (10Dzahn) 05Open→03Resolved flea power was drained by Jclark. I can ssh to mgmt (faster than before). Looking good so far. Works for now. If we see it again we will just reopen this. [22:30:17] 10serviceops, 10Parsing-Team, 10Parsoid, 10PHP 7.2 support: Parsoid-php doesn't get updated after a code deploy - https://phabricator.wikimedia.org/T236275 (10Arlolra) Some notes from today's deploy. Here's the log, https://phabricator.wikimedia.org/P9543 `restart_php` runs after `repool`, despite the at... [22:31:19] 10serviceops, 10Operations, 10ops-eqiad: Can't SSH to mw1290.mgmt - https://phabricator.wikimedia.org/T234153 (10Dzahn) repooled [22:37:24] 10serviceops, 10Parsing-Team, 10Parsoid, 10PHP 7.2 support: Parsoid-php doesn't get updated after a code deploy - https://phabricator.wikimedia.org/T236275 (10Arlolra) Oh, and the above log is from the beta cluster deploy. Production didn't seem to have the `restart_php` failure issue, however, it did hav...