[08:41:39] 10serviceops, 10DBA, 10Operations, 10Goal: Switchover backup director service from helium to backup1001 - https://phabricator.wikimedia.org/T236406 (10jcrespo) [11:04:32] 10serviceops, 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (CI & Testing services), 10Test-Coverage: Upgrade our php-xdebug package for php7.2 - https://phabricator.wikimedia.org/T234418 (10hashar) That is for the CI Docker containers running php7.2. It seems the production releases... [14:49:40] <_joe_> mark: around? [14:51:54] <_joe_> I've just been asked to assist on https://phabricator.wikimedia.org/T235188 and I don't think I can manage [14:55:14] 10serviceops, 10MediaWiki-General, 10Core Platform Team Workboards (Clinic Duty Team), 10Language-Team (Language-2019-October-December), and 4 others: Preemptive refresh in getMultiWithSetCallback() and getMultiWithUnionSetCallback() pollutes cache - https://phabricator.wikimedia.org/T235188 (10Joe) >>! In... [15:27:00] 10serviceops, 10Operations, 10Wikimedia-Apache-configuration: Build a black-box httpd testing framework - https://phabricator.wikimedia.org/T236699 (10RLazarus) [15:43:26] _joe_: how can I help? [15:44:33] <_joe_> mark: I am working on it now, I guess excusing my absence at the SRE meeting [15:46:38] need help from anyone? [15:47:01] <_joe_> I think maybe I'll ask rlazarus a shameful code review [15:47:15] best kind [16:51:08] 10serviceops, 10Wikidata, 10Wikidata-Termbox: Error when executing helmfile commands for the termbox service - https://phabricator.wikimedia.org/T236709 (10Jakob_WMDE) [16:56:20] 10serviceops, 10MediaWiki-General, 10Core Platform Team Workboards (Clinic Duty Team), 10Language-Team (Language-2019-October-December), and 4 others: Preemptive refresh in getMultiWithSetCallback() and getMultiWithUnionSetCallback() pollutes cache - https://phabricator.wikimedia.org/T235188 (10Joe) After... [16:59:43] 10serviceops, 10Wikidata, 10Wikidata-Termbox: Error when executing helmfile commands for the termbox service - https://phabricator.wikimedia.org/T236709 (10Joe) p:05Triage→03High a:03Joe @Jakob_WMDE this is a result of our temporary fix for a CVE affecting kubernetes. We will try to revert the situatio... [17:07:26] 10serviceops, 10MediaWiki-General, 10Core Platform Team Workboards (Clinic Duty Team), 10Language-Team (Language-2019-October-December), and 4 others: Preemptive refresh in getMultiWithSetCallback() and getMultiWithUnionSetCallback() pollutes cache - https://phabricator.wikimedia.org/T235188 (10Joe) To sho... [19:06:25] 10serviceops, 10Parsing-Team, 10Parsoid, 10PHP 7.2 support: Parsoid-php doesn't get updated after a code deploy - https://phabricator.wikimedia.org/T236275 (10Arlolra) When trying to deploy to the beta cluster today, we hit, ` Executing check 'restart_php' Unhandled error: deploy-local failed: {... [19:07:56] 10serviceops, 10Parsing-Team, 10Parsoid, 10PHP 7.2 support: Parsoid-php doesn't get updated after a code deploy - https://phabricator.wikimedia.org/T236275 (10Arlolra) > which presumably means that command: /usr/local/sbin/restart-php7.2-fpm is not on those servers. I don't have permission to ssh there an... [19:49:29] 10serviceops, 10Parsing-Team, 10Parsoid, 10PHP 7.2 support: Parsoid-php doesn't get updated after a code deploy - https://phabricator.wikimedia.org/T236275 (10ssastry) On beta cluster, we worked around this by continuing the deployment and restarting the php7.2-fpm service manually. When we proceeded to p... [20:18:53] 10serviceops, 10Parsing-Team, 10Parsoid, 10PHP 7.2 support: Parsoid-php doesn't get updated after a code deploy - https://phabricator.wikimedia.org/T236275 (10Dzahn) > The name org.freedesktop.PolicyKit1 was not provided by any .service files The error really does not make it obvious but this should be a... [20:28:04] 10serviceops, 10MediaWiki-General, 10Core Platform Team Workboards (Clinic Duty Team), 10Language-Team (Language-2019-October-December), and 4 others: Preemptive refresh in getMultiWithSetCallback() and getMultiWithUnionSetCallback() pollutes cache - https://phabricator.wikimedia.org/T235188 (10WDoranWMF)... [23:37:38] is there anyone around that understand what this error means? [23:37:39] Error: forwarding ports: error upgrading connection: pods "tiller-deploy-97d7ddd67-2kgh4" is forbidden: User "echostore" cannot create resource "pods/portforward" in API group "" in the namespace "echostore" [23:37:48] I was trying to do a helmfile diff [23:39:41] any helmfile commands seems to provoke this error, though [23:39:49] Google is not helpful here [23:44:28] 10serviceops, 10Parsing-Team, 10Parsoid, 10PHP 7.2 support, 10Patch-For-Review: Parsoid-php doesn't get updated after a code deploy - https://phabricator.wikimedia.org/T236275 (10Dzahn) @ssastry The error in production should hopefully be gone now: ` root@wtp1026:/etc/sudoers.d# sudo -u deploy-service s... [23:47:31] urandom: I think something broke for helmfile deploys as non-admins [23:47:52] sad_trombone.wav? [23:48:43] cdanis: admin == SRE? [23:55:32] urandom: yeah, I think so, someone with cluster admin roles. I'm not sure what... ah [23:55:35] https://gerrit.wikimedia.org/r/c/operations/deployment-charts/+/544158 [23:56:35] I'm not sure I understand or agree with the threat model here, but I'm also rather hesitant to just merge and push myself [23:58:42] how hard are you blocked urandom?