[07:23:37] <wikibugs>	 10serviceops, 10DBA, 10OTRS, 10Operations: Create a parallel OTRS database with a frozen snapshot of the production one - https://phabricator.wikimedia.org/T257928 (10jcrespo) 05Open→03Resolved Credentials have been setup and shared on client root dir, feel free to productionize as you see adequate: `...
[07:23:44] <wikibugs>	 10serviceops, 10OTRS, 10Operations, 10User-notice: Update OTRS to the latest stable version (6.0.x) - https://phabricator.wikimedia.org/T187984 (10jcrespo)
[09:03:57] <wikibugs>	 10serviceops, 10Operations, 10Product-Infrastructure-Team-Backlog, 10Push-Notification-Service, 10Patch-For-Review: Deploy push-notifications service to Kubernetes - https://phabricator.wikimedia.org/T256973 (10jijiki) @MSantos Is there an internal service/database push-notifications will be communicatin...
[09:35:34] <wikibugs>	 10serviceops, 10Operations: Update deprecated extension names in envoy config - https://phabricator.wikimedia.org/T258140 (10JMeybohm)
[09:37:13] <wikibugs>	 10serviceops, 10Operations, 10Product-Infrastructure-Team-Backlog, 10Push-Notification-Service, 10Patch-For-Review: Deploy push-notifications service to Kubernetes - https://phabricator.wikimedia.org/T256973 (10jijiki)
[10:39:45] <wikibugs>	 10serviceops, 10Operations, 10Product-Infrastructure-Team-Backlog, 10Push-Notification-Service, 10Patch-For-Review: Deploy push-notifications service to Kubernetes - https://phabricator.wikimedia.org/T256973 (10MSantos) >>! In T256973#6311423, @jijiki wrote: > @MSantos Is there an internal service/databa...
[13:11:18] <wikibugs>	 10serviceops, 10CX-cxserver, 10Language-Team (Language-2020-Focus-Sprint), 10Release-Engineering-Team (Pipeline): Migrate apertium to the deployment pipeline - https://phabricator.wikimedia.org/T255672 (10Rileych) a:03KartikMistry
[13:47:11] <_joe_>	 akosiaris, jayme any idea if there is a way to tell helmfile "execute this bash script before running"?
[13:47:14] <_joe_>	 I guess not
[13:48:42] <jayme>	 _joe_: are those triggers that run !log for example run after the actual action?
[13:49:07] <_joe_>	 they are before the actual action, but after helmfile.yaml gets interpreted
[13:49:20] <jayme>	 hm :/
[13:49:48] <_joe_>	 yeah I think there is no way around asking people to set env variables via a script
[13:51:11] <jayme>	 what about something dump as a shell hook? Or a wrapper script for helmfile that does it?
[13:51:17] <jayme>	 *dumb
[13:51:27] <_joe_>	 yeah I wanted to avoid the wrapper script
[13:51:37] <_joe_>	 we alredy have it
[13:51:49] <jayme>	 eheh, indeed...wrapper for a wrapper then
[13:51:55] <_joe_>	 what do you mean with "a shell hook"?
[13:52:12] <akosiaris>	 _joe_: we can work instead of not needing the .hfenv files
[13:52:17] <akosiaris>	 s/of/on/
[13:52:25] <_joe_>	 akosiaris: that needs helm 3, correct?
[13:52:40] <akosiaris>	 no, helm-diff used to block that, not sure if that's still true
[13:52:58] <akosiaris>	 essentially the issue was the kubeconfig: parameter wasn't honored
[13:53:24] <_joe_>	 that parameter from helmfile.yaml, correct?
[13:53:29] <akosiaris>	 yup
[13:53:35] <_joe_>	 because it's passed as a --kubeconfig parameter
[13:54:04] <akosiaris>	 yes
[13:54:15] <_joe_>	 and that would allow us to use environments
[13:54:17] <akosiaris>	 for the other variables, IIRC K8S_CLUSTER is there just for helmfile_log_sal
[13:54:30] <akosiaris>	 which we can work around it I guess
[13:54:36] <_joe_>	 definitely
[13:55:05] <akosiaris>	 HELM_HOME is to make sure that everyone uses the global /etc/helm so their don't all redownload the charts in the homes
[13:55:26] <akosiaris>	 I am not sure if that's passed, but we could put it in /etc/profile instead :P
[13:55:32] <_joe_>	 yeah so
[13:55:47] <_joe_>	 my idea was to add a file with a couple functions to /etc/profile.d anyways
[13:56:10] <akosiaris>	 and tillerNamespace is also a key now in helmfile.yaml from what I see
[13:56:24] <_joe_>	 yeah
[13:56:42] <akosiaris>	 there is however a minor gotcha
[13:56:52] <_joe_>	 so, I'm going to add a couple very simple functions to /etc/profile.d anyways
[13:56:58] <akosiaris>	 while we could work so that in the general case . .hfenv would not  be needed
[13:57:08] <akosiaris>	 we 'd still need it for stuff like kubectl get pods/events
[13:57:19] <_joe_>	 that's why I am writing those functions
[13:57:33] <_joe_>	 k8s_env mathoid eqiad
[13:57:45] <_joe_>	 and you get those :)
[13:58:12] <jayme>	 with "shell hook" I meant something running on changing the directory and sources the .hfenv automatically
[13:59:20] <ema>	 Pchelolo et al: I've drafted https://wikitech.wikimedia.org/wiki/Kafka_HTTP_purging -- please add mediawiki details if/when you have the chance
[14:03:50] <Pchelolo>	 looking good ema. will add details about MW today.
[14:05:03] <ema>	 <3
[14:12:43] <_joe_>	 Pchelolo: out of curiosity, did you or otto got to create a second partition for hte purges topics?
[14:13:38] <Pchelolo>	 _joe_: oh, no.. somehow that conversation went to something else and we forgot about it
[14:13:47] <Pchelolo>	 it's easy if it's needed, should we do it?
[14:14:05] <_joe_>	 Pchelolo: maybe next week, things seem stabilized rn
[14:14:15] <_joe_>	 I'll re-ping you
[14:14:22] <Pchelolo>	 ok, cool, whenever. 
[14:14:38] <_joe_>	 I'm off tomorrow and I don't want to leave any possible issue to others
[14:14:41] <_joe_>	 to deal with
[14:14:52] <Pchelolo>	 oh, have a nice long weekend :)
[14:15:16] <_joe_>	 yeah I have so much unspent vacation days I'm taking off every friday for a while
[14:15:36] <ottomata>	 yoooo happy to do it! but herron & filippo are managing kafka main more than me these days!  still happy to do it, ticket please!
[14:35:39] <_joe_>	 jayme: names for the bash functions? I was thinking of k8s_env and k8s_env_remove
[14:36:13] <jayme>	 or k8s_env_clear?
[14:36:35] <jayme>	 guessing that it's just resetting whatever is currently set?
[14:36:43] <_joe_>	 yes
[14:36:49] <jayme>	 or enable/disable like in virtualenv
[14:36:51] <_joe_>	 clear is better
[14:37:01] <_joe_>	 yeah I was trying to avoid k8s_enable
[14:37:07] <_joe_>	 and k8s_disable
[14:37:12] <jayme>	 :)
[14:45:40] <jayme>	 having a bash_complete function for that would be delightful 
[15:21:06] <wikibugs>	 10serviceops, 10Mobile-Content-Service, 10Page Content Service, 10Product-Infrastructure-Team-Backlog (Kanban): Investigate why mobileapps in k8s "/{domain}/v1/data/css/mobile/base" endpoint takes way longer than on scb to complete - https://phabricator.wikimedia.org/T258186 (10akosiaris)
[15:23:42] <wikibugs>	 10serviceops, 10Mobile-Content-Service, 10Page Content Service, 10Product-Infrastructure-Team-Backlog (Kanban): Investigate why mobileapps in k8s "/{domain}/v1/data/css/mobile/base" endpoint takes way longer than on scb to complete - https://phabricator.wikimedia.org/T258186 (10Mholloway) I believe the cul...
[15:23:56] <wikibugs>	 10serviceops, 10Mobile-Content-Service, 10Page Content Service, 10Product-Infrastructure-Team-Backlog (Kanban): Investigate why mobileapps in k8s "/{domain}/v1/data/css/mobile/base" endpoint takes way longer than on scb to complete - https://phabricator.wikimedia.org/T258186 (10akosiaris) p:05Triage→03H...
[15:25:02] <wikibugs>	 10serviceops, 10Mobile-Content-Service, 10Page Content Service, 10Product-Infrastructure-Team-Backlog (Kanban): Investigate why mobileapps in k8s "/{domain}/v1/data/css/mobile/base" endpoint takes way longer than on scb to complete - https://phabricator.wikimedia.org/T258186 (10akosiaris) >>! In T258186#63...
[20:23:54] <wikibugs>	 10serviceops, 10Operations, 10Parsoid, 10Parsoid-Tests: Move testreduce away from scandium to a separate Buster Ganeti VM - https://phabricator.wikimedia.org/T257906 (10Dzahn)
[20:51:17] <wikibugs>	 10serviceops, 10Performance-Team, 10Sustainability (Incident Prevention): Avoid php-opcache corruption in WMF production - https://phabricator.wikimedia.org/T253673 (10jijiki) In order to move this a little bit forward, we can try to reproduce and have a go at @ori 's suggestion. If we don't get anywhere, we...
[21:20:22] <wikibugs>	 10serviceops, 10Performance-Team, 10Sustainability (Incident Prevention): Avoid php-opcache corruption in WMF production - https://phabricator.wikimedia.org/T253673 (10Krinkle)
[21:26:41] <wikibugs>	 10serviceops, 10Operations, 10Parsoid, 10Parsoid-Tests: Move testreduce away from scandium to a separate Buster Ganeti VM - https://phabricator.wikimedia.org/T257906 (10Dzahn) Change 613278 merged by Dzahn: [operations/puppet@production] parsoid: create new role to install just testreduce  https://gerrit.w...
[23:24:50] <wikibugs>	 10serviceops, 10Performance-Team, 10Release-Engineering-Team: Create warmup procedure for MediaWiki app servers - https://phabricator.wikimedia.org/T230037 (10Krinkle) p:05Triage→03Low a:05dpifke→03None
[23:32:09] <wikibugs>	 10serviceops, 10Performance-Team, 10Release-Engineering-Team: Create warmup procedure for MediaWiki app servers - https://phabricator.wikimedia.org/T230037 (10Krinkle) We are currently restarting and wiping php/fpm/opcache/apcu on a regular basis. This isn't great and I think there's room for improvement her...
[23:34:54] <wikibugs>	 10serviceops, 10Wikimedia-production-error: Uncaught ConfigException: Failed to load configuration from etcd:  in /srv/mediawiki/php-1.35.0-wmf.38/includes/config/EtcdConfig.php:202 - https://phabricator.wikimedia.org/T256900 (10Krinkle) 05Open→03Resolved a:03Krinkle I guess it's normal that etcd can tim...