[00:34:32] <wikibugs>	 10serviceops, 10Operations: All wtp and parse servers have a bad partition scheme. - https://phabricator.wikimedia.org/T258775 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by dzahn on cumin1001.eqiad.wmnet for hosts: ` wtp2019.codfw.wmnet ` The log can be found in `/var/log/wmf-auto-reimage/2020...
[02:57:15] <wikibugs>	 10serviceops, 10Operations: All wtp and parse servers have a bad partition scheme. - https://phabricator.wikimedia.org/T258775 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['wtp2019.codfw.wmnet'] `  and were **ALL** successful.
[03:05:52] <mutante>	 jayme: that special case wtp2019 is now also working, after repeating the reimage and then restarting php-fpm/apache it passed the tests and is pooled now. 
[03:06:24] * mutante goes off now
[05:12:09] <_joe_>	 Pchelolo: it's an rsyslog filter IIRC
[05:12:26] <_joe_>	 so it's defined at the physical host level via puppet
[16:12:23] <ottomata>	 hiya jayme 
[16:12:38] <ottomata>	 i'm getting an error when running helmfile template
[16:12:45] <ottomata>	 that might be related to the chartmuseum change? not sure:
[16:13:01] <ottomata>	 Downloading kafka-dev from repo https://helm-charts.wikimedia.org/stable
[16:13:01] <ottomata>	 Save error occurred:  could not find : no matching version
[16:13:46] <ottomata>	 oh maybe thisi is because there is a requirements.lock file?  but it has version >= 0.0.1
[16:13:47] <ottomata>	 hm
[16:13:55] <ottomata>	 i'm working on eventgate chart
[16:14:04] <ottomata>	 any eventgate helmfile will do
[16:16:45] <ottomata>	 also, is there an easy way to browse what charts are in chartmuseum?
[16:20:06] <jayme>	 ottomata: there is a JSON api at https://helm-charts.wikimedia.org/api/stable/charts
[16:20:52] <jayme>	 taking a look at eventgate
[16:21:14] <ottomata>	 hm ok, it looks like all the kafka-dev charts exist
[16:21:59] <jayme>	 hm. I'm running "helmfile template" in deployment-charts/helmfile.d/services/staging/eventgate-main/ just fine
[16:22:14] <ottomata>	 hmmmmmmmm
[16:22:37] <jayme>	 This happens on your machine I guess?
[16:22:40] <ottomata>	 weird.
[16:22:46] <ottomata>	 no no i'm in deploy1001
[16:22:49] <jayme>	 ah
[16:23:18] <ottomata>	 16:23:02 [@deploy1001:/srv/ … /deployment-charts/helmfile.d/services/staging/eventgate-main] (master)[f84b83f] ± source .hfenv; helmfile template
[16:24:03] <jayme>	 okay, yeah. Can reproduce that there. Sorry
[16:28:21] <jayme>	 "helm fetch stable/kafka-dev" works as well...
[16:32:49] <jayme>	 uuf
[16:33:33] <jayme>	 when running that locally I get an error saying "Error: no repository definition for https://releases.wikimedia.org/charts. Please add the missing repos via 'helm repo add'"
[16:34:05] <jayme>	 Which makes sense as I probably should have bumped chart versions when updating requirements
[16:35:39] <jayme>	 but that brings me to the error level of deploy1001 then, so never mind
[16:41:24] <ottomata>	 hmmm
[16:41:49] <jayme>	 ottomata: what fails is "helm dependency build" for eventgate
[16:42:04] <ottomata>	 the kafka-dev chart really is just for developing
[16:42:12] <ottomata>	 we can make that a manual install for that if we like,i don't mind
[16:42:17] <ottomata>	 we can remove the requirements.yaml dep
[16:42:25] <jayme>	 I got that "fixed" with a local copy of the chart by adding an additional "/" to the repository in requirements.yaml
[16:43:13] <jayme>	 I'll try a CR
[16:43:18] <ottomata>	 ok
[16:46:06] <jayme>	 ottomata: https://gerrit.wikimedia.org/r/c/operations/deployment-charts/+/618787/
[16:46:15] <ottomata>	 +1, shall i merge and try?
[16:46:51] <ottomata>	 ah you got it danke
[16:47:19] <jayme>	 I'll kick of the packaging job when it's merged
[16:47:38] <ottomata>	 ok
[16:49:13] <jayme>	 ottomata: works...oh man *facepalm*
[16:49:33] <jayme>	 deja vu ... https://github.com/helm/helm/issues/1588
[16:51:01] <ottomata>	 haha
[16:52:23] <jayme>	 ottomata: I assume it works for you as well now?
[16:52:32] <jayme>	 would leave for today then...
[16:54:27] <ottomata>	 it does thank you jayme !
[16:54:29] <ottomata>	 :)
[16:56:01] <jayme>	 yw o/
[18:07:16] <_joe_>	 very interesting https://github.com/crazyxman/simdjson_php
[18:09:11] <Krinkle>	 _joe_: relevant tech talk, one  of my fav speakers :)  https://www.youtube.com/watch?v=wlvKAT7SZIQ
[18:10:01] <_joe_>	 Krinkle: given the amount of json we parse, it could be interesting to test out :)
[18:11:31] <Krinkle>	 _joe_: ack,  I'm not aware of large files or many files being parsed in the critical path though. mostly in CLI and jobs,  or for the one  cache  miss after fpm restart
[18:11:51] <Krinkle>	 for extension registry we need to parse ~ 1000 json  files(or whoever many extensions  we  have)
[18:11:57] <_joe_>	 oh right we cache that
[18:12:19] <Krinkle>	 which is going to be "too slow" no matter what unless  we  can find a 10-100X  speed up
[18:12:30] * Krinkle is on vacation
[18:12:48] * _joe_ is off for dinner :P
[18:13:05] <Krinkle>	 I'm  actually looking into turning that into an array file from  the same simplified  build step that  would handle localisation  files  and wmf-config cache for docker
[18:13:55] <Krinkle>	 also reasonably close to  not needing a wmf-config cache anymore
[18:14:45] <ottomata>	 oh ho
[18:14:46] <ottomata>	 https://github.com/luizperes/simdjson_nodejs
[18:14:49] <ottomata>	 for eventgate perhaps?
[18:15:27] <ottomata>	 hmmmm something is still pretty weird with helm
[18:15:31] <ottomata>	 just trying to to a helmfile status
[18:15:39] <ottomata>	   Error: Get "https://neon.eqiad.wmnet:6443/api/v1/namespaces/eventgate-logging-external/pods?labelSelector=app%3Dhelm%2Cname%3Dtiller": Forbidden
[18:20:24] <Krinkle>	 ottomata: https://bugs.chromium.org/p/v8/issues/detail?id=10398
[18:22:48] <Krinkle>	 consider starring it  :) /me just did
[18:23:06] <ottomata>	 :)
[18:27:33] <ottomata>	 Hmmm OHHHH ^^^ was because i had http proxy set
[18:34:03] <mutante>	 same here.one day i kept missing the http_proxy all the time.. then i added it to my global .bash_profile, now i always run into this and forget i have it set and that same thing happens to me.
[19:16:38] <jayme>	  ottomata: I can't reproduce that (on deploy1001 for eventgate-logging-external staging)
[19:24:42] <ottomata>	 oh jayme  sorry that was my fault, had http_proxy set ^^^
[19:24:53] <ottomata>	 i'm having another weird issue but i'm not yet sure if it is my image or what
[19:25:02] <ottomata>	 i can't contact the mw api at api-ro.discovery
[19:25:05] <ottomata>	 a deployed pod can
[19:25:13] <ottomata>	 but when I apply, the new pods cant
[19:28:04] <jayme>	 you still have the not working version running somewhere?
[19:28:36] <ottomata>	 not 100%, still investigating, i saw that once, am trhing on a different helmfile, i think i have a different problem too
[19:38:33] <jayme>	 Okay. Drop a line if you want me/us to have a look tomorrow morning
[19:38:52] * jayme off again
[19:40:25] <ottomata>	 thanks :)
[20:41:31] <ottomata>	 ok,  it was me, had a bug in the way I was passing on the Host header
[20:41:35] <ottomata>	 thanks anyway!
[22:26:46] <wikibugs>	 10serviceops, 10Operations, 10Patch-For-Review, 10User-Elukey: Reimage one memcached shard to Buster - https://phabricator.wikimedia.org/T252391 (10MMiller_WMF) @kostajh -- are you asking whether we should deactivate EditorJourney in all wikis, so as to stop it from recording data anywhere?  If so, I am fi...