[09:13:31] <wikibugs>	 10serviceops, 10wikimedia-irc-freenode: Add Group Contacts user to #wikimedia-serviceops access list - https://phabricator.wikimedia.org/T214282 (10Peachey88)
[10:07:13] <_joe_>	 godog: so fsero needs to create a synced container on swift for the docker registry
[10:07:33] <_joe_>	 he was thinking to add an exec (with an onlyif clause) to create it if not present
[10:07:44] <_joe_>	 but I am not sure it's how we manage swift containers
[10:07:54] <_joe_>	 or if we have a policy on how we manage them
[10:08:54] <fsero>	 thing is i do not see any trace of created containers in swift in the puppet repo, maybe other accounts handles the container creation in the application (like mw maybe)
[10:10:08] <_joe_>	 I strongly hope it doesn't
[10:10:12] <_joe_>	 :P
[10:10:48] <fsero>	 hopes exists to be crushed :P
[10:24:56] <godog>	 yeah container creation is handled by mw maintenance scripts
[10:25:50] <godog>	 fsero: I meant to ask but forgot, is there a task for the container sync research/work and docker registry ?
[10:26:36] <fsero>	 There is one about docker registry but I'll create a subtask for this specific task
[10:32:50] <wikibugs>	 10serviceops, 10Operations, 10Prod-Kubernetes, 10Kubernetes, 10Patch-For-Review: improve docker registry architecture - https://phabricator.wikimedia.org/T209271 (10fselles)
[10:56:33] <wikibugs>	 10serviceops, 10Operations, 10Prod-Kubernetes, 10Kubernetes: Make swift containers for docker registry cross replicated. - https://phabricator.wikimedia.org/T214289 (10fselles) p:05Triage→03Normal
[10:56:45] <fsero>	 godog: https://phabricator.wikimedia.org/T214289 here you go
[11:02:54] <wikibugs>	 10serviceops, 10Operations, 10Prod-Kubernetes, 10Kubernetes: Make swift containers for docker registry cross replicated. - https://phabricator.wikimedia.org/T214289 (10fselles) I've replicated this using a local SAIO setup and it seems to work, however obviously we are avoiding network latency here hence t...
[11:05:30] <wikibugs>	 10serviceops, 10Operations, 10Thumbor, 10Patch-For-Review, and 3 others: Upgrade Thumbor servers to Stretch - https://phabricator.wikimedia.org/T170817 (10Gilles)
[11:06:35] <godog>	 fsero: looks good! thank you
[12:37:44] <wikibugs>	 10serviceops, 10MediaWiki-Cache, 10Operations, 10Patch-For-Review, and 3 others: Apply -R 200 to all the memcached mw object cache instances running in eqiad/codfw - https://phabricator.wikimedia.org/T208844 (10jijiki)
[14:19:13] <fsero>	 this post seems to align what my gur feeling about  puppet and k8s (it also covers other options) http://leebriggs.co.uk/blog/2018/05/08/kubernetes-config-mgmt.html
[14:42:50] <Krenair>	 fsero, there was a followup: https://leebriggs.co.uk/blog/2018/11/07/kr8-kubernetes-config-mgmt.html
[14:47:48] <fsero>	 yep Krenair thanks! I'll take a look into the project it may fit our purposes but does look too alpha even in k8s terms :)
[14:48:09] <Krenair>	 yeah I don't know nearly enough to evaluate it
[14:48:20] <Krenair>	 just went looking around his blog and noticed the followup so thought I'd mention it
[15:48:41] <wikibugs>	 10serviceops, 10Continuous-Integration-Infrastructure, 10Developer-Wishlist (2017), 10Patch-For-Review, and 3 others: Relocate CI generated docs and coverage reports - https://phabricator.wikimedia.org/T137890 (10hashar) It is almost down, I could just use files received from rsync to be group writable. Th...
[16:12:02] <_joe_>	 fsero: I do think we could manage production value files from puppet
[16:12:08] <_joe_>	 or better, adopt a mixed approach
[16:12:33] <_joe_>	 I would never ever use puppet to define what's deployed on k8s
[16:12:47] <_joe_>	 or whatever other of these shits
[16:14:01] <_joe_>	 there is a fundamental flaw in all of his reasoning
[16:14:18] <_joe_>	 he considers k8s as if it was a set of on-disk applications you deploy
[16:14:29] <_joe_>	 instead, you interact with a database with an api, basically
[16:16:02] <_joe_>	 and thus probably the best way to manage it is.. a program interacting with that API
[16:16:27] <_joe_>	 so helm or some higher-level wrapper on top of it
[16:19:21] <fsero>	 I think we can agree on managing value files that way but I don't see a clean way to do it, like puppet writing the value file on deploy1001 in a known path and helm apply from deploy1001
[16:20:01] <_joe_>	 no please
[16:20:13] <_joe_>	 puppet shall never be able to deploy on kubernetes
[16:20:53] <_joe_>	 so ideally I would see a flow as follows:
[16:21:22] <wikibugs>	 10serviceops, 10Cloud-VPS, 10Operations, 10Traffic: Difficulties to create offline version of Wikipedia because of HTTP 429 response - https://phabricator.wikimedia.org/T213475 (10akosiaris) >>! In T213475#4883423, @Kelson wrote: > I'm not sure to fully understand the technical explanation. Is the problem...
[16:21:31] <_joe_>	 * puppet maintains a list of key-value pairs in a yaml file controlled by SRE
[16:22:25] <_joe_>	 * when someone wants to deploy, they'll write another value file containing all the values that are deployment-dependent, committed $somewhere (I don't have a great solution for that part)
[16:22:41] <_joe_>	 our wrapper around helm will combine both values file when deploying
[16:23:11] <_joe_>	 so that say you have to turn off kafka1001 - you will remove it from the values.yaml file controlled by puppet
[16:23:23] <_joe_>	 and then we'll need to redeploy all apps that use it
[16:23:33] <_joe_>	 but that will just be a simple command, hopefully
[16:24:18] <_joe_>	 the reason why I'm yelling "puppet stay the fuck away" is... I know how bad puppet is at coordination
[16:24:38] <_joe_>	 it's impossible to properly manage a coordinated change via puppet, ever
[16:43:22] <fsero>	 ack but we need to iterate over that idea, agree on leave puppet outside the coordination but we need to figure it which kind of data we need from puppet i.e host list for discovery? and how to merge them with the application values.yaml. If hosts enabled by services list is < 1MiB we can store it as a per namespace well known ConfigMap 
[16:56:23] <_joe_>	 so, helm supports multiple values files from the CLI
[16:56:35] <_joe_>	 as in, feeding it multiple vlaues files
[16:59:12] <fsero>	 yep, so something like helm install --set values.yaml --set values.puppet.yaml but i'd like to describe really well the whole process and also managing helm itself has its own quirks, currently my bet is on helmfile but i'll do a quick poc
[20:12:35] <wikibugs>	 10serviceops, 10Cloud-VPS, 10Operations, 10Traffic: Wikimedia varnish rules no longer exempt all Cloud VPS/Toolforge IPs from rate limits (HTTP 429 response) - https://phabricator.wikimedia.org/T213475 (10bd808)
[20:12:58] <wikibugs>	 10serviceops, 10Cloud-VPS, 10Operations, 10Traffic: Wikimedia varnish rules no longer exempt all Cloud VPS/Toolforge IPs from rate limits (HTTP 429 response) - https://phabricator.wikimedia.org/T213475 (10bd808)
[20:23:43] <wikibugs>	 10serviceops, 10Cloud-VPS, 10Operations, 10Traffic: Wikimedia varnish rules no longer exempt all Cloud VPS/Toolforge IPs from rate limits (HTTP 429 response) - https://phabricator.wikimedia.org/T213475 (10Cyberpower678) @bd808 just invited me here.  Ever since the Cloud VPS migration, Cyberbot has been hit...
[20:25:16] <wikibugs>	 10serviceops, 10Cloud-VPS, 10Operations, 10Traffic: Wikimedia varnish rules no longer exempt all Cloud VPS/Toolforge IPs from rate limits (HTTP 429 response) - https://phabricator.wikimedia.org/T213475 (10Cyberpower678) p:05Normal→03High I'm also boldly raising the priority as from what I gather I'm li...