[07:40:31] <wikibugs>	 10serviceops, 10OpenRefine, 10Operations, 10Traffic, 10Core Platform Team Workboards (Clinic Duty Team): Clients failing API login due to dependence on "Set-Cookie" header name casing - https://phabricator.wikimedia.org/T249680 (10MoritzMuehlenhoff) p:05Triage→03High
[07:53:34] <wikibugs>	 10serviceops, 10OpenRefine, 10Operations, 10Traffic, 10Core Platform Team Workboards (Clinic Duty Team): Clients failing API login due to dependence on "Set-Cookie" header name casing - https://phabricator.wikimedia.org/T249680 (10Joe) As far as I can see, apache (which sits beyond envoy) emits all cooki...
[08:06:33] <wikibugs>	 10serviceops, 10OpenRefine, 10Operations, 10Traffic, 10Core Platform Team Workboards (Clinic Duty Team): Clients failing API login due to dependence on "Set-Cookie" header name casing - https://phabricator.wikimedia.org/T249680 (10Pintoch) Switching on headers capitalization would be absolutely fantastic...
[09:54:40] <wikibugs>	 10serviceops, 10MediaWiki-API, 10Operations, 10Core Platform Team Workboards (External Code Reviews), 10MW-1.35-notes (1.35.0-wmf.27; 2020-04-07): CORS errors on commons on debug servers - https://phabricator.wikimedia.org/T249107 (10Tgr) 05Open→03Resolved a:03Tgr The error can't be reproduced on C...
[10:13:30] <_joe_>	 hnowlan: I added a couple comments to https://gerrit.wikimedia.org/r/#/c/operations/deployment-charts/+/587298/
[10:13:47] <_joe_>	 I'm happy to discuss the substance if you have doubts
[11:47:53] <hnowlan>	 _joe_: thanks! that CR has been merged, it was accidentally reopened I think. But the points are still valid - I'll test and add them to a new change 
[13:17:28] <wikibugs>	 10serviceops, 10Growth-Team, 10MediaWiki-Configuration, 10Parsoid, and 4 others: pywikibot encounters an internal API error with Flow on testwiki (but not other wikis) - https://phabricator.wikimedia.org/T249705 (10Joe)
[13:26:54] <wikibugs>	 10serviceops, 10Growth-Team, 10MediaWiki-Configuration, 10Parsoid, and 5 others: pywikibot encounters an internal API error with Flow on testwiki (but not other wikis) - https://phabricator.wikimedia.org/T249705 (10kostajh) > I just found your errors in the logs, and this thickens the cloud of mistery arou...
[14:18:20] <hnowlan>	 _joe_: seems that loading the file out of /etc won't work, helm limits access to the chart. Unless I misunderstood your idea and we should copy the cert to the chart itself as a file and load it that way as opposed to out of values.yaml 
[14:19:07] <_joe_>	 hnowlan: well or we can just have puppet copy the file over
[14:19:33] <_joe_>	 hnowlan: I'm fixing a patch, then I can assist with that
[14:19:46] <hnowlan>	 puppet would copy it into the helmfile directory? 
[14:19:48] <hnowlan>	 ah, cool 
[14:20:16] <_joe_>	 hnowlan: or it can read it and copy it in a proper place, but yes it's annoying
[14:20:44] <_joe_>	 hnowlan: so either it's in the chart, or it can't be read with Files.Get?
[14:20:55] <_joe_>	 meh
[14:21:00] <_joe_>	 it also kinda-makes sense
[14:21:09] <_joe_>	 so, uhm.
[14:21:09] <hnowlan>	 yeah, tiller doesn't know what's on the filesystem on install 
[14:22:06] <_joe_>	 so, gimme 10 minutes and I can try to help you get unblocked
[14:22:20] <hnowlan>	 cool, thanks! 
[14:22:29] <_joe_>	 the solution will be injecting it in a values.yaml-like file
[14:22:33] <_joe_>	 from puppet
[14:24:51] <_joe_>	 and sorry for sending you down the wrong implementation path
[14:25:11] <_joe_>	 this is something we'd solve if we had a CI system to build the charts
[14:26:40] <_joe_>	 hnowlan: so let's look at puppet class profile::kubernetes::deployment_server::helmfile
[14:27:45] <_joe_>	 look at line 83
[14:28:06] <_joe_>	 there is a template explicitly created to do what we need - pass values from puppet to the charts
[14:28:44] <_joe_>	 we can probably add the puppet CA for all charts :)
[14:28:58] <hnowlan>	 oh, nice!
[14:29:23] <hnowlan>	 or at least better - it still feels kinda ugly but it's better than doing it 1000 times 
[14:30:00] <_joe_>	 it's also better that it will change whenever the puppet ca will change
[14:30:32] <hnowlan>	 yeah, that'll be good
[14:30:40] <_joe_>	 I don't remember if the CA is available in puppet as a variable
[14:30:48] <_joe_>	 or if you will have to read the file somehow
[14:31:23] <hnowlan>	 I'll have a look
[15:54:40] <wikibugs>	 10serviceops, 10Operations, 10Kubernetes: New Deployment charts should allow exposing services via TLS - https://phabricator.wikimedia.org/T236008 (10Joe) 05Open→03Resolved
[15:54:43] <wikibugs>	 10serviceops, 10Operations, 10Kubernetes, 10Patch-For-Review: Add TLS termination to services running on kubernetes - https://phabricator.wikimedia.org/T235411 (10Joe)
[15:55:06] <wikibugs>	 10serviceops, 10Operations, 10Release-Engineering-Team: Hundreds of tags for `wikimedia/mediawiki-core` image - https://phabricator.wikimedia.org/T242775 (10Joe) p:05High→03Low
[16:08:50] <apergos>	 James_ F says 'probably' (the Tuesday preceding the 5 day stretch) will be a no-deploy day but I am waiting for greg -g to confirm.
[16:35:12] <wikibugs>	 10serviceops, 10OpenRefine, 10Operations, 10Traffic, and 2 others: Clients failing API login due to dependence on "Set-Cookie" header name casing - https://phabricator.wikimedia.org/T249680 (10Joe) 05Open→03Resolved a:03Joe From my tests, now we get all cookies correctly set: ` $ curl --http1.1 -sIL...
[16:36:44] <wikibugs>	 10serviceops, 10OpenRefine, 10Operations, 10Traffic, and 2 others: Clients failing API login due to dependence on "Set-Cookie" header name casing - https://phabricator.wikimedia.org/T249680 (10Joe) @Pintoch the behaviour should be restored now. Can you confirm old versions of OpenRefine work correctly now?
[16:55:15] <apergos>	 greg -g says: apergos: tl;dr: no train that week and no deploys on those days off. swats open for the non-off days though. cc thci  priani (right?)
[16:55:35] <apergos>	 so now waiting for thcipri ani to agree or not
[16:56:44] <wikibugs>	 10serviceops, 10OpenRefine, 10Operations, 10Traffic, and 2 others: Clients failing API login due to dependence on "Set-Cookie" header name casing - https://phabricator.wikimedia.org/T249680 (10Pintoch) Thanks a million, this is very kind of you!  I can confirm this works, edits are coming through again fro...
[17:05:09] <wikibugs>	 10serviceops, 10MediaWiki-Parser, 10Operations, 10Core Platform Team Workboards (Clinic Duty Team), and 2 others: API action=parse should be poolcounter-limited if a re-parse is necessary - https://phabricator.wikimedia.org/T243803 (10Anomie) Next step is to determine the limits and implement them for Wiki...
[18:17:57] <wikibugs>	 10serviceops, 10Growth-Team, 10MediaWiki-Configuration, 10Parsoid, and 5 others: pywikibot encounters an internal API error with Flow on testwiki (but not other wikis) - https://phabricator.wikimedia.org/T249705 (10Dvorapa) Just for the record, here is the error after @ssastry (? I guess) edited that parti...
[18:20:17] <wikibugs>	 10serviceops, 10MediaWiki-Page-derived-data, 10Performance-Team: Post-send DeferredUpdates unreliable due to fpm timeout before MW (missing edits in Watchlist and Recentchanges) - https://phabricator.wikimedia.org/T248564 (10Krinkle)
[19:59:46] <apergos>	 (09:31:23 μμ) greg -g: tl;dr: no train that week and no deploys on those days off. swats open for the non-off days though.  -- verified by thcipri ani
[20:00:18] <apergos>	 new calendar should be up Soon™
[23:52:54] <wikibugs>	 10serviceops, 10Growth-Team, 10MediaWiki-Configuration, 10Parsoid, and 5 others: pywikibot encounters an internal API error with Flow on testwiki (but not other wikis) - https://phabricator.wikimedia.org/T249705 (10Dvorapa) Occasionally also fails on mediawiki.org (https://travis-ci.org/github/wikimedia/py...