[00:39:35] <wikibugs>	 10serviceops, 10Performance-Team: Run latest Thumbor on Docker with Buster + Python 3 - https://phabricator.wikimedia.org/T267327 (10jijiki)
[00:44:56] <wikibugs>	 10serviceops, 10Performance-Team: Run latest Thumbor on Docker with Buster + Python 3 - https://phabricator.wikimedia.org/T267327 (10jijiki) @Gilles We do not have any other python applications running on Kubernetes, so we are exploring new ground here. I will discuss it a bit more with #serviceops to see how...
[04:51:08] <wikibugs>	 10serviceops, 10Machine Learning Platform, 10ORES, 10Okapi, and 3 others: ORES redis: max number of clients reached... - https://phabricator.wikimedia.org/T263910 (10Ladsgroup) This will very likely fix it: https://github.com/wikimedia/ores/pull/352 I can deploy it if @akosiaris or @calbon approve the PR
[08:56:31] <wikibugs>	 10serviceops, 10Performance-Team, 10User-jijiki: Run latest Thumbor on Docker with Buster + Python 3 - https://phabricator.wikimedia.org/T267327 (10jijiki)
[08:58:45] <wikibugs>	 10serviceops, 10Operations, 10Platform Engineering, 10Wikidata, and 3 others: Upgrade memcached cluster to Debian Stretch/Buster - https://phabricator.wikimedia.org/T213089 (10hashar)
[08:59:28] <_joe_>	 cdanis, rzl https://github.com/wikimedia/puppet/blob/production/modules/mediawiki/manifests/web/prod_sites.pp#L126-L150
[08:59:38] <_joe_>	 all wikipedias share the same vhost in apache
[09:02:14] <wikibugs>	 10serviceops, 10Operations, 10Platform Engineering, 10Wikidata, and 3 others: Upgrade memcached cluster to Debian Stretch/Buster - https://phabricator.wikimedia.org/T213089 (10hashar)
[10:10:08] <_joe_>	 I have a dockerfile question: is it better to COPY 4 files in one go as a dir, then having a RUN statement move them in the right places, or having 4 COPY statements?
[10:10:24] <_joe_>	 akosiaris / jayme ^^
[10:10:33] <_joe_>	 I can see arguments in both directions
[10:11:38] <akosiaris>	 performance wise, the former. It only creates 2 layers that might potentially be invalidated
[10:13:12] <akosiaris>	 if the 4 files however are going to change on very dockerfile build, that's pointless and you can use whatever approach you want. But that's the pathological case, I think
[10:16:21] <jayme>	 +1 to that. You can also give multiple sources in COPY (but one destination only). I usually like that more than giving a directory as source as it is more explicit and fails early if things change in the repo
[10:19:56] <_joe_>	 ok so, I might get away with 2 COPY commands
[10:20:11] <_joe_>	 and no, those files are probably almost never changing
[10:20:19] <jayme>	 if you have only two destinations, yes
[10:20:21] <_joe_>	 but COPY breaks caching anyways
[10:20:40] <_joe_>	 but at least it will be for two 300 bytes layers
[10:21:20] <jayme>	 COPY does not break cache if the file(s) have not changed IMHO
[10:23:51] <_joe_>	 right
[10:24:12] <_joe_>	 so maybe the best solution is
[10:24:24] <_joe_>	 copy all the files to tmp
[10:24:41] <_joe_>	 run installing the packages and moving the files in the right place in the next command
[10:25:02] <_joe_>	 it's a bit less clear to read in the dockerfile though
[10:29:03] <jayme>	 but you can still list the files to copy to tmp explicitely in COPY to make it obvious that they are required
[11:04:35] <wikibugs>	 10serviceops, 10Operations: Upgrade the MediaWiki servers to ICU 63 - https://phabricator.wikimedia.org/T264991 (10Joe) For the ICU transition it's crucially important that no machine with write access to the databases gets updated before the date of the migration. So please do not test this on the eqiad mwdeb...
[12:22:27] <wikibugs>	 10serviceops, 10Desktop Improvements, 10Operations, 10Product-Infrastructure-Team-Backlog, and 3 others: Connection closed while downloading PDF of articles - https://phabricator.wikimedia.org/T266373 (10Jgiannelos) I can also reproduce this on prod: GETing the prod endpoint to render a pdf for the article...
[12:34:52] <wikibugs>	 10serviceops, 10Desktop Improvements, 10Operations, 10Product-Infrastructure-Team-Backlog, and 3 others: Connection closed while downloading PDF of articles - https://phabricator.wikimedia.org/T266373 (10Jgiannelos) Something that could be interesting for debugging purposes is that on eg. `en.wikipedia.org...
[12:57:51] <wikibugs>	 10serviceops, 10Performance-Team, 10User-jijiki: Run latest Thumbor on Docker with Buster + Python 3 - https://phabricator.wikimedia.org/T267327 (10jijiki) @Gilles  Please go ahead bringing deps via wheel files, we do not have any alternative solution for this :)
[14:14:34] <wikibugs>	 10serviceops, 10Desktop Improvements, 10Operations, 10Product-Infrastructure-Team-Backlog, and 3 others: Connection closed while downloading PDF of articles - https://phabricator.wikimedia.org/T266373 (10Jgiannelos) I wrote  a quick script to fetch some random pages and check if the rendered PDF is valid h...
[14:59:15] <wikibugs>	 10serviceops, 10Operations, 10Performance-Team, 10Patch-For-Review, 10User-jijiki: MediaWiki to route specific keys to /*/mw-with-onhost-tier/ - https://phabricator.wikimedia.org/T264604 (10jijiki) @aaron We will merge your patches on Monday and enable onhost memcached on an API canary host :)
[15:41:01] <wikibugs>	 10serviceops, 10Desktop Improvements, 10Operations, 10Product-Infrastructure-Team-Backlog, and 3 others: Connection closed while downloading PDF of articles - https://phabricator.wikimedia.org/T266373 (10akosiaris) So this is not specific to frwiki it seems. Is there perhaps some correlation between page s...
[16:53:04] <wikibugs>	 10serviceops, 10Operations, 10Patch-For-Review: Upgrade the MediaWiki servers to ICU 63 - https://phabricator.wikimedia.org/T264991 (10thcipriani) >>! In T264991#6603285, @jijiki wrote: > @thcipriani we will be upgrading to ICU 63 on the 16th Nov 2020. Since we will be restarting php-fpm across the cluster t...
[17:10:07] <cdanis>	 rzl: have you ever had a desire to have httpbb support `assert_body_lacks` or `assert_body_matches_re` ?
[17:11:10] <rzl>	 `assert_body_lacks` is new but makes sense, `assert_body_matches` has basically just been waiting for a use case
[17:11:39] <cdanis>	 I was thinking about either or both wrt: testing the thankyouwiki change
[17:11:53] <rzl>	 yeah makes sense
[17:11:55] <cdanis>	 not a big deal ofc
[17:12:15] <cdanis>	 haha I guess `assert_body_sha256sum` wouldn't be horrible either
[17:12:20] <rzl>	 it's an easy patch though, happy to toss it in today
[17:12:34] <rzl>	 you and I have widely diverging ideas of what "wouldn't be horrible" means
[17:12:55] <cdanis>	 🥠
[17:14:09] <rzl>	 I have a weak preference for `_matches` over `_lacks`, do you have any opinion?
[17:15:03] <rzl>	 hmm I thought about it before and now I'm leaning the other way 🙃 either is basically fine by me though
[17:15:08] <cdanis>	 would it match the entire body with the multiline flag set?
[17:15:24] <rzl>	 yeah that's about what I was thinking
[17:15:34] <cdanis>	 either sgtm
[17:15:39] <rzl>	 other flags as included in the string, python re syntax
[17:15:43] <cdanis>	 ye
[17:15:50] <rzl>	 so case sensitivity etc is at your discretion
[17:16:18] <rzl>	 okay let's go with that
[17:17:28] <rzl>	 the only thing I'm not sure of is the least confusing syntax -- we already use slashes to indicate regex on assert_headers values
[17:17:35] <cdanis>	 mmm
[17:17:53] <rzl>	 so e.g. assert_headers: {"Content-Type": "text/plain", "Content-Length": "/\d+/"}
[17:17:59] <rzl>	 one is a string compare, the other is a regex
[17:18:33] <rzl>	 so it would be nice to be consistent with that but I'm not sure how exactly
[17:20:53] <cdanis>	 so, for the regex in I'd write here, and I think generally for ones that match against the body
[17:21:02] <cdanis>	 it's very likely you will have literal embedded `/`s as part of the match
[17:21:28] <rzl>	 sure
[17:22:25] <rzl>	 (backwards-incompatible changes aren't the end of the world here btw, there's not that many httpbb test files out there and we can update em all if we decide to change things)
[17:22:30] <cdanis>	 nod
[17:22:46] <rzl>	 (that might be the difference between doing it today vs. in a couple weeks though)
[17:23:10] <cdanis>	 yeah, tbh I don't super need it today -- it will be easy enough to curl the files on mwdebug
[17:23:30] <rzl>	 yeah, it's just this is exactly the kind of thing I want the tool to support
[17:24:12] <cdanis>	 I have a suggestion that you would hate
[17:24:29] <rzl>	 oh?
[17:25:02] <cdanis>	 if the string begins and ends with one of a few punctuation-like characters -- /, !, @ -- treat it as a regexp; otherwise don't
[17:25:04] <cdanis>	 (it's a bad idea)
[17:25:24] <rzl>	 heh agreed
[17:25:31] <rzl>	 I'd much rather just do a re: prefix or something in that case
[17:25:46] <rzl>	 (and retcon it to the header asserts too, then)
[17:26:29] <cdanis>	 nod
[17:27:06] <rzl>	 hmmm
[17:27:21] <rzl>	 if you'd still be happy with `_lacks` we could just do that and ship it out today
[17:27:30] <rzl>	 sidestep this whole question for another time
[17:28:07] <cdanis>	 I don't even think I need either to do a reasonable job on this tbh
[17:28:20] <cdanis>	 I could probably just match on the substring 'excludes' in this case
[17:35:18] <cdanis>	 hm yeah
[17:35:50] <cdanis>	 I thought I was going to _lacks (also, bikeshedding, I wonder if _omits is slightly clearer?) anyway I thought I was going to that on the app id string, but, it turns out it shows up there anyway
[17:38:33] <rzl>	 haha I was just gonna go with assert_body_does_not_contain because bytes are free
[17:39:40] <cdanis>	 haha sgtm
[17:39:41] <rzl>	 but, nod
[17:40:06] <cdanis>	 so the other thing you could do wrt consistency
[17:40:19] <cdanis>	 assert_body_matches_regexp and assert_headers_matches_regexp
[17:41:53] <rzl>	 yeah, it just felt a little weird to break up the assert_headers dict that way
[17:42:06] <rzl>	 but I guess it's probably unlikely anyone actually uses both
[17:42:13] <rzl>	 in the same request I mean
[17:45:38] <cdanis>	 right
[17:45:52] <cdanis>	 well -- I can see using it for different headers
[17:46:06] <cdanis>	 it would be a bit strange to see using them on the same header, but, I don't see a strong reason to disallow that either
[17:46:23] <rzl>	 right yeah
[17:47:29] <rzl>	 using it for different headers is what I had in mind, I think in a perfect world it would make sense to keep them together -- but on balance I think the thing you're suggesting is probably right
[17:48:20] <rzl>	 we can even do it in two passes to avoid breaking the existing uses, first add the new assert and then rip out regex support from the existing one
[17:48:28] <cdanis>	 right
[17:48:55] <cdanis>	 and yeah, splitting it feels like the least-bad option to me
[17:49:06] <rzl>	 yeah
[19:07:21] <wikibugs>	 10serviceops, 10Operations, 10Patch-For-Review, 10User-notice: Upgrade the MediaWiki servers to ICU 63 - https://phabricator.wikimedia.org/T264991 (10Krinkle)
[19:07:44] <wikibugs>	 10serviceops, 10Operations, 10Patch-For-Review: Upgrade the MediaWiki servers to ICU 63 - https://phabricator.wikimedia.org/T264991 (10Krinkle)
[19:13:06] <rzl>	 cdanis: assume it's documented either way, but check my instinct -- would you assume assert_body_regex is full-match or partial-match?
[19:13:51] <rzl>	 e.g. if the body is "<html><head><title>Hello and welcome to my haystack</title></head><body>needle</body></title>"
[19:14:00] <rzl>	 does "assert_body_regex: needle" pass?
[19:15:25] <cdanis>	 hmmmmm
[19:15:38] <cdanis>	 I would assume it is partial-match
[19:16:17] <cdanis>	 I don't think it matters a ton because you can always express one in terms of the other, but, that way keeps easy cases terser/DRY-er
[19:16:24] <rzl>	 so if you wanted that assertion to fail, you would expect to write "assert_body_regex: ^needle$"
[19:16:28] <rzl>	 yeah exactly
[19:16:37] <rzl>	 okay cool thanks
[19:19:07] <wikibugs>	 10serviceops, 10Machine Learning Platform, 10ORES, 10Okapi, and 3 others: ORES redis: max number of clients reached... - https://phabricator.wikimedia.org/T263910 (10calbon) @akosiaris Can you review it? I don't know enough about the nodes vs redis connection to intelligently review.
[19:35:18] <rzl>	 cdanis: mind if I send you the review for this?
[19:35:37] <cdanis>	 rzl: sure thing
[19:39:25] <rzl>	 📬
[20:03:14] <rzl>	 cdanis: thanks for the quick review! once this is merged, it gets updated by puppet, so you can run-puppet-agent wherever you want it, or wait 30m and it'll be everywher
[20:03:15] <rzl>	 e
[20:03:29] <cdanis>	 👍
[20:03:37] <cdanis>	 no rush, I'll be writing the testcase Monday
[20:03:56] <rzl>	 cool
[20:04:10] <rzl>	 I'll update wikitech now but I'm going to put off the assert_headers_regex update for another time
[20:04:27] <rzl>	 the update to existing test files, is what I meant but didn't write down
[20:04:48] <cdanis>	 yeah sounds good
[20:59:41] <wikibugs>	 10serviceops, 10Operations, 10Patch-For-Review: Upgrade the MediaWiki servers to ICU 63 - https://phabricator.wikimedia.org/T264991 (10jijiki) >>! In T264991#6608410, @Joe wrote: > For the ICU transition it's crucially important that no machine with write access to the databases gets updated before the date...
[21:25:38] <wikibugs>	 10serviceops, 10Operations, 10Patch-For-Review: Upgrade the MediaWiki servers to ICU 63 - https://phabricator.wikimedia.org/T264991 (10RLazarus) On a pointer from @joe (thanks!) I used the same basic approach as T86096#2325895 to identify wikis we'll need to touch with updateCollation.php.  {F32455073}  The...
[21:30:27] <wikibugs>	 10serviceops, 10Operations, 10CommRel-Specialists-Support (Oct-Dec-2020), 10User-notice: CommRel support for ICU 63 upgrade - https://phabricator.wikimedia.org/T267145 (10RLazarus) Yep, that text looks good to me -- the "eight of the ten biggest Wikipedias" language is the last thing I wanted to verify, an...