[00:00:16] thanks you too [00:32:25] 10serviceops, 10Operations, 10Thumbor, 10Patch-For-Review, and 2 others: Assess Thumbor upgrade options - https://phabricator.wikimedia.org/T209886 (10kaldari) @jijiki - Is there a way I can test it, other than locally? [06:32:44] 10serviceops, 10Operations, 10TechCom-RFC, 10Wikidata, and 5 others: New Service Request: Wikidata Termbox SSR - https://phabricator.wikimedia.org/T212189 (10Krinkle) [06:54:20] 10serviceops, 10Operations, 10TechCom-RFC, 10Wikidata, and 5 others: New Service Request: Wikidata Termbox SSR - https://phabricator.wikimedia.org/T212189 (10Krinkle) This task proposes a significant change to software architecture and should follow the [RFC process](https://www.mediawiki.org/wiki/Requests... [07:50:40] 10serviceops, 10Operations, 10TechCom-RFC, 10Wikidata, and 5 others: New Service Request: Wikidata Termbox SSR - https://phabricator.wikimedia.org/T212189 (10akosiaris) >>! In T212189#4833536, @daniel wrote: > "We should not introduce a service that is called by MediaWiki, and itself calls MediaWiki." Sl... [08:05:38] 10serviceops, 10Operations, 10Thumbor, 10Patch-For-Review, and 2 others: Assess Thumbor upgrade options - https://phabricator.wikimedia.org/T209886 (10Gilles) If the files are already on Beta, purge them, make sure your browser cache is cleared for these images, and you'll get thumbnails generated with lib... [09:49:27] akosiaris (and others) I was thinking about kubernetes authentication, we are using tokens right now but a more flexible and easy way could be integrating LDAP using and authentication webhook like in this video https://www.youtube.com/watch?v=ivVqZt8w1K8 [09:51:18] fsero: I 've been thinking a bit too about it. It does have quite some appeal (not having to deal with tokens ourselves is great) but it does also differ from the way we currently authenticate and authorize deployers (via puppet and the admin module) [09:51:49] * akosiaris takes a look at the video [09:57:44] I need to take a look into the admin module [10:44:21] <_joe_> fsero: yeah it's an interesting module indeed. Nowadays it could probably be rewritten making use of modern puppet constructs and native resources [10:46:57] <_joe_> And tbh now that I think of it, I try to avoid coupling critical systems to ldap availability. So we'd need a fallback for emergencies [10:48:11] <_joe_> Sorry I'm on the phone, I'll articulate more once I'm back at my keyboard [10:56:59] there's a lot more coming on the LDAP/authn/authz front in the coming quarters [10:58:47] it's still early and we haven't made any decisions, but I don't think relying purely on LDAP for authn is going to be something that we'll continue doing for much longer [11:20:26] i do agree to having a fallback mechanism if LDAP is down, but we need to rely authentication on something [12:14:03] <_joe_> fsero: sure, I'm saying for critical infrastructure I want a super-user to be able to act on the cluster even when the remote auth provider is unavailable [12:14:13] <_joe_> be it LDAP or $authn_service [12:16:34] 10serviceops, 10Operations, 10TechCom-RFC, 10Wikidata, and 5 others: New Service Request: Wikidata Termbox SSR - https://phabricator.wikimedia.org/T212189 (10mobrovac) [12:16:42] in k8s that is x509 admin certificates [12:16:44] :) [12:17:03] we had certificates as a fallback, SAML integration as main authn mechanism [12:37:31] <_joe_> SAML [12:37:33] <_joe_> oh my [12:43:00] 10serviceops, 10Operations, 10Thumbor, 10Patch-For-Review, and 2 others: Assess Thumbor upgrade options - https://phabricator.wikimedia.org/T209886 (10jijiki) @kaldari there is a thumbor debug log on `deployment-imagescaler03` under `/var/log` which was generated as I was testing https://commons.wikimedia.... [13:07:28] 10serviceops, 10Operations, 10TechCom-RFC, 10Wikidata, and 5 others: New Service Request: Wikidata Termbox SSR - https://phabricator.wikimedia.org/T212189 (10daniel) @Milimetric wrote: > In the simplest case, this code would be almost identical client and server-side. No matter where it's running, nodejs o... [13:18:38] 10serviceops, 10Operations, 10TechCom-RFC, 10Wikidata, and 5 others: New Service Request: Wikidata Termbox SSR - https://phabricator.wikimedia.org/T212189 (10mobrovac) >>! In T212189#4837768, @daniel wrote: > But for the case at hand, there might be a workaround: the PHP code that renders the (Wikibase Ent... [13:22:32] can't remember if we have already a task open for restbase logging 10x more when a single cassandra host is down? [13:24:39] anyways, I'm opening one [13:33:56] 10serviceops, 10Operations, 10TechCom-RFC, 10Wikidata, and 5 others: New Service Request: Wikidata Termbox SSR - https://phabricator.wikimedia.org/T212189 (10daniel) >>! In T212189#4837780, @mobrovac wrote: > When rendering the page, `index.php` knows the exact data that needs to be rendered already, corre... [13:39:13] 10serviceops, 10Operations, 10TechCom-RFC, 10Wikidata, and 5 others: New Service Request: Wikidata Termbox SSR - https://phabricator.wikimedia.org/T212189 (10mobrovac) >>! In T212189#4837824, @daniel wrote: > That works, but defies the purpose. The idea is to present a default rendering to clients that don... [13:55:20] 10serviceops, 10Operations, 10TechCom-RFC, 10Wikidata, and 5 others: New Service Request: Wikidata Termbox SSR - https://phabricator.wikimedia.org/T212189 (10daniel) > When rendering the page, index.php knows the exact data that needs to be rendered already, correct? I just had a brief chat with @Jakob_W... [13:58:09] 10serviceops, 10Operations, 10TechCom-RFC, 10Wikidata, and 5 others: New Service Request: Wikidata Termbox SSR - https://phabricator.wikimedia.org/T212189 (10daniel) > You first need to render the page on the server before you know whether the client supports JS/SW or not, so it will need to be rendered on... [14:42:02] mmm _joe_ i do see that our docker module in puppet repo it actually contains managing docker-engine, creating some base docker images and the docker-registry. What is the motivation for that? it seems it will be clearer AIUI if there is one module for docker-engine and maybe base docker images and another one for registry. So do you think its a good idea to split them? [14:44:51] <_joe_> well it's just a module collecting different docker-related functionalities [14:45:13] <_joe_> So while I think the base images probably don't belong there, I think the registry can be just a sub-namespace [14:45:19] <_joe_> so docker::registry::* [14:45:39] <_joe_> and docker::engine::* as well [14:46:16] <_joe_> but I don't think it really makes so much of a difference, so whatever you feel better doing is ok :) [14:48:42] <_joe_> akosiaris, mobrovac I edited https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/478637 (role::beta::docker_services) [14:48:54] <_joe_> following your suggestions [14:49:18] <_joe_> I think I'll test it with otto once eventgate is ready to deploy in beta next q [15:00:16] 10serviceops, 10Release Pipeline, 10Release-Engineering-Team (Kanban): Allow access to blubberoid.discovery.wmnet:8748 - https://phabricator.wikimedia.org/T212251 (10akosiaris) >>! In T212251#4835130, @dduvall wrote: >>>! In T212251#4834349, @akosiaris wrote: >>>>! In T212251#4834325, @hashar wrote: >>>>>! I... [15:01:36] _joe_: missed a comment [15:01:43] about -d and -r ? [15:01:52] <_joe_> oh maybe [15:01:54] <_joe_> sorry [15:02:12] <_joe_> lemme find it and fix it [16:14:36] 10serviceops, 10Operations, 10TechCom-RFC, 10Wikidata, and 5 others: New Service Request: Wikidata Termbox SSR - https://phabricator.wikimedia.org/T212189 (10Addshore) [16:14:45] 10serviceops, 10Operations, 10TechCom-RFC, 10Wikidata, and 5 others: New Service Request: Wikidata Termbox SSR - https://phabricator.wikimedia.org/T212189 (10Addshore) There sure has been a fair amount of discussion on this ticket! So I have created an updated interacting diagram showing off a few more de... [16:32:08] 10serviceops, 10Operations, 10TechCom-RFC, 10Wikidata, and 5 others: New Service Request: Wikidata Termbox SSR - https://phabricator.wikimedia.org/T212189 (10Milimetric) >>! In T212189#4838090, @Addshore wrote: >>>! In T212189#4835359, @Milimetric wrote: >> Now, I started looking through the code and it lo... [16:56:15] 10serviceops, 10Operations, 10TechCom-RFC, 10Wikidata, and 5 others: New Service Request: Wikidata Termbox SSR - https://phabricator.wikimedia.org/T212189 (10Jakob_WMDE) >>! In T212189#4838123, @Milimetric wrote: > My question here was more, how **can** the client render everything it needs, when some of t... [17:07:42] 10serviceops, 10Release Pipeline, 10Release-Engineering-Team (Kanban): Allow access to blubberoid.discovery.wmnet:8748 - https://phabricator.wikimedia.org/T212251 (10thcipriani) >>! In T212251#4837963, @akosiaris wrote: > There is however a 3rd overarching use case/requirement which is to allow developers to... [17:14:40] 10serviceops, 10Release Pipeline, 10Release-Engineering-Team (Kanban): Allow access to blubberoid.discovery.wmnet:8748 - https://phabricator.wikimedia.org/T212251 (10akosiaris) >>! In T212251#4838202, @thcipriani wrote: >>>! In T212251#4837963, @akosiaris wrote: >> There is however a 3rd overarching use case... [17:17:26] are we ok with this mixing of prod and wmcs? (relforge in prod allows connections from labs and now also wants them from contint prod) https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/479567/1/hieradata/role/common/elasticsearch/relforge.yaml [17:18:22] mutante: there's quite some discussion already in the task. godo.g has been looped in as well [17:18:39] akosiaris: i know, i added him :) [17:18:45] it makes me cringe tbh [17:19:03] the new request is "contint in prod to relforge in prod" [17:19:13] but the existing thing is "wmcs already connects to relforge" [17:19:23] and yea. .the way that it says wmcs offered the machines [17:19:57] i pinged moritz about the ferm / mixing realms part [17:46:47] 10serviceops, 10Operations, 10TechCom-RFC, 10Wikidata, and 5 others: New Service Request: Wikidata Termbox SSR - https://phabricator.wikimedia.org/T212189 (10Milimetric) Thanks @Jakob_WMDE, I think we're saying the same thing in slightly different terms, and it's because I'm not being precise. It's ok for... [17:49:32] 10serviceops, 10Core Platform Team (Session Management Service (CDP2)), 10Core Platform Team Kanban (Doing), 10User-Clarakosi, 10User-Eevans: Plan/design a session storage service - https://phabricator.wikimedia.org/T206015 (10Eevans) [18:04:55] re more than one process per pod , not impossible just makes debug things harder [18:06:49] in any case this https://kubernetes.io/docs/tasks/administer-cluster/cpu-management-policies/#cpu-management-policies could be interesting for us [18:06:56] specially for CPU bounded things [19:23:31] _joe_: https://gerrit.wikimedia.org/r/#/q/topic:apache-httpd+(status:open+OR+status:merged) puppetmaster, noc, simplelamp and then it's going to be "delete apache" module [19:24:09] the simplelamp part is going to be annoying because a bunch of users in wmcs somewhere [19:24:23] the puppetmaster part i'll try to tackle next then [19:24:38] just finished with all the analytics and ci ones [19:25:36] and noc is blocked on mediawiki module [19:26:59] akosiaris: moritz acked the ferm change is ok, fwiw [19:52:47] 10serviceops, 10Release Pipeline, 10Release-Engineering-Team (Kanban): Allow access to blubberoid.discovery.wmnet:8748 - https://phabricator.wikimedia.org/T212251 (10thcipriani) p:05Triage→03Normal Discussed in today's deployment pipeline meeting. Conclusion was that we would like to open this service u... [20:02:55] 10serviceops, 10Operations, 10Thumbor, 10Patch-For-Review, and 2 others: Assess Thumbor upgrade options - https://phabricator.wikimedia.org/T209886 (10kaldari) @jijiki - The first test seems to be hugely improved: On beta cluster: {F27689226} On Commons: {F27689222} It is frustrating though that the kernin... [22:47:37] installed php5* and nodejs package upgrades on phab prod machine as advised by moritz (he already did the others) [23:17:09] 10serviceops, 10Operations, 10Patch-For-Review: docker-registry.wikimedia.org caches images missing instead of revalidating - https://phabricator.wikimedia.org/T211719 (10akosiaris) 05Open→03Resolved a:03akosiaris With the merge of the above, this is probably resolved for now. Note that newly pushed im...