[11:27:46] !log toolsbeta Creating instance toolsbeta-test-k8s-etcd5 and adding to the etcd cluster (T267140) [11:27:49] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL [11:27:49] T267140: [toolsbeta] Rebuild servers to learn how to take down the services without downtime - https://phabricator.wikimedia.org/T267140 [11:44:54] !log toolsbeta etcd5 member added, creating instance toolsbeta-test-k8s-etcd6 and adding to the etcd cluster (T267140) [11:44:58] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL [11:44:58] T267140: [toolsbeta] Rebuild servers to learn how to take down the services without downtime - https://phabricator.wikimedia.org/T267140 [13:22:54] | head [13:23:01] fg [13:23:05] jjlu:w [13:26:33] * dcaro is going to nuke terminator xd [16:51:51] matanya, re: tool-adoption - I'm not a dev, so I'm not familiar with the nuances of that. Up to you, or ask the others :) (Sorry for the delayed reply) [18:08:35] musikanimal: yeah you have to use the sssd images now. if the old image is still referenced in docs somewhere, we should update it [19:48:46] legoktm: just one reference to it on wikitech at https://wikitech.wikimedia.org/wiki/Help:Toolforge/Kubernetes#Node.js. I can update image name but not sure if the rest of that sentence still holds true [19:49:40] well it has node10 now, as the image name suggests [19:49:51] :) [19:50:47] maybe also worth a link to https://docker-registry.toolforge.org/ which has all the images listed [21:09:50] !log ores deleted six ores.wmflabs.org VMs for donwsizing it (T242819) [21:09:53] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Ores/SAL [21:09:53] T242819: ores.wmflabs.org - 503 icinga alerts - https://phabricator.wikimedia.org/T242819 [22:13:55] !log paws deploying new paws changes for multiinstance readiness [22:13:57] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Paws/SAL [23:03:40] hi [23:04:02] https://en.wikipedia.org/wiki/User_talk:ShakespeareFan00 [23:04:22] I have some cross database queries that will break in January due to the replicas design changes [23:04:37] Can you provided a way to ensure the queries do NOT stop working? [23:04:47] The queries are written in Quarry [23:05:07] https://quarry.wmflabs.org/query/18894 [23:11:35] Hi ShakespeareFan00. So far, I've been working hard to ensure that quarry will be able to operate when the wikireplicas go multi-instance at all https://gerrit.wikimedia.org/r/c/analytics/quarry/web/+/632804 [23:12:45] Making them capable of cross-database joins requires a database server to host both schemas at the same time. Right now, we have all the sections replicate to the replica servers like a huge funnel. The problem is that it causes crashes, lag and performance issues as the system grows. [23:13:27] These are limitations in mariadb, not our implementation of it. There are some 900 databases [23:13:42] and hundreds of users (at least) [23:15:01] So far, I cannot conceive of a way that quarry can join across database servers, as written, and the system cannot handle all databases in one server anymore. [23:15:36] It would need to become much more than just an SQL machine to do it. [23:18:40] You might be able to generate the same thing using PAWS and multiple connections in the future. As of now, the multi-instance system isn't operational yet. [23:18:57] We want to give people time to experiment and work with it. [23:23:25] bstorm : Well given the attitude I got in #technical on discord I am not going to bother [23:23:38] trying to fix this myself [23:24:01] Perhaps you can leave a not here explaining to the bot author what they will need to change? [23:24:02] Ok. I'm not on discord, so I don't have any context around that, unfortunately. [23:24:21] https://en.wikipedia.org/wiki/User_talk:ShakespeareFan00#Quarry [23:24:29] Sure, I can take a look! [23:24:33] If you can explain to that user much appreciated [23:24:44] I've already removed the relevant query from Quarry [23:24:59] The bot author will need to provide an replacement. [23:25:21] We are hoping to get some code samples and things up to help people. I know this is a big change for the way a lot of people did things. I did try to find ways to make it simpler, but I could not find one. [23:28:02] Annoyinglu the first I heard about this was the note on the linked talk page [23:28:14] A breaking change like this should be more prominent [23:31:20] We have sent announcements on the cloud-announce email list and there's a discussion going on in the cloud email list around it. It's been tricky because over time, I haven't seen any one channel that everyone receives reliably. As we get closer to getting the system up, we will probably get a blog post out as well. We are hoping to have everything moved to it in January with the two systems in parallel as we go. [23:31:32] This isn't a deal where it will just all stop working [23:32:08] If that helps [23:32:22] We hope to have both systems running in parallel for a fair while [23:33:23] ShakespeareFan00: do you get emails from the cloud-announce list? https://lists.wikimedia.org/mailman/listinfo/cloud-announce [23:33:29] Nope [23:33:40] I am not a user that does techncial stuff [23:33:45] Generally... [23:33:53] I wrote the now removed query some time ago [23:34:11] ( If you want the code for it, you'll have to ask an admin or check a backup) [23:34:17] Fair enough! We don't spam that list much, but you would get a lot of messages about "maintenance happening" if you subscribed. [23:35:18] I stated over on Discord that ": [23:35:18] Breaking long standing tools isn't on" [23:35:25] but the person I was talking to didn't get it [23:35:40] I do have the code...and it will keep working for quite a long time if you wanted to put it back for now. [23:35:48] I can't [23:36:01] There isn't a wau to revert an edit to a quarry query [23:36:14] If you know of one then you know more than I do. [23:36:51] https://www.irccloud.com/pastebin/aXzbcWRK/ [23:37:01] I did save what was on my screen 😁 [23:38:06] I can't revert it from here. I know you can add a new revision. [23:38:30] Did just that [23:38:36] 👍🏻 [23:38:37] bstorm: To be fair though... [23:39:01] Shouldn't something like checking for an Image on Commons be an API function? [23:39:25] Perhaps by checking for filenames and SHA1 hashes directly? [23:39:26] Are there any communication channels you'd recommend for communicating things like this? I'm not on any Discords right now, myself. [23:39:35] bstorm: See my talk page [23:39:37] Yes, things like that are probably better done with an API [23:39:41] 👍🏻 [23:39:49] Bot authors will need to now [23:39:53] knwo about this [23:40:14] If this was mentioned in a tech news I must have missed it [23:40:29] (Tech News post to talk pages/ VP etc..) [23:41:00] bstorm : I would suggest you join the Wikipedia/Wikimedia discord. [23:42:16] See:- https://en.wikipedia.org/wiki/Wikipedia:Discord [23:42:48] I won't name the user I had a dispute with, because they were acting in good faith :) [23:43:22] It wasn't. I have briefly talked to someone who writes tech news last year. I need to go check who they are. We almost always communicate on here and on the cloud-announce list on WMCS, which maybe isn't enough. We also have a tech blog now...but that maybe doesn't help if people are already using other channels. [23:43:50] We usually expect anyone who uses toolforge or WMCS services to follow cloud-announce [23:44:12] Quarry users arent necessarily technical [23:44:44] They may use that service without necessarily knowing it's a Labs/Clouds service in my opinion [23:45:09] And of course I accept that some breakage when technical changes happen is inevitable [23:45:18] bstorm: On a related note :- [23:45:45] https://quarry.wmflabs.org/query/25986 will break in the same way [23:46:27] I'm sure there's a lot. I need to run some checks on the quarry database and make a list of how many will break and why. [23:46:31] as will- https://quarry.wmflabs.org/query/20698 [23:46:54] bstorm: I've already disabled the ones I've linked so any checker won't see them [23:47:15] The disabling mechanism I use to comment out the query concerned. [23:47:27] *use is sorry [23:48:58] T267989 [23:48:58] T267989: Do some checks of how many queries will break in a multiinstance environment - https://phabricator.wikimedia.org/T267989 [23:49:29] bstorm: Commons duplicate finding is something that should be API based, or something that's done server side as a Database report. [23:49:43] I think there are some of those that might also break.. [23:50:05] because they are SQL based... Not sure how many of those use cross wiki queries though.. [23:50:05] Yeah. I figure it might be possible to get an overall idea of what people are doing with these queries and help to provide alternatives in our communications [23:50:43] bstorm: Presumably as part of the replicas work, code on indvidual wikis is also being examined? [23:50:56] Some like English Wikisource pull data from Wikidata/Commons. [23:50:57] As I said, nothing like this will break for some time. Not until after the quarry is migrated to the new system (or I merge my patch...which isn't ready yet anyway) [23:51:14] bstorm: I am thinking ahead [23:51:44] It would be tiresome if something like Wikidata import broke because of Replicas changes :rofl: [23:52:04] Those don't touch the replicas. [23:52:34] Good [23:52:47] These are purely a cloud service for Toolforge and things that run on Cloud VPS, so production wikis are only affected by what Toolforge tools and CLoud VPS things break. [23:53:01] That's good to hear [23:53:01] The production wiki systems are already segmented like this [23:53:06] Good [23:53:11] That's part of why we have to split up the replicas [23:53:45] bstorm: Also by splitting the replicas, certain types of creative query can't be perfomed anymore IIRC [23:54:04] Not that Quarry allowed you do anything but SELECT queries anyway [23:54:43] (Aside: Suggest someone confirms you can only do SELECT queries, as opposed to INSERT or DROP..) [23:55:07] ShakespeareFan00: confirmed. the db user grants are read-only [23:55:32] I think the replicas becoming more like segmented will force more people to use API calls, dumps and possibly the PAWS service in order to use code between queries to link things up. It won't be easy, but we do hope to help people get there. [23:57:07] ShakespeareFan00: People need to connect to ToolsDB or their own database in CloudVPS to do INSERT, etc. Some people already use that as a workaround for more complex queries against the replicas. A long time ago, you could do some of that in "labsdb", but not for quite a while. [23:58:14] bstorm: Would you be open to a phabricator ticket asking for a better way to do what my query was doing? [23:58:26] Naemly finding 'shadows' content on local wikis? [23:58:30] *Namely [23:58:41] The query approach isn't ideal.