[00:02:46] how do I count how many tools I have? [00:03:32] It's okay, I also forgot how many tools I've made. [00:03:41] Come up with a number that feels correct to you. [00:03:57] Your response is going to be aggregated anyway, so I don't think precision is that important. [00:05:16] I tried writing 50+ and then it said "Please enter a valid number." :< [00:05:43] You are allowed to round down and just say 50 [00:06:00] legoktm: you can figure it out from toolsadmin.wikimedia.org or by looking in LDAP directly [00:06:29] ok, I just wrote 55 and submitted it [00:07:04] legoktm: ldap tells me that you are a maintainer of 22 tools [00:07:35] that's not right [00:07:49] https://screenshots.firefox.com/8yGCnmThLOXCtRzT/toolsadmin.wikimedia.org [00:07:55] 22 gets me to the g's in that list ;) [00:08:42] * harej has a separate LDAP account to send abandoned tool accounts [00:08:50] interesting. My ldap metadata query must not be using pagination correctly [00:12:31] bd808: maybe it was last year, (time flies!) but I was right behind Yuvi and Magnus in tool counts [00:13:03] we should make a bus-factor leaderboard [00:13:43] :< [00:13:46] Toolhub will allow that [00:17:15] https://tools.wmflabs.org/admin/tools#!/maintainer/legoktm [00:17:19] That says 71 [00:18:37] oof [00:19:31] 45 for me, but at least a few of those are just name squatting :) [01:11:31] !log git removing the old ui ( https://gerrit-review.googlesource.com/c/gerrit/+/116790 ) from gerrit-new.wmflabs.org [01:11:33] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Git/SAL [04:16:29] !log wikistats switched backend of web proxy to wikistats-kraken (stretch upgrade), migrated db dump from jessie instance cowgirl, deleted extra webproxy [04:16:31] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Wikistats/SAL [10:46:04] HAHA https://wikitech.wikimedia.org/w/index.php?title=Help:Toolforge&oldid=68315 [10:46:44] legoktm: >> https://wikitech.wikimedia.org/w/index.php?title=Help:Toolforge&direction=next&oldid=68315 [10:47:37] looooool [10:47:40] so long ago [10:47:48] just love how that was the start of the docs [10:53:52] Just read it and it looks lovely. Coren was very busy back in those days :) [18:34:37] !log tools.fireflytools + Adithyak1997 - Matanya (per their request due to cron spam) to maintainer list of tools.fireflytools T209147 [18:34:40] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.fireflytools/SAL [18:34:40] T209147: Adoption request for Firefly Tools - https://phabricator.wikimedia.org/T209147 [19:29:34] zhuyifei1999_, matanya, chicocvenancio, are any of you around to talk about the 'video' project? (If not, I can send an email) [19:29:47] I think I am [19:30:03] so, I'm thinking about the region migration for this project... [19:30:10] those VMs are HUGE as I recall [19:30:15] yeah [19:30:33] so it would probably be better to build a new, parallel cluster in the new region and move the workload over [19:30:35] and then delete the old ones [19:30:40] rather than moving the VMs [19:30:45] does that seem possible/reasonable? [19:31:29] the stuff is unpuppetized. I there are scripts documented on wikitech that automates most of the setup, but still complicated [19:31:45] hm... [19:32:02] well, I certainly /can/ move them, it'll just be an all-day affair [19:32:12] is there a way to depool a single worker while it moves? [19:32:16] yeah [19:32:27] for the encoding0* ones [19:33:07] videodev & gfg01 are independent and can move any time (they are mostly still dev stuffs) [19:33:13] ok — so how about if I move encoding01 starting in a few minutes? And then we can make sure that the connectivity is still good [19:33:20] I'm guessing the other VMs aren't so big [19:33:24] video-redis is a single point of failure unfortunately [19:33:34] ok [19:33:45] let me see if I can depool it [19:33:50] yeah, there'll be downtime when -redis moves :( [19:33:51] thanks! [19:34:01] I'll make a tracking task for this [19:34:04] ok [19:36:21] what is your username on phab? [19:36:45] zhuyifei1999 [19:36:58] encoding03 seems to be idle and not pooled [19:37:15] great! I'll start the move now and see what the eta is [19:43:04] zhuyifei1999_: it predicts only a couple of hours to move; I'll ping you when it's done and up [19:43:09] k [19:56:48] zhuyifei1999_: I'll move videodev & gfg01 now too if that's harmless [19:56:58] ok [20:20:08] !log tools.wikibugs Updated channels.yaml to: 13c6f92210f849c0c1b5ec701f9a20b2d663ef04 Merge "Add maps items to Infrastructure team channel" [20:20:10] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.wikibugs/SAL [20:49:20] andrewbogott: you scheduled the migration of dwl to 29.11., right? [20:54:46] doctaxon: looks like it, unless I got confused and scheduled it twice [20:55:37] andrewbogott: well, so I need a time window for this migration [20:55:59] how long does the migration of dwl last [20:57:01] doctaxon: I can do it right now if you want :) [20:57:14] Duration depends on the number of instances and the sizes of those instances. It's hard to predict. [20:58:55] andrewbogott: can you check dwl and guess a duration please? [21:01:29] each of your three VMs is using about 20Gb of storage. So it might take as long as 90 minutes to move all three. [21:02:09] I can schedule that project to be moved first, which would be around 15:00 UTC [21:02:46] ah, sorry, I have a meeting then, let's say 16:00 UTC [21:02:50] (10AM CST) [21:03:02] andrewbogott: I have time bound scripts running all the time, 90 minutes are too much [21:03:29] maximum 30 minutes [21:03:34] Sorry, it takes as long as it takes. I can't change the physics. [21:03:45] so please don't migrate dwl [21:03:49] It's not optional [21:04:00] could you move each instance separately? [21:04:03] The only alternative is that you can create a parallel project in the new region and move your traffic over there [21:04:15] and then shut down the project in the older region [21:04:26] does moving them one at a time help? [21:06:23] andrewbogott: can you move these three projects one by one, though there are running on one instance? [21:06:46] I don't understand what you're asking, sorry [21:06:53] Are we talking about dwl still, or about other things? [21:07:01] doctaxon: it's three VMs/instances in one project [21:07:37] andrewbogott: the VMs one by one? [21:07:57] yes, can move them one by one. [21:08:16] wait a minute [21:10:35] zhuyifei1999_: encoding01 is in its new home and running now. Can you tell if it's still working or not? (Might have to change some security groups for it to talk to the old VMs, if it does that…) [21:11:35] can you move each VM on 17:15 UTC(!) tuesday 27th, wednesday 28th and thursday 29th please? [21:11:43] andrewbogott: ^ [21:15:32] andrewbogott: or better all three on 29th starting 17:15 UTC(!) but urgently starting with the VM taxonbot? [21:15:50] doctaxon: sure, hang on... [21:16:13] "hang on"? [21:16:24] hang on is a term used to say wait :) [21:16:41] thx [21:17:26] doctaxon, can you pm me your email address so I can add you to my gcal entry? [21:17:54] take animalia@gmx.net [21:19:57] andrewbogott: but I have to be able to rely on this migration start time [21:20:25] just check in with me on IRC on that day, we'll get it done [21:22:05] an email an hour before will be better [21:22:37] google should do that automaticall [21:22:40] automatically [21:22:42] and thank you very much indeed [21:23:15] sure thing [21:24:04] This is all very messy… but the server those VMs are on is leased an getting sent back to the vendor on December so it's this or sending our cold, dead VMs back to the manufacturer :( [21:26:09] huh [21:26:29] TIL some of the labvirt servers are rented [21:28:33] Only two [21:29:03] There was a brief period when all of Ops leased their servers, it was some kind of business decision made elsewhere in the org [21:29:35] Which is fine in almost all cases (where we have clusters of interchangeable boxes) but it's not ideal for labvirts :( [21:38:27] andrewbogott: and please stop and start one VM after the other [21:39:01] andrewbogott: I get WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED! [21:40:58] zhuyifei1999_: that surprises me a bit since the contents of the VM is the same, including the host key... [21:41:03] but the IP is different [21:41:35] encoding02 reports encoding01 not pooled [21:41:53] do you know what port it uses to coordinate? [21:42:08] port? no, it connects to redis [21:42:14] on video-redis [21:42:37] ok, looks like redis is port 6379 [21:42:41] I'll open that, hang on... [21:43:29] ok, let's see if that helps... [21:43:34] you might need to nudge it to reconnect [21:43:55] I'm not sure if I should connect, given the ssh host key changed [21:44:48] zhuyifei1999_ that's expected i guess. [21:44:51] as the ip changed [21:44:52] I can do it — do you know what the service name is? [21:46:32] it should be something like v2ccelery [21:46:34] actually, hang on a minute, I need to fix up something with the domain on this host [21:47:00] I see it up [21:47:26] ok, and I the hostname looks right now [21:47:29] so, all good I think [21:48:14] I guess I'll reset the known hosts then [21:48:23] well, let me reboot it first [21:48:29] in case the screwed up hostname was the issue here... [21:49:54] ok, try now? [21:50:45] zhuyifei1999_: still get a host key warning now? (I'd expect yes but I'm curious) [22:01:22] andrewbogott: I reset it pre-reboot, loaded the new keys. after reboot no warning [22:01:29] ok [22:01:31] I guess the key changed permanantly [22:01:40] Want to depool 02 so I can move that? [22:01:41] sorry I'm off-and-on right now [22:01:54] no worries, I'm multi-tasking too [22:01:59] can you do 03? it's not pooled [22:02:03] sure [22:02:16] I'll depool 02 tonight [22:02:25] cool [22:02:26] thanks! [23:08:55] @access [23:09:09] I trust: bd808!.*@wikimedia/BDavis-WMF (2admin), .*@wikimedia/andrew-bogott (2admin), .*@wikimedia/mviswanathan-wmf (2admin), .*@wikimedia/.* (2trusted), .*@mediawiki/.* (2trusted), .*@wikipedia/.* (2trusted), [23:09:09] @trusted [23:09:27] hm [23:11:30] bd808: could you try "@relay-on" in here? I'd like to get https://meta.wikimedia.org/wiki/Wm-bot#Relay_of_messages_from_scripts_and_other_tools working so I can have my tools log themselves [23:12:11] @relay-on [23:12:11] Relay is already enabled [23:12:25] legoktm: ^ busted feature? [23:12:32] hm [23:12:38] the other thing I think is that someone already set a token? [23:12:51] legoktm@upgrader-04:~$ echo "#wikimedia-cloud test" | nc wm-bot2.wm-bot.eqiad.wmflabs 64834 -w2 [23:12:51] ERROR4 (invalid token): #wikimedia-cloud :test [23:13:07] @token-off [23:13:07] This channel will no longer require a token in order to relay messages into it [23:13:11] test [23:13:18] woot [23:13:48] is it ok if we don't have a token? I assume the abuse potential is low since I think all requests need to come from inside cloud services [23:14:06] wololololo [23:14:34] no ok if a lot of crap messages start coming in from the bot :) [23:15:09] alternatively we just give lego the token :D [23:15:14] wonder if this thing is Sigyn exempt [23:15:48] my idea was to create a git hook that on pull would send off a !log message [23:16:04] the wm-bot project doesn't have any floating ips right now, so it should be effectively firewalled to internal cloud vps traffic [23:16:23] even if it did you could restrict it to only internal traffic at the security group [23:16:45] or iptables [23:18:43] legoktm: give it a shot. If it turns out to be a target of abuse we can think up a different system that is less restrictive than the single token security model that mw-bot seems to provide [23:19:26] could always run your own ircecho