[10:12:06] !log admin Increasing the memory limit of osds in eqiad from 8589934592(8G) to 12884901888(12G) (T273851) [10:12:08] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [10:12:09] T273851: [ceph] Increase OSD ram limit to 12G - https://phabricator.wikimedia.org/T273851 [11:05:26] dear cloud users, I'm being lazy. Do you remember if I can ssh/scp to a tool account directly? i.e, `ssh mytool@login.toolforge.org` or `scp file mytool@login.toolforge.org:.` [11:18:01] arturo: I think you can only login as yourself [11:18:19] that's what I though ... [11:18:51] there was some cli tool that you can use to become the owner of some files you scp'd in, but I don't remember how it works [13:06:45] cat "myfile" | ssh login.toolforg.org 'bash -c "become mytool; cat > myfile"' or some similar hack? xd [13:12:29] perhaps, yes [16:19:07] Can any sysadmin tell me what VMs belong to the IP 185.15.56.1? [16:19:18] all of them [16:19:31] that's used for outbound NAT for all instances [16:19:36] Not toolforge though. [16:19:48] Toolforge apparently has .48 [16:20:35] hmm [16:21:06] It's assaulting the availability API on archive.org causing the server to error out on IABot. [16:21:06] .48 looks to be the toolforge login bastion which has a public ip [16:21:25] I think .1 is the general outbound nat ip which is used if the instance doesn't have a public ip [16:27:11] !log tools rebooting tools-package-builder-02 [16:27:15] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [16:27:19] Majavah: ok, good to know. A lot of requests are coming from that NAT IP and it's exceeding the rate limit by a huge margin. [16:27:53] I'm guessing the UA is some generic one? [16:28:32] The quick brown foks jumped over the fence. [16:29:49] wmcs people: do you have any solutions to that? ^ [16:30:12] complain on the cloud list? [16:31:25] not really, but we could try identify the source tool/bot [16:31:58] Nemo_bis: ^^ you? [16:34:01] One sec... IA sysadmin is running through the massive logs. [16:36:54] inb4 `python-requests/2.21.0 CPython/3.5.3 Linux/4.9.0-8-amd64` [16:38:06] list of VMs currently connecting to archive.org [16:38:09] https://www.irccloud.com/pastebin/joSrN6Nw/ [16:38:55] arturo: Thank you, any chance we can resolve those to host names? [16:39:42] https://www.irccloud.com/pastebin/pMZ89r1M/ [16:39:47] Cyberpower678: ^^^ [16:40:05] https://phabricator.wikimedia.org/P14208 [16:40:09] bah arturo was faster [16:41:04] so cyberbot, a few dumps instances and some toolforge exec nodes [16:41:16] So, we have IABot, which is expected, 4 dumps instances, and a bunch of shared tools. [16:41:17] Cyberpower678: I see many connections coming from sgeexec-0947, so this list of tooms may or may not be a valid smaller search scope [16:41:28] https://sge-status.toolforge.org/#host-tools-sgeexec-0947 [16:41:30] list of tools* [16:42:07] https://phabricator.wikimedia.org/P14209 [16:42:55] I suspect it's botwikiawk. Something there sounds familiar. [16:43:31] are crontabs public? [16:45:46] the job it's running on is fairly messy, but it's at least doing something related to archive.org [16:45:57] cat /data/project/botwikiawk/arcstat/arcstat [16:46:26] Yes. That's GreenC, he's my colleague. [16:48:53] Ok. I think we have what we need. [16:49:01] Thanks for the help. :-) [16:49:31] 🎉 [17:33:27] legoktm: not me ^_^ [17:33:39] Cyberpower678: lol intra-IA dos [17:33:41] sorry for the mis-ping then [17:34:07] no worries, I was happy to read this conversation [17:34:25] the dumps instances don't usually do anything with URLs [19:15:10] bstorm: I feel like I'm accidentally stumbling into a deeper rabbit hole...is there a reason updating meta_p doesn't happen automatically? isn't it all scripted? [19:52:26] legoktm: There have historically been tools that watch meta_p to detect when adding a wiki is complete. If it is automatic, then it would cause errors there (and I've been pinged about that many times). [19:52:34] That said, it is all scripted. [19:52:47] Separate scripts/hosts and now all in a notebook. [19:52:51] *cookbook [19:53:29] However, when we add something to DNS, it updates/repairs anything that has changed. When we add something to meta_p, it doesn't check every single database. [19:53:36] So there can be drift. [19:53:46] Plus meta_p and DNS are updated based on noc.wikimedia.org [19:53:54] gotcha [19:57:03] Overall, if you *really* want to be 100% getting it from the source, you go the noc route, since that's our source [19:57:33] I think we discussed putting the meta_p build on a timer somewhere in phab about a year ago... [19:57:50] yeah.... [19:58:09] That's less "heavy" on multiinstance than on the big converged ones [19:58:15] but as far as I know there has been literally 1 wiki with messed up data in meta_p in the last N years [19:58:29] We have moved some [19:58:48] But you can rebuild meta_p [19:59:21] legoktm: I'm also advising people based on the current state of operations, not what it could be :) [19:59:46] I honestly don't see much point in meta_p in the current layout. However, it works for some people. [20:00:08] There's nothing in there that isn't actually from another publicly accessible location [20:00:40] bstorm: yeah, I'm trying to figure out what I should be putting in my toolforge library after seeing what MA put in their symfony ToolforgeBundle [20:00:41] Especially since DNS and the action api also have that data, except the size...which is in noc [20:01:38] I mostly used it to join a section for getting the replica lag for a given wiki [20:02:40] can the DNS lookup be done locally? [20:06:31] I mostly use meta_p to get the list of all wikis more easily than the API [20:06:45] also replag [20:08:33] legoktm: it can be deom from inside Cloud VPS address space, but not from external networks AFAIK [20:08:38] *be done [20:09:12] AntiComposite: is the meta_p/replag connection just knowing the section number? [20:10:06] yeah [20:10:28] *nod* makes sense [20:13:03] from inside Cloud (toolforge or any other vps project) `host eswiki.web.db.svc.wikimedia.cloud` will return you the section number and the ip because the wiki db service names are all CNAME records pointing the the associated section's service name and io. [20:13:17] *ip [20:20:32] that's more work (from my end) than `SELECT lag FROM heartbeat_p.heartbeat WHERE shard = LEFT((SELECT slice FROM meta_p.wiki WHERE url = "https://en.wikipedia.org"), 2)` [20:21:54] most of my tools don't care about replag though, I mostly use meta_p instead of the sitematrix api [20:29:23] yeah, as far as I understand meta_p usage that is the main reason for it to exist [20:35:01] AntiComposite: just curious, what language is your tool in? [20:35:05] python [20:45:30] AntiComposite: it isn't actually more work for replag if you figure you can just: https://www.irccloud.com/pastebin/xFvIzUy9/ [20:45:45] You dont' need the shard name in the new model [20:45:54] Therefore, you don't need meta_p [20:46:06] The instance you connect to only knows it's own shard [20:47:21] Getting the list of wikis I can see [20:50:38] Depending on how you think about it, the splitting up of the instances may very well eliminate a lot of places where you actually have to check meta_p. For instance, if you want a list of what's on s5, you can really just connect to s5.analytics.db.svc.wikimedia.cloud and `show databases;` [20:50:51] https://www.irccloud.com/pastebin/o85UGocs/ [20:51:36] If you want a list of all wikis, connecting to s7 and checking meta_p is still going to be fastest [20:51:57] well, except maybe sitematix api [20:52:06] depending on how fast the replicas are being [20:53:53] I never thought about it, but the partitioning up of the replicas replaces some of the need for meta_p in general. [20:54:01] and for contacting noc.wikimedia.org [20:55:59] !log tools.lexeme-forms deployed 32b6b23f72 (German adverbs) [20:56:01] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.lexeme-forms/SAL [21:21:57] calling 'show databases;' seems like the most foolproof way tbh, and you don't have to contact anything externally [21:22:18] can't ever get out of sync either [23:16:35] I guess you could end up listing databases not holding wikis? [23:16:40] such as meta_p itself