[05:52:24] Hi all! I'm on a 5 GHz Wi-Fi connection which has been having up to 30% packet loss lately. English Wikipedia's "Preview" button seems to hardly ever work for me. Why not? Isn't TCP supposed to be able to work around packet-loss problems? [05:55:43] unforgettableid: hi, can you follow the steps on https://wikitech.wikimedia.org/wiki/Reporting_a_connectivity_issue ? also #wikimedia-operations is a better IRC channel for reporting problems like this [06:00:01] legoktm: Noted. I suspect that the problem is with my own WLAN or ISP, not with anything on the WMF's end. [06:00:01] legoktm: I've also been having problems loading other websites, such as bikesharemap.com. [06:00:01] legoktm: I think my question is really more about TCP networking theory than about how to fix my actual problem. [06:00:44] I'm not sure this channel can really help you with that [06:01:00] in any case, you can still lose packets over TCP, if that's your question [06:01:42] legoktm: oh. I think that _was_ my question. [06:01:42] legoktm: but shouldn't retransmission be able to cope with 30% packet loss? [06:02:51] legoktm: I would like that the packets should still eventually arrive, just slower. [06:02:51] legoktm: If the Web browser is set to offer a generous timeout limit (e.g. 60 seconds), that provides lots of time for lots of retransmit attempts. [06:02:53] I don't know [06:03:03] I'm sure there's a networking channel on freenode that could help you better [06:03:03] legoktm: OK; maybe I'll ask ##networking instead. [13:34:30] I'm having problems scping stuff using ProxyJump over primary.bastion. I'm allowed to ssh -J, but suddenly I can't scp -o "ProxyJump. [13:34:33] https://pastebin.com/vBGJPrd3 [13:34:52] Anyone that have an idea of what might be going on here? [14:11:40] kalle: I haven't used scp w/ProxyJump before but (at least on my system) scp observes my .ssh/config. So if you can ssh directly to wikispeech-tts-dev already then I'd expect scp to just work. [16:03:09] Ok so who wants to talk toolforge? [16:03:23] @kb BryanDavis-WMF [16:09:20] andrewbogott: Yeah, me too :C [16:40:59] Hello cloud people! I need some assistance. I am trying to get a new metric added to graphite. The purpose is to track the sizes of Jenkins workspaces (T258626). Based on https://wikitech.wikimedia.org/wiki/Graphite and experimentation I landed on using `echo "jenkins.job.mwext-php72-phan-docker.workspace-size 1126108 $(date +%s)" | nc -q0 cloudmetrics1002.eqiad.wmnet 2003`. However the new metric never shows [16:40:59] up in https://graphite-labs.wikimedia.org/ Please advise. Thanks! [16:41:00] T258626: Collect samples of workspace sizes on integration-agent-docker-* nodes - https://phabricator.wikimedia.org/T258626 [16:49:24] dancy: I just made a "bd808.test" metric by doing functionally the same thing although I used telnet rather than nc. So I guess one thing to work out is if there are some firewall restrictions or something that are blocking your test packets [16:49:58] hmm. interesting. [16:50:15] dancy: here's what I did to make my test -- https://phabricator.wikimedia.org/P12030 [16:52:20] bd808: Hmm.. ok. I ran from `integration-agent-docker-1001`. The connection is definitely being established. I just retried using telnet to see what happens. Thanks for testing! [16:58:18] I was able to add a datapoint to your metric. [16:59:01] dancy: interesting. but still not able to put one in the path you wanted? [16:59:33] Correct. I tried prefixing it with `dancy.` in case there are some kind of namespace restrictions... that didn't do the trick. [17:02:18] @kb mw-bot6 [17:02:19] Sorry but I don't see this user in a channel [17:02:31] @kb wm-bot6 [17:02:32] Sorry but I don't see this user in a channel [17:03:22] what's that about? [17:03:29] dancy: Troll. [17:03:32] an jerk who has been trolling [17:03:34] He's a WMF banned user [17:03:49] bd808: If you have flags in here you may want to set +t. [17:04:17] It's been a headache for months. He has hundreds of puppets blocked [17:04:48] Just so you know: he uses IPs from Spain, although now he also uses proxies [17:04:58] That's too much free time [17:05:19] Absolutely. [17:05:45] cteam: I have set /mode +t (topic lock) here for a bit :/ [17:07:27] (You should also replace the old topic) [17:08:09] I did? [17:08:28] I still see the one he left [17:09:07] MadriCR: Type /topic? [17:09:14] I see the new one [17:09:16] hey, we have slight problem with .wmflabs.org domain. We just released new service - wcqs-beta.wmflabs.org. We have a web proxy set up to this address, but if for some reason service is down, this address is permanently redirected to wcqs-beta.wmfcloud.org. Once the service is up, original address works again, but if some user tried to access it during the outage, user's browser probably cached 301. [17:09:23] Yup, my bad, sorry [17:09:45] Is there a an approved way of dealing with that? (other than creating reverse 301 redirect) [17:10:22] zpapierski_: advertise the wmcloud.org hostname. Problem solved [17:12:59] wmflabs.org isn't supported anymore? [17:14:17] Toolforge went from "tools.wmflabs.org" to "toolforge.org" recently. [17:14:20] the redirect to wmcloud.org should only happen if there is a named proxy for it, but maybe there is a bug in that logic. The redirect is happening because of how we have set things up for the wmcloud.org transition for sure [17:14:39] Texas: this is a different proxy layer, but sort of related [17:14:49] Ah [17:15:48] zpapierski_: so I do see a configured wcqs-beta.wmflabs.org and not a configured wcqs-beta.wmcloud.org proxy so I think the 301 you are seeing when the backend for the wcqs-beta.wmflabs.org proxy is down is a bug [17:16:28] if I move to wmfcloud.org, that bug is actually quite useful :) [17:16:48] what we intended to setup was 301 redirect from $X.wmlabs.org to $X.wmcloud.org only when there is no configured proxy for $X.wmflabs.org but there is for $X.wmcloud.org [17:16:49] :zpapierski_ Hey. I watched your talk yesterday. I used to work for a company that developed an RDF database (AllegroGraph) [17:17:26] zpapierski_: so I guess we should open a phab task and figure out why the redirect is being over eager [17:18:31] dancy: I'm familiar with AllegroGraph. Or was, few years back [17:19:20] unfortunately, our open source rules stand (at least until we are sure, that there isn't available open source solution) [17:20:05] Nod. Makes sense. [17:20:30] I'm not surprised that many RDF/SPARQL solutions are closed source, getting this right in terms of scalability is painfully difficult [17:20:46] It is very difficult. [17:21:14] zpapierski_: T258730 [17:21:15] T258730: Cloud proxy redirect from $X.wmflabs.org to $X.wmcloud.org happening without a configured wmcloud.org proxy record - https://phabricator.wikimedia.org/T258730 [17:21:40] bd808: thanks! Although, I will need to set up redirect myself once you fix that :) [17:21:55] when is transition suppose to happen, btw? [17:22:31] any time, we haven't started a forced migration process, but things are in place to allow anyone to move their proxy to the new domain [17:22:47] ok, thanks for the info [17:29:46] zpapierski_: when you say "if for some reason service is down", are you doing anything at all to change the proxy target using the Horizon dashboard? Or is this actually triggered by the response (or lack of response) that the upstream server sends back to the proxy? [17:29:58] * bd808 is reading the code and trying to figure out how this might happen [17:30:20] zpapierski_: when you said the proxy was redirected because the service was down, can you tell me what you mean by 'down'? [17:30:32] heh, same question as bd808 [17:30:41] :) great minds andrewbogott [17:31:11] looking at the LUA, I'm not sure why this would happen if the proxy record was static [17:31:30] right, I'm hoping 'down' means 'didn't yet exist' [17:32:16] I'm wondering about making the lua a bit smarter though. Changing the logic to only redirect if there is a configured $name.wmcloud.org record [17:33:07] that would be fine [17:33:18] and less lazy than not dong that :) [17:33:21] *doing [17:33:32] your hack was so nice and short though :) [17:40:33] andrewbogott: I'm verifying this more precisely - it didn't happen to me [17:40:45] thanks [17:41:50] there was a situation that web proxy was removed for a few hours, so potentially, but that was 2 days ago, and gehel reported issue today. But if, by chance, that was the time of the last interaction, that could be correct behaviour [19:23:07] On https://admin.toolforge.org/tool/coverme I see " No .description or toolinfo.json found " so I went to https://toolsadmin.wikimedia.org/tools/id/coverme where I saw a panel with a "+ new toolinfo" button. I now realize that the panel that was there was itself already a toolinfo, and now I have two of them. [19:23:14] Is there a way to delete the new "sub" toolinfo? [19:25:03] Krinkle: T198114 -- not yet, no [19:25:03] T198114: Allow tool maintainers to delete toolinfo via ToolsAdmin - https://phabricator.wikimedia.org/T198114 [19:28:14] bd808: I see, ok. [19:28:27] bd808: so do I need to do something else to make the toolinfo "real"? [19:28:45] the info from https://toolsadmin.wikimedia.org/tools/id/coverage seems to match on https://admin.toolforge.org/tool/coverage [19:28:50] but for coverme it doesn't [19:29:26] the reason I care is that when I go to cover-me.toolforge.org it (very nicely) forwards the user to https://admin.toolforge.org/tool/coverme where they would then ideally be able to find the webservice and open it [19:31:12] Krinkle: I'm not spotting any problems looking at https://toolsadmin.wikimedia.org/tools/id/coverme/info/id/340/edit. There is kind of a twisty and confusing code path to walk in the admin tool to try and see why it would reject a toolinfo record. [19:31:44] Krinkle: I can look into it deeper, but not right now. Phab task? [19:31:57] ok [19:32:07] but you would generally expect these two to be connected yes? [19:32:32] I did note that one talks about a file called toolinfo.json which I'm guessing is still supported but the tooladmin way is generally working? [19:33:11] yes, toolsadmin (striker) publishes those records to Hay's directory and then the admin tool reads the data from Hay's to augment any .description data that a tool may also publish [19:33:40] ok [19:35:58] the dump as toolinfo.json records is https://toolsadmin.wikimedia.org/tools/toolinfo/v1/toolinfo.json [19:36:04] filed https://phabricator.wikimedia.org/T258744 [21:06:44] !log paws pushing dbproxy docker image for new cluster into main quay.io repo T211096 [21:06:47] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Paws/SAL [21:06:47] T211096: PAWS: Rebuild and upgrade Kubernetes - https://phabricator.wikimedia.org/T211096 [21:08:28] !log paws pushing quay.io/wikimedia-paws-prod/paws-hub:newbuild to the main repo T211096 [21:08:30] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Paws/SAL [21:09:53] !log paws pushing quay.io/wikimedia-paws-prod/singleuser:newbuild to the main repo T211096 [21:09:54] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Paws/SAL [21:11:44] !log paws pushing quay.io/wikimedia-paws-prod/deploy-hook:newbuild to main repo T211096 [21:11:46] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Paws/SAL [21:14:36] !log paws pushing quay.io/wikimedia-paws-prod/nbserve:newbuild to main repo T211096 [21:14:39] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Paws/SAL [21:14:39] T211096: PAWS: Rebuild and upgrade Kubernetes - https://phabricator.wikimedia.org/T211096 [22:48:50] !log paws tagged the newbuild tags with "latest" to set sane defaults for all images in the helm chart T211096 [22:48:53] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Paws/SAL [22:48:53] T211096: PAWS: Rebuild and upgrade Kubernetes - https://phabricator.wikimedia.org/T211096 [22:51:38] !log paws deploying via the default 'latest' tag in the new cluster T211096 [22:51:40] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Paws/SAL [23:59:03] !log tools.openstack-browser Deploy 6a76383 (T258567) [23:59:07] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.openstack-browser/SAL