[00:21:50] shipped re-weighting of NS_HELP in wikitech search, the one query i was annoyed with is fixed at least :) [00:23:13] ebernhardson: thanks! [00:26:03] can't remember the specifics of what you had mentioned earlier... what was the issue? [04:53:54] https://wikitech.wikimedia.org/w/index.php?title=Special:Search&limit=500&offset=0&profile=all&search=irc+cloak&advancedSearch-current={} [04:53:54] Internal error [d5d4f322-9d69-44ea-a6b1-bee52852fdae] 2020-02-11 04:52:07: Fatal exception of type "WMFTimeoutException" [06:37:06] Any idea what the point of this edit was? https://wikitech.wikimedia.org/w/index.php?title=Server_Admin_Log&curid=7919&diff=1854656&oldid=1854655 [08:02:03] DSquirrelGM: perhaps reverting this edit: [08:02:06] https://wikitech.wikimedia.org/w/index.php?offset=20200205131730&title=Server_Admin_Log&action=history [10:44:20] !log tools.openstack-browser restart webservice to create new ingress object for toolforge.org [10:44:22] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.openstack-browser/SAL [12:40:00] !log admin [codfw1dev] delete unknown subnet pool 'cloudinstancesb-v4-pool0': `root@cloudcontrol2001-dev:~# openstack subnet pool delete d23a9b88-5c3d-4a53-ab88-053233a75365` (T244851) [12:40:03] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [12:40:05] T244851: Neutron: replace NAT customization with address scopes - https://phabricator.wikimedia.org/T244851 [12:40:44] !log admin [codfw1dev] delete unknown address scope 'wmcs-v4-scope': `root@cloudcontrol2001-dev:~# openstack address scope delete 078cfd71-117b-4aac-9197-6ebbbb7dd3de` (T244851) [12:40:46] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [13:46:48] !log admin [codfw1dev] creating some neutron objects to investigate T244851 (subnets, subnet pools, address scopes, ...) [13:46:51] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [13:46:51] T244851: Neutron: replace NAT customization with address scopes - https://phabricator.wikimedia.org/T244851 [16:45:25] So i set up a tool and i have it set so any maintainer of tools.zppixbot is a maintainer of the new tool but yet they cant access it any idea why [16:47:23] It also appears it only shows on https://toolsadmin.wikimedia.org/tools/id/zppixbot-test not on the https://tools.wmflabs.org/admin/tool/zppixbot-test [16:49:39] I'm in tools.zppixbot and get the below when trying to become (how we knew) [16:49:42] rhinosf1@tools-sgebastion-07:~$ become zppixbot-test [16:49:42] You are not a member of the group tools.zppixbot-test. [16:49:42] Any existing member of the tool's group can add you to that. [16:50:41] seems like you have to become zppixbot then become zppixbot-tets [16:50:44] test* [16:56:00] RhinosF1: per https://toolsadmin.wikimedia.org/tools/id/zppixbot-test, that is correct. Adding a tool as a maintainer of another tool is a weird thing that does not work to extend transitive membership. [16:56:19] bd808: thx [16:56:33] now just to try and get my head round scp again or fix my ftp client [16:56:44] or find a better one [17:00:31] the best scp client is `git pull` :) [17:00:52] bd808: it's copying private files for a one off [17:03:39] bd808: any idea of a mac ftp client that will support this and be free [17:04:31] RhinosF1: hmmm... I just use scp from the command line. I have never been into gui tools for (s)ftp [17:05:22] bd808: scp from my computer up gets permission denied (publickey,hostbased) [17:07:47] I would need more details to help debug that. I scp to my user's $HOME and ssh in to move things to a tool's $HOME, but I also do that very very infrequently myself. [17:08:01] yeah [17:08:14] I've told Zppix to add me the normal way [17:11:02] Hey there, anyone have any guidance on how to up the max size for a GET request for a web-based tool I implemented via tools.wmflabs.org and Kubernetes? [17:12:17] I am pretty sure I have upped the uwsgi buffer size properly, but now I think it's running into the GET size limit for the web server [17:31:56] !log wikibugs reboot because it wasn't reporting from phab [17:31:57] Reedy: Unknown project "wikibugs" [17:31:57] Reedy: Did you mean to say "tools.wikibugs" instead? [17:32:03] !log tools.wikibugs reboot because it wasn't reporting from phab [17:32:05] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.wikibugs/SAL [17:32:47] fuzheado: if you are hitting a GET limit that is probably enforced by the shared front proxy. It should be somewhere around 4K I think? [17:35:19] fuzheado: is your client getting a 414 response code? Or something else? [17:35:59] Correct - just over 4k and the thing fails [17:36:34] that would be the default Nginx limit I think [17:37:45] Dare I ask what your tool is doing that makes huge GET requests reasonable? That's a lot of search criteria. Content creating things really should be POST and not GET [17:41:30] It's useful if you want to pass info in via the URL. I should not that query.wikidata.org allows for large strings for the SPARQL parameter that are much larger than 4k [17:41:43] fuzheado: we actually have to send those queries via POST, too :) [17:41:47] s/too/though/ [17:42:47] Hi Lucas! So this is for the Knowledge Grapher tool I just made public. It cannot take a large set of Q numbers, but the same list of Q numbers that make my tool not work, does work when passed to WDQ [17:42:53] https://tools.wmflabs.org/knowledgegrapher/ [17:43:11] pointing out that some other system somewhere does something feels like whataboutism [17:43:21] for the Wikidata Query Service, long URLs are part of the URL *after the #* [17:43:24] which is never sent to the server [17:43:35] and then JS in the browser reads the query from there and sends it in a network request [17:43:39] Ah true, I could go that way [17:43:46] it tries GET first, but the queries can get too long, and then it retries as POST instead [17:44:04] (GET is better for caching in our case – it might be that in your case it might make more sense to always POST) [17:44:04] Good point, if that's a best practice I can do it that way [17:45:10] one of the issues with huge query strings is that they get logged in log files and can really blow up things as a result [17:46:35] bd808: whataboutism is a Wikipedia editing policy concept no? :) I'm just trying to identify our best practices here, so this seems to work best [17:47:21] (I’m not necessarily saying what we do is best practice, just wanted to point out that it’s not as easy as it looks for us either ^^) [17:48:11] but even if you manage to bump the nginx limit, at some point you’ll run into the browser limit (that’s the reason we had to implement POST in WDQS, IIRC), so changing nginx config will only get you so far [17:48:49] Lucas_WMDE: thanks for this, it's been a while since I last implemented a web-facing tool that handles Wikidata-sized chunks, so I'll go the POST route [17:49:30] ok :) [18:46:08] !log stashbot adding twentyafterfour to maintainers (cc bd808) [18:46:08] greg-g: Unknown project "stashbot" [18:46:08] greg-g: Did you mean to say "tools.stashbot" instead? [18:46:12] yes [18:46:16] !log tools.stashbot adding twentyafterfour to maintainers (cc bd808) [18:46:17] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.stashbot/SAL [18:47:23] thanks greg-g. I've also been thinking of some features to add to stashbot and the SAL. I'll taskify them once wmf.18 is overwith [18:47:31] * greg-g nods [18:52:41] what is in the x-forwarded-for field in requests through the cloud vps proxies? [18:58:50] fuzheado: FWIW URLs longer than 2K are somewhat fragile: https://stackoverflow.com/a/417184/323407 [18:59:28] Yeah I figured it's not a great idea :) [19:36:47] tgr: the XFF header should not be passed at all to upstream servers from either the domainproxy (Cloud VPS) or urlproxy (Toolforge). This is a feature that strips client IPs from the data seen by tools to reduce PII leaks. [19:38:51] greg-g: ack. twentyafterfour: welcome to the project. :) Sorry I was afk and didn't see the restart ask when it happened. [19:39:06] bd808: no worries, that's why we have more than 1 maintainer :) [19:39:18] I got it done within minutes, so it's OK [19:41:03] stashbot has been having some troubles talking to its elasticsearch store, but I haven't dug in too deeploy because I have an opne task to rebuild the elasticsearch cluster on buster that will probably end up changing some things. [19:41:03] See https://wikitech.wikimedia.org/wiki/Tool:Stashbot for help. [19:41:13] * bd808 pets stashbot [20:37:48] !log tools.heritage Triggered a manual categorisation job for ua_uk (T244333) [20:37:53] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.heritage/SAL [20:37:53] T244333: Re-enable categorization of Ukraine images - https://phabricator.wikimedia.org/T244333 [21:05:41] !log tools.heritage Triggered a images_of_monuments_without_id job, with template adding, for cz_cs (T242034) [21:08:20] ~log shinken stopping the shinken service on shinken-02 to give our inboxes a break during this outage [21:08:27] wm-bot157: Failed to log message to wiki. Somebody should check the error logs. [21:23:27] !log tools.integraality Deploy latest from Git master: c350e4f (T243998), fa12c21 [21:23:32] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.integraality/SAL [21:23:33] T243998: Queries feature is broken for total - https://phabricator.wikimedia.org/T243998 [23:14:17] !log tools.wsexport Redirected this tool to wsexport-test.wmflabs.org T242760 [23:14:20] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.wsexport/SAL [23:20:54] For folks who haven't migrated their tools from the legacy kubernetes cluster to the 2020 kubernetes cluster, we have a new way that is easier! `webservice migrate` will stop your existing webservice, change your active Kubernetes context, and start you webservice back up. [23:21:59] This command will also tell you if your webservice is not eligible for migration because it is not running on Kubernetes already or because it has already been migrated. [23:22:36] legoktm: ^ That was the thing you wanted me to make I think :) [23:22:46] awesome. /me does that [23:38:04] !log tools Depooling tools-sgewebgrid-lighttpd-0927 (T244791) [23:38:08] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [23:38:08] T244791: Scale up 2020 Kubernetes cluster for final migration of legacy cluster workloads - https://phabricator.wikimedia.org/T244791 [23:56:30] !log tools Depooling tools-sgewebgrid-lighttpd-0926 (T244791) [23:56:33] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [23:56:34] T244791: Scale up 2020 Kubernetes cluster for final migration of legacy cluster workloads - https://phabricator.wikimedia.org/T244791 [23:58:26] !log tools Depooling tools-sgewebgrid-lighttpd-0925 (T244791) [23:58:29] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL