[01:19:05] Is the "generic" webservice type still available? I recently tried a "webservice --backend=kubernetes generic start $script" and it said the "type" can't be "generic" [01:21:44] generic is only on the grid, not kubernetes [01:22:09] ah, gotcha [01:26:22] enterprisey: the golang and ruby Kubernetes containers work like generic works on the grid [01:26:42] and java? I think java too [01:26:49] what do the golang and ruby containers do that generic doesn't, if anything? [01:27:01] run on kubernetes :) [01:27:36] generic was the badly named "design your own webservice startup system" from the grid engine implementation [01:27:54] ah, I see [01:28:52] so when I say "work like generic" I guess I really mean "expects additional args telling how to start the webserver process inside the container" [01:28:53] question 2: I've started a webservice using golang111; where will the logs/stdout/stderr go, if anywhere? can I set it up so I can see them? [01:29:30] enterprisey: `kubectl logs ` is where stderr/stdout go by default [01:30:10] (`kubectl get pods` to get the pod name) [01:30:30] nothing will be written to your tool's $HOME. The log output will be buffered by the Kubernetes system with (some) default limit [01:31:21] if you know you only have one pod running: `kubectl logs -f $(kubectl get pods --output=jsonpath={.items..metadata.name})` will tail the log buffer for it [01:32:19] the `-f` is for --follow just like `tail` [01:32:46] (I feel like I'm in for a long night) [01:33:15] thank you both for your responses so far [01:36:24] any 502 debugging tips? logs show a successful server startup on stdout and then nothing, of course [01:36:49] I have the server printing to stdout when it gets a request, and it doesn't seem to have printed anything [01:37:37] enterprisey: is the pod running at all? [01:37:44] certainly is [01:38:55] I guess my process would be to first check the ingress object, then check the pod [01:40:03] if both looked fine I might double check that the 502 is not from routing being messed up by stopping the webservice entirely and checking that I get the expected "this tool is not running a webservice" response from the fourohfour tool [01:41:05] enterprisey: is your golang (or whatever other language) webservice code listening on port 8000 and speaking HTTP there? [01:41:25] Rust, I have it serving on the port given by the PORT env't variable, and it does speak HTTP [01:41:31] should I set it to 8000? [01:41:45] hopefully the PORT env is already 8000 [01:42:07] at least if you are running on Kubernetes, which I guess I didn't ask [01:43:50] * bd808 wonders off to find food [01:52:35] Should I make my tool listen on localhost, 127.0.0.1, or ::1? [01:59:29] to do what? [02:02:16] I can tell ya it won't be ::1 [02:09:47] Reedy: toolforge webservice, kubernetes backend [02:14:30] enterprisey: listen on 0.0.0.0 [02:14:36] enterprisey: also yay rust! [02:15:10] which tool is this? happy to help you debug [02:15:14] section-watchlist [02:15:51] I swear, if 0.0.0.0 was it... [02:16:19] also have you seen https://wikitech.wikimedia.org/wiki/User:Legoktm/Rust_on_Toolforge#webservice ? [02:16:31] I have indeed [02:16:45] :) [02:17:01] I'm starting it just like that, except I have the binary in the tool's home directory [02:17:35] very helpful page, thanks for writing it [02:18:32] :D yw [02:18:54] did you publish the source code yet? I'm curious what web framework you're using [02:18:57] you're precisely the person who I would hope would come online while I'm struggling with this, haha [02:19:07] raw hyper, no framework - I'm not doing anything fancy with routing [02:20:27] I will link the repo in just a minute; I'm force-pushing a different email address so I don't self-dox [02:20:46] ok, interesting [02:23:48] alright, legoktm: https://git.sr.ht/~enterprisey/section-watchlist [02:24:01] did 0.0.0.0 work btw? [02:24:44] restarting it as we speak [02:25:16] also it's probably best to just hardcode port 8000 [02:25:35] it worked!! thank you! [02:26:31] :D [02:26:32] yay [02:26:39] also, you might be interested in https://docs.rs/eventstreams/0.2.0/eventstreams/ [02:26:45] time to test oauth... [02:26:54] alright, now somebody go put that in some documentation [02:27:08] AntiComposite: https://wikitech.wikimedia.org/w/index.php?title=User%3ALegoktm%2FRust_on_Toolforge&type=revision&diff=1872477&oldid=1871655 :) [02:27:29] I was just about to myself [02:27:48] and re eventstreams, I recall trying that and/or sse-client at one point, but there was either an async runtime conflict or something like that [02:28:15] how much worse can it be than https://git.sr.ht/~enterprisey/section-watchlist/tree/master/backend/src/firehose.rs#L30 ? [02:28:33] excellent point [02:28:45] _that_ line worked first try, surprisingly [02:28:54] hahaha [02:29:26] a property that is probably exclusive to that line... [02:29:28] in any case, PRs welcome for eventstreams [02:30:46] I wrote it for an IRC bot I didn't finish yet [02:31:22] also I found that eventstreams is really finicky and just couldn't hold a connection open over a long time period [02:31:28] very nice, yeah a library would probably be a much better idea than what I have [02:31:42] oh, that's where my stability issues are coming from [02:31:45] enterprisey: I'm about to bbl for dinner, anything else you need help with now? [02:31:51] I would see disconnects every ~5 min [02:31:56] 10 min for me [02:32:02] um... [02:32:07] I think it's good for now [02:32:26] :D cool, I expect to come back to a fully working section-based watchlist ;-) [02:32:29] jk [02:32:45] it only works when lines are removed in a diff at the moment :( [02:32:58] but I have everything else sketched out don't worry [02:33:12] (or was it added...?) [03:13:16] enterprisey: also, I'm curious to what your experience with sourcehut has been [03:23:57] legoktm: great so far - it's the website I'd make if I had the free time for it [03:24:10] although I don't really use 99% of its features [05:41:15] can API:Options be used with just an access token from OAuth, or do I have to use API:Tokens first to get a CSRF token? [05:41:38] I suspect the latter, but it would be nice if there were a way to accomplish the former [06:00:14] pinging *shuffles deck* legoktm (sorry!) [06:00:42] enterprisey: you need to get a csrf token [06:01:12] how would I get a csrf token using an OAuth access token? The obvious API call doesn't work, returning only +\\ [06:01:18] The access token is equivalent to login cookies basically, it doesn't do anything else [06:02:12] so is there any way to call the Options API? [06:02:21] meta=tokens ? [06:02:48] yeah that doesn't work [06:02:55] I mean, get a csrf token from meta=tokens, and then call action=options.. [06:03:05] and you're passing the OAuth access token properly? [06:03:16] I sure hope so (header Authorization: ) [06:04:07] although I feel like it would be a security hole if I could get a csrf token from just an access token [06:05:06] enterprisey: OAuth 2 right? in that case it should look like https://www.mediawiki.org/wiki/OAuth/For_Developers#Authenticating_requests [06:05:17] how is that a security hole? [06:05:27] oh no I didn't include the word "Bearer" [06:05:56] I was gonna say, if the access token came from a limited permissions grant, there would be no way to similarly limit the csrf token, but that's a suspicious assumption [06:06:48] I wonder how I could access the rest.php/oauth2/resource/profile with just Authorization: $token instead of Authorization: Bearer $token - maybe some OAuth spec stuff [06:07:03] the token is just to protect against csrf attacks. permissions checking is done in the action=options module [06:07:51] I'm clearly a bit out of my depth here, haha [06:10:10] enterprisey: you got lucky, I spent last Thursday also writing an OAuth app in Rust ;) [06:10:17] I really did [06:10:28] excellent way to spend multiple days [06:10:48] I spent a day or so on a school assignment about 127.0.0.1 vs localhost vs ::1 last year, to boot [06:11:03] Python -> Rust has at least a few pitfalls [06:11:28] oof :| [06:11:38] I think the biggest thing is that a lot of libraries just don't exist yet [06:12:38] I am using https://docs.rs/rocket_oauth2/0.3.0/rocket_oauth2/ but I need to submit a PR for my MediaWiki version [06:13:16] (which is why I was very curious as to what framework and therefore oauth library you were using) [06:15:02] I got scared off by the nightly requirement for rocket, but otherwise I think it looks like a fantastic framework [06:16:51] I really like it so far [06:17:01] supposedly everything will be in the next rust stable release [06:17:30] https://github.com/SergioBenitez/Rocket/issues/19 [06:18:18] maybe one dependency left [06:26:47] hmm [06:27:04] so now I'm getting an invalid csrf token error even though I swear I'm passing the token in correctly [06:27:33] I get the token from meta=tokens (type=csrf) and pass it in the token parameter in the options API [06:31:42] enterprisey: and you're continuing to pass the access token right? [06:31:50] oh bologna [06:32:29] ok *that*'s a little unfair, they should've written that somewhere [06:32:44] (although it's pretty clear why in hindsight) [06:35:20] all requests need the access token [06:35:26] 23:01:18 The access token is equivalent to login cookies basically, it doesn't do anything else [06:35:32] you always need to pass along your login cookies :) [06:36:31] enterprisey: ideally we would add OAuth2's access token as a new authentication way in the mediawiki_rust library so we don't need to hand roll it [06:36:44] now that's a good idea [09:08:45] Hello, i need to become an op. [09:23:08] 🎉 hauskatze [09:23:15] :) [09:23:37] /mode #wikimedia-cloud -b Taichi!*@* [09:23:50] just in case the real taichi ever joins [09:25:11] I'd say /mode #wikimedia-cloud +b *!*@83.red-88-7-156.staticip.rima-tde.net [09:59:39] !log tools point DNS tools.wmflabs.org A record to 185.15.56.60 (tools-legacy-redirector) (T247236) [09:59:42] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [09:59:42] T247236: Toolforge: introduce a system to preserve old tools.wmflabs.org URLs - https://phabricator.wikimedia.org/T247236 [10:20:12] arturo: 03:02:49 <+shinken-wm_> PROBLEM - Toolforge Home Page on toolforge is CRITICAL: HTTP CRITICAL: HTTP/1.1 308 Permanent Redirect - string 'Magnus' not found on 'https://tools.wmflabs.org:443/admin/' - 428 bytes in 0.014 second response time [10:20:24] (in case you haven't seen it yet) [10:21:06] oops :D [10:28:51] legoktm: unable to sleep today? :-) [10:30:21] -> pm [10:30:59] legoktm: yes, I'm investigating how to refresh the check [10:32:38] ok, just wanted to make sure you were aware :) [10:34:01] refresh the check: take the check (with many figures), turn on the A/C, hand the check to hauskatze :P [10:37:00] legoktm: I think a simple https://gerrit.wikimedia.org/r/c/operations/puppet/+/610013 will do the trick [10:38:42] reviewed [10:39:52] left a comment [10:41:48] thanks you both Majavah and legoktm . What you commented make sense for the next iteration [10:42:48] shinken is going away very soon anyways [10:43:09] * legoktm crosses fingers [10:48:24] * arturo watches https://grafana-labs.wikimedia.org/d/7fTGpvsWz/toolforge-vm-details?orgId=1&var-VM=tools-legacy-redirector&from=now-3h&to=now [10:50:27] * Majavah notes that is using an outdated version of grafana [10:51:52] also would anyone mind if I took AutoWikiBrowser and replaced all outdated https://tools.wmflabs.org/ urls in wikitech? that should be fairly simple to do with some regex find-and-replacing [10:58:24] Majavah: I'd say wait a few days just to make sure we're not reverting back [11:03:25] Majavah: the problem I see is that many entries require some other side updates, example https://wikitech.wikimedia.org/w/index.php?title=Help:Toolforge/Performance&diff=1872589&oldid=1855691 [11:05:54] Amir1 musikanimal we are using the tools-info as example here: https://wikitech.wikimedia.org/w/index.php?title=Help:Toolforge/Performance&diff=1872589&oldid=1855691 would you be interested in refreshing the file in the webservice itself? [11:08:35] Majavah: that being said, edits welcome of course :-) [11:15:29] * Majavah found https://tools.wmflabs.org/.error/banned.html which still has tool labs logo :P [11:19:11] Majavah: your DNS might be outdated. That URL already redirects to toolforge.org which then redirects to wikitech [11:20:14] arturo: it was outdated (it's now https://toolforge.org/.error/banned.html), but I still want to fix that old logo :D [11:20:47] oh OK, I see now Majavah [11:21:09] https://commons.wikimedia.org/wiki/File:Toolforge_logo_with_text.svg probably? [11:22:31] Majavah: the file lives in the operations/puppet.git tree, see `modules/profile/manifests/toolforge/proxy.pp` [11:22:53] arturo: I found the files and am making a patch, just making sure that is the right logo to use [11:23:16] oh yes, sorry, that one would work I think Majavah [11:24:51] https://gerrit.wikimedia.org/r/c/operations/puppet/+/610026/ [11:35:02] Majavah: thanks! [13:13:49] Hello, this is urgent, i need to use Toolforge, and my request is https://toolsadmin.wikimedia.org/tools/membership/status/841 [13:13:55] Approve me now. [13:16:00] Hello, this is urgent, i need to use Toolforge, and my request is https://toolsadmin.wikimedia.org/tools/membership/status/841 [13:16:01] Approve me now. [13:16:08] Hello??? [13:16:22] Rociolaura: sure I will approve you right now [13:21:16] lool [13:21:36] sigh... [14:34:44] So either my PuTTY is being stupid or the tools-login bastion is having a hissy fit... it's being super slow [14:35:03] tools-sgebastion-07 [14:35:30] you're not the only one [14:35:43] Hello, I would need some help to setup some projects on Toolforge.I'm not sure if I need to request a Cloud VPS or if I should continue developing my tools on Toolforge, given the fact that I'm not very comfortable with Kubernetes and that I need a JS front-end (the only part that I have deployed on Toolforge so far), a Java backend, a MariaDB DB [14:35:44] and a continuously running bash script. Can someone help? [14:36:45] Majavah: Alright, glad it's not my PuTTY... I'll have patience [14:39:17] I'm guessing someone is running a cpu heavy script or it's NFS issues [14:39:21] * Reedy tries to login [14:40:59] Reedy: I see one pwb script and couple other python scripts [14:41:13] I havn't got a prompt yet [14:41:21] also one nodejs script [14:44:21] still not got a prompt [14:44:26] this is fun [14:44:58] Reedy: Yea, mine took a good while to get me a prompt, then just to do ls [14:46:22] pid 10889 (python= is using 18.2% of cpu [14:46:29] what's load showing as? [14:46:59] load average: 11.09, 12.54, 11.64 [14:48:28] Not that I can do much until I get on [14:48:35] Nearly 10 minutes now [14:49:17] I'm in but I can't do anything [14:53:12] Load is more reasonable again, ofc now [14:53:29] but definitely two python processes using quite a lot of cpu [14:56:29] !log killed python process from tools.experimental-embeddings using a lot of cpu on tools-sgebastion-07 [14:56:30] Reedy: Unknown project "killed" [14:56:39] !log tools.experimental-embeddings killed python process from tools.experimental-embeddings using a lot of cpu on tools-sgebastion-07 [14:56:40] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.experimental-embeddings/SAL [14:57:03] meno25 9023 2.0 0.4 218364 78260 pts/10 SN+ 09:51 6:16 python3 pwb.py cosmetic_changes -lang:arz -newpages:300000 [14:57:33] !log bastion killed meno25 process running pwb.py on tools-sgebastion-07 [14:57:34] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Bastion/SAL [14:58:19] isn't it discouraged to run scripts on bastions? [14:58:29] Yup [14:58:35] But it happens, repeatedly [14:58:48] I think it's "fine" for short running, testing, or low cpu scripts [14:58:55] Which is why they will just get killed [14:58:57] was that the right sal? afaik that wasn't on the bastion vps project [14:59:24] https://wikitech.wikimedia.org/wiki/Nova_Resource:Bastion [14:59:26] >The project to allow Cloud VPS users to connect to other instances within Cloud VPS via tunneling/proxy. [14:59:26] Global projects [14:59:28] * Reedy shrugs [14:59:48] load average: 0.24, 2.53, 6.78 [14:59:51] Problem is solved though [15:01:30] It might have been me : I was on tools-sgebastion-07 and was doing a git pull that took more than 10 minutes and seemed to be stuck [15:01:31] !log tools killed meno25 process running pwb.py on tools-sgebastion-07 [15:01:34] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [15:01:44] !log tools killed python process from tools.experimental-embeddings using a lot of cpu on tools-sgebastion-07 [15:01:45] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [15:01:58] Nonoxb: Probably not. You weren't running python processes using high CPU [15:02:11] Alright [15:05:32] thanks for doing the needful on the bastion Reedy [15:26:08] multiple scripts on toolforge have been broken today [15:26:08] is this multiple scripts being broken, or toolforge being broken [15:27:20] There's no enough information to help you [15:27:24] Which tools/scripts? [15:27:27] How were they broken? [17:51:40] Hola RhinosF1 [17:51:50] Please, request global ban to joaquinito01 [17:52:21] I need to use the Wikimedia Toolforge [17:53:38] give me op please [17:53:42] bd808 [17:54:49] hi [17:55:24] Please, request to use apache traffic server [17:56:41] Do you know what the first sign of madness is? [17:57:35] @kb Guest63 [17:57:42] That's better [17:57:49] @kb MarcoAurelio01 [17:57:58] kicking since they are trolls [17:57:59] Reedy: this? [17:58:04] Talking to yourself [17:58:16] I'm not sure why this guy has a grudge with me now [17:58:19] It's not even hard to see that they're the same person xD [17:58:27] hauskatze: Imitation is a form of flattery [17:58:38] So... shall I feel good? [17:58:43] hauskatze: he loves you. It gets worse, I would know. [17:59:18] Reedy: oh he's quite clearly on something. You should see what he's like when he's had drugs. [19:17:36] Please, request BanBot to #wikimedidia-cloud [19:17:41] hey [19:17:44] stop that [19:17:49] give me an op please [19:18:11] Por el amor de Dios... [19:19:03] hauskatze he need a ban [19:21:06] MacFan4000: ? [19:21:33] Hola RhinosF1 [19:21:34] @kb fcy [19:21:34] Sorry but I don't see this user in a channel [19:21:39] suppist01, fxy: please go away [19:21:41] I need to use the Wikimedia Toolforge [19:21:44] @kb fxy [19:21:51] @kb suppist01 [19:21:55] There [19:22:04] Ty [19:22:31] At least he's not posting nasty stuff [19:24:14] hauskatze: he's normally just a demanding, drunk pain in the neck [19:37:35] !log tools.admin Hard restart with --canonical enabled [19:37:37] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.admin/SAL [22:42:08] !log tools.fengtools Updated .lighttpd.conf and restarted for T257332 [22:42:11] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.fengtools/SAL [23:16:43] !log tools Building webservice v0.73 (T234617, T257229) [23:16:47] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [23:16:48] T234617: Toolforge. introduce new domain toolforge.org - https://phabricator.wikimedia.org/T234617 [23:16:48] T257229: webservice: replace .webservicerc with service.template - https://phabricator.wikimedia.org/T257229 [23:19:45] !log tools Deploying webservice v0.73 via clush (T234617, T257229) [23:19:48] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [23:22:53] !log tools Rebuilding all Docker images to pick up webservice v0.73 (T234617, T257229) [23:22:58] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [23:22:58] T234617: Toolforge. introduce new domain toolforge.org - https://phabricator.wikimedia.org/T234617 [23:22:58] T257229: webservice: replace .webservicerc with service.template - https://phabricator.wikimedia.org/T257229