[09:11:53] !log tools.zppixbot close ZppixBotWiki [09:11:56] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.zppixbot/SAL [09:12:01] Reception123: ^ see bots.mh.o [09:33:44] Majavah: is Majavah on the new wiki you? [09:45:32] RhinosF1: yep, wanted to test logging in because it said I would not be able to do that [09:46:22] Majavah: Ah [11:10:23] godog: I'm trying to use exim.mtail to get some prometheus metrics from our toolforge email servers (T256737) [11:10:24] T256737: Toolforge email: proper prometheus integration - https://phabricator.wikimedia.org/T256737 [11:10:55] but I see in prod some metrics that my deployment is not producing [11:11:05] for example `exim_messages_total` [11:11:37] reading the source code and the puppet manifest, I don't even understand how mtail knows which log file to scan [11:12:24] * arturo finds `mtail::logs` hiera config [11:13:16] who is reading that hiera key though? [11:14:39] I wonder if `modules/mtail/templates/default.erb` [11:18:18] yeah, that seems to do it [11:18:50] !log tools set some hiera keys for mtail in puppet prefix `tools-mail` (T256737) [11:18:53] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [11:18:53] T256737: Toolforge email: proper prometheus integration - https://phabricator.wikimedia.org/T256737 [11:26:50] !log redirect zppixbotwiki to bots.miraheze.org [11:26:51] RhinosF1: Unknown project "redirect" [11:27:00] !log tools.zppixbot redirect zppixbotwiki to bots.miraheze.org [11:27:01] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.zppixbot/SAL [11:30:52] MacFan4000: I've set canonical server but links aren't changing [11:45:40] MacFan4000: let's just hard drop it [11:58:26] !log tools.zppixbot delete wiki [11:58:29] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.zppixbot/SAL [15:00:46] musikanimal: you around by any chance? [15:42:18] Amir1: Is there a public interface where I can monitor the amount of logins my bot is doing? [15:51:32] remote: unfortunately not :( [15:51:42] kk [17:15:34] Hello, my projects folder's permission is changed to 2700, how do I change it to 2775 like everyone else's? [17:16:05] I'm trying WinSCP to change it, but getting pemission denied. [17:17:22] Tanvir: which tool name? [17:17:34] welcomebots-bn [17:18:59] You do have some strange file permissions set there. Let me see if I can reset them to something a bit more normal [17:19:58] bd808, could be. I've trying/testing multiple things for the last half an hour. Sorry. [17:21:22] !log tools.welcomebots-bn Reset permissions on /data/project/welcomebots-bn/ to 2775 (drwxrwsr-x) [17:21:23] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.welcomebots-bn/SAL [17:22:03] Tanvir: your $HOME should be fixed now. Do you need more changes inside there too? [17:23:04] looks like you have bad perms on more things inside there. maybe from winscp doing something strange [17:23:42] bd808, like? Please recommend. [17:24:35] Tanvir: I'm seeing permissions like "drwx--S--- 2 tools.welcomebots-bn tools.welcomebots-bn 4.0K Feb 13 2016 public_html/" [17:25:55] You could reset all you find problematic / not normal. [17:27:43] bd808, I noticed today I can't create folder or change it inside sh directory. Getting permission denied. [17:28:32] that directory ended up with "drwxr-sr-x" permissions with the owner being your user rather than the tool [17:29:09] your scp client is not being very helpful with umasks I would say. [17:29:35] bd808, what would be a good client then? [17:30:21] Tanvir: maybe we should back up. Can you explain the workflow you are trying to use? [17:31:11] bd808, I just run the welcoming bots there for bn projects. [17:32:40] Yes, but how are you moving files there and why is basically my question. What workflow are you trying to achieve? Knowing that I can maybe help figure out how to do it with fewer errors [17:35:15] bd808, I usually use SCP client to upload/change files and crontab to run them. Does that answer your question? If not, I failed to understand your question again. [17:36:22] Today, I realized that I couldn't change my sh files, and wondered what could go wrong. [17:38:56] Tanvir: and this has worked for you in the past but is now suddenly broken? I can see the bad permissions, but unfortunately not how they ended up being set. [17:39:37] I've no idea either. I don't access there very often as the bot runs without error. [17:40:12] But I'm happy to manage them in the way you see fit and easy. [17:43:23] I really don't have any recommendation for an scp based workflow. I personally find that using git is the easiest way for me to manage files for a tool. Git push from my laptop to some central repo and then git pull from toolforge into the tool's $HOME [17:43:48] but anyway, let me see if I can fix up these permissions to get you back in action [17:45:33] bd808, thanks in advance. Problem I'm facing I can't create/change files in sh directory from terminal; can't access pywikibot and public_html directory from WinSCP. [18:00:10] Tanvir: I've got most of them fixed. still trying to figure out what to do with the permissions inside $HOME/pywikibot. [18:04:51] bd808, Thank you very much. Just trying to understand the issue here. I just used touch sample.txt from terminal and it created the file. I couldn't save changes by opening through WinSCP or even changed the permissions. However, I could do so from terminal. Why is it happening? [18:06:16] I can also save changes to the file from from terminal using nano. [18:06:35] In the terminal you are a different user (tools.welcomebots-bn) than you probably are through winscp (tanvir). This is a thing that folks trying to use scp based workflows often get confused by [18:07:11] the `become welcomebots-bn` command that you run in the terminal switches you to being a different user [18:07:20] I see. Is there way I can change it in WinSCP? [18:08:28] https://wikitech.wikimedia.org/wiki/Help:Access_to_Toolforge_instances_with_PuTTY_and_WinSCP#How_to_set_up_WinSCP_for_direct_access_to_your_Toolforge_account [18:09:09] Thanks. Reading... [18:09:22] I think the magic bit is "sudo -u tools.PROJECT-NAME /usr/lib/sftp-server" in the SFTP server settings. Replace PROJECT-NAME with welcomebots-bn in your case [18:12:07] bd808, in my case the PROJECT-NAME is wikitanvirbot (my regular bot). Do you think that caused the problem? [18:13:06] well, it would not help. That is yet another shell account [18:13:37] Yeah, I know. [18:13:47] I don't think that is what messed up all the permissions, but maybe whatever you tried to do after they first got messed up did [18:14:09] or you messed them up by trying to make the 3rd account be able to access the 2nd? [18:14:47] Tanvir: it sounds to me like you need to setup multiple profiles in your winscp, one for each tool [18:15:05] I think I have mostly unbroken the permissions in /data/project/welcomebots-bn [18:15:27] Yeah, there are no options to select. So I'm setting up another profile. [18:15:31] any that are still messed up can probably be fixed through ssh using `take $FILENAME` [18:22:52] bd808, I created sperate profiles for separate tools and it worked fine. I hope it will reduce such mix-ups. [18:23:39] Tanvir: cool. good luch :) [18:23:42] *luck [18:24:24] bd808, thank you very much! [19:19:30] Hey folks, is something up with mail.tools.wmflabs.org today? I'm getting x509 unknown authority errors, even sending from Toolforge. [19:42:51] Naypta: can you tell me more about these errors? Is this an error in TLS handshaking when connecting to the mail server? [19:43:29] That's what it looks like; I've only actually seen it briefly in a stack trace in my application logs. Let me try sending something through Thunderbird, that'll give me a more informative log hopefully [19:44:40] We did add TLS to that service recently. It should be using a cert signed by Let's Encrypt [19:45:46] * bd808 verified that `mail` at least works from a bastion [19:49:37] Hrm, Thunderbird works fine [19:49:54] Maybe something is up with my SMTP library [19:55:35] Naypta: it seems likely that the problem is your library/language not having the LE root cert in it to verify the new cert on the mail server [19:56:54] Yeah, I'm just trying to work out whether golang has a specific set of root certs that it itself uses (which would be wack, but then, most things about Golang are) or whether there's some other system it uses [19:59:58] https://stackoverflow.com/a/40051432/8171 says it reads a bunch of places on the server [20:00:09] Looks like it's missing on toolforge [20:00:15] I wouldn't doubt that we are missing the current LE root [20:00:15] If you run `awk -v cmd='openssl x509 -noout -subject' '/BEGIN/{close(cmd)};{print | cmd}' < /etc/ssl/certs/ca-certificates.crt | grep "Let's Encrypt"`, you get nothing [20:01:02] Worth me opening a task to add the LE root? I could ship it specifically with my bot's binaries, but that seems a bit insane lol [20:01:37] hmm. curl was fine with it [20:01:47] so it must be installed somewhere [20:01:51] that's... weird [20:02:14] don't feel obligated to look into my mess haha, I'm sure you've got things to do :p not that I'm complaining about the help if you don't :D [20:04:29] possibly it's not following certificate chaining correctly [20:05:55] Naypta: ah. so the LE cert is not there, but its signer "DST Root CA X3" is [20:06:17] so maybe the mail server is not stapling the LE intermediate root [20:07:02] TLS was added late last week, so you may just be the first person to notice who is using a sending client that actually it trying LTS [20:07:08] oooh, that's a good shout [20:08:19] root@tools-mail-02:~# grep acmecert /etc/exim4 -r [20:08:26] /etc/exim4/exim4.conf:tls_certificate = /etc/acmecerts/tools_mail/live/ec-prime256v1.crt [20:08:42] Naypta: it is worth a ticket :) you could link it to T208281 [20:08:42] T208281: Set up SPF, DKIM, etc. for new cloud MX servers - https://phabricator.wikimedia.org/T208281 [20:09:09] or maybe better to the parent at T249237 [20:09:10] T249237: Fix Cloud VPS and Toolforge mail servers to work with the modern internet - https://phabricator.wikimedia.org/T249237 [20:09:12] think that should be /etc/acmecerts/tools_mail/live/ec-prime256v1.chained.crt [20:09:39] which will make it include the LE cert signed by DST [20:09:47] Krenair: that does sounds right. I wonder if that's hiera stuff for the profile? Seems likely [20:10:26] hmmmm: [20:10:28] modules/profile/templates/exim/exim4.conf.smarthost.erb:tls_certificate = /etc/acme/cert/<%= @cert_name.gsub(/[-.]/, '_') %>.chained.crt [20:10:40] modules/profile/templates/exim/exim4.conf.mailman.erb:tls_certificate = /etc/acmecerts/lists/live/rsa-2048.chained.crt [20:10:43] modules/role/templates/exim/exim4.conf.mx.erb:tls_certificate = /etc/acmecerts/mx/live/rsa-2048.chained.crt [20:10:45] but: [20:10:49] modules/profile/templates/toolforge/mail-relay.exim4.conf.erb:tls_certificate = /etc/acmecerts/<%= @cert_name %>/live/ec-prime256v1.crt [20:11:05] welp, worked out how to replicate the problem on the servers without golang at least! [20:11:11] `openssl s_client -connect mail.tools.wmflabs.org:25 -starttls smtp` [20:11:21] you can literally see the certificate chain ending early [20:11:55] Krenair: you want to make the patch or should I? [20:12:06] go for it [20:14:35] bd808: all yours ;) https://phabricator.wikimedia.org/T256806 [20:17:46] Naypta: if this stays broken until EU morning tomorrow will that cause you major problems? [20:18:14] save for a disaster, no :) I only use it for sending myself stack traces if a panic happens haha [20:18:42] cool. I'd like a.rturo to at least see the patch. :) [20:19:13] tysm for looking into it! it's greatly appreciated :) and you too, Krenair! [20:20:32] thanks for the nice recreation of the problem :) [20:21:01] heh, after twenty minutes of flailing about :D [20:22:42] np, thanks for raising [20:22:46] !log devtools managed to let certbot get LE certs for gerrit.devtools.wmflabs.org and the floating IP [20:22:47] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Devtools/SAL [20:24:56] I think openssl s_client's -verify_return_error option may be broken :| [20:25:52] sigh: https://github.com/openssl/openssl/issues/8079 [20:26:43] okay so... [20:26:44] -verify +int Turn on peer certificate verification [20:26:47] +int??? [20:31:38] okay so apparently that's depth [20:32:15] I kind of would've expected `openssl s_client -connect mail.tools.wmflabs.org:25 -starttls smtp -verify_return_error -verify 1 -partial_chain` to work but at this point I kind of doubt openssl s_client's behaviour is going to match my expectation :) [20:36:26] s_client is a weird beast from the very few interactions I've had with it... I don't know whether that says more about me or it [21:11:53] what's your issue? [21:12:37] you will need -show_certs to get the whole certs sent. btw [21:14:03] which doesn't add anything [21:14:05] quite simple [21:14:11] it is missing the intermediate certificate [21:14:58] just compare: [21:15:01] openssl s_client -connect mail.tools.wmflabs.org:25 -starttls smtp -showcerts [21:15:09] with [21:15:11] openssl s_client -connect mail.wikimedia.es:25 -starttls smtp -showcerts [21:15:49] Platonides: yeah, there's a patch already for it that Bryan sorted :) https://phabricator.wikimedia.org/T256806 [21:16:02] good [21:17:35] btw, it shouldn't fail sending mails [21:17:42] smtp starttls fails open [21:18:00] that's why there are thing like mta-sts, to close that gap [21:31:35] someone can help me with the toolforge.org migration of wmopbot? I removed the line "app.config['APPLICATION_ROOT'] = '/wmopbot/'" from flask code and removed 'wmopbot' in the .lighttpd.conf but it doesn't work [21:33:01] danilo: what doesn't work specifically? [21:33:22] I'm trying to make sure a tool I'm working on works on the new toolforge scheme with subdomains [21:33:23] https://wmopbot.toolforge.org/ [21:34:10] currently, it sets cookies with an /ia-upload/ path, but I guess in this new version it would have to detect where it's being run from and scope the cookie path correctly [21:34:33] which is not great... is there a better solution? like to redirect from /ia-upload/ to ia-upload.toolforge.org? [21:34:36] ningu: in --canonical mode you no longer need to scope the cookies [21:34:45] bd808: ok, thanks [21:34:54] so there's the answer, basically the wrapper handles it [21:35:05] so I should write it to be at the root level [21:36:15] ningu: yes. once you add --canonical to your webservice start command then the front proxy will redirect any visits to the old tools.wmflabs.org/$TOOL url space to $TOOL.toolforge.org [21:36:30] cool [21:36:38] danilo: maybe set it to '/' ? [21:36:42] yeah, I also didn't want to hardcode the urls if possible, so that's great [21:36:50] danilo: hmmm.. so you are getting a redirect loop from somewhere? [21:37:05] * bd808 becomes the tool to poke around [21:37:17] btw I ported this ia-upload tool from silex, a now-abandoned PHP framework, to slim -- hopefully that won't cause a big fuss for the maintainers [21:37:50] ningu: I suppose you should talk to the maintainers about that :) [21:37:59] Platonides: I will try that [21:38:12] but +1 for slim as a reasonably easy PHP framework to use [21:38:23] bd808: yes, I don't know how to make that work [21:39:16] danilo: is there something inside your tool that is issuing redirects? [21:40:56] this also looks like it could move to the Kubernetes cluster and make things a lot simpler on the management side [21:41:03] not that I remember, I search 'wmopbot' in the code and I didn't find anything that could do that [21:41:58] I tried "app.config['APPLICATION_ROOT'] = '/'" but didn't work [21:42:37] I will try to change to kubernetes [21:45:04] danilo: https://wikitech.wikimedia.org/wiki/Help:Toolforge/Web#python3.7_(Python3_+_Kubernetes) [21:46:47] I am trying with "webservice --backend=kubernetes python2 start" but it try to use uwsgi and return an error, how can I make it use lighttpd? [21:47:36] using python2 is not recommended. that container is deprecated and will be removed in the next 3 months or so. [21:48:08] The python kubernetes setup uses uwsgi directly without lighttpd being involved [21:48:45] there is a really complete tutorial at https://wikitech.wikimedia.org/wiki/Help:Toolforge/My_first_Flask_OAuth_tool [21:49:04] that should explain where all the code needs to go to "just work" [21:49:40] danilo: your $HOME/public_html directory is empty, so I don't think you really need lighttpd for anything [21:49:54] and your $HOME/web.py looks to be just a simple flask app [21:50:45] I wonder if the redirect loop is from your _force_https() function? [21:51:29] danilo: I don't see your code looking for the X-FORWARDED-PROTO header that will tell you if TLS is active to the front proxy or not [21:51:46] so I think your redirect loop is self inflicted? [21:53:54] I don't know, it always worked in tool.wmflabs.org [21:57:34] we enforce TLS for all tools now, so either way that code should be dead code [21:57:48] (for reference, https://github.com/legoktm/toolforge/commit/292203f8de3d08dcebe92b884974b4bf8a95ea0d#diff-f40b9763b6bdb74acba8d9c86da8ffe4 is what my force https flask function used to look like) [21:57:54] but I think it is causing the redirect loop [21:58:29] !log tools.zppixbot deploy web [21:58:30] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.zppixbot/SAL [21:58:39] bd808: do you block 301s external of toolforge [21:59:00] no, but we don't like them [21:59:36] bd808: I set one on https://zppixbot.toolforge.org/ and it shows blank [21:59:41] are you worried that the 4 of you who visit that site will forget where you moved it to RhinosF1? ;) [22:00:16] RhinosF1: do a view source. Your php is not being executred [22:00:17] bd808: it's had 3 hits according to google search console. [22:00:20] RhinosF1: whatever is serving that tool isn't interpreting the PHP code [22:00:23] *executed [22:00:51] from not us [22:01:17] bd808: it's a standard php webservice on kuberenetes [22:01:40] index.html is not a php extension [22:01:48] index.php is [22:02:06] and the code there isn't even actually php [22:02:21] that's an html comment that has some php like noise in it [22:02:27] It has to start with and be a php file, not an html file [22:02:53] we don't setup lighttpd to parse all html as php [22:03:03] * Reedy also wonders why it has an exit; and a ?> [22:03:05] I know what to do [22:03:55] RhinosF1: If y'all are shutting down the tool I don;t think you really need to leave a foreign redirector running [22:04:56] bd808: the tool will live for the next probably 2 week before I fully kill it. [22:05:14] * RhinosF1 is trying to update search console and it needs a 301 [22:07:32] bd808: I removed the part of the code that try to force https, but it still don't work, but https://wmopbot.toolforge.org/wmopbot/ works, something is trying to redirect to /wmopbot, but I am not finding what is doing that [22:08:44] maybe something in the lighttpd default config? [22:09:53] danilo: oh. have you set --canonical yet? Or are you trying to get it working on both URL schemes at the same time? [22:10:29] * bd808 sees it is the latter [22:10:46] I didn't try --canonical, I will try [22:11:07] there may be things in the lighttpd config that are messing with you. I know there is an alias that can cause problems [22:11:19] --canonical removes it [22:12:56] !log tools.zppixbot deploy web [22:12:58] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.zppixbot/SAL [22:13:00] I started with "webservice --backend=gridengine --canonical start", but it didn't work [22:13:59] what was the error? Something about a missing [type]? [22:14:25] `webservice --backend=gridengine --canonical lighttpd start` would be the full command [22:15:07] no, it started, but https://wmopbot.toolforge.org does not work, and https://wmopbot.toolforge.org/wmopbot works [22:16:08] let's look at the generated ligttpd.conf [22:17:52] nothing in there that would redirect [22:20:14] danilo: I think your redirect_login() method has issues with the new url scheme [22:20:56] got to https://wmopbot.toolforge.org/info while logged out and I think you will notice that it redirects to https://wmopbot.toolforge.org/info/login?returnto=%2F%3F [22:21:48] which is maybe flask [22:22:01] flask's url_for() getting confused? [22:22:42] I'm not sure how to debug your hand built wsgi container [22:25:22] danilo: just playing with the URLs in my browser your webservice is acting like it strips off the initial directory of the path no matter what it is and then sticks it back on when flask generates full URLs [22:25:29] I had the same problem to make ptwikis tool work in toolforge.org, I needed to put a rewrite rule in lighttpd.conf there to make it work, and there I don't have a login that make redirects [22:26:37] huh. I've moved 20+ tools myself and never had this sort of behavior [22:27:20] but I also don't have anything doing the super old setup for uwsgi that you are using [22:27:51] I guess its actually fcgi and not wsgi [22:30:25] is moving this tool to k8s/wsgi likely to cause more problems than the current ones? [22:31:06] today is the deadline of the OAuth migration? what will happen tomorrow with tools that use OAuth and didn't migrated? [22:31:19] nothing :) [22:31:50] I don't think a.rturo is planning on the final cutover this week honestly [22:32:32] danilo: but this tool doesn't actually use OAuth as far as I can tell [22:32:33] great, so I will have some more time to rewrite that code [22:32:55] oh.. I'm not good are reading [22:33:01] I was worry because you said in T254857 that today is the deadline [22:33:02] T254857: Toolforge: domain migration: track down missing OAuth grants - https://phabricator.wikimedia.org/T254857 [22:33:06] it is [22:33:18] and you should have been worried 30 days ago [22:33:34] yes, it uses to access the cloak request form [22:34:26] danilo: oh. look at your set_base_url() function [22:34:37] that's what is doing the strange things with the url [22:35:22] you have a special case there for 'ptwikis.toolforge.org' and not wmopbot.toolforge.org [22:35:42] copy pasta error [22:37:18] I will remove that function to test, but that is used only inside templates, I put that a hour ago to try to make the same fix I did in ptwikis [22:40:08] I removed that function, but the problem persists [22:44:44] danilo: if you go to https://wmopbot.toolforge.org/anything/ you can see that having any leading directory in the path makes things work. This really has to be in the python code somewhere, but I don't know where. [22:47:39] yeh, I noticed that when I was trying to make the migration in ptwikis, I tryied many things to debug that, but I didn find what is doing that [22:50:37] I will undo the changes to make it work in tools.wmflabs.org again, and I will update the code in the next days to make it work with uwsgi [23:00:53] !log paws added paws-public.wmflabs.org to the alt-names for acme-chief, which broke it until we hand off the zone to the paws project T195217 T255997 [23:00:57] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Paws/SAL [23:00:57] T255997: PAWS: figure out paws-public in the new service model - https://phabricator.wikimedia.org/T255997 [23:00:57] T195217: Simplify ingress methods for PAWS - https://phabricator.wikimedia.org/T195217 [23:02:43] bd808: can you approve the new ia-upload-dev consumer I just added? [23:02:57] samwilson suggested using beta commons for testing, so I changed applicable sites to * [23:03:36] ningu: {{done} , but that grant won't work on beta commons [23:03:47] that grant is only in the "production" wikis [23:04:00] hrm ok [23:04:06] so how do I get one for beta commons? [23:04:31] at https://meta.wikimedia.beta.wmflabs.org/wiki/Main_Page (well the right special page there) [23:05:04] https://meta.wikimedia.beta.wmflabs.org/wiki/Special:OAuthConsumerRegistration [23:06:10] thanks [23:06:16] it looks like I need a new account, too? [23:06:38] they're free there too ;P [23:06:41] yes. the beta wikis are a separate system from the main project wikis [23:08:10] captcha isn't appearing though [23:08:33] apparently I need to click the "request an account" link, which is an empty page [23:09:20] sweet [23:09:35] https://meta.wikimedia.beta.wmflabs.org/wiki/Meta:Request_an_account [23:09:39] I saw [23:09:52] If you hit refresh on the captcha a few times, you'll get a word [23:09:55] hehe ok [23:10:06] I dunno why numerous are blank... [23:10:57] Apparently there's nearly 30K captchas... [23:11:13] * Reedy attempts to clean up [23:12:33] I wonder if when people were debugging some captcha issues, they made empty captcha images [23:13:48] ok, it seems to be working. hooray [23:13:52] consumer too [23:20:13] Fixed [23:26:30] thanks!