[04:50:25] !log tools.admin-test Stopped old, broken copy of tools.admin [04:50:26] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.admin-test/SAL [14:21:04] !log admin stopping old VPS proxies in eqiad — T213540 [14:21:07] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [14:21:07] T213540: Migrate nova proxies to eqiad1 - https://phabricator.wikimedia.org/T213540 [14:21:16] !log project-proxy stopping old VPS proxies in eqiad — T213540 [14:21:18] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Project-proxy/SAL [14:23:59] !log tools T213418 allocate floating IPs for tools-docker-registry-03 & 04 [14:24:01] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [14:24:01] T213418: Toolforge: move docker nodes from eqiad to eqiad1 - https://phabricator.wikimedia.org/T213418 [14:34:52] !log tools T213418 point docker-registry.tools.wmflabs.org to tools-docker-registry-03 (was in -02) [14:34:55] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [14:34:55] T213418: Toolforge: move docker nodes from eqiad to eqiad1 - https://phabricator.wikimedia.org/T213418 [15:36:38] is the SAL tool currently not responding for anyone else? (e. g. https://tools.wmflabs.org/sal/) [15:37:19] * gtirloni checking [15:37:23] I joined the channel to to ask if it’s a known problem that search doesn’t seem to work in other SALs (e. g. Cloud VPS projects), but now I can’t view the SALs at all ^^ [15:37:28] (there’s still the on-wiki pages of course) [15:37:52] Lucas_WMDE: works for me? [15:37:54] ah, now it’s loading again [15:37:57] I don't know if someone restarted it just now, but it's working for me (it took a long time to load though) [15:37:59] oh well [15:38:23] in that case I can repeat my original question :) [15:38:28] * bd808 needs to update that tool to use the php7.2 image [15:38:35] e. g. https://tools.wmflabs.org/sal/wikidata-dev?p=0&q=wikidata-constraints&d= doesn’t show just messages matching “wikidata-constraints” [15:38:55] but on the prod and releng SALs, search seems to work [15:39:04] add double quotes [15:39:04] https://tools.wmflabs.org/sal/wikidata-dev?p=0&q=%22wikidata-constraints%22&d= [15:39:10] yeah, that ^ [15:39:13] ah, thanks [15:39:23] so was the wikidata part matching every message due to the project name? :D [15:40:05] I'm not sure :) text search isn't like in the old days of string matching, it's all very smart now :) [15:40:15] without the quotes the search tokenizer would turn 'wikidata-constraints' into 'wikidata OR contraints' [15:40:19] "smart" [15:40:27] chasemp: use double quotes :P [15:41:28] SAL is backed by Elasticsearch and uses its 'simple query string query' syntax -- https://www.elastic.co/guide/en/elasticsearch/reference/current/query-dsl-simple-query-string-query.html [15:41:44] I should copy the help page from bash over to sal -- https://tools.wmflabs.org/bash/help [15:50:47] KPADSIBIJSWJREDMAQUIMNJKNJJHLCQNEEPGSNWJ Technical Advice IRC meeting starting in 10 minutes in channel #wikimedia-tech, hosts: @addshore & @CFisch_WMDE - all questions welcome, more infos: https://www.mediawiki.org/wiki/Technical_Advice_IRC_Meeting [16:38:18] !log tools T213418 shutdown tools-docker-registry-01 and 02. Will delete the instances in a week or so [16:38:21] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [16:38:21] T213418: Toolforge: move docker nodes from eqiad to eqiad1 - https://phabricator.wikimedia.org/T213418 [17:15:14] !log bastion shutting down all bastions in the eqiad region [17:15:15] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Bastion/SAL [17:26:52] !log striker deleting VM striker-mwvdev [17:26:54] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Striker/SAL [17:29:36] !log tools depooling and moving tools-sgeexec-0904 tools-sgeexec-0906 tools-sgewebgrid-lighttpd-0904 [17:29:37] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [19:48:38] \o [19:48:53] anyone know what is wrong with wmde-dashboards.wmde-dashboards.eqiad.wmflabs ? [19:49:44] addshore, it doesn't appear to be responding to ping [19:49:59] ill give it a kick / reboot [19:49:59] but it is theoretically ACTIVE [19:50:06] you may want to look at the console log in horizon [19:50:11] then reboot and look again [19:50:31] looks like it was only created half an hour ago [19:50:35] andrewbogott, ^ [19:50:55] the console / log looked normal, and it seems to be in the default security group etc [19:51:06] i hit reboot, lets see what happens [19:51:08] * andrewbogott looks [19:51:41] :D [19:51:58] log is at "wmde-dashboards login: " now [19:53:33] mind if I add myself to the project so I can look around? [19:53:46] andrewbogott: go for it [19:53:49] tis currently empty [19:55:18] port 22 is definitely not open in the default security group. Let me set some defaults... [19:55:49] you'll want ICMP too IIRC [19:56:07] addshore: try now? [19:56:13] pinging is working now [19:56:15] is this due to the project being created entirely within the new region? [19:56:34] and ssh works for me [19:56:37] GoranSM: ^^ [19:56:41] Krenair: I'm not sure. addshore do you know when the project was created? [19:56:49] addshore: Got it, I'm in [19:56:56] andrewbogott: the project was created some time ago (let me find the ticket) [19:57:00] huh okay [19:57:15] was there ever any VMs in there until today addshore ? [19:57:22] Thanks wikimedia-cloud guys [19:57:33] andrewbogott: https://phabricator.wikimedia.org/T185429#3928759 [19:57:37] Krenair: No no I've created this one some 30 mins ago [19:57:39] Krenair: no vms until today [19:57:43] right [19:57:46] Hm, a year ago, ok [19:57:55] it's possible this got an entirely blank default security group in eqiad1-r [19:58:00] and no migration was done, so [19:58:04] :D [19:58:06] So because it was empty I never migrated anything over from the old region [19:58:07] yeah [19:58:18] your welcome for being an edgecase ;) [19:58:40] might be some more of those lurking about, would check but I don't think novaobserver can see security groups still [19:59:08] Krenair: I am about to run a bunch of installations on the new instance now - just let me know if I should not, for any reason [19:59:24] should be fine AFAIK [19:59:29] I don't even have access to the instance so [19:59:33] petan|lounge: as part of our perpetual upgrade treadmill, I'd like to move wm-bot.wm-bot.eqiad.wmflabs to another host. Are there good times or bad times for me to do that? (Shouldn't be down for more than 30 minutes or so) [19:59:34] don't worry about me [20:00:03] thanks for your help all (as always) [20:00:05] GoranSM: it's all yours, all I did was change some of the network filtering rules in your project [20:00:22] andrewbogott: Thanks! [20:00:50] http://informativoangolano.com [20:01:13] informativo Angolano [20:01:41] aude [20:03:00] help [20:19:30] When I try to sudo, what password do I use? [20:20:50] DatGuy: if it asks for a password, you don't have admin rights on that host. What are you trying to do? [20:21:11] I might not be supposed to have permission for it in general, but I'm trying to install pyexiv2 [20:21:33] which in turn requires exiv [20:25:00] DatGuy: Ok! Do you need this on toolsforge? Or on a cloud vps project? [20:25:06] Toolforge [20:30:28] DatGuy: Ok, for that the toolforge admins will have to install it. Could you create a phabricator ticket with the #toolforge tag, listing the package(s) that need to be installed, and on which hosts you need them (bastions / exec nodes / webgrid / kubernetes -- or just describe what you intend to do)? [20:33:28] https://phabricator.wikimedia.org/T213965 valhallasw`cloud looks good? [20:34:16] DatGuy: ah. Unfortunately, ppa's can't be used due to security concerns. But let me check something... [20:34:48] Would https://packages.ubuntu.com/trusty/python-pyexiv2 work? [20:35:28] yep, that should be OK. The other option is to pip install py3exiv2, and to have the dependencies installed using apt-get [20:35:54] need sudo access for that [20:36:05] https://pastebin.com/mpzCC6HX [20:37:22] andrewbogott: Do you have a minute to inspect an issue on the new wmde-dashboards instance with me? [20:38:21] DatGuy: we recommend using python virtual environments for things like this. That will let you `pip install ...` all you want but put the packages in a place where only your tool can see them [20:38:35] Does it not need special stuff? [20:38:38] such as scons [20:38:43] DatGuy: fwiw, python-pyexiv2 is already installed system-wide. [20:38:49] oh I see [20:38:52] that I did not know [20:39:22] but given that python2 is EOL in... a year or so, I think having the dev dependency installed might be the way to go [20:47:53] valhallasw`cloud: excuse me for being a bit rusty, but what command would I need to do to get pyexiv2 into my venv pip packages? [20:48:45] iirc you need to create the venv with something like --system-site-packages. But that means you get _all_ system site-packages, not just the lib you need [20:53:24] oh hi valhallasw`cloud :D [20:53:51] Hauskatze: *waves* [20:54:51] I found a pyvenv.cfg file and changed 'include-system-site-packages' from false to true, but not sure how to restart it so to say :p [20:56:46] GoranSM: what's up? [20:56:55] andrewbogott: Hey [20:57:23] DatGuy: virtualenvs are black magic -- in my experience you have to recreate them for stuff like this to work [20:57:25] andrewbogott: Well, I have Rstudio Server installed and running on port 8787; all security groups set and applied to the new instance [20:57:44] However... http://wmde-dashboards.wmflabs.org/ - you can try it yourself [20:57:50] damn alright [20:58:01] andrewbogott: I've also allowed 8787 on ufw... [20:58:33] might just rewrite all the code as well, there's more spaghetti there than in an Italian restaurant [20:58:59] GoranSM: looking... [20:59:55] andrewbogott: If it helps, we have the RStudio Server running on the same port on the wikidataconcepts instance which you can also access, but I really tried to make everything the same... [21:00:06] DatGuy: if you have an existing venv, you should be able to run `pip freeze > $SOMEFILE` to get the list of things you already have in there. Then you can add those same packages to a new venv with `pip install -r $SOMEFILE` [21:00:40] $SOMEFILE is often called requirements.txt but that's just a random convention [21:00:42] ah, ok — GoranSM you opened 8787/tcp from 10.0.0.0/8 but the proxy gateway isn't in that range. 10.0.0.0/8 is the old 'eqiad' region which is mostly unused these days. [21:00:45] the new region is 172.16.0.0/21 [21:02:03] andrewbogott: So what I should I place there: 172.16.0.0/21 ? [21:02:17] yep [21:02:26] andrewbogott: Let me try [21:04:17] andrewbogott: No change [21:04:29] GoranSM: works for me! [21:04:49] andrewbogott: ? [21:04:53] Let me check again [21:05:37] andrewbogott: so, you're saying that http://wmde-dashboards.wmflabs.org/ returns a login page for RStudio Server in your browser? [21:05:41] GoranSM: works for me too. One thing that may have happened to you is local cache of a negative DNS lookup. That should expire soon in theory [21:06:00] bd808: Hm. Thanks! [21:06:02] GoranSM: yeah I get redirected to https://wmde-dashboards.wmflabs.org/auth-sign-in and see the form there [21:06:24] bd808: What I get is: Hmm. We’re having trouble finding that site. We can’t connect to the server at wmde-dashboards.wmflabs.org. [21:07:27] is that your browser's default DNS failure page? [21:07:52] Sounds like the built-in page in firefox [21:07:56] bd808: I'd say yes; that was Firefox. Chromium gives me: This site can’t be reachedwmde-dashboards.wmflabs.org’s server IP address could not be found.DNS_PROBE_FINISHED_NXDOMAIN [21:08:05] like http://foobar.bd808.com/ would give the same response [21:08:16] bd808: I think so. [21:09:43] yeah. cached negative lookup somewhere between our DNS server and you. That happens when you hit the URL before the DNS record is fully added by the proxy creation. From my local laptop I get: `host wmde-dashboards.wmflabs.org` -> "wmde-dashboards.wmflabs.org has address 185.15.56.49" [21:09:44] bd808, valhallasw`cloud: thank you! seems to import [21:10:12] GoranSM: I think we put a 5 or 10 minute TTL on negative responses [21:11:52] bd808: In translation to English, that means: wait 10 minutes or so and than try again, I guess? [21:12:03] :) yes [21:12:09] bd808: :) Thanks! [21:12:22] or lookup how to clear your local DNS cache if you are in a hurry [21:12:44] bd808: Well, not in a hurry but I could learn how to do that, why not [21:16:21] andrewbogott: RStudio Server up and running; thx [21:16:27] bd808: Thank you! [21:42:50] !log tools.sge-status Force killed orphan webservice grid job and restarted webservice [21:42:51] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.sge-status/SAL [21:53:41] andrewbogott: Now this is silly: I cannot SSH to wmde-dashboards.eqiad.wmflabs anymore. [21:54:51] GoranSM: any chance you installed a firewall on that host by mistake? [21:55:36] it looks like port 22 to that host from a bastion is blocked [21:56:40] security groups look right to me, must be something installed on the host [21:56:45] that will be hard to undo :( [21:57:14] hm [21:57:32] we could do this stuff if we had remote console access [21:57:36] welp, I'm on my way out the door… good luck absent stranger [21:57:55] he left it on the ticket: https://phabricator.wikimedia.org/T204695#4886191 [21:59:00] no special puppet roles applied -- https://tools.wmflabs.org/openstack-browser/server/wmde-dashboards.wmde-dashboards.eqiad.wmflabs [21:59:27] I did something dumb with ferm/iptables once that IIRC had the effect of breaking SSHd [21:59:31] I think I restarted the instance [21:59:32] could try that [21:59:47] (or maybe Andrew restarted it that time, I forget) [22:01:11] hi GoranSM [22:01:23] Krenair: Hi [22:01:37] was there any iptables or other firewall stuff on the instance? [22:01:47] Krenair: ufw installed [22:02:07] !log wmde-dashboard Added BryanDavis (self) as project admin to help debug ssh issues [22:02:08] bd808: Unknown project "wmde-dashboard" [22:02:21] bd808, plural [22:02:24] !log wmde-dashboards Added BryanDavis (self) as project admin to help debug ssh issues [22:02:25] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Wmde-dashboards/SAL [22:02:35] GoranSM, and is it possible that started blocking SSHd? [22:02:43] if ufw installed that is probably the problem [22:03:26] Krenair: Well, I did not allow 22 from ufw manually, however, I have been logging in and out from that instance following the installation of ufw [22:03:46] Krenair: the only thing I've used ufw for was to allow for 8787 (RStudio Server default port) [22:04:14] which implies that there was a default block? [22:04:55] bd808: I wouldn't say so, because if so how could I login and out from that instance? [22:05:21] bd808: But better not rely on my understanding of networking services [22:05:27] why would you need to add a rule to allow a port if you weren't blocking ports? [22:06:27] bd808: Oh, I see - I understand the question now. No I did not want to have anything blocked - I was running into problems with RStudio Server on 8787 and thought that I might need to open the port manually. [22:06:48] bd808: That is why I have manually allowed 8787 from ufw (and installed ufw in the first place) [22:06:48] GoranSM: one thing we can try is rebooting and hoping that the instance local firewall is actually not setup to start on boot [22:07:01] bd808: I have already did a soft reboot [22:07:13] bd808: Nothing changed. Let me try just another thing please. [22:07:19] maybe someone can mount the instance's disk and overwrite the rules etc.? [22:08:26] Krenair: No no wait [22:08:30] Solved. [22:08:36] Krenair: although possible in theory, in practice that instance would have to be full of bitcoin wallets to make it worthwhile ;) [22:08:51] heh [22:08:57] I have access to the terminal through RStudio Server - I only forgot about it [22:09:03] and then... sudo ufw allow 22 [22:09:22] this thing provides a remote shell? [22:09:27] So I can ssh there again. Silly of me to play with ufw, however, I needed 8787 in place. [22:09:49] Krenair: Yes. And if you ask me, that is a *serious* security issue with RStudio Server. [22:10:13] depends whether you trust it as much as you trust SSHd [22:10:15] I wouldn't though [22:10:40] Krenair: However, I can disable the RStudio Server terminal from the config files if that would match our security policy. Just let me know. [22:11:16] Krenair: However, as you can see, having me onboard exemplified that the feature is not useless at all :) [22:11:34] I'm not aware of any particular rule that would disallow this [22:11:46] GoranSM: its up to you, but if somebody takes over your instance and does ugly things that will be your problem to clean up [22:11:51] Krenair: Ok than. Sorry for panicking [22:11:58] I would be careful with it [22:12:09] I wouldn't allow anyone to use it that doesn't already have root on the instance [22:12:21] bd808: I understand. Well I will probably disable the terminal, I can do ssh there when I need to [22:12:50] bd808: And they need a Wikitech account to get there first, of course [22:13:34] GoranSM: a wikitech account? Is your authentication using the LDAP server? [22:13:48] bd808: I'd say so [22:13:51] that's more concerning to hear [22:14:00] how do they authenticate with this? [22:14:05] do they input their LDAP credentials? [22:14:10] Krenair: what do you mean? [22:14:21] when they go to the RStudio server interface [22:14:28] Krenair: Ok, I get your meaning [22:14:28] what credentials do they use to login? [22:14:51] Krenair: to get to RStudio Server I use my instance (terminal, console) username, and then my Wikitech password [22:15:04] Krenair: It has always worked that way for me [22:15:20] ... I'm pretty sure there either is or should be a rule against that [22:15:34] Using LDAP auth is against the Cloud Services TOU -- https://wikitech.wikimedia.org/wiki/Wikitech:Cloud_Services_Terms_of_use#What_uses_of_Cloud_Services_do_we_not_like? [22:15:41] "Use of Wikimedia's LDAP server for authentication: Even with end-user notification that is required for any service that handles passwords or other sensitive data, use of the Wikimedia LDAP server for password authentication from within the Cloud Services environment is strictly prohibited." [22:15:54] Your instance is asking for some of the most sensitive passwords to which it has no right to see [22:16:21] you can use OAuth or local users/passwords, but not LDAP/Wikitech/Developer accounts [22:16:23] Krenair: Well, my onboarding with WMDE took place in March 2017; since then, that is how I use the CloudVPS instances (aka Labs) [22:16:44] Krenair: I mean: there is nothing that I did to make it work that way. [22:16:59] so how does it know what LDAP server etc. to authenticate with? [22:17:04] Krenair: Since I know for Labs/CloudVPS, the only password that ever worked there was that one [22:17:14] did you register with it using that password? [22:17:28] Krenair: Registered with what? [22:17:40] with the RStudioServer software running on your instance [22:18:29] Krenair: If you mean do I login to the RStudio Server instance running from our CloudVPS virtual machine, then yes [22:18:46] Did you register with RStudioServer using the same password [22:18:49] Krenair: If you mean have I used my LDAP password to register anywhere outside our world, then no [22:18:55] Or does RStudioServer communicate with LDAP to verify passwords? [22:19:04] Krenair: That I do not know [22:19:10] Sorry but your LDAP password is already compromised, it has been seen by a labs instance [22:19:27] Krenair: Oh... [22:20:04] Krenair: How can you explain the following: [22:20:07] And you are in the LDAP NDA group. [22:20:17] Krenair: Yes I have NDA [22:20:39] Krenair: So, on two virtual instances (wikidataconcepts, and now wmde-dashboards that was just created) [22:20:47] Krenair: without any intervention from my side [22:21:04] Krenair: RStudio Server, upon installation, accepts that and only that password? [22:22:12] looks like rstudio uses pam for other. Specifically the /etc/pam.d/other profile [22:22:19] Maybe RStudioServer is reading your instance's ldap.conf or PAM [22:22:25] this is not exciting to find [22:22:49] GoranSM, do you have logs showing who else has logged into this system? [22:23:36] Krenair: Where by "this system" you mean: the two instances of RStudio Server running on our two CloudVPS instances? [22:23:43] yes [22:23:46] RStudio Server [22:24:22] Krenair: Let me take a look at the logs, but I would be very surprised to learn that anyone ever reach them except me [22:24:35] ok [22:26:55] Krenair: from https://support.rstudio.com/hc/en-us/articles/200554766-RStudio-Server-Application-Logs [22:27:07] Krenair: The RStudio Server log messages are located in the system log, typically located at: [22:27:11] /var/log/syslog [22:27:57] Krenair: So I can only download all syslog files from the wikidataconcepts instance (in use for approx. 2 years) and search for rstudio server messages there [22:28:29] grep on that file might show it [22:28:57] Krenair: Let me try [22:33:25] Krenair: Well, I can't find nothing evil in the logs, but it also does not seem to log who accessed the system and when [22:33:52] maybe auth.log? [22:34:38] Krenair: Would that also be found under /var/log [22:34:52] yes [22:35:15] Krenair: Let me see [22:38:25] Krenair: No, it doesn't write such things there either