[00:24:11] yuvipanda, can we use tools-bastion-03 now? [00:24:58] not yet. I need to add it to gridengine. I'm going to do that in a few minutes! [00:25:38] OK. [00:33:29] yuvipanda: So the bastion would be available under tools-login or not? And how big is that instance? [00:34:26] tom29739: try it now? [00:36:09] ah, a xlarge instance [00:36:10] Luke081515: yup! it's xlarge, so twice as big [00:36:10] great [00:36:24] It works. [00:36:35] Puppet run 22 minutes ago. [00:36:40] how to use it? tools-login, tools-dev or bastion? [00:37:01] I had to SSH in though tools-bastion-05 [00:37:49] * Luke081515 uses bastion-01 [00:37:56] (not tools-bastion) [00:38:24] I've always used login.tools.wmflabs.org to get in. [00:38:27] yuvipanda: So the next step is setting this instance as default for tools-login? [00:38:43] I'm using bastion.wmflabs a lot, because I got much more instances [00:38:52] Makes sense. [00:39:01] I only use tools. [00:39:45] * Luke081515 has access to a small instance, and is project admin for currently one xlarge and two medium instances [00:42:09] * tom29739 has access to no instances at all. [00:42:48] I've had no need to have an instance for any of my tools. [00:44:23] Luke081515: yup! [00:44:37] I'll announce and do that soon [00:46:13] great [00:56:43] * Luke081515 gives yuvipanda a big barnstart for the quick setup of a bigger bastion-host [00:57:41] yuvipanda: General question: I guess bastion-03 is not used much at the moment, why is this instance consuming 7.5 GB ram in near idle state? Is that the connection to grid? [01:02:10] yuvipanda, remember when I came to you regarding PHP die claiming mysqli_fetch_all didn't exist? [01:05:50] Luke081515: ah, depends on what you mean by 'use'. Linux uses them for buffers as much as possible, so it isn't really using all 7.5 [01:05:53] Cyberpower678: ?no [01:06:20] yuvipanda, I beleive you fixed the problem by installing a different mysqli package for php [01:06:26] tools-bastion-03 seems to be running all the stuff that 05 ran. [01:06:43] yuvipanda: Yeah, but a fresh xlarge instance at my project has < 1 GB cached and buffered, so why is there so much ram in use? [01:06:59] Luke081515: because lots of people have been using it, so lots of buffer usage I guess? [01:07:14] Cyberpower678: oh? was there a ticket for it? [01:07:23] I beleive so. [01:07:28] but bastion-03 is fresh and already not linked? So why should so much people use it already? [01:07:42] But I don't know how to search for tickets on Phabricator. The interface eludes me. [01:07:42] oh I see what you mean. I thought you were talking about -05 not -03 [01:08:40] Luke081515: good question, and I must admit my knowledge of Linux isn't good enough to answer that question [01:08:44] Cyberpower678: there's a search bar on top. [01:10:34] yuvipanda: Did you install something on that instance, that could consume this? I mean more than 5 GB ram were gan in less than one hour [01:11:15] Luke081515: so we ran puppet that installed like a lot of packages and ran for a long time :D [01:11:35] maybe this is the reason for the ram I guess :D [01:11:45] some programms running in background [01:11:49] :D [01:12:17] On htop I can see that -03 is running practically the same stuff as -05 did. Same python commands, same tools, etc. [01:13:50] But if it's an xlarge instance, then it should be able to accommodate that and even more stuff. [01:18:49] 6Labs, 10Labs-Infrastructure: Labs proxy api (aka 'Invisible Unicorn') is a spof - https://phabricator.wikimedia.org/T131308#2163975 (10yuvipanda) So I could backend the db to tools-db, but that means we'll stop being able to modify records when tools-db goes down. Is that acceptable? [01:20:04] yuvipanda, I can [01:20:09] 't fin the ticket. [01:21:40] yuvipanda, oh wait. Got it. https://phabricator.wikimedia.org/T125758 [01:22:33] Ah so you used mysqlnd [01:25:09] :D [01:30:01] 10Labs-project-wikistats: move wikistats database to c2 - https://phabricator.wikimedia.org/T110391#2163990 (10Dzahn) p:5Low>3Lowest [01:38:58] I can't switch tools-login yet because my 2factor auth stuff is dead (my phone is dead) [01:49:34] * tom29739 wishes Horizon didn't need 2FA. [02:30:17] 6Labs, 10Horizon: DNS Domains view in Horizon for Tools project displays only one domain - https://phabricator.wikimedia.org/T131334#2164011 (10yuvipanda) [04:31:26] ^ I can't really change any DNS right now because of ^ [04:43:29] 6Labs, 10Horizon: DNS Domains view in Horizon for Tools project displays only one domain - https://phabricator.wikimedia.org/T131334#2164011 (10Andrew) Every domain is owned by a project. Because 'wmflabs.org' domain is used by a bunch of different projects, it's owned by the 'wmflabsdotorg' project. Proxies... [04:54:05] 6Labs, 10Horizon: Increase Horizon Session timeout to a... much longer - https://phabricator.wikimedia.org/T131336#2164074 (10yuvipanda) [07:04:28] 6Labs, 7Tracking: create hashtags project in labs - https://phabricator.wikimedia.org/T131339#2164153 (10Slaporte) [07:48:28] 6Labs: create hashtags project in labs - https://phabricator.wikimedia.org/T131339#2164191 (10Peachey88) [08:34:52] PROBLEM - Host tools-bastion-01 is DOWN: CRITICAL - Host Unreachable (10.68.17.228) [08:36:23] tools-bastion-01 has been out of use for a while. [08:37:41] Why is shinken-wm still reporting about it? [09:02:09] 6Labs, 7Graphite: Delete non-existent instances from Graphite for Cyberbot project - https://phabricator.wikimedia.org/T131259#2161380 (10fgiunchedi) deleted instances are archived automatically after a grace period, would that work or is there some urgency? [09:06:55] 6Labs, 10Labs-Infrastructure: Labs proxy api (aka 'Invisible Unicorn') is a spof - https://phabricator.wikimedia.org/T131308#2164356 (10scfc) a:5yuvipanda>3None Where is `novaproxy-01` hardcoded? In the proxy or in the Horizon UI? AFAIU `modules/openstack/files/liberty/horizon/proxy/views.py`, `backendIn... [09:08:03] 6Labs, 10Labs-Infrastructure: Labs proxy api (aka 'Invisible Unicorn') is a spof - https://phabricator.wikimedia.org/T131308#2164365 (10scfc) a:3yuvipanda (Phabricator doesn't tell me what assignee I removed, but I think it was @yuvipanda.) [09:14:26] !log tools.heritage Commented out the Russian Wikipedia in user-config.py for [[Phab:T131344]] [09:14:27] T131344: Change unused_monument_images.py to support other sites than Wikipedia - https://phabricator.wikimedia.org/T131344 [09:14:30] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.heritage/SAL, Master [09:21:02] 6Labs, 7Tracking: New Labs project requests (tracking) - https://phabricator.wikimedia.org/T76375#2164387 (10bd808) [09:21:04] 6Labs, 15User-bd808: create hashtags project in labs - https://phabricator.wikimedia.org/T131339#2164384 (10bd808) 5Open>3Resolved a:3bd808 https://wikitech.wikimedia.org/wiki/Nova_Resource:Hashtags [10:11:48] 6Labs, 10Labs-Infrastructure: Allow CORS access to tools.wmflabs.org from all of wmflabs.org - https://phabricator.wikimedia.org/T131346#2164537 (10Magnus) [10:47:52] Hi. I find that the page.page_latest is not the latest revision of a page in Tool Labs database replication. May I get the latest revision of all pages? [10:50:05] Hello? [10:53:58] Sorry, are there anyone here can help me? [10:59:42] 6Labs, 10Labs-Infrastructure: Allow CORS access to tools.wmflabs.org from all of wmflabs.org - https://phabricator.wikimedia.org/T131346#2164537 (10scfc) What server should add/not add what header? AFAICS, https://tools.wmflabs.org/wdq2sparql/ does not have any restricting `Access-Control-*` headers. I think... [11:06:42] Hello? [11:33:24] Hello? [11:34:34] I find that the page.page_latest is not the latest revision id of a page in Tool Labs database replication. May I get the latest revision id of all pages in one operation? [11:37:57] 6Labs, 7Graphite: Delete non-existent instances from Graphite for Cyberbot project - https://phabricator.wikimedia.org/T131259#2164623 (10Cyberpower678) I was told that they would have to be removed manually. There is no urgency to this. [11:51:58] 6Labs, 10Labs-Infrastructure: Allow CORS access to tools.wmflabs.org from all of wmflabs.org - https://phabricator.wikimedia.org/T131346#2164631 (10Magnus) To repeat the very first sentence from my original ticket: Situation: I am trying to use the WDQ2SPARQL conversion tool (https://tools.wmflabs.org/wdq2spa... [12:03:12] 6Labs, 10Labs-Infrastructure: Allow CORS access to tools.wmflabs.org from all of wmflabs.org - https://phabricator.wikimedia.org/T131346#2164633 (10scfc) I did not suggest that you should write a proxy but that, if you need to add a header to a request's response at `tools.wmflabs.org`, you should be able to d... [12:14:56] 6Labs, 10Labs-Infrastructure: Allow CORS access to tools.wmflabs.org from all of wmflabs.org - https://phabricator.wikimedia.org/T131346#2164635 (10Magnus) Well, I don't know how tools.wmflabs.org is set up, so I can't tell you what to fix there. I suspect the header to add is ``` Access-Control-Allow-Origin:... [12:36:52] 6Labs, 10Labs-Infrastructure: Allow CORS access to tools.wmflabs.org from all of wmflabs.org - https://phabricator.wikimedia.org/T131346#2164650 (10Magnus) 5Open>3Invalid OK, I checked and apparently such wildcards are not supported for CORS, only "*" and specific domain names. So sub-domain level CORS is... [13:15:02] 6Labs, 7Graphite: Delete non-existent instances from Graphite for Cyberbot project - https://phabricator.wikimedia.org/T131259#2164688 (10fgiunchedi) 5Open>3Resolved a:3fgiunchedi ok, thanks @Cyberpower678 ! I'm resolving this as they seem both already archived [14:28:52] hmm [14:29:05] "The last Puppet run was at Fri Mar 25 21:33:14 UTC 2016 (8215 minutes ago). Puppet is disabled. disabled to stage a few resource sharing measures. Feel free to disable if this becomes an issue -- Chase" no new ssh key sync? [14:31:21] we don't keep keys locally revi [14:31:30] tools-login [14:31:31] I mean [14:31:49] I don't follow, what's are you trying to solve? [14:32:10] I added a new key to login, it didn't work, delete and add again, didn't work [14:32:17] logged in as an old key, saw that [14:32:30] Try tools-bastion-03 [14:32:48] right, we lookup keys in real time from ldap [14:32:54] so puppet there has no effect either way [14:33:07] Why is puppet run so often then? [14:33:23] 30m intervals is standard for puppet pretty universally [14:34:15] What does it do? I understand that it automates things, but what does it automate? [14:34:21] configurations [14:35:20] Meh. Another thing I don't need to worry about on tools. [14:35:36] tom29739: it's a long complex topic the gist of is, it's a declarative language to describe the end state of a host [14:35:46] based on a pull model where the host calls up every interval to see what's new [14:35:59] so it can be anything [14:36:27] copying configs from old laptop to new one, deploying new server, etc etc.... [14:36:31] but yeah it's pretty inconsequential for tools users in general [14:37:08] (03PS1) 10Merlijn van Deen: Port wikibugs logging to forrestbot [labs/tools/forrestbot] - 10https://gerrit.wikimedia.org/r/280668 [14:37:43] (03CR) 10jenkins-bot: [V: 04-1] Port wikibugs logging to forrestbot [labs/tools/forrestbot] - 10https://gerrit.wikimedia.org/r/280668 (owner: 10Merlijn van Deen) [14:38:29] I just know about it because of the 'last puppet run was at x' thing when you log in. [15:15:38] haha, i'm getting a very strong warning on deployment-tin [15:15:52] is that just an artifact of it sharing configs with prod? [15:15:58] 'While it is perfectly working, this is not the active deployment server.' [15:22:02] is git deploy broken in deployment-prep? [15:22:57] /Stage[main]/Eventlogging/Package[eventlogging/eventlogging]: Could not evaluate: source 'Error processing lin' not in PSON! [15:24:28] ottomata: you should ask in #wikimedia-releng [15:25:08] ook! [15:31:22] Hi, "wmflabs.org" does not respond. The tile service is dead (http://tiles.wmflabs.org/osm-no-labels/13/4372/2714.png) [15:34:49] (03PS2) 10Merlijn van Deen: Port wikibugs logging to forrestbot, flake8 fixes [labs/tools/forrestbot] - 10https://gerrit.wikimedia.org/r/280668 [15:35:21] (03CR) 10jenkins-bot: [V: 04-1] Port wikibugs logging to forrestbot, flake8 fixes [labs/tools/forrestbot] - 10https://gerrit.wikimedia.org/r/280668 (owner: 10Merlijn van Deen) [15:36:07] (03PS3) 10Merlijn van Deen: Port wikibugs logging to forrestbot, flake8 fixes [labs/tools/forrestbot] - 10https://gerrit.wikimedia.org/r/280668 [15:36:39] (03PS4) 10Merlijn van Deen: Port wikibugs logging to forrestbot, flake8 fixes [labs/tools/forrestbot] - 10https://gerrit.wikimedia.org/r/280668 [15:36:41] (03CR) 10jenkins-bot: [V: 04-1] Port wikibugs logging to forrestbot, flake8 fixes [labs/tools/forrestbot] - 10https://gerrit.wikimedia.org/r/280668 (owner: 10Merlijn van Deen) [15:38:00] James_F: your wish has been granted [15:39:07] valhallasw`cloud: Yay. Now fix the V-1? ;-) [15:39:15] I did! [15:39:21] the -1 is for PS3 [15:39:37] Aha. [15:39:42] * James_F glares at grrrit-wm . [15:39:52] grrrit-wm doesn't report +1s [15:40:00] for jenkins, at least [15:40:55] It doesn't report which PS the jenkins-bot reports are for, though. [15:41:45] fair point [15:42:22] same for code-review [15:42:32] but I guess there it makes more sense [15:42:52] Well. You can only comment on non-latest PSes. [15:43:49] * tom29739 tries to decipher all this because he doesn't know how to use Gerrit or how it works. :( [15:44:55] mwh, bastion-05 is slow again [15:45:00] my git pull take a lot of time [15:45:15] !log tools.luke081515bot deploying the latest 2 diffs [15:45:18] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.luke081515bot/SAL, Master [15:45:53] I'm actually enjoying tools-bastion-03. Given that it isn't used much because it isn't on tools-login and login.tools is good. [15:46:13] Probably won't stay that way for long though... [15:46:14] James_F: I seem to not have broken stuff! [15:46:24] I'm just executing my small sheel script, so I don't need much, tom29739 [15:46:34] the script does: git pull, delete jobs, start them again [15:46:34] (03CR) 10Merlijn van Deen: [C: 032] "Test run seems to work correctly" [labs/tools/forrestbot] - 10https://gerrit.wikimedia.org/r/280668 (owner: 10Merlijn van Deen) [15:47:04] (03Merged) 10jenkins-bot: Port wikibugs logging to forrestbot, flake8 fixes [labs/tools/forrestbot] - 10https://gerrit.wikimedia.org/r/280668 (owner: 10Merlijn van Deen) [15:47:26] valhallasw`cloud: Surprise! [15:47:42] seems like the owner of the "contentidcloud" project is very active today? [15:47:55] * Luke081515 is at the IRC feed of wikitech and gets spammed with these messages [15:48:01] I've been trying to create a page that runs a shell script to deploy the latest code when I push to github. [15:48:24] Luke081515: I think you mean contint? that's continuous integration, which creates/deletes vms for unit testing changesets [15:48:42] (03CR) 10Glaisher: "I tried it on my tool and it works fine there as well. Maybe this wasn't deployed completely? Can we try this again?" [labs/tools/stewardbots] - 10https://gerrit.wikimedia.org/r/278925 (owner: 10Glaisher) [15:49:03] ah, ok, so when gerrit is active, the labslogbot is very active too? [15:49:20] this night at about 2 UTC it was very quiet there [15:49:44] /nick YuviPanda [15:49:56] fail [15:50:04] valhallasw`cloud: contintcloud autocreates instances for jenkins tho [15:50:08] Luke081515: huh? what has labslogbot to do with it? [15:50:23] valhallasw`cloud: I can see at lot of entrys like this: [15:50:36] 17:47 <@rc-pmtpa> [[Nova Resource:Ci-jessie-wikimedia-66912.contintcloud.eqiad.wmflabs]] B https://wikitech.wikimedia.org/w/index.php?diff=404573&oldid=404572 * Labslogbot * (+50) Auto update of instance info. [15:50:42] that's why [15:51:41] !log tools.forrestbot deployed r280668, adapted crontab to log to ~/forrestbot.log [15:51:44] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.forrestbot/SAL, Master [15:52:14] Luke081515: oh, right. Yes, that's just triggered by account creation [15:52:41] (03CR) 10MarcoAurelio: "Sure, we can :-) Let's talk on IRC about this, since it might be easier than messaging us on gerrit :)" [labs/tools/stewardbots] - 10https://gerrit.wikimedia.org/r/278925 (owner: 10Glaisher) [15:55:10] 6Labs, 10Horizon: Increase Horizon Session timeout to a... much longer - https://phabricator.wikimedia.org/T131336#2165052 (10Andrew) [15:55:12] 6Labs, 10Horizon, 13Patch-For-Review, 7Tracking: Increase horizon session length - https://phabricator.wikimedia.org/T130621#2165053 (10Andrew) [15:56:25] 6Labs, 10DBA: Querying the logging table on labs is slow - https://phabricator.wikimedia.org/T131266#2165056 (10Sigma) Is there an ETA for their completion? I notice that some of those listed tasks are months old. [16:02:39] (03PS1) 10Glaisher: Improve elections.php (second attempt) [labs/tools/stewardbots] - 10https://gerrit.wikimedia.org/r/280684 [16:04:31] 6Labs, 10Labs-Infrastructure: Labs proxy api (aka 'Invisible Unicorn') is a spof - https://phabricator.wikimedia.org/T131308#2165079 (10yuvipanda) It's using a floating IP to talk to it. The SPOF is because the API uses an sqlite database backend, and so that file is only on novaproxy-01 [16:05:29] 6Labs, 10DBA: Querying the logging table on labs is slow - https://phabricator.wikimedia.org/T131266#2165095 (10jcrespo) Log table for enwiki has very recently finished importing. Do you still have issues? [16:07:05] 6Labs, 10Horizon: Proxy corner case: proxy name foo.wmflabs.org == domain name foo.wmflabs.org - https://phabricator.wikimedia.org/T131367#2165096 (10Andrew) [16:08:18] 6Labs, 10Horizon: Proxy corner case: proxy name foo.wmflabs.org == domain name foo.wmflabs.org - https://phabricator.wikimedia.org/T131367#2165108 (10Andrew) For the record, I'm not sure that fixing this is a better solution than just saying "Don't do that then" [16:08:38] (03PS2) 10MarcoAurelio: Improve elections.php (second attempt) [labs/tools/stewardbots] - 10https://gerrit.wikimedia.org/r/280684 (owner: 10Glaisher) [16:09:32] (03CR) 10MarcoAurelio: [C: 032] Improve elections.php (second attempt) [labs/tools/stewardbots] - 10https://gerrit.wikimedia.org/r/280684 (owner: 10Glaisher) [16:09:57] (03Merged) 10jenkins-bot: Improve elections.php (second attempt) [labs/tools/stewardbots] - 10https://gerrit.wikimedia.org/r/280684 (owner: 10Glaisher) [16:10:26] 6Labs, 10DBA: Querying the logging table on labs is slow - https://phabricator.wikimedia.org/T131266#2165124 (10Sigma) MariaDB [enwiki_p]> select log_timestamp, log_action from logging where log_namespace=0 and log_title='Great_Nuclear_Doge_Event' and log_type='delete' and log_timestamp > '20160121065733'; +--... [16:15:55] 6Labs, 10DBA: Querying the logging table on labs is slow - https://phabricator.wikimedia.org/T131266#2165131 (10jcrespo) p:5Low>3Normal So this is **not** directly caused by the import (it could still be a case of analytics being wrong due to the reimport). Let me play around a bit with the index stats. [16:43:58] 6Labs, 10DBA: Querying the logging table on labs is slow - https://phabricator.wikimedia.org/T131266#2165280 (10jcrespo) p:5Normal>3Low This is not an issue with analytics either, it is due to the filtering. Querying the same table in production and labs has the same performance, it is **when using the vie... [16:46:17] I need an wikitech admin [16:48:16] !admin [16:48:29] is there a command like this? [16:48:31] what [16:48:32] Luke081515: no. [16:48:43] some of you got wikitech admin permissions? [16:48:48] you've been here long enough to know you should either just ask your question or file a bug [16:49:11] Luke081515: ask your question [16:50:53] Hi.. Why is the stewardbots project web serving php 5.3 while CLI is php 5.5? Is it something configured there or on somewhere up on tool labs configuration? [16:51:07] ok, solved via query [16:51:11] you probably need to set up the webservice to run on trusty? [16:51:55] Yeah, is it configured differently on the project itself or on tool labs? [16:52:01] valhallasw`cloud: an user is vandalizing wikitechwiki, and has an abusive username which should be blocked with hideuser flag [16:52:17] mafk: he already blocked it, but maybe, yes [16:52:34] well, harassment shouldn't be allowed [16:52:45] yeah, that's true [16:52:52] oh hmm [16:52:56] glaisher@tools-bastion-05:/data/project/stewardbots$ cat service.manifest [16:52:59] web: lighttpd-precise [16:53:09] maybe that is changing it? ^ [16:53:21] does simply changing precise to trusty work? [16:53:27] and restarting the webservice [16:53:30] Glaisher: webservice --help [16:55:23] valhallasw`cloud: thanks :) [16:56:21] (03CR) 10Merlijn van Deen: [C: 032] Report labs/private to #wikimedia-operations [labs/tools/grrrit] - 10https://gerrit.wikimedia.org/r/266769 (owner: 10Merlijn van Deen) [16:56:33] (03CR) 10Merlijn van Deen: [C: 032] Add wikidata/.* to wikidata-feed [labs/tools/grrrit] - 10https://gerrit.wikimedia.org/r/247831 (owner: 10Addshore) [16:56:51] (03CR) 10Merlijn van Deen: [C: 032] Remove analytics/limn-wikidata-data [labs/tools/grrrit] - 10https://gerrit.wikimedia.org/r/273428 (owner: 10Addshore) [16:57:02] (03CR) 10Merlijn van Deen: [C: 032] Add analytics/wmde/.* to #wikimedia-de-tech [labs/tools/grrrit] - 10https://gerrit.wikimedia.org/r/273429 (owner: 10Addshore) [16:57:06] (03Merged) 10jenkins-bot: Report labs/private to #wikimedia-operations [labs/tools/grrrit] - 10https://gerrit.wikimedia.org/r/266769 (owner: 10Merlijn van Deen) [16:57:16] (03Merged) 10jenkins-bot: Add wikidata/.* to wikidata-feed [labs/tools/grrrit] - 10https://gerrit.wikimedia.org/r/247831 (owner: 10Addshore) [16:57:44] (03CR) 10jenkins-bot: [V: 04-1] Remove analytics/limn-wikidata-data [labs/tools/grrrit] - 10https://gerrit.wikimedia.org/r/273428 (owner: 10Addshore) [16:57:47] (03CR) 10jenkins-bot: [V: 04-1] Add analytics/wmde/.* to #wikimedia-de-tech [labs/tools/grrrit] - 10https://gerrit.wikimedia.org/r/273429 (owner: 10Addshore) [16:57:55] (03CR) 10jenkins-bot: [V: 04-1] Remove analytics/limn-wikidata-data [labs/tools/grrrit] - 10https://gerrit.wikimedia.org/r/273428 (owner: 10Addshore) [16:57:58] (03CR) 10jenkins-bot: [V: 04-1] Add analytics/wmde/.* to #wikimedia-de-tech [labs/tools/grrrit] - 10https://gerrit.wikimedia.org/r/273429 (owner: 10Addshore) [16:58:58] 10Tool-Labs-tools-Database-Queries, 10Phabricator: Archive Tool-Labs-tools-Database-Queries project - https://phabricator.wikimedia.org/T107699#2165358 (10jcrespo) I am going to be bold and create a "Backlog (help welcome)" as a column of #DBA to generalize all kind of help related to MySQL/Postgres, but I do... [17:01:21] 10Tool-Labs-tools-Database-Queries, 10DBA: HELP! Database is getting Slow: A test which took less than 10 min, now it takes 3 hours. I cannot do my experiments. - https://phabricator.wikimedia.org/T105964#2165384 (10jcrespo) [17:04:41] !log tools.stewardbots Merged https://gerrit.wikimedia.org/r/#/c/280684/ [17:04:45] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.stewardbots/SAL, Master [17:05:26] !log tools.stewardbots Webservice restart: webservice --release=trusty start [17:05:29] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.stewardbots/SAL, Master [17:08:59] (03PS1) 10Merlijn van Deen: Remove analytics/limn-wikidata-data Add analytics/wmde/.* to #wikimedia-de-tech [labs/tools/grrrit] - 10https://gerrit.wikimedia.org/r/280689 [17:09:14] (03CR) 10Merlijn van Deen: [C: 032] Remove analytics/limn-wikidata-data Add analytics/wmde/.* to #wikimedia-de-tech [labs/tools/grrrit] - 10https://gerrit.wikimedia.org/r/280689 (owner: 10Merlijn van Deen) [17:09:41] (03Abandoned) 10Merlijn van Deen: Add analytics/wmde/.* to #wikimedia-de-tech [labs/tools/grrrit] - 10https://gerrit.wikimedia.org/r/273429 (owner: 10Addshore) [17:09:46] (03Abandoned) 10Merlijn van Deen: Remove analytics/limn-wikidata-data [labs/tools/grrrit] - 10https://gerrit.wikimedia.org/r/273428 (owner: 10Addshore) [17:09:56] (03Merged) 10jenkins-bot: Remove analytics/limn-wikidata-data Add analytics/wmde/.* to #wikimedia-de-tech [labs/tools/grrrit] - 10https://gerrit.wikimedia.org/r/280689 (owner: 10Merlijn van Deen) [17:26:54] YuviPanda: halp [17:27:01] how do I use the docker build hosts? [17:27:12] because the internet is slow here [17:27:29] valhallasw`cloud: ah, use tools-docker-builder-03 [17:27:54] valhallasw`cloud: clone on /srv and use docker build -t yuvipanda/grrrit (I think?) . [17:28:02] YuviPanda: ok, will try [17:28:07] ssh is going to be painful [17:28:13] ouch [17:28:22] valhallasw`cloud: I can handle it if you want [17:28:25] (I can't ssh there from bastion, so no mosh) [17:29:06] I'll try [17:30:56] working now [17:32:03] andrewbogott: thanks for horizon! so nice \o/ [17:32:25] valhallasw`cloud: ok [17:33:25] marxarelli: great! [17:54:00] (03PS1) 10MarcoAurelio: Bring back requests.php [labs/tools/stewardbots] - 10https://gerrit.wikimedia.org/r/280695 [17:57:34] (03CR) 10MarcoAurelio: [C: 04-2 V: 04-1] "This won't work as it stands." [labs/tools/stewardbots] - 10https://gerrit.wikimedia.org/r/280695 (owner: 10MarcoAurelio) [17:57:56] (03PS2) 10MarcoAurelio: Bring back requests.php [labs/tools/stewardbots] - 10https://gerrit.wikimedia.org/r/280695 [17:58:16] (03CR) 10MarcoAurelio: [V: 04-1] Bring back requests.php [labs/tools/stewardbots] - 10https://gerrit.wikimedia.org/r/280695 (owner: 10MarcoAurelio) [17:58:18] 6Labs, 10DBA, 10wikitech.wikimedia.org: Changing rights via userrights-interwiki with '@labswiki' throws database-error - https://phabricator.wikimedia.org/T131385#2165553 (10Luke081515) [18:00:17] 6Labs, 10DBA, 10wikitech.wikimedia.org: Changing rights via userrights-interwiki with '@labswiki' throws database-error - https://phabricator.wikimedia.org/T131385#2165570 (10MarcoAurelio) The error message reported above is what I got when I tried changing my userrights at Meta using @labswiki as database. [18:03:13] 6Labs, 10DBA, 10wikitech.wikimedia.org: Changing rights via userrights-interwiki with '@labswiki' throws database-error - https://phabricator.wikimedia.org/T131385#2165604 (10Krenair) 5Open>3declined Yes, and this isn't going to be changed. [18:05:33] (03PS3) 10MarcoAurelio: Bring back requests.php [labs/tools/stewardbots] - 10https://gerrit.wikimedia.org/r/280695 [18:08:10] 6Labs, 10DBA, 10wikitech.wikimedia.org: Changing rights via userrights-interwiki with '@labswiki' throws database-error - https://phabricator.wikimedia.org/T131385#2165615 (10MarcoAurelio) I guess asking for an explanation other than "because I say so" would be too much asking? We were asked to OS something... [18:10:29] 6Labs, 10Labs-Infrastructure: Labs proxy api (aka 'Invisible Unicorn') is a spof - https://phabricator.wikimedia.org/T131308#2165619 (10scfc) [18:10:54] 6Labs, 10Labs-Infrastructure: Labs proxy api (aka 'Invisible Unicorn') is a spof - https://phabricator.wikimedia.org/T131308#2162479 (10scfc) Okay, I updated the task description. [18:18:45] 6Labs, 10DBA, 10wikitech.wikimedia.org: Changing rights via userrights-interwiki with '@labswiki' throws database-error - https://phabricator.wikimedia.org/T131385#2165626 (10Krenair) labswiki is supposed to be isolated like this, the firewall shouldn't be opened up to allow connections from any appserver ju... [18:26:04] 6Labs, 10DBA, 10wikitech.wikimedia.org: Changing rights via userrights-interwiki with '@labswiki' throws database-error - https://phabricator.wikimedia.org/T131385#2165553 (10jcrespo) @Krenair, I wouls like to respectfully disgree with the declined. Either this is allowed, or it is not, and it is an legitima... [18:29:03] 6Labs, 10DBA, 10wikitech.wikimedia.org: Changing rights via userrights-interwiki with '@labswiki' throws database-error - https://phabricator.wikimedia.org/T131385#2165650 (10Krenair) It is not to be allowed, it is not a legitimate bug, this is not a case of "not worth working on", it should not be done, the... [18:30:26] 6Labs, 10DBA, 10wikitech.wikimedia.org: Changing rights via userrights-interwiki with '@labswiki' throws database-error - https://phabricator.wikimedia.org/T131385#2165666 (10Krenair) You could open a task against MediaWiki core to have such DB connection failures during interwiki-userrights changes handled... [18:31:20] !log tools.lolrrit-wm Built new image,now ran kubectl --user=lolrrit-wm --namespace=lolrrit-wm delete pod grrrit-v9g1e and deo volente... [18:31:23] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.lolrrit-wm/SAL, Master [18:32:44] yuvipanda@tools-k8s-master-01:/home/valhallasw$ kubectl --user=lolrrit-wm --namespace=lolrrit-wm logs grrrit-fqbg3 [18:32:44] Error from server: Internal error occurred: Pod "grrrit-fqbg3" in namespace "lolrrit-wm": container "grrrit" is in waiting state. [18:32:47] what does that even mean [18:33:38] Error: ENOENT, no such file or directory '/srv/grrrit/connections.yaml' [18:33:39] ... [18:35:19] 6Labs, 10DBA, 10wikitech.wikimedia.org: Changing rights via userrights-interwiki with '@labswiki' throws database-error - https://phabricator.wikimedia.org/T131385#2165672 (10jcrespo) Yes, that was my point. Not allowing that, but failing gracefully. The errors shows at https://logstash.wikimedia.org/#dashbo... [18:35:32] valhallasw`cloud: I think that's in .gitignore [18:35:36] baaaaaaaaah [18:35:38] yes [18:35:45] why does the docker build succeed? [18:35:48] it really shouldn't [18:36:30] and ocnfig.yaml also needs updating [18:36:31] gah docker [18:37:03] valhallasw`cloud: yeah am pushing [18:37:05] we should really just loda config.yaml from NFS :P [18:37:08] oh ok [18:37:12] valhallasw`cloud: https://gerrit.wikimedia.org/r/280699 [18:37:25] YuviPanda: have you also updated config.yaml? [18:37:29] as in, are the passwrods in there? [18:37:47] valhallasw`cloud: nope [18:37:58] or shouldn't they be? I'm super confused about all of this [18:42:02] please document :> [18:44:38] valhallasw`cloud: I think there was supposed to be a password, and then we lost it at some point [18:44:44] there is a password [18:44:56] in the tool dir on toollabs [18:44:58] so when I built the image I think I didn't use the password [18:45:07] I don't know if we need to [18:45:17] probably not, although it might crash without? not sure [18:45:43] I am building it now [18:45:50] ok [18:54:37] 10Tool-Labs-tools-stewardbots, 6Stewards-and-global-tools, 13Patch-For-Review: Bring back requests.php - https://phabricator.wikimedia.org/T130028#2165742 (10MarcoAurelio) I've uploaded the old code of our tool we had on the toolserver, but it still needs major work to work. [18:54:52] 10Tool-Labs-tools-stewardbots, 6Stewards-and-global-tools, 13Patch-For-Review: Bring back requests.php - https://phabricator.wikimedia.org/T130028#2165755 (10MarcoAurelio) p:5Triage>3Normal [18:58:23] 10Tool-Labs-tools-Global-user-contributions, 7Easy, 7JavaScript: GUC project total count doesn't increment - https://phabricator.wikimedia.org/T123879#2165762 (10Krinkle) [19:20:49] valhallasw`cloud: is back up now. no password [19:38:26] PROBLEM - SSH on tools-exec-1404 is CRITICAL: Server answer [19:43:28] RECOVERY - SSH on tools-exec-1404 is OK: SSH OK - OpenSSH_6.9p1 Ubuntu-2~trusty1 (protocol 2.0) [19:54:19] Are global npm package installs supposed to fail? According to the npm website the errors I've been getting are permission errors. Should I just open a phabricator task to ask for it to be installed? [19:54:34] I'm trying to install Bower BTW. [19:57:08] tom29739: yes, because you don't have rights to install stuff globally [19:57:17] OK. [19:57:20] install it locally for the tool user (don't know the details of how that works for npm) [19:58:19] It has to be installed globally, apparently, because it contains a terminal command. It's only the ones which don't that can be installed locally. [19:58:37] that doesn't make sense [19:58:49] just add the local install location to PATH? [19:59:18] I could try that... [20:03:22] PROBLEM - SSH on tools-exec-1210 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:03:23] npm didn't moan when it was installing, which is good. [20:08:12] RECOVERY - SSH on tools-exec-1210 is OK: SSH OK - OpenSSH_6.6.1p1 Ubuntu-2ubuntu2~wmfprecise2 (protocol 2.0) [20:08:35] valhallasw`cloud, it works, just not very reliably, and it's very slow. [20:09:28] Don't know whether that's just the bastion being slow, or the tool being slow. [20:10:12] tom29739: probably a node tool needing many seperate files, which is slow over nfs [20:10:50] 6Labs, 10Labs-Infrastructure: Stop using 'testlabs' as the defacto auth project for 'novaadmin' - https://phabricator.wikimedia.org/T131395#2165953 (10Andrew) [20:11:15] Isn't that the type of thing that causes NFS outages and the like? [20:11:58] I don't think so [20:15:39] depends on teh volume more or less [20:18:19] 6Labs, 10Labs-Infrastructure, 13Patch-For-Review: Stop using 'testlabs' as the defacto auth project for 'novaadmin' - https://phabricator.wikimedia.org/T131395#2166018 (10Andrew) Ooh, it turns out I already did this for wikitech, which was the scary bit. [20:34:17] 6Labs, 10Labs-Infrastructure, 13Patch-For-Review: Stop using 'testlabs' as the defacto auth project for 'novaadmin' - https://phabricator.wikimedia.org/T131395#2166162 (10Andrew) 5Open>3Resolved I just removed the 'admin' role from novaadmin/testlabs. [20:37:32] How one can revoke access to an user to a tool labs project? [20:37:57] and a way to see all users with access? [20:38:34] mafk: Wait a moment [20:38:37] I can explain it [20:38:43] 6Labs: Maybe delete jitsi project - https://phabricator.wikimedia.org/T130050#2166193 (10Andrew) 5Open>3Resolved deleted. [20:38:54] * Luke081515 needs to search for his 2FA data first [20:39:11] Luke081515: if it's going to be a disturbance please don't [20:39:21] no, I finished [20:39:29] so, go here: https://wikitech.wikimedia.org/wiki/Special:NovaServiceGroup [20:39:34] 6Labs, 7Tracking: Make OpenStack Horizon useful for production labs - https://phabricator.wikimedia.org/T87279#2166200 (10Andrew) [20:39:36] 6Labs, 10Labs-Sprint-103, 10Labs-Sprint-104, 10Labs-Sprint-105: Upstream: Limit available images on horizon - https://phabricator.wikimedia.org/T91782#2166197 (10Andrew) 5stalled>3Resolved a:3Andrew This is no longer needed; the upstream added the 'disabled' property instead. [20:39:44] at the sidebar it's the link "manage Service groups" [20:40:01] there you can click: manage members [20:40:22] Luke081515: don't use service groups in any project outside of 'tools,' they're about to be deprecated. [20:40:36] yeah, but it is tools :D [20:40:41] oh, sorry, didn't read the backscroll, ignore me :) [20:40:45] np ;) [20:40:48] kneejerk [20:41:06] mafk: So for stewardbot, this would be here: https://wikitech.wikimedia.org/w/index.php?title=Special:NovaServiceGroup&action=managemembers&projectname=tools&servicegroupname=tools.stewardbots&returnto=Special%3ANovaServiceGroup [20:41:12] 6Labs, 7Tracking: Make OpenStack Horizon useful for production labs - https://phabricator.wikimedia.org/T87279#987805 (10Andrew) [20:41:18] and then you can add or remove members [20:41:49] and when I add a member there he's given access to the tool.labs project? [20:41:56] nothing more to do? [20:42:05] 6Labs, 13Patch-For-Review, 5WMF-deploy-2016-03-22_(1.27.0-wmf.18), 5WMF-deploy-2016-03-29_(1.27.0-wmf.19): Switch to using Horizon/Designate for labs public dns - https://phabricator.wikimedia.org/T124184#2166208 (10Andrew) [20:42:20] no chmod strange stuff to be done on the tool itself? [20:42:53] mafk: Then he can become the tools account [20:43:04] and the tools account is the owner of the most data of your project [20:43:08] so no need to change chmod [20:43:15] okay, thanks [20:43:18] mafk: But be careful, then the user is able to remove you too [20:43:22] from the project [20:43:25] xD [20:43:36] or I can remove them [20:43:49] besides, I can lock their accounts [20:43:59] oh wait... they are stewards too [20:44:30] 10MediaWiki-extensions-OpenStackManager: Cant add a security group to an existing instance - https://phabricator.wikimedia.org/T42525#2166229 (10Andrew) 5stalled>3Resolved a:3Andrew Yep, works in Horizon. [20:44:48] mafk: Yeah, but once locked, you can't unlock, so it's a criteria who is faster xD [20:44:53] or has DB access [20:49:31] * Krinkle notices that tools-login MOTD says that puppet has been disabled since March 25 [20:49:36] is the maintenance done? [20:50:07] (bastion-05) [20:52:54] YuviPanda, ^ [20:53:19] Krenair: hey! I think chasemp staged some cgroups stuff there. [20:53:27] and I don't think that's gonna go away. [20:53:33] Krinkle was asking, I'm Krenair [20:53:41] heh [20:53:50] bah [20:53:54] :p [20:54:51] give me a sec here and I'll release that lock [20:55:03] 6Labs, 10Labs-Infrastructure, 10MediaWiki-extensions-OpenStackManager, 13Patch-For-Review: Clean up after ldap->mysql keystone migration - https://phabricator.wikimedia.org/T126758#2166251 (10Andrew) [20:55:05] 6Labs, 10Labs-Infrastructure, 10labs-sprint-117, 10labs-sprint-118, and 2 others: Move project membership/assignment from ldap to keystone mysql - https://phabricator.wikimedia.org/T115029#2166250 (10Andrew) [20:55:16] I hate to leave it stale but it was in dire state for awhile [20:55:28] 6Labs, 10Labs-Infrastructure, 10labs-sprint-117: switch to keystone api v3 - https://phabricator.wikimedia.org/T115027#2166255 (10Andrew) [20:55:30] 6Labs, 10Labs-Infrastructure, 10labs-sprint-117, 10labs-sprint-118, and 2 others: Move project membership/assignment from ldap to keystone mysql - https://phabricator.wikimedia.org/T115029#1712891 (10Andrew) 5Open>3Resolved This is all done except for cleaning up leftover ldap junk. [21:05:23] PROBLEM - SSH on tools-exec-1210 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:16:11] tools-bastion-05 and tools-bastion-03 are both currently running. Is one planned to take over the other or something, or are they just going to stay in parallel? [21:31:08] tom29739: I think -03 will take over today. -05 will continue to run for a while tho [21:33:14] It's acting up a bit. The CPU is at 0.2%, but my SSH is jittery. Probably due to my net connection, rather than the bastion, but it isn't usually like that. [21:34:55] tom29739: try out mosh [21:56:18] !mosh [21:56:21] !ssh [21:56:21] https://labsconsole.wikimedia.org/wiki/Help:SSH [21:56:33] -.- [21:56:47] !ssh is https://wikitech.wikimedia.org/wiki/Help:SSH [21:56:47] This key already exists - remove it, if you want to change it [21:56:49] !ssh del [21:56:50] Successfully removed ssh [21:56:52] !ssh is https://wikitech.wikimedia.org/wiki/Help:SSH [21:56:53] Key was added [21:58:03] !mosh is You should use mosh, you'll thank me later: https://en.wikipedia.org/wiki/Mosh_%28software%29 [21:58:04] Key was added [22:01:23] PROBLEM - SSH on tools-exec-1210 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:06:13] RECOVERY - SSH on tools-exec-1210 is OK: SSH OK - OpenSSH_6.6.1p1 Ubuntu-2ubuntu2~wmfprecise2 (protocol 2.0) [22:37:22] PROBLEM - SSH on tools-exec-1210 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:42:52] 6Labs: Add 'novaadmin' to new projects, always - https://phabricator.wikimedia.org/T131411#2166554 (10Andrew) [22:44:58] 6Labs: Add 'novaadmin' to new projects, always - https://phabricator.wikimedia.org/T131411#2166554 (10yuvipanda) This causes a puppet failure on labservices1001 & holmium if we want a new project to have a floating IP that's reachable from inside labs (https://gerrit.wikimedia.org/r/#/c/280768/) [22:47:15] RECOVERY - SSH on tools-exec-1210 is OK: SSH OK - OpenSSH_6.6.1p1 Ubuntu-2ubuntu2~wmfprecise2 (protocol 2.0) [23:01:23] I just logged in to horizon, it looks nice, btw. google authenticator is sanctioned against Iran, I had to do a trick to get it installed in my phone [23:02:48] andrewbogott: Currently here? [23:03:16] for another 5 minutes or so [23:03:29] Amir1: the /app/ is sanctioned? Weird. [23:03:48] andrewbogott: Something is strange with dns [23:03:48] code.google.com is blocked by google for Iranian IPs [23:03:52] I guess we should make a page listing alternatives, there are plenty of other apps that support the same algorithm [23:03:56] Amir1: ah! I see, makes sense. [23:04:11] my old proxy got the domain: .wmflabs.org. if I create a new one, there is no '.' after the org [23:04:20] strangely matlab website is also bloacked, [23:04:29] it's like we can't use proxies in Iran ;) [23:04:31] Luke081515: the terminal . was causing breakage with the backend, so we changed things [23:04:37] and it was freebase [23:04:45] *blocked [23:04:46] ok, but the missing '.' will not cause errors? [23:04:54] Luke081515: no, but having it might [23:19:11]