[00:47:30] RECOVERY - Puppet failure on tools-exec-1205 is OK: OK: Less than 1.00% above the threshold [0.0] [00:48:19] it should be called "Pupet run is OK" [00:48:26] not "Puppet failure is OK" :p [00:48:45] always gotta look twice [01:07:02] 6Labs, 10Beta-Cluster-Infrastructure: New labs instance fails running `block-for-project-export` before running mount - https://phabricator.wikimedia.org/T126568#2017518 (10thcipriani) 3NEW [01:09:28] Think I've got it, 160,000 in 52 minutes (0.02 sec per page). page_id and page_namespace don't mix well. [01:47:52] labservices1001 - out of disk space [01:47:58] doesnt seem good [01:48:06] tries [01:56:09] 6Labs, 10Labs-Infrastructure, 6operations: labservices1001 ran out of disk space - https://phabricator.wikimedia.org/T126572#2017616 (10Dzahn) 3NEW [01:56:47] a / under 10G is always gonna be trouble [01:57:10] even with logrotate, just too easy to have 4G of that filled with logs [01:57:21] like in this case, designate/mdns [01:57:43] might explain some earlier DNS issues? [02:01:21] 6Labs, 10Labs-Infrastructure, 6operations: labservices1001 ran out of disk space - https://phabricator.wikimedia.org/T126572#2017627 (10Dzahn) deleted some of the older rotated logs, gzipped designate-central.log which is also large [02:15:55] (03CR) 10Legoktm: [C: 032] -releng and -devtools changes [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/269737 (owner: 10Greg Grossmeier) [02:16:30] (03Merged) 10jenkins-bot: -releng and -devtools changes [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/269737 (owner: 10Greg Grossmeier) [02:16:46] !log tools.wikibugs Updated channels.yaml to: 164d65c02aa22ec6e53f05ec74c35dfd58d11c24 -releng and -devtools changes [02:16:51] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.wikibugs/SAL, Master [04:05:07] andrewbogott, i created a labs vm instance in the wikitextexp project, but I am getting a permission denied ... [04:05:22] [subbu@earth ~] ssh ssastry@nlwiki.wikitextexp.eqiad.wmflabs [04:05:23] Permission denied (publickey). [04:26:19] subbu: your instance is broken in a way that was reported earlier today by MarkT.raceur as well. I'm going to file a bug [04:26:54] in the mean time I would suggest spinning up another vm and hoping the problem is transient [04:40:16] 6Labs: Instances broken on initial provision with dns setup issues - https://phabricator.wikimedia.org/T126580#2017808 (10bd808) 3NEW [04:43:43] 6Labs, 10Beta-Cluster-Infrastructure: New labs instance fails running `block-for-project-export` before running mount - https://phabricator.wikimedia.org/T126568#2017817 (10greg) [04:47:59] 6Labs, 10Beta-Cluster-Infrastructure: New labs instance fails running `block-for-project-export` before running mount - https://phabricator.wikimedia.org/T126568#2017820 (10bd808) Did @yuvipanda start making changes to remove NFS from the non-MediaWiki nodes in beta cluster? I can't find a task but remember an... [07:40:16] Hey guys, Is https://wikitech.wikimedia.org/wiki/RCStream down? I don't see any result even from the provided examples. [07:56:53] 6Labs, 6operations, 10wikitech.wikimedia.org: Rename specific account in LDAP, Wikitech, Gerrit and Phabricator - https://phabricator.wikimedia.org/T85913#2017918 (10adrianheine) 5Open>3Resolved Thank you very much, @demon. I've yet to find any issue :) [08:10:28] 6Labs, 10Beta-Cluster-Infrastructure: Completely remove Beta Cluster dependency on NFS - https://phabricator.wikimedia.org/T102953#2017949 (10hashar) [08:10:30] 6Labs, 10Beta-Cluster-Infrastructure: Disable /data/project for instances in deployment-prep that do not need it - https://phabricator.wikimedia.org/T125624#2017948 (10hashar) [08:11:01] 6Labs, 10Beta-Cluster-Infrastructure: Disable /data/project for instances in deployment-prep that do not need it - https://phabricator.wikimedia.org/T125624#1992944 (10hashar) Similar to T102953 itself blocked on Swift [08:11:59] 6Labs, 10Beta-Cluster-Infrastructure: New labs instance fails running `block-for-project-export` before running mount - https://phabricator.wikimedia.org/T126568#2017962 (10hashar) A recent task is {T125624} [08:12:55] 6Labs, 10Beta-Cluster-Infrastructure: New labs instance fails running `block-for-project-export` before running mount - https://phabricator.wikimedia.org/T126568#2017966 (10hashar) Forgot to quote the relevant part from Yuvi: > The way to disable this would be to set mount_nfs hiera variable to false in Hiera:... [08:20:25] 6Labs, 10Beta-Cluster-Infrastructure: New labs instance fails running `block-for-project-export` before running mount - https://phabricator.wikimedia.org/T126568#2017975 (10hashar) IIRC the NFS server whitelists instances of a project on a per IP basis. So potentially deployment-tin with IP 10.68.17.240 would... [09:15:07] PROBLEM - Free space - all mounts on tools-worker-1001 is CRITICAL: CRITICAL: tools.tools-worker-1001.diskspace.root.byte_percentfree (<22.22%) [10:28:46] 6Labs, 6operations, 10wikitech.wikimedia.org: Rename specific account in LDAP, Wikitech, Gerrit and Phabricator - https://phabricator.wikimedia.org/T85913#2018140 (10Krenair) @demon, are all the steps you took documented? [10:59:51] Hey guys, Is https://wikitech.wikimedia.org/wiki/RCStream down? I don't see any result even from the provided examples. [11:04:39] http://stream.wikimedia.org says "404 Not Found" but of course I am not sure if this ever was supposed to say something else or not [11:28:39] ebraminio, it's working for me [11:28:50] ebraminio, you're not supposed to connect over http like that [11:29:15] with an ordinary browser anyway [11:30:10] Thank you Krenair, I see this with the provided python sample [11:30:13] self.handshake_response = handshake(self.sock, *addrs, **options) [11:30:14] File "/usr/local/lib/python2.7/site-packages/websocket/_handshake.py", line 70, in handshake [11:30:14] raise WebSocketException("Invalid WebSocket Header") [11:30:15] websocket._exceptions.WebSocketException: Invalid WebSocket Header [11:30:28] My interest of course is nodejs bot I was running before [11:30:47] which is not working anymore :/ [11:31:35] this code works: https://phabricator.wikimedia.org/P2592 [11:35:37] Thank you Krenair, it seems the issue is just limited to my local machine [11:35:53] might be your version of socketIO [12:28:39] Hello everyone [13:17:50] As a new git user, i can't add 'public_html' folder by 'git add' from the tool instance, returning permission denied. On tools.vocabulary-index. [13:19:15] drwxrwsr-x 2 tools.vocabulary-index tools.vocabulary-index 4096 Jan 30 12:47 public_html [13:30:23] sorry i reached to 'git add public.html' via y-verciti@tools [13:53:39] then i commit but when i clone i don't receive the public_html folder [14:01:42] seems there is a rebase in progress on deplyoment-puppetmaster (I'm trying to cherry-pick my change to test it on lab). How do we coordinate those kind of things ? [14:02:21] I suspect the rebase is being done by thcipriani|afk or twentyafterfour [14:02:22] bd808, ah, ok. [14:02:55] which rebase? [14:03:02] hmm.. [14:03:24] quote: You are currently rebasing branch 'production' on '7f43aeb'. [14:03:26] gehel: it's ok [14:03:35] let me see... [14:04:11] gehel: no that's not me rebasing [14:04:45] * gehel wonders how to find out who it is [14:05:38] bd808, no luck after i deleted the old instance and spun up a new one. [14:06:06] twentyafterfour: you and thcipriani|afk are the users logged in currently (that was the extent of my suspicion). Don't even know how I would go about finding who that is... everything seems to be done as sudo on this repo... [14:06:46] 6Labs: Instances broken on initial provision with dns setup issues - https://phabricator.wikimedia.org/T126580#2018529 (10ssastry) I deleted this instance and spun up a new one. The new one failed with the same issue. [14:06:47] .git/rebase-merge/* has some info [14:06:51] still not sure how [14:07:22] I'll start digging ... [14:07:31] gehel: the only other person who's logged in to that machine is dcausse [14:07:46] dcausse: you left a rebase in progress on deployment-puppetmaster? [14:07:54] usually, how do you ensure that there is not 2 persons doing changes at the same time = [14:08:05] twentyafterfour: no? [14:08:08] gehel: we don't have any way to ensure that [14:08:26] dcausse: ok, sorry you showed up in the last log... [14:08:41] twentyafterfour: and usually it never happens, I'm just lucky for my first cherry-pick ;-) [14:09:59] gehel: I guess abort the rebase? [14:10:54] twentyafterfour: my assumption is that there is a reason for this rebase to be in progress... [14:11:35] 7f43aeb is https://gerrit.wikimedia.org/r/#/c/269828/ [14:12:16] well I don't know... maybe me but I did not run any git command today :/ [14:13:44] dcausse: 7f43aeb is the HEAD of origin/master [14:14:33] may be it's me and I screwed my cherry-pick just on my own ... [14:15:37] ebernhardson: you should use an older SocketIO version [14:15:42] aaah [14:15:45] wrong tab, sorry [14:16:25] Youni: you probably don't have write access to the .git folder? Try 'take .git' [14:20:40] i'm trying it now with the user account [14:21:23] 6Labs, 6operations, 10wikitech.wikimedia.org, 5Patch-For-Review: Deploy Mathoid for Wikitech too, or texvc as fallback - https://phabricator.wikimedia.org/T126468#2018558 (10Dereckson) 5Open>3Resolved a:3Dereckson [14:21:26] 6Labs, 10Wikimedia-Site-Requests, 10wikitech.wikimedia.org, 7Blocked-on-Operations, 5Patch-For-Review: Enable math extension on wikitech - https://phabricator.wikimedia.org/T126338#2018560 (10Dereckson) [14:21:46] I'm aborting the rebase on deployment-puppetmaster [14:22:57] It seems to be ok i have this: [14:23:10] $>drwxr-sr-x 8 y-verciti tools.vocabulary-index 4096 Feb 11 13:55 .git [14:24:50] merge issue on deployment-puppet seems related to git-sync-upstream [14:25:48] * gehel getting some coffee before having a deeper look into failed automatic rebase [14:38:10] Then i reached to add the public html folder and i commit but when i clone on my local station i don't see the folder index_html [14:50:04] 6Labs: Instances broken on initial provision with dns setup issues - https://phabricator.wikimedia.org/T126580#2018640 (10ssastry) [15:12:37] Youni: no, that's not OK -- the directory is owned by y-verciti, not tools.vocabulary-index [15:15:59] !log deployment-prep fixed deployment-puppetmaster rebase conflict by removing commit 814f12bc - author is informed [15:16:03] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Deployment-prep/SAL, Master [15:20:33] Youni: as tools.vocabulary-index, run `take .git`, then try the whole git add and git commit routine again [15:24:36] gehel: did you figure out that the rebase was caused by cron? The self-hosted puppetmasters have a cron job to rebase on the upstream. It gets broken when people leave stale cherry-picks in beta cluster or there is real merge conflict with upstream changes [15:25:15] gehel: also if you need a tour of logstash things let me know. I'm the responsible party for most of the setup. [15:25:58] bd808: yep, figured out and fixed (well, removed the incriminating commit after talking to yurik) [15:26:14] cool [15:26:33] its not my fault officer! someone slipped it into my pocket [15:26:44] bd808: kool ! I'm ready to apply my patch and start sending logs from elasticsearch (lab) [15:27:16] yurik: the commit was incriminating, I did not say who was incriminated by the commit, but your reaction is telling ... [15:27:17] nice. You did guard against the logstash ES boxes sending logs to themselves right? [15:27:34] :))) [15:28:32] bd808: it is configured only for the discovery elastic search cluster (as far as I understand it). If you want to have a look at https://gerrit.wikimedia.org/r/#/c/269100/ to double check ... [15:31:06] that will apply to the ES on deployment-logstash2 as well [15:31:17] bd808: now that you say, I might have assumed something wrong ... [15:31:48] bd808: I assumed the puppet-logstash module was self contained, I should have checked (just did) [15:31:50] we use the same puppet class to provision ES everywhere [15:32:06] which is a good thing ! [15:32:17] most of the time, yes :) [15:32:59] what is the correct differentiator between "logstash elasticsearch" and "other elastic search" [15:33:08] there is a host based hiera file for deployment-logstash2 where you can turn things off [15:33:34] in prod there is a role hiera file but that bit of magic doesn't work in Labs [15:34:01] gehel: https://github.com/wikimedia/operations-puppet/blob/production/hieradata/labs/deployment-prep/host/deployment-logstash2.yaml [15:34:51] deployment-logstash2 gets settings from that file first and then falls back to common.yaml for things that are missing [15:35:09] (plus there is Hiera: namesapce stuff possible on wikitech but yuck) [15:36:06] valhallasw`cloud: thanks i'm trying right now [15:36:19] so I could set `elasticsearch::graylog_host` to undef in deployment-logstash2 and set it in common.yaml (is that even possible? need to check) [15:37:09] gehel: for prod things look at hiera_lookup in puppet repo should help work out how the hieararchy works [15:37:20] and then for labs instances there is a slightly diff path look at [15:37:37] gehel: yeah, I'm not sure if you can set something to "undef" [15:37:41] modules/puppetmaster/files/labs.hiera.yaml [15:38:03] you can but '' is just as effective [15:38:06] idk whqt is better [15:38:50] valhallasw`cloud: ok the owner changed for the tool account [15:39:19] i might re-introduce a boolean flag to activate (or not) sending logs to logstash. Relying on undef is in the end much harder to understand... [15:39:47] keep the separation between activating logs and where to send them [15:41:25] this is mostly a problem of the roles hiera mechanism not working for Labs projects [15:41:40] in prod you will be able to separate the config based on role [15:52:53] bd808: but in prod, I'd probably still want to "enable log shipping for everyone except logstash", which is awkward if disabling log shipping is done on a parameter being undef [15:53:55] gehel: for prod your config can go in -- https://github.com/wikimedia/operations-puppet/blob/production/hieradata/role/common/elasticsearch/server.yaml [15:54:14] which will not apply to the ES cluster that role::logstash uses [15:54:46] that cluster is configured by -- https://github.com/wikimedia/operations-puppet/blob/production/hieradata/role/common/logstash/elasticsearch.yaml [15:55:24] bd808: I'd still think that which logstash server is used in prod is a common definition that should not be related to a specific role. [15:55:52] valhallasw`cloud: I cloned more once, no index_html folder... than put on new file on my new clone ; i can add it commit it but i cant't push it. git returns : error: Cannot access URL http://tools-static.wmflabs.org/vocabulary-index/vocabulary-index.git/, return code 22 [15:57:03] i need a lunch break, i'll come back! [16:09:43] bd808: about picking a random server, you mean at runtime? The GELF appender that I'm using does not support it. [16:10:48] Hi all [16:10:51] gehel: yeah I figured that would be the case. Puppet has some consistent hash by hostname magic that would maybe at least let you configure each host to pick one of N available backends. [16:10:55] bd808: do you think keeping the unused appender in the config is cleaner? I find it confusing to have dead config... [16:11:08] at the Wikipedia Library, we have been running into issues configuring a server for a contractor: https://phabricator.wikimedia.org/T118247 [16:11:26] bd808: I can add round-robin to the appender, but that's going to take more time... [16:11:31] gehel: it's a 50/50 thing I guess. I dislike if blocks in the erb [16:12:03] bd808: don't we have some kind of load balancer in front of the logstash cluster ? [16:12:03] gehel: *nod* spreading the load isn't a blocker but something we should be mindful of [16:12:07] nope [16:12:18] and an LB wouldn't work for GELF [16:12:23] Youni: no, that's not going to work. The easiest is to use a remote git server (WMF gerrit, bitbucket, github, ...) and push/pull from there [16:12:44] bd808: right, reassembling would not work... [16:12:45] GELF tosses out multiple UDP datagrams per-message and they need to be reassembled on the logstash side [16:13:00] GELF is crap honestly [16:13:27] bd808: you have anything better? I'm opened to suggestions... [16:13:38] but java things and node things tend to make stacktraces that won't work with other UDP transports [16:14:21] yep, my thinking as well... [16:14:43] we have a small percentage of log loss for MW even where the syslog encoding cuts off a json blob [16:14:52] well, let's try to move this forward, and already test if it works... can improve later ... [16:14:55] 6Labs, 6operations, 10wikitech.wikimedia.org, 5Patch-For-Review: Failing wikitech logins - https://phabricator.wikimedia.org/T126322#2018828 (10Andrew) 5Open>3Resolved ok, I'm happy with how this is fixed now. [16:15:01] gehel: *nod* [16:15:29] andrewbogott: according to your announce "[Labs-l] [Labs-announce] Wikitech maintenance and CI downtime tomorrow, 2015-02-12 16:00 UTC": what is "CI"? [16:15:53] doctaxon: Continuous Integration, e.g. jenkins [16:17:04] andrewbogott: does it affect my work on toollabs as /data/project/taxonbot [16:17:30] only if you’re using gerrit.wikimedia.org for your source control [16:18:27] thx [16:29:20] Change on 12wikitech.wikimedia.org a page Nova Resource:Tools/Access Request/Volans was created, changed by Volans link https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/Access_Request/Volans edit summary: Created page with "{{Tools Access Request |Justification=Get to know the tool labs from the user point of view as I started working with the wikimedia-operations team. |Completed=false |User Nam..." [16:38:52] 6Labs: track growth of project and home directories - https://phabricator.wikimedia.org/T126623#2018951 (10chasemp) 3NEW [16:49:22] having some irc issues here, but it may be linode fyi [17:06:05] valhallasw`cloud: Ok so i will try to complete this step : https://wikitech.wikimedia.org/wiki/Help:Tool_Labs#Requesting_a_Gerrit.2FGit_repository_for_your_tool thanks and see you soon! [17:10:37] 6Labs: track growth of project and home directories - https://phabricator.wikimedia.org/T126623#2019086 (10valhallasw) Historical data is available in graphite.wm.o as servers.labstore1001.diskspace._srv_project_tools.byte_used: {F3333968} The current growth rate is ~200GB/24h: {F3333970} However, we probabl... [17:21:01] 6Labs, 10Wikimedia-Site-Requests, 10wikitech.wikimedia.org: Allow wikitech to write files - https://phabricator.wikimedia.org/T126628#2019112 (10Dereckson) 3NEW [17:24:18] 6Labs, 10Wikimedia-Site-Requests, 10wikitech.wikimedia.org: Allow wikitech to write files - https://phabricator.wikimedia.org/T126628#2019126 (10Krenair) Maybe we just want to set wmgMathFileBackend = 'local-multiwrite' on wikitech (does that exist? no idea how to list them)? Note that Captcha on wikitech is... [17:30:03] 6Labs, 10Beta-Cluster-Infrastructure: New labs instance fails running `block-for-project-export` before running mount - https://phabricator.wikimedia.org/T126568#2019152 (10greg) p:5Triage>3Unbreak! [17:30:27] 6Labs, 10Beta-Cluster-Infrastructure: New labs instance fails running `block-for-project-export` before running mount - https://phabricator.wikimedia.org/T126568#2017518 (10greg) UBN! because this is blocking Beta Cluster updates. [17:31:38] 6Labs, 10Labs-Infrastructure, 10wikitech.wikimedia.org: Update Semantic Forms - https://phabricator.wikimedia.org/T63104#2019157 (10Krenair) Yeah, SemanticForms is deployed from master like normal extensions... [17:32:27] valhallasw`cloud: I'm getting errors using your tutorial at https://merlijn.vandeen.nl/2015/flask-mwoauth-on-tools.html [17:33:01] 6Labs, 10Labs-Infrastructure, 10wikitech.wikimedia.org: Update Semantic Forms - https://phabricator.wikimedia.org/T63104#2019163 (10Krenair) Oh right, I made that change. Umm... [17:34:00] 6Labs, 10wikitech.wikimedia.org: Old versions of Semantic* extensions break Wikitech (after removing long-living deprecated functions from core) - https://phabricator.wikimedia.org/T123599#2019176 (10Krenair) Also https://gerrit.wikimedia.org/r/#/c/270012/ [17:34:53] 6Labs, 10Labs-Infrastructure, 10Continuous-Integration-Infrastructure: Bump labs quota for 'integration' project - https://phabricator.wikimedia.org/T126557#2019177 (10Paladox) Where would we change this so it is raised. [17:35:11] bd808: I included your comments in my elasticsearch / logstash patch [17:35:34] bd808: unless you have a counter indication, I'm going to try deploying it on labs [17:35:48] go for it! [17:36:01] * gehel is crossing fingers [17:36:29] 6Labs, 10Labs-Infrastructure, 10Continuous-Integration-Infrastructure: Bump labs quota for 'integration' project - https://phabricator.wikimedia.org/T126557#2019188 (10greg) @Paladox: this task is for the WMF Labs admins to take care of. Quotas are handled in the OpenStackManager on wikitech (iow: not in Ger... [17:36:59] 6Labs, 10Labs-Infrastructure, 10Continuous-Integration-Infrastructure: Bump labs quota for 'integration' project - https://phabricator.wikimedia.org/T126557#2019190 (10Paladox) @greg Ok. [17:38:41] tom29739: namely? [17:39:27] 6Labs, 10Wikimedia-Site-Requests, 10wikitech.wikimedia.org: Allow wikitech to write files - https://phabricator.wikimedia.org/T126628#2019195 (10Dereckson) @Andrew Could you enable Math extension on https://labtestwikitech.wikimedia.org/ and check if that local-multiwrite file backend works? [17:40:46] 6Labs, 10Wikimedia-Site-Requests, 10wikitech.wikimedia.org: Allow wikitech to write files - https://phabricator.wikimedia.org/T126628#2019199 (10Dereckson) Nevermind, I see it's managed by our main operations/mediawiki-config repo, using //labtestwikitech// as database key. [17:43:39] 10MediaWiki-extensions-OpenStackManager, 10Continuous-Integration-Config, 5Patch-For-Review, 5WMF-deploy-2016-02-02_(1.27.0-wmf.12), 7WorkType-Maintenance: ApiDocumentationTest failure: Undefined property: AuthPlugin::$boundAs - https://phabricator.wikimedia.org/T124613#2019217 (10hashar) [17:43:42] 10MediaWiki-extensions-OpenStackManager, 10Continuous-Integration-Infrastructure, 5Patch-For-Review: CI slaves need package php5-ldap for OpenStackManager/LdapAuthentication - https://phabricator.wikimedia.org/T125158#2019215 (10hashar) 5Open>3Resolved Patch has been merged in operations/puppet [17:46:03] 6Labs, 10Labs-Infrastructure, 10Continuous-Integration-Infrastructure: Bump labs quota for 'integration' project - https://phabricator.wikimedia.org/T126557#2019225 (10Andrew) 5Open>3Resolved a:3Andrew done! [17:46:32] 6Labs, 10Labs-Infrastructure, 10Continuous-Integration-Infrastructure: Bump labs quota for 'integration' project - https://phabricator.wikimedia.org/T126557#2019229 (10hashar) Wonderful @Andrew \O/ Let us know if integration/contintcloud cause too much stress on labs infra. [17:49:59] 10MediaWiki-extensions-OpenStackManager: Failed to create instance: OpenStackManager don't tells me why - https://phabricator.wikimedia.org/T126633#2019240 (10Luke081515) 3NEW [17:50:46] valhallasw`cloud: A 'Traceback' EOFError: EOF when reading a line [17:51:06] valhallasw`cloud: Here's a paste of my log: http://pastebin.com/ADnGU12s [17:51:20] It's just the same, repeating over and over [17:51:51] 6Labs, 10Wikimedia-Site-Requests, 10wikitech.wikimedia.org: Allow wikitech to write files - https://phabricator.wikimedia.org/T126628#2019255 (10Krenair) Actually it's `labtestwiki`. [17:52:05] tom29739: yes, you did not fill in the consumer key and secret in app.py [17:52:12] oh [17:52:14] I did [17:52:16] you filled it in in the raw_input [17:52:28] it should read consumer_key = '' [17:53:00] raw_input asks you for input, but that fails because it's trying to load it as web app [17:53:49] Should I restart the webserver then, or will it pick it up? [17:54:56] I think you need to restart [17:57:17] valhallasw`cloud, I restarted the webserver, but it still gives the same error [17:58:50] tom29739: you still have input() around it [17:59:48] valhallasw`cloud, It's working now, thanks [18:01:51] 6Labs, 10Beta-Cluster-Infrastructure: New labs instance fails running `block-for-project-export` before running mount - https://phabricator.wikimedia.org/T126568#2019281 (10Paladox) [18:04:21] Damn, I've just reverted commit e07e4719 by mistake on deployment-puppetmaster [18:05:30] just re-applied it, sorry ... [18:19:23] 10Quarry, 10Analytics: Build an internal Quarry instance to share data & sample queries between researchers (and other analytics users?) - https://phabricator.wikimedia.org/T75142#2019467 (10Milimetric) @YuviPanda have you changed your mind about this in favor of jupyter goodness? [18:23:18] 10MediaWiki-extensions-OpenStackManager: Failed to create instance: OpenStackManager don't tells me why - https://phabricator.wikimedia.org/T126633#2019495 (10scfc) [18:23:20] 6Labs, 10Labs-Infrastructure: Make instance creation failures more verbose - https://phabricator.wikimedia.org/T47483#2019496 (10scfc) [18:24:09] 6Labs, 10Labs-Infrastructure, 10MediaWiki-extensions-OpenStackManager: unhelpful error message from Special:NovaInstance when you reach instance quota - https://phabricator.wikimedia.org/T65794#2019502 (10scfc) [18:24:11] 6Labs, 10Labs-Infrastructure: Make instance creation failures more verbose - https://phabricator.wikimedia.org/T47483#513101 (10scfc) [18:33:15] valhallasw`cloud: the doc say that: Log in to https://gerrit.wikimedia.org/ with your Labs account. But there i dont have any login options not any user option? i logged in media wiki mith my basic user account than i filled a request but it doesn work. i'm lost in the doc [18:34:04] Youni: there's a 'sign in' link on the top right? [18:34:22] and what do you mean with 'I filled a request but it doesn't work'? it's a manual process. [18:34:39] telling this: status:open [18:35:31] I have no clue what you mean with that [18:35:39] * valhallasw`cloud is off to dinner, will be back in an hour or so [18:36:08] i'm ashame my screen is too small i found the sign-in link sorry - i'm trying now [18:56:16] 6Labs: Create temporary test mailman mailing list to test synchronization with https://discourse.wmflabs.org/ - https://phabricator.wikimedia.org/T126547#2019703 (10Krenair) [18:56:28] 6Labs: Create temporary test mailman mailing list to test synchronization with https://discourse.wmflabs.org/ - https://phabricator.wikimedia.org/T126547#2016945 (10Krenair) I guess this should be treated as a labs project request? [18:57:46] Change on 12wikitech.wikimedia.org a page Nova Resource:Tools/Access Request/Volans was modified, changed by Tim Landscheidt link https://wikitech.wikimedia.org/w/index.php?diff=301118 edit summary: [19:17:35] PROBLEM - SSH on tools-exec-1216 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:27:28] RECOVERY - SSH on tools-exec-1216 is OK: SSH OK - OpenSSH_6.6.1p1 Ubuntu-2ubuntu2~wmfprecise2 (protocol 2.0) [19:43:23] 6Labs, 10Tool-Labs: scripttopic____ uses large amount of memory and swap - https://phabricator.wikimedia.org/T126647#2019872 (10valhallasw) 3NEW [19:43:41] 6Labs, 10Tool-Labs: scripttopic____ uses large amount of memory and swap - https://phabricator.wikimedia.org/T126647#2019881 (10valhallasw) (I killed the job with qdel 3319714) [19:51:16] 6Labs, 10Labs-Infrastructure, 10labs-sprint-117, 10labs-sprint-118, and 3 others: Move project membership/assignment from ldap to keystone mysql - https://phabricator.wikimedia.org/T115029#2019888 (10scfc) >>! In T115029#1712921, @Krenair wrote: >> How, then, will pam/ssh determine project membership? I do... [20:00:08] hey, trying to rebuild BC and we're seeing weird behavior for reverse dns resolution [20:00:21] dig +short deployment-tin.deployment-prep.eqiad.wmflabs [20:00:26] 10.68.17.240 [20:00:50] dig +short -x 10.68.17.240 [20:00:58] nslcd-1005.testlabs.eqiad.wmflabs. [20:00:58] deployment-tin.deployment-prep.eqiad.wmflabs. [20:00:59] kafka02.analytics.eqiad.wmflabs. [20:01:14] i.e. three answers for reverse lookup of the ip [20:01:37] marxarelli: iirc hashar also ran into this in the CI project; there should be a phab task (but I can't find it right now) [20:02:16] valhallasw`cloud: kk [20:02:21] any workaround? [20:02:37] aside from rebuilding the instance :) [20:03:28] marxarelli: https://phabricator.wikimedia.org/T126518 [20:03:40] valhallasw`cloud: ty! [20:03:52] I don't know if there's a workaround unfortunately [20:05:24] andrewbogott: ^ any idea? [20:06:09] it may not be a blocker btw. we just ran into it with ssh host verification [20:06:12] we'll see [20:07:21] Finally i created two request for guerrit repository - sorry - I can see them in https://www.mediawiki.org/wiki/Git/New_repositories/Requests. I guess i have to wait now. [20:08:07] 6Labs: Duplicate entries in labs internal dns - https://phabricator.wikimedia.org/T126518#2019936 (10dduvall) We saw this behavior today as well, while rebuilding the Beta Cluster deployment server. ```lang=irc [12:00:08] hey, trying to rebuild BC and we're seeing weird behavior for reverse dns re... [20:13:09] marxarelli: there is a task I made a few days ago same issue, we know it happens [20:13:11] unsure why atm [20:13:40] chasemp: kk [20:13:56] i think we're ok without a fix [20:14:10] marxarelli: can you add too? https://phabricator.wikimedia.org/T115194 [20:14:53] chasemp: oh, i already commented on https://phabricator.wikimedia.org/T126518 [20:15:07] i'll mark that one a dupe [20:15:33] too many isues :) [20:15:34] issues [20:15:41] 6Labs: Duplicate entries in labs internal dns - https://phabricator.wikimedia.org/T126518#2019951 (10dduvall) [20:15:44] 6Labs, 6operations: RDNS for some labs instance IPs resolve to multiple different instances - https://phabricator.wikimedia.org/T115194#2019952 (10dduvall) [20:17:23] 6Labs, 6operations: Some labs instances IP have multiple PTR entries in DNS - https://phabricator.wikimedia.org/T115194#2019965 (10hashar) [20:25:05] 6Labs, 6operations: Some labs instances IP have multiple PTR entries in DNS - https://phabricator.wikimedia.org/T115194#2019987 (10dduvall) [20:25:23] chasemp: ^ nm. it is blocking us :/ [20:26:02] partially due to how we're resolving the host name in scap [20:26:09] socket.getfqdn() [20:31:12] 6Labs, 10Labs-Infrastructure: LVM failures on Debian Jessie labs VMs - https://phabricator.wikimedia.org/T87310#2020006 (10scfc) 5Open>3Resolved a:3scfc >>! In T87310#1119198, @coren wrote: > That depends on how you define //issue//. There is still a problem with Jessie refusing to boot properly if any... [20:31:19] 6Labs, 10Labs-Infrastructure: LVM failures on Debian Jessie labs VMs - https://phabricator.wikimedia.org/T87310#2020009 (10scfc) a:5scfc>3None [20:37:53] 6Labs, 10Labs-Infrastructure, 10DBA, 6operations: db1069 is running low on space - https://phabricator.wikimedia.org/T124464#2020049 (10jcrespo) at 78% and going down. [20:55:57] I'm not sure we know why it's happening atm, but maybe we can do a one-time cleanup to unblock [20:56:11] andrewbogott: ^ can we correct this one record as a one-off in the short term? [20:59:34] chasemp: just landed a patch in scap that might have us out of the weeds https://phabricator.wikimedia.org/D126 [20:59:54] but, yeah, fixing the record would be fantastic [21:04:27] chasemp: yes, probably, I will look shortly [21:16:50] chasemp, andrewbogott: something I just thought of -- we are bumping the minimum PHP version for MediaWiki to 5.5.9. This means that MW master won't run on precise hosts. That may cause some SGE issues for Tool Labs users. [21:17:25] I haven't done it, but I've seen notes on wikitech about running MW on SGE from a shared clone of master [21:17:35] legoktm: ^ [21:17:54] If people are running mediawiki on the grid engine that’s news to me [21:18:48] I think it's done or been done but not sure of status [21:18:50] bd808: hmm, the only person I know who does that is liangent [21:19:15] 6Labs, 10Tool-Labs, 10pywikibot-core: pywikibot broken on wikimedia labs - https://phabricator.wikimedia.org/T126666#2020304 (10Jogo.obb) [21:19:17] isn't there a way to force the webservice to run on trusty? [21:19:26] there is [21:20:03] but I was going to ask next that if I sent out and email about that, are there enough trusty runners to deal with people switching? [21:23:24] I think we've mostly discouraged people from running MW inside tool labs [21:23:45] people are probably still running it on precise vms [21:29:55] Hello, I'm trying to get this to work: https://tools.wmflabs.org/oauth-hello-world/ on here: https://tools.wmflabs.org/ircredirector/ but it requires a .ini file somehwere other than the webserver root. How can I set the permissions so that PHP can read the file? Please help. [21:36:38] MW on SGE is possible but horribly slow [21:36:53] (I remember a task about adding an opcode cache) [21:37:12] Help [21:37:26] tom29739: php runs as the tool user, so it can read anything the tool user can read [21:38:01] so tools.? [21:38:11] yes [21:39:03] bd808: I think we should be OK -- the average load on the webgrid-14* nodes is maybe 50% (i.e. load 2 on a 4-core machine) [21:40:22] valhallasw`cloud, are you maintainer of shared pywikibot on labs? it's broken [21:40:36] amir built it, but I can try to run the crontabs by hand [21:47:36] valhallasw`cloud: can you load any wikipedia page in the wikiversum? [21:48:00] wikipedia is down!!! [21:48:38] https://en.wikipedia.org/wiki/Main_Page works for me [21:48:49] doctaxon: what exactly are you trying to do? [21:49:12] I think, all the access from Europe [21:49:16] ah some reports from EU in -ops [21:49:52] https://de.wikipedia/wiki/Main_Page doesn't load [21:50:52] ^ server not found on Portugal -europe [21:51:21] DNS_PROBE_FINISHED_NXDOMAIN [21:51:54] doctaxon, it's missing the .org [21:51:56] https://de.wikipedia.org/ [21:52:12] forgotten [21:52:19] oh, that explains :-p [21:52:25] en.wp works for me [21:52:34] https://en.wikipedia.org/wiki/Main%20Page doesn't work, too [21:52:49] valhallasw`cloud, when you mentioned amir you where talking to me about pywikibot on labs? [21:52:54] Alchimista: yes [21:53:06] No any Wikipedia works for me too. [21:53:25] seems to be a local European issue [21:53:30] yes [21:53:38] valhallasw`cloud, Ah. if you have the power, could you check what happened? [21:54:23] doctaxon, ato_ i'm in Europe (at least i think, who really knows where we are), and they're working for me [21:54:32] Alchimista: I'm not checking what happens, I'm just re-running the crontab [21:54:37] hoping that will fix it [21:54:40] Germany [21:54:46] Czech? [21:55:07] Germany [21:55:37] Alchimista: here in Augsburg is nothing [21:56:05] petan: re: cookie code: don't build cookies from api responses, just handle the set-cookie header correctly [21:56:49] doctaxon, you should move to portugal, more sunny wheather and beaches. Accessing wikipedia is a plus XD [21:57:00] okay, Southern Germany [21:57:08] joking, there's an DNS issue -> http://status.wikimedia.org/ [21:57:50] Alchimista: ugh, that crontab / git clone is super slow [21:58:02] thanks guys we know, it appears some EU locations have issues [21:58:45] Alchimista: but core seems to be back [22:00:32] it's working, thanks valhallasw`cloud . [22:03:37] 6Labs, 10Tool-Labs, 10pywikibot-core: pywikibot broken on wikimedia labs - https://phabricator.wikimedia.org/T126666#2020483 (10valhallasw) I manually ran the crontab, and this seems to have fixed it (at least for now). The underlying issues seems to be something like this: The code to update runs: ``` # Cu... [22:03:44] 6Labs, 10Tool-Labs, 10pywikibot-core: pywikibot broken on wikimedia labs - https://phabricator.wikimedia.org/T126666#2020485 (10Alchimista) p:5High>3Low valhallasw re-ran the update crontab and seems to be working now. Letting open so far, to a post-mortem description of the bug cause. [22:07:06] 6Labs, 10Beta-Cluster-Infrastructure: New labs instance fails running `block-for-project-export` before running mount - https://phabricator.wikimedia.org/T126568#2020557 (10greg) [22:07:22] 6Labs, 10Beta-Cluster-Infrastructure: New labs instance fails running `block-for-project-export` before running mount - https://phabricator.wikimedia.org/T126568#2017518 (10greg) [22:08:28] Hallelujah! Wikipedia is back :) [22:08:55] ato_, did it got down for you, recently up? [22:09:07] yes [22:10:17] It was a 50 minutes pause. [22:13:10] ato_, can I ask your provider? [22:17:12] Kabel Baden-Wuerttemberg GmbH, unitymedia.de [22:19:07] But only the Wikipedia [22:19:45] All others did working [22:54:36] Is Tool Labs veery slow for everyone right now or just me? [22:56:03] 6Labs: Duplicate entries in labs internal dns - https://phabricator.wikimedia.org/T126518#2020895 (10Andrew) 5duplicate>3Resolved I have cleaned up all the duplicates I can find. I'm not sure when the leaks were happening; as best I can tell we aren't leaking things anymore. [22:58:44] 6Labs, 10Labs-Infrastructure, 10labs-sprint-117, 10labs-sprint-118, and 3 others: Move project membership/assignment from ldap to keystone mysql - https://phabricator.wikimedia.org/T115029#2020897 (10Andrew) @scfc, that is almost correct. Certain ldap entries (the groups used by pam/ssh) will remain as be... [23:29:57] 6Labs, 10Wikimedia-Site-Requests, 10wikitech.wikimedia.org: Allow wikitech to write files - https://phabricator.wikimedia.org/T126628#2021081 (10Dereckson) >>! In T126628#2019255, @Krenair wrote: > Actually it's `labtestwiki`. I'm fixing a slight error in this case.