[08:34:52] PROBLEM - Host tools-bastion-01 is DOWN: CRITICAL - Host Unreachable (10.68.17.228) [10:22:46] RECOVERY - Puppet failure on tools-worker-1001 is OK: OK: Less than 1.00% above the threshold [0.0] [13:02:53] hello. i'm trying to install visualeditor extension using vagrant. After, vagrant roles enable visualeditor, I get this error. [13:02:59] Error: /Stage[main]/Citoid/Service::Node[citoid]/Npm::Install[/vagrant/srv/citoid]/Exec[/vagrant/srv/citoid_npm_install]/returns: change from notrun to 0 failed: Command exceeded timeout [13:02:59] Warning: /Stage[main]/Citoid/Service::Node[citoid]/Service[citoid]: Skipping because of failed dependencies [13:55:36] (03CR) 10Hashar: "recheck" [labs/tools/stewardbots] - 10https://gerrit.wikimedia.org/r/275163 (https://phabricator.wikimedia.org/T128503) (owner: 10Glaisher) [14:08:54] srijan: does your mediawiki vagrant box has access to internet ? It is probably trying to git clone the citoid repository or doing 'npm install' [14:09:11] srijan: you can try running puppet manually with debug mode which will give you some more informations: puppet agent -tv --debug [14:10:25] 6Labs, 10DBA: Move labs pdns database off of m5-master - https://phabricator.wikimedia.org/T128737#2094267 (10Andrew) I tidied up both servers a bit -- there should be plenty of space for db servers now. [14:12:35] 6Labs, 10DBA: Move labs pdns database off of m5-master - https://phabricator.wikimedia.org/T128737#2094270 (10jcrespo) how much memory can you dedicate to mysql? The on disk size is 130MB, so it should be lower than that. [14:18:13] 6Labs, 10DBA: Move labs pdns database off of m5-master - https://phabricator.wikimedia.org/T128737#2094275 (10Andrew) The memory metrics for labservices1001 were very noisy until last week (I suspect because of the ridiculously verbose logs that were getting written), but free memory never dropped below 1G. R... [14:18:35] 6Labs, 10DBA: Move labs pdns database off of m5-master - https://phabricator.wikimedia.org/T128737#2094276 (10Andrew) (Oh, and the picture is much rosier on Holmium since it isn't doing much.) [14:31:48] Error loading data from the server 503 [14:31:57] when i try to switch to visual editing [14:50:57] 6Labs, 6Operations: revise/fix labstore replicate backup jobs - https://phabricator.wikimedia.org/T127567#2094318 (10chasemp) I found some old stale snapshots on labstore2001 from failed backups in the past on friday. I things up and the replicate jobs seem to be running fine over the weekend. At the same ti... [15:14:51] 6Labs, 10Horizon: Horizon - Can't execute actions - https://phabricator.wikimedia.org/T127440#2094466 (10hashar) Dropping #labs-infrastructure since we now have #horizon [15:15:02] 6Labs, 10Horizon, 5Continuous-Integration-Scaling: Labs project admin can not delete per project image on Horizon - https://phabricator.wikimedia.org/T110936#2094482 (10hashar) Dropping #labs-infrastructure since we now have #horizon [15:28:46] 6Labs, 10Datasets-General-or-Unknown: Labs dumps mirror directories in /public/dumps/public/ no longer world-readable - https://phabricator.wikimedia.org/T128945#2090963 (10ArielGlenn) baaahhhh let me check into this. [15:29:30] 6Labs, 10Datasets-General-or-Unknown: Labs dumps mirror directories in /public/dumps/public/ no longer world-readable - https://phabricator.wikimedia.org/T128945#2094538 (10ArielGlenn) a:3ArielGlenn [15:50:47] 6Labs, 10Datasets-General-or-Unknown: Labs dumps mirror directories in /public/dumps/public/ no longer world-readable - https://phabricator.wikimedia.org/T128945#2094569 (10ArielGlenn) I've fixed these by hand; if there's a problem tomorrow morning after the sync I'll look at the rsync script (but it claims to... [16:12:16] (03PS1) 10ArthurPSmith: Updated nuclides charts to display using SVG; includes some refactoring into common components [labs/tools/ptable] - 10https://gerrit.wikimedia.org/r/275521 [16:15:17] (03CR) 10ArthurPSmith: [C: 031] "Take a look. One question I have about this is concerning the external javascript library I've used for the chart rescaling: 'svn-pan-zoom" [labs/tools/ptable] - 10https://gerrit.wikimedia.org/r/275521 (owner: 10ArthurPSmith) [16:16:23] (03CR) 10ArthurPSmith: "That's 'svg-pan-zoom' - https://github.com/ariutta/svg-pan-zoom - I think my autocorrect did something there!" [labs/tools/ptable] - 10https://gerrit.wikimedia.org/r/275521 (owner: 10ArthurPSmith) [16:40:54] 6Labs, 10Tool-Labs, 6Operations, 10Traffic, 7HTTPS: Make Magnus tools on tools.wmflabs.org work in HTTPS - https://phabricator.wikimedia.org/T102457#2094745 (10Magnus) Just FYI, treeviews is now completely https (the last remaining http element was the tree image...) https://tools.wmflabs.org/glamtools/t... [16:42:35] 6Labs, 10Tool-Labs, 6Operations, 10Traffic, and 2 others: Detect tools.wmflabs.org tools which are HTTP-only - https://phabricator.wikimedia.org/T128409#2073537 (10Magnus) @Dzahn: Does "using http" mean "available over http", or "do not work on https"? [16:54:28] 6Labs, 10Labs-Infrastructure, 10Beta-Cluster-Infrastructure, 6Operations: beta: Get SSL certificates for *.{projects}.beta.wmflabs.org - https://phabricator.wikimedia.org/T50501#2094832 (10Paladox) [16:57:18] 6Labs, 10Labs-Infrastructure, 10Beta-Cluster-Infrastructure: Setup real ssl certs for Beta Cluster using a restricted project - https://phabricator.wikimedia.org/T75919#785524 (10Paladox) letsencrypt.org is now available but in beta mode. [17:15:07] (03CR) 10Ricordisamoa: [C: 04-1] Updated nuclides charts to display using SVG; includes some refactoring into common components (038 comments) [labs/tools/ptable] - 10https://gerrit.wikimedia.org/r/275521 (owner: 10ArthurPSmith) [17:15:35] (03CR) 10Ricordisamoa: "recheck" [labs/tools/ptable] - 10https://gerrit.wikimedia.org/r/275521 (owner: 10ArthurPSmith) [17:29:33] 6Labs, 10Tool-Labs, 6Operations, 10Traffic, and 2 others: Detect tools.wmflabs.org tools which are HTTP-only - https://phabricator.wikimedia.org/T128409#2095064 (10scfc) @Magnus: In this context, it means that a request came in with `http` and was not redirected or failed. This could be by someone typing... [17:59:43] yuvipanda: wdqb-test and the rest are ready for the changes :) [17:59:59] w00t! [18:00:09] SMalyshev: I'm in meetings for the next bits I'll do it right after [18:00:21] yuvipanda: great [18:00:49] SMalyshev: do you know if I can potentially restart the instances (if necessary)? I don't expect it to but just checking [18:01:26] yuvipanda: yes, you can restart them just tell me afterwards if you do so I check if all services came back up ok (it's a test install so it's messy :) [18:01:36] SMalyshev: thanks [18:08:54] harej: I think so (see quarry), but some communities do not appreciate detailed statistics (e.g. dewiki and edit counters) [18:09:27] 6Labs, 10Labs-Infrastructure, 6Operations, 13Patch-For-Review: labservices1001 ran out of disk space - https://phabricator.wikimedia.org/T126572#2095280 (10Andrew) 5Open>3Resolved [18:10:38] 6Labs, 10Tool-Labs: tools-login.wmflabs.org points to tools-bastion-02 - https://phabricator.wikimedia.org/T127686#2095288 (10Andrew) 5Open>3Resolved [18:14:15] andrewbogott: ping [18:14:46] Krinkle: in a meeting, should be a short one [18:15:34] andrewbogott: k. Was gonna ask you to hotfix silver one more time, this time add PT to the static.php rewrite rules in https://github.com/wikimedia/operations-puppet/blob/production/modules/mediawiki/files/apache/sites/public-wiki-rewrites.incl. I think the descrepancy is caused by hhvm vs zend [18:15:52] which populates $_SERVER differently [18:16:00] 6Labs, 13Patch-For-Review: Periodic internal labs dns outages - https://phabricator.wikimedia.org/T124680#2095306 (10Andrew) [18:16:02] 6Labs: pdns trying to resolve wikimedia.org.eqiad.wmflabs - https://phabricator.wikimedia.org/T128123#2095304 (10Andrew) 5Open>3stalled I'm not sure this is a thing we care about/can do anything about [18:16:03] e.g. [PT] instead of [L] [18:16:33] For https://phabricator.wikimedia.org/T128747 [18:16:39] 6Labs: pdns trying to resolve wikimedia.org.eqiad.wmflabs - https://phabricator.wikimedia.org/T128123#2095307 (10Andrew) a:5Andrew>3None [18:19:00] https://gerrit.wikimedia.org/r/#/c/275582/ [18:19:07] 6Labs, 7Tracking: Make OpenStack Horizon useful for production labs - https://phabricator.wikimedia.org/T87279#2095314 (10Andrew) a:3Andrew [18:26:14] (03CR) 10ArthurPSmith: "Thanks for the css/javascript coding advice, looking into it now!" [labs/tools/ptable] - 10https://gerrit.wikimedia.org/r/275521 (owner: 10ArthurPSmith) [18:26:30] 6Labs, 10DBA: Move labs pdns database off of m5-master - https://phabricator.wikimedia.org/T128737#2095340 (10Andrew) a:5Andrew>3jcrespo [18:27:16] 6Labs, 6Operations: Untangle labs/production roles from labs/instance roles - https://phabricator.wikimedia.org/T119401#2095346 (10yuvipanda) a:5yuvipanda>3None [18:27:47] 6Labs, 10DBA: Move labs pdns database off of m5-master - https://phabricator.wikimedia.org/T128737#2095348 (10jcrespo) So, 100MB ok for the buffer pool? [20:38:20] 6Labs, 6Operations, 10wikitech.wikimedia.org, 13Patch-For-Review: Update wikitech-static OS/PHP version - https://phabricator.wikimedia.org/T126385#2096207 (10Dzahn) ``` root@wikitech-static-jessie.wikimedia.org's password: ___________________________ < wikitech-static on jessie > < T126385 > -----... [21:01:45] 6Labs, 6Operations, 10wikitech.wikimedia.org, 13Patch-For-Review: Update wikitech-static OS/PHP version - https://phabricator.wikimedia.org/T126385#2096372 (10Dzahn) @Krenair i added your ssh key to that ^ got root , like on the current wikitech-static [22:05:40] 6Labs, 6Operations, 10wikitech.wikimedia.org, 13Patch-For-Review: Update wikitech-static OS/PHP version - https://phabricator.wikimedia.org/T126385#2096990 (10Dzahn) [22:11:28] andrewbogott: regarding your drive space issues - did you test virt-sparsify ? [22:14:45] matanya: nope, it’s worth a look though [22:17:33] Krinkle: ok, I’m briefly undistracted… can you tell me literally what line you want me to add? [22:22:55] andrewbogott: etc/apache/sites-enabled/pub rewrite.incl [22:23:09] the bottom 3 lines that rewrite to static.php have [L] and the idea is to change them to [PT] [22:23:36] andrewbogott: https://gerrit.wikimedia.org/r/#/c/275582/ [22:25:57] Krinkle: ok, I’ll change that on silver, just a moment... [22:26:33] Krinkle: done — any better? [22:27:17] andrewbogott: graceful restart? [22:27:21] yep, did [22:27:27] Yep! [22:27:29] Fixed it [22:27:30] yay [22:27:34] cool [22:27:54] want to remove your -1 from https://gerrit.wikimedia.org/r/#/c/275582 ? [22:28:27] andrewbogott: Yeah, I'll just double check on an app server before I do [22:28:38] since the existing config worked fine on an app server, but not on silver [22:34:22] 6Labs, 6Operations, 10wikitech.wikimedia.org, 13Patch-For-Review: Wikitechwiki has 4xx responses to requests for some static assets inc. poweredby_mediawiki_88x31.png and WikiEditor's button-sprite.svg - https://phabricator.wikimedia.org/T128747#2097133 (10Krinkle) Yeah, so that was the issue. On silver... [22:35:08] 10Tool-Labs-tools-Other, 6Community-Tech, 7Community-Wishlist-Survey, 7Milestone: Pageview Stats tool - https://phabricator.wikimedia.org/T120497#2097134 (10DannyH) [22:55:42] hello, I want to have one tool forward to a path within another tool, but with "masking". In short, https://tools.wmflabs.org/topviews to keep it's URL but show https://tools.wmflabs.org/pageviews/topviews [22:55:56] anyone know how you could accomplish this with the .lighttpd.conf or other means? [22:56:37] I tried a simple `alias.url += ( "/topviews" => "/pageviews/topviews" )` with no luck [23:01:39] alternatively, I would be quite content overriding the "document.root" (and just have a separate repo for /topviews), but it doesn't appear that you can do this with lighttpd. The first rule set by tool-labs has precedence and can't be overridden [23:06:22] not sure if anyone is reading this, but anyway check https://tools.wmflabs.org/topviews-test and not /topviews. The latter appears to do what I want but it actually just has the code copied/pasted in a parent directory [23:15:49] 6Labs: Add DNS entry for promethium.wikitextexp.eqiad.wmflabs - https://phabricator.wikimedia.org/T129181#2097345 (10ssastry) [23:17:27] MusikAnimal: IMO you should just have them be different tools [23:17:42] for maintenance rasons and allowing other people to be added to it [23:18:05] yuvipanda: well, the thing is that they share a lot of the same code [23:18:11] MusikAnimal: git :) [23:18:13] I could use git submodules, I get [23:18:17] but those are messy [23:18:24] they can also just be the same git repository [23:18:30] not even git submodules [23:18:33] that's what is now [23:18:33] just a config variable, etc [23:18:39] just clone them in multiple places :) [23:18:54] I tried that, with some succuss [23:19:13] this is also ruby, right? [23:19:21] nope! for once it is not [23:19:21] we have terrible support for ruby and should get better :| [23:19:23] aaah [23:19:25] :D [23:19:26] PHP [23:19:27] ok [23:19:44] basically my set up is I have a ~/pageviews which is where the repo lives, then a deploy script that copies over what we need to /public_html [23:20:23] cool [23:20:27] so I could just have it copy the /topviews within the git repo to ~/public_html, but then when the PHP tries to render the partials at ../_head.php it looks in the home directory [23:20:33] and I don't really want code there [23:20:51] if I could just override document.root then all would be well [23:21:18] is that not possible? [23:21:33] I want document.root to be ~/public_html/topviews [23:22:35] Google was not very friendly in getting such answers [23:22:47] 6Labs: Add DNS entry for promethium.wikitextexp.eqiad.wmflabs - https://phabricator.wikimedia.org/T129181#2097345 (10scfc) Does https://wikitech.wikimedia.org/wiki/Help:Proxy#Creating_a_web_proxy not work for you? [23:22:56] fun [23:24:01] yuvipanda: where does the default .lighttpd.conf (or whatever filename) live? is it there one for each tool, such that I could modify it? [23:24:14] and set document.root to what I need [23:24:48] with Apache you can also do Alias and it works like masking, it appears that is not the case with alias in lighttpd [23:25:04] you can setup a .lighttpd.conf to override. you can find the default in the puppet repo I think [23:25:07] under 'tool-lighttpd' [23:25:18] you can't really modify that tho [23:25:27] I highly suspect that the git solution will end up being easier in the end [23:25:49] well git isn't the problem, it's the routing [23:26:12] with the same repo and the structure, I'd have to browse to /topviews/topviews, as /topviews would be the same as /pageviews [23:26:45] for PHP I usually end up symlinking to get around these [23:27:07] I'm happy with that! you don't think that'd be bad for performance or anything? [23:27:16] I just figured I should avoid symlinks [23:29:22] or maybe I could even use the lighttpd alias to make it fetch the right .php partials [23:29:31] I'll give that a shot [23:32:00] MusikAnimal: nope [23:32:16] MusikAnimal: symlinks are fine wrt. php [23:34:05] 6Labs: Add DNS entry for promethium.wikitextexp.eqiad.wmflabs - https://phabricator.wikimedia.org/T129181#2097408 (10ssastry) >>! In T129181#2097377, @scfc wrote: > Does https://wikitech.wikimedia.org/wiki/Help:Proxy#Creating_a_web_proxy not work for you? Nope. This is one-off raw hardware. Not a labs VM. So, i... [23:43:27] 6Labs, 10Labs-Infrastructure, 10Tool-Labs-tools-meetbot: restore Meetbot logs from around 2015-06 lost in NFS outage - https://phabricator.wikimedia.org/T113000#2097478 (10yuvipanda) 5Open>3declined They're all long gone, unfortunately. [23:48:03] chasemp: forgot to tell you but https://phabricator.wikimedia.org/T129002 - another instance became unreachable [23:49:01] 6Labs, 6Discovery, 10Wikidata, 10Wikidata-Query-Service: Move wikidata-query project from /home NFS to scratch - https://phabricator.wikimedia.org/T128815#2097522 (10yuvipanda) a:3yuvipanda [23:57:17] 6Labs: tools-worker-1002 is un-sshable - https://phabricator.wikimedia.org/T129002#2092215 (10chasemp) Was this precise? [23:57:37] 6Labs: tools-worker-1002 is un-sshable - https://phabricator.wikimedia.org/T129002#2097543 (10yuvipanda) Nope, Jessie.