[07:08:45] <madhuvishy>	 Hi all, feel free to look into this whenever but leaving this here - when I make an instance self hosted after creating a new one in the ifttt project I get this error for the puppet run
[07:08:50] <madhuvishy>	 https://www.irccloud.com/pastebin/69rD0KCh/
[07:09:07] <madhuvishy>	 and if I try to apply a role after, it shows
[07:09:23] <madhuvishy>	 https://www.irccloud.com/pastebin/nHGFNGer/
[07:43:48] <harej>	 madhuvishy: the page views service is great by the way. i look forward to telling the us census bureau that wikipedia articles citing their data were viewed 104 million times -- in february
[07:44:03] <madhuvishy>	 harej: thank you!
[07:46:37] <madhuvishy>	 woah, that is cool!
[09:02:40] <valhallasw`cloud>	 madhuvishy: maybe start puppetmaster manually?
[10:23:38] <wikibugs>	 6Labs, 10Labs-Infrastructure, 10DBA, 13Patch-For-Review: Get HA db support for labs internal services - https://phabricator.wikimedia.org/T126251#2117491 (10Nemo_bis) p:5High>3Normal
[10:24:11] <wikibugs>	 6Labs, 10DBA: Measure capacity and utilization of labsdb*** boxes - https://phabricator.wikimedia.org/T107070#2117492 (10Nemo_bis) p:5High>3Normal
[10:46:04] <wikibugs>	 6Labs: New project: Petscan - https://phabricator.wikimedia.org/T129749#2117509 (10WMDE-Fisch) Hi,  >  Is there a catgraph instance running? The links I clicked on were all dead. Is it is running, is is covering all wikis (including Commons etc)? Is is up-to-date, as in lag in seconds?  The link on https://wikit...
[10:50:59] <wikibugs>	 6Labs: New project: Petscan - https://phabricator.wikimedia.org/T129749#2117516 (10Magnus) These all say "ns14". Does that mean they all store the tree of categories, but not the pages/files/etc. in them? Unless it does store page links, it is rather useless to me; I can get category trees reasonably cheap from...
[10:58:14] <wikibugs>	 6Labs: New project: Petscan - https://phabricator.wikimedia.org/T129749#2117536 (10WMDE-Fisch) Yes this is only storing the tree of categories.
[13:15:41] <grrrit-wm>	 (03CR) 10ArthurPSmith: [C: 031] "Fine with me - a .gitignore file is a good idea, thanks Tobias. We only just recently switched to python3 so __pycache__ is a little new. " [labs/tools/ptable] - 10https://gerrit.wikimedia.org/r/277070 (owner: 10Tobias47n9e)
[13:39:48] <grrrit-wm>	 (03CR) 10Ricordisamoa: [C: 032] Add .gitignore and ignore pycache directory [labs/tools/ptable] - 10https://gerrit.wikimedia.org/r/277070 (owner: 10Tobias47n9e)
[13:40:28] <grrrit-wm>	 (03Merged) 10jenkins-bot: Add .gitignore and ignore pycache directory [labs/tools/ptable] - 10https://gerrit.wikimedia.org/r/277070 (owner: 10Tobias47n9e)
[14:05:01] <wikibugs>	 6Labs: New project: Petscan - https://phabricator.wikimedia.org/T129749#2114793 (10Andrew) I recently deleted the hostname 'sylvester.wmflabs.org' because it was not bound to an actual instance.  Recreating it and binding it to sylvester.catgraph.wmflabs.org is certainly easy, if useful.  It looks to me like the...
[14:41:52] <chasemp>	 madhuvishy: it's just a guess but try running /usr/loca/bin/puppet-run or do an aptitude update before hand, our normal op does this to avoid stale package data that can manifest in this way 
[14:56:44] <grrrit-wm>	 (03CR) 10ArthurPSmith: "I need to think about this for a bit. There's a bunch of things broken for mobile devices, I'm not sure this is particularly helpful. But " [labs/tools/ptable] - 10https://gerrit.wikimedia.org/r/277072 (owner: 10Tobias47n9e)
[15:33:27] <wikibugs>	 6Labs: New project: Petscan - https://phabricator.wikimedia.org/T129749#2118498 (10Magnus) I think the two do have some overlap, but I would really like to have the petscan project.  I assume I can create a VM with an externally visible host name myself then?
[15:38:59] <wikibugs>	 6Labs: New project: Petscan - https://phabricator.wikimedia.org/T129749#2118557 (10Andrew) If all you need is http(s) then getting a public hostname is super easy, you just use the 'manage web proxies' panel.  If you need a full-fledged public IP (e.g. for non-web services) then we can do that too, but I have to...
[15:40:02] <wikibugs>	 6Labs, 7Tracking: New Labs project requests (tracking) - https://phabricator.wikimedia.org/T76375#2118561 (10Andrew)
[15:40:04] <wikibugs>	 6Labs: New project: Petscan - https://phabricator.wikimedia.org/T129749#2118558 (10Andrew) 5Open>3Resolved a:3Andrew I created the Petscan project with Magnus as the only admin.  Magnus, you can add other people and admins as you see fit; generally I like there to be at least two people responsible for a g...
[15:59:04] <wikibugs>	 6Labs, 10Labs-Infrastructure, 10DBA: Lost database changes on s2 for 3 hours on labs replicas - https://phabricator.wikimedia.org/T129432#2118635 (10jcrespo) The plan here is going to be:  * Reimport, one by one, the appropiate ranges from page, revision, user and recentchanges  This will fix the most import...
[16:27:37] <wikibugs>	 6Labs: Labs proxy not working - https://phabricator.wikimedia.org/T129884#2118817 (10Magnus)
[17:11:06] <wikibugs>	 6Labs: Labs proxy not working - https://phabricator.wikimedia.org/T129884#2118817 (10scfc) Your [[https://wikitech.wikimedia.org/wiki/Nova_Resource:Petscan1.petscan.eqiad.wmflabs|instance]] has the security group "default" which (by default) does not allow incoming requests.  So you either need to set up a secur...
[17:22:42] <grrrit-wm>	 (03CR) 10ArthurPSmith: "Tobias, this looks very nice, thanks for the layout adjustments, it's definitely an improvement. There are still issues for mobile/touch s" (031 comment) [labs/tools/ptable] - 10https://gerrit.wikimedia.org/r/277072 (owner: 10Tobias47n9e)
[17:30:48] <wikibugs>	 6Labs, 10Phlogiston, 15User-bd808: phlogiston-2 hangs every week - https://phabricator.wikimedia.org/T129891#2119161 (10JAufrecht)
[17:39:27] <shinken-wm>	 PROBLEM - ToolLabs Home Page on toollabs is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[17:44:23] <shinken-wm>	 RECOVERY - ToolLabs Home Page on toollabs is OK: HTTP OK: HTTP/1.1 200 OK - 802725 bytes in 5.885 second response time
[17:45:02] <madhuvishy>	 joal: around? 
[17:45:31] <joal>	 I am :)
[17:49:28] <matanya>	 bd808: ping ?
[17:50:11] <matanya>	 any wikitech admin around ? 
[17:50:37] <Reedy>	 matanya: ?
[17:50:52] <matanya>	 our ip was blocked due to too many account creations
[17:51:09] <matanya>	 Reedy: can you please create the account NirZabari
[17:51:41] <Reedy>	 matanya: Email address and shell username?
[17:51:54] <Reedy>	 PM if you want
[17:52:07] <bd808>	 Reedy: or just make matanya an admin there :)
[17:52:10] <matanya>	 NirZabari@gmail.com
[17:52:26] <matanya>	 and shell: NirZabari
[17:52:58] <matanya>	 sorry
[17:53:03] <matanya>	 nirzabari
[17:53:20] <Reedy>	 yeah, I already lowercased ;)
[17:53:46] <Reedy>	 It's taking a while...
[17:56:06] <Reedy>	 wtf is it doing
[18:02:17] <Reedy>	  17:53, 14 March 2016 User account NirZabari (talk | contribs | block) was created by Reedy (talk | contribs | block) and password was sent by email (Per matanya on #wikimedia-labs)
[18:02:22] <Reedy>	 Yet, the page hasn't returned
[18:04:13] <matanya>	 thanks Reedy !
[18:04:17] <Reedy>	 Timed out
[18:04:19] <Reedy>	 gj MediaWiki
[18:07:31] <matanya>	 Reedy: ok, let me know if it worked
[18:07:41] <matanya>	 i need several more please  :)
[18:10:21] <matanya>	 Reedy: mind if i give myself admin rights and handle ?
[18:17:33] <matanya>	 bd808: can you give me rights ?
[18:17:40] <bd808>	 matanya: yup.
[18:17:46] <matanya>	 i can remove them after this pre-hackthon meeting
[18:17:49] <matanya>	 thanks
[18:18:51] <bd808>	 matanya: gave you admin + crat
[18:19:05] <matanya>	 thanks!
[18:19:34] * bd808 is sure that matanya isn't a hat collector ;)
[18:21:22] <matanya>	 :)
[18:27:30] <grrrit-wm>	 (03PS1) 10Andrew Bogott: Add a bunch of dummy username/passwords for labs [labs/private] - 10https://gerrit.wikimedia.org/r/277305 
[18:28:04] <grrrit-wm>	 (03CR) 10Andrew Bogott: [C: 032 V: 032] Add a bunch of dummy username/passwords for labs [labs/private] - 10https://gerrit.wikimedia.org/r/277305 (owner: 10Andrew Bogott)
[18:31:39] <wikibugs>	 6Labs: Labs proxy not working - https://phabricator.wikimedia.org/T129884#2119551 (10Magnus) Thanks, works now!
[18:31:48] <wikibugs>	 6Labs: Labs proxy not working - https://phabricator.wikimedia.org/T129884#2119552 (10Magnus) 5Open>3Resolved a:3Magnus
[18:35:11] <wikibugs>	 6Labs, 10Phlogiston: phlogiston-2 hangs every week - https://phabricator.wikimedia.org/T129891#2119563 (10bd808) a:5bd808>3None This looks to be a bursty i/o problem. The kernel is too busy doing something (probably flushing buffered disk writes to disk) to do anything else.  Do you have some cron job(s) t...
[18:36:51] <wikibugs>	 6Labs, 10Phlogiston: phlogiston-2 hangs every week - https://phabricator.wikimedia.org/T129891#2119581 (10bd808) Some things to read: * http://www.blackmoreops.com/2014/09/22/linux-kernel-panic-issue-fix-hung_task_timeout_secs-blocked-120-seconds-problem/ * https://lonesysadmin.net/2013/12/22/better-linux-disk...
[18:57:15] <wikibugs>	 6Labs, 10Tool-Labs: Puppet errors on tools-web-static-01 and tools-web-static-02 - https://phabricator.wikimedia.org/T128411#2119635 (10scfc) a:3scfc
[19:17:27] <wikibugs>	 6Labs, 10Tool-Labs, 6Developer-Relations, 6WMF-Legal: Set up process / criteria for taking over abandoned tools - https://phabricator.wikimedia.org/T87730#997462 (10scfc) This task has been idling for about a year now, with the RfC going nowhere, and I don't remember any other requests apart from T127494....
[19:37:15] <shinken-wm>	 PROBLEM - Puppet run on tools-elastic-03 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0]
[19:55:42] <shinken-wm>	 PROBLEM - Puppet run on tools-elastic-02 is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0]
[21:08:24] <grrrit-wm>	 (03CR) 10Tobias47n9e: "Hey Arthur!" [labs/tools/ptable] - 10https://gerrit.wikimedia.org/r/277072 (owner: 10Tobias47n9e)
[21:09:33] <tom29739>	 Would I be allowed to use Google reCAPTCHA on tool labs?
[21:11:42] <chasemp>	 tom29739: good question, it may be considered user tracking...idk, bd808?^
[21:12:39] <tom29739>	 Like just to somewhat drive away bots, etc. Wikipedia uses it on the create account form, and I think it has a BSD license.
[21:13:48] <bd808>	 Noting on tool labs should cause the visiting browser to interact with a server outside of Wikimedia Foundation control
[21:13:52] <bd808>	 *Nothing
[21:14:06] <tom29739>	 So no then.
[21:15:01] <chasemp>	 That would be my thought as well bd808, captcha seems good but calling out the googleverse not so much
[21:15:02] <bd808>	 Some sort of captcha tests is fine but only if it doesn't involve external servers
[21:15:09] <chasemp>	 :)
[21:15:22] <bd808>	 jikx!
[21:15:28] <bd808>	 *jinx
[21:17:03] <bd808>	 chasemp: on a related note... I can't find that prohibition on https://wikitech.wikimedia.org/wiki/Wikitech:Labs_Terms_of_use
[21:17:31] <chasemp>	 seems like it should be in https://wikitech.wikimedia.org/wiki/Wikitech:Labs_Terms_of_use#What_uses_of_Labs_do_we_not_like.3F
[21:17:36] <bd808>	 and I do know of Labs projects that violate it :/
[21:17:52] <chasemp>	 or clarified under Misuse of Private Information
[21:17:54] <chasemp>	 yeah
[21:23:23] <tom29739>	 It isn't exactly spelled out disallowing it, so it's probably best to add it into the Terms of Use. I have seen some projects that use reCAPTCHA though
[21:24:40] <bd808>	 tom29739: yeah. similarly there are things which are using external CDNs for jquery, bootstrap, etc.
[21:25:06] <bd808>	 I'll make a task (or find an existing one) about getting the terms of use updated with "conventional wisdom"
[21:25:43] <tom29739>	 bd808, am I allowed to use jQuery then?
[21:27:01] <bd808>	 yes, but you should either host it inside your tool or get it from https://tools.wmflabs.org/static-browser/ 
[21:27:26] <bd808>	 e.g. <script src="//tools-static.wmflabs.org/static/jquery/2.1.0/jquery.min.js">
[21:27:52] <tom29739>	 Wow, I didn't even know we had that. Good to know
[21:28:17] <bd808>	 this is one of the many things I'd like to see better documentation on
[21:28:25] * bd808 takes baby steps
[21:30:24] <tom29739>	 The trouble with Tool Labs is that the docs are all over the place, and some things don't even have docs at all, you just have to hear about them from other users
[21:30:32] <bd808>	 amen
[21:30:41] <tom29739>	 :D
[21:30:53] <chasemp>	 yes truly we all want to improve it
[21:31:06] <chasemp>	 there is a lot of awesome and we make it hard to find
[21:31:17] <bd808>	 organizing things is hard and documentation doesn't come naturally for a majority of developers
[21:32:28] <chasemp>	 see: everything ever
[21:32:59] <bd808>	 and docs for some things are good, but not on wikitech -- https://meta.wikimedia.org/wiki/Research:Quarry
[21:33:11] <tom29739>	 The 'Portal:Tool Labs' page works(ish), but it doesn't link to some of the pages directly, so you have to go to say 'Help:Tool Labs' and then another page and then another, and so on and so forth. Some of the docs are even in people userspaces
[21:33:41] <halfak>	 bd808, IMO, those docs don't belong on Wikitech. 
[21:33:41] <tom29739>	 It would make sense to have it all under say 'Help:Tool Labs' for instance
[21:33:57] <tom29739>	 For Tool Labs stuff
[21:34:21] <halfak>	 FWIW, I was referring to the quarry usage docs. 
[21:34:29] <tom29739>	 The new Main Page is a great start though
[21:34:32] <halfak>	 The quarry maintenance docs and tool labs help seem to belong on wikitech. 
[21:34:40] <bd808>	 halfak: I totally understand why quarry has docs on meta, but it should be discoverable on wikitech too
[21:35:09] <bd808>	 I did a bit of work to make wikitech at least have a pointer to the meta docs
[21:36:03] <bd808>	 tom29739: do you mean that the giant wall of text page should be even longer?
[21:47:50] <tom29739>	 bd808, no, it's great as it is. I'm a newish user on Tool Labs, and the previous page you had was terrible. You couldn't find anything on there, there were that many links. It did have the advantage of that if you wanted to find something, it was probably easier to just go read the Main Page.
[21:49:18] <bd808>	 tom29739: I'd love to hear more from you (and any/all other Tool Labs users) about specific things that were hard to find docs on. Even more I'd love to see people make docs themselves as they discover things.
[21:49:34] <bd808>	 I'll be writing some emails to labs-l "soon" abut such things
[21:49:59] <tom29739>	 It must be harder for seasoned users because they already have much more experience
[21:50:36] <bd808>	 my big campaign on this is blocked on getting the analysis from the last Tool Labs user survey published
[22:01:31] <wikibugs>	 6Labs: Ensure that Terms of Use document restrictions on third-party web interactions - https://phabricator.wikimedia.org/T129936#2120419 (10bd808)
[22:09:12] <wikibugs>	 6Labs: Ensure that Terms of Use document restrictions on third-party web interactions - https://phabricator.wikimedia.org/T129936#2120419 (10tom29739) Give examples in the Terms of Use, eg reCAPTCHA, CDN, etc. Suggest that tools-static be used instead.
[23:43:00] <wikibugs>	 6Labs: Ensure that Terms of Use document restrictions on third-party web interactions - https://phabricator.wikimedia.org/T129936#2120419 (10scfc) I'm not sure if the premise is actually true.  We had a user redirecting http://tools.wmflabs.org/$tool to their own private site (which is a very, eh, complete third...