[07:15:09] @notify hashar [07:15:09] I'll let you know when I see hashar around here [09:18:31] !log glam created new instance for the project that will be used as the test server : i-0000081a https://wikitech.wikimedia.org/wiki/Nova_Resource:I-0000081a [09:18:33] Logged the message, Master [09:35:42] Hi I have created one database for my tool instance then how can I connect using php ? [09:38:05] harshkothari: the only doc I know of is https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/Help#Database_access [09:38:25] harshkothari: apparently your db username/pass is in the file replica.my.cnf in your homedir [09:39:04] hashar: but in database name I have to place USERNAME__DBNAME ? [09:39:56] """Users can create databases with names starting with username__ (where username is your credentials' user, which can be found in your ~/replica.my.cnf""" [09:40:01] harshkothari: apparently ) [09:40:42] hashar: so have to change lots of thing :( [09:40:53] hashar: thanks for the help [10:28:39] hashar: hey [10:28:41] around? [10:28:42] hashar: when I updated it manually last time It stayed there, but not this time. any idea why so? [10:28:56] YuviPanda: no idea [10:29:07] YuviPanda: maybe the extension got unbroken in gerrit ? [10:29:21] and how did you manually update it? git pull origin master / git reset --hard origin/master (or) origin/head? [10:29:30] I did push a master update in mediawiki/extensions.git [10:29:38] will have to catch up with Chad to get ti fixed properly though [10:29:47] I got a copy of mw/ext [10:29:51] hashar: hmm, but it was still broken, since it was reverting to july [10:29:51] aah [10:29:51] right [10:29:51] hmm, I don't have a checkout [10:29:53] of mw/ext [10:29:57] went to VisualEditor manually updated [10:30:01] right right [10:30:02] then did a commit of that [10:30:08] I'll do that next time :) [10:30:14] but yeah that is very confusing :( [10:30:15] if there is a next time [10:30:27] I don't know how it worked last time, since I just did a pull [10:30:30] there [10:31:02] hashar: either way, thanks for taking care of that :) [10:31:27] :) [10:31:38] why is tool lab appearing slow now? [10:31:56] I think the NFS server is a bit overloaded [10:32:10] I can't find it in ganglia though :-( [10:32:11] yeah, betalabs is also slooow [10:32:20] ganglia for labs is very broken, IIRC [10:32:30] YuviPanda: we got Ganglia slightly fixed last week :-] [10:32:43] whee, so from very broken to just broken! :D [10:32:43] at least beta has some metrics http://ganglia.wmflabs.org/latest/?r=hour&cs=&ce=&s=by+name&c=deployment-prep&tab=m&vn= [10:33:17] I have zero idea what labsnfs.pmtpa.wmnet is really [10:33:31] hashar: 'Could not connect: Can't connect to local MySQL server through socket '/var/run/mysqld/mysqld.sock' (2)' got this error [10:33:39] if only we had documentation.... [10:33:45] harshkot_: set host to tools-db [10:33:48] not localhost [10:33:54] ok YuviPanda thanks [10:34:08] isn't there a basic PHP script for people to reuse ? [10:34:26] some basic glue that people could include to have access to their db easily [10:34:36] hashar: well, it is fairly trivial... [10:34:43] set host=tools-db, and username and password [10:34:51] would be nice to have though, yeah [10:35:06] YuviPanda: thanks its working :) [10:35:10] :) [10:35:13] congratulations harshkot_ ! [10:35:34] hashar: :) FInally db is connected :) [10:35:59] so the NFS servers seems to be labstore3 and labstore4 [10:36:58] just labstore3 [10:37:03] which is under heavy load http://ganglia.wikimedia.org/latest/?r=week&cs=&ce=&m=cpu_report&s=by+name&c=Labs+NFS+cluster+pmtpa&h=&host_regex=&max_graphs=0&tab=m&vn=&sh=1&z=small&hc=4 [10:38:08] hi [10:38:17] is here anyone that need any help with tools project? :0 [10:38:25] * petan is lazy to read [10:38:44] * Danny_B [10:38:53] O: [10:39:39] hashar: is it manybubbles and his solr again? :P [10:39:55] petan: do you know if mod_rewrite is enabled for our apaches? [10:40:03] let me check [10:40:08] petan: harshkot_ [10:40:14] YuviPanda you can always check on beta which is same [10:40:16] and if it is allowed to be setup from .htaccess [10:40:26] true, but you already know it :D [10:40:46] YuviPanda: yeah that is solr I guess [10:40:55] yes it is [10:40:56] http://ganglia.wmflabs.org/latest/?c=solr&m=load_one&r=custom&s=by%20name&hc=4&mc=2&cs=7%2F13%2F2013%208%3A00&ce=7%2F13%2F2013%2020%3A00 [10:40:57] hashar: yeah, that needs to go elsewhere, I'd think. [10:40:58] YuviPanda yes [10:41:10] petan: hmm, okay. and is overridable from .htacess [10:41:11] solr spike started between 7/13/2013 8:00 and 7/13/2013 20:00 [10:41:20] how do I check it? :o [10:41:21] just like the NFS one http://ganglia.wikimedia.org/latest/?r=custom&cs=7%2F13%2F2013+8%3A00&ce=7%2F13%2F2013+20%3A00&m=cpu_report&s=by+name&c=Labs+NFS+cluster+pmtpa&h=&host_regex=&max_graphs=0&tab=m&vn=&sh=1&z=small&hc=4 [10:41:52] Quoting him : "I'm stressing NFS in labs. If you're suffering from super poor performance and I'm not around feel free to kill my job by restarting solr-mw2.pmtpa.wmflabs." [10:41:53] hashar: yeah, we need to do something about that. move it maybe. [10:42:01] heh [10:42:10] that should be done on real hardware or a dedicated NFS server [10:42:11] we should setup a cron that kills it right after he starts it. every time :P [10:42:21] meanwhile we need someone to reboot solr-mw2.pmtpa.wmflabs :-] [10:42:23] hashar: who? [10:42:30] Quoting him [10:42:32] someone [10:42:32] ? [10:42:33] +1 to real hardware, I'm not sure how much you can trust perf numbers off this [10:42:34] ah [10:42:38] petan: manybubbles [10:42:42] petan: many bubbles, he posted on labs-l [10:42:47] why the heck is he stressing nfs [10:42:51] I am filling a bug with the stuff above [10:42:54] it's already slow [10:45:50] [bz] (8NEW - created by: 2Antoine "hashar" Musso, priority: 4Unprioritized - 6normal) [Bug 51350] solr stress out the NFS server - https://bugzilla.wikimedia.org/show_bug.cgi?id=51350 [10:46:10] petan: can you restart it? [10:46:31] restart what [10:46:38] solr-mw2 [10:46:41] whole box? [10:46:46] harshkot_: ask questions here, much easier than PM :) [10:46:48] petan: yes [10:47:17] YuviPanda: petan is there any way so that we can copy files from local to remote for this tool instance ? [10:47:26] use scp [10:47:46] hashar: http://en.wikipedia.beta.wmflabs.org/wiki/Special:Version is also 500ing. is that because of the NFS? [10:48:08] YuviPanda: yup apergos mentioned that to me this morning [10:48:11] same cause [10:48:11] YuviPanda what project is that instance member of? [10:48:18] YuviPanda I can't find it [10:48:20] petan: solr, I suppose [10:48:23] Special:Version shells out to "git show" for each extension registered [10:48:23] not in search [10:48:24] ah [10:48:29] and some commands takes up to 40 seconds [10:48:31] I am not a member of search project [10:48:38] hashar: aaah. [10:48:39] * solr [10:48:49] that means I can't restart it [10:48:49] petan: sigh. [10:48:49] the NFS issue is https://bugzilla.wikimedia.org/show_bug.cgi?id=51350 [10:48:49] you need to wait for someone with powers [10:49:03] hashar: so if manybubbles isn't around we can't actually restart it :D [10:49:10] since ryan and andrewbogott_afk are also probably not here [10:49:12] and coren too [10:49:35] yes they should get some more EU based labs admins... unfortunately they are not hiring now [10:49:51] otherwise I would apply :D [10:50:27] heh [10:53:06] petan: we should jsut ask him to add a bunch of us as admins [10:53:12] so that someone is there ot restart it [10:53:20] of course, moving it *off* labs would help too [10:54:45] I am talking with apergos about it [10:55:02] he mentioned to me earlier that Special:Version on beta was doing 503 [10:55:07] which is related to that NFS slowness [10:56:32] hmm, right [10:56:46] need to get out [10:56:51] I am @home with daughter sick [10:56:53] cya! [10:57:00] ow. [10:57:07] take care, hashar. [10:57:45] yes take care hashar :) [10:58:28] YuviPanda: petan is there any way to avoid use of 'become' command ? [10:58:56] why do you want to do that? [10:59:30] YuviPanda: so that I will directly log in to my project tool [10:59:36] you can't do that, no [10:59:42] (from what I know) [11:00:04] so how can I use scp command ? [11:00:51] scp tools-login.wmflabs.org:/data/project/ [11:00:52] works [11:01:09] petan: we should write docs about scp and take on help, if it isn't already there [11:01:10] * YuviPanda looks [11:01:41] YuviPanda: Thanks :) [11:12:25] !log solr apergos rebooted solr-mw2.pmtpa.wmflabs to kill out the heavy I/O load on labstore3 NFS server. See {{bug|51350}} [11:12:27] Logged the message, Master [11:13:41] YuviPanda: liangent : labs tools should be better now :) [11:13:56] hashar: was it restarted? [11:14:02] i'm trying to send an email using python's smtplib. on toolserver, i can connect to localhost via smtp and send mails. on labs, i tried using tools-mail; there is no error, but no mail gets delivered. [11:14:15] hashar: it seems so [11:14:18] nice thanks [11:14:28] how is sending mail supposed to work? [11:14:36] JohannesK_WMDE: I don't think we've an smtp server, yet. and Coren said something about tools-mail waiting for legal (or something like that?) [11:14:46] oh. ok. [11:14:54] YuviPanda: yup I asked apergos to reboot the instance to disable the solr search script [11:15:07] now I am out to do some cooking [11:16:01] ah, nice [11:16:10] JohannesK_WMDE: you should file a bug tohugh [11:16:11] *though [11:20:56] [bz] (8NEW - created by: 2Johannes Kroll (WMDE), priority: 4Unprioritized - 6normal) [Bug 51359] tools-mail doesn't deliver mails - https://bugzilla.wikimedia.org/show_bug.cgi?id=51359 [11:21:55] YuviPanda: done. ^^ [11:21:55] JohannesK_WMDE: ty! [11:47:56] hi all [11:48:01] hi nutz [11:48:07] :) [11:49:06] would it be okay, if i created a dump of a subset of a table to import into my own database? i need to do a lot of queries + i want to use an ORM (Django) so it's kinda complicated to work with qsub there. should i create that dump with a qsub as well? [12:08:00] .. it's disabled, isnt it? [12:19:01] !flow [12:19:01] sudo tcpflow -i any -C -e port 11211 [12:22:58] [bz] (8NEW - created by: 2Željko Filipin, priority: 4Unprioritized - 6normal) [Bug 51362] GettingStarted extension broken at commons.wikimedia.beta.wmflabs.org - https://bugzilla.wikimedia.org/show_bug.cgi?id=51362 [12:28:02] [bz] (8RESOLVED - created by: 2Željko Filipin, priority: 4High - 6normal) [Bug 47194] GettingStarted extension not at beta.wmflabs.org - https://bugzilla.wikimedia.org/show_bug.cgi?id=47194 [12:28:03] [bz] (8NEW - created by: 2Željko Filipin, priority: 4Unprioritized - 6normal) [Bug 51362] GettingStarted extension broken at beta.wmflabs.org - https://bugzilla.wikimedia.org/show_bug.cgi?id=51362 [12:32:31] badger badger badger badger badger badger badger badger badger badger badger mushroom, mushroom [12:33:08] this isn't the offtopic channel. oops [13:12:56] SNAAAAAKE! [13:13:17] * YuviPanda bites Coren [13:14:00] JohannesK_WMDE: Actually, /outgoing/ mail works already. [13:14:06] JohannesK_WMDE: User senmail. [13:14:10] sendmail* [13:14:28] Coren: would be nice if we had a server with stuff setup [13:14:41] "stuff"? [13:15:33] Coren: and things [13:15:35] y'kno! [13:15:36] :P [13:15:56] Coren: I mean an SMTP + a forwarder so people can hit reply, and appropriate records [13:16:00] so that it can be verified [13:16:29] That will have to wait faw duh legalz. [13:16:55] Coren: can you poke their tail again [13:19:20] Because legal says so. :-) Something about confusion created by people thinking volunteers are legally associated with the WMF because of the domain name. [13:19:42] aaah [13:19:48] wmflabs vs wikimedia ? :D [13:20:04] 'so wmflabs is not like google labs? bullshit!' [13:20:18] !ping [13:20:18] pong [13:20:18] pong [13:20:34] Coren: for your information, the Solr labs instance stressed out the tools/beta.. NFS server for /home since saturday noon (utc) [13:20:56] Coren: got solved by rebooting Nik solr instance a few hours ago. the bug is https://bugzilla.wikimedia.org/show_bug.cgi?id=51350 [13:21:32] Ah, I wqas hoping the job was done. [13:21:39] Changed default instance of #wikimedia-labs to wm-bot [13:21:53] Coren: um, things like "echo stuff | mail junk-mailbox@gmx.de" didn't work for me either [13:22:20] Hm. It should. sge sends mail okay. [13:22:22] Coren: I was wondering if solr could be made to run on its own hard drives instead of the shared one. But that might be overkill [13:22:23] and for the script, i would rather use the smtplib directly rather than firing off a process. [13:22:41] hashar: According to Ryan, "local" OpenStack drives would be even worse. [13:22:42] maybe you can look into that [13:22:58] JohannesK_WMDE: If you want to use smtplib, tools-mail is a relay. [13:23:10] Coren: I was more along the way of setting up yet another labstore that would be dedicated to the solr instance :-] [13:23:34] hashar: That'd work; but I don't think we have the hardware for it. [13:24:20] :( [13:24:28] Coren: tools-mail accepts mail with smtp but it apparently doesn't deliver it. at least i never got the ones i sent :) [13:24:33] will talk about it with nik tonight, he will follow up [13:24:40] JohannesK_WMDE: Lemme to check logs. [13:24:52] k [13:26:14] 2013-07-15 10:23:22 1Uyfw9-00075n-O8 ** junk-mailbox@gmx.de R=dnslookup T=remote_smtp: SMTP error from remote mail server after initial connection: host mx01.gmx.net [213.165.67.115]: 554-gmx.net (mxgmx109) Nemesis ESMTP Service not available\n554-No SMTP service\n554 invalid DNS PTR resource record [13:26:38] JohannesK_WMDE: Your server don't like ours because it doesn't have a reverse. That can be arranged. [13:27:28] okay. sounds good ^^ [13:30:01] bug is here Coren: http://www.der-postillon.com/2013/07/newsticker-477.html [13:30:05] oops. [13:30:11] i meant https://bugzilla.wikimedia.org/show_bug.cgi?id=51359 [13:30:27] but the above is also fun if you speak german [13:32:54] !wm-bot2 is http://lists.wikimedia.org/pipermail/wikitech-l/2013-July/070375.html [13:32:54] Key was added [13:35:02] @infobot-share-trust+ #wikimedia-labs-offtopic [13:35:02] You inserted channel #wikimedia-labs-offtopic to shared db [13:36:04] morning/whatever time it is for you, everyone! [13:36:28] I see that I screwed everyone's storage again this weekend [13:44:05] manybubbles: No so much "screwed up" as "made cry in pain" :-) [13:45:34] On the plus side, the NFS server itself resisted the onlaught. [13:51:09] Coren: I guess that is good. so, what should I do to stop this from happening? [13:52:01] manybubbles: We're going to look into providing you some alternate storage; I don't think we can really prevent mysql from hammering the storage -- it /is/ a DB engine after all. [13:52:24] manybubbles: Once the SF people get on in a few hour, I'll have a chat with them. [13:52:52] Coren: cool. I'm certainly happy to leave it off as long as we need to. [13:53:37] Coren: It might be useful to have an nfs server that is only used for this sort of thing - that way we can hurt it to our heart's content.... [13:53:49] Though I really wish I had a better solution [13:54:09] manybubbles: Perhaps; but we don't have quite that many available storage bricks. :-) [13:54:25] manybubbles: I'll have to speak with the other opsen before we know which direction we are heading for. [13:55:11] Coren: I tried local storage and that is actually slower and seems to cause other problems. [13:55:34] Coren: Could we use some traffic shaper so that no instance can get more than x % of the available NFS bandwidth? [13:56:36] scfc_de: Like I said, we'll be discussing options later. :-) [13:57:09] Coren: k [14:05:14] Coren: Nemo asked at https://wikitech.wikimedia.org/wiki/User_talk:Tim_Landscheidt#Queries_stats if we have an equivalent to http://munin.toolserver.org/Database/Summary/queries.html at Labs. I can't even find the DB servers at http://ganglia.wikimedia.org/. Where are they? [14:06:36] scfc_de: We don't. I'm not sure about where the DB servers would be in ganglia, they are really Asher's domain; but I'd be surprised if they weren't there. [14:07:58] Coren: Searched for "labsdb" and looked under various groups to no avail. [14:08:07] Coren: I'll ask Asher. [14:09:08] (BTW, such statistics would probably be interesting for "regular" DB servers as well, so the same situation as with replication lag graphs.) [14:10:37] Coren: after talking to hashar I started the process again this time running in the most crippled slow method I could think of. It'll get me somewhere while hopefully not putting any real load on the nfs. [14:10:46] please let me know if it does more than I expected [14:10:57] manybubbles: We'll keep an eye on it. [14:14:03] manybubbles: so yeah ionice will probably not help since it is only acting on the instance itself which I don't think has any knowledge about the NFS server current load / processes [14:14:29] manybubbles: maybe it is possible to mount the NFS point mysql is using with some options that would limit it s activity [14:14:44] hashar: so the plan in -labs is to wait until more opsen come in and try to work out a solution along the lines of a different nfs instance. or something. [14:15:17] yeah a dedicated labstore would be nice [14:15:19] hashar: but in the mean time I'm trying to make it 'nicer' [14:26:16] Not Found [14:26:17] The requested URL /wiki/Special:NovaInstance was not found on this server. [14:26:18] wtf [14:26:22] Coren ^ [14:26:27] wikitech is gone? [14:26:59] o_O [14:27:20] does it work to you? [14:27:26] Or, perhaps more to the point, 'Dafu?' [14:27:40] No, it's dead here too. How... "interesting" [14:27:45] meh [14:33:53] [Mon Jul 15 14:32:51 2013] [error] [client redacted] File does not exist: /srv/org/wikimedia/controller/wikis/w/index.php/ [14:33:53] root@virt0:/srv/org/wikimedia/controller/wikis/w# ls -l /srv/org/wikimedia/controller/wikis/w/index.php [14:33:53] -rw-r--r-- 1 root root 2327 May 14 21:07 /srv/org/wikimedia/controller/wikis/w/index.php [14:34:20] The trailing "/" looks odd. [14:34:44] scfc_de: No, that's normal RewriteRule fun for a mw installation with sane paths. [14:35:07] scfc_de: Besides, the error is ENOENT, not EDIR, so that's not the issue. [14:35:59] (Php gets passed the part after a / as query string) [14:36:29] What error gets thrown when a directory in the path is missing its "x"? [14:41:24] NOPERM [14:42:55] Which host runs wikitech, BTW? [14:44:58] vtrt0 [14:45:02] virt0* [14:45:19] * Coren doesn't get it. Every bit of config is okay but apache sees nothing. Whu? [14:46:16] Anything in Puppet's log in the past two hours? [14:47:10] That's what I'm looking at, but this is also the puppetmaster so the logs are... verbose. [14:47:27] Last change in operations/puppet was at 10:35, so that can't be it. [14:47:45] Oh, that's local -- ignore. [14:51:47] Coren: Anything in /etc touched in the last two hours? [14:54:46] [bz] (8NEW - created by: 2MZMcBride, priority: 4Unprioritized - 6normal) [Bug 51368] https://wikitech.wikimedia.org/wiki/ is inaccessible (404, not found) - https://bugzilla.wikimedia.org/show_bug.cgi?id=51368 [15:02:34] andrewbogott: Any good ideas? [15:03:37] scfc_de, I just noticed the problem, catching up… [15:03:43] At least favicon.ico still works :-). [15:03:47] Does anyone have a guess about how long it's been out? [15:05:03] I used it less than two hours ago IIRC. [15:05:33] When do the errors start in the logs? [15:09:57] And wikitech's up again? [15:10:05] is it? [15:10:14] For me, yes. [15:10:28] I'm even still logged in. [15:10:29] I can access wikitech.wikimedia.org (in a web browser) [15:10:49] hm... [15:11:00] Coren, did you force a puppet run on virt0? [15:11:13] Ah, apparently I did, and I killed it. Sorry! [15:11:18] andrewbogott: :-) [15:11:29] andrewbogott: viz.: -operations. :-) [15:13:57] wikitech is back up, but I'm not sure we found the root cause yet. [15:14:01] sumanah: ^^ [15:14:11] nod [15:19:02] Welp, puppet didn't undo the fix so things should remain okay. [15:19:12] * Coren starts investigating why that happened in the first place. [16:08:15] :o [16:09:14] Hi, I'm porting the "citation bot" aka "doi bot" to labs from the tool server, how do I get access to create a mysql database? [16:10:23] wrought_: do you already have a tool user? [16:10:57] local-doibot or similar? [16:11:09] wrought_: You just need to prefix it with the username being used to log into mysql followed by two underscores. For example, you might name it "p50380g12345__foo", if the username in your credentials file is "p50380g12345" [16:11:47] wrought_: https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/Help#Creating_databases has more [16:30:17] [bz] (8NEW - created by: 2Quim Gil, priority: 4Normal - 6enhancement) [Bug 51050] Connecting wikitech.wikimedia.org user profiles with community metrics - https://bugzilla.wikimedia.org/show_bug.cgi?id=51050 [17:07:59] Coren, indexing tables provides the advantage of information being readily available right? [17:08:32] More precisely, it makes the information easier to macth against or sort. [17:09:00] How do I index tables in my personal database? [17:09:59] Coren, ^^^^ [17:11:23] "create index" [17:11:53] Cyberpower678: same as regular mysql [17:12:41] Coren, so say I want to create a table called lala and columns fart, farts. How would I go about to create and index that table? [17:13:14] YuviPanda, I am a N00B in regards to SQL. I only know basic stuff. [17:13:28] Cyberpower678: I found a very nice guide to indexes, looking [17:13:36] Cyberpower678: http://www.tutorialspoint.com/mysql/mysql-indexes.htm [17:14:44] What's the difference, performace wise, between unique and non-unique? [17:14:55] Coren, YuviPanda: ^ [17:14:59] and http://www.sitepoint.com/optimizing-mysql-application/ [17:15:40] Coren, any chance of userindexing the externallinks table for enwiki? [17:18:00] brb [17:18:18] ... there are no user columns in that table. What in blazes are you talking about? [17:18:45] Coren, I meant indexing that table. Sorry. [17:18:46] (and that table has no redaction, so any indices are preserved) [17:20:25] Coren, I meant indexing that table. Sorry. [17:20:58] Cyberpower678: Please stop pining me and repeating your statements; I read them the first time. :-) [17:21:17] Wasn't sure. You're not responding. [17:21:18] :p [17:22:00] There are five indexes on that table; it's very indexed already. But besides, that's moot since the indices that exist in replication are exactly those of production and need to remain the same. [17:22:47] Cyberpower678: "[13:18:46] [13:20:25] " That's not even a minute and a half of delay. You need to learn a bit of patience. [17:23:25] Coren, I swear that was 5 minute. :p [17:24:29] I'm going to be using that table a lot in the future. [17:24:39] For my newest script. [17:25:10] Which I'm trying to speed up from an 80 hour run to something significantly smaller. [17:38:52] Anyone around? [17:39:13] * Mono needs help [17:39:40] Depends on what kind of help. This is for village idiots only. [17:39:43] :D [17:40:09] Well, I have trouble with the Labs between the projects, instances, etc. [17:40:24] Which project? [17:40:28] basically I want a public-facing web server to run some Python code (like the Toolserver does) [17:40:44] Are you using tools? [17:41:24] No, I'm not - basically I need to start from scratch because the last time I used WMF Labs it was in early beta and there was no tools project [17:42:01] I have an account on wikitech [17:42:11] Coren, ^ [17:42:30] * YuviPanda waves at Mono [17:42:33] ah there https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/Access_Request/Mono [17:42:46] * Mono waves back at YuviPanda [17:42:48] Mono: http://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/Help :) [17:43:00] Mono: and you can run python stuff as CGI for now (WSGI coming *soon*) [17:43:05] Mono: You've been added. [17:43:10] great [17:43:29] ooooo [17:43:32] we're getting WSGI [17:43:35] ? [17:43:36] fwilson: yeah, uwsgi [17:43:45] shiny [17:43:51] fwilson: emperor+tyrant mode. Coren said should be coming soon (this week, maybe0 [17:43:51] ) [17:43:56] fwilson: should be much faster [17:44:00] ooo also shiny [17:44:01] * Mono generates a code [17:48:10] can I SFTP in from Windows Explorer or do I need extra software? [18:00:48] Coren: https://gerrit.wikimedia.org/r/#/c/73790/ [18:01:10] https://gerrit.wikimedia.org/r/#/c/73530/ <— I have serious doubts anyone is going to review this one, so I'll just merge it [18:46:37] did s7 replication ever come online? [18:56:51] NFS is slowing down again [18:56:59] hashar: ^ [18:57:08] manybubbles: is that you again? :) [18:57:33] I should have pasted some ganglia links on https://bugzilla.wikimedia.org/show_bug.cgi?id=51350 :D [18:57:35] YuviPanda: I've got mine running in slow slow slow mo. I'll turn it off totally and you can see if it get better [18:57:53] off [18:57:56] http://ganglia.wikimedia.org/latest/?c=Labs%20NFS%20cluster%20pmtpa&h=labstore3.pmtpa.wmnet&m=cpu_report&r=hour&s=descending&hc=4&mc=2 [18:58:02] system CPU skyrockected [18:58:09] so that seems to be a different issue than yesterday [18:58:57] need ops to look at labstore3 :-] [19:03:52] Coren you realize the load of webserver 01? [19:03:56] like 90 [19:04:05] and what about tools-login? [19:04:10] no idea [19:04:13] * Nettrom tried to cat a file [19:04:22] load average: 102.48, 65.82, 31.13 [19:04:35] eh [19:04:41] fucking nfs [19:05:45] yep. NFS server is shitting itself again [19:06:12] 112 load on login :/ [19:07:11] Ryan_Lane, me and YuviPanda were wondering if you weren't about to hire more labs admins... we often need someone with access in time when nobody of ops is around [19:07:32] the load is likely due to nfs shitting itself [19:07:35] I was specifically wondering about more people being able to reboot manybubbles' instances :P [19:07:41] if there is no budget to hire more labs admins, did you consider to empower some volunteers? :0 [19:07:42]