[00:00:44] !tools-help | Cyberpower678 [00:00:44] Cyberpower678: https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/Help [00:01:33] Thanks [00:07:35] Looks like the RfX scripts aren't entirely continuous supported. [00:07:51] That bad RfX that was transcluded today, crashed my script. [00:24:33] scfc_de, I'm impressed with myself. My PC tagging script has been in continuous operation for 11 days now. :) [00:47:50] Any particular reason labs is being toolserver? [00:47:51] Current Load Avg (15, 5, 1m): [00:47:51] 202%, 285%, 339% [00:47:51] Avg Utilization (last hour): [00:47:51] 91% [00:48:12] Load is high again.\ [00:49:07] hmmm [00:49:13] this is annoying [00:49:30] I moved to labs to avoid this. [00:50:02] And I'm trying to do some tests for my new script. [00:51:09] Tools' load spiked incredibly twice in the last half hour. [00:51:26] What all is involved in creating a labs account? [00:51:45] Go to wikitech and signup. There. [00:52:14] Load back down to 42 [00:52:20] for now. [00:54:51] WHO DELETED MY CRONTAB? [01:04:27] LABS IS DOWN AGAIN> :| [01:04:44] arrrrrg [01:05:13] I'm trying out my new i18n code [01:06:16] And it's up again. [01:07:02] :| [01:59:14] Cyberpower678: What in blazes are you talking about? [01:59:16] 01:58:58 up 14 days, 7:22, 26 users, load average: 0.47, 0.68, 1.29 [01:59:54] Coren: you really expect Cyberpower678 to be able to answer that? :p [02:00:47] Coren, tool labs was going up and down. Kind of annoying when I'm trying to do stuff. I actually am calling those moments Toolserver v2 from now on. [02:01:23] What I pasted up there was straight from Ganglia Tools section [02:04:06] Coren, when I notice my connection slowing down, then failing, I look at the tools in Ganglia. As my connection was failing, load was higher than 300 at that moment and then dropped off again. [02:04:22] Then it happened again a few minutes later and dropped off again. [02:04:46] Not sure what's going on there but something is not right. [02:06:23] I'm seeing a period of ~ 2.5 minutes during which idle went down. Hardly "down". OTOH, there are regular spikes as some users have overly busy crontab needlessly, or do things like reload their crontab every minute. [02:07:13] Coren, can't blame for that anymore. :p [02:07:24] And, interestingly enough, the only place where stuff is supposed to be running -- the grid -- is fine. So any properly running job wouldn't have been affected. [02:07:53] Mysteriously enough though, my crontab vanished somehow. [02:08:38] Coren, jsub sends to the grid right? [02:08:45] Well, given that you had an automated process that replaced it at interval if anything went wrong with it that might happen. Yet another reason why running cron every minute to replace your crontab wouldn't have been a good idea. [02:08:50] Yes, it does. [02:09:37] Then I'm running my scripts correctly. :D [02:10:10] They're all jsubbed. 10 of the previously cron initialized scripts are now continuous. [02:12:08] Coren, you should enforce a new rule and broadcast it globally. [02:12:54] Any user not running jobs correctly will have a wiped crontab. Second offence, the crontab will be locked out. If there are still problems, the account will be disabled. [02:13:23] Coren, does that sound reasonable? [02:13:29] No, it doesn't. [02:13:43] Why? [02:14:06] The other users aren't being fair when running the stuff they want to run. [02:14:58] Because you're missing the point of collaborative environment. (Including, I should add, the fact that shared resources are _shared_ and that occasional dips in performance art part and parcel). There are finite resources; if you need all the processeing power for your task, you need a VPS. :-) [02:17:26] Coren, let's not forget that WMF is providing us with these resources free of charge. [02:17:39] Yes, that too. [02:17:40] :-) [02:18:07] The least you guys could expect from us is that we follow the rules and run our crap in the correct place, right? :) [09:11:44] heh [09:11:48] tools are dying again [09:11:58] nfs fail [09:58:38] http://ganglia.wikimedia.org/latest/?c=Labs%20NFS%20cluster%20pmtpa&h=labstore3.pmtpa.wmnet&m=cpu_report&r=hour&s=descending&hc=4&mc=2 [09:58:40] petan: ^ [09:58:59] some project doing heavy I/O [10:00:34] petan: maybe tools-webserver-01 http://ganglia.wmflabs.org/latest/?c=tools&h=tools-webserver-01&m=load_one&r=hour&s=by%20name&hc=4&mc=2 [10:00:44] which got a nice 100% CPU usage hehe [10:01:25] started around 7:40 UTC (aka a bit more than 2 hours ago) [10:45:57] its been flakey for the last 12-18 hours [11:12:17] [05:11:59] nfs fail [11:12:26] What are you talking about? The filesystem is fine. [11:19:58] * Cyberpower678 thinks Coren is having somewhat of an off day. ;) [11:21:04] * YuviPanda_ logs in to check [11:21:07] Cyberpower678: is this on -login? [11:21:11] or what? [11:22:01] Cyberpower678: wfm too [11:22:32] YuviPanda_, ? [11:22:38] as in [11:22:40] works fine for me [11:22:41] toollabs [11:23:58] I see a burst of IO some hours ago. This would have made performace laggy for a bit, not broken anything. [11:24:10] (And, indeed, logs show nothing broken) [11:24:36] Coren: what's yoru TZ again? [11:24:41] *your [11:24:49] EDT (atm, UTC-4) [11:24:55] Yes, it's early. [11:24:58] hmm, you're up early! :) [11:25:12] Coren: is using icinga to monitor processes a good idea? [11:25:23] Coren: I'm trying to figure out a way to alert me if my processes are not running [11:25:23] !coffee Coren [11:25:23] * Helpmebot pours Coren a nice hot cup of coffee. [11:25:30] Ol.o [11:25:32] o.O [11:25:32] Coren: or my web services are down [11:25:42] Cyberpower678: I thought we kicked that bot off here into the -offtopic [11:25:43] channel [11:25:52] I thought that too. [11:25:54] YuviPanda_: Well, it's done to monitor some critical processes (like mysql daemons), but it's generally considered overkill for most things. [11:26:10] @kick Helpmebot [11:26:10] Permission denied [11:26:14] ah [11:26:29] Coren: right. not on the regular icinga but perhaps on somtehing else? [11:26:35] Coren: or do you have some other suggestions [11:26:36] "or my web services are down"? [11:26:37] I see no reason to kick it now unless it's being abused here. [11:26:46] Oh, you mean monitor them. [11:26:59] * Coren ponders. [11:27:03] Coren: yeah. 'check wp-signpost/cgi-bin/api.py, make sure it is not 500' [11:27:07] and things like that [11:27:15] anyone know d3.js here ? [11:27:18] we can have our own icinga instance maybe... [11:27:28] OrenBochman, no [11:27:43] OrenBochman: ask in -analytics when more of the US is up :) [11:27:47] It might be worthwhile to have a tool maintainer icinga; we certainly don't want enduser tools to page opsen or spam -operations [11:28:00] Coren: yeah, agreed. -operations icinga should be ignored :P [11:28:06] Coren: how hard would it be? [11:28:13] * YuviPanda_ opens docs [11:28:15] And you'd want maintainers to be able to acknowledge outages, etc. [11:28:27] yeah. Email, I suppose? [11:28:49] YuviPanda_: Open a feature request bz? It's a fair idea, but will require thought. [11:28:56] I guess I't will have to wait [11:28:58] Coren: yeah. let me do that. [11:29:20] OrenBochman: I have no knowledge of that thing. [11:29:24] Sowwy. :-( [11:30:44] [bz] (8NEW - created by: 2Yuvi Panda, priority: 4Unprioritized - 6normal) [Bug 51434] Setup an icinga instance to monitor tools on tool-labs - https://bugzilla.wikimedia.org/show_bug.cgi?id=51434 [11:31:11] Coren: done. [11:31:33] Coren: also how is uwsgi coming along? I guess you've been distracted with making NFS not suck [11:31:48] You guess properly. [11:32:19] Strictly speaking, it only sucked once every other week. :-) [11:32:52] sure, but if you average that over the month, the spikes bring up the average :) [11:32:55] WSGI is my todo today. [11:32:59] nice! [11:44:28] Coren: for like 40 minutes servers were unresposible with load about 100 [11:44:37] I think it was nfs [11:44:53] hmm, http://tools.wmflabs.org/?status isn't loading for me right now [11:45:05] and my ssh hung [11:45:06] it was like 5 hours ago [11:45:08] is that my internet? [11:45:15] (this is happening right now) [11:45:28] nah, it was 3 hours [11:45:53] no [11:45:53] i mean [11:45:57] my ssh is hung right now [11:46:02] and ?status is hugn [11:47:47] <_acs_> hi guys, I am trying to access as always the instance "ssh -v acs@korma.wmflabs.org" [11:47:58] <_acs_> but it is blocked for some minutes now in [11:48:04] <_acs_> debug1: Offering RSA public key: /home/acs/.ssh/id_rsa [11:48:17] <_acs_> do you have any clues? [11:48:51] petan: ^ [11:49:13] yes labs are fucked now [11:49:22] I don't have powers to do anything, you need to talk to Coren [11:49:24] yeah [11:49:28] it seems fucked. [11:49:47] !ping [11:49:47] pong [11:49:47] pong [11:49:48] Just veeery slow [11:49:57] @q Helpmebot [11:50:05] at least something works [11:50:06] Someone is kicking the filesystem's behind. [11:50:09] !ping [11:50:09] pong [11:50:13] *everything* works. [11:50:29] Coren: -login has load over 20 are you sure? [11:50:39] Yes. [11:50:44] is it manybubbles again? [11:50:50] can we please please kick him off labs? [11:50:51] * petan eats many bubble [11:51:01] :D [11:51:06] Not only am I on it, but I've also just ssh'ed again to it. It's sluggish, but it works fine. [11:51:07] or the NFS server, I mean. [11:51:17] can we please punch him in eye [11:51:20] yeah, I can ssh again but it was down for a minute or so [11:51:33] petan: no, we can punch solr. No violence against people! :) [11:51:51] But I think I'll have to crack down on people running stuff on -login. *sigh* [11:51:51] Coren: there is difference between sshing with root key and sshing like regular user who'm /home is mounted on nfs [11:52:00] Coren: if it works fine to you doesn't really mean anything [11:52:18] petan: I know that; and that's why I didn't. [11:52:31] I don't see anything running on -login [11:52:37] most expensive process is gmond [11:52:49] which IMHO should have priority lowered by default [11:52:54] eg nice+ [11:52:59] petan: Really? You don't see all the screens? And tmux? and crons? [11:53:06] oh man, wonderful weather. I should go out [11:53:08] that is supposed to live there or not? [11:53:17] I thought that -login is for screens and shit [11:53:22] just not bots [11:53:25] and heavy jobs [11:53:30] (and mosh?) [11:53:42] One should not run stuff in a detached screen. [11:53:55] If it's long enough to run in a screen noninteractively, it should be sent to the grid. [11:54:21] well but what about stuff like shell :P [11:54:24] or ssh session [11:54:29] that doesn't eat almost anything [11:54:40] or some tail -f [11:54:43] in screen [11:54:52] or vim [11:55:07] these things are IMHO ok, (except for emacs in screen) because emacs suck [11:55:12] petan: Then it shouldn't be in a *detached* session, shouldn't it? [11:55:18] why not [11:55:26] what if you need to go away and want to return to it later [11:55:33] like if you move from work to home etc [11:55:42] I do that a lot [11:55:54] basically I work always in a screen just to make sure if my net die it doesn't break [11:56:34] proper idle processes, like vim or bash do not eat resources [11:56:42] I see no reason why these couldn't run in background [11:56:49] gmond seems more evil to me :P [12:00:04] Back to the primary point; I see nothing but occasional spikes in NFS useage that lead to somewhat laggy disk use; in particular, when " yes labs are fucked now" I've seen a delay of all of 5-6 seconds to log in. [12:00:28] I had delay like 20 minutes that time [12:00:50] basically all my open terminals stuck (I could just move cursor) [12:01:06] OH [12:01:07] nvm [12:01:15] ... the use spike lasted all of 3 minutes. Check your network connection. [12:01:18] I thought you talked about that message from morning [12:01:24] not that one [12:01:30] this one was like 5 seconds lag [12:01:50] Hardly justifies "labs is fucked" then. [12:10:37] !part [12:11:56] @unq Helpmebot [12:11:56] Sorry but I don't see this user in a channel [12:13:05] petan: do you want helpmebot superuser to be able to make it join/part channel? Is there a reason you don't already? [12:30:20] T13|needsCoffee probably I am not trusted enough :D [12:30:22] Bleh. I need to find who's hammering on the disk this much. [12:30:27] Coren: is there no problem now? [12:30:32] because I can't do anything now [12:30:35] !pin [12:30:35] pong [12:30:42] at least this bot works :D [12:30:42] It's laggy. That's called a *shared resource* [12:30:48] load 28 [12:31:00] wait, since when shared resource is called laggy :P [12:31:09] petan: Will you *please* stop obsessing about load? [12:31:13] okok [12:31:37] that's usualy indicator that something is wrong :P [12:31:50] No, it's not. It's an indicator of the number of processes that are ready to run. [12:32:09] it's number of processes that are ready to run but do not run because they are waiting for something [12:32:16] well in fact it's neither of that [12:32:19] but it's close :P [12:32:20] And given the severe misuse of cron on -login, a one minute lag causes that. [12:33:19] (plus the time to catch up) [12:35:11] good morning! I can't ssh to my solr-mw2 machine this morning [12:35:39] or not. it let me in eventually I think [12:35:53] manybubbles: Are you hammering on NFS again? [12:36:23] Coren: no. I haven't been running it since yesterday [12:37:11] I'd really like to get back to it at some point but I won't do it until I can no break everyone else [12:38:19] * Coren disables crontabs. [12:43:06] petan: You have Helpmebot superuser now :P Now you can make it join/part channels [12:43:19] aha cool [12:43:34] Coren disabling where? [12:46:25] Hm. ContinuityBotLabs.jar [12:46:35] Seems to be the evul disk hog [13:12:35] I think we should buy like 2tb of RAM and create a huge ramdisk for labs :P [13:12:46] that would be finally good IO [13:21:52] Huh. [13:22:02] * Coren wonders why puppet hammers on the NFS. [13:23:20] no wonder :0 puppet is evil [13:23:28] @search evil [13:23:28] No results were found, remember, the bot is searching through content of keys and their names [13:23:34] !evil is puppet [13:23:34] Key was added [13:23:47] @search scary [13:23:47] Results (Found 1): puppet, [13:23:50] here we go [13:23:51] !puppet [13:23:51] http://scary.hostei.com/ascarywaterorg-/img/scary+puppet1.jpg learn: http://docs.puppetlabs.com/learning/ troubleshoot: http://docs.puppetlabs.com/guides/troubleshooting.html [13:24:13] Worse is ua31's bot on -login though. [13:24:13] retsreklawts: you there? [13:25:43] Coren: can't we kill bots on -login? [13:25:55] That's what I'm looking into atm. [13:26:00] sweet. [13:26:05] * YuviPanda_ gets out of his way. [13:29:10] I think we should do killall python :D [13:29:31] that would solve a lot of troubles :o [13:30:24] ... [14:25:55] petan: FYI, if you like graphs, the gangllia for labstore3 are easy to interpret: red spikes were the Bug of Doom. Peach/organge peaks are actual IO traffic. [14:26:14] !ganglia [14:26:18] linky [14:26:22] !labstore3 [14:26:26] @search ganglia [14:26:26] Results (Found 2): load, load-all, [14:26:32] !load-all [14:26:32] http://ganglia.wikimedia.org/2.2.0/?c=Virtualization%20cluster%20pmtpa&m=load_one&r=hour&s=by%20name&hc=4&mc=2 [14:27:07] http://ganglia.wikimedia.org/latest/graph_all_periods.php?h=labstore3.pmtpa.wmnet&m=cpu_report&r=hour&s=descending&hc=4&mc=2&st=1373975819&g=cpu_report&z=large&c=Labs%20NFS%20cluster%20pmtpa [14:27:15] !labstore3 is http://ganglia.wikimedia.org/2.2.0/?c=Labs%20NFS%20cluster%20pmtpa&h=labstore3.pmtpa.wmnet&m=cpu_report&r=hour&s=by%20name&hc=4&mc=2 [14:27:16] Key was added [14:27:27] sorry but nfs is slowing down again :/ [14:30:42] liangent: I'd expect the ones who should be sorry are the ones doing lots and lots of IO. :-) That said, that seems to be already over. [14:30:55] That was a comparatively small bump. [14:32:09] Coren: maybe we should write e-mail with instructions how to write IO effective software [14:32:14] like use /tmp for temporary stuff etc :D [14:32:45] btw where are dumps? [14:32:49] is it nfs as well? [14:33:00] I think people running dump scans may eat a lot of IO [14:41:31] What's a 'lot' of IO on the labs cluster? Is dumping 50MB user database into a backup file an issue? [14:43:07] tb4: Almost certainly not. [14:43:44] tb4: The bursts I've seen were several hundred megs of writes. [14:43:56] Okaydokay ta - IO lag may have coincided with my backup script running was all. [14:45:06] I'm thinking one of the bigger culprit may be creating lots of small files. [14:45:28] As opposed to writing a large contiguous file at one; which NFS handles fairly well. [14:49:14] !ping [14:49:14] pong [14:52:23] !ping [14:52:23] pong [14:52:25] works [14:52:25] nice [14:54:00] AzaToth ping [14:54:07] pong [14:54:39] AzaToth I have some packaging questions [14:54:55] shout [14:55:27] how do I turn that debian folder into .deb - doesn't it require the sources to be present somewhere? how does it know where the produced binaries should be installed to? where I define that [14:56:40] you usually uses "git buildpackage" (when building from a git repo, otherwize, debuild) to build it [14:57:13] !log deployment-prep restored /data/project/apache/common-local/php-master/extensions/Diff/Diff.php got deleted somehow by git [14:57:15] Logged the message, Master [14:57:53] !log deployment-prep Retriggering the database updating job manually https://integration.wikimedia.org/ci/job/beta-update-databases/ [14:57:53] Logged the message, Master [14:58:02] AzaToth: debuild complains about missing orig.tar file [14:58:11] which is what makes perfect sense [14:58:21] where should that tar be? dh_make removed it [14:58:36] usually in the parent dir [14:58:48] but I started debuild from root dir [14:58:54] there is no parent [14:59:41] should I start debuild from debian and copy tar to root [15:00:14] the orig tar is usually located in the directory above the directory containing the package [15:00:23] o.O [15:00:48] ok so I have root, then I should have root/debian and then I should have root/debian/blah/tarfile ?? [15:01:10] this is really not well documented anywhere [15:01:25] most of debian documentations expect you already knows everything [15:03:56] no [15:04:19] if you have $HOME/builds/package/debian then the orig tar should be in $HOME/builds [15:04:32] a.k.a. the parent dir [15:04:55] no I don't have that [15:05:19] I have $HOME/Documents/root_of_git/debian [15:05:40] and root_of_git is supposed to be root, which I just don't want to move behind [15:06:11] basically what I want to is being able to do [15:06:15] a) git clone url [15:06:21] b) cd [15:06:27] c) ./makepackage.sh [15:06:38] d) be happy & have fully working blabla.deb [15:07:10] that's why everything needs to be in git [15:09:20] btw AzaToth why that .tar needs to have so shitty structure? why is that required? like not having stuff in a subfolder [15:09:51] if I do tar -zxf .orig.tar.gz it unpack all files into current directory instead of / [15:10:01] which is what most of people probably expect from tar [15:10:24] you probably failed then, as 99.999999999% of all orig.tars have a dir named the package [15:10:49] no I didn't failed, I was told by someone in that debian channel that it must be that [15:11:06] my original tar was creating subfolder [15:11:08] .999999999? [15:11:13] ? [15:11:37] petan: your original tar should be the orig tar [15:11:47] right [15:14:11] AzaToth so if it unpack all stuff to .orig/ [15:14:20] would it work with debian packaging [15:14:25] no need to unpack [15:14:49] I think the problem is that you are trying to make a non-native package in a native environment [15:14:50] I suppose that .tar is going to be unpacked at some point, otherwise I fail to see why it needs it [15:14:58] ? [15:15:04] what these 2 things mean [15:15:32] a native package (lacking -N at the end of the version) has no orig tar [15:15:42] this has version [15:15:45] i.e. a package without a separate upstream [15:15:50] it's _.orig [15:15:59] I am just lazy to type all that [15:19:50] petan: create a file debian/gbp.conf with following data: http://paste.debian.net/16385/ [15:20:10] upstream-tree=branch is needed as long you aint build the debian package from distinct tags [15:20:31] defualt debian branch is master and default upstream branch is upstream, so we need to override this [15:20:52] now when you run git buildpackage, you should get "gbp:info: pidgeon_1.2.5.0.orig.tar.gz does not exist, creating from 'master'" [15:29:14] petan: copy over? [17:37:01] !ping [17:37:02] pong [17:37:55] !pong [17:37:55] don't play with me [17:38:55] <^demon> !pang [17:39:05] <^demon> Aww :( [17:40:18] @labs-info korma [17:40:18] I don't know this instance, sorry, try browsing the list by hand, but I can guarantee there is no such instance matching this name, host or Nova ID unless it was created less than 23 seconds ago [17:40:30] @labs-info korma.wmflabs.org [17:40:30] I don't know this instance, sorry, try browsing the list by hand, but I can guarantee there is no such instance matching this name, host or Nova ID unless it was created less than 35 seconds ago [17:40:39] @labs-host korma.wmflabs.org [17:40:45] I am running http://meta.wikimedia.org/wiki/WM-Bot version wikimedia bot v. 1.20.0.0 my source code is licensed under GPL and located at https://github.com/benapetr/wikimedia-bot I will be very happy if you fix my bugs or implement new features [17:41:09] @labs-resolve korma.wmflabs.org [17:41:10] I don't know this instance, sorry, try browsing the list by hand, but I can guarantee there is no such instance matching this name, host or Nova ID unless it was created less than 14 seconds ago [17:41:13] * Ryan_Lane sighs [17:41:25] @labs-resolve korma [17:41:25] I don't know this instance, sorry, try browsing the list by hand, but I can guarantee there is no such instance matching this name, host or Nova ID unless it was created less than 5 seconds ago [17:41:33] * Ryan_Lane rolls his eyes [17:41:34] <_acs_> korma.wmflabs.org exists, for sure! :) [17:42:01] what project is this in? [17:42:23] _acs_: ^^ [17:42:39] ah. contributors [17:42:43] <^demon> Ryaaannnnnnnn! I'm baaaaaacccckkkkkk [17:42:53] ^demon: yep ;) [17:43:11] <^demon> Your days of not getting bugged about my one-line puppet changes are over ;-) [17:43:23] _acs_: I can't ssh into it from inside of labs either [17:43:29] I bet you guys OOM'd the instance [17:43:30] <^demon> aka: vacation [17:43:34] @labs-info [17:43:43] <_acs_> OOM'd? [17:43:52] <_acs_> Ryan_Lane: the web is working correctly [17:43:57] <_acs_> http://korma.wmflabs.org/browser/ [17:44:00] ran it our of memory [17:44:04] *out [17:44:25] <_acs_> Ryan_Lane: pretty strange ... we are doing pretty basic things there [17:44:34] <_acs_> just copying files [17:44:40] <_acs_> and serving the files using http [17:45:39] <_acs_> Ryan_Lane: maybe some issue in the auth service? [17:45:48] no [17:45:52] unlikely [17:46:01] let's reboot it [17:46:02] <_acs_> Ryan_Lane: maybe we can reboot the instance [17:46:05] <_acs_> yeah! [17:46:06] since we can't ssh into it [17:47:36] <_acs_> Ryan_Lane: I should leave now ... I will contact you later. Thank youuuu! [17:47:50] _acs_: you can reboot the instance through wikitech [17:47:55] via Manage instances [17:48:57] petan: Eh, why did you approve Madihakanwal222's shell request as well (as Clairemarcc's)? [17:49:21] scfc_de: something wrong with them? [17:49:24] non-existent? [17:50:00] andrewbogott: it looks good to me [17:50:16] Ryan_Lane: Look like spammers to me. [17:50:40] yep [17:50:42] they are [17:50:45] revoke shell [17:50:45] Ah, okay, Madihakanwal222's request is not approved. [17:50:46] Ryan_Lane, ok, thanks. Will also check in with otto since he wrote the original class. [17:50:50] I'm going to deploy an LDAP fix [17:50:58] and we can block them [17:51:08] petan: I take it all back. [17:52:14]