[01:25:02] marktraceur: updated patchset with another bugfix for skipping tutorial [01:54:40] zz_YuviPanda: What a strange channel to use [02:40:49] [bz] (8NEW - created by: 2Matthew Flaschen, priority: 4Unprioritized - 6enhancement) [Bug 50999] Install git-review on Labs - https://bugzilla.wikimedia.org/show_bug.cgi?id=50999 [04:00:11] [bz] (8NEW - created by: 2Tilman Bayer, priority: 4Unprioritized - 6normal) [Bug 50556] Total number of Wikipedia articles is ca. 11 million too high - https://bugzilla.wikimedia.org/show_bug.cgi?id=50556 [06:23:39] * zhuyifei1999 wonders why no one is saying a word [07:44:18] * zhuyifei1999 wonders why zeljkof is flooding this channel [07:48:54] zhuyifei1999: it should stop soon [07:49:00] I spoke with him [07:49:17] he's disabling his client b/c he's not on a very good connection right now [07:49:55] Jasper_Deng: ok. Thanks for telling me. [07:52:32] hi [07:54:11] * zhuyifei1999 says hi back [08:03:43] Hi petan :) [08:15:14] Unable to parse the feed from http://rss.gmane.org/messages/excerpts/gmane.org.wikimedia.labs this url is probably not a valid rss, the feed will be disabled, until you re-enable it by typing @rss+ mail [08:16:25] @rss+ mail [08:16:26] Permission denied [10:58:14] [bz] (8RESOLVED - created by: 2Matthew Flaschen, priority: 4Unprioritized - 6enhancement) [Bug 50999] Install git-review on Labs - https://bugzilla.wikimedia.org/show_bug.cgi?id=50999 [11:35:26] Cyberpower678: Can you wait for your cloak before you join the channel? [11:35:54] zhuyifei1999, ??? [11:36:45] (11:34:29) Cyberpower678!~Cyberpowe@spruce-4.stoweaccess.com just joined the channel [11:36:46] (11:34:29) Cyberpower678!~Cyberpowe@spruce-4.stoweaccess.com has quit [Changing host] [11:36:47] (11:34:29) Cyberpower678!~Cyberpowe@wikipedia/Cyberpower678 just joined the channel [11:37:03] time converted to UTC [11:37:08] Out of my control. [11:38:04] but you can wait for your cloak in !status [11:39:38] Cyberpower678: http://freenode.net/faq.shtml#nocloakonjoin [11:55:22] Cyberpower678: That's better [11:55:48] zhuyifei1999, I didn't do anything. The network is just stupid [11:56:12] :o [11:57:37] Yeah, you're using X-chat, which is in http://freenode.net/faq.shtml#nocloakonjoin [12:00:29] Cyberpower678: But how come I havn't got a cloak yet? [12:00:43] I submitted long ago [12:02:45] LOL [12:38:12] hi, how is prior calculated for sge jobs? [12:57:46] can someone help me with a labs instance that goes into the "stopped" state when I try to reboot it? [13:03:15] so now my instance is up but seems to be in 10.4.0 instead of 10.4.1 and I get no route to host when I try to connect to it. [13:04:23] manybubbles: seems "they" are still sleeping [13:04:31] perhaps Thehelpfulone can help though [13:04:58] AzaToth: man, sleeping, what a pain. I wish I didn't have to do it. [13:05:04] heh [13:05:17] * AzaToth is listening to Pokarekare Ana (Vocalise) by Traditional [13:06:58] o.o [13:07:23] AzaToth what's up [13:07:36] space [13:08:24] clouds [13:08:39] birds [13:10:11] * YuviPanda is getting a new Linode VPS [13:33:10] YuviPanda: Linode rokcs [13:33:18] drdee indeed [13:33:29] drdee: i have one at prgmr, but they *were* cheaper. Linode is better at same cost right now... [13:33:50] drdee: I tried to sign up for hetzner, but they asked me for a copy of passport + credit card photocopy.... not happening :) [14:29:31] I'm sure no one is in who can fix this either but I'll just throw it out there - when I try to use nfs for /data/project it just hangs. [14:29:51] Coren: ^ [14:29:57] (he was active elsewhere a short while ago) [14:30:33] "when you try to use nfs"? Context, instance? [14:30:45] Coren: solr-mw2. [14:30:58] solr is on NFS? [14:31:33] Coren: solr-mw2 is my host, sorry. when I apply role::labsnfs::client then `ls /data/project/` I hang [14:32:58] Did you reboot it after applying? autofs is completely unable to cope with changing a mount properly. [14:33:11] * Coren checks status [14:34:28] Coren: I didn't. let me try [14:34:46] Yeah, automount is wedged. [14:35:22] Automount works very well as long as you don't try to fiddle its settings while it runs. :-) [14:36:49] Coren: what logs did you check for automount? [14:37:22] manybubbles: Not logs, I just looked at the running processes and noted they were unresponsive to SIGUSR1 [14:37:25] :-) [14:37:57] Coren: ok! [14:38:11] Coren: well I'll keep going from here. thanks! [14:38:29] I note that the instance is back up and properly uses NFS now. \o/ [14:43:37] hey Coren [14:43:47] when are you going to start that mediawiki hosting project? [14:44:06] want any help w that? :0 [14:44:12] Probably after Wikimania. [14:44:16] ok [14:46:31] YuviPanda: So whats the status on wsgi? I saw the bug mail [14:46:45] lazyktm: yeah, should be done by this week, says Coren [14:46:52] yay :D [14:46:55] I know that there are puppet patches being merged, so :) [14:47:05] lazyktm: it also won't be dealing with apache, so will be significantly faster too [14:47:25] Nonono. I said it /might/ be in this week, but that's optimistic and that next week is more likely. [14:47:56] I can't hear you! :) [14:47:58] YuviPanda: There's still an Apache in front to proxy so that I can rewrite the headers. [14:48:22] Coren: of course, but the CGI overhead will be gone, and uwsgi is in general much faster than apache+mod_wsgi (or so I am told) [14:49:29] Coren: you know you can use toolsbeta to test apaches? :P [14:49:53] I'm not touching the apache config, really. [14:50:01] I think you will need to [14:50:08] We already /have/ a proxy. :-) [14:50:11] but... I meant upgrade to 2.4 more than configs [14:50:32] Ah, yes, that upgrade is postponed for quite a while. It'll go to toolsbeta first certainly. [14:50:37] Coren: hmm, also are the web servers accesible from other labs instances? [14:50:38] cool [14:51:14] YuviPanda: They are, but not through the floating IPs (Openstack limitation) [14:51:27] 'floating IPs'? [14:54:01] hi [14:54:46] anyone around who can increase our CPU quotas for the wikidata-dev project ? [14:56:01] andrewbogott: ^ [14:56:12] or Coren [14:56:14] sure, how many do you need? [14:56:45] Abraham_WMDE, increase by how much? [14:56:50] we might also like an instance for [14:56:52] solr stuff [14:57:12] i don't know how much we need [14:57:17] 10 more CPUs would be fine [14:57:19] :) [14:57:22] yeah [14:57:23] thx. [14:57:37] and one public IP address would also be very helpful [14:58:09] denny is moving some of his tools to labs [14:58:14] wikidata stuff [14:58:27] Abraham_WMDE, ok, should be all set. [14:58:55] andrewbogott: great! thank you [14:59:10] thanks! [15:00:33] [bz] (8NEW - created by: 2Chris McMahon, priority: 4Unprioritized - 6major) [Bug 50622] Special:NewPagesFeed error - https://bugzilla.wikimedia.org/show_bug.cgi?id=50622 [15:00:47] [bz] (8REOPENED - created by: 2Chris McMahon, priority: 4Unprioritized - 6major) [Bug 50623] Entering AFTv5 feedback causes error - https://bugzilla.wikimedia.org/show_bug.cgi?id=50623 [15:01:09] andrewbogott: the public IP is not posible to allocate, does this take some time? [15:01:30] Abraham_WMDE, sorry, I missed the thing about IPs. One moment. [15:03:48] Unable to parse the feed from http://rss.gmane.org/messages/excerpts/gmane.org.wikimedia.labs this url is probably not a valid rss, the feed will be disabled, until you re-enable it by typing @rss+ mail [15:04:35] Abraham_WMDE: OK, raised. [15:06:21] @rss- mail [15:06:21] Item was removed from db [15:10:08] [bz] (8NEW - created by: 2orenbochman, priority: 4High - 6major) [Bug 50222] tool-labs is not configured correctly to run phpunit tests - https://bugzilla.wikimedia.org/show_bug.cgi?id=50222 [15:10:22] [bz] (8NEW - created by: 2Yuvi Panda, priority: 4Unprioritized - 6normal) [Bug 50422] Replicate the Gerrit mysql database to labsdb - https://bugzilla.wikimedia.org/show_bug.cgi?id=50422 [15:10:23] [bz] (8NEW - created by: 2Peter Bena, priority: 4Normal - 6normal) [Bug 48930] (Tracking) Database replication services - https://bugzilla.wikimedia.org/show_bug.cgi?id=48930 [15:11:10] * dan-nl waves at andrewbogott [15:11:18] [bz] (8RESOLVED - created by: 2billinghurst, priority: 4Low - 6normal) [Bug 35488] bots.wmflabs.org no https - https://bugzilla.wikimedia.org/show_bug.cgi?id=35488 [15:12:49] andrewbogott: still having a problem accessing a labs instance … i can get into bastion, but i'm never allowed into an instance. created a new instnace and tried to get into it without success. do you know what might be wrong? [15:13:04] [bz] (8NEW - created by: 2Tim Landscheidt, priority: 4Low - 6enhancement) [Bug 50585] Silence the qacct transfer jobs and monitor them with Icinga instead - https://bugzilla.wikimedia.org/show_bug.cgi?id=50585 [15:51:51] Coren: Is /shared/viewstats/ abandoned, or is whatever copies the files just broken since May 31? [15:52:36] anomie: That data is provided for by one of the volunteers, so I don't know for certain how it is updated. I'd ask on labs-l [15:53:33] what is that? [15:53:56] Coren: Ok. Someone asked something on wikitech-l that seems to be related (subject line "[Wikitech-l] How to get the number of pages in a category"), so I mentioned its existence. [15:54:27] ... that doesn't seem related, actually. :-) [15:55:39] Coren: He asked about page view stats for pages in the categories, too [15:55:52] Ah, /that/ is related. :-P [15:56:04] I see the thread. [15:59:43] is anyone else having an issue logging into their instances? [16:04:00] hi is there a list of the all the available dbases being replicated on labs ? [16:05:12] OrenBochman: It should be "all of them", but you can get a cheat list by taking a look at /etc/hosts on any of the tool labs instances; they are enumerated there. [16:14:36] thanks - i'm adding support to my tools for all dbs in a bit ... [16:19:41] Coren: Could you update the list at /Help so that we have something authoritative on what is supposed to work? [16:25:24] is there an issue with creating new instances and logging into them? [16:25:57] just created another instance and while i can get into bastion1 i cannot log into the instance [16:27:24] i am logging into toollabs, after a while [16:27:48] how did i get http://dpaste.com/1292684/ as my .profile file> [16:27:49] ? [16:27:57] * aude did not do that! [16:29:03] aude: so you get into bastion1 and then you're able to ssh into tool labs? [16:29:34] ssh tools-login [16:29:50] and i can get into other instances [16:30:34] aude: thanks for checking … i'm getting nothing on my instances, even a new one i just created [16:31:06] no idea why [16:31:08] aude: You're the only user with such a .profile (which file identifies as "VAX COFF executable" -- wow). [16:31:28] yikes [16:31:50] i think this is my second time logging in [16:32:33] * aude does not know what VAX COFF executable is [16:34:14] aude: Did you compile anything on Tools? (Even though it would be highly unlikely that a compilation product is saved as ~/.profile.) [16:34:52] My assembler-fu is too weak to dissect that further. [16:35:28] scfc_de: i doubt it [16:35:30] * aude checks [16:36:19] who can i ask for help in troubleshooting why i cannot log into an instance? [16:36:20] wtf, it seems my tool got deleted! [16:36:27] * aude can't find my stuff [16:37:20] gah! [16:37:38] andrewbogott: Can you help dan-nl? [16:37:46] aude: What was the tool's name? [16:37:47] Coren: help! [16:37:50] geonotice [16:37:58] ah, yes, dan-nl, thought I missed you. [16:38:06] * aude making something that displays the geonotices [16:38:10] * andrewbogott reads backscroll [16:38:26] i see the directory still in /data/projects [16:38:37] aude: /data/project/geonotice is *file*, should be a directory. [16:38:44] wtf! [16:38:50] i do have a copy, but still [16:38:59] who's deleting tools? or how? [16:39:02] Another "VAX COFF executable". [16:39:45] Coren: Could you take a look at that? It seems as it has been this way since May 27 12:38. [16:40:31] doesn't leave me confident to have tools there [16:40:33] andrewbogott: i may be doing something wrong of course, but the login routine i'm using is the same i've been using for about 9 months … i can get into bastion1 and then when i ssh into an instance ssh just hangs on offering the rya pub key … it indicates the correct path for the key, but it never connects with the instance [16:41:17] aude: That's the only file in /data/project, all the other tools seem fine. Was it Java/Perl/PHP/Python/...? [16:41:22] dan-nl, tell me project and instance again? [16:41:29] php i think [16:41:40] if that [16:41:46] could be just html and js [16:42:03] andrewbogott: ryan accidentally deleted the original so i created a new one this afternoon …. https://wikitech.wikimedia.org/wiki/Nova_Resource:I-0000080b [16:42:38] ah, yes i remember...... [16:42:49] aude: But nothing where something would be compiled and a "--prefix $HOME" could screw something up? [16:42:53] it was just a js file i think [16:42:59] and maybe index.html or index.php [16:43:11] * aude doesn't know how to do that [16:43:20] no python [16:43:55] Coren: Could aude's problems be an artifact of the XFS troubles? [16:44:06] http://en.wikipedia.org/w/index.php?title=User:Aude/common.js&diff=prev&oldid=557189866 [16:44:13] dan-nl, try again now? [16:44:18] i used the js file initially for testing my changes [16:44:31] and then with the intention to have something to view the geonotices [16:44:48] to see what areas they cover [16:44:49] andrewbogott: that worked, thanks! what was the problem? [16:45:16] /home wasn't mounting properly because of a stuck process on the file server. [16:46:38] andrewbogott: thanks. now i can get to setting everything up again … [16:47:00] andrewbogott: is there anyway i could have figured that out on my own? [16:47:09] dan-nl, not really :( [16:47:31] aude: We had a filesystem corruption last week (cf. labs-l), so this may be related to that. But we need to wait for Coren to investigate that. [16:47:50] scfc_de: could be [16:48:01] since i had not much there, not a big deal [16:48:23] but reminder to always have a copy of everything in github or gerrit [16:48:42] aude: +1000 :-). [16:49:28] my .profile is in group adminbot [16:49:32] is that normal? [16:50:08] my .bashrc is owned by reza? [16:50:17] (another user?) [16:50:28] my.cache owned by amire80 [16:50:31] .cache [16:50:55] the .bashrc is nonsense [16:51:02] liek a list of wiki pages [16:51:06] like [16:51:37] .bash_logout also owned by amir [16:52:30] aude: No, that's not normal. [16:52:36] not normal at all [16:52:59] aude: All files in your user directory should be owned by aude.wikidev. [16:53:09] yeah :) [16:54:42] * aude waits for coren [17:10:14] * Coren is back. [17:11:24] Coren: any idea what happened to my tool and toollabs account? [17:11:41] aude: There was filesystem corruption, but as far as I could tell there was no data loss except for the rollback (because we restored from a snapshot dating ~1.5h before the problem) [17:11:43] home directory files not owned by me [17:11:57] when did this occur? [17:12:12] aude: You're the only user having reported any issues, so it might be unrelated. [17:12:16] aude: Early last week. [17:12:22] ok [17:12:32] aude: May I go take a look by myself? [17:12:34] there's no way i could have done this stuff [17:12:35] yes [17:12:58] like my .bash_rc is interesting [17:13:08] owned by someone else and its contents [17:13:19] .bashrc [17:13:31] That is... wow. How completely random. [17:13:37] yeah! [17:13:48] and my tool directory disappearing [17:13:59] What's the name of your tool? [17:14:03] geonotice [17:14:15] ! [17:14:15] There are multiple keys, refine your input: !log, $realm, $site, *, :), ?, access, account, account-questions, accountreq, add, addresses, addshore, afk, airport-centre, alert, amend, ask, bang, bastion, beta, bible, blehlogging, blueprint-dns, bot, bots, botsdocs, broken, bug, bz, chmod, cmds, console, cookies, coren, Coren, credentials, cs, Cyberpower678, damianz, damianz's-reset, db, del, demon, deployment-beta-docs-1, deployment-prep, doc, docs, domain, enwp, epad, etherpad, extension, failure, flow, forwarding, gerrit, gerritsearch, gerrit-wm, ghsh, git, git-puppet, gitweb, google, group, hashar, help, helpmebot, hexmode, home, htmllogs, hyperon, info, initial-login, instance, instance-json, instancelist, instanceproject, ip, keys, labs, labsconf, labsconsole, labsconsole.wiki, labs-home-wm, labs-l, labs-morebots, labs-nagios-wm, labs-project, labswiki, leslie's-reset, link, linux, load, load-all, logs, logsearch, mac, magic, mail, manage-projects, meh, mobile-cache, monitor, morebots, msys-git, nagios, nagios.wmflabs.org, nagios-fix, nc, newgrp, new-labsuser, new-ldapuser, nova-resource, op_on_duty, openstack-manager, origin/test, os-change, osm-bug, pageant, password, pastebin, pathconflict, petan, petan..., petan-build, petan-forgot, ping, pl, pong, port-forwarding, project-access, project-discuss, projects, proxy, puppet, puppetmaster::self, puppetmasterself, puppet-variables, putty, pxe, pypi, python, pythonguy, pythonwalkthrough, queue, quilt, rb, remove, replicateddb, report, requests, resource, revision, rights, rq, rt, rules, Ryan, Ryan_Lane, ryanland, sal, SAL, say, screenfix, search, searchlog, security, security-groups, seen, sexytime, shellrequests, single-node-mediawiki, snapshits, socks-proxy, ssh, sshkey, start, stats, status, stucked, sudo, sudo-policies, sudo-policy, svn, taskinfo, terminology, test, Thehelpfulone, tooldocs, tools-admin, tools-bug, tools-help, tools-request, tools-web, tunnel, tygs, unicorn, venue, vim, vmem, whatIwant, whitespace, wiki, wikitech, wikiversity-sandbox, windows, wl, wm-bot, wmflabs, [17:14:16] i didn't really do much work on it yet, so no data loss [17:14:24] blergh [17:14:25] That... wow. [17:14:26] huh :) [17:14:30] !coren [17:14:30] The toolmeister: http://www.mediawiki.org/wiki/User:MPelletier_(WMF) [17:14:45] * addshore gently slaps Coren with a fish for the ping :P [17:14:46] AzaToth: Yeah, I wasn't talking to wm-bot. :-) [17:14:50] hehe [17:15:03] !:) [17:15:03] /me laughs [17:15:11] !* [17:15:12] $* [17:15:19] !addshore [17:15:19] fail [17:15:22] :D [17:15:22] !!log [17:15:22] petan needs a new hobby :P [17:15:27] * Coren inventories the other tools' homes. [17:15:44] gwicke: can't decide? [17:16:29] AzaToth: switching from wireless to wired triggers a bip reconnect.. [17:16:58] k [17:17:01] aude: As far as I can tell, your home is the only victim. What in blazes could have been different about you and your tool? [17:17:40] my tool was nothing essentially [17:17:45] aude: Your tool's home has been... replaced by a regular file. [17:17:54] i had a js file in it and maybe index.php or index.html [17:17:56] # file geonotice [17:17:56] geonotice: VAX COFF executable - version 24279 [17:17:57] Coren: not even a link? [17:17:59] yeah [17:18:11] * Coren boggles. [17:18:36] seems to be a random file, unless someone is actually using vax [17:19:04] Coren: strings geonotice? [17:19:06] AzaToth: I haven't actually used vaxen in ~15 years, I doubt I still have executables lying around. :-) [17:19:12] anything intresting in it [17:19:13] * aude doesn't know what vax is [17:19:25] aude: Do you remember when your last login was? [17:19:26] aude: what your daddy used [17:19:37] :) [17:19:39] Coren: I have never used VAX [17:19:41] probably end of may [17:19:49] AzaToth: No text; just almost-strings from code. It's either actual program text or its random binary junk. [17:20:03] hmm [17:20:06] http://en.wikipedia.org/w/index.php?title=User:Aude/common.js&diff=prev&oldid=557223001 [17:20:12] * aude removed the js from my user js then [17:20:17] and I assume objdump doesn't reveal anything either? ツ [17:20:33] not sure i did anything since then, though possible i logged in again [17:20:34] AzaToth: I'd have to install multiplatform binutils. [17:20:43] aude: I think he meant login to the server [17:20:55] aude: I see only aude pts/41 bastion1.pmtpa.w Tue Jul 9 16:26 - 17:01 (00:34) in July [17:20:56] i logged in today [17:21:06] i was in bastion and then ssh tools-login [17:21:18] not sure that's the right way to do but should work, right? [17:22:10] aude: It's not necessary, but it works (you can login directly to tools-login.wmflabs.org also) [17:22:13] Coren: did you chmod the geonotice? [17:22:15] chown* [17:22:25] I see it's owned by local-geonotice now at least [17:22:27] aude: Do you mind if I take a few more minutes to inspect this before I fix your accounts? [17:22:54] AzaToth: That doesn't mean much, my toolwatcher daemon would have chowned it. [17:22:55] go ahead [17:23:17] k [17:25:16] * liangent guess d8927129973ab0fd1ea1b58332555cfdf9a5f827^ is enough [17:25:54] Coren: on tools-login I made ls -l /home [17:26:01] and the groups are fucked up imo [17:26:12] AzaToth: How so? [17:26:32] * Coren sees nothing amiss. [17:26:33] 30% is group svn [17:26:44] AzaToth: That's normal for hysterical raisins. [17:26:50] oh [17:26:59] AzaToth: It depends how the account was originally created for gerrit. [17:27:03] ok [17:27:17] (I.e. group svn have users who existed back then) [17:27:33] I assume as well the ownership of /home/icinga is correct as well then? [17:28:09] Also, but for other reasons. [17:28:12] k [17:28:53] anyway, /home/pleclown is chmodded 777 [17:28:58] which must be wrong [17:29:20] aude: I can't seem to find rhyme or reason to the oddities in your home. [17:29:21] that's not good [17:29:32] [bz] (8NEW - created by: 2Quim Gil, priority: 4Unprioritized - 6enhancement) [Bug 51050] Connecting wikitech.wikimedia.org user profiles with community metrics - https://bugzilla.wikimedia.org/show_bug.cgi?id=51050 [17:29:33] Coren: we can just see if it happens again [17:29:52] as always, once i have anything i care about, i will have a backup and stuff in git [17:29:55] Coren: is /home/nettrom a "real" normal file, or an other fubar? [17:29:58] AzaToth: It is -- but it's also not entirely surprising. User education on why to not chmod 777 their homes is... harder than first appears. :-) [17:30:11] although worried about my files ending up some place else [17:30:17] Coren: I don't think you as a normal user can rechmod your own home dir? [17:30:26] AzaToth: Sure you can. You own it. [17:30:33] aint that dependent on the permission of /home? [17:30:37] e.g. for bots, not sure where people put passwords etc [17:30:40] * Coren shakes head. [17:30:44] AzaToth: I chmoded it to copy files. [17:30:45] k [17:30:51] would be bad for something like that of someones to end up some place else [17:31:02] or be readable [17:31:19] aude: As far as I can tell, that's a freak accident and you seem to be the only one to whom that happened. I'm going to preuse the entire filesystem to see if anything else is out of place, but it seems unlikely. [17:31:21] I was able to log as projectname, but I couldn't access anything.... [17:31:27] Coren: ok [17:31:43] Coren: /home/nettrom? [17:31:44] and can you restore my tool to a new state [17:31:49] e.g. recreate it [17:31:52] Also, it looks like things may have been misplaced, but permissions unaffected (as one would expect if inodes got shuffled around) [17:32:00] yeah [17:32:09] * aude would like to work more on the tool [17:32:19] So it's not a major security concern, even though it's troubling. [17:32:27] Coren: ok [17:32:28] aude: Your tool is back. I'll clean up your home now. [17:32:32] yay! [17:33:57] Oh! [17:34:07] huh? [17:34:13] You seem to have gotten *hardlinks* to random other files. [17:34:30] Including, get this, a hardlink to someone else's home *directory* [17:34:35] oh really? [17:35:14] ... a hardlink to a directory is definitely filesystem corruption. Damn. [17:35:25] makes sense [17:35:28] Apparently, the snapshot I restored from already had some corruption. [17:35:38] It just hadn't surfaced yet. [17:35:41] hopefully it's isolated [17:36:15] I've had nobody else report issues, and it's been over a week, so what cases there may be seem to be rare. [17:36:26] hope so [17:36:38] i'm still getting used to toollabs [17:36:40] * Coren finishes cleanup and tries to find out more precisely the extent of the problem. [17:36:46] not logging in so often yet [17:37:25] aude: Yeah, that should hopefuly be a freak even that will not reoccur; having a filesystem crash this way is not commonplace. [17:37:39] sure [17:37:39] anyone who can add me to deployment-prep? [17:38:27] aude: was that 'sure' for me? :D [17:38:39] YuviPanda: no [17:38:41] aww [17:39:02] don't know if i have permission to do that [17:39:11] YuviPanda: Normally, any member of the project can add you. [17:39:17] looks like i can [17:39:20] i can add you [17:39:23] aude: woo! [17:39:36] Coren: hmm, chrismcmahon says he doesn't have the perms? [17:39:44] success [17:40:00] woo [17:40:02] yurik: Oh, it's possible that this was restricted to project /admins/ [17:40:13] tabcomplete strikes again! [17:40:16] i don't know about project admin, though [17:40:41] let me log in [17:40:51] looks like most people are project admin [17:40:55] even me :) [17:40:56] chrismcmahon: you are project admin, so you should've been able to add me. [17:40:56] yeah [17:41:05] aude: I think I've cleaned up the residue right. [17:41:10] is... everyone project admin?! [17:41:11] Coren: thanks [17:41:17] YuviPanda: most everyone [17:41:28] * aude trusts you not to go around deleting stuff [17:41:36] :D [17:42:07] !log deployment-prep added Yuvipanda to the project [17:42:09] Logged the message, Master [17:42:31] aude: Just in case you're curious, a cursory random sample of other homes and project dirs show no obviously out of place files. [17:42:40] Coren: interesting [17:42:50] * aude wonders how i got singled out :) [17:43:16] anyway, thanks Coren for fixing stuff [17:43:24] things happen.... [17:43:30] aude: That's what I'm here for. [17:43:36] :) [17:43:44] alright, time to go home and eat [17:43:49] * aude around later [17:44:37] * Damianz looks at Coren [17:44:43] Coren: I seem to be having the same problem dan-nl was having earlier. ssh's to bastion, can't to others from there. stuck at 'debug1: Offering RSA public key: /Users/yuvipanda/.ssh/licheking' [17:45:28] * YuviPanda adds more vs [17:45:49] debug2 tells me debug2: we sent a publickey packet, wait for reply [17:45:59] YuviPanda: Needz moar contexts! What instance? [17:46:08] Coren: deployment-sql02 [17:46:11] in deployment project [17:46:22] nothing extra in debug3 [17:46:51] YuviPanda: At first glance, the box is dead/dying. [17:47:06] too many things seem to be dying around here :( [17:47:29] YuviPanda: AFAICT, it's trashing its poor little core out. Runaway processes? [17:47:40] Coren: probably. i can ssh to other places fine [17:47:45] deployment-bastion works [17:47:54] hmm, I see *** /dev/vdb will be checked for errors at next reboot *** [17:47:57] wonder if that is ominous [17:48:05] mm, petan is listed as 'emergency contact' :D [17:48:34] petan: around? [17:48:58] YuviPanda: It normally isn't (omnious); debian preemptively checks filesystems at interval and after a number of mounts. [17:49:06] ah, okay [17:50:27] YuviPanda: Yeah, that box's userspace is dead or dying; but I can't log in with root to see what's going on. What's on the console log? [17:51:10] Coren: just DHP noise [17:51:21] Coren: but last entry was in Jul 2 06:29:49 deployment-sql02 dhclient: bound to 10.4.0.248 -- renewal in 54 seconds. [17:52:23] Krinkle: would you know where I can run maintanance scripts on deployment-prep? [17:52:53] YuviPanda: I think we need to call TOD on that box. [17:53:01] TOD? [17:53:06] Total "O" "D"? [17:53:06] Time of Death. :-) [17:53:08]