[04:00:18] ? [09:35:09] grml, i hate those highlights when ever someone works on the SSL Terminators ;) [09:50:10] lol [09:50:24] at least there is no longer "log" idling here :P [09:52:21] ^^ [12:10:15] andre__: We will be featuring the June 24 bug triage meeting in the current issue of Tech News [12:10:28] andre__: That might not help with attendance, but surely will not hurt! [12:10:41] yeah, saw it in the draft. Heh, let's see, but sounds like a cool idea :) [12:10:58] An IRC meeting to organize old high priority MediaWiki bug reports will take place on June 24 at 17:00 UTC on the channel #wikimedia-office on freenode (time conversion). You are welcome to participate! [5] [12:11:25] "organize" popped up as a simple synonym for 'triage', so :-) [12:12:13] "mop" is a popular term in wikimedia projects [17:57:07] discussing skin revamp in #wikimedia-office in 5 min https://www.mediawiki.org/wiki/Architecture_meetings/RFC_review_2014-06-20 [18:04:12] ok, we are talking about https://www.mediawiki.org/wiki/Separating_skins_from_core_MediaWiki in #wikimedia-office [21:04:02] TParis: contact me asap [21:10:07] ugh [21:10:10] who is yuvipanda? [21:10:20] whoever this was logged in to the UTRS server and deleted the entire database [21:11:08] YuviPanda: ^^^ [21:11:10] YuviPanda: wtf [21:12:26] YuviPanda: at Fri, 20 Jun 2014 19:21:12 GMT you issued the following command on the UTRS labs instance: rm -rf mysql/ [21:12:36] YuviPanda: please explain this [21:12:56] UTRS is down as a result and all appeals are lost [21:13:17] BadPanda [21:14:36] if someone knows how to get ahold of him please do so [21:15:12] It's 02:30 in India, so good luck. [21:16:30] ugh [21:16:57] he's idle since 90 minutes. [21:17:08] dammit [21:17:28] * Nemo_bis forgot Yuvi is now following almost-human bio TZ [21:18:44] i'm trying to follow his command history, it looks like he copied /var/lib/mysql around a few different places and copied it back and stopped and started mysql several times [21:18:55] somehow the utrs database is just gone [21:19:08] all database users are gone [21:19:24] SerajewelKS: well, probably because mysql is looking in the wrong place? [21:19:51] i.e. he copied it to $someplace, but did not symlink /var/lib/mysql to /someplace ? [21:19:52] i'm running find, we will see what it produces [21:20:05] he copied it to several home directories [21:20:13] but those copies don't have utrs databases either [21:20:20] no idea what he did [21:20:27] all i know is afterwards, the utrs database doesn't exist [21:26:52] SerajewelKS: i don't know anything about utrs, but http://lists.wikimedia.org/pipermail/labs-l/2014-June/002688.html [21:27:35] -_- thanks [21:27:42] so he was trying to recover it [21:27:54] not sure why tparis didn't contact me first [21:28:25] hope it can be recovered ... [21:28:26] :/ [21:28:39] the mount the datadir is in isn't even mounted [21:28:45] trying to mount it i get a bad superblock error [21:28:54] * SerajewelKS cracks his knuckles [21:29:12] YuviPanda: ^^^ my apologies, saw rm -fr mysql in your .bash_history and freaked out, particularly since i had not been contacted [21:29:35] SerajewelKS: Its hard to get ahold of you these days :P [21:29:36] i would poke andrewbogott or yuvi when they are around [21:29:55] Superblock has an invalid journal (inode 8) [21:30:39] TParis: :( [21:30:40] he did put a copy of /var/lib/mysql in my home dir [21:30:47] yeah [21:30:53] i started digging around [21:31:01] our database is not stored in /var/lib/mysql [21:31:08] so... *wipes brow* [21:31:12] he didn't delete anything important [21:31:24] it's in /mnt/mysql [21:31:29] but the /mnt filesystem is corrupt [21:31:51] i'm looking at my options now. i'm hesitant to let fsck run wild without backing up the volume first. [21:32:36] SerajewelKS: Whatever you do is going to be 100x more informed about the situation than what I can do [21:33:30] appears to be a 20GB volume [21:33:37] backing this up over the internet is likely to be very slow [21:35:03] do we have any labs roots here that could clone the volume in case the fsck makes matters worse? [21:35:22] Seemingly not in here [21:35:39] But we can probably find you someone to do it [21:36:07] would be much appreciated [21:38:00] looks like it was rebooted 21.7ish hours ago and at that point the journal was corrupt [21:38:58] Jun 19 22:28:53 utrs-primary kernel: [7612421.970025] Aborting journal on device vdb. [21:39:03] Jun 19 22:28:55 utrs-primary kernel: [7612423.445422] EXT3-fs (vdb): error: remounting filesystem read-only [21:39:06] yay [21:39:35] Could be easier creating a new vm and copying stuff you need over to it [21:40:43] yay is a good thing or a sarcastic thing? [21:41:04] SerajewelKS: YOu can backup to /data/project (shared NFS storage) [21:41:25] bd808: everyone is cool if i dump a 20GB disk image in there? [21:41:36] We're not stuck for 20GB [21:41:37] ;) [21:41:44] oh 14T available heh [21:42:25] dd is running [21:42:37] TParis: sarcastic ;) [21:42:48] TParis: means that while mounted the filesystem suffered a failure, the kernel remounted it read-only to protect it [21:43:09] well that certainly coincides with the timeline [21:43:11] sucky though [21:43:12] when the box was rebooted 21 ish hours ago it failed to mount at all due to the failure. right now i can't even mount it read-only. [21:43:33] wait, I think I might've done the reboot 21 hours ago [21:43:44] At the time, mysql showed utrs but none of the tables worked [21:43:55] I couldn't query any tables [21:44:10] with a damaged FS that's not unlikely [21:44:15] okay [21:44:19] the question is how damaged is it [21:44:34] which we will find out when this backup completes [21:46:14] i am going to throw up a notice for utrs.wmflabs.org for now [21:47:14] I've updated WP:UTRS/on [21:51:15] TParis: okay the whole site should be down now [21:51:15] SerajewelKS: My plan if we couldn't get this fixed was to restore the Nov 13 backup and then increment the autoincrementers to the nearest logical number from what I recall the last being [21:51:32] So I was going to push the appeal_id to 15000 and the user_id to 150 [21:51:38] yeah i'm going to put a cron job to take nightly backups after this [21:51:44] yeah, that was my plan too [21:51:48] and to delete them after 7 days [21:51:50] there is no reason we shouldn't have more consistent backups [21:52:06] (shrug) Only ever occured to me when it crashed :X [21:52:11] it always does ;) [21:54:08] ahhhhh [21:54:11] * SerajewelKS wipes brow [21:54:30] You've cured cancer? [21:54:38] select count(*) from appeal; --> 11172 [21:54:47] select count(*) from user; --> 103 [21:55:23] those are looking well [21:55:29] comments? [21:55:42] 64087 [21:55:47] :) [21:56:02] i ran a table check and the usermgmtlog table showed as "crashed" [21:56:26] it has 348 rows though, so i guess it was able to recover [21:57:18] just took a backup [21:57:58] that is probably the least important table out of all of them [21:58:02] other than LoggedInusers [21:58:56] site is back up, i can log in [21:59:05] everything looks sane [21:59:17] You're a hero [21:59:57] except for the yelling at random people who tried to help [22:00:28] who was yelling? [22:00:33] i was heh [22:00:48] hehe, well thank you for figuring it out, I was way over my head [22:01:05] np [22:01:11] * SerajewelKS takes some breaths [22:01:45] I thought it was gone for good [22:02:31] i was worried it might be when i saw the filesystem wouldn't mount [22:03:44] Alright, I have to go get ready for work [22:03:51] k, catch you later [22:03:55] thanks everyone else who assisted too [22:27:55] SerajewelKS: I've had "We are the Champions" in my head ever since you fixed it [22:28:09] hah [23:42:05] SerajewelKS: :) [23:44:20] SerajewelKS: glad you could recover.