[00:07:42] <gerrit-wm_>	 New patchset: Lcarr; "Script to change initcwnd to 10 packets see http://wikitech.wikimedia.org/view/TCP_Tuning#initcwnd_10" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2521
[00:08:30] <gerrit-wm_>	 New review: Lcarr; "(no comment)" [operations/puppet] (production); V: 0 C: 2;  - https://gerrit.wikimedia.org/r/2521
[00:08:30] <gerrit-wm_>	 Change merged: Lcarr; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2521
[00:14:13] <drdee>	 LeslieCarr: no they are preparing for celebrating the revolution
[01:09:37] <maplebed>	 whee!!!  db42 ate itself.
[01:09:38] <maplebed>	 http://ganglia.wikimedia.org/latest/?r=day&cs=&ce=&m=&c=MySQL+pmtpa&h=db42.pmtpa.wmnet&tab=m&vn=&metric_group=
[01:09:47] * maplebed  goes to try and smack it.

[01:11:03] <maplebed>	 RobH: you're not around, are you
[01:14:57] <RobH>	 maplebed: kinda whassup?
[01:16:38] * robla  also looks at the email alert about udp2log

[01:17:18] <robla>	 anyone been doing anything on emery?
[01:18:25] <maplebed>	 RobH:  db42 bit the dust and I'm trying to remember where the docs are for its LOM
[01:18:38] <RobH>	 ahhhh, ibm.
[01:18:40] <RobH>	 lemme link
[01:18:52] <maplebed>	 all I need is a powercycle.
[01:18:53] <RobH>	 everything links from platform specific docs
[01:19:01] <RobH>	 oh, power cycle
[01:19:03] <RobH>	 or reset.
[01:19:09] <RobH>	 http://wikitech.wikimedia.org/view/IBM_X3650M3
[01:19:25] <RobH>	 unfortunately, the two IBMs do not appear to accept IPMI commands
[01:19:26] <maplebed>	 thanks.
[01:19:29] <RobH>	 or my nifty script would work
[01:19:38] <RobH>	 maplebed: btw, ipmi_mgmt scirpt is on sockpuppet
[01:19:43] <maplebed>	 the other half of the question, of course, was 'what make is db42?'
[01:19:44] <RobH>	 for the Cseries and such
[01:19:46] <maplebed>	 which you already answered.
[01:20:01] <RobH>	 yea, db42 is an ibm
[01:20:09] <RobH>	 no way to konw without logging in and recognizing it, or pulling up racktables
[01:20:16] <RobH>	 i just recognize the login prompts =P
[01:20:21] <maplebed>	 I'll try reset first.
[01:20:49] <maplebed>	 ok!
[01:20:50] <RobH>	 console 1 starts console, and the escape for serial is ctrol+{ then shift (
[01:20:57] <maplebed>	 well, thanks, db42.
[01:21:11] <RobH>	 i am about until its fixed, i assumed you rather learn the software ;]
[01:21:31] <RobH>	 'learn these two one off servers we wont be buying again!'  ;]
[01:21:34] <maplebed>	 meh.  dario said if it doesn't come back up easily it's cool to wait till monday.
[01:21:45] <RobH>	 oh, and the mgmt required a nifty little one by one inch bit of circuit
[01:21:56] <RobH>	 that is the hardware logic and licensing chip
[01:22:05] <RobH>	 i hate that its called a licensing chip
[01:22:28] <RobH>	 the actual servers are nice though, just annoying since we dont have a bunch
[01:22:32] <RobH>	 but the internals are slick
[01:22:37] <RobH>	 (the case design and the like)
[01:23:59] <maplebed>	 looks like it's booting.
[01:25:05] <maplebed>	 RobH:  back up and able to ssh in.
[01:25:08] <maplebed>	 thanks for the link.
[01:25:26] <RobH>	 sweet
[01:25:29] <RobH>	 glad to help
[01:25:54] <robla>	 Erik Zachte just forwarded me a nagios alert from emery.  "CRITICAL: filters absent: /var/log/squid/filters/sqstat3,".  It was followed pretty quickly by "OK: all filters present".
[01:26:07] <robla>	 I'm assuming transient issue; no big deal
[01:26:33] <robla>	 ...but he was surprised to get the emails, and didn't know what to do with them
[01:26:55] <robla>	 I'm assuming I can poke around in puppet somewhere and see who gets those
[01:27:23] <maplebed>	 there was just a conversation a week or two ago about who should get alert emails from emery and locke...
[01:27:54] <robla>	 I vaguely recall that conversation.  I know Diederik set a bunch of that stuff up
[01:28:04] <robla>	 drdee: you aren't around by any chance, are you?
[01:28:26] <maplebed>	 umm..
[01:28:35] <maplebed>	 what's on disk on emery doesn't match what's in puppet.
[01:28:45] <woosters>	 he is on tech channel i think
[01:29:22] <maplebed>	 it's nimish that's doing it.
[01:29:27] <robla>	 he's logged into this channel, but it's 8:30pm where he is
[01:29:45] <robla>	 is Nimish futzing on emery now?
[01:30:47] <robla>	 maplebed: you at your desk at work?
[01:31:03] <maplebed>	 only he and I are logged in and the timestamp on the config file is 3 minutes ago.
[01:31:18] <maplebed>	 yes, I am at my desk.
[01:31:37] <robla>	 actually, maybe I should call him
[01:31:48] <maplebed>	 well...  there are actually two root shells connected.
[01:31:51] <maplebed>	 so maybe someone else is here.
[01:32:11] <maplebed>	 binasher: arey ou here?
[01:32:12] <robla>	 can't hurt to call and ask
[01:32:18] <maplebed>	 he's holding the other one.
[01:32:58] <RobH>	 I am going to go play videogames and be a goddamn bum.
[01:33:08] <RobH>	 so kinda around if shit breaks.
[01:33:36] <binasher>	 mahey
[01:33:44] <robla>	 just tried calling him...no answer
[01:33:59] <binasher>	 i'm fucking around on emery
[01:34:30] <binasher>	 the monitors.. they work!
[01:34:53] <maplebed>	 ah, puzzle solved!
[01:34:54] <robla>	 binasher!
[01:35:25] <robla>	 watcha doin'?
[01:36:02] <binasher>	 replacing the filter i wrote
[01:36:06] <maplebed>	 hooray for last and ps!
[01:36:24] <binasher>	 maybe
[01:36:34] <robla>	 why are you doing that at 5:30pm on a Friday?
[01:37:30] <binasher>	 because i'm not doing anything fun for another 90 minutes
[01:38:21] <woosters>	 good answer ;-p
[01:38:34] <robla>	 not really :-P
[01:39:04] <robla>	 binasher: does Diederik know what you're up to?
[01:39:39] <maplebed>	 binasher: did you catch the db42 fun?  it crashed ~5hrs ago so I powercycled it.  slaving restarted on its own, so I tihnk maybe it's ok.
[01:39:51] <binasher>	 robla: does deidrek need to know whether or not something i wrote on off time which isn't used by anyone else is running at any given moment?
[01:40:10] <robla>	 yes
[01:40:17] <robla>	 if it's on emery, absolutely
[01:40:35] <robla>	 locke and emery are ridiculously fragile systems
[01:40:53] * robla  digs up rant from a month ago the last time things went sideways there

[01:43:33] <binasher>	 and to think i was just going to replace something that uses an entire core with something way more efficient…
[01:43:37] <binasher>	 nevermind
[01:44:39] <maplebed>	 keep going asher.
[01:44:51] <robla>	 that's really cool if you're making it more efficient.  we just need to know rather than getting surprised by it, especially because it might be that it doesn't work as designed
[01:44:53] <maplebed>	 replacing the that thing is well worth it.
[01:45:05] <woosters>	 robla - there are a few guys i trust on the system …Asher is one of them
[01:45:10] <woosters>	 so we are in good hands
[01:45:15] <maplebed>	 robla: the one binasher is working on takes 3x the resources of any other script, and probably more than all the others combined.
[01:45:44] <robla>	 woosters: he doesn't necessarily know where all of the landmines are with that particular system, and I don't think anyone should be above peer review
[01:48:08] <robla>	 diederik put those monitors in place because the last time we had a problem, we didn't know who it was that screwed it up, and didn't detect it for a very long time
[01:48:21] <binasher>	 peter put those monitors in place and i helped write them..
[01:48:24] <binasher>	 anyways
[01:49:16] <maplebed>	 +1 monitor.  +1 detect "false positives" and keep going.
[01:49:20] <maplebed>	 ::sigh::
[01:52:27] <robla>	 I guess I'm just confused
[01:54:55] <maplebed>	 g'night.
[04:39:39] <gerrit-wm_>	 New review: Demon; "Do this for core2 instead." [test/mediawiki/core] (master); V: 0 C: -2;  - https://gerrit.wikimedia.org/r/2507
[04:39:46] <gerrit-wm_>	 Change abandoned: Demon; "(no reason)" [test/mediawiki/core] (master) - https://gerrit.wikimedia.org/r/2507
[04:40:00] <gerrit-wm_>	 Change abandoned: Demon; "(no reason)" [test/mediawiki/core] (master) - https://gerrit.wikimedia.org/r/2506
[10:55:01] <gerrit-wm_>	 New patchset: Hashar; "adding gitreview config file" [test/mediawiki/core2] (master) - https://gerrit.wikimedia.org/r/2522
[17:08:17] <RobH>	 !log db9 shutting down to move racks, offline during this includes: blogs, bugzilla, racktables, rt, survey, etherpad, observium
[17:08:19] <morebots>	 Logged the message, RobH
[17:08:53] <RobH>	 cmjohnson1: Once db9 turns off, you can move it.
[17:08:58] <cmjohnson1>	 okay
[17:12:53] <RobH>	 heh, everything I use for datacenter work is now offline (rt and racktables)
[17:17:37] <cmjohnson1>	 robh: powered on
[17:18:07] <RobH>	 ok, logging into drac and watching it post
[17:20:26] <RobH>	 282 days without fsck, im letting it run
[17:22:00] <cmjohnson1>	 wow...puppet is supposed to that...right?
[17:22:23] <RobH>	 i mean i tell it to start manually and it fails
[17:23:26] <maplebed>	 anybody else here already poking at the blog?
[17:23:26] <RobH>	 maplebed: you know more aboutmysql than i
[17:23:31] <RobH>	 db9 is down
[17:23:36] <maplebed>	 lame.
[17:23:37] <RobH>	 we moved it, now it wont fire up mysql
[17:23:41] * maplebed  logs in

[17:23:42] <RobH>	 i just pinged asher
[17:23:52] <maplebed>	 so you moved it to a new rack
[17:23:56] <maplebed>	 and the OS came back up
[17:23:58] <maplebed>	 but not mysql?
[17:24:00] <RobH>	 it has a ton of gmond failures for every single mysql metric
[17:24:02] <RobH>	 yep
[17:24:07] <maplebed>	 ok.
[17:24:16] <maplebed>	 I don't think mysql is set to start on boot for most of our DBs
[17:24:20] <RobH>	 i am on the sysetm but if you are poking i wont run anything
[17:24:25] <RobH>	 i try to start it manually, it fails
[17:24:45] <maplebed>	 how'd you try to start it?
[17:24:49] <maplebed>	  /etc/init.d/mysql
[17:24:53] <RobH>	 yea
[17:24:57] <maplebed>	 or /usr/local/mysql/bin/mysqld_safe?
[17:25:09] <RobH>	 didnt try safe yet, you litterally hopped on
[17:25:16] * maplebed  just rat it

[17:25:19] <maplebed>	 looking at logs now
[17:25:38] <RobH>	 huh, its up now
[17:25:44] <RobH>	 so hgave to safe start it, interesting
[17:25:58] <apergos>	 fb version, yeah
[17:26:05] <maplebed>	 error log looks fine
[17:26:09] <RobH>	 !log db9 moved, all systems online
[17:26:11] <morebots>	 Logged the message, RobH
[17:26:17] <apergos>	 yes, I was just looking at it (the log)
[17:26:24] <RobH>	 slow, but pulling up all servies to check
[17:26:54] <maplebed>	 yeah, it's very misleading that we have /etc/init.d/mysqld and that it doesn't work.
[17:27:08] <RobH>	 so we are running the FB version on all dbs
[17:27:09] <RobH>	 ?
[17:27:17] <RobH>	 so i need to safestart everthing ?
[17:27:21] <maplebed>	 I think so.
[17:27:26] <maplebed>	 but there's probably an exception
[17:27:31] <apergos>	 I'm sure there is
[17:27:33] <RobH>	 well good to know.
[17:27:40] <apergos>	 "if it's already running don't start by hand" :-P
[17:28:05] <RobH>	 !log manual test of each affected service complete, db9 fully online.
[17:28:07] <morebots>	 Logged the message, RobH
[17:28:10] <maplebed>	 maybe 'if it doesn't start on boot, start the fb version' would be a good heuristic...
[17:28:23] <maplebed>	 RobH: you actually have a list of all those services?!
[17:28:54] <RobH>	 well i have the database list and my knowledge of which are actually used
[17:29:32] <maplebed>	 gotcha
[17:33:50] <gerrit-wm_>	 New review: Demon; "(no comment)" [test/mediawiki/core2] (master); V: 1 C: 2;  - https://gerrit.wikimedia.org/r/2522
[17:33:50] <gerrit-wm_>	 Change merged: Demon; [test/mediawiki/core2] (master) - https://gerrit.wikimedia.org/r/2522
[17:34:58] <maplebed>	 cya
[17:36:45] <gerrit-wm_>	 New review: Demon; "Rather than creating a new commit to revert the old one, you should abandon the original change so i..." [analytics/reportcard] (master) C: 0;  - https://gerrit.wikimedia.org/r/2419
[20:31:13] <apergos>	 !log restarted lightty on dataset2
[20:31:15] <morebots>	 Logged the message, Master
[20:32:53] <RobH>	 apergos: watchmouse page?
[20:33:03] <RobH>	 i just got it, i assume you just fixeD?
[20:33:04] <apergos>	 yeah, that's why I restarted it
[20:33:06] <apergos>	 uh huh
[20:33:37] <apergos>	 once every couple weeks it needs a kick in the bum
[20:33:45] <RobH>	 when i get a watchmouse page it plays the mario game over tone.
[20:33:49] <apergos>	 hahaha
[20:34:42] <RobH>	 oh well, going back afk since its fixed ttyl
[20:34:49] <apergos>	 enjoy
[23:51:15] <gerrit-wm_>	 New patchset: Asher; "frontend cache tier udplog data for graphite" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2523
[23:51:47] <gerrit-wm_>	 New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/2523