[00:08:11] <icinga-wm>	 PROBLEM - RAID on searchidx1001 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds.  
[00:09:11] <icinga-wm>	 RECOVERY - RAID on searchidx1001 is OK: OK: optimal, 1 logical, 4 physical  
[01:20:37] <grrrit-wm>	 (03PS1) 10Jeremyb: brwikimedia: fix import sources [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/103730 
[01:20:42] <grrrit-wm>	 (03PS1) 10Jeremyb: fix import sources for all chapter wikis [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/103731 
[01:20:43] <grrrit-wm>	 (03PS1) 10Jeremyb: import sources: move chapter wikis to own section [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/103732 
[01:23:08] <grrrit-wm>	 (03CR) 10Nemo bis: [C: 031] "We usually use the one-letter prefixes IIRC (see below), but there are both (see above), so who cares" [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/103730 (owner: 10Jeremyb)
[01:25:59] <grrrit-wm>	 (03CR) 10Nemo bis: fix import sources for all chapter wikis (032 comments) [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/103731 (owner: 10Jeremyb)
[01:26:22] <grrrit-wm>	 (03CR) 10Nemo bis: fix import sources for all chapter wikis (031 comment) [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/103731 (owner: 10Jeremyb)
[01:26:59] <grrrit-wm>	 (03CR) 10Jeremyb: "Nemo says:" [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/103730 (owner: 10Jeremyb)
[01:27:03] <grrrit-wm>	 (03CR) 10Nemo bis: "no opinion" [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/103732 (owner: 10Jeremyb)
[01:30:26] <grrrit-wm>	 (03CR) 10Jeremyb: fix import sources for all chapter wikis (032 comments) [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/103731 (owner: 10Jeremyb)
[01:31:43] <grrrit-wm>	 (03CR) 10Jeremyb: fix import sources for all chapter wikis (031 comment) [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/103731 (owner: 10Jeremyb)
[01:39:31] <icinga-wm>	 PROBLEM - Puppet freshness on virt1007 is CRITICAL: Last successful Puppet run was Tue 24 Dec 2013 07:29:50 PM UTC  
[02:10:16] <logmsgbot>	 !log LocalisationUpdate completed (1.23wmf7) at Thu Dec 26 02:10:16 UTC 2013
[02:10:35] <morebots>	 Logged the message, Master
[02:17:31] <icinga-wm>	 PROBLEM - Puppet freshness on db9 is CRITICAL: Last successful Puppet run was Wed 25 Dec 2013 05:14:45 PM UTC  
[02:18:37] <logmsgbot>	 !log LocalisationUpdate completed (1.23wmf8) at Thu Dec 26 02:18:37 UTC 2013
[02:18:54] <morebots>	 Logged the message, Master
[02:35:43] <logmsgbot>	 !log LocalisationUpdate ResourceLoader cache refresh completed at Thu Dec 26 02:35:43 UTC 2013
[02:35:57] <morebots>	 Logged the message, Master
[04:40:31] <icinga-wm>	 PROBLEM - Puppet freshness on virt1007 is CRITICAL: Last successful Puppet run was Tue 24 Dec 2013 07:29:50 PM UTC  
[05:18:31] <icinga-wm>	 PROBLEM - Puppet freshness on db9 is CRITICAL: Last successful Puppet run was Wed 25 Dec 2013 05:14:45 PM UTC  
[07:41:31] <icinga-wm>	 PROBLEM - Puppet freshness on virt1007 is CRITICAL: Last successful Puppet run was Tue 24 Dec 2013 07:29:50 PM UTC  
[08:17:32] <grrrit-wm>	 (03CR) 10TTO: [C: 04-1] import sources: move chapter wikis to own section (031 comment) [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/103732 (owner: 10Jeremyb)
[08:19:31] <icinga-wm>	 PROBLEM - Puppet freshness on db9 is CRITICAL: Last successful Puppet run was Wed 25 Dec 2013 05:14:45 PM UTC  
[08:50:32] <grrrit-wm>	 (03CR) 10Chad: [C: 04-1] "I wonder if we can kill sillyshell entirely while we're here, rather than copying it over to the module." (037 comments) [operations/puppet] - 10https://gerrit.wikimedia.org/r/100760 (owner: 10Matanya)
[10:32:58] <grrrit-wm>	 (03CR) 10Nemo bis: [C: 04-1] Disable interwiki magic for wikimedia (chapter) sites [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/103649 (owner: 10TTO)
[10:39:09] <grrrit-wm>	 (03PS2) 10Nemo bis: fix import sources for all chapter wikis [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/103731 (owner: 10Jeremyb)
[10:41:12] <grrrit-wm>	 (03CR) 10Nemo bis: fix import sources for all chapter wikis (032 comments) [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/103731 (owner: 10Jeremyb)
[10:42:31] <icinga-wm>	 PROBLEM - Puppet freshness on virt1007 is CRITICAL: Last successful Puppet run was Tue 24 Dec 2013 07:29:50 PM UTC  
[11:20:31] <icinga-wm>	 PROBLEM - Puppet freshness on db9 is CRITICAL: Last successful Puppet run was Wed 25 Dec 2013 05:14:45 PM UTC  
[11:43:02] <grrrit-wm>	 (03CR) 10Nemo bis: "Thanks for actually explaining on the bug now. :)" [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/103649 (owner: 10TTO)
[11:49:03] <grrrit-wm>	 (03CR) 10Odder: [C: 031] "It is! Well done, thanks Mateusz!" [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/99768 (owner: 10M4tx)
[13:43:31] <icinga-wm>	 PROBLEM - Puppet freshness on virt1007 is CRITICAL: Last successful Puppet run was Tue 24 Dec 2013 07:29:50 PM UTC  
[14:21:31] <icinga-wm>	 PROBLEM - Puppet freshness on db9 is CRITICAL: Last successful Puppet run was Wed 25 Dec 2013 05:14:45 PM UTC  
[14:29:01] <icinga-wm>	 RECOVERY - Puppet freshness on db9 is OK: puppet ran at Thu Dec 26 14:28:55 UTC 2013  
[14:29:37] <akosiaris>	 !log killed stuck puppet agent on db9 and forced manual puppet run.
[14:29:55] <morebots>	 Logged the message, Master
[14:30:34] <matanya>	 hi akosiaris, no vacation?
[14:31:24] <akosiaris>	 matanya: hey. Just lower productivity
[15:38:32] <icinga-wm>	 PROBLEM - Host virt1007 is DOWN: PING CRITICAL - Packet loss = 100%  
[15:43:41] <icinga-wm>	 RECOVERY - Host virt1007 is UP: PING OK - Packet loss = 0%, RTA = 0.32 ms  
[16:05:14] <andrewbogott>	 cmjohnson1: working today?
[16:05:34] <cmjohnson1>	 yes...but not going to DC till 2
[16:05:53] <cmjohnson1>	 what's up
[16:06:23] <andrewbogott>	 Two of those virt boxes are giving me trouble, and I don't really know how to diagnose.
[16:06:43] <cmjohnson1>	 what kind of trouble?
[16:06:51] <andrewbogott>	 virt1007… partman fails because it can't find
[16:07:09] <andrewbogott>	 well, it maybe can't find /any/ drives, I can't tell.  It has "primary	unknown/dev/sdh1"  many different volumes
[16:07:30] <andrewbogott>	 I can copy/paste a bunch of the partman output if that's useful...
[16:08:06] <cmjohnson1>	 okay
[16:08:26] <cmjohnson1>	 the Cisco servers are very temperamental.   
[16:08:58] <andrewbogott>	 here's a snip of partman output:  https://dpaste.de/5XeY
[16:09:01] <andrewbogott>	 there's a lot more like that :)
[16:09:16] <cmjohnson1>	 what boot cfg are you using?
[16:10:07] <andrewbogott>	 virt-raid10-cisco.cfg
[16:10:29] <andrewbogott>	 which worked for seven of the nine.
[16:10:48] <cmjohnson1>	 fyi: these were the same issues I ran into with an1007
[16:11:08] <andrewbogott>	 This is a different box, right?
[16:11:37] <andrewbogott>	 Seems like if it can't see any drives then probably a controller card is fried or just needs re-seating or something?
[16:11:40] * andrewbogott  takes a wild guess

[16:14:41] <cmjohnson1>	 I believe it has something to do with the controller. I do have a spare on-site so I can swap out.
[16:15:01] <cmjohnson1>	 and see if that works.  I need to view the settings first
[16:15:20] <cmjohnson1>	 cisco support is really sketchy for these since they were donations
[16:15:34] * andrewbogott  nods

[16:15:56] <cmjohnson1>	 I am going to DC at 2...to swap out main board for an1012...I will look at it then
[16:16:54] <cmjohnson1>	 andrebogott: which two? 
[16:17:13] <cmjohnson1>	 virt1007 and ?
[16:17:56] <andrewbogott>	 1006 is doing a different thing
[16:18:05] <andrewbogott>	 of course :)
[16:18:14] <andrewbogott>	 When it boots, it gets to the point of requesting an IP
[16:18:28] <andrewbogott>	 but then it just hangs there.  Brewster offers and offers but virt1006 hangs
[16:18:48] <andrewbogott>	 I'm hopeful that that's some kind of networking thing rather than a hardware failure, but… no idea really.
[16:19:12] <andrewbogott>	 LeslieCarr observed the same behavior on Tuesday and didn't have any idea either.
[16:19:47] <cmjohnson1>	 i will poke at it 
[16:20:32] <andrewbogott>	 thanks
[16:22:02] <matanya>	 andrewbogott: did you try to run the client in debug mode?
[16:22:29] <andrewbogott>	 matanya: I'm not sure I know what that means :)
[16:25:25] <matanya>	 andrewbogott: are you using dhclient?
[16:26:14] <andrewbogott>	 Oh, um… this is pxe boot on a new server.  So, no proper OS at all… I wouldn't know how to do things interactively.
[16:28:41] <matanya>	 andrewbogott: want me to through some ideas or you ar good? i have 2 or 3
[16:29:24] <andrewbogott>	 I think if cmjohnson1 is willing to look I'm happy to leave it to him.  Most likely this box will land on the scrap heap :(
[16:30:07] <matanya>	 andrewbogott: sure
[16:30:26] <cmjohnson1>	 matanya andrewbogott: feel free to poke around
[16:30:36] <cmjohnson1>	 I will not get to it till later today. 
[16:30:51] <cmjohnson1>	 let me know if you find anything
[16:44:31] <icinga-wm>	 PROBLEM - Puppet freshness on virt1007 is CRITICAL: Last successful Puppet run was Tue 24 Dec 2013 07:29:50 PM UTC  
[17:05:22] <grrrit-wm>	 (03PS1) 10Ottomata: Adding jmxtrans classes for datanode and nodemanager [operations/puppet/cdh4] - 10https://gerrit.wikimedia.org/r/103755 
[17:06:34] <grrrit-wm>	 (03PS2) 10Ottomata: Adding jmxtrans classes for datanode and nodemanager [operations/puppet/cdh4] - 10https://gerrit.wikimedia.org/r/103755 
[17:06:49] <grrrit-wm>	 (03CR) 10Ottomata: [C: 032 V: 032] Adding jmxtrans classes for datanode and nodemanager [operations/puppet/cdh4] - 10https://gerrit.wikimedia.org/r/103755 (owner: 10Ottomata)
[17:19:10] <grrrit-wm>	 (03PS1) 10Aaron Schulz: Made CDB scripts less verbose [operations/puppet] - 10https://gerrit.wikimedia.org/r/103759 
[17:30:40] <grrrit-wm>	 (03CR) 10Chad: [C: 031] Made CDB scripts less verbose [operations/puppet] - 10https://gerrit.wikimedia.org/r/103759 (owner: 10Aaron Schulz)
[18:00:16] <grrrit-wm>	 (03CR) 10Dekel E: [C: 031] Enable $wgImportSources for Hebrew Wikivoyage [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/103118 (owner: 10Odder)
[18:27:03] <grrrit-wm>	 (03CR) 10Ori.livneh: [C: 032] Made CDB scripts less verbose [operations/puppet] - 10https://gerrit.wikimedia.org/r/103759 (owner: 10Aaron Schulz)
[18:45:04] <grrrit-wm>	 (03CR) 10Jdlrobson: [C: 031] "Merge at will! :)" [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/103611 (owner: 10Kaldari)
[18:52:43] <grrrit-wm>	 (03PS1) 10BryanDavis: Revert "Configure Varnish not to cache scholarship app reqs" [operations/puppet] - 10https://gerrit.wikimedia.org/r/103768 
[18:53:06] <grrrit-wm>	 (03CR) 10jenkins-bot: [V: 04-1] Revert "Configure Varnish not to cache scholarship app reqs" [operations/puppet] - 10https://gerrit.wikimedia.org/r/103768 (owner: 10BryanDavis)
[18:55:17] <grrrit-wm>	 (03PS2) 10BryanDavis: Revert "Configure Varnish not to cache scholarship app reqs" [operations/puppet] - 10https://gerrit.wikimedia.org/r/103768 
[18:56:06] <grrrit-wm>	 (03CR) 10BryanDavis: "Patch set 2 was a manual rebase." [operations/puppet] - 10https://gerrit.wikimedia.org/r/103768 (owner: 10BryanDavis)
[19:04:37] <grrrit-wm>	 (03PS4) 10Andrew Bogott: Added 'adminadd' tool to auto-generate new user entries. [operations/puppet] - 10https://gerrit.wikimedia.org/r/98700 
[19:06:29] <grrrit-wm>	 (03PS1) 10Ottomata: Adding jmxtrans README and convenience worker and master classes [operations/puppet/cdh4] - 10https://gerrit.wikimedia.org/r/103769 
[19:06:51] <grrrit-wm>	 (03PS5) 10Andrew Bogott: Added 'adminadd' tool to auto-generate new user entries. [operations/puppet] - 10https://gerrit.wikimedia.org/r/98700 
[19:06:56] <grrrit-wm>	 (03CR) 10Ottomata: [C: 032 V: 032] Adding jmxtrans README and convenience worker and master classes [operations/puppet/cdh4] - 10https://gerrit.wikimedia.org/r/103769 (owner: 10Ottomata)
[19:09:33] <grrrit-wm>	 (03PS1) 10Aaron Schulz: Prevent two people from running scap at the same time [operations/puppet] - 10https://gerrit.wikimedia.org/r/103770 
[19:10:08] <grrrit-wm>	 (03PS6) 10Andrew Bogott: Added 'adminadd' tool to auto-generate new user entries. [operations/puppet] - 10https://gerrit.wikimedia.org/r/98700 
[19:10:34] <Aaron|home>	 ori: those 3 lines work for me in a script in ~ on tin
[19:14:57] <grrrit-wm>	 (03PS1) 10Andrew Bogott: Add Katie Filbert (Aude) to mortals. [operations/puppet] - 10https://gerrit.wikimedia.org/r/103771 
[19:14:59] <grrrit-wm>	 (03PS1) 10Ottomata: Changing default run_interval to 15 and log_level to info. [operations/puppet/jmxtrans] - 10https://gerrit.wikimedia.org/r/103772 
[19:15:12] <grrrit-wm>	 (03CR) 10Ottomata: [C: 032 V: 032] Changing default run_interval to 15 and log_level to info. [operations/puppet/jmxtrans] - 10https://gerrit.wikimedia.org/r/103772 (owner: 10Ottomata)
[19:16:54] <grrrit-wm>	 (03PS1) 10Ottomata: Now using jmxtrans instead of built in ganglia support for Hadoop [operations/puppet] - 10https://gerrit.wikimedia.org/r/103773 
[19:17:30] <grrrit-wm>	 (03CR) 10Ottomata: [C: 032 V: 032] Now using jmxtrans instead of built in ganglia support for Hadoop [operations/puppet] - 10https://gerrit.wikimedia.org/r/103773 (owner: 10Ottomata)
[19:30:31] <cmjohnson1>	 andrewbogott: going to try and swap nic card out on virt1006 with the spare cisco I received from tampa. it's not the cable, connected to right port, the config looks right and vlan is correct
[19:30:53] <andrewbogott>	 cmjohnson1: ok!  Let me know when it's swapped and I'll try another pxe boot
[19:33:50] <grrrit-wm>	 (03PS2) 10Aude: Add Katie Filbert (Aude) to mortals. [operations/puppet] - 10https://gerrit.wikimedia.org/r/103771 (owner: 10Andrew Bogott)
[19:33:52] <grrrit-wm>	 (03CR) 10Ori.livneh: [C: 031] Add Katie Filbert (Aude) to mortals. [operations/puppet] - 10https://gerrit.wikimedia.org/r/103771 (owner: 10Andrew Bogott)
[19:36:36] <grrrit-wm>	 (03CR) 10Andrew Bogott: [C: 032] Add Katie Filbert (Aude) to mortals. [operations/puppet] - 10https://gerrit.wikimedia.org/r/103771 (owner: 10Andrew Bogott)
[19:37:01] <ori>	 \o/
[19:37:35] <andrewbogott>	 ori, can you brief aude about what's involved in actually having the 'mortal' right?
[19:37:58] <andrewbogott>	 Specifically, which things she should or shouldn't do without checking in with other folks first?  It's a pretty big hammer.
[19:38:16] <aude>	 ok :)
[19:38:23] <ori>	 1) always blame Reedy
[19:38:24] <^demon|away>	 Nobody gave me that talk!
[19:38:29] <andrewbogott>	 ori:  or if you think she should be in a different group…
[19:38:34] <^demon|away>	 It was more like "here you go, enjoy"
[19:38:50] <ori>	 andrewbogott: no, it's appropriate
[19:39:04] * aude  will just look at stuff like flourine at least to start

[19:39:31] <aude>	 maybe ori and reedy can explain more when i come to sf in january
[19:39:49] <andrewbogott>	 aude: I think the main point (which probably goes without saying) is -- you can now deploy new code, but please don't actually do that without scheduling a window and coordinating with people and such.
[19:40:01] <aude>	 obviously
[19:40:20] <andrewbogott>	 yeah, obvious, but I still feel like I should say it out loud :)
[19:40:28] <ori>	 aude: the joking-aside bits are: don't use the prod key for anything other than prod; don't leave your computer unattended / unlocked; if you lose your computer, get in touch with ops asap and report it so that the key can be revoked
[19:40:28] <aude>	 yep
[19:40:40] * aude  nods

[19:41:57] <ori>	 don't download random stuff from the internet onto production boxes, etc. do you have a good ~/.ssh/config ?
[19:42:12] <aude>	 yep
[19:42:49] * ori  invites others to chime in, if i'm missing something.

[19:42:55] <ori>	 aude: have you tried logging in yet?
[19:42:58] <aude>	 not yet
[19:43:27] <andrewbogott>	 I was thinking, we should write all this down… then I remembered that we sort of did.  A quick read:  https://wikitech.wikimedia.org/wiki/Server_access_responsibilities
[19:45:31] <icinga-wm>	 PROBLEM - Puppet freshness on virt1007 is CRITICAL: Last successful Puppet run was Tue 24 Dec 2013 07:29:50 PM UTC  
[19:49:01] <matanya>	 did i miss something important ori ?
[19:49:17] <aude>	 ori: andrewbogott it works
[19:49:35] <andrewbogott>	 cool!
[19:50:28] <aude>	 probably takes time for puppet to run everywhere
[19:50:36] <ori>	 matanya: hm?
[19:50:47] <andrewbogott>	 aude, if you're waiting on a particular box I can give it a push
[19:51:06] <ori>	 fluorine
[19:51:25] <aude>	 no hurry
[19:51:27] * aude  can wait

[19:51:49] <ori>	 i'm really glad this worked out
[19:52:08] <andrewbogott>	 Oh, I've been spelling fluorine wrong
[19:52:09] <aude>	 thanks ori
[19:52:11] <ori>	 it makes me feel good about the wmf
[19:52:28] <aude>	 can bug ree-dy less now
[19:53:37] <ori>	 aude: there was a lot of work by quim, mutante, and robla to work out the requirements -- you should thank them, not me :) and andrewbogott for doing the deed
[19:53:52] <aude>	 :)
[19:54:10] <andrewbogott>	 aude, does fluorine like you now?
[19:54:16] <matanya>	 i saw :<ori> don't download random stuff from the internet onto production boxes, etc. do you have a good ~/.ssh/config ?
[19:54:29] <matanya>	 that looked like something important
[19:54:41] <ori>	 matanya: http://ur1.ca/edq22 (in topic)
[19:55:26] <manybubbles>	 broken: https://gdash.wikimedia.org/dashboards/jobq/
[19:55:28] <aude>	 just ssh fluorine ?
[19:55:43] <andrewbogott>	 aude, depends on how you're doing it.
[19:55:54] <andrewbogott>	 Are you using proxycommand, or forwarding via bast1001, or...?
[19:56:02] <aude>	 bast1001
[19:56:28] <andrewbogott>	 hm… yep, you have a home account there.
[19:56:51] <andrewbogott>	 on fluorine I mean
[19:57:05] <ottomata>	 andrewbogott: have you had the pleasure of trying to get new nrpe checks to show up in icinga before?
[19:57:08] <ottomata>	 and
[19:57:10] <ottomata>	 if I restart icinga
[19:57:21] <ottomata>	 will everyone get text messages on their holiday vacas?
[19:57:22] <ottomata>	 :p
[19:57:23] <andrewbogott>	 aude, I'd advise you to set things up with proxy command like this, though:  https://wikitech.wikimedia.org/wiki/Server_access_responsibilities#SSH
[19:57:37] <andrewbogott>	 I was dragging my feet about that but just got things put together last week and it is /way/ better than key forwarding.
[19:58:01] <aude>	 yeah, that's what i have
[19:58:29] <andrewbogott>	 Ah, so… then the ssh command kind of depends on what you have in your config.
[19:58:56] <aude>	 -A is for if i have to make changes for gerrit (unlikely)
[19:59:12] <andrewbogott>	 ottomata: I haven't done much with icinga.  I don't think it'll page people, but… you should probably ask someone who knows before trying :)
[19:59:22] <andrewbogott>	 aude: yep, sounds like you're doing things right then
[19:59:59] <ottomata>	 yeahhhh
[20:00:05] <ottomata>	 maybe i'll wait on figuring that out :p
[20:00:34] <ori>	 AaronSchulz: cool, i didn't know about using exec o create file handles
[20:00:52] <ori>	 *to
[20:01:06] <andrewbogott>	 aude, so, not working?
[20:04:34] <aude>	 not yet
[20:06:01] <andrewbogott>	 I see a fair number of access requests on fluorine that are maybe you… they're all root@ though
[20:06:08] <andrewbogott>	 you should be able to log in as yourself, aude@
[20:06:27] <grrrit-wm>	 (03CR) 10Ori.livneh: [C: 04-1] "The error message that is printed when scap cannot acquire the lock should mention the lock file name, so you don't have to fish around th" [operations/puppet] - 10https://gerrit.wikimedia.org/r/103770 (owner: 10Aaron Schulz)
[20:07:09] <subbu>	 greg-g, ori, andrewbogott I have to push config changes to Parsoid  (config broke last week when we removed all "default" entries from our config file, but that broke VE on a few wikis which I got an email about today) .. so, given that this is code freeze week, who do I co-ordinate this with?
[20:07:21] <ori>	 subbu: greg-g
[20:08:02] <ori>	 subbu: will it require purging varnish?
[20:08:06] <subbu>	 ori, do you know if he is working / online today.
[20:08:08] <subbu>	 ori, no
[20:08:11] <aude>	 i do ssh -i (mykey) aude@bast1001.wikimedia.org
[20:08:21] <aude>	 -i not needed probably
[20:08:39] <andrewbogott>	 Oh, ok, lemme look at the logs on bast1001
[20:08:43] <andrewbogott>	 (I was on fluorine before)
[20:09:07] <subbu>	 ori, but, we might require someone to restart cluster in case the restart command doesn't work .. ryan was finetuning and testing that and not sure if it will work.
[20:09:09] <aude>	 ok
[20:09:25] <andrewbogott>	 whoah, puppet has a bad cert on bast1001.   What the heck?
[20:09:43] <andrewbogott>	 This would explain your troubles, but may take me a few minutes to sort out.
[20:09:43] <aude>	 -i is indeed not needed but i think is better/faster
[20:09:55] <aude>	 ok
[20:10:12] <aude>	 for labs, i just do ssh tools-dev from bastion
[20:10:16] <aude>	 or whatnot
[20:10:41] <ori>	 subbu: greg-g should be around today, yeah. if VE is actually broken on some wikis i can't imagine he'd say no. let's wait for him to answer, but in the meantime i think you can prep.
[20:11:01] <subbu>	 ori, will do, thanks.
[20:12:25] <Aaron|home>	 ori: I don't see the use of --force
[20:13:38] <grrrit-wm>	 (03PS2) 10Aaron Schulz: Prevent two people from running scap at the same time [operations/puppet] - 10https://gerrit.wikimedia.org/r/103770 
[20:13:48] <Aaron|home>	 seems like it's asking for trouble
[20:14:33] <ori>	 Aaron|home: OK, printing the lock filename is probably sufficient, in the unlikely scenario that someone needs to urgently scap and bypass some lingering lock
[20:14:37] <Aaron|home>	 that would make sense with a "this is my deploy window lock", but not with "don't clobber data" lock
[20:15:11] <Aaron|home>	 and cntrl-c on scap kills the lock anyway
[20:15:14] <ori>	 Aaron|home: OK, I am persuaded.
[20:15:20] <Aaron|home>	 it would be pretty rare to mess that up
[20:15:20] <Aaron|home>	 ok
[20:15:53] <ori>	 waiting for jenkins
[20:20:07] * aude  is spelling challenged

[20:20:29] <andrewbogott>	 aude, try bast1001 now?
[20:20:39] <aude>	 now i just get "Permission denied (publickey)."
[20:20:45] <aude>	 i can try logout/in again
[20:21:15] <aude>	 no luck
[20:21:19] <grrrit-wm>	 (03CR) 10Ori.livneh: [C: 032 V: 032] Prevent two people from running scap at the same time [operations/puppet] - 10https://gerrit.wikimedia.org/r/103770 (owner: 10Aaron Schulz)
[20:21:44] <andrewbogott>	 Hm...
[20:22:08] <andrewbogott>	 Failed publickey for aude from <redacted>
[20:22:40] <aude>	 no idea what i have to do different or what
[20:22:57] <andrewbogott>	 aude, sometime recently a user solved this problem by restarting their terminal program… something about ssh and the shell caching a bad staet.
[20:23:04] <andrewbogott>	 I would think that -i would address that in any case though...
[20:23:07] <Aaron|home>	  Ryan_Lane: curious, do you use -t with ssh-add ?
[20:23:33] <Ryan_Lane>	 is that the option that asks for permission every time?
[20:23:41] <Aaron|home>	 no, that's -d or something
[20:23:43] <Aaron|home>	 -t is ttl
[20:24:11] <Ryan_Lane>	 nope
[20:24:16] <paravoid>	 -c is confirm
[20:24:39] <Aaron|home>	 Ryan_Lane: it's kind of cool...but does not get along with scap :)
[20:25:40] <Ryan_Lane>	 why the need for ttl?
[20:25:58] <Ryan_Lane>	 stop using scap. problem solved! :D
[20:26:20] <Aaron|home>	 Ryan_Lane: in case one forgets ssh-add -D before leaving
[20:26:56] <Ryan_Lane>	 ah. good reason
[20:27:10] <Aaron|home>	 but if it expires mid-scap it's kind of funny
[20:28:01] <paravoid>	 if you want to do a simple incremental update to scap, just make a separate account with a separate key
[20:28:10] <paravoid>	 and have tin have that private key and be done with it
[20:28:33] <paravoid>	 <- ssh key forwarding hater
[20:29:52] <Ryan_Lane>	 I also hate ssh forwarding
[20:30:23] <Ryan_Lane>	 but I'm a salt proponent
[20:35:42] * Aaron|home  wonders what ori thinks of that

[20:41:08] <greg-g>	 subbu: heya, yeah, please fix :)
[20:41:22] <subbu>	 ok, thanks :-)
[20:41:24] <greg-g>	 sorry, wifi at coffee shop wasn't working, and they were busy as heck, so just had to walk back home
[20:41:57] <ori>	 subbu / greg-g: i think andrewbogott is available to help with the restarting
[20:42:10] <ori>	 subbu: so you can just coordinate with him, since you got the green light from greg-g
[20:42:39] <subbu>	 ori, ok, I'm going to deploy now then .. ok, thanks. 
[20:43:05] <subbu>	 I see that Ryan_Lane is also around in case service-restart doesn't do the trick.
[20:43:17] <Ryan_Lane>	 ish. I'm a volunteer now
[20:43:43] <subbu>	 right. ok.
[20:43:45] <greg-g>	 "don't wanna do it" carries more weight now
[20:43:48] <greg-g>	 ;)
[20:44:10] <subbu>	 andrewbogott, you around to help with parsoid cluster restart, if necessary?
[20:44:24] <andrewbogott>	 I'm around but only if you spoonfeed me the actual commands to run
[20:45:09] <subbu>	 hmm .. ok, let me look up my notes.
[20:45:33] <Ryan_Lane>	 greg-g: well, if I'm around and not billing for other work I'd probably do it :)
[20:45:44] <Ryan_Lane>	 I'm supposed to be billing for labs migration work right now
[20:46:35] <subbu>	 andrewbogott, ok have them. ok, will sync config changes in just a bit.
[20:48:09] * Aaron|home  tries scap now

[20:48:25] <ori>	 Aaron|home: probably better to wait until parsoid deploy
[20:48:29] <ori>	 until after, I mean
[20:48:44] * Aaron|home  cancels

[20:48:48] <Aaron|home>	 meh, ok
[20:49:26] * Aaron|home  didn't see anything on https://wikitech.wikimedia.org/wiki/Deployments

[20:49:57] <subbu>	 !log updated localsettings.js to unbreak VE on wikis with "-" in their name (ex: nds-nl)
[20:50:06] <ori-away>	 Aaron|home: it's an emergency deployment; coordination happened in this channel over the last 30 mins or so
[20:50:14] <morebots>	 Logged the message, Master
[20:50:56] <greg-g>	 oh right, pushbot...
[20:51:13] <subbu>	 Ryan_Lane, FYI about service-restart. https://gist.github.com/subbuss/c8b6e7bc00ce36caf156
[20:51:24] <subbu>	 andrewbogott, so, you'll have to restart cluster.
[20:51:29] <Ryan_Lane>	 not surprising
[20:51:33] <andrewbogott>	 that was fast!
[20:51:35] <andrewbogott>	 OK...
[20:51:43] <Ryan_Lane>	 related to the init script
[20:51:49] <Ryan_Lane>	 you guys really need to switch to the upstart
[20:52:03] <subbu>	 this is what I have in my notes: using salt:  salt -b 10% -G 'deployment_target:parsoid' parsoid.restart_parsoid parsoid
[20:52:06] <subbu>	 Ryan_Lane, yes, we haven't pushed the updated code yet.
[20:52:09] <andrewbogott>	 yeps
[20:52:20] <subbu>	 we have migrated to upstart and deployed in rt testing.
[20:52:24] * greg-g  took too long to find the quote, but here it is anyways:

[20:52:25] <greg-g>	 13:28 <       ori> AaronSchulz: do the scripts need some sort of concurrency control?
[20:52:28] <greg-g>	 yep
[20:52:29] <greg-g>	 ;)
[20:52:32] <andrewbogott>	 !log restarting parsoid cluster via salt
[20:52:47] <morebots>	 Logged the message, Master
[20:52:58] <Ryan_Lane>	 greg-g: which scripts?
[20:53:08] <greg-g>	 Ryan_Lane: what Aaron|home is working on
[20:53:21] <Ryan_Lane>	 ah, ok
[20:53:44] <andrewbogott>	 subbu: how's that?
[20:54:03] <subbu>	 looks like that did it.
[20:54:14] <subbu>	 thanks. now to verify the fix does what it is supposed to.
[20:54:19] <subbu>	 andrewbogott, thanks
[20:54:30] <greg-g>	 well, look at that: https://gerrit.wikimedia.org/r/#/c/103770/ "Prevent two people from running scap at the same time
[20:54:33] <greg-g>	 "
[20:55:45] <subbu>	 Aaron|home, greg-g .. deployment done.
[21:02:03] <grrrit-wm>	 (03PS1) 10Aaron Schulz: Use LOCK_EX for making the CDB MD5 file for sanity [operations/puppet] - 10https://gerrit.wikimedia.org/r/103846 
[21:03:05] <greg-g>	 sweet
[21:03:17] <logmsgbot>	 !log aaron started scap
[21:03:35] <morebots>	 Logged the message, Master
[21:07:37] <MaxSem>	 !log Recreating GeoData index
[21:07:55] <morebots>	 Logged the message, Master
[21:08:08] <andrewbogott>	 subbu: you're done with me, right?
[21:08:30] <subbu>	 andrewbogott, yes.
[21:08:47] <subbu>	 btw, are you in mpls or out somewhere?
[21:09:51] <andrewbogott>	 mpls this week :)
[21:10:41] <subbu>	 and escaping somewhere after that? its been too fricking cold here .. anyway, we should find halfak and catch up again sometime.
[21:10:47] <andrewbogott>	 I was in FL last week
[21:10:58] <andrewbogott>	 going to singapore in mid-January.
[21:11:07] <andrewbogott>	 But yeah, I'll be around for a few weeks so we can meet up.
[21:11:09] <subbu>	 oh my, ok :)
[21:14:20] <grrrit-wm>	 (03PS1) 10Ottomata: Removing disabled views [operations/puppet] - 10https://gerrit.wikimedia.org/r/103847 
[21:14:34] <grrrit-wm>	 (03CR) 10Ottomata: [C: 032 V: 032] Removing disabled views [operations/puppet] - 10https://gerrit.wikimedia.org/r/103847 (owner: 10Ottomata)
[21:15:58] <grrrit-wm>	 (03CR) 10Ori.livneh: [C: 032] Use LOCK_EX for making the CDB MD5 file for sanity [operations/puppet] - 10https://gerrit.wikimedia.org/r/103846 (owner: 10Aaron Schulz)
[21:16:09] <grrrit-wm>	 (03PS1) 10Ottomata: Adding Hadoop Ganglia view [operations/puppet] - 10https://gerrit.wikimedia.org/r/103848 
[21:16:20] <grrrit-wm>	 (03PS2) 10Ottomata: Adding Hadoop Ganglia view [operations/puppet] - 10https://gerrit.wikimedia.org/r/103848 
[21:16:25] <grrrit-wm>	 (03CR) 10Ottomata: [C: 032 V: 032] Adding Hadoop Ganglia view [operations/puppet] - 10https://gerrit.wikimedia.org/r/103848 (owner: 10Ottomata)
[21:18:51] <logmsgbot>	 !log aaron finished scap
[21:19:07] <morebots>	 Logged the message, Master
[21:20:09] <grrrit-wm>	 (03PS1) 10Ottomata: Including hadoop view [operations/puppet] - 10https://gerrit.wikimedia.org/r/103850 
[21:20:27] <grrrit-wm>	 (03CR) 10Ottomata: [C: 032 V: 032] Including hadoop view [operations/puppet] - 10https://gerrit.wikimedia.org/r/103850 (owner: 10Ottomata)
[21:23:57] <grrrit-wm>	 (03CR) 10Chad: [C: 031] Revert "Configure Varnish not to cache scholarship app reqs" [operations/puppet] - 10https://gerrit.wikimedia.org/r/103768 (owner: 10BryanDavis)
[21:25:41] <cmjohnson1>	 ottomata: so an1012 is still broke...the 3rd mainboard came today and was damaged in shipping or packing so next week sometime now
[21:25:43] <grrrit-wm>	 (03PS1) 10Ottomata: Fixing hadoop stacked graphs. [operations/puppet] - 10https://gerrit.wikimedia.org/r/103851 
[21:25:52] <grrrit-wm>	 (03CR) 10Ottomata: [C: 032 V: 032] Fixing hadoop stacked graphs. [operations/puppet] - 10https://gerrit.wikimedia.org/r/103851 (owner: 10Ottomata)
[21:26:19] <ottomata>	 haha
[21:26:22] <ottomata>	 awesome, thanks cmjohnson1
[21:28:11] <cmjohnson1>	 it was unbelievable....a corner was completely smashed ...the box was fine so it was packed that way
[21:29:14] <ottomata>	 ha
[22:15:30] <grrrit-wm>	 (03PS1) 10Odder: Enable local TimedText on Portuguese Wikipedia [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/103861 
[22:20:04] <grrrit-wm>	 (03CR) 10Siebrand: "Should be merged once all Wikimedia sites run 1.23wmf9." [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/96472 (owner: 10Siebrand)
[22:46:31] <icinga-wm>	 PROBLEM - Puppet freshness on virt1007 is CRITICAL: Last successful Puppet run was Tue 24 Dec 2013 07:29:50 PM UTC  
[23:41:40] <grrrit-wm>	 (03PS1) 10Aaron Schulz: Sync out the wikiversions.cdb in scap-1 only [operations/puppet] - 10https://gerrit.wikimedia.org/r/103870 
[23:43:44] <grrrit-wm>	 (03PS5) 10BryanDavis: [WIP] Logstash puppet class [operations/puppet] - 10https://gerrit.wikimedia.org/r/100395 
[23:59:33] <logmsgbot>	 !log aaron started scap
[23:59:50] <morebots>	 Logged the message, Master