[00:04:33] PROBLEM - Puppet freshness on hume is CRITICAL: Puppet has not run in the last 10 hours [00:04:39] New patchset: Ori.livneh; "(Bug 37787) Uninstall CustomUserSignup extension" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/18201 [00:14:58] ori-l: sad to lose CustomUserSignup extension :( [00:15:27] aude: i presume / desperately hope that you are being sarcastic? [00:17:02] i have deployment rights (as of yesterday, I think) but fenari is still rejecting my key. is it simply a matter of waiting for a puppet run, or is there some additional steps that needs to be done? [00:17:13] *are there some [00:17:20] would be nice if someone would maintain the extension [00:18:02] aude, we (e3) are going to give the sign up interface some much-needed love [00:18:08] the current signup interface is horrible [00:18:17] good! [00:18:31] > puppet ran at Thu Aug 9 00:17:23 UTC 2012 [00:18:35] for fenari [00:18:50] (per nagios) [00:18:55] 09 00:18:30 < jeremyb> > puppet ran at Thu Aug 9 00:17:23 UTC 2012 [00:19:02] so, i think that's good enough [00:19:14] odd! any idea what else could be the cause? [00:19:32] pastebin a verbose ssh ? [00:19:42] maybe -vvv ? i can't remember what a good invocation is [00:19:42] jeremyb: sure [00:19:50] i think -vv is the max? [00:20:10] ration problems. Multiple -v options increase the verbosity. The maximum is 3. [00:21:02] * aude sees that kaldari made Fancycaptcha-createaccount slightly better [00:21:14] still ugly [00:22:06] ori-l: what about an audio captcha for accessibility purposes? [00:22:21] that's one of the other problems [00:22:39] don't we just send people to ACC? [00:23:37] jeremyb: we do but that's not great [00:24:02] * aude wants to turn my screen off and be able to make an account with screen reader and keyboard [00:24:33] aude: sure. but it's not like we just forgot about them entirely [00:24:59] jeremyb: true [00:27:03] jeremyb / aude: http://dpaste.com/783520/ [00:28:19] i don't think there's a whole lot to it other than "fenari is rejecting my key" :) [00:29:50] ori-l: what username ? [00:29:56] and is it the key in puppet ? [00:30:05] LeslieCarr: olivneh, and yes [00:30:37] ori-l: has it actually worked anywhere yet? [00:30:51] hrm, no home directory [00:31:06] i'm homeless! i can ssh into emery, for example [00:31:20] ori-l: to be safe let's also see `ssh-keygen -l -f ~/.ssh/khorsabad_rsa.pub` [00:31:38] LeslieCarr: same happened with krinkle account [00:31:55] LeslieCarr: pm'd [00:32:08] I'm being rejected by gerrit now [00:32:19] as of ~ 20 minutes [00:32:19] hah [00:32:30] gerrit just let me in [00:32:36] Krinkle: you mean 22 or 29418? [00:32:53] krinkle@gerrit.wikimedia.org:29418/ [00:33:06] e.g. git remote update / git review -R [00:33:37] Krinkle: wfm [00:34:28] mutante: btw, last week I could ssh into integration.mediawiki.org; but now I can't anymore. fenari also still doesn't work. [00:34:46] LeslieCarr: thanks for looking, btw [00:35:18] Krinkle: ugh, checking on integration [00:35:29] created the home dirs, let's see if keys works or not [00:36:39] ok, it seems gerrit ssh works again. weird. [00:36:44] LeslieCarr: nope [00:36:47] Krinkle: your home and key are still there [00:36:53] olivneh try now [00:37:09] mutante: and integration also works again. [00:37:12] Krinkle: please try again, watching auth.log now [00:37:15] Krinkle: try now [00:37:17] weird, no clue what's going on. [00:37:18] Krinkle: oh? ok... [00:37:18] LeslieCarr: still no :/ [00:37:20] both work again. [00:37:21] added authorized_keys manually to both [00:37:37] fenari still denied though [00:38:11] ori-l: try one last time on fenari [00:39:06] ori-l: run that other command? [00:39:06] 09 00:31:20 < jeremyb> ori-l: to be safe let's also see `ssh-keygen -l -f ~/.ssh/khorsabad_rsa.pub` [00:39:16] ok mutante has got you guys set :) [00:39:17] bye! [00:39:24] Krinkle: one more time on fenari? [00:39:25] LeslieCarr: works! [00:39:29] woot [00:39:30] confirmed, i see the login :) [00:39:44] mutante: still denied [00:40:22] heh, wall :) [00:40:32] Krinkle: now? [00:40:43] session opened for user krinkle [00:40:50] checking with -v to see if the key is offered properly [00:40:54] wow, whole bunch of debug output [00:40:57] yes, Im in now [00:41:09] alright [00:41:11] thx [00:41:23] -v is also great for when non-technical friends are looking over your shoulder and you want to appear very impressive [00:41:50] lol [00:42:03] I remember doing $ dir -r /s on MS-DOS at some point [00:42:09] in the rooot [00:42:13] takes an hour sometimes [00:42:21] ori-l: what about xxd ? [00:42:21] and say "I'm wiping your hard drive" [00:42:29] Krinkle: hah [00:42:34] even though all it does is list everything in every dir [00:42:54] Change merged: Kaldari; [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/18201 [00:43:32] mutante: thanks! where do you guys keep all the warez? [00:43:43] i can see no /appz or /gamez dir anywhere [00:44:32] okay, jokes about mission critical production systems unappreciated by operations people responsible for their upkeep. noted :P [00:44:41] anyhow, thanks for the help [00:45:22] ori-l: haha :) no problem. [00:45:36] * ori-l slinks out [00:45:49] ori-l: apt-cache search game [00:45:52] cya [01:26:35] PROBLEM - Puppet freshness on labstore1 is CRITICAL: Puppet has not run in the last 10 hours [01:39:30] PROBLEM - Puppet freshness on neon is CRITICAL: Puppet has not run in the last 10 hours [01:41:18] PROBLEM - MySQL Slave Delay on storage3 is CRITICAL: CRIT replication delay 206 seconds [01:41:36] PROBLEM - MySQL Slave Delay on db1025 is CRITICAL: CRIT replication delay 227 seconds [01:47:54] PROBLEM - Misc_Db_Lag on storage3 is CRITICAL: CHECK MySQL REPLICATION - lag - CRITICAL - Seconds_Behind_Master : 603s [01:54:39] RECOVERY - MySQL Slave Delay on db1025 is OK: OK replication delay 17 seconds [01:58:15] RECOVERY - Misc_Db_Lag on storage3 is OK: CHECK MySQL REPLICATION - lag - OK - Seconds_Behind_Master : 11s [01:58:33] RECOVERY - MySQL Slave Delay on storage3 is OK: OK replication delay 10 seconds [02:02:36] PROBLEM - Puppet freshness on ocg3 is CRITICAL: Puppet has not run in the last 10 hours [02:19:33] PROBLEM - Puppet freshness on labstore2 is CRITICAL: Puppet has not run in the last 10 hours [02:41:36] RECOVERY - Puppet freshness on labstore2 is OK: puppet ran at Thu Aug 9 02:41:13 UTC 2012 [02:42:38] PROBLEM - Puppet freshness on ms-be1003 is CRITICAL: Puppet has not run in the last 10 hours [02:57:30] PROBLEM - Puppet freshness on virt1001 is CRITICAL: Puppet has not run in the last 10 hours [03:10:05] RECOVERY - Puppet freshness on srv281 is OK: puppet ran at Thu Aug 9 03:09:45 UTC 2012 [03:12:20] RECOVERY - Router interfaces on cr1-sdtpa is OK: OK: host 208.80.152.196, interfaces up: 78, down: 0, dormant: 0, excluded: 0, unused: 0 [03:12:38] PROBLEM - Puppet freshness on virt1002 is CRITICAL: Puppet has not run in the last 10 hours [03:27:29] PROBLEM - Puppet freshness on virt1003 is CRITICAL: Puppet has not run in the last 10 hours [03:37:05] RECOVERY - Puppet freshness on virt0 is OK: puppet ran at Thu Aug 9 03:36:46 UTC 2012 [04:56:36] PROBLEM - Puppet freshness on ms-be1006 is CRITICAL: Puppet has not run in the last 10 hours [04:56:36] PROBLEM - Puppet freshness on ms-be1009 is CRITICAL: Puppet has not run in the last 10 hours [05:33:01] New patchset: Tim Starling; "Split PoolCounter off into a separate class" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/18207 [05:33:40] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/18207 [05:41:08] PROBLEM - Host search32 is DOWN: PING CRITICAL - Packet loss = 100% [05:43:33] RECOVERY - Host search32 is UP: PING OK - Packet loss = 0%, RTA = 0.49 ms [05:49:49] New patchset: Tim Starling; "Split PoolCounter off into a separate class" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/18207 [05:50:28] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/18207 [05:59:17] New patchset: Tim Starling; "Split PoolCounter off into a separate class" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/18207 [05:59:55] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/18207 [06:01:25] Change merged: Faidon; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/18207 [06:02:35] PROBLEM - Puppet freshness on singer is CRITICAL: Puppet has not run in the last 10 hours [06:04:03] New patchset: Faidon; "Add a missing import for poolcounter.pp" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/18208 [06:04:40] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/18208 [06:04:53] Change merged: Faidon; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/18208 [09:06:27] Change merged: Faidon; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/16654 [09:14:55] PROBLEM - Puppet freshness on spence is CRITICAL: Puppet has not run in the last 10 hours [09:25:25] RECOVERY - Puppet freshness on labstore1 is OK: puppet ran at Thu Aug 9 09:24:53 UTC 2012 [09:28:52] PROBLEM - Puppet freshness on ms-be1007 is CRITICAL: Puppet has not run in the last 10 hours [09:28:52] PROBLEM - Puppet freshness on ms-be1010 is CRITICAL: Puppet has not run in the last 10 hours [09:28:52] PROBLEM - Puppet freshness on ms-be1011 is CRITICAL: Puppet has not run in the last 10 hours [10:05:56] PROBLEM - Puppet freshness on hume is CRITICAL: Puppet has not run in the last 10 hours [11:18:04] PROBLEM - Host search32 is DOWN: PING CRITICAL - Packet loss = 100% [11:20:10] RECOVERY - Host search32 is UP: PING OK - Packet loss = 0%, RTA = 1.62 ms [11:40:53] PROBLEM - Puppet freshness on neon is CRITICAL: Puppet has not run in the last 10 hours [12:02:38] PROBLEM - Host search32 is DOWN: PING CRITICAL - Packet loss = 100% [12:03:50] PROBLEM - Puppet freshness on ocg3 is CRITICAL: Puppet has not run in the last 10 hours [12:05:56] RECOVERY - Host search32 is UP: PING OK - Packet loss = 0%, RTA = 0.26 ms [12:24:58] PROBLEM - Host search32 is DOWN: PING CRITICAL - Packet loss = 100% [12:43:52] PROBLEM - Puppet freshness on ms-be1003 is CRITICAL: Puppet has not run in the last 10 hours [12:58:52] PROBLEM - Puppet freshness on virt1001 is CRITICAL: Puppet has not run in the last 10 hours [13:13:52] PROBLEM - Puppet freshness on virt1002 is CRITICAL: Puppet has not run in the last 10 hours [13:28:53] PROBLEM - Puppet freshness on virt1003 is CRITICAL: Puppet has not run in the last 10 hours [14:30:13] !log cp1001 squid services going offline for troubleshooting [14:30:23] Logged the message, RobH [14:32:00] New patchset: Ottomata; "Preparing to install Precise on analytics1011-1022 eqiad row C dell machines using partman/analytics-dell.cfg" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/18235 [14:32:21] meeeeester RobH good morning to you! [14:32:48] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/18235 [14:34:25] PROBLEM - Frontend Squid HTTP on cp1001 is CRITICAL: Connection refused [14:35:19] PROBLEM - Backend Squid HTTP on cp1001 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:46:44] PROBLEM - Host cp1001 is DOWN: PING CRITICAL - Packet loss = 100% [14:47:02] ottomata: you want any assistance setting those things up? [14:47:13] yeeeeehaaaaw [14:47:16] yeah, so I just committed [14:47:20] I want a review on that [14:47:28] and then I will pxe boot and see how it goes [14:47:52] also, I'd love to keep this page up to date with my servers: [14:47:53] http://wikitech.wikimedia.org/view/Category:Servers [14:48:00] but um, I don't thikn I have a wikitech account [14:48:22] need review here: [14:48:22] https://gerrit.wikimedia.org/r/#/c/18235/ [14:49:18] you should have someone else review that. I have never set up a new row file. that siad, looks fine to me [14:49:30] oh, you should get a racktables account [14:49:38] that's where we actually keep track of servers [14:49:41] inventory [14:49:47] that page is soooooooo outdated [14:49:57] and we committed to not updating it by hand [14:50:19] RobH: can you review the above ^ [14:50:29] RECOVERY - Host cp1001 is UP: PING OK - Packet loss = 0%, RTA = 35.50 ms [14:51:32] i have one [14:51:35] oh really [14:51:35] ok [14:51:42] how does it get updated? [14:51:54] i wanted a place to write down the MACs [14:51:58] although…i guess they are in puppet now [14:52:15] \yeah, also, the more I look, that checkin is a-ok [14:52:17] I +1 [14:52:25] or, want me to +2? [14:52:43] i think it shoudl be ok, they are new servers so, it won't hurt anything if I did something all wrong [14:52:59] yeah [14:53:03] unless I accidentally told them to self destruct or something [14:53:08] Change merged: Pyoungmeister; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/18235 [14:53:20] ok, you shoudl merge it on sockpuppet now [14:53:28] !log cp1001 returned to service, resolving rt 3212 [14:53:38] Logged the message, RobH [14:53:52] hmm, how do I merge [14:53:54] cd /etc/puppet [14:53:55] git pull? [14:54:01] ottomata: as long as you didnt use the $self-destruct => true, we will be ok [14:54:05] RECOVERY - Backend Squid HTTP on cp1001 is OK: HTTP OK HTTP/1.0 200 OK - 27407 bytes in 0.143 seconds [14:54:17] ottomata: its on labs console, the directions to merge, lemme find [14:54:41] ottomata: https://labsconsole.wikimedia.org/wiki/Git#Public_repo [14:55:30] ok, trying that [14:55:37] ottomata: cd /root/puppet/ [14:55:57] git fetch && git diff HEAD origin/production [14:56:05] git merge origin/production [14:56:09] aye cool [14:56:19] * cmjohnson1 leaves for DC [14:56:26] cmjohnson1: travel safely! [14:56:34] cool, merged! [14:56:41] ok, going to try to pxe boot analytics1011 [14:56:45] oh wait [14:56:47] wait [14:56:50] root@stafford.pmtpa.wmnet's password: [14:56:52] force puppet run on brewster [14:56:57] oh right yeah [14:56:58] oh [14:57:00] brewster ok [14:57:02] uh [14:57:03] wiat [14:57:07] why brewster? [14:57:13] root@sockpuppet:~/puppet# git merge origin/production [14:57:13] Merge made by the 'recursive' strategy. [14:57:13] files/autoinstall/netboot.cfg | 1 + [14:57:13] files/autoinstall/subnets/analytics1-c-eqiad.cfg | 19 +++++++ [14:57:13] files/dhcpd/linux-host-entries.ttyS0-115200 | 61 ++++++++++++++++++++++ [14:57:14] 3 files changed, 81 insertions(+) [14:57:14] create mode 100644 files/autoinstall/subnets/analytics1-c-eqiad.cfg [14:57:15] root@stafford.pmtpa.wmnet's password: [14:57:25] your agent is not properly forwarded [14:57:29] bwaaaaa [14:57:32] it should be grrrr [14:57:44] it would seem that it's not ;) [14:57:48] aye [14:57:50] PROBLEM - Puppet freshness on ms-be1009 is CRITICAL: Puppet has not run in the last 10 hours [14:57:50] PROBLEM - Puppet freshness on ms-be1006 is CRITICAL: Puppet has not run in the last 10 hours [14:57:51] ok, will figure that out, hmm [14:57:56] kk [14:57:56] once I get that, can I just run merge again? [14:58:04] won't do anything [14:58:16] why is merge asking me for my pw? [14:58:33] because it has a hook to sync everything to stafford [14:58:41] that is where the puppet actually happens [14:58:56] hm k [14:58:57] like, sockpuppet only does cert signing [14:59:03] stafford handles all the traffic [14:59:20] ag, so I need the hook to fire [14:59:21] so merges happen on sockpuppet, pull happens on stafford [14:59:25] yeah [14:59:29] so, you can check in more code [14:59:35] hah, ok [14:59:35] or do it by hand on stafford [14:59:42] hmmm, might be cool to learn that [14:59:47] but I'm not 100% on what "it" is [14:59:51] I think just a git pull [15:00:08] not a git repo in /etc/puppet [15:00:10] on stafford [15:00:25] /var/lib/git [15:00:26] I believe [15:01:03] hah [15:01:07] # Changes not staged for commit: [15:01:07] # modified: files/misc/GeoIP.dat [15:01:07] /var/lib/git/operations [15:01:08] oh that's ok [15:01:10] that is a dl update [15:01:11] weird ok [15:01:17] ok git pulling [15:01:21] wait [15:01:28] too late [15:01:29] eh? [15:01:38] looks good, it pulled my stuff [15:01:46] ok [15:02:20] PROBLEM - check_all_memcacheds on spence is CRITICAL: MEMCACHED CRITICAL - Could not connect: 10.0.8.20:11000 (Connection timed out) [15:02:22] ok, now force puppet run on brewster [15:02:40] (our imaging box) [15:02:49] and that should get your stuff in place to pxe boot [15:02:53] k... [15:03:21] Change merged: Mark Bergsma; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/18096 [15:03:36] Change merged: Mark Bergsma; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/18099 [15:03:47] Change merged: Mark Bergsma; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/18101 [15:04:00] Change merged: Mark Bergsma; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/18102 [15:04:15] Change merged: Mark Bergsma; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/18103 [15:04:34] Change merged: Mark Bergsma; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/18105 [15:05:11] RECOVERY - check_all_memcacheds on spence is OK: MEMCACHED OK - All memcacheds are online [15:05:13] hm, uh oh [15:05:13] err: /Stage[main]/Misc::Install-server::Dhcp-server/Service[dhcp3-server]: Failed to call refresh: Could not start Service[dhcp3-server]: Execution of '/etc/init.d/dhcp3-server start' returned 1: at /var/lib/git/operations/puppet/manifests/misc/install-server.pp:234 [15:05:30] should I try to start it to see what the prob might be? [15:05:46] yes [15:06:19] hmm syntax errors in /etc/dhcp3/linux-host-entries.ttyS0-115200 and /etc/dhcp3/dhcpd.conf, although I don't think I edited dhcpd.conf [15:06:23] oh maybe that is the trace [15:06:24] yeah [15:06:42] ah got it [15:06:58] kk [15:07:18] ok, lemme see if I can +2 [15:07:21] New patchset: Ottomata; "Fixing syntax error in linux-host-entries.ttyS0-115200" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/18240 [15:07:32] nope [15:07:33] can't [15:08:02] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/18240 [15:08:36] ah, kk [15:08:50] Change merged: Pyoungmeister; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/18240 [15:09:34] ok [15:09:40] I did sockpuppet [15:09:43] force on brewster [15:10:01] oh ok [15:10:09] just did that [15:10:15] will let you know when done [15:10:18] haha [15:10:19] ok [15:10:38] ok, punch it [15:11:39] pxe boot? [15:11:43] yeah [15:13:02] ok its going, analytics1011 [15:13:07] cool [15:19:19] so i don't see it doing much in console [15:19:25] last was Checking NVRAM [15:20:18] ah poo and I just lost my console [15:20:23] i closed it and tried to open another one [15:20:40] woo [15:20:45] eh, just give it some time [15:20:47] It's serial so might be a little weird, usualy if it's blank there's nothing there or it's not redirected properly [15:20:54] oh [15:20:55] hit tab [15:21:05] anything? [15:21:13] maybe the word yes or the word no? [15:22:38] ergh, I closed my console though [15:22:41] can't get a new one [15:22:44] Info: SOL payload already active on another session [15:23:11] ah [15:23:15] yes.... [15:23:18] hurray dracs [15:23:23] just give it some time [15:23:25] ok... [15:23:29] how will I know though? [15:23:44] or ask RobH if there's a way to boot your old session [15:23:50] he may know wizardry [15:24:04] if its stuck [15:24:16] you can do the consoleclose command in the impi_mgmt script ;] [15:24:52] ottomata: ^ [15:25:19] ooo [15:25:20] cool [15:29:12] mark: does this look like what you wanted? https://gerrit.wikimedia.org/r/#/c/18194/ [15:30:21] er [15:30:27] why are you removing the existing file, it's still in use [15:35:17] ok, in console, no movement notpeter + RobH [15:36:13] So what exactly is happening now? You are attempting to install analytics1011 and attached to the serial console via the impi script? [15:37:34] mark: ah, correct... will ammend [15:39:30] yup [15:39:40] brewster is updated with the dhcp boot stuff [15:39:49] i did bootpxe, powercycle, console [15:42:57] New patchset: Pyoungmeister; "moving sudo definitions from sudo::applicationserver into the correct modules" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/18194 [15:43:36] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/18194 [15:51:11] still sitting there, notpeter, RobH [15:51:48] ottomata: you arent seeing a post at all? [15:52:09] naw its just sitting there [15:52:25] ok, disconnect and let me take a look [15:52:56] k done [15:53:38] ipmi_mgmt analytics1011.mgmt.eqiad.wmnet console [15:55:12] ottomata: thats what i just did, these take a minute to post [15:55:22] but i am going to attach a physical console to verify operation [15:55:25] k [15:57:40] cmjohnson1: think search32 has power supply problems? [15:57:44] kinda the only thing left.... [15:58:31] ottomata: didnt this work yesterday? [15:58:35] its not working for me either [15:58:52] we didn't get this far yesterday [15:59:02] this is the first bootpxe i've done [15:59:09] the console did work though, with bootbios [15:59:12] i did that today [15:59:22] and I did see output on console after bootpxe + powercycle [15:59:25] but only a bit [15:59:35] ottomata: but i mean now the serial redirection shows NOTHING [15:59:38] im not even seeing post now [15:59:52] hrmm, wait, now it is, wtf [16:00:04] cmjohnson1: cool cool [16:00:07] thanks! [16:01:11] ottomata: uhh, these are in the wrong file to start with [16:01:18] you have them in ttyS0, its S1 [16:01:21] its com port 2 [16:01:53] but there may be other issues, still checking [16:02:15] but all those entries you put in linux-host-entries.ttyS0-115200 need to be in linux-host-entries.ttyS1-115200 [16:02:18] as they are dells, not ciscos [16:02:26] (the 1001-1010 are in linux-host-entries.ttyS0-115200) [16:03:44] so, once it does hit the install server, it wont output the installer correctly [16:03:49] whaaaaa [16:03:49] PROBLEM - Puppet freshness on singer is CRITICAL: Puppet has not run in the last 10 hours [16:03:49] ok [16:03:51] ottomata: did you see it hit the dhcp server? [16:03:58] haha [16:04:01] ottomata: sorry I didn't catch that :( [16:04:11] nope, not sure how to look for dhcp server hits [16:04:14] well, it may not be the only problem [16:04:20] ok, lemme fix dell stuff [16:04:21] i see it trying to dhcp and not hitting [16:04:21] ok [16:06:09] New patchset: Ottomata; "analytics dells are on S1 (com port 2), not S0" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/18254 [16:06:40] ottomata: its not hitting brewster [16:06:47] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/18254 [16:06:55] so i think the network isnt setup right (either the port isnt in the right vlan, or dhcp relay isnt setup) [16:06:57] LeslieCarr: ^ [16:07:40] ah [16:07:50] ok, is that analytics-1-c ? [16:07:53] yep [16:08:06] and why'd you break it rob ? [16:08:37] my guess is petty spite. [16:09:12]